Manifolds in Machine Learning

Basic Intuition

A manifold is a space that looks locally like ordinary Euclidean space. The essential slogan is \[\boxed{\text{A manifold is locally Euclidean.}}\] This means that, near each point, the manifold behaves like \(\mathbb{R}^d\) for some dimension \(d\).

Simple examples include:

• a line, which is a one-dimensional manifold;

• a circle \(\mathbb{S}^1 \subset \mathbb{R}^2\), which is also one-dimensional;

• the surface of a sphere \(\mathbb{S}^2 \subset \mathbb{R}^3\), which is two-dimensional.

A circle is curved as a whole, but a sufficiently small arc looks like a line. Similarly, the surface of the Earth is curved globally, but a small region looks approximately flat. This is the central intuition: manifolds may be globally curved, but locally they admit flat coordinates.

Ambient Dimension and Intrinsic Dimension

A manifold can be embedded in a larger Euclidean space. This is often written as \[\mathcal{M}\subset \mathbb{R}^D.\] In this notation:

• \(D\) is the ambient dimension;

• \(d\) is the intrinsic dimension of the manifold \(\mathcal{M}\).

For example, the circle satisfies \[\mathbb{S}^1 \subset \mathbb{R}^2.\] Its ambient dimension is \(D=2\), because it is represented in the plane. Its intrinsic dimension is \(d=1\), because one coordinate, the angle \(\theta\), is enough to describe a point on the circle: \[(x,y) = (\cos\theta,\sin\theta).\]

For the surface of a sphere, \[\mathbb{S}^2 \subset \mathbb{R}^3,\] the ambient dimension is \(D=3\), while the intrinsic dimension is \(d=2\). A point on the sphere can be described by two coordinates, such as latitude and longitude.

Why Manifolds Matter in Machine Learning

Machine learning data are often represented in very high-dimensional spaces. A \(100\times 100\) grayscale image, for instance, can be viewed as a vector in \[\mathbb{R}^{10000}.\] A hidden representation in a Transformer may live in \[\mathbb{R}^{768}, \qquad \mathbb{R}^{4096}, \qquad \text{or even higher-dimensional spaces.}\]

However, meaningful data typically do not fill the entire ambient space. Most random vectors in \(\mathbb{R}^{10000}\) do not look like natural images. Most random vectors in a high-dimensional hidden representation space do not correspond to meaningful language representations.

This motivates the manifold hypothesis: \[\boxed{\text{High-dimensional data often concentrate near a low-dimensional manifold.}}\] In other words, data may be stored in a large vector space, while the meaningful variations may be governed by far fewer degrees of freedom.

Example: Face Images

Consider \(64\times64\) grayscale face images. Each image has \[64\cdot64 = 4096\] pixels, so each image can be represented as a point in \(\mathbb{R}^{4096}\).

Realistic face images are not arbitrary vectors in this space. They vary according to structured factors such as identity, pose, lighting, facial expression, camera angle, and background. These factors may be numerous, but they are not \(4096\) independent degrees of freedom.

For this reason, the set of realistic face images can often be modeled as a lower-dimensional structure: \[\mathcal{M}_{\mathrm{faces}} \subset \mathbb{R}^{4096}.\] The ambient dimension is \(4096\), while the intrinsic dimension is expected to be much smaller.

This perspective helps explain the importance of representation learning, autoencoders, variational autoencoders, diffusion models, and manifold-learning methods. These approaches aim to uncover, exploit, or sample from hidden low-dimensional structure in high-dimensional data.

Local Coordinates and Charts

The main reason manifolds are mathematically manageable is that each small neighborhood can be described using ordinary coordinates.

Strictly speaking, a chart maps a local region of a manifold into \(\mathbb{R}^d\). Its inverse can be viewed as a local parameterization: \[\varphi : U \subset \mathbb{R}^d \longrightarrow \mathcal{M}.\] This means that a small patch of the manifold can be described using \(d\) real coordinates.

For the circle, a local parameterization is \[\varphi(\theta) = (\cos\theta,\sin\theta).\] For the sphere, a common parameterization is \[\varphi(\theta,\phi) = (\sin\theta\cos\phi,\, \sin\theta\sin\phi,\, \cos\theta).\]

The important point is that a manifold may be curved in \(\mathbb{R}^D\), but small neighborhoods can still be studied using coordinates in \(\mathbb{R}^d\).

What Does \(C^1\) Mean?

A \(C^1\) manifold is a manifold whose coordinate maps are continuously differentiable. Informally, it has no sharp corners or sudden breaks. A \(C^1\) manifold has a well-defined tangent space at every point.

At each point \(x\in\mathcal{M}\), the tangent space is denoted by \[T_x\mathcal{M}.\] For a circle, the tangent space at a point is the tangent line. For a sphere, the tangent space at a point is the tangent plane.

This notion is important in machine learning because many local tools rely on smooth local geometry: gradients, Jacobians, local PCA, nearest-neighbor dimension estimation, and local linear approximations.

Manifold Smoothness and Function Smoothness

In learning theory, it is useful to distinguish between two objects: \[\underbrace{\mathcal{M}}_{\text{where the data live}} \qquad\text{and}\qquad \underbrace{f:\mathcal{M}\to\mathbb{R}}_{\text{the target function to learn}}.\] These two objects have different notions of smoothness.

Why Approximation Rates Depend on \(d\) Rather Than \(D\)

Suppose that the data lie on a \(d\)-dimensional manifold \[\mathcal{M}\subset \mathbb{R}^D,\] and that the target function satisfies \[f\in C^\beta(\mathcal{M}).\] In many approximation-theoretic settings, the difficulty of approximating \(f\) is governed by the intrinsic dimension \(d\), not by the ambient dimension \(D\).

Typical approximation rates may take the form \[N^{-\beta/d},\] or, for squared error, \[N^{-2\beta/d}.\] Here \(N\) denotes a measure of model capacity. In sparse Mixture-of-Experts settings, this quantity may correspond to the active parameter budget \(N_{\mathrm{act}}\), leading to rates such as \[N_{\mathrm{act}}^{-2\beta/d}.\]

This distinction is important because \(D\) can be extremely large, while \(d\) may be much smaller.

Manifolds and Neural Networks

Neural networks are universal approximators, but approximation in high dimension can be difficult. If a function is defined on all of \(\mathbb{R}^D\), approximation rates may suffer from the curse of dimensionality: \[N^{-\beta/D}.\] If the function only needs to be learned on a manifold \(\mathcal{M}\) of dimension \(d\), the relevant rate may become \[N^{-\beta/d}.\] When \(d\ll D\), this can be a major improvement.

This explains why manifold assumptions are common in machine-learning theory. They provide a mathematical reason why learning in high-dimensional representation spaces can still be possible.

Manifolds and Representation Learning

A neural network maps an input through a sequence of representations: \[x \longmapsto h_1(x) \longmapsto h_2(x) \longmapsto \cdots \longmapsto h_L(x).\] Each layer transforms the geometry of the data. Ideally, these transformations make the representation simpler and more useful: classes become more separated, irrelevant variations are compressed, and meaningful factors become easier to extract.

From this perspective, representation learning can be interpreted as the process of reshaping the data manifold into a geometry that is easier for the final model layers to use.

This geometric view is related to the study of intrinsic dimension, hidden-state geometry, local PCA, clustering in embedding spaces, neural collapse, and manifold flattening.

Manifolds and Generative Models

Generative models are naturally connected to manifolds. A generator often starts from a latent vector \[z\in\mathbb{R}^d\] and maps it to a high-dimensional object \[G(z)\in\mathbb{R}^D.\] The image of the generator is a manifold-like set: \[\mathcal{M}= \{G(z): z\in\mathbb{R}^d\} \subset \mathbb{R}^D.\] Thus, a generator can be viewed as learning a parameterization of the data manifold.

This viewpoint is useful for understanding autoencoders, variational autoencoders, generative adversarial networks, and diffusion models. These models aim to learn or sample from the structured region of high-dimensional space where realistic data are concentrated.

Manifolds and Transformers

In Transformers, a token hidden state may live in a high-dimensional space: \[h_t \in \mathbb{R}^D.\] However, not every vector in \(\mathbb{R}^D\) corresponds to a meaningful token representation. Hidden states produced by a trained Transformer often occupy a structured region, which can be modeled as \[h_t \in \mathcal{M}\subset \mathbb{R}^D.\]

A Transformer layer can then be viewed as learning a function on, or near, this manifold: \[T:\mathcal{M}\to\mathbb{R}^D.\] If the relevant function is smooth along \(\mathcal{M}\), then approximation and generalization behavior may depend on the intrinsic dimension \(d\), rather than only on the ambient dimension \(D\).

Manifolds and Mixture-of-Experts Models

Mixture-of-Experts models have a natural geometric interpretation. Suppose the data manifold \(\mathcal{M}\) is complicated. Instead of using one dense feed-forward network everywhere, an MoE model can divide the manifold into regions and assign different experts to different regions.

A simplified expression is \[f(x) \approx \sum_{m=1}^M g_m(x)E_m(x),\] where:

• \(E_m\) is expert \(m\);

• \(g_m(x)\) is the router weight for expert \(m\);

• only a small number of experts are active for each input \(x\).

Sparse routing can be written as \[\left|\{m: g_m(x)\neq 0\}\right| \le k.\] The router selects the local experts that are most relevant for the current region of the manifold. In this sense, MoE models can be interpreted as collections of local approximators combined by a sparse routing mechanism.

Partition of Unity: A Geometric Bridge

A partition of unity is a classical tool from differential geometry. It makes it possible to build a global object by combining local objects.

Suppose the manifold is covered by local patches: \[\mathcal{M}\subseteq U_1\cup U_2\cup\cdots\cup U_M.\] On each patch \(U_m\), an expert \(E_m\) approximates the target function locally. Router weights \(g_m\) combine these local approximations: \[f(x) \approx \sum_{m=1}^M g_m(x)E_m(x).\] The weights satisfy \[g_m(x)\ge 0, \qquad \sum_{m=1}^M g_m(x)=1.\] If the partition is sparse, only a few weights are nonzero at each point: \[\left|\{m:g_m(x)\neq 0\}\right|\le k.\] This closely resembles sparse MoE routing from a geometric point of view.

A Mental Picture

The data manifold can be imagined as a curved road in a huge empty desert. The ambient space \(\mathbb{R}^D\) is the whole desert, while the data live only on or near the road.

A dense model uses one shared mechanism for the entire road. An MoE model places specialists along different parts of the road. The router decides which specialists should be used for each point.

The theory suggests that, when the road is smooth and the target function changes smoothly along it, the number of active specialists helps control approximation quality. The routing mechanism also introduces additional statistical complexity, because the model must choose among many possible experts.

Common Confusions

Takeaway for MoE Transformer Theory

In MoE Transformer theory, the manifold assumption can formalize the idea that token representations may have low-dimensional structure: \[\mathcal{M}\subset \mathbb{R}^D, \qquad \dim(\mathcal{M})=d, \qquad d\ll D.\] A common additional assumption is that the target function is smooth along this manifold: \[f\in C^\beta(\mathcal{M}).\] Under such assumptions, the approximation error may scale as \[N_{\mathrm{act}}^{-2\beta/d}.\] This connects geometry to scaling laws: \[\text{data geometry} \quad\Longrightarrow\quad \text{approximation rate} \quad\Longrightarrow\quad \text{scaling exponent}.\]