Information Geometry

Probability distributions as points on a Riemannian manifold.

A family of probability distributions $p(x; \theta)$ can be viewed as a manifold where each point corresponds to a distribution and the parameters $\theta$ are the coordinates. Information Geometry uses differential geometry to study the intrinsic shape of these manifolds.

The Fisher information metric measures distance between nearby distributions by how distinguishable they are from samples, independent of how they are parametrized. It is the expected outer product of the score, written in coordinates $\theta$ as:

$$ g_{ij}(\theta) = \mathbb{E}_{p_\theta}\!\left[ \frac{\partial \log p_\theta}{\partial \theta^i} \, \frac{\partial \log p_\theta}{\partial \theta^j} \right] $$

For a small parameter step, $\mathrm{KL}[p_\theta \Vert p_{\theta+\Delta\theta}] \approx \tfrac12 \Delta\theta^\top g(\theta)\,\Delta\theta$, so $g_{ij}$ measures how reliably a sample can tell the two distributions apart.

One sentence: Information geometry treats a statistical model as a curved surface whose metric is the Fisher information, so lengths and geodesics measure distinguishability rather than parameter differences.

1. The Probability Simplex and Geodesics

The simplest statistical manifold is the probability simplex: the set of all categorical distributions over $n$ outcomes. For $n=3$, this is a triangle where each point $(p_1, p_2, p_3)$ satisfies $p_i \ge 0$ and $\sum p_i = 1$.

Under the Fisher metric, the simplex is a portion of a hypersphere. The shortest paths (geodesics) between two distributions are great circles on this sphere.

Concretely, the map $p_i \mapsto 2\sqrt{p_i}$ sends each distribution onto the positive orthant of a sphere of radius 2. Distance along the manifold is then the great-circle distance, called the Fisher-Rao distance:

$$ d_{\mathrm{FR}}(p, q) = 2 \arccos\!\left( \sum_i \sqrt{p_i \, q_i} \right) $$

Figure 1 · Geodesics on the 3-outcome Simplex

Euclidean path (mixture) Fisher-Rao geodesic

Drag the two endpoints (A and B) on the triangle. Compare the "straight" mixture path to the curved information-theoretic geodesic.

A "straight line" in Euclidean space corresponds to a mixture: $(1-t)P + tQ$. A Fisher-Rao geodesic is the great-circle arc between $2\sqrt{p}$ and $2\sqrt{q}$ on the sphere, traversed at constant angular speed.

2. Dual Flatness: e-projections and m-projections

Exponential families (like the Gaussian, Poisson, or Bernoulli) have a special geometric structure called dual flatness. They can be described by two sets of coordinates: natural parameters ($\eta$) and expectation parameters ($\mu$).

Such a family has density

$$ p(x; \eta) = h(x) \, \exp\!\left( \eta^\top T(x) - A(\eta) \right), $$

and the log-partition function $A(\eta)$ ties the two coordinate systems together: its gradient carries natural parameters to expectation parameters, $\mu = \nabla A(\eta)$.

This duality gives us two types of "straight lines":

e-geodesics: Straight lines in natural parameters (log-linear mixtures).
m-geodesics: Straight lines in expectation parameters (standard mixtures).

Figure 2 · Dual Projections: e-projection vs. m-projection

m-projection (Forward KL / MLE) e-projection (Reverse KL / VI)

The chart shows the parameter space of a Bernoulli distribution. Drag the target point $P$ and the constraint line.

For two members of the family, KL divergence is the Bregman divergence generated by the log-partition function:

$$ D_{\mathrm{KL}}\!\left[\, p_{\eta_1} \,\Vert\, p_{\eta_2} \,\right] = A(\eta_2) - A(\eta_1) - \nabla A(\eta_1)^\top (\eta_2 - \eta_1) $$

Minimizing Forward KL divergence is an m-projection (finding the point in a sub-family that has the same average statistics). Minimizing Reverse KL divergence is an e-projection. MLE therefore projects along m-geodesics and VI along e-geodesics, and for the same target and constraint set the two land on different points (Figure 2).

3. The KL Landscape and its Tangent Quadratic

Using KL as height lifts the divergence into a third dimension. The surface exposes how the bowl steepens toward the boundary of the simplex, where the Fisher quadratic approximation peels away from the true KL, and how the two dual geodesics of §2 trace through the space.

Consider two independent Bernoulli variables with means $(\mu_1, \mu_2) \in (0,1)^2$. The KL divergence from a fixed anchor $\theta_0 = (\mu_{0,1}, \mu_{0,2})$ to a varying point $\theta = (\mu_1, \mu_2)$ is a sum over coordinates:

$$ D_{\mathrm{KL}}(\theta_0 \,\Vert\, \theta) = \sum_{i=1}^{2} \mu_{0,i} \log \frac{\mu_{0,i}}{\mu_i} + (1 - \mu_{0,i}) \log \frac{1 - \mu_{0,i}}{1 - \mu_i} $$

The Fisher quadratic, the second-order Taylor expansion of KL at the anchor, uses the inverse of the Fisher information metric:

$$ \hat D(\theta) = \tfrac{1}{2} \sum_{i} \frac{(\mu_i - \mu_{0,i})^2}{\mu_{0,i}(1 - \mu_{0,i})} $$

At the anchor, surface and paraboloid agree to second order. Move away and they diverge. The bowl is asymmetric: true KL grows much faster as $\theta$ approaches the boundary than as it moves toward the center.

Figure 3 · The KL Bowl, its Fisher Quadratic, and Two Geodesics

KL bowl (height = divergence) Fisher quadratic (wireframe) m-geodesic e-geodesic

Anchor θ₀ μ₀₁ 0.40 μ₀₂ 0.60

Endpoint θ₁ μ₁₁ 0.85 μ₁₂ 0.15

Layers Fisher quadratic m-geodesic e-geodesic

Drag the canvas to rotate. The bowl is capped at a finite height so its boundary behavior stays legible.

The two geodesics agree at the endpoints but otherwise trace different curves over the bowl. The m-geodesic ($\mu$ linear in $t$) is the chord of the base square. The e-geodesic ($\eta = \log(\mu / (1-\mu))$ linear in $t$) bows toward the interior, where the Fisher metric stretches less.

What next

Revisit the foundations.

Geometry

Uninformative Priors, Fisher Information & MLE

Review the Fisher information metric and its role in defining the geometry of statistical models.

Referenced by

Other notes on this site that link to this page.

The Exponential Family in the text Probability & Statistics
KL Divergence in the text Information Theory