Subspace Geometry

PCA, principal angles, Procrustes fits, and relative representations explain how two nearly different-looking bases can still carry the same structure.

Representation papers often compare spaces rather than single neurons. A probe may work after projecting to the top PCA components; two models may have nearly orthogonal principal bases; an affine map may still align their examples. The geometry is easier to see as a moving point cloud than as a table of cosines.

The figures on this page are schematic: the angles and transfer scores illustrate the geometry, not measured results from a specific model.

Figure 1 · Drag or scrub the rotation and watch the principal angle open

rotation 54

1. PCA chooses a subspace, not a mechanism

PCA finds directions of high variance. In probe transfer work, the top-k subspace can be a useful compression before fitting an alignment map. That does not mean variance caused the feature. It means the feature was readable after a particular low-rank projection.

Principal angles summarize how two subspaces sit relative to one another. Small angles mean the planes overlap; angles near 90 degrees mean the bases can look almost orthogonal. But a bad-looking basis comparison is not the same as a failed example alignment.

Figure 2 · Procrustes rotation versus affine fit

alignment rotation-only

fit progress 0.55

2. Alignment maps answer a different question

Procrustes alignment rotates one centered cloud toward another while preserving distances. An affine fit can also stretch, shear, and translate. If Procrustes fails but affine succeeds, the result says the examples share a recoverable coordinate relationship, not that the two models used identical geometry.

Near-orthogonal bases can still transfer. A top PCA basis can point almost nowhere near another top PCA basis while the examples themselves remain alignable after an affine map. The important question is whether the selected subspace keeps task-relevant neighborhoods and directions.

Figure 3 · Subspace selection beats rank alone

basis angle 87

3. Relative representations use anchors

Absolute coordinates are fragile. Rotate every point and a coordinate probe has to relearn its weights. Distances or cosines to anchor examples can stay stable under the same rotation, giving a relative representation that survives changes of basis.

Figure 4 · Rotate coordinates; anchor signatures stay fixed

coordinate rotation 45

Related pages

Dependency Trees & Structural Probes for the tree-distance probe that often gets aligned across models.
Interchange Interventions & DAS for learned rotations used as causal interventions.
Residual Stream & Directions for the local "feature as direction" approximation.

What next

Application

Structural Probes

Where cross-model alignment pressure appears.

Method

Interchange Interventions

How trained low-rank slices become causal tests.