The Information Geometry of Representation Formation

Sanjukta Krishnagopal

cs.LGOct 25, 2026

The Information Geometry of Representation Formation

Authors: Sanjukta Krishnagopal

Organizations: Department of Computer Science, University of California, Santa Barbara

Abstract

Most analyses of neural representations focus on trained endpoints, leaving the process by which internal structure forms comparatively unmeasured. We instead treat representation formation itself as an object of study, asking whether the path a representation takes through training reveals information that its endpoint does not. We formalize this idea by viewing representation learning as a geometric trajectory, using Bures geometry to measure movement through representation space and quantum Fisher information to characterize local sensitivity to input factors. We find that learning concentrates local sensitivity on task-relevant factors, and that changing only the prediction target reorganizes this geometry around the corresponding visual factor. More strikingly, representation trajectories become systematically less direct as the learned computation requires higher-order interactions among input factors. This effect persists under matched-performance and training-duration controls and transfers from synthetic vector inputs to residual CNNs on rendered and natural images. Importantly, formation history contains predictive information about the stability of the learned representation: controlling for interaction order, path tortuosity predicts its subsequent drift under continued training and sensitivity to parameter perturbations, whereas endpoint displacement does not. Finally, representation-aware optimization lets us actively shape these trajectories, showing that formation geometry is not only measurable but actionable. Together, these results establish representation formation as a new source of information about learning: its geometry reveals properties of the computation being constructed, predicts subsequent representational stability, and provides a target for intervention during optimization.

Figures & tables

Figure 1: Local and path geometry of representation formation. Left: training produces a trajectory of covariance states in Bures representation space, whose directness is summarized by tortuosity. Right: QFI measures which controlled input-factor perturbations induce the largest local representation movement.

Figure 2: Local geometry tracks task relevance. (a) Learning concentrates QFI on the two relevant synthetic factors. (b) QFI captures task relevance beyond raw activation sensitivity, measured by the representation-Jacobian baseline, without using the labels required by the loss-gradient reference. (c) On dSprites, changing only the prediction target reorganizes QFI around the task-relevant visual factor. Error bars are 95% confidence intervals.

Figure 3: Interaction order shapes representation path complexity. (a) Checkpoint spacings; (b) fixed-output matched-accuracy tasks; (c) dSprites CNNs; and (d) composite CIFAR-10 tasks at approximately matched validation performance.

Figure 4: Metric intervention. (a) Validation-matched paired comparison. (b) RepNG remains more direct than Adam as the shared parameter subspace grows. Error bars are 95% confidence intervals.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

$k$	Test accuracy	$L_{B}$	$D_{B}$	$\tau$
1	$97.95\,[97.53,98.37]$	$3.831\,[3.618,4.044]$	$1.272\,[1.256,1.288]$	$3.011\,[2.843,3.179]$
2	$95.53\,[94.96,96.10]$	$6.041\,[5.699,6.382]$	$1.228\,[1.202,1.253]$	$4.929\,[4.594,5.263]$
3	$88.63\,[79.41,97.85]$	$8.525\,[7.103,9.946]$	$1.192\,[1.167,1.218]$	$7.132\,[6.033,8.230]$

Appendix

Table 1: Fixed-output dSprites residual-CNN results over ten paired seeds. Entries are means with 95% Student-

t

confidence intervals.

Seed	$k$	$L_{B}$	$D_{B}$	$\tau$
1	1	1.431	1.139	1.256
1	2	4.380	1.083	4.045
1	3	7.299	1.057	6.903
2	1	2.110	1.179	1.790
2	2	5.192	1.083	4.794
2	3	7.759	1.119	6.937

Appendix

Table 2: CIFAR-10 path geometry at the selected epoch-level checkpoints. Seeds are anonymized as 1–5.

Figure 6: Robustness. (a) QFI relevance is stable with sample size and increases across hidden layers. (b) The path-complexity ordering persists under three global distances. Error bars are 95% confidence intervals over ten seeds.

Layer	$\epsilon$	$n=200$	$n=500$	$n=1000$
1	$2\times 10^{-3}$	0.6257	0.6283	0.6285
2	$1\times 10^{-3}$	0.9229	0.9284	0.9323
2	$2\times 10^{-3}$	0.9229	0.9284	0.9323
2	$5\times 10^{-3}$	0.9229	0.9284	0.9323

Appendix

Table 3: Mean relevant-factor QFI mass over ten seeds. The first row reports hidden layer 1 at the default step; the remaining rows report hidden layer 2.

Transformation	Max logit error	$L_{B}$	$D_{B}$	$\tau$
Identity	0	2.729	1.008	2.708
Hidden-unit permutation	$<2\times 10^{-15}$	2.729	1.008	2.708
Common orthogonal rotation	$<6\times 10^{-15}$	2.729	1.008	2.708
Compensated per-unit rescaling	$<2\times 10^{-15}$	2.595	0.937	2.772

Appendix

Table 4: Function-preserving feature-coordinate audit over ten seeds. The same transformation is applied at every checkpoint;

L_{B}

D_{B}

, and

\tau

are seed means.

Spacing $s$	$k=1$	$k=2$	$k=3$	$k=4$
1	1.261	1.607	1.838	2.049
2	1.255	1.601	1.832	2.043
5	1.222	1.563	1.792	2.000
10	1.161	1.477	1.697	1.900
20	1.094	1.351	1.537	1.713

Appendix

Table 5: Mean Bures tortuosity over ten seeds after retaining every

s

th checkpoint.

Updates	$k=1$	$k=2$	$k=3$	$k=4$
20	1.178	1.711	1.804	1.970
40	1.209	1.940	1.956	2.133
60	1.225	2.294	2.284	2.597
100	1.245	2.575	2.708	3.428
200	1.263	2.791	3.069	4.590
300	1.270	2.878	3.152	5.142

Appendix

Table 6: Mean tortuosity at matched training budgets over ten paired seeds. Every interaction order receives exactly the listed number of updates.

		Stopped at 75% validation accuracy			Fixed 300 updates
$k$	Stop epoch	$L_{B}$	$D_{B}$	$\tau$	$L_{B}$	$D_{B}$	$\tau$
1	1.9	0.294	0.290	1.014	1.600	1.259	1.270
2	22.7	1.331	0.775	1.723	3.045	1.061	2.878
3	31.9	1.501	0.803	1.872	3.240	1.028	3.152
4	175.3	3.693	0.889	4.177	4.587	0.893	5.142

Appendix

Table 7: Decomposition of the accuracy-stopped and 300-update parity trajectories.

L_{B}

is path length and

D_{B}

is endpoint displacement.

Optimizer	Accuracy (%)	Steps	$L_{B}$	$D_{B}$	$\tau$
RepNG	71.43	65.77	1.207	0.636	1.895
Output NG	71.12	31.70	1.426	0.717	1.987
SGD	71.27	62.30	1.549	0.748	2.084
Adam	71.02	47.33	1.945	0.736	2.663

Appendix

Table 8: Optimizer means over 30 paired seeds at unit checkpoint spacing. Accuracy is test accuracy;

L_{B}

is path length and

D_{B}

is endpoint displacement.

Method	$L_{B}$	$D_{B}$	$\tau$
RepNG	1.679	0.674	2.506
Output NG	2.639	0.753	3.520

Appendix

Table 9: Twenty-seed fixed-budget control with identical maximum update norms.

Outcome	Std. $\beta_{\tau}$	Cluster $p$	Perm. $p$
Continued-training drift	0.87	$9.2\times 10^{-4}$	0.0015
Parameter-perturbation drift	0.51	$2.4\times 10^{-4}$	0.0010

Appendix

Table 10: Fresh 30-seed representational-stability sweep (120 task–seed runs). Clustered

p

-values use interaction-order fixed effects; permutation tests shuffle complete task profiles across seeds.

Neural Representation Geometry

Neural representation geometry studies how learned features are arranged and how that structure relates to model behavior and performance.

See all 256 papers

Oct 17, 2026

Representations Have Trajectories: Characterizing Temporal Representation Dynamics in Neural Networks

Recent work has shown that deep neural networks exhibit an implicit low-rank simplicity bias. In this paper, we investigate the questions that naturally arise from this observation. What are the temporal consequences of low-rank simplicity bias during optimization? How does representation geometry evolve throughout optimization rather than only at convergence? Do different neural architectures exhibit distinct geometric optimization behaviors? To answer these questions, we conduct a large-scale empirical study tracking representational geometry throughout training. We find that neural networks follow three recurring geometric trajectories: persistent high-rank, collapse-and-recovery, and persistent low-rank. We further identify Destructive Representation Collapse (DRC) as a pathological regime in which representations collapse to rank-one and fail to recover, leading to a sustained loss of representational capacity. While collapse-and-recovery is consistently associated with successful optimization in controlled settings, experiments on real-world models show that the relationship between geometric trajectories and optimization outcomes is configuration dependent. Overall, our findings suggest that representation geometry evolves dynamically during training and provide a new perspective on how deep neural networks learn.

Shivaprasad B. Gowda, Sparsh Mittal

Indian Institute of Information Technology, Nagpur · Indian Institute of Technology, Roorkee

Low-Rank Representation Learning Neural Representation Geometry Neural Network Training Dynamics Representation Learning Neural Network Optimization Representation Collapse

Ask PDF

Oct 11, 2026

Matched Specialization, Different Adaptation: Second-Moment Structure in Shared Representations

Small Adam-trained shared-representation networks can reach equivalent selected mean readout measurements yet respond differently to the same further learning. We use state interventions to separate second-moment scale and coordinate-pattern effects, then examine their formation and time dependence. In the central two-target experiment, the more precisely readable target switches within ten updates in 67.5% of acquisition-formed checkpoints and 0% of update-formed checkpoints. First-moment exchange leaves the ten-update contrast change within its pre-specified margin, whereas second-moment exchange reverses the contrast. A coordinate-pattern effect remains at common mean adaptive scale. Layer and movement controls locate an effect beyond overall update size. Formation records connect small encoder second moments with stronger cancellation between task gradients. A prospective gradient-projection experiment changes this retained state; transferring the resulting second moment alone reduces the ten-update adaptation response in a fixed recipient. The matched comparison extends to three Gaussian targets, while a room-sensor extension retains component effects but fails endpoint matching. Across tested regimes, component effects survive more widely than reversal or reset equivalence. Effect magnitude and duration depend on learning conditions.

Toru Takahashi

Human Informatics and Systems Lab, Doshisha University · Linked Open Data Initiative, NPO · Keio Research Institute at SFC +1

Multi-Task Learning

Ask PDF

May 6, 2026

Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior

Neural representations carry rich geometric structure; but does that structure causally shape behavior? To address this question, we intervene along paths through activation space defined by different geometries, and measure the behavioral trajectories they induce. In particular, we test whether interventions that respect the geometry of activation space will yield behaviors close to those the model exhibits naturally. Concretely, we first fit an activation manifold

M_h

to representations and a behavior manifold

M_y

to output probability distributions. We then test the link

M_h \leftrightarrow M_y

via interventions: we find that steering along

M_h

, which we term manifold steering, yields behavioral trajectories that follow

M_y

, while linear steering -- which assumes a Euclidean geometry -- cuts through off-manifold regions and hence produces unnatural outputs. Moreover, optimizing interventions in activation space to produce paths along

M_y

recovers activation trajectories that trace the curvature of

M_h

. We demonstrate this bidirectional relationship between the geometry of representation and behavior across tasks and modalities. In language models, we use reasoning tasks with cyclic and sequential geometries as well as in-context learning tasks with more complex graph geometries. In a video world model, we use a task with geometry corresponding to physical dynamics. Overall, our work shows that geometry in neural representation is not merely incidental, but is in fact the proper object for enabling principled control via intervention on internals. This recasts the core problem of steering from finding the right direction to finding the right geometry.

Daniel Wurgaft, Can Rager, Matthew Kowal +13

Stanford University · University College London · Northeastern University +2

Neural Representation Geometry Manifold Learning Causal Interventions in Language Models Activation Steering

Ask PDF

Abstract

Neural Representation Geometry

Explore similar work

Representations Have Trajectories: Characterizing Temporal Representation Dynamics in Neural Networks

Matched Specialization, Different Adaptation: Second-Moment Structure in Shared Representations

Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior