Traversing the solution space of neural networks with Hessian Null Space Continuation

Ann Huang; Mitchell Ostrow; Zhouyang Lu; William T. Redman; Leo Kozachkov; Kanaka Rajan

cs.LGSep 29, 2026

Traversing the solution space of neural networks with Hessian Null Space Continuation

Authors: Ann Huang, Mitchell Ostrow, Zhouyang Lu, William T. Redman, Leo Kozachkov, Kanaka Rajan

Organizations: Harvard University · Harvard Medical School · Kempner Institute · Massachusetts Institute of Technology ·

Abstract

On a single task, deep networks can learn many solutions, depending on their optimizer, training data, architecture, and hyperparameters. Many of these solutions are mode-connected: rather than isolated points in weight space, they are connected by low-loss regions. Yet how their internal computation varies within these regions is unknown. A parallel line of work has identified the degeneracy of neural representations: many networks reach similar training loss with distinct internal structures. However, it is unclear how these solutions are related in weight space. We unify these subfields and show for the first time that many different internal mechanisms exist within a local mode-connected region in weight space. To do so, we introduce Hessian Null Space Continuation (HNC), a scalable method that uses local curvature to traverse regions of weight space that preserve network function, and can be steered toward solutions with specified properties. In RNNs trained on a memory task, HNC reaches drastically different representations and dynamics with maintained behavior. In ImageNet-trained Vision Transformers, HNC finds representations that differ more from the original network than any independently trained model with a different architecture or objective. In reinforcement-learning agents, HNC uncovers a distinct navigation strategy at comparable return and exposes reward hacking in an AI Safety Gridworld. Finally, HNC measures the local geometry of the solution set, showing how model size and task complexity shape its dimension and functional sensitivity. Our results show that a surprisingly large amount of representational diversity exists near a single trained solution, unseen by standard gradient-based optimization. HNC identifies and quantifies this diversity, opening new possibilities for mechanistic understanding of solution spaces and for model merging, editing, and fine-tuning.

Figures & tables

Figure 1: Hessian Null Space Continuation (HNC) overview. (A) Around a trained network ${\bm{\theta}}_{0}$ (the anchor), the function-matching loss Eq. 1 is sharp along a few directions and flat along many. HNC follows the flat directions to an alternative network ${\bm{\theta}}_{T}$ within the connected low-loss region. (B) The two networks share the same input-output mapping, but their hidden-state representations of the same inputs can differ. (C) The Hessian spectrum at ${\bm{\theta}}_{t}$ separates a few sharp directions from a high-dimensional null space with eigenvalues below a threshold $\epsilon$ . (D) Each iteration takes a step within the null space (flat step), then re-minimizes the function-matching loss (restore), yielding ${\bm{\theta}}_{t+1}$ . (E) The flat step can be steered to optimize a differentiable objective $\varphi$ by following its gradient projected onto the null space.

Figure 2: HNC drives RNNs to learn drastically different dynamics in a memory task. (A) The 3-bit flip-flop task and the anchor’s solution. Left: one input channel and the corresponding output, where the output holds the last nonzero input. Right: hidden-state trajectories of the trained anchor projected onto their top three principal components, colored by the output state, with the eight stable fixed points (black diamonds) at the vertices of a cube. (B) Undirected, CKA-steered, and DSA-steered HNC walks from the same anchor. Left: task loss plotted against the walk step. Middle and right: representational ( $1-\mathrm{CKA}$ ) and dynamical (DSA) distance to the anchor against relative weight movement. (C) Hidden-state trajectory projected onto their top three principles components, and output traces from networks reached during the HNC walks. (D) Left and middle: hidden states of the undirected endpoint during input-free memory periods, projected onto the top two PCs of the readout and readout-null subspaces, colored by output state as in (A). Right: per-step hidden-state movement in the readout (dotted) and readout-null (solid) subspaces. (E) Fraction of output retained over increasing steps without input (left) and recovered after hidden-state perturbations (right). Thick lines: networks shown in (C); thin lines: endpoints of other independent HNC walks and the other trained anchors. (F) MDS embeddings of 131 networks (the anchor, ten independently trained seeds, and all checkpoints from three HNC walks of each type), using pairwise representational distance ( $1-\mathrm{CKA}$ , left) or relative weight distance (right). Marker size and opacity increase along each walk.

Figure 3: HNC reveals representational freedom underexplored by gradient descent in ViTs. (A) Max-over-layers CKA and $k$ -nearest-neighbor (kNN) to the anchor along the HNC walk, both on images it was steered on (the probe set) and on images never seen during HNC (the held-out set). (B) Left: agreement between the anchor and HNC endpoint’s predictions on Places-365 images. Right: top-1 to top-5 ImageNet accuracy. (C) Max-over-layers CKA to the anchor, between our HNC endpoint, an untrained ViT, and seven trained models. In (A-C), lines show the mean and shaded bands indicate $\pm 1$ s.d. across five random probe sets. (D) Pairwise cosine similarities between penultimate-layer representations of probe set images, for the anchor and one HNC endpoint. (E) Multidimensional scaling (MDS) embedding of the similarities in (D) for 20 images from 4 classes. (F) For both the anchor and HNC endpoint, we take the five image pairs represented as most similar, and score each pair’s similarity on three attributes: semantic content (DINOv2 feature similarity), color, and spatial layout. Scores are percentiles among all pairs of probe images, where higher means the network’s closest pairs share that attribute more than chance. Dashed line is the level of a random pair.

Figure 4: HNC finds behaviorally distinct, high-reward policies in RL. (A) Plume Tracking task where the agents navigate to the source of an odor plume, which travels away from the source under the dynamics of the wind in the environment. Anchor and alternative policy’s navigation trajectory are shown for three trials. (B) Action divergence and return plotted against relative weight change during HNC (dotted vertical line: the policy that we analyze in (A) and (C)). (C) Number of casts, defined as the number of times the agent sweeps across the wind and reverses its heading direction, and mean distance from the plume centerline for 240 shared initial conditions. (D) Boat race task. The proxy reward gives $+3$ for entering an arrow tile clockwise; the true return measures net clockwise progress around the track but is hidden to the agent during standard training. The anchor follows the track as intended, while HNC exposes a reward-hacking policy that steps on and off a single arrow tile, collecting the proxy reward without making clockwise progress. (E) Action divergence, proxy return, and true return plotted against relative weight change during HNC for ten seeds.

Figure 5: Model size and task difficulty shape local flatness and curvature. (A) Effective null fraction as a function of model width and task difficulty (number of image classes). (B) Normalized curvature along HNC-stepped directions. Curvature is summarized by the median over steps within each walk. Shading indicates standard error across 3 seeds.

Appendix figures & tables46 assets

Supplementary material from the paper’s appendix.

Appendix

	Dense eigendecomposition	Matrix-free LOBPCG
Time	$\mathcal{O}(P^{3})$	$\mathcal{O}(kP)$
Memory	$\mathcal{O}(P^{2})$	$\mathcal{O}(kP)$
Memory, 3BFF RNN ( $P=4611$ )	$85$ MB	$1.2$ MB ( $k=64$ )
Time, 3BFF RNN	$10$ s	$39$ s ( $k=64$ )
Memory, ViT-S ( $P=22$ M)	$1.9$ PB	$5.6$ GB ( $k=64$ )
Time, ViT-S	$2\times 10^{7}$ products to assemble ( $\sim$ months of GPU time), factorization infeasible	$\approx 1$ s per product; one continuation step $\approx 1$ min

Appendix

Table 1: Cost of a dense eigendecomposition of the Hessian against the matrix-free LOBPCG route used by HNC.

P

is the number of parameters and

k

the block size. RNN timings are for the 3BFF anchor of Section 3 on one CPU node with

8

threads; ViT-S timings are from the walks of Section 4 on one NVIDIA H200 with 512 probe images.

Figure 6: Ablation experiment on the function-restoring step. Undirected null-space walks at three null-space step sizes

\eta

, with

m\in\{0,3,6,9,12\}

restore steps. (A) The task loss against the normalized weight travel during HNC walks, where the dotted line is the anchor’s loss. (B) HNC endpoint’s task loss relative to the anchor’s against

m

. Larger null-space steps drift faster, and the restore step corrects for part of the drift rather than eliminating it.

Figure 7: Ablation experiment on the null-space projection of the steering gradient. CKA-steered walks from the same 3BFF anchor network at varying magnitudes of damping. Here, the maximum loss threshold is removed so that we can compare the task loss reached during HNC under different damping. A larger damping factor allows the steering gradient along sharper directions to pass through, therefore leading to faster representational divergence at the price of higher task loss. In contrast, with a highly strict damping factor (

\mu_{\mathrm{rel}}=10^{-9}

), neither the task loss nor the representation changes at all.

Figure 8: Ablation experiment on the relinearization step. (A) Constraint task loss against the normalized weight travel; dotted lines mark the anchor’s loss and the walk’s usual ceiling. (B) Endpoint task loss relative to the anchor’s against different relinearization interval. Open markers mark walks whose loss blew up.

Figure 9: HNC against training with a similarity penalty. Left: the highest task loss reached during 600 steps against the divergence from the anchor,

\Delta\mathrm{CKA}=1-\mathrm{CKA}

, for HNC at four flatness thresholds

\mu_{rel}

and for penalized training at three penalty weights

\beta

(five seeds each; markers show the median, bars the range). Dotted lines mark the anchor’s loss and the loss ceiling. Middle, right: task loss and divergence over the same 600 steps for one HNC walk (

\mu_{\mathrm{rel}}=10^{-5}

) and one penalized run (

\beta=1

) that end at similar divergence. At every level of task loss, HNC reaches more divergence. The penalized run crosses the loss ceiling within its first twenty steps, gains nearly all of its divergence during that excursion, and only then returns toward low loss, whereas HNC never leaves the low-loss region.

Figure 10: Task-loss and function-matching walks along the worst-case flat direction on 3BFF. A, B : task loss relative to its anchor value, and function-matching loss to the outputs of the anchor, over

300

null-space steps from four anchors (median, band from minimum to maximum). Solid lines follow the flat direction along which the task loss rises fastest; the paler dashed lines follow the CKA-steered heading and a random flat direction from one anchor. (C) : change of the task loss per step along the task-loss walk, split into the first-order term set by the gradient and the second-order term set by the curvature. (D) : number of negative Hessian eigenvalues at each anchor for the two losses. The task-loss walk drifts because the gradient, not the curvature, moves the loss, whereas the function-matching walk holds both losses in place.

Figure 11: Constraint strength of three preserved quantities on ViT-S. CKA-steered walks from one anchor that hold fixed, on a probe of

512

labeled ImageNet validation images, the full logit vector, the per-image cross-entropy, or the mean cross-entropy (median over probe seeds, band from minimum to maximum). (A) : representational divergence from the anchor along the walk. (B) : fraction of images that keep the top-1 prediction of the anchor, on the probe images (solid) and on

5{,}000

held-out images (dotted). (C) : held-out top-1 accuracy. (D) : probe prediction match against divergence, so the walks are compared at matched representational change rather than at matched step. The fewer constraints a quantity imposes per image, the more predictions change, on the probe and beyond it.

		Pred. match (probe) with $\Delta\mathrm{CKA}=$
Preserved quantity	# constraints	$0.2$	$0.4$	$0.6$
Full logits	$512{,}000$	$0.996$	$0.994$	$0.993$
Per-image cross-entropy	$512$	$0.916$	$0.894$	$0.875$
Mean cross-entropy	$1$	$0.831$	$0.811$	$0.793$

Appendix

Table 2: Preserved quantities on ViT-S, ordered by the number of constraints they impose on a probe of

512

labeled ImageNet validation images over

1{,}000

classes. Prediction match is the fraction of probe images given the top-1 prediction of the anchor, at three values of representational divergence (max over layers, mean and standard deviation over probe seeds).

Figure 12: Alternating retraining with the DSA-steered HNC walk yields a chain of dynamically distinct 3BFF solutions at preserved function. Starting from the anchor $T_{0}$ , each round first runs a DSA-steered walk from the current trained network ( $H_{k}$ ), then trains a fresh network with a penalty on dynamical similarity to every network found so far ( $T_{k+1}$ ). (A) Held-out task loss along the chain (purple: retraining; orange: HNC walk; dotted line: loss ceiling). Every trained network reaches the anchor’s accuracy and every walk stays well below the ceiling. (B) Weight distance between consecutive networks, in units of the anchor’s weight norm; the grey band is the range of distances between independently trained seeds. Retraining lands as far away as an unrelated seed, whereas a walk moves less than one anchor norm. (C) Pairwise DSA distance between the eight networks. (D) Two-dimensional embedding of the same distances; the star is the anchor and the arrows follow the chain. (E) For each new network, its DSA distance to the nearest and farthest earlier network, against the range between independently trained seeds (grey band). Every network in the chain lies farther from all its predecessors than two seeds lie from each other.

Figure 13: The chain visits qualitatively different dynamical mechanisms. Hidden-state trajectories of every network in the chain on the same trials, projected onto each network’s top three principal components and coloured by memory state, with stable fixed points as black diamonds. Top row: trained networks; bottom row: the HNC endpoints reached from the network above; grey arrows give the order of the chain. The anchor stores each memory state at its own fixed point. The first walk keeps only a few fixed points and spreads the activity into a cloud, the two retrained networks that follow hold the memory with no stable fixed point at all, and the last round returns to a fixed-point solution with more fixed points than memory states. All eight networks solve the task with bit accuracy above

0.999

Figure 14: Eigenvalue spectra of the recurrent dynamics along the chain. Eigenvalues of a linear model fitted to each network’s hidden-state trajectories, with the unit circle for reference; the distance between these spectra is the quantity that both the retraining penalty and the steered walk push apart. The anchor’s eigenvalues fan out from the real axis. The networks without fixed points concentrate their eigenvalues at

+1

and

-1

, a slowly decaying mode and a mode that alternates every step, and the third walk endpoint adds a pair of modes that repeat every four steps.

Figure 15: Hessian spectrum of the trained 3BFF RNNs. (A) All

4611

eigenvalues at the anchor, sorted. (B) The same eigenvalues divided by the largest one. (C) Null fraction, the fraction of parameter directions with

\lambda_{i}<\mu_{\mathrm{rel}}\lambda_{\max}

, as the threshold varies. Curves and bands are the median and range over five independently trained anchors. Dotted lines mark the threshold of the undirected walks,

\mu_{\mathrm{rel}}=10^{-3}

$\mu_{\mathrm{rel}}$	absolute flatness threshold	median dimension	fraction of $P$	$\max\Delta\mathcal{L}$
$1.0\times 10^{-7}$	$1.1\times 10^{-6}$	1701	36.9%	$5.4\times 10^{-7}$
$1.0\times 10^{-6}$	$1.1\times 10^{-5}$	2896	62.8%	$5.4\times 10^{-6}$
$1.0\times 10^{-5}$	$1.1\times 10^{-4}$	3684	79.9%	$5.4\times 10^{-5}$
$1.0\times 10^{-4}$	$1.1\times 10^{-3}$	4145	89.9%	$5.4\times 10^{-4}$
$1.0\times 10^{-3}$	$1.1\times 10^{-2}$	4418	95.8%	$5.4\times 10^{-3}$
$1.0\times 10^{-2}$	$1.1\times 10^{-1}$	4565	99.0%	$5.4\times 10^{-2}$

Appendix

Table 3: Dimension of the flat subspace of the 3BFF RNN (

P=4611

) as the relative flatness threshold

\mu_{\mathrm{rel}}

varies. A direction is counted as flat when its eigenvalue satisfies

\lambda<\mu_{\mathrm{rel}}\lambda_{1}

, where

\lambda_{1}

is the top eigenvalue of the Hessian (median

10.8

across seeds).

Hyperparameter	Value
Flat directions	explicit basis of eigenvectors with $\lambda<10^{-3}\lambda_{1}$
Step direction	one random heading direction uniformly sampled from $\mathcal{V}_{0}$ , held fixed and re-projected onto the current basis at every relinearization
Restore steps	$12$ per null-space step
Null-space step size $\eta$	starts at $1.0$ , halved on rejection and grown $1.1\times$ on acceptance of a step; lower bounded by $10^{-4}$ and upper bounded by $4.0$
Loss ceiling	$10^{-2}$
Relinearization	every $20$ accepted steps or on rejection, at most $60$ times

Appendix

Table 4: Settings of the undirected walks in Fig. 2 .

Hyperparameter	Value
Flat directions	soft projection with damping $\mu=10^{-7}\lambda_{1}$
Step direction	gradient of the steering objective, after a seeded random first step
Steering objective	$1-\mathrm{CKA}$ between the hidden states of the current network and the anchor, on a probe of $32$ trials
Solver	plain CG, $200$ iterations, started from zero at every step (Appendix A.3 )
Restore steps	$12$ per null-space step
Null-space step size $\eta$	starts at $0.03$ ; halved on rejection down to $5\times 10^{-4}$

Appendix

Table 5: Settings of the CKA-steered walks in Fig. 2 .

Hyperparameter	Value
Flat directions	soft projection with damping $\mu=10^{-7}\lambda_{1}$
Step direction	gradient of the steering objective, after a seeded random first step
Steering objective	DSA distance between the hidden-state dynamics of the current network and the anchor ( $5$ delays, rank $10$ ), on a probe of $32$ trials
Solver	plain CG, $200$ iterations, started from zero at every step (Appendix A.3 )
Restore steps	$12$ per null-space step
Null-space step size $\eta$	starts at $0.03$ ; halved on rejection down to $5\times 10^{-4}$

Appendix

Table 6: Settings of the DSA-steered walks in Fig. 2 .

Figure 16: A set of diverse 3BFF solutions produced by the function-preserving null-space walks. Hidden-state trajectories on the probe trials, projected onto each snapshot’s own top three activity PCs and coloured by the current memory state, with the snapshot’s stable zero-input fixed points as black diamonds.

Figure 17: MDS embeddings of the solutions reachable from ten anchors, by representation and by weights. Metric MDS of 470 networks: ten independently trained anchors (hollow squares), four checkpoints from each of their undirected, CKA-steered, and DSA-steered walks, and forty further independently trained networks (grey). Walks steered away from an earlier endpoint rather than from the anchor share the colour of the metric they steer. Marker size and opacity grow along each walk. Left: representational distance ( $1-\mathrm{CKA}$ ). Right: relative weight distance. In weights every anchor forms its own cluster with its walks around it. In representation all anchors and trained networks collapse onto one point, and the walks from every anchor spread far beyond it.

Figure 18: The prune-steered walk makes the network hard-prunable at preserved function. (A) Fraction of removable units (outgoing group norm below the dead threshold) against the normalized weight movement, the walk arclength divided by the anchor weight norm. (B) Hard-truncation test: the RNN is physically rebuilt keeping only the units with the largest outgoing group norms; task MSE against the removed fraction, for the anchor and the prune-steered endpoint. The dotted line marks the largest removed fraction at which the pruned network is still bit-perfect (

53/64=0.83

11

units kept). (C) Bit accuracy of the same truncations: the pruned endpoint stays at

100\%

down to

11

units, while the anchor is already imperfect at the first truncation and degrades from there.

Figure 19: Resumed SGD maintains the steered divergence while finding a lower-loss solution. Plain SGD resumed from each steered 3BFF endpoint (and from the anchor and a random initialization), tracking the task loss and the quantities the steering moved: representation (CKA distance to the anchor), dynamics (DSA distance to the anchor), and structure (fraction of near-zero

W_{hh}

columns).

Figure 20: SGD stays local. (A) How far the weights move during the resumed training, from each start. (B) Distance of each steered solution to the anchor before (light) and after (solid) retraining.

Figure 21: Out-of-distribution input probes of the walk endpoints. Accuracy of the anchor and the three walk endpoints when the input statistics differ from training. The dotted line marks the trained value. (A) All input pulses scaled by a common magnitude. (B) Pulse probability per step. Rare pulses leave long stretches without input within a trial.

Figure 22: Steering the mutual $k$ NN score finds alternative representations at preserved input-output mapping. (A) The max-over-layers mutual $k$ NN similarity to the anchor falls along the walk, both on the probe images and on held-out images never seen during HNC. The max-over-layers CKA falls as well, even though the walk does not steer it. (B) The top- $k$ predictions on Places-365 images and the top- $k$ accuracy on ImageNet are largely preserved along the walk. (C) At the endpoint, the max-layer mutual $k$ NN to the anchor sits below every independently trained model and above only an untrained initialization. (D) MDS embedding of 20 images from four classes, for the anchor and the endpoint. (E) Nearest-neighbor graph of the same images. Grey edges mark neighbors shared by the anchor and the endpoint; blue edges mark neighbors found in only one of the two networks. (F) The most similar image pairs of the endpoint match those of the anchor in semantic content (DINOv2) and color, but their spatial-layout similarity falls toward chance.

Hyperparameter	Value
Anchor	ImageNet-trained ViT-S/16, $22$ M parameters
Probe set	$512$ Places-365 images
Flat directions	soft projection with damping $\mu=10^{-7}\lambda_{1}$
Solver	preconditioned conjugate gradient, $40$ iterations
Steering objective	$\varphi_{\mathrm{CKA}}$ of Eq. 12 , $\beta=30$
First step	random direction, soft-projected

Appendix

Table 7: Settings of the CKA-steered ViT-S walk in Fig. 3 .

Hyperparameter	Value
Anchor	ImageNet-trained ViT-S/16, $22$ M parameters
Probe set	$512$ Places-365 images
Flat directions	soft projection with damping $\mu=10^{-7}\lambda_{1}$
Solver	preconditioned conjugate gradient, $40$ iterations
Steering objective	$\varphi_{\mathrm{kNN}}$ of Eq. 13 , $\beta=30$ , $k=10$
First step	random direction, soft-projected

Appendix

Table 8: Settings of the mutual-

k

NN-steered ViT-S walk in Figure 22 .

Figure 23: Accuracy, prediction match and error consistency of the CKA-steered endpoints on four held-out test sets. (A) Top-

k

accuracy of the anchor, the walk endpoints, a diff-seed network (the anchor’s final fine-tuning stage repeated with a different seed) and an independently trained ViT-S/16 (ViT-S/in1k) on clean ImageNet validation images and on ImageNet-V2, ImageNet-Rendition and ImageNet-Sketch. None of these images constrained the walk. (B) Fraction of images on which a network makes the same top-1 prediction as the anchor. (C) Error consistency with the anchor (chance-corrected agreement on which images are classified correctly). Endpoint curves and bars show the mean and s.d. over the five walks of Figure 3 ; diff-seed curves and bars the mean and s.d. over two seeds. The anchor and ViT-S/in1k are single networks; their bands and bars show the s.d. over bootstrap resamples of the test images.

Figure 24: CKA to the anchor against accuracy lost, for random weight noise and for the HNC walk. Max-over-layers CKA against the drop in top-1 accuracy on held-out ImageNet validation images, for Gaussian weight noise of increasing scale (three seeds) and for the walk checkpoints up to the reported endpoint (ringed). Band: independently trained models; dotted: untrained.

Figure 25: Accuracy, prediction match and error consistency of the

k

NN-steered endpoints on four held-out test sets. Same panels as Figure 23 . (A) Top-

k

accuracy of the anchor, the walk endpoints, the diff-seed network and ViT-S/in1k. (B) Fraction of images on which a network makes the same top-1 prediction as the anchor. (C) Error consistency with the anchor (chance-corrected). Endpoint curves and bars show the mean and s.d. over three

k

NN-steered walks with different probe draws; diff-seed curves and bars the mean and s.d. over two seeds; anchor and ViT-S/in1k bands and bars the s.d. over bootstrap resamples of the test images.

Figure 26: Alignment to the anchor against accuracy lost, for random weight noise and for the HNC walk. Max-over-layers mutual-

k

NN alignment against the drop in top-1 accuracy on held-out ImageNet validation images, for Gaussian weight noise of increasing scale (three seeds) and for the walk checkpoints up to the reported endpoint (ringed). Band: independently trained models; dotted: untrained.

Figure 27: Steering a single layer: the pre-logits feature. Top row: CKA-steered walk; bottom row: mutual-

k

NN-steered walk. Left: similarity of the steered layer to the anchor (thick, steered metric; thin, its unsteered companion) on the probe and held out. Middle and right: function preservation along the same walk.

Figure 28: Steering a single early layer: CKA of the block-1 class token. Same panels as Figure 27 , shown up to a weight distance of

0.005

Figure 29: ViT-S can be steered most of the way to ViT-B without changing what it computes. (A) CKA to the frozen ViT-B over the walk, one thin curve per depth (shaded early to late) under the steered mean, and the same mean against the similarity ViT-S keeps to its own starting representation. (B) Function: prediction match to the anchor on the constrained probe and on held-out ImageNet images, and ImageNet top-1 to top-5 accuracy. (C) Probe image kernels, in one ordering taken from ViT-B, for ViT-B, the ViT-S anchor, and the ViT-S endpoint.

Figure 30: A larger probe set allows more metric displacement and a better-preserved function, at proportionally more compute. Each point is one function-preserving max-over-layers mutual-

k

NN walk on ViT-S. (A) Metric displacement at the null-space walk’s endpoint (B) Function drift measured as the change in top-1 accuracy on

5{,}000

held-out ImageNet validation images that the walk never constrained on. (C) Total wall clock runtime. The per-step cost grows about linearly in

N

Figure 31: Solutions reached by CKA- and kNN-steered HNC from a pretrained ViT-S/16, under three distances. Metric MDS of every checkpoint of CKA-steered (blue) and mutual-kNN-steered (red) walks on five probe sets each, together with walks steered away from those endpoints rather than from the anchor, drawn in the colour of the metric they steer. Marker size and opacity grow along each walk. Distances are computed on 512 held-out Places-365 images that no walk was steered on: representational distance by CKA (left) and by mutual-kNN alignment (middle) of the penultimate layer, and relative weight distance (right). The two steering metrics leave the anchor in separate directions under every distance. kNN-steered walks look compact under CKA distance but spread widely under kNN distance, so how far a representation has moved depends on what is measured.

Figure 32: Steering one network at fixed function moves its Brain-Score as much as switching architectures. IT predictivity on MajajHong2015 (PLS-25, ceiling-normalized, the same pooled pipeline for every bar) for two anchors and their up and down HNC endpoints, alongside the other leaderboard models. (A) The top model of the neural leaderboard, a ViT-B with relative-position embeddings. Its up endpoint scores above every trained model, and the range between its two endpoints (

0.618

0.719

) is wider than the whole spread across models (

0.075

). (B) A plain ViT-S/16, the lowest scorer of the seven. Its range (

0.546

0.627

) still matches the spread across models, and its down endpoint falls below every trained model. Colors mark what mainly distinguishes each comparison model from the panel’s anchor; shaded rows mark the anchor and its endpoints.

Hyperparameter	Value
Anchor	PPO-trained $64$ -unit recurrent policy
Buffer	$8$ episodes of the current policy, re-collected every $5$ steps
Flat directions	soft projection with damping $\mu=10^{-5}\lambda_{1}$
Solver	conjugate gradient, $50$ iterations
Steering objective	$\phi_{B}^{\mathrm{MSE}}$ of Eq. 23
Restore steps	$15$ per null-space step

Appendix

Table 9: Settings of the plume-tracking walk in Fig. 4 .

Hyperparameter	Value
Anchor	ten PPO-trained two-layer policies (Appendix I.6 )
Buffer	$32$ episodes of the current policy with $\epsilon$ -greedy exploration, $\epsilon=0.25$ ; re-collected every $3$ steps
Flat directions	soft projection with damping $\mu=10^{-5}\lambda_{1}$
Solver	conjugate gradient, $50$ iterations
Steering objective	$\phi_{B}$ of Eq. 20
Restore steps	$15$ per null-space step

Appendix

Table 10: Settings of the boat race walks in Fig. 4 .

Hyperparameter	Value
Anchor	PPO-trained feed-forward actor with $256$ hidden units
Buffer	$4{,}000$ transitions of the current policy, re-collected every $5$ steps
Flat directions	soft projection with damping $\mu=10^{-5}\lambda_{1}$
Solver	conjugate gradient, $50$ iterations
Steering objective	$\phi_{B}^{\mathrm{MSE}}$ of Eq. 23
Restore steps	$15$ per null-space step

Appendix

Table 11: Settings of the MuJoCo Ant walk in Appendix I.7 .

Figure 33: Success rate along the plume-tracking walk. The fraction of

60

fixed initial conditions in which the agent reaches the odor source, plotted against the relative weight distance from the anchor. Markers show the evaluated walk steps, and shading shows the standard error over initial conditions. The horizontal dotted line marks the anchor’s success rate. The vertical dotted line marks the alternative policy shown in Fig. 4 .

Figure 34: The alternative plume-tracking policy generalizes better than its anchor on every shifted condition. Homing rate against the HNC walk step for the walk of Fig. 4 , evaluated under the constant-wind training condition and three shifted conditions. Lines and shading show the mean

\pm

s.e.m. over eight batches of

30

paired initial conditions with common random numbers; rates are comparable across steps within a condition but not across conditions, since each condition has its own initial-condition grid. The dotted line marks the alternative policy analyzed in the main text (Fig. 4 A,C).

Figure 35: On Ant-v3, the walk moves the actions while the return is unchanged. (A) Action-KL from the anchor along the walk. (B) Return, mean

\pm

SE over the

30

fixed initial states, against the anchor level (dotted) and against random actor-space directions rescaled to the same action-KL as the snapshot at that step. (C) Average forward speed over the same episodes.

Figure 36: The alternative policy walks with a different inter-leg pattern and fails in a different way. Legs are named by position: front-right (FR), front-left (FL), back-left (BL), and back-right (BR). (A) Gait diagram from a shared initial state. A bar marks the swing of one leg, with the anchor above and the alternative below within each leg row, and time measured in gait cycles (one bounce of the torso, dotted lines) so that policies running at different tempos remain comparable. (B) The tempo of each leg in the same units, mean

\pm

SE over the episodes in which the agent walks (

25

30

for the anchor and

27

for the alternative; a stalled episode provides no gait to measure). The pink bands in (A) and (B) mark the FL-BR diagonal, the pair that is re-timed. (C) Torso paths seen from above, one line per episode. (D) Episodes that end badly, where stall denotes an episode that ran to the

1000

-step limit at under

0.5

m/s and early-end an episode stopped by the simulator because the torso left the healthy height range.

Width $w$	$P$ at $C=7$	$P$ at $C=100$
$16$	$75{,}703$	$81{,}748$
$24$	$169{,}039$	$178{,}060$
$32$	$299{,}367$	$311{,}364$
$48$	$670{,}999$	$688{,}948$
$96$	$2{,}673{,}703$	$2{,}709{,}508$

Appendix

Table 12: Parameter count

P

of the CNN grid.

Figure 37: Null fraction across the grid at every flatness threshold. Each column is one threshold

\epsilon

. Top: number of classes at each width. Bottom: width at each number of classes. Mean

\pm

s.e.m. over

3

seeds.

Figure 38: Normalized curvature across the grid at four relative flatness thresholds. Each column is one cutoff

\mu/\lambda_{\max}

. Top: number of classes at each width. Bottom: width at each number of classes. Gauss–Newton curvature

d^{\top}Gd/(nC)

of each accepted unit step, median over steps, then median

\pm

s.e.m. over

3

seeds.

Figure 39: Harder tasks push the representation into more dimensions; model size barely does. (A) Effective dimensionality (participation ratio) of the held-out representation vs width (at

C{=}10

) and vs the number of classes (at

w{=}48

); band = min-max over

3

seeds. (B) PCA cumulative variance of the representation (mean and min-max band over

3

seeds): width leaves the curves nearly identical (left), task complexity fans them apart (right).

Mode Connectivity

Mode connectivity studies low-loss paths between separately trained neural-network solutions and what those paths reveal about optimization and model behavior.

See all 11 papers

Apr 13, 2026

Inter-Layer Hessian Analysis of Neural Networks with DAG Architectures

Modern automatic differentiation frameworks (JAX, PyTorch) return the Hessian of the loss function as a monolithic tensor, without exposing the internal structure of inter-layer interactions. This paper presents an analytical formalism that explicitly decomposes the full Hessian into blocks indexed by the DAG of an arbitrary architecture. The canonical decomposition

H = H^{GN} + H^T

separates the Gauss--Newton component (convex part) from the tensor component (residual curvature responsible for saddle points). For piecewise-linear activations (ReLU), the tensor component of the input Hessian vanishes (

H^{T}_{v,w}\!\equiv\!0

a.e.,

H^f_{v,w}\!=\!H^{GN}_{v,w}\!\succeq\!0

); the full parametric Hessian contains residual terms that do not reduce to the GGN. Building on this decomposition, we introduce diagnostic metrics (inter-layer resonance~

\mathcal{R}

, geometric coupling~

\mathcal{C}

, stable rank~

\mathcal{D}

, GN-Gap) that are estimated stochastically in

O(P)

time and reveal structural curvature interactions between layers. The theoretical analysis explains exponential decay of resonance in vanilla networks and its preservation under skip connections; empirical validation spans fully connected MLPs (Exp.,1--5) and convolutional architectures (ResNet-18,

{\sim}11

M~parameters, Exp.,6). When the architecture reduces to a single node, all definitions collapse to the standard Hessian

\nabla^2_θ\mathcal{L}(θ)\in\mathbb{R}^{p\times p}

Maxim Bolshim, Alexander Kugaevskikh

ITMO University, Saint Petersburg, Russia

Neural Network Optimization Multilayer Perceptrons ReLU Neural Networks

Ask PDF

Jul 8, 2026

Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks

The Hessian of the training loss governs the local geometry of the loss landscape, yet despite existing explanations for its largest eigenvalues, the origin of the vast multitude of vanishingly small eigenvalues remains elusive. We argue that the bulk consists of the weakly lifted pseudo-Goldstone modes of the continuous symmetries of the network parametrization. In deep linear networks these symmetries are exact: they generate flat directions and hence exact zero modes, whose eigenvectors we construct explicitly. Introducing a ReLU nonlinearity as a perturbation, we show that it breaks these symmetries weakly and explicitly. Resolving the spectrum at the level of eigenvectors, we find that the high-curvature directions are orthogonal to the symmetry subspace, while the bulk lies almost entirely within it. We demonstrate the mechanism in a two-layer ReLU student--teacher model and in a network trained on CIFAR-10. A convolutional example demonstrates that the same diagnostic extends beyond fully connected layers. Together, these results link the Hessian bulk to weakly broken symmetries and clarify the origin of near-zero modes.

Marcel Kühn, Bernd Rosenow

Institut für Theoretische Physik, Universität Leipzig, 04103 Leipzig, Germany · ScaDS.AI Dresden/Leipzig, Universität Leipzig, 04105 Leipzig, Germany

ReLU Neural Networks Symmetry Breaking

Ask PDF

Jul 15, 2026

How the Hessian-Spectrum of Neural Networks Depends on Data

The Hessian matrix is an important quantity of interest when it comes to studying the loss landscape and optimization dynamics in deep learning, as well as designing measures of generalization, second-order learning algorithms, etc. Prior works have focused on empirical results or pursued a theoretical treatment under overly simplified settings. In this work, we derive the eigenvalues of the Hessian of linear networks with arbitrary widths and depths, and datasets with an arbitrary number of samples, features, and labels. Importantly, for classification tasks with MSE loss, we identify that the sharpness of the solution is directly related to the maximum proportion of samples belonging to any class. We empirically validate our predictions and systematically analyze the effects of shedding the impractical assumptions one at a time, as well as incorporating nonlinearities. We observe that our predictions are considerably robust in most cases, allowing us to extend our conclusions to more practical learning setups.

Jasraj Singh, Enea Monzio Compagnoni, Antonio Orvieto

ELLIS Institute, Tübingen, Germany · Universität Basel, Basel, Switzerland · MPI-IS, Tübingen, Germany

Loss Landscape Sharpness Neural Network Optimization Deep Linear Networks Neural Network Training Dynamics

Ask PDF

Abstract

Mode Connectivity

Explore similar work

Inter-Layer Hessian Analysis of Neural Networks with DAG Architectures

Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks

How the Hessian-Spectrum of Neural Networks Depends on Data