Mode Dynamics: A Framework for Neural Network Learning

Hossein Honarvar

cs.LGOct 29, 2026

Mode Dynamics: A Framework for Neural Network Learning

Authors: Hossein Honarvar

Organizations: Independent Researcher

Abstract

We introduce Mode Dynamics, a framework that connects coordinated parameter motion during neural network learning to changes in model outputs. Starting from a recorded parameter trajectory, we decompose its updates into collective modes and map each mode to its state-dependent output effect. The framework organizes this link through three mechanisms. Decomposition identifies collective modes in parameter motion and their time-varying amplitudes. Reconstruction tests whether the effects induced by those modes recover local and accumulated output change. Concentration quantifies how behavioral contribution is distributed across modes. Controlled scalar regression and next-token experiments directly test these mechanisms. They show that small sets of leading modes reconstruct behavior, induced effects depend on the training path, and modal complexity varies with task and data. We then evaluate the framework on held-out image and language-model outputs using CIFAR-100 autoencoding and a public Pythia-70M pretraining trajectory. Across these settings, the results connect compact modal subspaces of recorded parameter motion to evolving behavior.

Figures & tables

Figure 1: Leading-mode reconstructions approximate local output changes. A: Median local

R^{2}

for

K\in\{1,3,5,10\}

and the full basis over 15 trajectories. Capped bars show the interquartile range and the vertical axis is symmetric-log. B: The observed and reconstructed one-step output change for a representative run at epoch 20k. Modes follow the parameter-energy ordering of the update SVD.

Figure 2: Path-dependent induced effects improve accumulated reconstruction. A: Mean Top-10 accumulated reconstruction

R^{2}

over 15 runs for along-path and initialization-frozen induced effects. B: Representative along-path Top-

K

function reconstruction at epoch 20k with the initialization-frozen full-basis baseline. C: Controls at epoch 20k over five width-64 seeds, scored on a separate analysis set held out from behavioral-energy ranking. Capped bars show one standard deviation. Panel A uses logarithmic epoch spacing, Panel C uses symmetric-log vertical scaling, and Panels A–B use trajectory-level behavioral-energy ordering. Protocols and control definitions are in Appendix C .

Figure 3: Behavioral PR varies across next-token tasks and data conditions. At epoch 50k, the full 675-run grid shows participation ratio versus sequence length in Panel A, content vocabulary in Panel B, and transformer depth in Panel C. Error bars denote one standard deviation over the corresponding conditions and seeds. All metrics are evaluated on training sequences. Matched-condition optimization, accuracy, and PR trajectories are in Appendix D .

Figure 4: Small sets of leading modes reconstruct next-token training-set behavior. A–B: Training-set logit $R^{2}$ and Top-1 prediction agreement compare each Top- $K$ reconstruction with the trained full-network output, averaged over five seeds. Capped bars show one population standard deviation, and modes use trajectory-level behavioral-energy ordering. C–E: Selected training examples illustrate low-to-high- $K$ recovery at epoch 50k. Rows progress through cumulative Top- $K$ modal reconstructions for $K\in\{1,3,5,10\}$ to the same trained full-network output. Diamonds mark targets and stars mark row predictions, overlapping when correct. Selection details are in Appendix D.4 .

Analysis	Retained set / control	Metric	Result
CIFAR-100 autoencoding
Test reconstruction	Top-50	pooled RGB-change $R^{2}$	$0.912\pm 0.012$
Image-level fidelity	Top-50	median per-image $R^{2}$	$0.892$
Parameter-motion energy	Top-50	fraction retained	$43.8\%\pm 0.4\%$
Coordinate control (training split)	signed permutation	pooled RGB-change $R^{2}$	$-0.642\pm 0.344$
Pythia-70M next-token pretraining

Table 1: Summary of CIFAR-100 and Pythia-70M results. CIFAR “

\pm

” values are population standard deviations over five trajectories, except the control, which is over three seed-0 signed permutations. Pythia rows above the Post-hoc sensitivity break use analysis settings fixed before any Pile-test access. Those below vary checkpoint density and quadrature afterwards on the same frozen test contexts. The bracketed interval is a 95% context-bootstrap confidence interval.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

setting	task / model	reconstructed output	mode ordering	reported measures
Scalar regression	sine fitting one-layer MLP	scalar prediction curve	parameter-energy (local) behavioral-energy (accumulated)	local/accumulated Top- $K$ $R^{2}$ PR and auxiliary $K_{\rm eff}$
Next-token prediction	copy / induction / random labels decoder-only transformer	last-position logits and predicted-token decisions	behavioral-energy	Top- $K$ logit $R^{2}$ and Top-1 prediction agreement PR and auxiliary $K_{\rm eff}$
Pixel-space autoencoding	CIFAR-100 reconstruction convolutional autoencoder	RGB decoder-output change	parameter-energy	Top-50 RGB-change $R^{2}$ retained parameter-motion energy, no full PR
Public pretraining trajectory	Pythia-70M validation ranking / locked Pile-test scoring	vocabulary-centered final-position next-token logit change	behavioral-energy on dedicated ranking contexts	held-out pooled $\Delta$ logit $R^{2}$ , PR, $K_{\rm eff}$ post-hoc KL and Top- $r$ prediction agreement

Appendix

Table 2: Experimental overview and metric coverage.

Figure 5: Regression output-change energy is concentrated across training. A: Mean ranked behavioral energy at epoch 10k, its cumulative share, and the corresponding participation ratio. B: Mean participation ratio across the five analyzed nonterminal checkpoints through 40k, with capped one-standard-deviation bars. Panel A averages the same five-seed, three-width sine grid.

Figure 6: Regression optimization context. Mean width-64 sine-target MSE through 80k. Shading shows one standard deviation over five seeds. Vertical markers locate the representative 1k and 20k analysis checkpoints.

Figure 7: Seed, width, and checkpoint-spacing robustness. A: Sine-target Top-10 accumulated

R^{2}

at 20k, aggregated over five seeds at each width. B: Sine Top-10 accumulated

R^{2}

at 20k versus checkpoint spacing for width 64. Top-10 uses trajectory-level behavioral-energy ordering. The default spacing of 10 averages five seeds, while additional spacing runs use seed 0 only.

Figure 8: Next-token optimization and participation-ratio trajectories. A–C: Cross-entropy, training accuracy, and participation ratio over time for copying, induction, and random labels in the matched reference configuration (depth 1, sequence length 4, content vocabulary 6). Lines and shading are mean

\pm

standard deviation over five seeds, and the dotted line marks the 50k checkpoint used for the main-text task/data comparison.

Figure 9: Next-token behavioral-concentration diagnostics. A:

K_{\mathrm{eff}}(\tau)

varies with the selected energy threshold from 0.70 to 0.99, with dashed horizontal lines showing the threshold-independent mean PR for each task. B: PR strongly agrees with the auxiliary

K_{\mathrm{eff}}(0.9)

count.

Figure 10: Behavioral PR versus parameter count. Across the controlled grid (approximately 13k–39k parameters), PR separates more strongly across task groups than across the displayed parameter-count range. This is a descriptive bounded-grid association rather than a variance decomposition or claim of scale invariance.

Figure 11: Dynamic trajectory and static endpoint summaries. A: Behavioral PR of realized-contribution energy at epoch 50k and endpoint head-spectrum PR across tasks. B: Per-run comparison. The quantities are not commensurate. The dynamic statistic uses full-parameter realized contributions at 50k, whereas the static statistic uses the 0–100k output-head change and is capped by head rank.

Figure 12: Selected per-example cumulative logit reconstructions. A–C: Copying. D–F: Induction. G–I: Random labels. Each subplot is one training example. Rows progress through cumulative Top- $K$ modal reconstructions to the trained-model full-network reference. White diamonds and smaller gold stars are centered on the task target and row prediction, respectively, so a correct prediction appears as a star over a diamond.

Figure 13: A small parameter-energy-ranked subspace reconstructs the RGB change induced by CIFAR-100 training. A: Training-split seed-0 eight-image

K

curve, five-seed Top-50 mean

\pm

population standard deviation (100 training images per seed), and three seed-0 norm-matched controls. B: Fraction of parameter-motion energy retained. C: Training-split per-image Top-50

R^{2}

(100 images per box), with extreme fliers hidden only visually. Panel A reconstructs the trained decoder’s RGB change from initialization, not the CIFAR target, and modal scoring uses training images.

Figure 14: Matched CIFAR-100 decoder outputs and cumulative modal reconstructions. Columns are fixed evaluation images. Rows show the ground truth, final decoder, epoch-0 decoder, and cumulative Top-1, Top-3, Top-5, Top-10, and Top-50 path-integrated predictions from the detailed seed-0 run. These examples illustrate pixel-level recovery, while aggregate claims use the five-seed, 100-image results in Figure 13 .

Figure 15: Pythia-70M centered-logit change is concentrated in a behaviorally ranked trajectory subspace. A: Locked confirmatory Pile-test pooled centered-

\Delta

logit

R^{2}

across validation-ranked

K

-mode subsets. Blue evaluates effects along the checkpoint path. Red is the initialization-frozen Top-

K

reconstruction, with

K=50

equal to the initialization-frozen all-mode reconstruction. The dotted line is full-update. Shading is a 95% context-bootstrap interval, and the annotation gives the signed-permutation 50-mode control. B: Validation-ranking cumulative behavioral energy in behavioral-energy ordering and cumulative parameter-motion energy in parameter-energy ordering. Legends report PR. Guides mark the 90% behavioral-energy threshold at 13 modes. The curves use separate rankings and are not a common-order comparison.

Checkpoints	Modes	PR / modes	$K_{\rm eff}(0.9)$ / modes	Full-update pooled centered $\Delta$ logit $R^{2}$ $Q=2/4/8$	Full-update mean KL (nats) $Q=2/4/8$	Full-update Top-1 prediction agreement $Q=2/4/8$
51 (locked)	50	$5.09$ (10.2%)	$13$ (26.0%)	$0.999809/0.999966/0.999997$	$0.1463/0.0366/0.00246$	$77.3\%/95.3\%/97.7\%$
101 (post-hoc nested)	100	$7.82$ (7.8%)	$21$ (21.0%)	$0.999705/0.999947/0.999994$	$0.2954/0.0312/0.00425$	$77.3\%/96.1\%/96.9\%$

Appendix

Table 3: Post-hoc Pythia checkpoint and quadrature sensitivity. Concentration uses the fixed validation ranking contexts. The

Q=2

columns use the all-mode modal reconstruction, while

Q=4,8

use the exact checkpoint displacement. The all-mode endpoint differs from full-update only at numerical precision. All output metrics use the same 128 frozen Pile-test contexts. The reported

R^{2}

is the pooled centered-

\Delta

logit score in Equation ( 16 ). KL is mean

D_{\mathrm{KL}}(p_{\mathrm{trained}}\|p_{\mathrm{recon}})

in nats. Top-1 prediction agreement compares the trained model’s and reconstruction’s highest-logit tokens. PR and

K_{\mathrm{eff}}

fractions divide each statistic by the available mode count.

Figure 16: Post-hoc Pythia checkpoint-resolution and quadrature sensitivity. Columns hold the trajectory discretization fixed at 51 checkpoints (50 modes) or 101 nested checkpoints (100 modes). Rows report pooled centered-

\Delta

logit

R^{2}

, mean full-vocabulary

D_{\mathrm{KL}}(p_{\mathrm{trained}}\|p_{K})

in nats on a log scale, and

Q=8

Top-

r

prediction agreement. Lower KL is better. Here

K

is the retained-mode count, while Top-

r

prediction agreement is top-

r

accuracy with the trained model’s highest-logit token in place of the ground-truth label: the fraction of contexts where that token lies among the reconstruction’s

r\in\{1,10,50\}

highest-logit candidates. It is therefore neither true-token accuracy nor candidate-set intersection, and

r=1

is exactly the Top-1 prediction agreement reported elsewhere. Even

r=50

covers only 0.10% of the vocabulary. Panel E also marks the signed-permutation control at full retention, scored at the same

Q=8

. It is defined only for the 51-checkpoint 50-mode path, so panel F has no counterpart. The 51-checkpoint

Q=8

sweep starts at

K=20

modes (40% retention), so panel E has no 20% or 30% points. Within a schedule, quadrature leaves the mode order fixed. The 101-checkpoint

Q=4,8

endpoints use the exact checkpoint displacement, numerically equivalent to retaining all modes.

	pooled centered $\Delta$ logit $R^{2}$		Top-1 prediction agreement
Retained	51 ckpt	101 ckpt	51 ckpt	101 ckpt
$20\%$	$0.8582$	$0.9357$	$0\%$	$0\%$
$40\%$	$0.9668$	$0.9877$	$0\%$	$1.6\%$
$60\%$	$0.9962$	$0.99795$	$10.9\%$	$17.2\%$
$80\%$	$0.99958$	$0.99947$	$46.1\%$	$59.4\%$
$100\%$	$0.99981$	$0.99970$	$77.3\%$	$77.3\%$

Appendix

Table 4: Matched retained fractions across checkpoint schedules. Two-node (

Q=2

) held-out scores on the same 128 frozen Pile-test contexts, with modes ranked on the validation contexts. Retained fraction is the kept mode count divided by the available mode count, so

80\%

is 40 of 50 modes for the 51-checkpoint basis and 80 of 100 for the 101-checkpoint basis. The reported

R^{2}

is the pooled centered-

\Delta

logit score in Equation ( 16 ).

Figure 17: Illustrative untouched-test next-token distributions across cumulative modal reconstructions. Each row uses one Pile-test scoring context. Columns show cumulative 1-, 5-, 20-, and 50-mode reconstructions followed by the trained-model reference. Bars show the five highest-probability tokens, green marks the true next token, and each panel reports its rank. Examples were selected deterministically among contexts the trained model predicts correctly. The reported true-token rank is therefore also the rank of the trained model’s predicted token and measures the same quantity that Figure 16 E–F aggregates as Top-

r

prediction agreement, here for the locked two-node analysis rather than the eight-node sweep. These examples are explanatory rather than aggregate evidence.

Rank cutoff $r$	$1$	$3$	$5$	$10$	$20$	$50$
51 checkpoints, 40/50 modes	$52.3\%$	$67.2\%$	$71.9\%$	$79.7\%$	$89.1\%$	$94.5\%$
101 checkpoints, 80/100 modes	$64.1\%$	$77.3\%$	$84.4\%$	$88.3\%$	$91.4\%$	$96.9\%$

Appendix

Table 5: Top-

r

prediction agreement at 80% retention. Eight-node (

Q=8

) scores on the same 128 frozen Pile-test contexts, corresponding to Figure 16 E–F. A context counts as agreeing when the trained model’s highest-logit token lies among the reconstruction’s

r

highest-logit candidates.

Neural Network Training Dynamics

Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.

See all 520 papers

Oct 26, 2026

Dynamical coarse-graining of neural-network training

Dynamical coarse-graining of neural networks gives macroscopic observables and enables training to be predicted, and controlled in a state space far smaller than the parameter space. In this paper, we first define the learning operator following the training law of neural networks and establish a descent theorem for the learning operator, which serves to determine whether the operator is sufficient to characterize the learning dynamics. Then the exact closure theorem and the dynamic coarse-graining theorem are presented in sequence. The three theorems established in this paper provide a systematic framework for constructing and assessing dynamical coarse-grained models of neural-network training. Finally, we demonstrate the applicability of the theorems with a finite-width nonlinear ReLU network. Its dynamic coarse-grained model for a fixed pattern is exactly derived and composed of five independent moments. The model exactly predict the training outputs and evolving learning operator. The framework may also guide the development of approximate coarse-grained models for more general neural networks when exact closure is unavailable.

Lin-Juan Wang, Da Li, Huiling Duan +1

School of Astronautics, Beihang University, Beijing 100191, China · School of Mechanics and Engineering Science, Peking University, Beijing 100871, China · CAPT-HEDPS, and IFSA Collaborative Innovation Center of MoE, Peking University, Beijing 100871, China

Neural Network Training Dynamics

Ask PDF

May 5, 2026

Physics-Modeled Neural Networks

We introduce Dynamical Physics-Modeled Neural Networks (DynPMNNs), a continuous-time deep learning architecture in which each hidden layer is defined as the solution of an ordinary differential equation. Unlike classical feed-forward networks, this approach replaces static activation functions with time-evolving dynamical systems, providing a biologically inspired interpretation of hidden-layer behavior and enabling the integration of physically meaningful models. The framework is rigorously grounded in Reproducing Kernel Banach Spaces (RKBSs), allowing DynPMNNs to be characterized as finite-dimensional solutions of an abstract training problem and revealing structural connections with standard neural networks. We present a concrete implementation based on the FitzHugh--Nagumo model for neuronal activation, where numerical ODE solvers are embedded into the computational graph via Euler-type schemes. Both network weights and dynamical parameters are trained jointly. Through experiments on the California Housing dataset, we compare DynPMNNs with Neural ODEs (NODEs) and Closed-form Continuous-Time Networks (CfCs). Despite using fewer trainable parameters, DynPMNNs achieve competitive performance. These results position DynPMNNs as a principled bridge between dynamical systems and deep learning, with promising directions for further research in expressivity, stability, and physics-based modeling.

Raul Felipe-Sosa, Angel Martin del Rey, Maria Flores Ceballos

aDepartamento de Matemáticas, Escuela Politénica Superior, Universidad Nebrija, 28015 Madrid, Spain · bDepartment of Applied Mathematics, IUFFyM, Universidad de Salamanca, 37008 Salamanca, Spain · cUnidad de Excelencia en Luz y Materia Estructuradas (LUMES), Universidad de Salamanca, 37008 Salamanca, Spain

Dynamic Neural Networks Neural ODEs

Ask PDF

Mar 11, 2026

JEDI: Jointly Embedded Inference of Neural Dynamics

Animal brains flexibly and efficiently achieve many behavioral tasks with a single neural network. A core goal in modern neuroscience is to map the mechanisms of the brain's flexibility onto the dynamics underlying neural populations. However, identifying task-specific dynamical rules from limited, noisy, and high-dimensional experimental neural recordings remains a major challenge, as experimental data often provide only partial access to brain states and dynamical mechanisms. While recurrent neural networks (RNNs) directly constrained neural data have been effective in inferring underlying dynamical mechanisms, they are typically limited to single-task domains and struggle to generalize across behavioral conditions. Here, we introduce JEDI, a hierarchical model that captures neural dynamics across tasks and contexts by learning a shared embedding space over RNN weights. This model recapitulates individual samples of neural dynamics while scaling to arbitrarily large and complex datasets, uncovering shared structure across conditions in a single, unified model. Using simulated RNN datasets, we demonstrate that JEDI accurately learns robust, generalizable, condition-specific embeddings. By reverse-engineering the weights learned by JEDI, we show that it recovers ground truth fixed point structures and unveils key features of the underlying neural dynamics in the eigenspectra. Finally, we apply JEDI to motor cortex recordings during monkey reaching to extract mechanistic insight into the neural dynamics of motor control. Our work shows that joint learning of contextual embeddings and recurrent weights provides scalable and generalizable inference of brain dynamics from recordings alone.

Anirudh Jamkhandi, Ali Korojy, Olivier Codol +2

Mila - Quebec AI Institute · Université de Montréal

Latent Dynamics Modeling Dynamical Systems Recurrent Neural Networks System Identification

Ask PDF

Abstract

Neural Network Training Dynamics

Explore similar work

Dynamical coarse-graining of neural-network training

Physics-Modeled Neural Networks

JEDI: Jointly Embedded Inference of Neural Dynamics