Neural Dynamics as the Composition of Quantized Units

Jacopo Minniti; Aravinth Kulanthaivelu; Richard Sproat

cs.LGSep 26, 2026

Neural Dynamics as the Composition of Quantized Units

Authors: Jacopo Minniti, Aravinth Kulanthaivelu, Richard Sproat

Organizations: Minerva University · Sakana AI

Abstract

Deep learning is commonly interpreted at two levels: the macroscopic, through aggregate trends in loss summarized by scaling laws, and the microscopic, through neurons, features, and circuits. A central challenge is understanding how these levels connect, so that we can explain how elementary computations compose and collectively shape macroscopic behavior. To this end, we study an intermediate abstraction in which training is described as the ordered acquisition of quanta: reusable computations acquired suddenly and binary-activated across examples to reduce loss. By approximating population-gradient updates, we derive quanta's acquisition dynamics. This yields an acquisition priority governed by demand, how frequently a computation is required across examples, and conditional complexity, how difficult that computation is to acquire given those already available. In a Boolean compositional task, we derive predictions for acquisition order and show how staggered discrete acquisitions can produce smooth aggregate loss and, under certain geometries of quanta composition, give rise to scaling laws. We then train a Transformer to map numerals to English number names and recover candidate quanta from its checkpoint trajectory. From these units, we construct a model that preserves much of the Transformer's behavior while exposing interpretable latent computations and acquisition dynamics consistent with the theory. Separately, the quanta structure can serve as training targets to improve transformer generalization. Together, these results suggest the quanta abstraction can provide useful computational atoms for studying a variety of macroscopic phenomena.

Figures & tables

Figure 1: Examples of Quanta Descriptions of a Neural Model. A smooth loss curve can be decomposed into the staggered acquisition of discrete computations. Scaling with model size becomes growth of the acquired frontier, whose unmet demand yields a power law. An opaque neural network can be rewritten as a more interpretable Q-model which follows the quanta dynamics.

Figure 2: HSP construction. A length- $10$ bit vector is read through sparse, local XORs and recursively composed by NAND. The learner receives only the vector, an opaque node identifier, and the target (Appendix A ).

Figure 3: Quantized loss dynamics in HSP. (a) In a flat HSP, individual held-out losses transition sharply at different times, while their weighted aggregate (red) is much smoother: its transition is

22.6\times

broader than the median task transition. Color runs from frequent (yellow) to rare (blue) tasks. (b) In a

32

-task composed HSP, one task (blue) pauses near the exact

1/2

-bit conditional level after acquiring its private constituent, then collapses when its shared constituent is acquired. Dashed lines mark the two

80\%

functional-mode crossings; other task losses are gray and red is their weighted aggregate.

Figure 4: Macroscopic effects of poset geometry. (a) At ratio

1.1

parents are acquired first and the direct-task and ancestor-closed loss reconstructions coincide, consistent with a compositional poset. (b) At ratio

5

, most children are acquired first and the direct-task reconstruction follows neural loss more closely, consistent with a shortcut interpretation. Curves show medians over three seeds; shading spans the range.

Figure 5: Fitted Scaling Laws. (a) parameter scaling across ten widths. Across the three prescribed demand exponents, fitted slopes match the theoretical prediction with

1.4\%

3.5\%

, and

5.7\%

relative error (

R^{2}\approx 0.89

). (b) online-IID data scaling at fixed width

2048

, plotted over the central learning regime between an initial and late plateau;

D=BS

is processed examples, with effective batch

B

and optimizer steps

S

. Fitted slopes have

9.9\%

26.7\%

, and

13.7\%

relative error from the prediction (

R^{2}\approx 0.90

). Each color denotes the demand exponent

\alpha

; parenthesized values give the fitted parameter exponent

\widehat{\alpha}

or data exponent

\widehat{b}

. Solid lines are log–log fits, and dashed lines give the prescribed slopes (

\alpha

on the left and

\alpha/(1+\alpha)

on the right).

Figure 6: From a neural trajectory to an executable Q-model. (a) The third output step of the

346\rightarrow

three, hundred, forty, six, EOS greedy trace. The source and Q-model both predict forty ; the bars show the Q-model’s top-token probabilities. Orange nodes mark five active quantum gates, but only four writers emit nonzero contributions: in the current Q-model, gate eligibility and executed writer activity can disagree. Four frozen held-out probe labels are shown. Solid arrows denote selected inter-quantum message channels; dotted lines show the layerwise residual aggregation. (b) Normalized cumulative population-gradient priority for the 13 frozen candidates in the selected

3\%

factorization. The dashed held-out NLL-improvement curve tracks their mean along this trajectory (

r=0.991

). (c) Across a finer 22-factor diagnostic basis, later factors are less frequent and their priority-weighted source updates involve more effective MLP groups.

Training	Balanced	Zero gap	Short tail
CE only	$84.9$	$61.7$	$50.5$
$+$ Q target	$\mathbf{94.7}$	$\mathbf{89.0}$	$\mathbf{89.9}$

Table 1: Q-guided training. Exact-sequence accuracy for a matched

10

-block Transformer. The zero-gap and short-tail columns contain

4{,}031

and

1{,}553

train-disjoint examples.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

$\alpha$	Target	Full fit	Earlier range / slope	Later range / slope
$0.20$	$0.1667$	$0.1832$	$2.2$ – $4.2$ M / $0.1882$	$4.3$ – $8.4$ M / $0.1072$
$0.25$	$0.2000$	$0.2534$	$1.9$ – $3.5$ M / $0.2838$	$3.6$ – $6.7$ M / $0.1532$
$0.339$	$0.2533$	$0.2879$	$1.5$ – $3.0$ M / $0.4339$	$3.1$ – $6.2$ M / $0.1164$

Appendix

Table 2: Data-fit window sensitivity. Equal-log-span halves of the acquisition/plateau-selected interval; ranges are optimizer steps and every fit uses

D=4096S

Output	Active	Active	Total	Mean/min	Event interpretation
	Qs	writers	activity	writer F1
three	4	6	39.600	0.812 / 0.741	sequence start, unit lexeme, value read
hundred	3	4	12.088	0.696 / 0.584	hundreds and low-chunk structure
forty	5	6	33.803	0.847 / 0.771	after-hundred, tail, value read/write
six	4	7	20.187	0.821 / 0.650	unit value and tail completion
EOS	7	3	6.095	0.794 / 0.684	termination and completed tail

Appendix

Table 3: Event summary for the

346

greedy trace. Counts and activity are exact for the displayed execution; writer F1 statistics use the frozen owner-conditional descriptions.

Output	Quantum	Held-out F1	Selected description
three	L0.Q3	0.615	unit lexeme or 5-digit input
three	L1.Q2	0.705	after sequence start or 3-digit input
three	L1.Q3	0.544	chunk has hundreds
three	L2.Q2	0.662	unit lexeme
hundred	L0.Q0	0.698	chunk has tail
hundred	L1.Q0	0.760	one-hundred rule or EOS decision

Appendix

Table 4: All active quanta for

346

. Each description is selected on calibration events and its F1 is evaluated unchanged on the disjoint semantic test.

Output	Writer	Activity	Conditional F1	Selected description
three	L0.Q3.W10	0.218	0.741	after sequence start
three	L1.Q2.W8	1.802	0.824	after sequence start or hundreds
three	L1.Q3.W9	0.359	0.783	unit form or after sequence start
three	L2.Q2.W6	14.219	0.882	value read
three	L2.Q2.W7	10.736	0.869	value read
three	L2.Q2.W8	12.266	0.771	emit value

Appendix

Table 5: All active writers for

346

. Activity is the gated contribution magnitude in this execution. F1 is held out and conditioned only on events where the writer’s owner quantum is active.

Figure 7: A second Q-model execution, $800$ . Orange cells mark active quantum gates; the number inside is the count of active scalar writers in that gate’s three-writer bank. Blank cells are inactive gates. Each column is one greedily emitted token. The source and Q-model both match the target at all three steps.

Neural Network Training Dynamics

Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.

See all 516 papers

Jan 10, 2026

Architecture Before the Formula: Training-Dependent Computational Semantics in Transformers

An operation's input-output function does not, by itself, determine its computational role within a represented process. That role also depends on the processes producing its inputs and receiving its outputs, a philosophical stance termed deep learning externalism (DLE). We develop a mathematical account of represented computation that distinguishes functional equivalence, retained predecessor distinctions, restricted receiver capability, and context-dependent effective architectures. The resulting equivalences depend on which computational relations a scientific description must preserve. We prove that two injective producer contexts can preserve every predecessor distinction while changing whether fixed local and attention receiver families are equivalent. We investigate these relations in a dimension-labelled PreNorm Transformer and their evolution under training. Residual bypasses restrict complete-state information identifications to two merges per block, while finer branches can change receiver accessibility. Exact gradient-descent models yield covariance-dependent geometric attenuation under learned producer routing and algebraic

(t^{-1/2})

attenuation under Q/K-softmax training, with optimal bounded-receiver recovery errors derived in both cases. Coupled producer-attention dynamics establish how learning jointly reorganizes the presented features and receiver allocation. Finally, present functional or receiver-profile equivalence need not determine future behavior under an optimizer. In an input-conditioned attention model, two accessibility measurements form an exact reduced predictive state, whereas one alone does not.

Luis F. Rosario Freytes

Transformer Training Dynamics Transformer Attention Neural Network Interpretability Neural Network Training Dynamics

Ask PDF

Oct 20, 2026

Transformers as Generic Deep Neural Networks: Exact Compilation, the Cost of Removing Activation-Activation Products, and Multi-Agent Attention Clustering

Transformers are deployed as a special model class. Neural processing units (NPU) add dedicated operators for attention and normalization, and quantization relies on Transformer-specific algorithms. We show the computation does not force this. Every Transformer block, with attention, LayerNorm and a GELU MLP, compiles exactly into a generic deep neural network built from fixed affine maps, routing and the square, sigmoid and inverse square root, plus GELU or its sigmoid approximation. Without activation-activation products, every transform-wise compilation adds at least

N^2

scalar nonlinear evaluations per head for dense scores. Keeping that one operation type removes this cost; its elementwise form, present on every surveyed NPU, suffices with the sigmoid and inverse square root. For top-

k

attention with exchangeable tail scores, we prove that exact or conditional-mean tail mass gives an unbiased, minimum-mean-square-error estimate of the attention distribution, while tail drop is conditionally biased. When queries or agents cluster, centroids choose the keys first, reducing the number of computed scores from

N^2

Nk

per head. On DeiT-Tiny, MambaIRv2-Light and Pythia-70M the compiled models match their references with all keys, standard post-training quantization applies, and the exact tail-mass estimate beats tail dropping with half as many keys on DeiT-Tiny and, for

k\le8

, on Pythia-70M.

Sergey Salishev, Zeev Volkovich, Oleg Granichin

St. Petersburg State University, Saint Petersburg 199034, Russia · Department of Software Engineering, Braude College of Engineering, Karmiel 2161002, Israel

Efficient Transformer Inference Transformer Attention Efficient Attention

Ask PDF

Aug 13, 2026

Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws

Neural networks trained by gradient descent on a smooth cost function can nevertheless learn in steps: the cost holds on long plateaus and then drops abruptly. Meanwhile, training losses instead follow smooth power laws. Variants of both behaviors occur in architectures with very different microscopic structures, which is the signature of a few relevant collective variables. We show that a symmetry fixes what those variables are: a network layer is a sum over interchangeable units, so relabeling the units leaves it unchanged; given smoothness and the condition that a unit's gradient vanish at the origin, symmetry then enforces a universal leading form for the expansion about the near-zero weights present at the start of training, the quadratic

\Tr[WW^{\top}A(x)]

, in which every architectural detail is confined to a single

structure matrix" $A(x)$ that we compute for each architecture. Perceptrons, attention layers, mixtures of experts, and convolutions become one model at different $A$. Its training dynamics then close on the

order parameter"

M=WW^{\top}

and, whenever the data matrices share an eigenbasis, reduce to a Lotka--Volterra equation whose modes switch on one after another. The smaller the initial weights, the further apart the switch-on times, and the plateaus appear as a singular limit of a smooth flow; when many modes are unresolved the same events merge into a power law in training time whose exponent the theory predicts. We confirm both numerically across training methods and architectures.

Liu Ziyin, Yizhou Xu, Tomaso Poggio +1

1Massachusetts Institute of Technology · 2 ´Ecole Polytechnique F´ed´erale de Lausanne

Dynamical Systems Neural Network Training Dynamics Neural Scaling Laws

Ask PDF

Abstract

Neural Network Training Dynamics

Explore similar work

Architecture Before the Formula: Training-Dependent Computational Semantics in Transformers

Transformers as Generic Deep Neural Networks: Exact Compilation, the Cost of Removing Activation-Activation Products, and Multi-Agent Attention Clustering

Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws