Deep learning is commonly interpreted at two levels: the macroscopic, through aggregate trends in loss summarized by scaling laws, and the microscopic, through neurons, features, and circuits. A central challenge is understanding how these levels connect, so that we can explain how elementary computations compose and collectively shape macroscopic behavior. To this end, we study an intermediate abstraction in which training is described as the ordered acquisition of quanta: reusable computations acquired suddenly and binary-activated across examples to reduce loss. By approximating population-gradient updates, we derive quanta's acquisition dynamics. This yields an acquisition priority governed by demand, how frequently a computation is required across examples, and conditional complexity, how difficult that computation is to acquire given those already available. In a Boolean compositional task, we derive predictions for acquisition order and show how staggered discrete acquisitions can produce smooth aggregate loss and, under certain geometries of quanta composition, give rise to scaling laws. We then train a Transformer to map numerals to English number names and recover candidate quanta from its checkpoint trajectory. From these units, we construct a model that preserves much of the Transformer's behavior while exposing interpretable latent computations and acquisition dynamics consistent with the theory. Separately, the quanta structure can serve as training targets to improve transformer generalization. Together, these results suggest the quanta abstraction can provide useful computational atoms for studying a variety of macroscopic phenomena.
Figures & tables
Figure 1: Examples of Quanta Descriptions of a Neural Model. A smooth loss curve can be decomposed into the staggered acquisition of discrete computations. Scaling with model size becomes growth of the acquired frontier, whose unmet demand yields a power law. An opaque neural network can be rewritten as a more interpretable Q-model which follows the quanta dynamics.
Figure 2: HSP construction. A length- 10 bit vector is read through sparse, local XORs and recursively composed by NAND. The learner receives only the vector, an opaque node identifier, and the target (Appendix A ).
Figure 3: Quantized loss dynamics in HSP. (a) In a flat HSP, individual held-out losses transition sharply at different times, while their weighted aggregate (red) is much smoother: its transition is 22.6× broader than the median task transition. Color runs from frequent (yellow) to rare (blue) tasks. (b) In a 32 -task composed HSP, one task (blue) pauses near the exact 1/2 -bit conditional level after acquiring its private constituent, then collapses when its shared constituent is acquired. Dashed lines mark the two 80% functional-mode crossings; other task losses are gray and red is their weighted aggregate.
Figure 4: Macroscopic effects of poset geometry. (a) At ratio 1.1 parents are acquired first and the direct-task and ancestor-closed loss reconstructions coincide, consistent with a compositional poset. (b) At ratio 5 , most children are acquired first and the direct-task reconstruction follows neural loss more closely, consistent with a shortcut interpretation. Curves show medians over three seeds; shading spans the range.
Figure 5: Fitted Scaling Laws. (a) parameter scaling across ten widths. Across the three prescribed demand exponents, fitted slopes match the theoretical prediction with 1.4% , 3.5% , and 5.7% relative error ( R2≈0.89 ). (b) online-IID data scaling at fixed width 2048 , plotted over the central learning regime between an initial and late plateau; D=BS is processed examples, with effective batch B and optimizer steps S . Fitted slopes have 9.9% , 26.7% , and 13.7% relative error from the prediction ( R2≈0.90 ). Each color denotes the demand exponent α ; parenthesized values give the fitted parameter exponent α or data exponent b . Solid lines are log–log fits, and dashed lines give the prescribed slopes ( α on the left and α/(1+α) on the right).
Figure 6: From a neural trajectory to an executable Q-model. (a) The third output step of the 346→ three, hundred, forty, six, EOS greedy trace. The source and Q-model both predict forty ; the bars show the Q-model’s top-token probabilities. Orange nodes mark five active quantum gates, but only four writers emit nonzero contributions: in the current Q-model, gate eligibility and executed writer activity can disagree. Four frozen held-out probe labels are shown. Solid arrows denote selected inter-quantum message channels; dotted lines show the layerwise residual aggregation. (b) Normalized cumulative population-gradient priority for the 13 frozen candidates in the selected 3% factorization. The dashed held-out NLL-improvement curve tracks their mean along this trajectory ( r=0.991 ). (c) Across a finer 22-factor diagnostic basis, later factors are less frequent and their priority-weighted source updates involve more effective MLP groups.
Training
Balanced
Zero gap
Short tail
CE only
84.9
61.7
50.5
+ Q target
94.7
89.0
89.9
Table 1: Q-guided training. Exact-sequence accuracy for a matched 10 -block Transformer. The zero-gap and short-tail columns contain 4,031 and 1,553 train-disjoint examples.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
α
Target
Full fit
Earlier range / slope
Later range / slope
0.20
0.1667
0.1832
2.2 – 4.2 M / 0.1882
4.3 – 8.4 M / 0.1072
0.25
0.2000
0.2534
1.9 – 3.5 M / 0.2838
3.6 – 6.7 M / 0.1532
0.339
0.2533
0.2879
1.5 – 3.0 M / 0.4339
3.1 – 6.2 M / 0.1164
Appendix
Table 2: Data-fit window sensitivity. Equal-log-span halves of the acquisition/plateau-selected interval; ranges are optimizer steps and every fit uses D=4096S .
Output
Active
Active
Total
Mean/min
Event interpretation
Qs
writers
activity
writer F1
three
4
6
39.600
0.812 / 0.741
sequence start, unit lexeme, value read
hundred
3
4
12.088
0.696 / 0.584
hundreds and low-chunk structure
forty
5
6
33.803
0.847 / 0.771
after-hundred, tail, value read/write
six
4
7
20.187
0.821 / 0.650
unit value and tail completion
EOS
7
3
6.095
0.794 / 0.684
termination and completed tail
Appendix
Table 3: Event summary for the 346 greedy trace. Counts and activity are exact for the displayed execution; writer F1 statistics use the frozen owner-conditional descriptions.
Output
Quantum
Held-out F1
Selected description
three
L0.Q3
0.615
unit lexeme or 5-digit input
three
L1.Q2
0.705
after sequence start or 3-digit input
three
L1.Q3
0.544
chunk has hundreds
three
L2.Q2
0.662
unit lexeme
hundred
L0.Q0
0.698
chunk has tail
hundred
L1.Q0
0.760
one-hundred rule or EOS decision
Appendix
Table 4: All active quanta for 346 . Each description is selected on calibration events and its F1 is evaluated unchanged on the disjoint semantic test.
Output
Writer
Activity
Conditional F1
Selected description
three
L0.Q3.W10
0.218
0.741
after sequence start
three
L1.Q2.W8
1.802
0.824
after sequence start or hundreds
three
L1.Q3.W9
0.359
0.783
unit form or after sequence start
three
L2.Q2.W6
14.219
0.882
value read
three
L2.Q2.W7
10.736
0.869
value read
three
L2.Q2.W8
12.266
0.771
emit value
Appendix
Table 5: All active writers for 346 . Activity is the gated contribution magnitude in this execution. F1 is held out and conditioned only on events where the writer’s owner quantum is active.
Figure 7: A second Q-model execution, 800 . Orange cells mark active quantum gates; the number inside is the count of active scalar writers in that gate’s three-writer bank. Blank cells are inactive gates. Each column is one greedily emitted token. The source and Q-model both match the target at all three steps.
Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.
St. Petersburg State University, Saint Petersburg 199034, Russia · Department of Software Engineering, Braude College of Engineering, Karmiel 2161002, Israel