Percolation Dynamics in Optimization: Variance Cascades and Nested Symmetry

Sai Niranjan Ramachandran; Suvrit Sra

cs.LGSep 2, 2026

Percolation Dynamics in Optimization: Variance Cascades and Nested Symmetry

Authors: Sai Niranjan Ramachandran, Suvrit Sra

Organizations: School of Computation, Information and Technology, Technical University of Munich, Germany · Munich Center for Machine Learning (MCML)

Abstract

We study the dynamics of Stochastic Gradient Descent (SGD), which is known to steer deep neural networks toward invariant sets that correspond to simpler subnetworks. How this steering unfolds over time remains poorly understood. We answer this by modeling the stochastic gradient flow (SGF) as a percolation process, in which nested architectural symmetries force subnetworks to merge in discrete blocks rather than by single-edge attachment. These structural transitions register as variance spikes in a macroscopic order parameter echoing physical phase transitions. We further state sufficient conditions under which the trapping argument carries over to Adam and AdamW under heavy-tailed gradient noise and measure them on a trained Transformer.

Figures & tables

Figure 1: Block merges under nested symmetry. Binary tree (solid) and flat layer (dashed) at $N=16$ and $32$ over $32$ seeds. (a) Only the tree reaches a single cluster. (b) Jumps of $+16$ , merging two half-trees, occur only in the tree. (c) Hitting times per level are evenly spaced under weight decay.

Figure 2: Grokking (modular arithmetic). Delayed generalization co-occurs with dimensionality collapse and a log-linear cascade of variance peaks (epochs $4500$ , $9500$ , $20000$ ; $\lambda_{t}=2.11$ ). Other datasets show similar cascades (Appendix F ).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Experimental validation of symmetry-induced topological collapse and the scaling of peak times. Top Row (Experiment 1): Kinematic simulation of hierarchical coalescence ( $N=3$ ), displaying discrete jumps in the order parameter (a), peaks in relative variance exactly at the imposed merge points (b), and forced continuous spline trajectories (c). Middle Row (Experiment 2): Empirical SGD dynamics ( $N=6$ ), showing macroscopic dimensionality collapse (a), distinct precursory variance peaks $p_{1},p_{2}$ during the condensation phase, and a reactive fragmentation peak $p_{3}$ following a task shift at $t=3000$ (b), alongside the raw stochastic weight trajectories (c). Bottom Row: Log-linear progressions of peak times: Experiment 1 (Left) recovers the imposed ratio $\lambda_{t}=2.00$ , and Experiment 2 (Right) gives the two-peak ratio $\lambda_{t}\approx 2.00$ ; the finite- $N$ prediction for the first two merges of $N=6$ units is $2.5$ (Section E ).

Arm	Clusters before $\to$ after	Fragmenting seeds	Post/pre $\max\mathcal{R}_{v}$	Maximum at
Target and noise (Fig. 3 )	$1.93\to 2.00$	$0.12$	$1.19$	$t=4995$
Noise only	$1.93\to 2.08$	$0.10$	$0.87$	$t=3045$
Target only	$1.93\to 2.11$	$0.30$	$1.61$	$t=4155$
Control	$1.93\to 1.95$	$0.03$	$1.05$	$t=4155$

Appendix

Table 1: Task-shift ablation (

N=6

,

40

realizations, common random numbers; interventions at

t=3000

). Fragmenting seeds end with at least half a cluster more, on average over the last

300

steps, than just before the shift.

	Tree $N=16$	Flat $N=16$	Tree $N=32$	Flat $N=32$
Full collapse (share of seeds)	$19\%$	$0\%$	$47\%$	$0\%$
Final mean cluster count	$3.6$	$2.3$	$1.7$	$2.3$
Single-attachment share	$0.74$	$0.68$	$0.65$	$0.60$
Exact-doubling share	$0.22$	$0.11$	$0.11$	$0.03$
Macroscopic jumps ( $\geq N/4$ ) share	$0.09$	$0.05$	$0.05$	$0.02$
Largest jump observed	$+4$	$+5$	$+16$	$+12$

Appendix

Table 2: Growth of the record largest component under SGD for the two architectures (

32

seeds,

6000

SGD steps). “Exact doubling” is the share of record-growth events with

|C_{\max}|_{\mathrm{after}}=2\,|C_{\max}|_{\mathrm{before}}

, and “macroscopic” is the share with jump

\geq N/4

. Hitting ratios omit the first level for the flat layer at

N=32

, where one pair is inside the tube at initialization (

T_{2}=18

).

Task	Model	Monitored matrix	Optimization	Data
Heart Disease, German Credit, Abalone, Digits	$d_{\mathrm{in}}\to 64\to 32\to C$ , GELU	first layer ( $64\times d_{\mathrm{in}}$ )	SGD, lr $0.05$ , batch $32$ , $100$ epochs	OpenML heart-statlog , credit-g , abalone (rings $\leq 8$ , $9$ – $10$ , $\geq 11$ ); scikit-learn digits
Moons, Swiss Roll	$d_{\mathrm{in}}\to 64\to 32\to 2$ , GELU	first layer ( $64\times d_{\mathrm{in}}$ )	SGD, lr $0.1$ , batch $64$ , $100$ epochs	$2000$ samples, noise $0.1$
MNIST, FashionMNIST	$784\to 128\to 64\to 10$ , GELU	first layer ( $128\times 784$ )	SGD, lr $0.05$ , batch $128$ , $50$ epochs	$5000$ -image subset
Modular arithmetic	1 layer, $d=128$ , 4 heads, FF $512$ , GELU, dropout $0.1$ , no LayerNorm	token embedding ( $18\times 128$ )	AdamW, lr $3\times 10^{-3}$ , $\lambda=0.5$ , batch $128$ , $30000$ epochs	$(a+b)\bmod 17$ , input $[a,b,=]$ , $70/30$ split ( $202/87$ )

Appendix

Table 3: Settings of the empirical suite. Categorical features are one-hot encoded (first level dropped) and tabular features standardized; models are trained on the full dataset, since the observable is the training dynamics. UCI runs use

20

initializations and the other tasks

15

. UCI peak detection uses macro

\sigma=20

, local

\sigma=1.5

and minimum peak distance

8

; the other tasks use macro

\sigma=E/5

, local

\sigma=E/50+1

and distance

E/20

for

E

epochs; the prominence is

0.05

throughout.

Dataset	Peaks	$\lambda_{t}$	$R^{2}$	FPR (%)
Heart Disease	4	1.71	0.98	4.9%
Abalone	6	1.28	0.97	15.6%
German Credit	4	1.99	0.86	80.2%
Digits	1	N/A	N/A	N/A

Appendix

Table 4: Empirical peak-time cascades and statistical null-model evaluations for UCI tabular datasets (aggregated over

20

independent initializations). A cascade fit needs at least three peaks, so none is reported for Digits.