The Geometry of Multi-Task Grokking: Transverse Instability, Superposition, and Weight Decay Phase Structure

Yongzhong Xu

cs.LGFeb 19, 2026

The Geometry of Multi-Task Grokking: Transverse Instability, Superposition, and Weight Decay Phase Structure

Authors: Yongzhong Xu

Abstract

Grokking -- the abrupt transition from memorization to generalization long after near-zero training loss -- has been studied mainly in single-task settings. We extend geometric analysis to multi-task modular arithmetic, training shared-trunk Transformers on dual-task (mod-add + mod-mul) and tri-task (mod-add + mod-mul + mod-sq) objectives across a systematic weight decay sweep. Five consistent phenomena emerge. (1) Staggered grokking order: multiplication generalizes first, followed by squaring, then addition, with consistent delays across seeds. (2) Universal integrability: optimization trajectories remain confined to an empirically invariant low-dimensional execution manifold; commutator defects orthogonal to this manifold reliably precede generalization. (3) Weight decay phase structure: grokking timescale, curvature depth, reconstruction threshold, and defect lead covary systematically with weight decay, revealing distinct dynamical regimes and a sharp no-decay failure mode. (4) Holographic incompressibility: final solutions occupy only 4--8 principal trajectory directions yet are distributed across full-rank weights and destroyed by minimal perturbations; SVD truncation, magnitude pruning, and uniform scaling all fail to preserve performance. (5) Transverse fragility and redundancy: removing less than 10% of orthogonal gradient components eliminates grokking, yet dual-task models exhibit partial recovery under extreme deletion, suggesting redundant center manifolds enabled by overparameterization. Together, these results support a dynamical picture in which multi-task grokking constructs a compact superposition subspace in parameter space, with weight decay acting as compression pressure and excess parameters supplying geometric redundancy in optimization pathways.

Figures & tables

$\lambda$	Grok (Add)	Grok (Mul)	Gap	Timescale
1.0	$13{,}333\pm 2{,}620$	$13{,}133\pm 2{,}748$	$200$	fast
0.5	$16{,}267\pm 685$	$16{,}200\pm 712$	$67$	fast
0.3	$28{,}467\pm 1{,}586$	$28{,}267\pm 1{,}190$	$200$	intermediate
0.2	$45{,}167\pm 1{,}307$	$45{,}067\pm 1{,}184$	$100$	intermediate
0.1	$97{,}967\pm 3{,}172$	$98{,}833\pm 2{,}001$	$-866$	slow
0.0	—	—	—	(never)

Table 1: Dual-task grokking steps (mean

\pm

std across 3 seeds). Multiplication consistently leads.

$\lambda$	Seed	Grok (Add)	Grok (Mul)	Grok (Sq)
1.0	42	30,500	17,900	24,800
	137	19,000	15,700	17,500
	2024	13,900	12,300	13,000
0.5	42	27,000	24,800	26,200
	137	26,800	25,900	26,800
	2024	25,000	24,300	24,300

Table 2: Tri-task grokking steps (per-seed). The ordering mul

\to

\to

add is consistent.

Panel of Figure 1 — Figure 1: Multi-task grokking dynamics. (a) Dual-task: multiplication leads addition. (b) Tri-task: a three-way staggered ordering emerges.

Panel of Figure 2 — Figure 2: PC1% decreases with task count. (a) Dual-task: 55–77%. (b) Tri-task: 49–56%. The manifold is no longer rank-1 but remains strongly low-dimensional.

Panel of Figure 3 — Figure 3: PC1% declines over training in multi-task settings, unlike single-task grokking where concentration increases.

Figure 4: Grok (WD=1.0) vs. no-WD (WD=0.0) eigenspectra for tri-task (seed 42). No-WD has higher PC1%, consistent with a simpler memorization trajectory.

Panel of Figure 5 — Figure 5: Task-specific head weights are nearly orthogonal. The shared trunk learns a representation where task readouts are geometrically separated.

Panel of Figure 6 — Figure 6: The execution manifold is empirically integrable in multi-task settings. The invariance measure $\rho\approx 1.000$ across all tested conditions, extending the single-task result.

Panel of Figure 7 — Figure 7: Cross-task gradient structure. (a) Same-task cosines are high ( ${\sim}0.8$ ), cross-task cosines are moderate ( ${\sim}0.3$ ). (b) Cross-task defect has roughly the same magnitude as total-loss defect.

Panel of Figure 8 — Figure 8: Hessian curvature depth scales with weight decay. (a) Time series of $\lambda_{\text{min}}$ for three WD values. (b) Grok time vs. curvature depth across WD, with $\lambda=0$ as the decoupled outlier.

Seed	$\lambda_{\text{min}}(\text{mul})$	$\lambda_{\text{min}}(\text{add})$	Ratio	First to grok
42	$\mathbf{-35.0}$	$-22.8$	1.54 $\times$	mul
137	$-17.4$	$\mathbf{-22.9}$	0.76 $\times$	mul
2024	$\mathbf{-35.1}$	$-21.4$	1.64 $\times$	mul

Table 3: Per-task Hessian eigenvalues at mid-training (dual-task, WD=1.0). The task with deeper

\lambda_{\text{min}}

tends to grok first (2/3 seeds).

Figure 9: Per-task bottom Hessian eigenvalues (seed 42, WD=1.0). Mul (blue) has deeper negative curvature early and recovers first, consistent with its earlier grokking.

$\lambda$	Grok (Add)	Grok (Mul)	$\lambda_{\text{min}}$ (total)	$\lambda_{\text{min}}$ (add)	$\lambda_{\text{min}}$ (mul)
1.0	$13{,}333\pm 2{,}620$	$13{,}133\pm 2{,}748$	$-63.4\pm 11.9$	$-50.8\pm 5.6$	$-43.8\pm 2.2$
0.5	$16{,}267\pm 685$	$16{,}200\pm 712$	$-35.7\pm 6.1$	$-21.2\pm 3.0$	$-33.8\pm 18.4$
0.1	$97{,}967\pm 3{,}172$	$98{,}833\pm 2{,}001$	$-17.4\pm 7.9$	$-11.8\pm 0.7$	$-11.8\pm 4.2$
0.0	—	—	$-49.5\pm 15.6$	$-53.8\pm 36.3$	$-40.7\pm 12.9$

Table 4: Multi-seed Hessian summary (dual-task, 3 seeds

\times

4 WD values = 12 runs).

Panel of Figure 10 — Figure 10: Tri-task Hessian analysis replicates the dual-task pattern: stronger WD drives deeper negative curvature.

Figure 11: Grokking timescale vs. weight decay (dual-task, 3 seeds, log scale). The relationship is nearly log-linear for $\lambda>0$ , with a sharp jump between $\lambda=0.5$ and $\lambda=0.1$ .

$\lambda$	Mean Grok Step	Mean Onset Step	Mean Lead (steps)	Lead %
1.0	13,133	6,600	6,533	52.9%
0.5	16,200	15,400	800	4.9%
0.3	28,200	19,133	9,067	32.0%
0.2	45,067	17,667	27,400	60.8%
0.1	97,933	29,467	68,467	69.9%

Table 5: Defect onset lead over grokking (dual-task, mean across 3 seeds per WD).

$\lambda$	Mean Grok Step	Mean Onset Step	Mean Lead (steps)	Lead %
1.0	12,089	1,833	8,900	82.2%
0.5	25,333	5,767	18,800	76.5%
0.1	152,356	5,000	147,144	96.7%

Table 6: Tri-task defect onset lead over grokking (mean across 3 tasks

\times

3 seeds = 9 conditions per WD).

Panel of Figure 12 — Figure 12: Tri-task defect onset always precedes grokking ( $27/27$ , $p=2^{-27}$ ). Unlike the dual-task V-shape, the tri-task lead fraction increases monotonically with decreasing WD, reaching ${\sim}97\%$ at $\lambda=0.1$ .

Panel of Figure 13 — Figure 13: Dual-task defect onset always precedes grokking (15/15 conditions), with a non-monotonic lead fraction (minimum at $\lambda=0.5$ ).

Panel of Figure 14 — Figure 14: V-shaped lead fraction across weight decay (dual-task only). (a) Per-seed lead times: $\lambda=0.5$ has the smallest lead ( ${\sim}5\%$ ), with larger leads at both higher and lower WD. (b) All 15 dual-task defect traces on a common normalized time axis.

Figure 15: Defect traces across five WD values (seed 42). $\lambda\geq 0.5$ : smooth monotonic rise. $\lambda\leq 0.3$ : intermittent spiky pattern with transient explosions.

$\lambda$	Final Defect Range	Integrability ( $\rho$ )	Grad Cosine (same-task)
1.0	82–144	1.000	0.77–0.84
0.5	837–1,288	1.000	0.47–0.76
0.1	${\sim}0$	1.000	0.14–0.49
0.0	0–52	varies	0.04–0.42

Table 7: Dual-task final-checkpoint defect and integrability across WD.

$\lambda$	Seed 42	Seed 137	Seed 2024	Mean	Var. at $k^{*}$
1.0	7	6	6	6.3	99.7–99.9%
0.5	5	5	5	5.0	99.6–99.7%
0.3	5	6	6	5.7	99.4–99.7%
0.2	6	10	7	7.7	99.5–99.7%
0.1	9	7	9	8.3	99.2–99.6%

Table 8: Reconstruction threshold

k^{*}

for dual-task (both tasks

>90\%

Panel of Figure 16 — Figure 16: Reconstruction threshold. The grokking solution requires 5–10 PCA directions, with $k^{*}$ increasing at lower WD.

$\lambda$	Seed 42	Seed 137	Seed 2024	Mean
1.0	5	5	3	4.3
0.5	4	4	5	4.3
0.3	7	5	7	6.3
0.2	5	5	6	5.3
0.1	5	5	8	6.0

Table 9: Reconstruction threshold

k^{*}

for tri-task (all three tasks

>90\%

Panel of Figure 17 — Figure 17: The grokking solution is incompressible by any post-hoc method. Only trajectory-informed PCA succeeds.

Delete %	Add Grok	Mul Grok	Delay
0%	14,100	14,000	(baseline)
1%	18,100	18,100	$+29\%$
2%	21,200	20,900	$+50\%$
3%	23,500	23,300	$+67\%$
5%	37,000	36,800	$+163\%$
7%	42,600	42,400	$+203\%$

Table 10: Dual-task orthogonal deletion dose-response (WD=1.0, seed 42, 50k step budget).

Delete %	Add Grok	Mul Grok	Sq Grok	Mean	Delay
0%	25,600	18,200	23,400	22,400	(baseline)
1%	35,200	21,200	28,800	28,400	$+27\%$
2%	42,400	30,200	36,000	36,200	$+62\%$
3%	35,000	29,000	33,600	32,500	$+45\%$
5%	35,400	26,800	33,800	32,000	$+43\%$
7%	56,000	44,600	49,600	50,100	$+124\%$

Table 11: Tri-task orthogonal deletion dose-response (WD=1.0, seed 42, 60k step budget).

Figure 18: Orthogonal deletion dose-response for both tri-task and dual-task. Grok delay increases monotonically with deletion fraction until a sharp cliff at ${\sim}10\%$ , beyond which grokking fails entirely. X-marks indicate failure to grok within the step budget.

Figure 19: Phase portrait for tri-task arithmetic (seed 42, layer 0) in the $(\sigma_{1}-\sigma_{2},\;\left\|[W_{Q},W_{K}]\right\|_{F})$ plane. Left: colored by training step. Right: colored by mean test accuracy across the three tasks. The three tasks grok at different positions along a single shared trajectory: multiplication (step 17,900) near the commutator peak, $a^{2}\!+\!b^{2}$ (step 24,800) during descent, and addition (step 30,500) in the alignment region. The competition–instability–alignment structure is preserved from the single-task case.

Figure 20: Grokking vs. memorizing phase portraits for dual-task arithmetic (modular addition + multiplication, seed 42, layer 0). Left: grokking run ( $\mathrm{wd}=1$ ) shows the characteristic competition–instability–alignment loop, with both tasks grokking near the transition to alignment. Right: memorizing run ( $\mathrm{wd}=0$ ) shows a diffuse random walk with $\sim\!3\times$ higher commutator values and no directed escape.

Dataset	Seed	L0 peak	L1 peak	Ratio
Dual-task (add+mul)	42	23.7	14.3	1.7 $\times$
	137	20.5	12.2	1.7 $\times$
	2024	21.3	13.1	1.6 $\times$
Tri-task (add+mul+sq)	42	24.7	11.3	2.2 $\times$
	137	27.3	11.6	2.4 $\times$
	2024	24.6	12.8	1.9 $\times$

Table 12: Layer-wise peak commutator magnitude. Layer 0 (bottom) consistently develops

1.3

–

2.4\times

stronger non-commutativity than Layer 1 (top).

Figure 21: Layer-wise phase portrait overlay for tri-task arithmetic (seed 42). Layer 0 (blue) traces a large loop reaching peak commutator $\approx 24.7$ at step 17,600, while Layer 1 (orange) remains below $\approx 11.3$ . The bottom layer bears the dominant geometric instability. The tri-task amplifies the asymmetry relative to the single-task case ( $2.2\times$ vs. $1.1\times$ ).

Setting	$\boldsymbol{\lambda}$	Grok step	$g_{23}^{\mathrm{early}}$	$g_{23}^{\mathrm{grok}}$	Decline	$R_{\mathrm{early}}$	$k^{*}_{\mathrm{term}}$	Decl.
Single-task (mean $\pm$ std over 3 seeds)
add	1.0	2500–3100	$14.6\pm 0.2$	$0.56\pm 0.4$	$50\times$	$1.40\pm 0.05$	1	✓
	0.0	—	$33.0\pm 9.3$	—	—	$2.72\pm 0.22$	—	—
mul	1.0	2600–3000	$14.7\pm 0.1$	$0.39\pm 0.2$	$49\times$	$1.41\pm 0.05$	1	✓
	0.0	—	$23.6\pm 10.7$	—	—	$2.85\pm 0.11$	—	—
sub	1.0	3300–3700	$14.4\pm 0.1$	$0.48\pm 0.2$	$35\times$	$1.31\pm 0.02$	1	✓

Table 13: Rolling-window Gram matrix spectral analysis (

W=10

g_{23}^{\mathrm{early}}

: peak

g_{23}

before grokking;

g_{23}^{\mathrm{grok}}

g_{23}

at the grokking step; Decline

=g_{23}^{\mathrm{early}}/g_{23}^{\mathrm{grok}}

;

R

: gap ratio at signal rank

k^{*}

. Single-task operations that do not grok within

2\times 10^{5}

steps (x2_xy_y2, x3_xy) are omitted.

Panel of Figure 22 — Figure 22: Constraint-induced compression. (a) Dual-task models require $k^{*}=5$ – $10$ PCA directions depending on WD, with the orthogonal complement carrying 0.2–0.8% of trajectory variance. (b) Tri-task models require fewer components ( $k^{*}=3$ – $8$ ), consistent with stronger superposition pressure from additional task constraints. See Tables 15 and 14 for per- $k$ accuracy breakdowns.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 23 — Figure 23: Top-10 eigenspectra for dual-task and tri-task. Both show a dominant first eigenvalue, but the gap is smaller in multi-task settings.

Panel of Figure 24 — Figure 24: SVD of weight deltas. Top-5 singular values grow concurrently in multi-task settings, with head-specific SVDs showing growth patterns at different timings consistent with staggered grokking.

Panel of Figure 25 — Figure 25: Commutator defect time series for both multi-task settings.

Figure 26: Tri-task defect vs. per-task test accuracy across three WD values ( $\lambda=1.0,0.5,0.1$ ), 3 seeds overlaid per panel. Black curves (left axis): commutator defect on log scale. Colored curves (right axis): per-task test accuracy (blue = add, red = mul, green = sq). Shaded bands mark the grok region for each task. Dotted verticals mark defect spike detection. The defect spike clearly precedes grokking in all panels, with the temporal gap increasing at lower WD.

Figure 27: Hero figure: defect predicts grokking in the tri-task setting (add task, $\lambda=0.1$ , seed 42). Blue: commutator defect (left axis, log scale) with IQR ribbon. Red: test and train accuracy (right axis). The defect spike occurs at step $3{,}700$ , while 90% test accuracy is reached at step $162{,}900$ —a lead time of $159{,}200$ steps ( $97.7\%$ of the grokking time).

Panel of Figure 28 — Figure 28: Combined defect magnitude and integrability over training. Integrability remains at 1.000 throughout while defect varies.

Figure 29: Hessian curvature scaling across WD values (multi-seed). Curvature depth increases monotonically with WD in the grokking regime.

Figure 30: Tri-task cross-WD defect summary. WD=0.5 shows the largest final defect (analysis captures the model mid-transition), while WD=0.1 collapses to zero post-grokking.

Figure 31: Dual-task PCA eigenspectrum across weight decay values. The top 5–10 eigenvalues capture $>$ 99% of trajectory variance; the remaining eigenvalues form a rapidly decaying tail. The number of non-negligible components increases at lower WD, consistent with the $k^{*}$ trend reported in Section 10.5 . See Figure 33 for the corresponding reconstruction heatmaps.

Panel of Figure 32 — Figure 32: Detailed comparison of reconstruction threshold $k^{*}$ between dual-task and tri-task settings (companion to Figure 22 in the main text). (a) Dual-task models require more PCA directions; the chance-to-perfect transition is visible as a sigmoid-like jump. (b) Tri-task models exhibit the same sharp transition but shifted leftward to lower $k$ , indicating stronger constraint-induced compression. The systematic $k^{*}_{\text{dual}}>k^{*}_{\text{tri}}$ ordering holds across all five WD values and three random seeds (see Tables 15 and 14 for numerical details).

Panel of Figure 33 — Figure 33: Heatmap comparison of reconstruction accuracy across WD and $k$ for dual-task (a) and tri-task (b) . The boundary between chance-level (dark) and near-perfect (bright) accuracy occurs at lower $k$ for the tri-task setting, providing a visual summary of constraint-induced compression across the full WD phase diagram.

Figure 34: Dual-task gradient projection ablation (averaged across 3 seeds, WD=1.0). PCA projection at strength $s=0.25$ delays grokking by 70–140%, while random projection at the same strength has no effect.

$\lambda=1.0$ ( $k^{*}=5$ )					$\lambda=0.5$ ( $k^{*}=4$ )
$k$	CumVar	Add	Mul	Sq	$k$	CumVar	Add	Mul	Sq
1	91.7%	20.4	36.7	47.5	1	93.9%	9.9	18.0	57.4
2	95.9%	5.6	8.2	7.5	2	97.1%	25.0	49.3	48.1
3	98.0%	74.4	89.2	94.8	3	98.8%	87.5	98.5	96.1
4	98.8%	72.5	76.2	87.7	4	99.3%	91.8	96.0	93.0
5	99.3%	96.5	99.9	99.3	5	99.5%	96.3	99.5	98.9

Appendix

Table 14: Tri-task reconstruction accuracy vs. number of PCA components

k

(seed 42). Each sub-table shows one weight decay value. CumVar is cumulative variance explained. Bold row =

k^{*}

(first

k

where all tasks

>90\%

). PC1 captures 89–94% of variance but carries no generalization capability.

$\lambda=1.0$ ( $k^{*}=7$ )				$\lambda=0.5$ ( $k^{*}=5$ )
$k$	CumVar	Add	Mul	$k$	CumVar	Add	Mul
1	94.9%	3.5	7.4	1	96.8%	4.5	5.3
2	97.0%	2.4	5.4	2	98.2%	4.5	8.9
3	98.5%	36.1	35.3	3	98.8%	39.1	57.7
4	99.0%	21.9	19.7	4	99.5%	69.6	79.5
5	99.4%	68.0	83.5	5	99.6%	92.5	95.5

Appendix

Table 15: Dual-task reconstruction accuracy vs. number of PCA components

k

(seed 42). Same format as Table 14 . The dual-task transition is equally sharp but requires more components (

k^{*}=5

–

9

) than tri-task (

k^{*}=3

–

8

$\alpha$ (%)	Add Grok	Mul Grok	Mean	$\Delta t$ (steps)	Delay
0 (baseline)	14,100	14,000	14,050	—	—
1	18,100	18,100	18,100	$+4{,}050$	$+29\%$
2	21,200	20,900	21,050	$+7{,}000$	$+50\%$
3	23,500	23,300	23,400	$+9{,}350$	$+67\%$
5	37,000	36,800	36,900	$+22{,}850$	$+163\%$
7	42,600	42,400	42,500	$+28{,}450$	$+203\%$

Appendix

Table 16: Dual-task transverse gradient ablation: detailed results (WD=1.0, seed 42, 50k step budget). Columns show per-task grokking steps, the mean across tasks, absolute delay, and percentage delay. The sharp cliff at

\alpha=0.10

and non-monotonic recovery at

\alpha=0.50

are clearly visible.

$\alpha$ (%)	Add Grok	Mul Grok	Sq Grok	Mean	$\Delta t$ (steps)	Delay
0 (baseline)	25,600	18,200	23,400	22,400	—	—
1	35,200	21,200	28,800	28,400	$+6{,}000$	$+27\%$
2	42,400	30,200	36,000	36,200	$+13{,}800$	$+62\%$
3	35,000	29,000	33,600	32,500	$+10{,}100$	$+45\%$
5	35,400	26,800	33,800	32,000	$+9{,}600$	$+43\%$
7	56,000	44,600	49,600	50,100	$+27{,}700$	$+124\%$

Appendix

Table 17: Tri-task transverse gradient ablation: detailed results (WD=1.0, seed 42, 60k step budget). Same format as Table 16 . The tri-task setting shows the same fragility cliff at

\alpha=0.10

but no recovery at any higher deletion level.

Explore similar work

Feb 18, 2026

Low-Dimensional and Transversely Curved Optimization Dynamics in Grokking

Grokking -- the delayed transition from memorization to generalization in small algorithmic tasks -- remains poorly understood. We present a geometric analysis of optimization dynamics in transformers trained on modular arithmetic. PCA of attention weight trajectories reveals that training evolves predominantly within a low-dimensional execution subspace, with a single principal component capturing 68-83% of trajectory variance. To probe loss-landscape geometry, we measure commutator defects -- the non-commutativity of successive gradient steps -- and project them onto this learned subspace. We find that curvature grows sharply in directions orthogonal to the execution subspace while the trajectory remains largely confined to it. Importantly, curvature growth consistently precedes generalization across learning rates and hyperparameter regimes, with the lead time obeying a power law in the grokking timescale. Causal intervention experiments show that motion along the learned subspace is necessary for grokking, while artificially increasing curvature is insufficient. Together, these results support a geometric account in which grokking reflects escape from a metastable regime characterized by low-dimensional confinement and transverse curvature accumulation. All findings replicate across this learning-rate range, a qualitatively different slow regime (lr=5e-5, wd=0.1, 3 layers), and three random seeds, though alignment dynamics differ quantitatively between regimes. Causal intervention experiments establish that orthogonal gradient flow is necessary but not sufficient for grokking: suppressing it prevents generalization with a monotonic dose-response across four operations, while artificially boosting curvature defects has no effect.

Yongzhong Xu

Grokking Neural Network Training Dynamics Gradient Descent Dynamics Loss Landscape Geometry

Ask PDF

Feb 19, 2026

Early-Warning Signals of Grokking via Loss-Landscape Geometry

Grokking -- the abrupt transition from memorization to generalization after prolonged training -- has been linked to confinement on low-dimensional execution manifolds in modular arithmetic. Whether this mechanism extends beyond arithmetic remains open. We study two sequence-learning benchmarks: SCAN compositional generalization and Dyck-1 depth prediction. Across both tasks and a wide range of learning rates, the commutator defect -- a curvature measure derived from non-commuting gradient updates -- rises well before generalization, with lead times following a superlinear power law (alpha approximately 1.18 for SCAN, approximately 1.13 for Dyck), consistent with prior results on modular arithmetic. Weight-space PCA reveals that spectral concentration is not a universal precursor; the commutator defect is. Causal interventions demonstrate a mechanistic role: amplifying non-commutativity accelerates grokking (roughly 32% on SCAN, roughly 50% on Dyck), while suppressing orthogonal gradient flow delays or prevents it. The three task families form a spectrum of causal sensitivity -- modular arithmetic is rigid, Dyck is responsive, SCAN is intermediate -- yet suppression delays or prevents grokking in all cases, establishing necessity as a universal finding. These results identify the commutator defect as a robust, architecture-agnostic, causally implicated early-warning signal for delayed generalization in transformers.

Yongzhong Xu

Grokking Neural Network Generalization Loss Landscape Geometry

Ask PDF

Jul 6, 2026

Grokking Is Conditional and Fragile: A Fully-Tractable, Multi-Seed Study at 12K Parameters

Grokking -- the delayed onset of generalization long after a network has fit its training set - -is usually studied in models too large to read completely and reported from single training runs. We instead study a publicly released ~11,856-parameter Llama-style transformer (Glimmer-1-Base) on modular arithmetic, small enough to enumerate its weights, attention, and full input-output map, and we measure grokking as a multi-seed rate rather than a single outcome. In this fully-tractable regime grokking is a conditional, fragile phase transition. It is gated by training-set coverage, whose threshold tracks output cardinality (the modulus) more than task structure, an ordering that holds above the transition and across a ten-fold change in domain size. Weight decay reproduces the Omnigrok inverted-U at 12K parameters, a positive control on the rate measurement. Grokking also sits on a numerical knife-edge: two perturbations of the floating-point environment -- CPU thread count (reduction order) and CPU-versus-GPU execution -- each flip a minority of same-seed outcomes without a detectable shift in the aggregate rate. Decomposition into sub-task specialists helps chiefly by making coverage cheap rather than by adding supervision. Methodologically, multi-seed control under a fixed numerical environment overturns three dramatic single-run narratives in our own data, each a seed confound. The unit of evidence for grokking must therefore be a multi-seed rate under a pinned numerical environment, checked where possible against a direct reading of the model.

Yoshiyuki Ootani

Independent Researcher

Phase Transitions ML Reproducibility Modular Arithmetic Grokking

Ask PDF