Mapping the Solution-Set Structure of Grokking on Latin Squares

Zihan Gu; Lulu Zhao; Tianyi Ma; Hua Zhang; Wenbo Su; Yanan Cao; Yue Hu

cs.LGOct 26, 2026

Mapping the Solution-Set Structure of Grokking on Latin Squares

Authors: Zihan Gu, Lulu Zhao, Tianyi Ma, Hua Zhang, Wenbo Su, Yanan Cao, Yue Hu

Organizations: Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Alibaba Group · Shanghai Jiaotong University

Abstract

Grokking is a delayed transition from memorisation to generalisation. A network first fits observed examples while remaining near chance on held-out inputs, then generalises abruptly. Prior work has focused mainly on this delay, finding that the network's main loss becomes weight decay after the fitting phase. We name the solution to this optimization problem as the minimum-norm endpoint, and study which tasks make all minimum-norm endpoints generalise from partial observations. Specifically, we model pairwise input tasks as edge colourings of $K_{q,q}$ , with Latin colourings as our structured family, and classify each observed instance as all-good, mixed, or no-good by its endpoint set. Furthermore, we prove two contrasting results: (1) Finite-abelian-group families are asymptotically all-good from partial observations; (2) Under a uniform random Latin square, all-good tasks become asymptotically rare for observation-based learners, leaving mixed or no-good behaviour. Moreover, experiments show that relabellings of addition generalise despite their dense polynomial representations. Finally, on tasks of composite order, finite-training results suggest that predictors learn a coarse part of the output before the full label.

Figures & tables

Figure 1: Coordinates, endpoint sets, and the two poles. Left: $x+y\bmod 97$ ; a relabelling of it, $9{,}118$ terms and degree $190$ ; and a random-matching colouring outside the orbit. Middle: the minimum-norm interpolators $\mathcal{S}_{q}(\mathcal{D})$ agree on the observed cells, may differ elsewhere; their extremes $A_{q}^{-}\leq A_{q}^{+}$ cut $\tau$ into all-good , mixed , no-good . Right: each orbit maps to a simplex point; abelian group families sit at the all-good vertex (cyclic addition illustrated), while the uniform Latin ensemble sits on the opposite edge, where the mixed/no-good split is open.

Figure 2: Relabelled addition groks like addition.

	final outcome over $6$ runs
Latin family	trans.	med. acc.	${\geq}\,0.99$
$C_{2}^{6}$	$6/6$	$1.000$	$6/6$
$C_{64}$	$6/6$	$0.949$	$3/6$
$C_{8}{\times}C_{8}$	$6/6$	$0.941$	$2/6$
$\mathrm{Dih}_{64}$	$6/6$	$0.870$	$1/6$
$C_{8}{\times}R_{8}$	$4/6$	$0.231$	$1/6$

Table 1: Final trained predictors at

q{=}64

30\%

coverage, six runs per family. “Transition” is the plateau-relative event of Section D.1 ;

0.99

is a reporting cutoff unrelated to

\tau

Figure 3: Coarse factors are recovered before exact labels. Colour encodes label resolution: exact label and the prespecified

2

4

8

-way factor quotients. Left: held-out accuracy at nominal

30\%

coverage (runs small, medians outlined). Right: nested masks at

20\%

30\%

40\%

; coarse factors saturate one coverage level before the exact label.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$q,[q],S_{q},X_{q},N$	Task order, symbol set $\{0,\ldots,q-1\}$ , its symmetric group, cell grid $[q]^{2}$ , and $q^{2}$ .
$C,\mathcal{C}_{q},\mathcal{L}_{q},C_{q}^{\mathrm{add}}$	A colouring, all pairwise colourings, Latin colourings, and modular addition.
$G_{q},C_{G_{q}}$	A finite abelian group of order $q$ and its group table, after identifying its elements with $[q]$ .
$\Omega,T,\nu_{q},\alpha,n_{q}$	Observed cells, held-out cells, mask law, coverage, and mask size.
$\mathcal{D}(C,\Omega)$	Observed triples $(u,v,C(u,v))$ .
$F_{\theta},\theta,\mathcal{S}_{q}(\mathcal{D})$	Score map, network parameters, and minimum-norm endpoint set.

Appendix

Table 2: Notation used in the main text. Appendix-local symbols are defined where they enter.

order $q$	$47$	$79$	$113$	$211$
$\widehat{\alpha}_{q}$	$0.33$	$0.25$	$0.23$	$0.20$
grid cells $q^{2}$	$2209$	$6241$	$12769$	$44521$
observed cells $\lfloor\widehat{\alpha}_{q}q^{2}\rfloor$	$728$	$1560$	$2936$	$8904$

Appendix

Table 3: Empirical observation threshold for modular addition.

\widehat{\alpha}_{q}

is the smallest fraction on a grid of resolution

0.01

at which all five optimisation seeds reach exact held-out accuracy

1.000

. Cell counts are the nominal

\lfloor\widehat{\alpha}_{q}q^{2}\rfloor

of ( 1 ); the implementation rounds, so realised sizes can differ by one cell.

representation	tasks	nonzero coefficients	$(\deg_{X},\deg_{Y})$	total degree
visible addition	1	$2$	$(1,1)$	$1$
hidden addition	10	$9108$ – $9134$	$(95,95)$	$190$
sequential Latin	8	$9107$ – $9127$	$(95,95)$	$190$

Appendix

Table 4: Reduced-polynomial reconstruction at

q=97

. Support counts are out of

97^{2}=9409

coefficients.

family	seed	nonzero	support (%)	degrees	total	file
visible addition	0	2	0.021	$(1,1)$	1	visible_addition_task_000.csv
hidden addition	0	9118	96.907	$(95,95)$	190	hidden_addition_task_000.csv
hidden addition	1	9133	97.067	$(95,95)$	190	hidden_addition_task_001.csv
hidden addition	2	9121	96.939	$(95,95)$	190	hidden_addition_task_002.csv
hidden addition	3	9109	96.812	$(95,95)$	190	hidden_addition_task_003.csv
hidden addition	4	9134	97.077	$(95,95)$	190	hidden_addition_task_004.csv

Appendix

Table 5: Index of the machine-readable sparse coefficient archives. Each CSV stores the nonzero triples

(i,j,a_{ij})

over

\mathbb{F}_{97}

family	exact accuracy	prespecified binary factor quotient
$C_{2}\times C_{32}$	$0.9094$	$0.9977$
$C_{2}\times C_{4}\times R_{8}$	$0.5938$	$0.9922$
$C_{2}^{2}\times R_{16}$	$0.0165$	$0.9823$
$C_{2}\times R_{32}$	$0.00330$	$0.9816$
$R_{64}$	$0.00069$	—

Appendix

Table 6: Final medians at nominal

30\%

label-stratified coverage; six runs per family.

family	resolution	$20\%$	$30\%$	$40\%$
$C_{2}^{3}\times R_{8}$	binary factor quotient	$0.948$	$0.999$	$1.000$
	$4$ -way factor quotient	$0.846$	$0.998$	$1.000$
	$8$ -way factor quotient	$0.398$	$0.998$	$1.000$
	exact	$0.0169$	$0.882$	$0.985$
$C_{2}^{2}\times R_{16}$	binary factor quotient	$0.495$	$0.982$	$0.992$
	$4$ -way factor quotient	$0.344$	$0.960$	$0.984$

Appendix

Table 7: Final held-out accuracy under nested masks. A dash denotes a factor quotient absent from the factor chain.

excision	parameter	U-Acc (%)	GO-Epoch
square $i,j\leq k$	$k=30$	$99.04$	$1500$
	$k=40$	$84.50$	$1700$
	$k=50$	$77.39$	$3400$
	$k=60$	$57.31$	$2200$
strip $i\leq t$	$t=40$	$89.89$	$1300$
	$t=50$	$80.10$	$1600$

Appendix

Table 8: Excised training distributions at

p=97

, reproduced from Gu et al. (2025) . U-Acc is the accuracy ceiling after the transition, GO-Epoch the epoch at which it occurs.

Grokking

Grokking is delayed generalization in which neural networks memorize training data before abruptly learning a rule that works on unseen examples.

See all 81 papers

Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activations
Jul 15, 2026Chon-Fai Kam, Xavier Cadet, Miloud Bessafi +1Feedforward Neural Networks Neural Network Expressivity
University Paris City & University of Reunion, Paris, France · Dipartimento di Fisica e Chimica Emilio Segrè, Università degli Studi di Palermo, Palermo, Italy · Thayer School of Engineering, Dartmouth College, Hanover, NH 03755, USA +2
Where Does Jagged Competence Come From?
Oct 2, 2026Ioannis TsiokosNeural Network Generalization Compositional Representation Learning
Automorph Inc., Wilmington, DE, USA
A Basin-Selection Perspective on Grokking via Singular Learning Theory
Mar 1, 2026Ben Cullen, Sergio Estan-Ruiz, Riya Danait +1Grokking Neural Network Training Dynamics
Department of Computer Science, University of Pisa, Italy · Department of Mathematics, Imperial College London, United Kingdom · Mathematical Institute, University of Oxford, United Kingdom +1

Jul 15, 2026J/K move · Enter open · S save

Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activations

Chon-Fai Kam, Xavier Cadet, Miloud Bessafi, Frederic Cadet

University Paris City & University of Reunion, Paris, France · Dipartimento di Fisica e Chimica Emilio Segrè, Università degli Studi di Palermo, Palermo, Italy · Thayer School of Engineering, Dartmouth College, Hanover, NH 03755, USA+2

Feedforward Neural Networks Neural Network Expressivity Modular Arithmetic Grokking

Open paper AskPDF arXiv

Neural networks trained on modular arithmetic exhibit grokking, a delayed transition from memorisation to generalisation known to depend on model capacity: too little and the network memorises slowly or not at all, too much and it generalises almost immediately. What happens at the extreme of this spectrum, when the architecture's expressible function class collapses to a finite-dimensional algebraic variety? We study two-layer networks with a holomorphic monomial activation sigma(z)=z^k, trained on modular tasks encoded via roots of unity. Here the network output, regardless of hidden width, is confined to a (k+1)-dimensional subspace of characters of (Z_p)^2, an O(k/p^2) slice of the full function space. We give a complete algebraic characterisation of this subspace: a task is representable if and only if its discrete Fourier support lies on the diagonal u+v = k (mod p), which for linear-phase targets reduces to the arithmetic criterion m+n=k. This is not merely a constraint on eventual generalisation but on memorisation itself: because the outputs are algebraically confined, a non-representable target cannot be fit even on the training set, and we prove a positive lower bound on the training loss, independent of width. Across 585 runs the algebraic prediction matches the observed outcome with 99.8% accuracy, with no memorisation regime and no grokking; outcomes split cleanly into instant success and outright failure. This binary behaviour is the limiting case of the capacity-grokking relationship: when the expressible class shrinks to a fixed algebraic object, the question of when a network will grok dissolves into whether it can represent the target at all. A bottleneck ablation connects this extreme to standard networks, tracing a continuous path from representational failure, through memorisation without generalisation, to grokking with a shrinking gap as capacity grows.

Abstract

Grokking

Explore similar work

Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activations

Where Does Jagged Competence Come From?

A Basin-Selection Perspective on Grokking via Singular Learning Theory