cs.LGMar 2, 2026

Intrinsic Task Symmetry Drives Generalization in Algorithmic Tasks

Authors: Hyeonbin Hwang, Yeachan Park

Organizations: KAIST · Sejong University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Grokking, the sudden transition from memorization to generalization, is characterized by the emergence of low-dimensional representations, yet the mechanism underlying this organization remains elusive. We propose that intrinsic task symmetries primarily drive grokking and shape the geometry of the model's representation space. We identify a consistent three-stage training dynamic underlying grokking: (i) memorization, (ii) symmetry acquisition, and (iii) geometric organization. We show that generalization emerges during the symmetry acquisition phase, after which representations reorganize into a structured, task-aligned geometry. We validate this symmetry-driven account across diverse algorithmic domains, including algebraic, structural, and relational reasoning tasks. Building on these findings, we introduce a symmetry-based diagnostic that anticipates the onset of generalization and propose strategies to accelerate it. Together, our results establish intrinsic symmetry as the key factor enabling neural networks to move beyond memorization and achieve robust algorithmic reasoning.

Figures & tables

Figure 1: Modular Arithmetics Embeddings
Figure 2: Graph Metric Completion and Comparison Embeddings.
Figure 3: Type of Graphs.
Figure 4: Full PCA visualization of modular addition embeddings.
Figure 5: Symmetric patterns within a dataset lead the network to learn intrinsic symmetry.
Figure 6: Three-stage dynamic of grokking for modular addition task. (1) Memorization : The model first memorizes as it quickly reaches perfect train accuracy. (2) Symmetry Acquisition Then with the symmetry criterion decreasing to certain threshold, perfect test accuracy is reached. (3) Geometric Organization Finally, as the representation collapses into lower-dimension, embedding forms a geometric organization.
Figure 7: Symmetry violation metric vs. test accuracy across different seeds. A threshold in symmetry violation separates Solved from Unsolved runs, supporting symmetry alignment as a predictor of generalization.
Figure 8: We train a two-layer MLP on modular addition using an 80% training fraction, without weight decay.
Figure 9: Comparison of convergence speed on Modular Addition, Graph Completion (Lattice), and Comparsion (2D) task with Geometry-promoting priors ( nuclear , entropy , Lipschitz ); Symmetry-prompting losses ( commutativity / associativity /etc); Optimizer-based accelerants ( GrokFast ). Markers denote the first point where test accuracy reaches 100%. Consistency refers to using both Commutativity and Associativity.
Appendix figures & tables46 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Grokking Dynamics in Modular Addition.
Figure 11: Grokking Dynamics in Modular Subtraction.
Figure 12: Grokking Dynamics in Modular Squared Sum
Figure 13: Grokking Dynamics in Modular Cubed Sum
Figure 14: Grokking Dynamics in Modular Multiplication
Figure 15: Grokking Dynamics in Modular Affine–bilinear
Figure 16: Grokking Dynamics in Graph Metric Completion: Cycle
Figure 17: Grokking Dynamics in Graph Metric Completion: Path
Figure 18: Grokking Dynamics in Graph Metric Completion: Cylinder
Figure 19: Grokking Dynamics in Graph Metric Completion: Hypercube
Figure 20: Grokking Dynamics in Graph Metric Completion: Lattice
Figure 21: Grokking Dynamics in Graph Metric Completion: 3D Lattice
Figure 22: Grokking Dynamics in 2D Comparison
Figure 23: Grokking Dynamics in 3D Comparison
Figure 24: Maximum Symmetry Violation and Generalization Status for modular arithmetic tasks.
Figure 25: Symmetry Loss vs. Test Accuracy for Graph Metric Completion Tasks
Figure 26: Symmetry Loss vs. Test Accuracy for Comparison Tasks
Figure 27: PCA projection of embeddings of model trained on modular addition task.
Figure 28: PCA projection of embeddings of model trained on modular subtraction task.
Figure 29: PCA projection of embeddings of model trained on Task 3.
Figure 30: PCA projection of embeddings of model trained on Task 4.
Figure 31: PCA projection of embeddings of model trained on modular multiplication task.
Figure 32: PCA projection of embeddings of model trained on Task 6.
Figure 33: PCA projection of embeddings of model trained on the Path graph metric completion.
Figure 34: PCA projection of embeddings of model trained on the Cycle graph metric completion.
Figure 35: PCA projection of embeddings of model trained on the Cylinder graph metric completion.
Figure 36: PCA projection of embeddings of model trained on the Hypercube graph metric completion.
Figure 37: PCA projection of embeddings of model trained on the 2D Lattice graph metric completion.
Figure 38: PCA projection of embeddings of model trained on the 3D Lattice graph metric completion.
Figure 39: PCA projection of embeddings of model trained on the 2D Comparison .
Figure 40: PCA projection of embeddings of model trained on the 3D Comparison .
Figure 41: Convergence speed on modular addition when using various regularization.
Figure 42: Convergence speed on modular subtraction when using various regularization.
Figure 43: Convergence speed on modular squared sum when using square sum.
Figure 44: Convergence speed on modular cubed sum when using various regularization.
Figure 45: Convergence speed on modular multiplication when using various regularization.
Figure 46: Convergence speed on modular affine–bilinear when using various regularization.
Figure 47: Convergence speed on cycle graph metric completion when using various regularization.
Figure 48: Convergence speed on path graph metric completion when using various regularization.
Figure 49: Convergence speed on cylinder graph metric completion when using various regularization.
Figure 50: Convergence speed on hypercube graph metric completion when using various regularization.
Figure 51: Convergence speed on 2d lattice graph metric completion when using various regularization.
Figure 52: Convergence speed on 3d lattice graph metric completion when using various regularization.
Figure 53: Convergence speed on Comparison (2D) when using various regularization.
Figure 54: Convergence speed on Comparison (3D) when using various regularization.
Figure 55: Comparison of embedding initialization and output scaling on Modular Addition, 1-Layer Transformer
More in this topic

Neural Network Generalization

Neural network generalization studies why trained models perform well on unseen data and how architecture, data, and training shape that ability.

See all 299 papers

Explore similar work

CardsList

  1. Breaking Data Symmetry is Needed For Generalization in Feature Learning Kernels

    Mar 31, 2026Marcel Tomàs Bernal, Neil Rohit Mallinar, Mikhail BelkinSymmetry BreakingKernel Methods

    CFIS UPC · UC San Diego

  2. How to Tame Grokking: Representation Geometry as a Control Signal

    Jul 13, 2026Maksim A KazanskiiNeural Network GeneralizationSpectral Regularization

    Independent Researcher

  3. Radial Suppression Accelerates Algorithmic Generalization: A Geometric Analysis of Delayed Generalization

    Jun 30, 2026Srijan Tiwari, Aditya Chauhan, Manjot SinghNeural Network GeneralizationNeural Representation Geometry

    Indian Institute of Technology Roorkee