Modular Addition

Latest papers

11 papers2 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Probability-Signature Dynamics: Unpacking Modular Addition Learning Within Two-Layer Networks
Oct 8, 2026Yunji Wang, Junjie Yao, Linyu Liu +2Neural Network Training Dynamics Frequency-Domain Feature Learning
School of Mathematical Sciences, Shanghai Jiao Tong University · Huawei Technologies Ltd · Shanghai University of Finance and Economics +1
A Spectral Theory of Grokking: Weight Decay induces Feature Learning
Sep 22, 2026Lenz Pracher, Pascal de Jong, Oskar Lieshaus +2Neural Network Generalization Weight Decay
Arnold-Sommerfeld-Center for Theoretical Physics, Ludwig-Maximilians-Universität München, Munich, Germany · Department of Applied Physics, Standford University, Standford, CA, USA · Department of Mathematics, University of Cambridge, Cambridge, UK
Symmetry without a manifold: intrinsic dimension on orbits
Sep 15, 2026Chon-Fai Kam, Miloud Bessafi, Frédéric CadetIntrinsic Dimensionality Modular Addition
Dipartimento di Fisica e Chimica, Universit`a degli Studi di Palermo, via Archirafi 36, I-90123 Palermo, Italy · EnergyLab, Universit´e de La R´eunion, F-97715 Saint-Denis, France · Universit´e Paris Cit´e and Universit´e de La R´eunion, BIGR, INSERM UMR S1134, F-75014 Paris, France +1
A Counterexample to Fourier Alignment in Single-Neuron Modular Addition
Aug 5, 2026Gautam Neelakantan MemanaShallow Neural Networks ReLU Neural Networks
Beyond Neural Collapse: Task-Intrinsic Geometry Governs Neural Representations in Modular Arithmetic
Jun 8, 2026Hu Tan, Kuo Gai, Shihua ZhangNeural Representation Geometry Classification
Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China; and School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai, China · Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China; School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China; and Key Laboratory of Systems Health Science of Zhejiang Province, School of Life Science, Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Chinese Academy of Sciences, Hangzhou 310024, China
BRo-JEPA: Learning Modular Arithmetic in Latent Space
May 31, 2026Divyansh Jha, Yuanfang Xie, Varan Mehra +1Zero-Shot Learning Joint-Embedding Predictive Architecture
1Georgia Institute of Technology · 2NYU Langone Health
Assign and Add: A Mechanistic Study of Compositional Arithmetic
May 29, 2026Brady Exoo, Alberto Bietti, John SousTransformer Interpretability Transformer
Yale University · Flatiron Institute
Learning Large-Scale Modular Addition with an Auxiliary Modulus
May 8, 2026Hanato Kikuchi, Ryosuke Masuya, Kazuhiko Kawamoto +1Boolean Function Learning Distribution Shift
Chiba University · National Institute of Informatics
Grokking of Diffusion Models: Case Study on Modular Addition
Apr 20, 2026Joon Hyeok Kim, Yong-Hyun Park, Mattis Dalsætra Østby +1Pixel-Space Diffusion Models Mechanistic Interpretability
Department of Computer and Information Science University of Pennsylvania Philadelphia, PA 19104, USA
On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking
Feb 18, 2026Jianliang He, Leda Wang, Siyu Chen +1Lottery Ticket Hypothesis Modular Addition
Department of Statistics and Data Science, Yale University
A Mechanistic Study of Transformers Training Dynamics
Oct 31, 2024Ambroise Odonnat, Wassim Bouaziz, Vivien CabannesTransformer Interpretability Attention Mechanisms
Inria, Univ. Rennes 2 · Mistral AI; work done while at Meta. · FAIR at Meta.

Oct 8, 2026J/K move · Enter open · S save

Probability-Signature Dynamics: Unpacking Modular Addition Learning Within Two-Layer Networks

Yunji Wang, Junjie Yao, Linyu Liu, Pinyan Lu+1

School of Mathematical Sciences, Shanghai Jiao Tong University · Huawei Technologies Ltd · Shanghai University of Finance and Economics+1

Neural Network Training Dynamics Frequency-Domain Feature Learning Modular Addition

Open paper AskPDF arXiv

Neural networks trained on modular addition tasks often develop Fourier-structured representations that support exact generalization. While prior work has identified these Fourier circuits, the mechanism by which gradient-based training selects them from the data distribution remains unclear. We address this question using probability signatures, which express leading gradient interactions through conditional statistics of the training distribution. For modular addition, these signatures are cyclic shift operators and are diagonalized by the discrete Fourier transform, yielding approximately decoupled Fourier-mode dynamics. This explains the emergence of Fourier sparsity, frequency matching, and phase alignment. The same framework resolves a puzzle under label noise: corrupted examples can show faster early loss decrease than clean examples, despite lacking a coherent generalization rule. We show that noise increases conditional label collisions, strengthening early shared-coordinate reinforcement. Finally, this method can be applied to other operators. Taking XOR as an example, we observed the predicted frequency in experiments.