Arithmetic OOD Failure Unfolds in Stages in Minimal GPTs

Seine A. Shintani

cs.CLMar 27, 2026

Arithmetic OOD Failure Unfolds in Stages in Minimal GPTs

Authors: Seine A. Shintani

Organizations: Department of Biomedical Sciences, College of Life and Health Sciences, Chubu University, Kasugai, Aichi 487-8501, Japan · Center for Mathematical Science and Artificial Intelligence, Chubu University, Kasugai, Aichi 487-8501, Japan · Institute for Advanced Research, Nagoya University, Nagoya, Aichi 464-8601, Japan

Abstract

Arithmetic benchmarks are often reduced to a single held-out score, but that score can conflate qualitatively different failures. We study a controlled minimal GPT trained on exhaustive 2-digit addition, where all local digit transitions are already present in training, and ask why 3-digit generalization still fails. The failure is staged. First, there is a layout barrier: a learned absolute-position model collapses under a pure 3-digit layout shift, and mixed-layout exposure is the only intervention that materially weakens this barrier. Second, after layout repair, the hundreds position behaves like a carry flag rather than a semantic hundreds digit; targeted carry probes reverse the relevant logit margin, whereas a matched extra-data control does not. Third, after carry repair, the main remaining bottleneck is conditional recomposition: high-conditioned tail data outperforms a matched control, high-only data, and tail-only data on all true-3-digit suites, and the same ordering reappears in a larger 2-layer bridge experiment. The residual errors after recomposition are then overwhelmingly tens-only, and a separate 10-seed late-stage study shows that a sign-aware tens repair raises exact match on the hardest thousands-carry suite from 0.664 to 0.822. We therefore provide an experimentally testable decomposition of arithmetic OOD failure into layout, carry-semantics, recomposition, and late tens-residual stages.

Figures & tables

Figure 1: Overview of the benchmark ladder and repair program. Exhaustive 2-digit addition already covers all 200 local digit transitions, so later 3-digit failures cannot be explained by missing local rules alone. The evaluation suites are ordered so that each new suite adds one qualitatively different source of difficulty. The repair families are matched to the active barrier rather than being generic augmentation.

Human-readable label	What it tests	Example target string
2-digit in-support	Standard 2-digit format seen during training (code name: in_support_2digit )	49+07=056
3-digit layout-only, low range	Same 0–99 value range, but written in 3-digit layout (code name: layout_shift_only_lowrange3 )	049+007=0056
True 3-digit, no incoming carry	Genuine 3-digit addition with no incoming carry to hundreds ( $c_{2}=0$ ; code name: true3_hundreds_no_incarry )	241+320=0561
True 3-digit, incoming carry	Genuine 3-digit addition with an incoming carry to hundreds ( $c_{2}=1$ ; code name: true3_hundreds_with_incarry )	289+173=0462
Thousands-carry	Carry into a new thousands digit ( $c_{3}=1$ ; includes both $c_{2}=0$ and $c_{2}=1$ cases; code name: true3_thousands_carry )	782+289=1071

Table 1: Evaluation suites. The suites are ordered so that the first failure can be interpreted cleanly. The layout-only suite changes formatting but not value range. The later suites keep 3-digit layout and add semantic difficulty through carry structure.

Split	Local coverage	Local accuracy	Sym. accuracy	Main missing structure
IID 80/20 random hold-out	1.000	1.000	1.000	none
Units-pair OOD	0.000	0.000	0.850	unseen ones-pair combinations
Tens-carry OOD	0.000	0.000	0.701	unseen tens+carry combinations
Carry-chain OOD	0.000	0.000	0.000	sustained carry propagation
Ordered commutativity OOD	0.091	0.091	1.000	compositional symmetry

Table 2: Why IID random hold-out is insufficient. “Local” refers to a baseline that reconstructs answers from observed local digit transitions; “Sym.” adds commutative symmetry. The IID split is already solved by complete local coverage, whereas structured splits expose qualitatively different barriers.

Family	2-digit	Layout-only	Layout token acc.	True 3-digit (no incarry)
Absolute	0.9200	0.0008	0.201	0.0002
Mixed-layout absolute	0.5886	0.4426	0.854	0.0000
Coupled significance	0.7990	0.0000	0.332	0.0022
Digit-aware	0.4288	0.0002	0.133	0.0000
Symmetry-aware	0.7482	0.0000	0.357	0.0006
No position	0.0944	0.0006	0.081	0.0002

Table 3: Five-seed position-family comparison. Only the mixed-layout family sees the 3-digit rendering of the same low-range sums during training; the other families test whether positional inductive bias alone weakens the first barrier. Exact-match means use constrained decoding. The final column reports exact match on the no-incoming-carry true-3-digit suite.

Exact match
Family	No incoming carry	Incoming carry	Thousands-carry
ctrl3	0.310	0.416	0.390
highonly	0.326	0.414	0.379
tail	0.396	0.512	0.468
tailhigh	0.541	0.700	0.585
$P(\mathrm{exact}\mid\mathrm{high2\ correct})$

Table 4: Analytic carry-repaired binding ablation (5 seeds; 1-layer, width-16). All families are compared from the same carry-repaired base with a volume-matched final augmentation pack. The decisive ingredient is high-conditioned tail binding.

Figure 2: 10-seed late-stage confirmation: pooled exact-match means on the three true-3-digit suites. This is a downstream confirmation study in the larger 2-layer, width-32 model. tenspolarity is best on the no-incoming-carry and thousands-carry suites, while tensboundary is narrowly best on the incoming-carry suite.

Figure 3: 10-seed late-stage confirmation: recomposition after the upper digits are already correct. The metric is $P(\mathrm{exact}\mid\mathrm{high2\ correct})$ . Improvements here are important because they show that later gains are not explained only by fixing the upper digits.

Family	No incoming carry	Incoming carry	Thousands-carry
ctrl4	0.7057	0.7492	0.6640
tensboundary	0.7572	0.8579	0.7674
tenspolarity	0.7973	0.8550	0.8219

Table 5: 10-seed late-stage confirmation means and strongest paired evidence. The strongest paired claim is reserved for the hardest thousands-carry suite.

Figure 4: 10-seed late-stage confirmation: signed tens residuals. Negative values indicate systematic under-shooting of the tens digit; positive values indicate systematic over-shooting. tenspolarity moves both $c_{2}=0$ and $c_{2}=1$ branches closer to zero, most clearly on the hardest thousands-carry suite.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Family	No-inc. exact	Inc. exact	1000-carry exact	$\Delta_{h0}$	$\Delta_{h1}$	$\Delta_{t1}$
base	0.0000	0.0000	0.0072	+288.0	+98.9	+238.8
ctrl_dup	0.0000	0.0002	0.0116	+335.9	+101.7	+271.5
probe_all	0.0332	0.0450	0.0250	-109.9	-105.2	-30.1

Appendix

Table 6: Matched carry-probe summary (5 seeds). Exact-match means are shown for the three true-3-digit suites. Positive margins indicate preference for the flag digit over the true digit at the corresponding probe step.

Family	Suite	Upper-correct rate	Lower-both-correct given upper
base	No incoming carry	0.0002	0.0000
base	Incoming carry	0.0000	—
base	Thousands-carry	0.0406	0.1773
ctrl_dup	No incoming carry	0.0012	0.0000
ctrl_dup	Incoming carry	0.0014	0.1429
ctrl_dup	Thousands-carry	0.0208	0.4231

Appendix

Table 7: Upper-digit repair versus lower-digit recomposition (5 seeds).

Exact match
Family	No incoming carry	Incoming carry	Thousands-carry
ctrl3	0.418	0.496	0.458
highonly	0.456	0.540	0.476
tail	0.535	0.568	0.530
tailhigh	0.733	0.753	0.716
$P(\mathrm{exact}\mid\mathrm{high2\ correct})$

Appendix

Table 8: Large-capacity bridge of the conditional-binding ablation (5 seeds; 2-layer, width-32).

Family	Suite	high2_correct	low2_correct	$P(\mathrm{exact}\mid\mathrm{high2\ correct})$
ctrl4	No incoming carry	0.8942	0.7650	0.7878
tensboundary	No incoming carry	0.8968	0.8289	0.8445
tenspolarity	No incoming carry	0.8931	0.8819	0.8928
ctrl4	Incoming carry	0.9424	0.7843	0.7915
tensboundary	Incoming carry	0.9625	0.8889	0.8903
tenspolarity	Incoming carry	0.9474	0.9007	0.9020

Appendix

Table 9: Pooled 10-seed recomposition summary.

Family	Suite branch	Mean tens delta	Interpretation
ctrl4	No incoming carry ( $c_{2}=0$ )	-2.149	under-shoot
tensboundary	No incoming carry ( $c_{2}=0$ )	-2.148	under-shoot
tenspolarity	No incoming carry ( $c_{2}=0$ )	-1.499	reduced under-shoot
ctrl4	Incoming carry ( $c_{2}=1$ )	+1.319	over-shoot
tensboundary	Incoming carry ( $c_{2}=1$ )	+1.044	reduced over-shoot
tenspolarity	Incoming carry ( $c_{2}=1$ )	+0.961	further reduced over-shoot

Appendix

Table 10: Mean signed tens residuals in the pooled 10-seed confirmation.

Numerical Reasoning in Language Models

Numerical reasoning in language models studies how systems represent, calculate with, and explain numbers across mathematical and real-world tasks.

See all 150 papers

The Shape of Addition: Geometric Structures of Arithmetic in Large Language Models
May 29, 2026Liuyuan Wen, Xun Zhu, Lihao Huang +2Neural Representation Geometry LLM Interpretability
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China.
Post-Routing Arithmetic in Llama-3: Last-Token Result Writing and Rotation-Structured Digit Directions
Feb 22, 2026Yao YanNumerical Reasoning in Language Models Circuits in Language Models
Chongqing Normal University Chongqing Key Lab of Cognitive Intelligence and Intelligent Finance Chongqing, China
Separators Make Carry Propagation Learnable:The Geometry of Latent Carry in a Multiplication Transformer
Oct 5, 2026Sama Satariyan, Raphael Cousin, G{é}rard BiauNumerical Reasoning in Language Models Transformer Interpretability
SCAI · LPSM,IUF,MEGAVOLT

May 29, 2026J/K move · Enter open · S save

The Shape of Addition: Geometric Structures of Arithmetic in Large Language Models

Liuyuan Wen, Xun Zhu, Lihao Huang, Wenbin Li+1

State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China.

Neural Representation Geometry LLM Interpretability LLM Mathematical Reasoning

Open paper AskPDF arXiv

Large Language Models exhibit paradoxical fragility in fundamental arithmetic, implying a disconnect between internal computation and discrete output. By analyzing the residual stream geometry during multi-operand addition, we identify the Iso-Raw-Sum Trajectory (IRST), a geometric structure where representations are anchored by semantic digits and modulated by continuous carry fibers. We propose the Noisy Quantization Model to explain this geometry, framing arithmetic errors as Geometric Slippages caused by internal neural noise pushing a continuous, latent Carry Potential across quantization thresholds. This geometric framework further elucidates Probe Versatility, explaining how lightweight probes can disentangle coexisting latent signals (such as ground truth versus hallucination) from a single activation vector. Finally, we validate these insights through a geometric consistency check method that effectively detects and corrects these quantization failures during inference. Our code is available at https://github.com/RL-MIND/Shape-of-Addition.

Abstract

Numerical Reasoning in Language Models

Explore similar work

The Shape of Addition: Geometric Structures of Arithmetic in Large Language Models

Post-Routing Arithmetic in Llama-3: Last-Token Result Writing and Rotation-Structured Digit Directions

Separators Make Carry Propagation Learnable:The Geometry of Latent Carry in a Multiplication Transformer