Post-Routing Arithmetic in Llama-3: Last-Token Result Writing and Rotation-Structured Digit Directions

Yao Yan

cs.AIFeb 22, 2026

Post-Routing Arithmetic in Llama-3: Last-Token Result Writing and Rotation-Structured Digit Directions

Authors: Yao Yan

Organizations: Chongqing Normal University Chongqing Key Lab of Cognitive Intelligence and Intelligent Finance Chongqing, China

Abstract

We study three-digit addition in Meta-Llama-3-8B (base) under a one-token readout to characterize how arithmetic answers are finalized after cross-token routing becomes causally irrelevant. Causal residual patching and cumulative attention ablations localize a sharp boundary near layer~17: beyond it, the decoded sum is controlled almost entirely by the last input token and late-layer self-attention is largely dispensable. In this post-routing regime, digit(-sum) direction dictionaries vary with a next-higher-digit context but are well-related by an approximately orthogonal map inside a shared low-rank subspace (low-rank Procrustes alignment). Causal digit editing matches this geometry: naive cross-context transfer fails, while rotating directions through the learned map restores strict counterfactual edits; negative controls do not recover.

Figures & tables

Layer range	Last-token strict $\uparrow$	Non-last strict $\uparrow$
0–16	0.00	$\approx$ 0.99
17–32	1.00	$\approx$ 0.00

Table 1: Cross-sample resid_post patching (strict transfer). Early layers are controlled by non-last tokens; after the transition near layer 16, overwriting only the last-token residual transfers the source answer, while overwriting non-last tokens has negligible effect. Full per-layer curves and additional diagnostics appear in Appendix A .

Ablated Layers	Accuracy (%)
None (baseline)	100.0
17–32	$\approx$ 0.0
18–32	$\geq$ 99.9

Table 2: Accuracy on baseline-correct subset (thus baseline=100%).Strict accuracy after cumulative attention ablation. Attention is causally necessary through the result-writing boundary, but becomes dispensable in the post-routing regime (layers

\geq 17

) under the one-token readout. Full curves and confidence intervals are in Appendix A .

Figure 1: Layerwise context-conditioned dictionary isomorphism (two digit-shifted settings). For each layer

L

, metrics are averaged over off-diagonal context pairs

(c_{1},c_{2})

. (a) Unaligned cosine decreases with depth, indicating non-invariance of individual bucket directions. (b) After low-rank orthogonal Procrustes alignment (Sec. C.2 ), aligned cosine remains near unity while

\mathrm{RelFro}

stays small, consistent with a rotation-like re-parameterization of a shared low-rank core. Solid: stripped-tens context

\rightarrow

ones-sum; dashed: result-hundreds-digit context

\rightarrow

result-tens digit.

Figure 2: Single-layer anchor: context-conditioned rotation at a fixed depth. At a representative post-routing layer, unaligned similarity between context dictionaries varies substantially (top), while low-rank Procrustes alignment collapses similarity to near unity (bottom).

Figure 3: Rotation is necessary for cross-context digit editing. Strict counterfactual success vs. layer (baseline-correct subset). rotated recovers toward direct , while transfer collapses; wrong-condition and random-

R

fail to recover.

Figure 4: Edit-magnitude sensitivity across two settings. Strict success vs.

|\Delta|

(layers 24–26, weighted by

n_{\mathrm{total}}

). Left:

H\!\rightarrow\!t

. Right:

T\!\rightarrow\!o

. transfer degrades with

|\Delta|

, while rotated remains close to direct ; random-

R

and wrong-condition controls do not recover.

Figure 5: Cross-prompt transport of digit-editing interventions. (a) Strict edit success vs. layer under the canonical template and three paraphrases. (b) Right: bar plot of answer accuracy under each prompt.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Cross-sample patching curves. Strict transfer vs. layer for last-token and non-last patching on baseline-correct source–target pairs. A sharp crossover occurs near layer 16.

	ones-sum dict (stripped-tens)			tens-sum dict (stripped-hundreds)
Layer	Unaligned	Procrustes	RelFro	Unaligned	Procrustes	RelFro
17	$0.94\pm 0.02$	$1.00\pm 0.00$	0.09	$0.87\pm 0.10$	$1.00\pm 0.00$	0.10
18	$0.95\pm 0.02$	$1.00\pm 0.00$	0.06	$0.84\pm 0.19$	$1.00\pm 0.00$	0.09
19	$0.88\pm 0.08$	$1.00\pm 0.00$	0.04	$0.84\pm 0.11$	$1.00\pm 0.00$	0.04
20	$0.81\pm 0.14$	$1.00\pm 0.00$	0.04	$0.79\pm 0.26$	$1.00\pm 0.00$	0.07
21	$0.72\pm 0.21$	$1.00\pm 0.00$	0.05	$0.77\pm 0.15$	$1.00\pm 0.00$	0.05

Appendix

Table 3: Full layerwise summary for digit-sum dictionary rotations. Mean

\pm

std over context pairs for unaligned (

k

-aligned) cosine and Procrustes-aligned cosine, along with relative Frobenius reconstruction error (RelFro), reported for layers

L\in[17,28]

in the ones-sum (stripped-tens context) and tens-sum (stripped-hundreds context) settings.

Numerical Reasoning in Language Models

Numerical reasoning in language models studies how systems represent, calculate with, and explain numbers across mathematical and real-world tasks.

See all 150 papers

Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts
May 1, 2026Sheridan Feucht, Tal Haklay, Usha Bhalla +9Numerical Reasoning in Language Models LLM Interpretability
aNortheastern University · bTechnion IIT · cHarvard University +1
The Readout Shortcut: Positional Number Copying Dominates Arithmetic CoT Readout in Small Language Models
May 20, 2026Ming LiuNumerical Reasoning in Language Models CoT Faithfulness
Amazon
Separators Make Carry Propagation Learnable:The Geometry of Latent Carry in a Multiplication Transformer
Oct 5, 2026Sama Satariyan, Raphael Cousin, G{é}rard BiauNumerical Reasoning in Language Models Transformer Interpretability
SCAI · LPSM,IUF,MEGAVOLT

May 1, 2026J/K move · Enter open · S save

Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts

Sheridan Feucht, Tal Haklay, Usha Bhalla, Daniel Wurgaft+8

aNortheastern University · bTechnion IIT · cHarvard University+1

Numerical Reasoning in Language Models LLM Interpretability Mechanistic Interpretability Causal Abstraction Representation Geometry in Language Models LLM Mathematical Reasoning

Open paper AskPDF arXiv

Does structure in representations imply structure in computation? We study how Llama-3.1-8B reasons over cyclic concepts (e.g., "what month is six months after August?"). Even though Llama-3.1-8B's representations for these concepts are circularly structured, we find that instead of directly computing modular addition in the period of the cyclic concept (e.g., 12 for months), the model re-uses a generic addition mechanism across tasks that operates independently of concept-specific geometry. First, it computes the sum of its two inputs using base-10 addition (six + August=14). Then, it maps this sum back to cyclic concept space (14->February). We show that Llama-3.1-8B uses task-agnostic Fourier features to compute these sums--in fact, these features have periods that respect standard base-10 addition, e.g., 2, 5, and 10, rather than the cyclic concept period (e.g., 12 for months). Furthermore, we identify a sparse set of 28 MLP neurons re-used across all tasks (approximately 0.2% of the MLP at layer 18) that can be partitioned into disjoint clusters, each computing the sum for a Fourier feature with a different period. Our work highlights how an interplay between causal abstraction and feature geometry can deepen our mechanistic understanding of LMs.

Abstract

Numerical Reasoning in Language Models

Explore similar work

Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts

The Readout Shortcut: Positional Number Copying Dominates Arithmetic CoT Readout in Small Language Models

Separators Make Carry Propagation Learnable:The Geometry of Latent Carry in a Multiplication Transformer