List Counting Failures Are Not One Phenomenon

Iyad Ait Hou; Saad Mankarious; Aya Zirikly; Rebecca Hwa

cs.LGSep 3, 2026

List Counting Failures Are Not One Phenomenon

Authors: Iyad Ait Hou, Saad Mankarious, Aya Zirikly, Rebecca Hwa

Organizations: Department of Computer Science The George Washington University Washington, D.C., USA

Abstract

Counting the items in a bracketed list looks trivial, yet open-weight chat models often get it wrong. Prior work usually blames input bottlenecks such as subword fragmentation or attention dilution, which predict that different models should fail in roughly the same way. Across seven instruct models on identical prompts, however, wrong answers form distinct modes: Qwen and Gemma 27B often flip odd lengths to a nearby even integer, OLMo concentrates errors on a few mid-sized integers, and Llama tends to under-count. These modes are useful labels rather than a stable family law (Gemma 9B does not reproduce Gemma 27B's odd-to-even drop), and heavier subword fragmentation does not make counting harder on our benchmark. When the model answers incorrectly, a linear probe can usually still recover the true count from the residual stream. Matching the same odd-to-even error also does not imply the same late-MLP magnitude fix: scaling a late MLP output helps Qwen modestly but is near null on Gemma 27B under the same protocol, while residual steering can move both only by trading odd gains for even losses. These results caution against transferring that magnitude fix across models without a transfer check.

Figures & tables

Figure 1: Overview: same surface failure need not mean the same mechanism. The standard story predicts a shared input bottleneck and one transferable late-layer fix. On identical prompts we instead find distinct modes, a usually recoverable residual count when wrong (

0.85

–

0.96

), and non-transfer of a late-MLP magnitude intervention across a shared odd-to-even drop (Qwen

0.20

–

0.25

vs. Gemma 27B

0.055

; Table 3 ; § 4.3 – 4.4 ).

Model	Role	Prec.
Qwen 2.5–14B	within-family scale	bf16
Qwen 2.5–32B	reference	bf16
Qwen 2.5–72B	within-family scale	nf4
Gemma 2–9B	within-family scale	bf16
Gemma 2–27B	cross-family	bf16
OLMo 2–32B	cross-family (+ stages)	bf16

Table 1: Models. Core seven: main eval + probes; Gemma 9B / Llama 8B: scale checks (Appendix B ); Mistral-Small-24B: held-out reverse_half test. Stages: Qwen base vs. instruct; OLMo base

\to

DPO

\to

instruct (§ 4.5 ).

Condition	Varies	Purpose
common_words	high-freq. English	Familiar baseline
rare_words	rare English	Frequency control
ml_single	one non-English lang.	Script/surface OOD
ml_mixed	mixed scripts	Script-mixing stress
random_strings	alphanumeric pseudowords	Tokenization load

Table 2: Surface conditions under a fixed English ask. ml_*

=

multilingual; multilingual rows stress list surface form, not multilingual counting competence. Examples in Appendix A .

Model	Gap $[95\%$ CI $]$	Shape	Acc c	Acc r	$\Delta$	$P($ u $\mid$ w $)$	Top wrong
Qwen-32B	$+0.40$ $[0.33,0.47]$	parity	0.70	0.92	$+0.22$	0.46	12, 14, 8
R1-Distill	$+0.39$ $[0.32,0.46]$	parity	0.45	0.70	$+0.24$	0.92	12, 6, 8
Qwen-14B	$+0.36$ $[0.29,0.44]$	parity	0.58	0.77	$+0.19$	0.75	12, 13, 8
Gemma-27B	$+0.24$ $[0.18,0.30]$	parity	0.38	0.46	$+0.08$	0.94	10, 11, 12
Qwen-72B	$+0.10$ $[0.02,0.18]$	weak parity	0.86	0.93	$+0.07$	0.11	13, 14, 15
OLMo-32B	$+0.05$ $[-0.01,0.12]$	mid-range	0.30	0.45	$+0.16$	0.95	11, 8, 9

Table 3: Error modes and tokenization control on identical prompts. Gap

=

even

-

odd accuracy (

\{10,12,14\}

vs.

\{11,13,15\}

;

300

/side; bootstrap

95\%

CIs). Acc c / Acc r

=

mean accuracy on common_words / random_strings ;

\Delta{=}

Acc

{}_{\mathrm{r}}{-}

Acc c (random uses

{\sim}4\times

more tokens at

L{=}15

P(

\mid

)

=

P(\hat{y}{<}y\mid

wrong

)

on common_words (separates mid-range concentration from pure under-count). Top wrong

=

leading wrong emissions on common_words (lattice-specific; not a length-invariant attractor). Full grids in Appendices C – D .

model	$n$	model acc.	probe acc.	probe right, model wrong	% at $\{11,13,15\}$
OLMo 2–32B	520	0.302	0.962	346 (66.5%)	32%
R1-Distill Qwen 32B	520	0.402	0.935	290 (55.8%)	33%
Gemma 2–27B	520	0.392	0.919	285 (54.8%)	35%
Llama 3.1–70B (4-bit)	520	0.465	0.979	268 (51.5%)	43%
Qwen 2.5–14B	520	0.575	0.927	197 (37.9%)	47%
Qwen 2.5–32B	520	0.704	0.923	131 (25.2%)	69%

Table 4: Cross-model probe-vs-model accuracy on the same

520

-sequence common_words cell (

L\!\in\!\{3,\ldots,15\}

40/L

). “Probe right, model wrong” counts sequences where the linear probe at

\ell_{\text{last}}

predicts the correct integer while the model emits the wrong digit. The last column reports the share of those cases at odd lengths

\{11,13,15\}

model	model acc.	probe acc.	$P(\checkmark\mid\times)$	$P(\checkmark\mid\checkmark)$	round-acc	on errors
			true-count probe		perm.	emit-probe
Qwen 2.5–32B	0.704	0.923	0.851 [0.79, 0.90]	0.954	0.071	0.448
Qwen 2.5–14B	0.575	0.927	0.891 [0.84, 0.93]	0.953	0.066	0.552
Qwen 2.5–72B (4-bit)	0.887	0.958	0.864 [0.76, 0.93]	0.970	0.073	0.441
Gemma 2–27B	0.392	0.919	0.902 [0.86, 0.93]	0.946	0.066	0.665
OLMo 2–32B	0.302	0.962	0.953 [0.93, 0.97]	0.981	0.076	0.730

Table 5: Probe controls on the

520

-sequence common_words cell, last-layer answer-position residual, all seven models.

P(\checkmark\mid\times)

P(\checkmark\mid\checkmark)

: held-out true-count probe round-accuracy on model-wrong / model-correct subsets (Wilson

95\%

CIs). perm. : mean round-accuracy under five label permutations (chance

{\approx}0.077

). emit-probe on errors : probe trained on the model’s emitted integer, evaluated on model-wrong residuals. On the same residuals where the true count is decodable at

0.85

–

0.96

, the wrong emission is decodable at only

0.40

–

0.76

Figure 2: Causal MLP scale interventions. Top: per-layer fraction of odd-length failures fixed by scaling that layer’s MLP output by $s\in\{-0.5,0\}$ over the late-layer candidate set (Appendix G ; small-sample panels are upper bounds). Selection-corrected full-failure held-out rates: $0.20\pm 0.06$ (Qwen 32B), $0.25\pm 0.05$ (Qwen 14B), $0.055\pm 0.022$ (Gemma 27B). Bottom: even- vs. odd-target accuracy as the MLP output scale $s$ sweeps on Qwen 32B ( $s{=}1$ is baseline). Layer indices differ by analysis (discovery vs. split-half vs. continuous sweep); see Appendix G .

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Condition	$L$	Example list (items only)	Tok
common_words	5	beach door window letter bird	5
common_words	11	water office car fish friend story rock fire road forest phone	11
rare_words	11	truculent ineffable quotidian …sesquipedalian	31
ml_single	11	Arabic nouns (e.g. school, garden, …market)	23
ml_mixed	11	mixed Chinese/Spanish/Arabic/Russian nouns	21
random_strings	11	7nyh5 wtcp 88h8jc 3bwrz …pvkp5	49

Appendix

Table 6: Example lists from the released eval (Qwen tokenizer token counts at right). Same strings are shown to every model; multilingual rows are abbreviated here for layout (full Unicode lists ship in the CSV release).

Model	Gap cw $[95\%$ CI $]$	Shape	$P(\mathrm{u}\mid\mathrm{w})$	Top wrong
Gemma-27B	$+0.25$ $[0.13,0.38]$	parity	0.99	10, 12, 11
Gemma-9B	$-0.30$ $[-0.42,-0.18]$	fixed mid-range	0.79	11, 12
Llama-70B	$+0.05$ $[0.00,0.11]$	under-count	1.00	11, 10, 12
Llama-8B	$+0.00$ $[0.00,0.00]$	under-count	1.00	10, 12, 9

Appendix

Table 7: Within-family behavioral scale checks on the same

1{,}300

-prompt lattice ( common_words even

-

odd gap over

L\in\{10,\ldots,15\}

; bootstrap

95\%

CIs).

P(\mathrm{u}\mid\mathrm{w})=P(\hat{y}{<}y\mid

wrong

)

. Llama 8B matches Llama 70B’s under-count class. Gemma 9B’s negative gap is a fixed mid-range collapse on this lattice, not a flipped 27B-style odd-to-even drop.

condition	3	4	5	6	7	8	9	10	11	12	13	14	15
common_words	1.00	1.00	0.90	1.00	0.80	0.85	0.65	1.00	0.05	0.75	0.25	0.85	0.00
rare_words	1.00	1.00	0.95	0.65	0.80	0.80	0.80	1.00	0.30	0.95	0.50	0.85	0.15
ml_single	1.00	0.95	1.00	1.00	0.60	0.85	0.25	0.85	0.25	0.95	0.40	0.75	0.25
ml_mixed	1.00	1.00	0.85	0.60	0.60	0.70	0.00	0.55	0.00	0.50	0.05	0.05	0.00
random_strings	1.00	0.95	1.00	1.00	0.85	0.95	0.90	1.00	0.55	1.00	0.95	0.90	0.90

Appendix

Table 8: Full accuracy grid for the reference model Qwen 2.5–32B,

L\in\{3,\ldots,15\}

and the five surface conditions.

20

sequences per cell;

1{,}300

total. Bold marks odd lengths

\{11,13,15\}

	3	4	5	6	7	8	9	10	11	12	13	14	15
common_words	.00	.00	.00	.00	-.10	-.05	-.15	.00	+.95	+.25	+.15	-.10	-.65
rare_words	.00	.00	+.05	+.35	+.20	+.10	+.20	.00	-.20	+.05	-.20	+.40	.00
ml_single	.00	-.05	.00	.00	-.20	-.05	-.35	-.30	+.25	+.05	-.50	-.45	-.45
ml_mixed	.00	.00	-.05	-.10	-.40	-.35	-.75	-.55	-1.25	-1.00	-1.35	-1.75	-2.70
random_strings	.00	-.05	.00	.00	-.05	-.05	-.15	.00	-.45	.00	-.05	-.10	.00

Appendix

Table 9: Mean signed error (predicted

-

true) for Qwen 2.5–32B. On common_words ,

L{=}11

errors are

+1

on average (model says “12”);

L{=}15

errors are

-1

on average (model says “14”). ml_mixed undercounts severely.

condition	3	4	5	6	7	8	9	10	11	12	13	14	15
common_words	0.95	0.95	0.80	0.95	0.50	0.75	0.50	0.75	0.20	0.95	0.05	0.15	0.00
rare_words	1.00	0.90	0.80	0.75	0.55	0.60	0.25	0.55	0.15	0.70	0.35	0.65	0.10
ml_single	1.00	0.85	0.80	0.90	0.65	0.85	0.20	0.75	0.15	0.70	0.55	0.30	0.20
ml_mixed	1.00	0.90	0.70	0.75	0.30	0.60	0.00	0.05	0.00	0.25	0.00	0.45	0.00
random_strings	1.00	1.00	1.00	0.90	0.75	0.80	0.50	0.90	0.50	0.90	0.55	0.65	0.55

Appendix

Table 10: Qwen 2.5–14B accuracy grid. Odd-length collapse at

\{11,13,15\}

is sharp on every condition; random_strings remains highest.

condition	3	4	5	6	7	8	9	10	11	12	13	14	15
common_words	1.00	1.00	1.00	1.00	0.95	0.95	0.85	0.80	0.95	0.60	0.85	0.60	0.65
rare_words	1.00	1.00	1.00	1.00	1.00	1.00	0.75	0.95	0.45	1.00	0.80	0.10	0.35
ml_single	1.00	1.00	0.95	1.00	1.00	0.85	0.50	0.80	0.55	0.70	0.70	0.50	0.40
ml_mixed	1.00	0.95	0.95	0.95	0.75	0.75	0.15	0.40	0.15	0.80	0.20	0.25	0.10
random_strings	1.00	1.00	1.00	1.00	1.00	0.95	0.85	1.00	1.00	0.90	0.90	0.90	0.55

Appendix

Table 11: Qwen 2.5–72B (4-bit nf4) accuracy grid. The odd-length collapse at

\{11,13,15\}

is much weaker than at 14B/32B; the odd–even gap is partly preserved on rare_words and ml_mixed but absent on common_words .

condition	3	4	5	6	7	8	9	10	11	12	13	14	15
common_words	1.00	1.00	0.90	0.55	0.25	0.15	0.15	0.75	0.10	0.10	0.00	0.00	0.00
rare_words	1.00	1.00	0.90	0.50	0.35	0.40	0.30	0.95	0.25	0.20	0.05	0.10	0.25
ml_single	1.00	1.00	0.65	0.20	0.15	0.30	0.40	0.55	0.05	0.15	0.00	0.00	0.00
ml_mixed	0.95	0.95	0.50	0.10	0.05	0.15	0.10	0.25	0.05	0.00	0.00	0.05	0.00
random_strings	1.00	1.00	0.75	0.35	0.25	0.55	0.20	0.60	0.05	0.20	0.35	0.55	0.15

Appendix

Table 12: Gemma 2–27B accuracy grid. Overall accuracy is lower across the board; the odd-length collapse is most visible on common_words , rare_words , and ml_mixed .

condition	3	4	5	6	7	8	9	10	11	12	13	14	15
common_words	0.95	0.85	0.35	0.45	0.00	0.85	0.10	0.50	0.30	1.00	0.05	0.50	0.00
rare_words	1.00	1.00	0.80	0.90	0.30	0.95	0.60	0.50	0.50	0.85	0.10	0.65	0.00
ml_single	1.00	1.00	0.50	0.45	0.25	0.70	0.15	0.65	0.25	0.65	0.35	0.35	0.30
ml_mixed	0.90	0.70	0.20	0.15	0.00	0.15	0.00	0.10	0.00	0.40	0.00	0.25	0.00
random_strings	1.00	0.70	0.55	0.75	0.70	1.00	0.25	0.95	0.60	0.85	0.40	0.65	0.65

Appendix

Table 13: DeepSeek-R1-Distill-Qwen-32B accuracy grid (reasoning enabled). The odd-length collapse at

\{11,13,15\}

is preserved through R1 distillation; even–odd gap (0.39) is essentially identical to its base Qwen 2.5–32B (0.40).

condition	3	4	5	6	7	8	9	10	11	12	14	15
common_words	1.00	0.55	1.00	0.90	0.30	0.00	0.10	0.00	0.00	0.00	0.00	0.00
rare_words	0.95	0.35	0.85	0.30	0.50	0.15	0.00	0.00	0.35	0.10	0.50	0.00
ml_single	0.95	0.45	0.80	0.65	0.75	0.25	0.60	0.20	0.15	0.10	0.40	0.15
ml_mixed	0.70	0.05	0.55	0.55	0.65	0.50	0.55	0.15	0.65	0.10	0.80	0.10
random_strings	0.85	0.75	0.55	0.60	0.40	0.45	0.20	0.25	0.85	0.35	0.65	0.00

Appendix

Table 14: OLMo 2–32B accuracy grid. Overall accuracy is lower than the Qwen/Gemma row even at short lengths, but there is no odd-length collapse: odd lengths are no worse than adjacent even lengths on most conditions. This is the no-collapse baseline that the cross-family argument in § 4.1 rests on.

condition	3	4	5	6	7	8	9	10	11	12	13	14	15
common_words	1.00	1.00	0.90	1.00	0.75	0.75	0.10	0.15	0.00	0.00	0.00	0.00	0.00
rare_words	1.00	1.00	1.00	0.95	0.80	0.95	0.55	0.30	0.95	0.55	0.50	0.00	0.70
ml_single	0.90	0.90	0.65	0.80	0.65	0.45	0.55	0.35	0.80	0.05	0.50	0.10	0.30
ml_mixed	1.00	0.95	0.95	1.00	0.75	0.35	0.70	0.45	0.95	0.05	0.15	0.05	0.40
random_strings	0.95	1.00	0.75	0.90	0.45	0.65	0.35	0.30	0.75	0.25	0.40	0.00	0.30

Appendix

Table 15: Llama 3.1–70B (4-bit nf4) accuracy grid. On common_words , the model is accurate at short lengths and then collapses above

L\!=\!8

to a small-integer under-counting attractor rather than to the Qwen/Gemma odd-length collapse.

Figure 3: Length-extrapolation falsifier. Mean probe-predicted count, by held-out true count, for a probe trained on $L\leq 10$ . At every layer (only four shown), the predicted mean collapses to the train-set ceiling ( $\sim\!9.9$ ) for every held-out count; round-accuracy is $0.00$ . The dashed line is $y=x$ , the prediction a single global cardinality axis would produce.

$L$	probe pred mean	probe round-acc	model acc
3	$3.09$	$0.95$	$1.00$
4	$4.01$	$1.00$	$1.00$
5	$5.01$	$1.00$	$0.90$
6	$6.01$	$0.95$	$1.00$
7	$7.02$	$0.98$	$0.80$
8	$7.98$	$0.98$	$0.85$

Appendix

Table 16: Per-length probe vs. model accuracy at the answer-position residual (layer

63

, Qwen-32B). The probe column is the held-out (5-fold CV) ridge prediction; the model column is the model’s own greedy emit on the same prompts. Bold rows are the odd-length collapse. Overall:

r=0.997

\text{MAE}=0.20

, round-acc

0.923

Setting	Acc	Chance
one answer $\to$ model	0.27	0.14
one answer $\to$ family	0.62	0.25
one wrong answer $\to$ model	0.39	0.14
bag of 40 wrong answers $\to$ model	0.93	0.14
bag of wrong answers $\to$ family	1.00	0.25

Appendix

Table 17: Error-mode separability on the shared benchmark. Single answers are weak identifiers; bags of wrong answers separate models well in this closed seven-model set. Family identity separates almost perfectly at the bag level. This is descriptive structure, not a claim that counting errors fingerprint models in the wild.

Model	Family	Err rate	Top wrong emissions	Even share
Gemma 27B	gemma	0.62	10:0.26, 12:0.13, 11:0.09, 5:0.08, 8:0.08	0.64
Llama 70B	llama	0.43	11:0.30, 13:0.13, 12:0.12, 10:0.09, 9:0.07	0.35
OLMo 32B	olmo	0.62	11:0.25, 14:0.17, 9:0.11, 7:0.11, 8:0.09	0.40
Qwen 14B	qwen	0.42	12:0.29, 14:0.13, 8:0.13, 10:0.11, 6:0.10	0.78
Qwen 32B	qwen	0.29	12:0.27, 10:0.18, 14:0.13, 8:0.12, 7:0.07	0.82
Qwen 72B	qwen	0.21	12:0.25, 13:0.23, 10:0.14, 14:0.11, 11:0.09	0.61

Appendix

Table 18: Wrong-emission modes on the

1{,}300

-prompt eval. Even share is the fraction of wrong emissions that are even integers. Qwen-family models are even-heavy; among OLMo’s wrong answers, mass peaks at “

11

”; Llama is odd-heavy / under-counting.

Figure 4: Cross-model overwrite maps for sampled odd-length failures. Each row is one prompt; color shows $P(\text{model's wrong units digit})-P(\text{correct units digit})$ under a layerwise logit-lens readout. Green regions indicate layers where the correct digit is favored; red regions indicate layers where the eventual wrong digit dominates. Qwen14, Qwen32, and Gemma27B all show a mid-to-late transition toward the wrong digit, a read-only diagnostic of late failure. Causal reverse_half confirms a usable late-MLP lever only in Qwen (§ 4.4 ).

Figure 5: Accuracy vs. list length by surface condition (odd lengths $\{11,13,15\}$ shaded). Read for the cross-panel shape split (parity oscillation vs. flat vs. under-count), not per-condition zig-zags; main-text summary is Table 3 and Fig. 1 .

Figure 6: Reference-model odd-length collapse and signed-error diagnostics for Qwen 2.5–32B. The odd-length failures are directional rather than diffuse: $L=11$ tends upward toward “12”, while $L=15$ tends downward toward “14”.

Figure 7: Qwen 2.5–32B logit-lens trajectories. Correct-digit mass appears in middle layers and is suppressed in late layers, consistent with a late overwrite rather than an absent count signal.

Figure 8: Cross-model second-digit logit lens on odd-length failures. For each of Qwen 14B, Qwen 32B, and Gemma 2 27B, we force the leading “ $1$ ” on length- $\{11,13,15\}$ failures and read off the per-layer probability of the correct vs. wrong units digit. In every model the correct digit is favored at intermediate depth and the wrong digit dominates only late, with mean final $P(\text{wrong})\in[0.77,0.92]$ . Read-only evidence of late failure, not by itself a claim that the same MLP lever is causal in every family (Fig. 2 ).

Figure 9: Digit-unembedding PCA for Qwen 2.5–32B. Digit geometry is ordinal; even/odd is not cleanly separated, supporting the claim that the parity pattern is not a simple head-side parity direction.

Figure 10: Probe-basis per-failure flip-layer analysis for Qwen 2.5–32B. The plot tracks when the probe agrees with the true count versus the model’s wrong emitted count on probe-right/model-wrong examples. It is a probe-basis diagnostic, not the model’s own logit-lens basis, and complements the logit-lens trajectories in Figure 7 .

Figure 11: Model-output prior proxy: marginal wrong-emission distributions by model. Each family concentrates wrong answers on a small set of integers, but the attractor set differs across families.

Figure 12: Accuracy versus model-output-prior proxy. For Qwen 32B and OLMo, per-count accuracy is lower when neighboring integers have high wrong-emission mass. This is a proxy for learned output prior, not a direct pretraining-frequency measurement.

condition	main eval seed	fine-grained seed
common_words	$2$	$12$
rare_words	$3$	$13$
multilingual_single	$4$	$14$
multilingual_mixed	$5$	$15$
random_strings	$6$	$16$
random-string pool	$303$

Appendix

Table 19: Per-condition data-generation seeds. The same seeds are used across all seven models so each per-model accuracy grid is computed on the same

1{,}300

200

sequences. The random-string pool of

200

pseudowords is built once under seed

303

and shared across both evals.

Numerical Reasoning in Language Models

Numerical reasoning in language models studies how systems represent, calculate with, and explain numbers across mathematical and real-world tasks.

See all 150 papers

The Right Answer, the Wrong Direction: Why Transformers Fail at Counting and How to Fix It
May 5, 2026Gabriel GarciaNumerical Reasoning in Language Models Transformer Interpretability
Independent Researcher
Repeated-Token Counting Reveals a Dissociation Between Representations and Outputs
May 10, 2026Sohan VenkateshLLM Interpretability Mechanistic Interpretability
Manipal Institute of Technology Bengaluru
From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks
Apr 1, 2026Ayan Datta, Mounika Marreddy, Alexander Mehler +2Symbolic Reasoning in Language Models LLM Interpretability
IIIT Hyderabad · Goethe University, Frankfurt am Main, Germany · University of Sheffield, UK

May 5, 2026J/K move · Enter open · S save

The Right Answer, the Wrong Direction: Why Transformers Fail at Counting and How to Fix It

Gabriel Garcia

Independent Researcher

Numerical Reasoning in Language Models Transformer Interpretability LLM Fine-Tuning Object Counting Language Modeling Representation Geometry in Language Models

Open paper AskPDF arXiv

Large language models often fail at simple counting tasks, even when items to count are in the prompt. We investigate whether this failure occurs because transformers do not represent counts internally, or because they cannot convert representations to the correct output tokens. Across three model families: Pythia, Qwen3, and Mistral, ranging from 0.4B to 14B parameters, we find evidence for the second explanation. Linear probes recover the correct count from intermediate layers with

R^2>0.99

, showing that the information is present. However, the internal directions that encode counts are nearly orthogonal to digit-token output-head rows (

|\cos| \leq 0.032

). In other words, the model stores the count in a form that the digit logits do not naturally read out. We localize this failure with two interventions. Updating only the digit rows of the output head (36,864 parameters) substantially improves constrained digit prediction (60.7--100.0% on four tasks), but it does not fix unconstrained generation (0%); we do not claim that digit-row repair fixes open-ended text. By contrast, small LoRA on attention Q/V (7.67M parameters) improves upstream routing and achieves 83.1%

\pm

7.2% in true greedy autoregressive generation (deployable fix). Logit-lens at layer 35 (entity counting; correct-digit rank): (i) median over 3 seeds drops from order-

10^4

to 1; (ii) seed 42 shows

54{,}332 \to 838

(median top-1 while one seed stays far below). Norm, logit-lens, and cross-task analyses generalize the bottleneck to counting, addition, and list length; nulls on MMLU and GSM8K and limited DROP transfer. These results identify counting failure as a geometric readout bottleneck, not an internal-representation failure: the model knows the count but the output pathway is misaligned with tokens needed to express it.

Abstract

Numerical Reasoning in Language Models

Explore similar work

The Right Answer, the Wrong Direction: Why Transformers Fail at Counting and How to Fix It

Repeated-Token Counting Reveals a Dissociation Between Representations and Outputs

From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks