Repetition, Not Length: Isolating the Counting Failure in Neural Text-to-Speech

Kirill Borodin; Vasilii Kudryavtsev; Maxim Maslov; Grach Mkrtchian

cs.CLSep 29, 2026

Repetition, Not Length: Isolating the Counting Failure in Neural Text-to-Speech

Authors: Kirill Borodin, Vasilii Kudryavtsev, Maxim Maslov, Grach Mkrtchian

Organizations: BitmanagerAI, Dubai, UAE · lab260, Yerevan, Armenia · MTUCI, Moscow, Russia

Abstract

Text-to-speech models loop, truncate and lose count on text that repeats a phrase many times. We show that repetition itself is what breaks them, not the length that comes with it. Every repeated sentence in our test set is paired with a control of matched sentence and word count in which no word ever repeats back-to-back. Six models from three architectures render the controls almost perfectly and fail the repeated twins: 94.3% against 18.2% exactly right at k >= 6. The gap survives greedy decoding, repetition-penalty sweeps, four independent speech recognisers and 420 analysis specifications without once reversing sign; a held-out fourth architecture lands within a point of its predicted gap, and one of two non-autoregressive baselines shows the same failure. Varying the period of the text shows the failure grows smoothly with periodicity, half of it surviving when no word is adjacent to itself.

Figures & tables

Scorer	NLL/token		ctl. less probable
	rep.	ctl.	(% of pairs)
phi-2	3.69	4.89	87
GPT-2-large	3.36	5.34	100

Table 1: Likelihood of the paired texts under two independent scorers: mean NLL per token (lower = more probable) and the share of 54 pairs whose control is less probable. A panel-internal scorer agrees (supplementary material).

Quantity	est.	95% interval
Exact-rate gap, per checkpoint ( $n{=}6{}$ )	76.7	$[68.4{},\,84.4{}]$
Exact-rate gap, per family ( $n{=}3{}$ )	75.4	$[71.1{},\,78.9{}]$
Exact-rate gap, mixed effects (widest of six)	76.4	$[63.2{},\,89.2{}]$
Count-error gap, per checkpoint	10.4	$[5.6{},\,13.9{}]$
Exact-rate gap, whole ladder ( $n{=}1559{}$ )	46.8	—
Predicted gap, unseen architecture	65.3	$[5.9{},\,93.3{}]$

Table 2: Headline estimates with 95% intervals, in exactly-right points (the count-error row: points of median relative error). The last two rows are the pre-registered out-of-panel prediction and its test.

Model	size	count err.		exact (%)
		rep.	ctl.	rep.	ctl.
Llasa-1B [ 20 ]	1B	$-12.5$	0.0	13.3	91.5
Llasa-3B [ 20 ]	3B	$-8.3$	0.0	16.9	94.1
Llasa-8B [ 20 ]	8B	$-16.7$	0.0	11.8	93.2
XTTS-v2 [ 2 ]	0.4B	$-12.5$	0.0	16.7	87.8
Qwen3-TTS-0.6B [ 8 ]	0.6B	$-12.5$	0.0	7.9	100.0

Table 3: Every system tested,

k\geq 6

, CTC judge: median

(\hat{c}-k)/k

and exactly-right rate, repeated (rep.) against control (ctl.). Systems in the lower block are outside the panel; their count errors are in the supplementary material.

Requested count	repeated	control
$k{=}48$	25.0	46.0
$k{=}64$	25.0	59.5
$k{=}96$	22.0	64.0
$k{=}128$	17.0	66.5

Table 4: The

k\leq 128

extension, 4 checkpoints: median rendered count against the requested count. Repeated renderings regress while controls advance.

	$p{=}1$	$p{=}2$	$p{=}3$	$p{=}4$	$p{=}6$	$p{=}8$
Ladder scan ( $n{=}1539{}$ )	10.4	47.9	—	73.6	—	93.1
At $k{=}24{}$ ( $n{=}1202{}$ )	8.5	51.2	62.3	75.2	90.0	92.9

Table 5: The period ladder: exactly-right rate (%) against the period

p

of the text at fixed carrier, word count and requested count, pooled over the four checkpoints measured. The second row re-measures at

k{=}24{}

with the odd and non-power steps added.

	gap (pts)
Panel, $k\geq 6$	76.1
Exclusions off ( $n{=}1620{}$ )	61.7
No penalty (Llasa $\times$ 3)	78.9
Greedy (Qwen3-TTS-0.6B)	72.2
Sampled (Qwen3-TTS-0.6B)	75.0
RAS on (Qwen3-TTS-0.6B)	94.3

Table 6: The gap under decoding and analysis variations. RAS is repetition-aware sampling [ 3 ] .

	free	oracle duration
Below $k{=}12{}$ ( $n{=}30{}$ )	43.3	76.7
At or above $k{=}12{}$ ( $n{=}60{}$ )	16.7	16.7

Table 7: F5-TTS handed the oracle total duration: exactly-right rate (%).

Model	capacity gain		ceiling
	rep.	ctl.	rep.	ctl.
Llasa-1B [ 20 ]	18.2 $[9.8{}{,}25.7{}]$	56.7 $[50.9{}{,}63.5{}]$	181	247
Llasa-3B [ 20 ]	12.4 $[-1.8{}{,}23.3{}]$	58.9 $[52.8{}{,}68.0{}]$	178	249
Llasa-8B [ 20 ]	15.0 $[11.0{}{,}20.3{}]$	45.5 $[37.2{}{,}53.1{}]$	188	228
XTTS-v2 [ 2 ]	24.0 $[16.9{}{,}31.1{}]$	34.8 $[29.1{}{,}40.7{}]$	81	105
Qwen3-TTS-0.6B [ 8 ]	20.9 $[16.3{}{,}25.9{}]$	34.9 $[30.6{}{,}39.6{}]$	81	115

Table 8: State capacity per checkpoint: growth of distinguishable states

dN_{\mathrm{eff}}/d\log k

with 95% intervals, and the saturating-fit ceiling of

N_{\mathrm{eff}}

, repeated against control.

Checkpoint	repeated	control
Llasa-1B [ 20 ]	1.01	0.86
Llasa-3B [ 20 ]	0.99	0.83
Llasa-8B [ 20 ]	0.86	0.59
Qwen3-TTS-0.6B [ 8 ]	0.74	0.95
Qwen3-TTS-1.7B [ 8 ]	0.90	0.89

Table 9: Probe MAE past the ladder (

k{=}48

–

128

) as a fraction of a constant predictor’s; below 1 the count is still decodable. The control column runs the same probe on control items.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Count	What it tests
word repetition	60	a single word repeated $k$ times in a fixed carrier
matched control	54	the same carrier, $k$ distinct filler words (no repetition)
sentence repetition	32	a whole short sentence repeated $k$ times
tongue twisters	24	a twister sentence (with internal repetition) repeated $k$ times
numbers	10	English number phrases with intrinsic digit-word repetition
Total	180

Appendix

Table S1: Item families.

ID	Prefix	Target	Suffix	Filler pool (for the control)
t1	The dog was	very	big and it ran across the field.	quite, really, truly, fairly, rather, somewhat, extremely, notably
t2	She said	no	to the offer and then left the room.	yes, maybe, sure, fine, okay, well, right, hmm
t3	He walked	far	into the forest before turning back.	deep, fast, long, wide, high, low, near, past
t4	The light was	blue	before the storm arrived.	green, bright, pale, dim, warm, cold, sharp, soft
t5	The runner moved	quick	along the narrow path.	swift, smooth, steady, light, sharp, loose, tight, clean
t6	They were	really	tired after the long journey.	truly, quite, very, rather, deeply, clearly, plainly, surely

Appendix

Table S2: Word-repetition carriers: fixed prefix, target repeated

k

times, fixed suffix; the filler pool builds the matched control at the same

k

ID	Phrase	Target	Count
n01	six hundred sixty six thousand six hundred sixty six	six	6
n02	seven hundred seventy seven thousand seven hundred seventy seven	seven	6
n03	nine hundred ninety nine thousand nine hundred ninety nine	nine	6
n04	three hundred thirty three thousand three hundred thirty three	three	6
n05	six hundred sixty six million six hundred sixty six thousand six hundred sixty six	six	9
n06	eight hundred eighty eight thousand eight hundred eighty eight	eight	6

Appendix

Table S3: Number-phrase items, with the target digit-word and its true count.

ID	Sentence	Target	Per-copy
s1	The bell rang twice.	bell	—
s2	He counted the stones.	stones	—
s3	Rain fell on the roof.	rain	—
s4	The door stayed open.	door	—
w1	She sells sea shells by the sea shore.	sea	2
w2	Peter Piper picked a peck of pickled peppers.	peck	1

Appendix

Table S4: Sentence-repetition items (

k\in\{1,2,3,4,6,8,12,16\}

, expected count

k

) and tongue-twister items (

k\in\{1,2,4\}

, expected count per-copy occurrences of the target

\times\,k

Recogniser	Material	$k\geq 4$ ratio	Notes
CTC	periodic (repeated word)	1.00	22/24 exact; 2 off by 1 ( $k{=}16,32$ )
CTC	repeated sentence (control)	1.00	23/24 exact; 1 off by 1 ( $k{=}32$ )
Whisper large-v3	periodic (repeated word)	0.19	no transcript at $k{=}16,32$
Whisper large-v3	repeated sentence (control)	0.25	no transcript at $k{=}16,32$

Appendix

Table S5: Head-to-head validation on identical synthetic ground truth. Median counted/true ratio,

k\geq 4

pooled. The repeated-sentence arm is periodic at the sentence grain, not an aperiodic control.

	primary	HuBERT	robust-ft	Whisper
control exact (%)	94.3	89.5	82.1	92.9
repeated exact (%)	18.2	17.9	17.5	17.1
gap (points)	$+76.7$	$+72.3$	$+65.1$	$+75.5$
Llasa-1B	78.1	78.6	65.2	69.9
Llasa-3B	77.2	74.8	65.9	76.0
Llasa-8B	81.4	80.3	69.7	71.7

Appendix

Table S6: Four judges on the full panel population: exact rates and repeated-versus-control gap in points (

k\geq 6

, paired panel), then the gap by checkpoint and by family.

checkpoint	exact-rate (pts)	median-error (pts)	capacity
Llasa-1B [ 20 ]	+78.1	+12.5	+38.5
Llasa-3B	+77.2	+8.3	+46.5
Llasa-8B	+81.4	+16.7	+30.5
Qwen3-TTS-0.6B [ 8 ]	+92.2	+12.5	+14.0
Qwen3-TTS-1.7B	+60.0	+0.0	+8.1
XTTS-v2 [ 2 ]	+71.1	+12.5	+10.7

Appendix

Table S7: The three per-checkpoint contrasts, signed so positive means the deficit is present: exact-rate and median-error gaps in percentage points, the capacity gap in distinguishable states per unit

\log k

contrast	mean	95% CI	$p$	positive
exact-rate gap (pts)	76.7	[68.4, 84.4]	0.031	6/6
median-error gap (pts)	10.4	[5.6, 13.9]	0.062	5/6
capacity gap	24.7	[13.7, 36.5]	0.031	6/6

Appendix

Table S8: Panel-level summary of the three contrasts: mean, bootstrap 95% CI, two-sided exact sign-test

p

, and the count of positive checkpoints.

Construction	$n$ clusters	Estimate	95% interval	Criterion
Mixed model, crossed RE, Wald (reference only)	3 fam.	76.4	[71.9, 80.9]	$z$
the six constructions the main text’s “all six” counts:
Mixed model, crossed RE, $t(2)$	3 fam.	76.4	[66.6, 86.3]	$t(2)$
OLS, clustered by family	3	76.2	[68.7, 83.7]	$t(2)$
OLS, clustered by checkpoint	6	76.2	[63.2, 89.2]	$t(5)$
Family-level means, $t$ -interval	3	75.4	[65.6, 85.2]	$t(2)$

Appendix

Table S9: Interval constructions for the arm effect (control

-

repeated, percentage points). The Wald row is reported for reference and is not one of the six intervals the main text’s “all six” refers to.

Level	Unit	Mean	95% credible interval
Family	Llasa	79.3	[73.4, 84.4]
Family	Qwen3-TTS	74.1	[67.7, 79.9]
Family	XTTS	73.7	[63.5, 82.2]
Checkpoint	Llasa-1B	79.5	[70.5, 86.9]
Checkpoint	Llasa-3B	77.2	[67.0, 85.5]
Checkpoint	Llasa-8B	81.0	[72.7, 88.0]

Appendix

Table S10: Posterior arm-effect gap (percentage points, control

-

repeated), partially pooled; all

P(\mathrm{gap}>0)=1.000

to the precision shown. Llasa [ 20 ] , Qwen3-TTS [ 8 ] , XTTS-v2 [ 2 ] .

Prior on $\sigma$	Mean	95% CrI	$P(\mathrm{gap}>0)$
HalfNormal(0.5), tighter, favourable	71.7	[26.6, 90.9]	1.000
HalfNormal(1.0), reported	65.3	[5.9, 93.3]	0.996
HalfNormal(3.0), much wider, unfavourable	52.3	[ $-$ 1.6, 96.1]	0.956

Appendix

Table S11: Prior sensitivity on the four group-level scales

\sigma

: new-family predictive gap (points).

Outcome	$n$	Range (pts)	Median	Positive	Zero / negative
Exact-rate gap	210	[34.4, 86.4]	70.5	210/210	0 / 0
Median-error gap	210	[0.0, 19.5]	8.3	197/210	13 / 0
Both outcomes combined	420	—	—	407/420	13 / 0

Appendix

Table S12: Specification curve;

n

is the number of specifications under that outcome definition.

Axis	Spread (pts)	Median gap by level
$k$ threshold	22.8	$k{\geq}1$ : 58.0, $k{\geq}6$ : 74.7, $k{\geq}8$ : 78.3, $k{\geq}12$ : 80.7, $k{\geq}16$ : 69.4
Exclusion rule	9.7	panel: 75.8, +templates: 68.8, +cap-hits: 76.7, +degenerate: 75.7, everything: 67.1
Aggregation level	2.3	pooled: 70.6, checkpoint: 71.3, family: 69.1
Control construction	2.3	cycled: 72.0, aperiodic: 69.7
Seed subset	1.1	all seeds: 69.7, seed 0 only: 70.8

Appendix

Table S13: Which forking choice moves the exact-rate gap (median gap at each level, percentage points), sorted by spread, descending.

model	sat. SSE	prop. SSE	$b$	soft $N^{\ast}$	best
Llasa-1B [ 20 ]	477.5	35.0	0.855	85	proportional
Llasa-3B	464.2	79.3	0.913	119	soft horizon
Llasa-8B	353.5	2.0	0.830	71	proportional
Qwen3-TTS-0.6B [ 8 ]	499.3	1.4	0.953	338	proportional
Qwen3-TTS-1.7B	668.8	5.1	1.107	640	proportional
XTTS-v2 [ 2 ]	378.0	0.8	0.860	90	proportional

Appendix

Table S14: Constant, proportional and soft-horizon fits to the median rendered count above

k=6

: residual sums of squares, fitted slope

b

, fitted horizon

N^{\ast}

, and the winning form.

arm	template rule	token budget	degenerate
repeated ( $n=1080$ )	16.7%	10.5%	4.4%
control ( $n=972$ )	16.7%	5.6%	3.7%

Appendix

Table S15: Exclusion rates by arm, before any other filter.

population	$n$	exact rep	exact ctl	gap
panel (as reported)	1234	18.2%	94.3%	76.1
$+$ judge-unmeasurable template	1424	17.2%	79.8%	62.6
$+$ budget-truncated items	1362	16.3%	93.1%	76.7
$+$ degenerate audio (scored 0)	1242	17.9%	93.9%	76.0
nothing excluded at all	1620	14.8%	76.5%	61.7

Appendix

\fnum@table : The gap with each exclusion rule lifted in turn, and with nothing excluded at all.

Scorer	Role	Rep. mean	Ctl. mean	Gap	Ctl. higher
Llasa-1B	panel backbone	14.85	16.05	1.20	89%
phi-2	independent, primary	3.69	4.89	1.20	87%
gpt2-large	secondary cross-check	3.36	5.34	1.98	100%

Appendix

\fnum@table : Naturalness check, three scorers, 54 matched pairs. “Ctl. higher”: fraction of pairs in which the control has the higher (less probable) per-token cross-entropy.

Model	$k^{\ast}_{\mathrm{word}}$	tok. (word)	$k^{\ast}_{\mathrm{sent}}$	tok. (sent)
Llasa-1B	0	n/a	3	351
Llasa-3B	1	209	3	457
Qwen3-TTS-0.6B	4	53	4	80
Qwen3-TTS-1.7B	8	80	4	86
XTTS-v2	4	89	4	128
XTTS-v2 (no penalty, ablation)	2	63	1	29

Appendix

\fnum@table : Unit invariance, as committed. Token counts are medians at the reported

k^{\ast}

; n/a where no

k

cleared the

0.5

threshold.

Model	Raw	+diversity	correct-only
Llasa-1B	0.42	0.50	0.01 ( $n{=}14$ /32)
Llasa-3B	0.28	0.34	0.80 ( $n{=}24$ /22)
Llasa-8B	0.43	0.75	— ( $n{=}0$ /0)
Qwen3-TTS-0.6B	0.57	—	0.31 ( $n{=}42$ /84)
Qwen3-TTS-1.7B	0.78	—	0.47 ( $n{=}64$ /82)
XTTS-v2	0.60	—	0.56 ( $n{=}22$ /41)

Appendix

\fnum@table : Capacity-confound ratios (repeated/control slope of

\mathcal{N}_{\mathrm{eff}}

against

\log k

n

pairs are repeated-side/control-side rows entering each regression; — marks an inestimable cell (fewer than 12 usable rows, or fewer than 4 distinct

k

values).

Model	Repeated retention	Control retention
Llasa-1B	0.97	1.09
Llasa-3B	0.87	0.99
Llasa-8B	0.96	1.09
Qwen3-TTS-0.6B	1.46	1.74
Qwen3-TTS-1.7B	0.77	1.12
XTTS-v2	1.92	2.85

Appendix

\fnum@table : Ridge-probe retention (late-window best-layer

R^{2}

/ early-window best-layer

R^{2}

). All entries

n{=}54

items.

Architecture	Penalty	Rep. exact (%)	Ctl. exact (%)	Control
XTTS-v2	1.00	15.3	30.9	collapsed
XTTS-v2	2.00	15.6	92.2	intact
XTTS-v2	3.00	21.1	91.1	intact
XTTS-v2 (shipped)	5.00	16.7	87.8	intact
XTTS-v2	8.00	15.6	84.4	intact
Qwen3-TTS-0.6B	1.00	10.0	100.0	intact

Appendix

\fnum@table : The repetition-penalty sweep on the two architectures that ship one; a collapsed control arm says nothing about repetition.

checkpoint	$p{=}1$	$p{=}2$	$p{=}4$	$p{=}8$	$p{=}k$	$n$ ( $1,2,4,8,k$ )
Llasa-1B	3.6	54.6	83.3	92.5	75.0	28, 130, 84, 40, 44
Qwen3-TTS-0.6B	6.8	41.3	73.3	100.0	86.7	44, 179, 90, 45, 45
XTTS-v2	11.1	51.7	75.6	82.2	57.8	45, 180, 90, 45, 45
MEAN	7.2	49.2	77.4	91.6	73.2

Appendix

\fnum@table : The full period ladder: exactly-right rate (%) by period at

k\in\{16,24,32\}

under the ordered-scan rule, with cell sizes.

rule	$D(1)$	$D(2)$	$D(4)$	$D(2)/D(1)$	verdict
ordered scan (headline)	84.4	42.4	14.2	0.502	PERIODICITY
unbounded recount	82.9	60.9	20.1	0.734	PERIODICITY
strict conjunction	82.9	61.2	21.2	0.738	PERIODICITY
cap hits kept (censored bound)	81.9	43.5	13.5	0.531	PERIODICITY
by family	84.4	42.4	14.2	0.502	PERIODICITY

Appendix

\fnum@table : The period-2 recovery under five scoring rules: deficits

D(p)=E(8)-E(p)

in points, the decisive ratio

D(2)/D(1)

, and the pre-committed verdict.

variant	Qwen3	XTTS	Qwen3	XTTS	words	chars	dur. (s)
	exact rate		recovered $R$		confound means
bare	6.8	11.1	—	—	31.8	170.6	11.65
comma	6.8	8.9	$+0.000$	$-0.031$	31.8	194.3	13.11
stop	9.8	13.6	$+0.031$	$+0.036$	31.6	191.6	13.42
and	15.6	11.1	$+0.094$	$+0.000$	54.8	263.0	16.23

Appendix

\fnum@table : The disambiguation arm at period 1: exactly-right rate and recovered fraction

R

by boundary variant, with per-variant confound means.

adjacent steps	gap (pts)	95% interval
$E(3)-E(2)$	$+10.9$	$[+2.1,\,+16.8]$
$E(4)-E(3)$	$+12.9$	$[+3.6,\,+21.2]$
$E(6)-E(4)$	$+14.7$	$[+2.2,\,+28.3]$
$E(8)-E(6)$	$+2.7$	$[-2.4,\,+7.7]$

Appendix

\fnum@table : Adjacent-step gaps in exactly-right points on the

k{=}24

ladder, carrier-cluster bootstrap 95% intervals.

checkpoint	$E$ periodic	$E$ shuffled	$\Delta$	95% CI
Qwen3-TTS-1.7B	80.0	97.8	$+17.8$	$[+5.6,+34.4]$
Qwen3-TTS-0.6B	73.3	91.1	$+17.8$	$[+6.7,+32.2]$
Llasa-1B	74.1	68.9	$-5.2$	$[-22.0,+11.3]$
XTTS-v2	73.3	43.3	$-30.0$	$[-53.3,-6.7]$

Appendix

\fnum@table : Shuffled order at fixed composition: exactly-right rate on periodic and shuffled

p{=}4

items per checkpoint, with the bootstrap 95% interval on the difference.

model	repeated exact (%)	control exact (%)	gap (pts)
Llasa-1B	13.3	91.5	+78.1
Llasa-3B	16.9	94.1	+77.2
Llasa-8B	11.8	93.2	+81.4
Qwen3-TTS-0.6B	7.8	100.0	+92.2
Qwen3-TTS-1.7B	38.9	98.9	+60.0
XTTS-v2	16.7	87.8	+71.1

Appendix

\fnum@table : The published arms on the two non-autoregressive baselines, beside the panel: exactly-right rates and the gap in points.

arm	setting	exact rep	exact ctl	gap
stock	—	67.6%	49.1%	$-18.5$
duration	duration-noise scale 1.6	53.7%	20.4%	$-33.3$
rate	speaking rate 1.35	61.1%	31.5%	$-29.6$

Appendix

\fnum@table : VITS under perturbed duration settings; the gap is control minus repeated, negative in every arm.

$k$	free exact (%)	given exact (%)	free median	given median
6	46.7	80.0	-0.167	+0.000
8	40.0	73.3	-0.125	+0.000
12	33.3	40.0	-0.083	-0.083
16	20.0	26.7	-0.062	-0.062
24	6.7	0.0	-0.167	-0.125
32	6.7	0.0	-0.094	-0.125

Appendix

\fnum@table : F5-TTS handed the oracle total duration: exactly-right rate and median relative error, free against given, by

k

checkpoint	layers	$n$ pairs	$q$ repeated [95% CI]	$q$ control [95% CI]	paired $p$
Llasa-1B	16	29	38.1 [32.9, 51.4]	31.8 [27.3, 39.8]	0.017
Llasa-3B	28	26	118.8 [101.9, 156.5]	86.4 [77.2, 137.3]	0.269
Llasa-8B	32	29	347.7 [258.1, 393.5]	185.4 [142.1, 236.5]	0.256
Qwen3-TTS-0.6B	28	44	21.0 [19.7, 24.2]	21.9 [20.6, 23.2]	0.931
Qwen3-TTS-1.7B	28	45	24.7 [23.1, 25.7]	23.6 [21.4, 25.1]	0.177

Appendix

\fnum@table : Largest singular value

q

of the per-repetition map at the headline setting (lag

\tau

, whole stack), per arm, with the paired repeated-versus-control test. Checkpoints as in the main text’s panel table.

checkpoint	repeated: range over 16 cells	control: range over 16 cells
Llasa-1B	[3.90, 78.10]	[3.50, 46.61]
Llasa-3B	[9.33, 182.80]	[7.60, 148.28]
Llasa-8B	[11.43, 499.13]	[12.28, 329.33]
Qwen3-TTS-0.6B	[3.35, 29.47]	[3.41, 28.23]
Qwen3-TTS-1.7B	[3.45, 37.83]	[3.35, 31.18]

Appendix

\fnum@table : Range of

q

over all sixteen (lag

\times

sub-stack) cells, per arm. Checkpoints as in the main text’s panel table.

flatness measure	repeated	control
largest single share	+0.59 [+0.35, +0.75]	+0.01 [-0.34, +0.37]
block entropy	+0.43 [+0.16, +0.65]	+0.17 [-0.20, +0.49]
deviation from uniform	+0.55 [+0.30, +0.72]	+0.05 [-0.31, +0.39]

Appendix

\fnum@table : Within-cell correlation between attention flatness and count error, pooled by Fisher

z

; positive means flatter items are counted better.

checkpoint	arm	probe MAE	constant MAE	$R^{2}$
Llasa-1B	repeated $k=2\ldots 32$	0.63	1.12	$+0.62$
Llasa-3B	repeated $k=2\ldots 32$	0.86	1.12	$+0.41$
Llasa-8B	repeated $k=2\ldots 32$	0.75	1.12	$+0.52$
Qwen3-TTS-0.6B	repeated $k=2\ldots 32$	0.79	1.12	$+0.47$
Qwen3-TTS-1.7B	repeated $k=2\ldots 32$	0.79	1.12	$+0.47$
Llasa-1B	repeated $k=48\ldots 128$	0.50	0.50	$-0.02$

Appendix

\fnum@table : Ridge-probe decoding of the requested count against the constant predictor, below the horizon (

k\leq 32

) and past it (

k=48\ldots 128

	MAE ratio vs. constant			MAE, repeated arm
checkpoint	rep.	reading	ctl.	length-only	state probe
Llasa-1B	1.01	indistinguishable	0.86	0.61	0.51
Llasa-3B	0.99	indistinguishable	0.83	0.56	0.49
Llasa-8B	0.86	recovers the count	0.59	0.45	0.43
Qwen3-TTS-0.6B	0.74	recovers the count	0.95	0.30	0.37
Qwen3-TTS-1.7B	0.90	recovers the count	0.89	0.36	0.45

Appendix

\fnum@table : Past-horizon verdicts: probe-to-constant MAE ratios with the

\pm 0.02

tie band, and the length-only control on the repeated arm.

checkpoint	P1 ratio	worst cell	bound	verdict
Qwen3-TTS-0.6B	1.38	$14\%$	$9\%$	readable null
Qwen3-TTS-1.7B	1.12	$9\%$	$6\%$	readable null
Llasa-1B	0.57	$25\%$	$20\%$	readable null
Llasa-8B	0.90	$42\%$	—	cannot tell

Appendix

\fnum@table : Rank-1 count-coordinate patching on four checkpoints in two families [ 20 , 8 ] ,

2{,}178

scored runs, thresholds pre-committed before any audio. P1 ratio: unrelated-to-informative disruption. Worst cell: largest median shift across donor kinds as a fraction of a full donor-to-receiver transfer (noise-floor bar: below

25\%

). Bound:

90\%

TOST equivalence bound on the informative cross-

k

cell in the same units, the smallest excludable effect.

Explore similar work

Repeated-Token Counting Reveals a Dissociation Between Representations and Outputs
May 10, 2026Sohan VenkateshLarge Language Models Fail Counting
Manipal Institute of Technology Bengaluru
Taming Long-form Text-to-Speech
Sep 15, 2026Rongxiang Wang, Berkin Durmus, Aysegul Orhon +2Autoregressive Text-To-Speech Voice Conversion
Argmax, Inc. · University of Virginia · Bilkent University
Reliable Neural-Codec Text-to-Speech by ASR Self-Verification and Distillation: Near-Zero Catastrophic Failures Across Models and Codecs
Jun 16, 2026Ali Asaria, Tony Salomone, Deep GandhiAutoregressive Text-To-Speech
Transformer Lab

May 10, 2026, cs.CLJ/K move · Enter open · S save

Repeated-Token Counting Reveals a Dissociation Between Representations and Outputs

Sohan Venkatesh

Manipal Institute of Technology Bengaluru

Large Language Models Fail Counting Multilayer Perceptrons Residual Stream Dissociation Reasoning Benchmark

Open paper AskPDF arXiv

Large language models fail at counting repeated tokens despite strong performance on broader reasoning benchmarks. These failures are commonly attributed to limitations in internal count tracking. We show this attribution is wrong. Linear probes on the residual stream decode the correct count with near-perfect accuracy at every post-embedding layer, across all model depths. This holds even at the exact layers where the wrong answer crystallizes while the model simultaneously outputs an incorrect count. Attention patterns show no evidence of collapse over repeated tokens and tokenization artifacts account for none of the failure. Instead, a format-triggered multi-layer perceptron (MLP) block overwrites the correctly-encoded count with a fixed wrong answer at roughly 88--93,% network depth. This prior fires for repeated word-tokens in space-separated list format and is absent for repeated digit-tokens. It is suppressed by comma-separated delimiters in larger models but persists in smaller ones. The finding holds across Llama-3.2 (1B and 3B) and Qwen2.5 (1.5B, 3B and 7B) at consistent relative depth. Counting failure is a failure of routing not of representation and the two require different interventions.

Abstract

Explore similar work

Repeated-Token Counting Reveals a Dissociation Between Representations and Outputs

Taming Long-form Text-to-Speech

Reliable Neural-Codec Text-to-Speech by ASR Self-Verification and Distillation: Near-Zero Catastrophic Failures Across Models and Codecs