Refinement Buys Intelligibility, Search Buys Identity: What Test-Time Compute Buys in Masked-Diffusion TTS

Nityanand Mathur; Hamees Sayed; Ayush Pratap Singh

cs.AIOct 2, 2026

Refinement Buys Intelligibility, Search Buys Identity: What Test-Time Compute Buys in Masked-Diffusion TTS

Authors: Nityanand Mathur, Hamees Sayed, Ayush Pratap Singh

Organizations: Work done outside of Blackstar Inc. · Work done at Smallest AI.

Abstract

Diffusion language models for text-to-speech combine two forms of computation: model depth (parameters) and refinement steps (inference budget). We ask whether they scale equally across capabilities. We train 15 masked-diffusion codec TTS models varying depth (19-133M parameters, 3 seeds) on 2,000 hours of speech and sweep refinement steps T in [1,16] at inference, measuring zero-shot synthesis via ASR word error rate (intelligibility) and speaker verification (identity) on 174 held-out speakers. Against measured floors, refinement closes 86.2% of the intelligibility range but only 46.4% of the identity range - a 1.86x asymmetry robust across multiple error metrics. Retraining at 3x and 6x schedule attenuates but does not reverse this gap (1.84 to 1.36 to 1.23x), because intelligibility saturates with steps while identity continues improving. Best-of-K search recovers speaker identity where refinement fails, with 64.6-79.0% win rates across four independent encoders. Depth and steps are not interchangeable: separable B(d)B(T) fits significantly better (Delta AICc=+69.3) than substitution models. Analysis shows 62% of remaining identity deficit lies in the codec, not the generator. We conclude that refinement and depth target different bottlenecks and should be optimized separately.

Figures & tables

Panel of Figure 1 — Figure 1: The two ways a masked-diffusion TTS model buys sequential computation. Phonemes ( Bernard and Titeux, 2021 ) and a 3-second prompt are always visible; the generated region of the $8\times\text{frames}$ token grid starts fully masked . The backbone spends depth ( owned serial compute, fixed at training time), and the sampler spends $T$ refinement steps per codebook level ( rented serial compute, chosen at serve time). One training grid over $({\color[rgb]{0.7695,0.3281,0.1133}w},{\color[rgb]{0.1367,0.2773,0.6133}d})$ therefore yields a whole $({\color[rgb]{0.7695,0.3281,0.1133}w},{\color[rgb]{0.1367,0.2773,0.6133}d},{\color[rgb]{0.0586,0.4805,0.4453}T})$ surface at inference. This paper asks whether the rented kind helps every capability equally.

${\color[rgb]{0.0586,0.4805,0.4453}T}$	intelligibility (WER)	identity (SIM-o)	ratio
	fraction of reachable range closed
2	32.2% [29.7, 34.4]	20.0% [19.1, 20.9]	1.61 $\times$
4	69.8% [66.0, 73.1]	35.7% [34.3, 37.0]	1.96 $\times$
8	81.8% [78.3, 84.8]	43.5% [42.1, 44.9]	1.88 $\times$
16	86.2% [82.8, 89.1]	46.4% [45.0, 47.8]	1.86 $\times$
32	87.9% [84.9, 90.5]	47.5% [46.2, 48.8]	1.85 $\times$

Table 1: What refinement buys, as a fraction of what is reachable. Share of the distance from one-step decoding to a measured reference, Whisper’s word error on the real recordings (0.0345) and the speaker similarity of our codec’s round trip (0.5554), closed by step budget

{\color[rgb]{0.0586,0.4805,0.4453}T}

, on the full 45-run grid. Percentile CIs from a 2000-replicate run-level bootstrap. No model is fitted, and the ratio is invariant to monotone reparameterisation of the error (1.68

\times

–1.86

\times

, Table 2 ).

error coordinate	$\tau_{\mathrm{WER}}$	$\tau_{\mathrm{SIM}}$	$\Delta\tau$	gap-closed ratio
identity (pre-registered)	0.8362	0.7260	+0.1102	1.86 $\times$
log(1+err)	0.6665	0.6913	-0.0248	1.85 $\times$
sqrt(err)	0.4325	0.6842	-0.2517	1.68 $\times$
err capped at 1	0.7093	0.7260	-0.0167	n/a
err squared	1.5613	0.8122	+0.7491	1.82 $\times$
affine 3+2*err (v1.0 check)	0.7600	0.7249	+0.0351	n/a

Table 2: The exponent contrast is a property of the error coordinate. The same 75 measurements, refitted after a monotone reparameterisation of both errors.

\tau

is invariant to the affine map, as our original (v1.0) pre-registration assumed; under the non-affine maps

\Delta\tau

changes sign, while the model-free ratio of Table 1 does not. This is why the paper’s claim is stated as a share of a reachable range rather than as an exponent difference.

training	Whisper	wav2vec2-large-960h-lv60-self	share	ratio	95% CI
	intelligibility share		identity
30k	90.6%	88.5%	49.3%	1.84 $\times$	1.73, 1.97
90k	97.1%	99.2%	71.5%	1.36 $\times$	1.28, 1.44
180k	97.0%	99.1%	78.8%	1.23 $\times$	1.18, 1.29

Table 3: Refinement’s intelligibility share saturates; its identity share does not. The same 9 runs (C1/C3/C5

\times

3 seeds) retrained to three budgets and measured identically: share of the reachable range closed from

{\color[rgb]{0.0586,0.4805,0.4453}T}{=}1

{\color[rgb]{0.0586,0.4805,0.4453}T}{=}16

. The second intelligibility column re-transcribes the identical audio with wav2vec2-large-960h-lv60-self, a CTC recogniser sharing neither architecture, objective nor training corpus with Whisper; it is reported so that saturation cannot be an artefact of one recogniser. Intervals cluster the bootstrap on configuration (seeds within a config are not draws from config space, so the effective

n

is 3) and resample evaluation items as a second dimension.

Figure 2: Left: iso-error contours in

({\color[rgb]{0.7695,0.3281,0.1133}w},{\color[rgb]{0.1367,0.2773,0.6133}d})

{\color[rgb]{0.0586,0.4805,0.4453}T}{=}16

. The two capabilities do not share a preferred shape, but the width amplitude saturates its bound, so the anisotropy ratio

\rho

is not identified. Right: iso-WER contours in

(\log{\color[rgb]{0.0586,0.4805,0.4453}T},\log{\color[rgb]{0.1367,0.2773,0.6133}d})

: a single substitution factor

d\,T^{\kappa}

cannot reproduce both the depth ordering at fixed

T

and the

T

ordering at fixed depth.

axis	change	$\Delta$ SIM-o	cost	scorer
training compute	30k $\to$ 90k steps	+0.0827	$3\times$ training	WavLM-L
(further)	90k $\to$ 180k	+0.0155	$2\times$ again	WavLM-L
test-time search	best-of-8 vs ${\color[rgb]{0.0586,0.4805,0.4453}T}{=}16$	+0.0365	+0.0425 WER, $8\times$ NFE	WavLM-L
parameters	20 M $\to$ 276 M	+0.0531	$14\times$ parameters	WavLM-L
guidance	$\gamma=0.5$	+0.0087	+4.41 WER points	WavLM-L
rate matching	per-item duration	+0.0044	free	WavLM-L

Table 4: What buys speaker identity. Absolute SIM-o gain per axis, set against the 0.0747 of headroom between our best system and a codec round trip of real audio. The headroom is a WavLM-large quantity, so every row is reported under that same encoder (final column); the cross-encoder agreement evidence for the search row is separated out into Table 6 rather than mixed into this column. Training compute and test-time selection buy the most. Guidance buys a little and charges for it in WER; rate matching and longer prompts buy nothing.

Figure 3: Identity is mostly not the model’s to win. (A) what each axis buys in speaker similarity, against the headroom that remains to the codec ceiling; bars are coloured by which lever they are ( rented at serve time , owned in training , shape ). Refinement’s bar is the largest only because it is measured from one-step decoding, which is not an operating point anyone ships; from the deployable

{\color[rgb]{0.0586,0.4805,0.4453}T}{=}16

default it buys nothing further, and test-time search is what moves identity from there. (B) of the distance between real audio and our best system, 62% is lost in the codec itself.

NFE	refinement	search	$\Delta$ ECAPA	95% CI	WER (refine / search)
128	${\color[rgb]{0.0586,0.4805,0.4453}T}{=}16$	$K{=}2$	+0.0153	[0.0127, 0.0179]	0.1530 / 0.1938
256	${\color[rgb]{0.0586,0.4805,0.4453}T}{=}32$	$K{=}4$	+0.0322	[0.0307, 0.0338]	0.1335 / 0.1947
512	${\color[rgb]{0.0586,0.4805,0.4453}T}{=}64$	$K{=}8$	+0.0444	[0.0416, 0.0470]	0.1316 / 0.1955
replicated at NFE 512 outside the grid: 276 M budget +0.0422 [0.0393, 0.0448]; $3\times$ training +0.0368 [0.0321, 0.0428]

Table 5: Refinement and search buy different capabilities at the same cost. Matched NFE (

8T

per candidate); best-of-

K

candidates are selected by WavLM-SV and scored by a separate ECAPA encoder. Refinement wins intelligibility at every tier, and search wins identity at every tier.

∗ selects the candidates, so this row is circular and is not counted as agreement.
encoder	family / role	win rate	95% CI	raw $\Delta$
ECAPA-TDNN ( Desplanques et al., 2020 )	ECAPA, gated	72.3%	[70.6, 73.8]	+0.0444
x-vector ( Snyder et al., 2018 )	TDNN	79.0%	[77.5, 80.5]	+0.0075
GE2E d-vector ( Wan et al., 2018 )	LSTM d-vector	69.1%	[67.5, 70.8]	+0.0164
WavLM-base+ SV	WavLM, other scale	64.6%	[62.9, 66.3]	+0.0173
WavLM-large SV ∗	WavLM / selector	84.9%	[83.6, 86.1]	+0.0803

Table 6: The search result is not an artefact of the selector’s lineage. The same 200 items per run, 15 runs, rescored end-to-end by five speaker encoders spanning four model families. The selector is WavLM-large SV, so its own row is circular and is shown only for reference. A per-item win rate (how often an encoder ranks the search output above the refinement output at matched NFE) is invariant to any monotone rescaling of that encoder’s cosine, which a difference of means is not; we therefore compare win rates across encoders and report each raw

\Delta

only beside its own scale. Every family, including the three whose absolute cosines fail our ground-truth gate (median same-speaker cosine

\geq 0.50

, cross-speaker

\leq 0.25

) purely on scale while separating speakers at AUC

\geq 0.98

, puts search ahead with an interval clear of one half.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

id	claim	verdict
H-D1	fixed- $T$ anisotropy $\Delta\rho>0$	not supported
H-D2	$\Delta\tau>0$ , metric-selective steps	supported §
H-D3	depth–step exchange rate $\kappa$	not supported
H-D4	small budgets predict the largest	not supported
H-E1a	$T^{*}_{\mathrm{WER}}>16$	supported
H-E1b	extended $\tau$ inside the v1.0 CI	refuted

Appendix

Table 7: The complete pre-registration ledger. Every hypothesis declared before the first run (H-D: main grid, v1.0; H-E: extensions; H-S: speaker identity), with its verdict, including the four that failed and the two that split. We report the failures beside the successes because a ledger that shows only its winners cannot be used to judge the winners. discordant is a declared outcome class, used where the two pre-specified lenses (primary and secondary tests) disagree, and is not a synonym for refuted.

13/15 runs positive, but a per-item sign test gives $p=0.051$
intervention	setting	$\Delta$ SIM-o	95% CI	$\Delta$ WER (pts)
contrastive guidance	$\gamma=0.5$	+0.0087	[0.0063, 0.0115]	+4.41
	$\gamma=1.0$	-0.0021	[-0.0059, 0.0023]	+8.94
	$\gamma=2.0$	-0.0322	[-0.0367, -0.0274]	+19.10
rate matching	per-item duration	+0.0044	[0.0017, 0.0067]	+0.86

Appendix

Table 8: Inference-time interventions that do not buy identity. Speaker-contrastive guidance trades intelligibility for similarity and degrades monotonically in

\gamma

; per-item rate matching gives a small mean gain that is not reliable item-by-item.

prompt	SIM-o	WER
1.5 s	0.3213	0.1578
3 s (training length)	0.3839	0.1474
6 s	0.3724	0.7692
9 s	0.3759	0.8642

Appendix

Table 9: Longer prompts do not buy identity, and destroy intelligibility. Prompt length swept at inference on 15 runs of the 125 M budget, 397 items, every arm scored against the same fixed 3 s reference. Training used 3 s prompts only, so the 6 and 9 s arms test extrapolation. There the output stays fluent and speaker-consistent, but it decouples from the conditioning text.

schedule	WER	SIM-o	UTMOS	degenerate
coarse [25,1,…,1]	0.2191	0.3057	2.17	0.1%
uniform [4,…,4]	0.3144	0.3477	2.51	0.5%
fine [1,…,1,25]	1.1480	0.2246	1.56	39.1%

Appendix

Table 10: Where the refinement passes are spent matters as much as how many. Three frozen per-level schedules (steps per codebook level, coarsest first) at matched total NFE

{}=32

on the 125 M budget, 15 runs, 400 items. Spending every pass on the coarsest codebook level rather than the finest cuts WER by 5.2

\times

, and costs identity and naturalness against a uniform split.

Metric	$\alpha$ ( width )	$\beta$ ( depth )	$\tau$ ( steps )	$\kappa$	$\Delta$ AICc sub-sep	$\Delta$ AICc full-N
WER	1.574	2.040	0.836 [0.825, 0.851]	3.000 †	$+69.3$	$-15.5$
SIM-o	0.919	0.553	0.726 [0.709, 0.744]	3.000 †	$+71.3$	$-5.5$
† at the fitted bound, i.e. not identified; $\Delta\tau=\mathbf{0.110}$ , CI [0.092, 0.131].

Appendix

Table 11: Fitted exponents on the 75-point surface (

M_{\mathrm{sep}}

for

\alpha,\beta,\tau

;

M_{\mathrm{sub}}

for

\kappa

), with 95% run-level bootstrap intervals on the step exponents. The primary result is

\Delta\tau=0.110

, CI [0.092, 0.131], which excludes zero.

\kappa

sits on its bound in both rows, and the substitution form is worse than the separable one by

\Delta\mathrm{AICc}_{\mathrm{sub-sep}}\gg+4

, so the depth–step exchange rate has no finite estimate. The negative

\Delta\mathrm{AICc}_{\mathrm{full}-N}

(

M_{\mathrm{full}}

, the separable form without its

T

term, fitted at

T{=}16

, against a single power law in

N

) says that shape matters at fixed

N

variation	what changes	$\Delta\tau$	95% CI
none (75 points, 45 runs)	n/a	0.110	[0.092, 0.131]
fourth budget, 276 M	10 further runs	+0.1095	[0.0951, 0.1241]
$3\times$ training compute	3 configs at 90k steps	+0.1767	[0.0789, 0.2854]
ASR swapped	Whisper-medium.en	+0.1283	[0.1064, 0.1492]
unbounded amplitudes	$\log A,\log B,\log C$ free	+0.1260	[0.1043, 0.1474]

Appendix

Table 12:

\Delta\tau

survives variation in scale, training compute and the measurement stack. Each row refits the surface with one factor changed: a fourth parameter budget,

3\times

training compute, a different ASR, and an unbounded reparameterisation of the fitted amplitudes. Every interval excludes zero.

Figure 4: Test-time scaling is metric-selective. Error versus refinement steps for all 15 configs, each normalised to its own

{\color[rgb]{0.0586,0.4805,0.4453}T}{=}1

value; thick lines are grid means. (A) WER error falls 0.952 in absolute terms over

{\color[rgb]{0.0586,0.4805,0.4453}T}{=}1\to 16

and is still falling at

{\color[rgb]{0.0586,0.4805,0.4453}T}{=}16

. (B) SIM-o error falls 0.161 over the same range, 5.9

\times

less. Same models, same frozen sampler, same items; only

T

differs.

Figure 5: H-D4 extrapolation. The two smaller budgets are fitted and the largest budget predicted, under

M_{\mathrm{full}}

and the

N

-only model. SIM-o extrapolates (mean absolute percentage error, MAPE, 2.4 % vs 2.2 %); WER does not (20.2 % vs 20.4 %).

Explore similar work

Beyond Token Revision: Investigating Mask-and-Replace Diffusion for Zero-Shot Text-to-Speech
Oct 7, 2026Hounsu Kim, Joonyong Park, Yuki Saito +2Zero-Shot TTS Audio Diffusion Models
KAIST, Republic of Korea · The University of Tokyo, Japan · National Institute of Advanced Industrial Science and Technology (AIST), Japan
LLaDA-TTS: Unifying Speech Synthesis and Zero-Shot Editing via Masked Diffusion Modeling
Mar 27, 2026Xiaoyu Fan, Huizhi Xie, Wei Zou +1TTS Synthesis Zero-Shot TTS
BRVoice Team, Bairong, Inc., China
Diffusion Language Models for Speech Recognition
Apr 15, 2026Davyd Naveriani, Albert Zeyer, Ralf Schlüter +1Automatic Speech Recognition Diffusion Language Model Decoding
Machine Learning and Human Language Technology Group, RWTH Aachen University, Germany · AppTek, Germany

Oct 7, 2026J/K move · Enter open · S save

Beyond Token Revision: Investigating Mask-and-Replace Diffusion for Zero-Shot Text-to-Speech

Hounsu Kim, Joonyong Park, Yuki Saito, Satoru Fukayama+1

KAIST, Republic of Korea · The University of Tokyo, Japan · National Institute of Advanced Industrial Science and Technology (AIST), Japan

Zero-Shot TTS Audio Diffusion Models Speech Language Models

Open paper AskPDF arXiv

Unlike autoregressive models, discrete diffusion-based models for zero-shot text-to-speech generate speech tokens in parallel and can revisit earlier predictions. Mask-and-replace training extends mask-only training by randomly replacing some tokens, and its gains are commonly attributed to self-correction, the ability to revise previously generated tokens. However, exposure to randomly perturbed context during training may itself improve generation, raising the question of whether these gains require inference-time token revision. To investigate this question, we use DeMaR, which combines mask-and-replace training with confidence-ranked mask-only sampling while preserving the total training corruption probability. Trained from scratch on LibriTTS, DeMaR achieves lower word error rates (WER) than autoregressive and mask-only diffusion baselines using the same speech tokenizer. This advantage persists when each token remains unchanged after first being unmasked. Matched training conditions on two heterogeneous speech tokenizers show that both noisy-context augmentation and replacement supervision improve WER under this restriction. These findings demonstrate training-side benefits of replacement beyond enabling inference-time token revision.

Abstract

Explore similar work

Beyond Token Revision: Investigating Mask-and-Replace Diffusion for Zero-Shot Text-to-Speech

LLaDA-TTS: Unifying Speech Synthesis and Zero-Shot Editing via Masked Diffusion Modeling

Diffusion Language Models for Speech Recognition