GRAFT: Restoring the Missing Sampling Interface in Continuous Diffusion Language Models

Haoning Yang; Lili Pan

cs.CLOct 21, 2026

GRAFT: Restoring the Missing Sampling Interface in Continuous Diffusion Language Models

Authors: Haoning Yang, Lili Pan

Organizations: University of Electronic Science and Technology of China

Abstract

Continuous diffusion language models generate a whole sequence in parallel, never drawing a single word at any intermediate step; only the final step turns embeddings into text. Models such as ELF now report autoregressive-level quality at a fraction of the sampling steps of discrete-token diffusion models, and this design principle -- no step-wise discretization -- is what their own ablations credit for the gain. We show that this same principle carries a hidden cost: a collapse of the lexical long tail. Measured against held-out human text, ELF's output matches the human rate on common words almost exactly, yet its rarest words appear at only 0.30x the human rate, a gap that does not close from 105M to 652M parameters and that no sampler setting these models expose can reach. A same-size GPT-2, in contrast, spans zero to above the human rate by its decoding rule alone. We trace the difference to what continuous diffusion models lack: with no step that ever draws a discrete word, every step is instead driven by an average over the candidates still in play, and averaging a rare candidate together with the common word competing for its slot lands on the common word. An autoregressive model avoids this because its decoding rule samples one candidate rather than averaging over all of them. We propose GRAFT, a training-free method that grafts this missing choice back onto a frozen model: while the model is still undecided, GRAFT commits the rare words the average would otherwise discard, then regenerates the sequence from scratch with those commitments held in place. Across three ELF scales and on LangFlow, an independent continuous dLLM, GRAFT restores the rare-word share of generated text from 0.42-0.77x the human rate to 0.82-1.10x, a relative gain of 34-109%, and moves the entire word-frequency profile closer to human on every model tested, by 2.7x on ELF-B and 12x on LangFlow.

Figures & tables

Figure 1: Lexical-tail coverage depends on whether a model samples discrete words during generation. (a) ELF-L matches human usage for common words but substantially underproduces rare ones, reaching only

0.72\times

0.32\times

, and

0.30\times

of the human rate across the three rarest Zipf bands. (b) GPT-2 and LLaDA, which sample discrete words during decoding, reach or approach the human rare-word rate, while ELF and LangFlow remain below it. Graft restores the missing tail, raising ELF and LangFlow to

1.13\times

and

1.25\times

of the human rate (three-seed means, Appendix E.6 ).

Zipf bucket (% of word tokens)	ELF-B	ELF-M	ELF-L	Human	ELF-L / Human
$\geq 7$	20.60	20.62	21.02	20.38	$1.03\times$
$[5,7)$	53.07	51.59	48.13	46.20	$1.04\times$
$[4,5)$	20.81	20.69	22.64	21.27	$1.06\times$
$[3,4)$	3.14	4.93	5.82	8.12	$\mathbf{0.72\times}$
$[2,3)$	0.44	0.68	0.80	2.51	$\mathbf{0.32\times}$
$<2$	0.27	0.24	0.26	0.86	$\mathbf{0.30\times}$

Table 1: Word-frequency profile of unconditional generation (1,000 samples each) against held-out human text from the training distribution. High-frequency buckets match human almost exactly; the deficit is confined to the tail, and the rarest bucket does not move from 105M to 652M parameters.

Figure 2: Where rare words are lost. ELF uses a denoising head (MSE) at every timestep and a decoding head (CE) only at the final readout. Right: on clean held-out human embeddings, the decoding head recovers all words across

124{,}212

positions (Finding 1). Left: on noised text at an intermediate generation level, errors increase sharply only after denoising and primarily on rare words: on ELF-B, the gap grows from

1.0\%

vs.

0.7\%

in the commonest band to

21.9\%

vs.

7.3\%

[3,4)

(Finding 2).

Figure 3: Graft commits rare words from the first denoising step and uses restarts to preserve them. Left: each of the $R$ rounds starts from the same noise with all previously committed words fixed, runs only to the read time, and commits $m$ additional rare words; a final full trajectory then generates the output with all commitments held in place. Top right: at the read time, Graft commits a rare word that retains substantial probability but loses to the top choice, while the argmax control commits that top choice instead. Bottom: three rules prevent committed words from creating non-words.

	lexical tail, and its control			distance to human profile
	tail %	$\times$ human	non-word %	JSD $\,\downarrow$	$\|\Delta\zeta\|\,\downarrow$	types
Human reference	11.88	1.00	1.14	—	—	17 726
ELF-B (105M)
Stock	5.01	0.42	2.64	.0134	.182	11 795
Stock, best-of-4	5.97	0.50	1.79	.0101	.175	11 482
Argmax control	5.31	0.45	2.42	.0133	.209	12 124

Table 2: Main results on unfiltered output (

n{=}200

per arm). Graft rows and the stock rows of ELF-L and LangFlow average three seeds; the remaining rows are single arms. The LangFlow tuned row uses the held-out dose of § 7.1 ; no ELF arm is tuned. Best-of-4 is the equal-budget control of § 7.4 . Bold marks the value closest to human within each model.

	tail %	non-word %	fidelity %
stock	4.54	2.35	—
$+$ rare commitment, no rule	7.92	4.14	86.7
$+$ atomic commitment	8.86	2.71	87.4
$+$ splicing	9.61	2.47	100.0
$+$ word-boundary co-pinning	9.10	1.78	100.0

Table 3: Ablation of the three commitment rules on ELF-B. Each row adds one rule to the previous setting; Fidelity is the fraction of committed words preserved in the final output. The last row also changes the sampler’s step distribution, so its

0.5

-point tail reduction reflects both effects. The full Graft setting further includes the two gates and the anti-repetition correction.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Alternative explanation	Measurement that rules it out
The decoding head favors common words	Clean embeddings of human text are decoded at $100.00\%$ , with a retention of $1.000$ in every band ( $n{=}124{,}212$ positions).
Rare words have smaller decision cells, the regions of embedding space that decode to them	They do, but only under noise far larger than the denoising head’s own error. With isotropic noise of per-dimension $\sigma{=}4$ , $96.1\%$ of the rarest words flip, that is, decode to another word, compared with $55.4\%$ of the commonest; at $\sigma{\leq}1$ , which covers the denoising head’s own errors ( $\sigma{\approx}0.3$ – $0.9$ ), at most $0.3\%$ of any band flips. On the segment from the competitor’s embedding ( $\kappa{=}0$ ) to the true word’s ( $\kappa{=}1$ ), the flip boundary is symmetric, at $\kappa^{*}{=}0.146$ for rare words and $0.152$ for common ones, both far from $0.5$ .
The output is an average of two candidates	Projected onto this segment, the denoising head’s output sits at $\hat{\kappa}{=}0.429$ and would still decode to the true word, but the perpendicular residual is $0.65\times$ the segment’s length, so the output is not on the segment.
The self-conditioning feedback of Zhang et al. (2026)	Sweeping the self-conditioning strength $\alpha$ over $0$ , $0.25$ , $0.5$ , $0.75$ , and $1$ leaves the tail flat ( $4.85$ , $4.84$ , $4.29$ , $4.49$ , and $4.72$ ), even at $\alpha{=}0$ , where Zhang et al. (2026) report generative perplexity rising to $97.5$ (ELF-B, 64 steps).
A single error direction, removable by a one-dimensional fix	The frequency-axis direction, the normalized difference between the mean embeddings of common and rare positions, carries only $4$ – $15\%$ of the error norm, and removing it does not recover accuracy ( $2.87\%\to 3.12\%$ ).
The denoising head picks the commonest candidate	At $35{,}513$ contested positions, taken at $t{=}0.25$ where the decoding head is uncertain and its most common candidate is not its most probable, the denoising head chooses the most common candidate only $10.1\%$ of the time (the most probable one, $20.8\%$ ). Where its word is among the candidates ( $47.4\%$ of these positions), its mean frequency rank is $2.69$ , against $3.08$ for a random candidate: a real but weak bias.

Appendix

Table 4: Eight alternative explanations for the collapse, each with the measurement that rules it out (ELF-B unless noted). Section 5 gives the mechanism that remains.

Arm	accept %	tail@acc	tail (unfiltered)	non-word (unfiltered)
GPT-2 greedy	0.0	—	0.00	0.00
GPT-2 top- $k$ 50	90.3	5.28	5.56	0.24
GPT-2 nucleus $0.95$	84.9	9.80	10.06	1.17
GPT-2 sample $T{=}1$	51.7	13.36	14.60	3.01
GPT-2 medium greedy	0.0	—	0.26	0.00
GPT-2 medium top- $k$ 50	92.6	5.92	6.14	0.25

Appendix

Table 5: Autoregressive reference arms under the full accept-then-measure protocol with the dual bar (

n{=}496

–

500

; greedy,

n{=}100

). Pure sampling reaches the deepest tail at the worst quality, and nucleus sampling is the quality point. The contrast of § 4 concerns whether such a knob exists, not the quality at its extremes.

Arm	tail (unfiltered)	non-word (unfiltered)	top-5k tail
LangFlow stock	8.59	2.44	11.05
LangFlow champion	15.09	3.27	13.70
LangFlow champion $+$ co-pinning	16.20	2.41	16.48
LangFlow ladder (5)	12.35	1.98	15.87
ELF-B stock	4.54	2.35	6.61
ELF-B champion	9.45	2.52	11.26

Appendix

Table 6: Left: unfiltered tail and non-word rates, with no acceptance filter. Right: the top-5k tail of Appendix A.5 on the accepted set (dual bar). The tail gains survive without filtering, and unfiltered non-words do not rise with co-pinning or the ladder, although they do for the two champion rows. Orderings are unchanged under the top-5k tail, on which all three non-stock LangFlow rows overshoot the human level of

12.35

; we report the overshoot as measured.

Excess error (denoising / decoding)	Zipf $\geq 6$	$[4,6)$	$[3,4)$	$[2,3)$	$<2$
ELF-B	$1.28\times$	$1.91\times$	$2.99\times$	$2.83\times$	$2.93\times$
ELF-M	$1.00\times$	$1.23\times$	$1.91\times$	$2.20\times$	$2.25\times$
ELF-L	$1.45\times$	$1.47\times$	$2.21\times$	$2.51\times$	$1.97\times$
ELF-B error rates (%)	$1.0/0.7$	$9.3/4.9$	$21.9/7.3$	$19.8/7.0$	$16.1/5.5$

Appendix

Table 7: Per-position error rate of the denoising head divided by that of the decoding head, per Zipf band, on the same network and the same input at noise level

t{=}0.5

on ELF’s time axis (matched input, with the decoding head’s time argument at

0.714

). The last row gives the ELF-B error rates behind the ratios (denoising / decoding, %), which Figure 2 shows. The setup and the other three noise levels are given in the text.

Model	champion $\Delta$ tail@acc	champion $+$ co-pinning: accept % / tail@acc % / $\Delta$
ELF-S (43.7M)	$+0.9$ (unfiltered, ${\sim}1$ /sample)	—
ELF-B (105M)	$+4.59$ to $+4.92$	69.5	9.68	$+5.39$
ELF-M (342M)	$+5.96$ to $+6.30$	64.0	11.81	$+6.58$
ELF-L (652M)	$+4.86$ to $+5.95$	63.5	12.98	$+6.32$

Appendix

Table 8: Gains across model scale under the accept-then-measure protocol with the dual bar (ungated runs; the champion runs without the step-schedule shift on ELF-M and ELF-L). The champion column gives the range over three seeds, and the champion

+

co-pinning columns give the first seed. ELF-M and ELF-L reach and pass the human accepted-set tail of

11.17

. The ELF-S row is an unfiltered tail gain rather than an accepted-set gain (see text).

Sampler	accept %	tail@acc	$\Delta$ tail@acc [95% CI]	4-gram rep.	non-word %
Stock	60.5	8.28	—	0.53	2.44
MCB $T{=}1.0$	77.9	7.77	$-0.51$ [ $-1.02,-0.02$ ], $p{=}.038$	0.00	1.87
MCB $T{=}0.8$ , $p{=}0.9$	19.8	4.77	$-3.51$ [ $-4.49,-2.48$ ]	20.10	0.74
MCB $T{=}0.6$ , $p{=}0.9$	${\sim}0$	—	off-manifold breakdown	35.26	4.23
champion $+$ co-pinning	52.0	13.61	$\mathbf{+5.33}$ [ $+3.40,+7.46$ ]	1.28	2.41

Appendix

Table 9: Comparison with MCB on the same LangFlow checkpoint, under our protocol. Stock and MCB run at an equal number of function evaluations (NFE), and MCB is our port of the authors’ released sampler to this checkpoint. The champion

+

co-pinning row (ungated) costs about

2.1

full trajectories (Appendix E.1 ) and is its seed-

42

run, the seed of the MCB arms; over three seeds, its gain is

+3.07

+5.33

, and the gated Graft arm gains

+4.80

on seed

42

(Table 15 ). In the Sampler column,

p

is MCB’s top-

p

threshold; in the

\Delta

column, it is a bootstrap

p

-value. At every temperature we could run, MCB trades cleanliness for tail in the wrong direction.

Arm	tail (unfiltered)	4-gram rep.	types/100k	accept (%)	$\Delta$ tail@acc vs. argmax
argmax (interface shut)	6.89	80.9	5,755	7.2	(reference)
low-confidence, $T{=}0.5$	6.85	77.6	6,002	13.5	$-1.56$ , n.s.
low-confidence, $T{=}1$	7.15	63.8	7,919	17.0	$+0.46$ , n.s.
random-order, $T{=}1$	11.16	0.0	28,599	55.8	$+3.41$ $[+1.01,+5.96]$ , $p{=}0.002$
Human (OpenWebText)	11.49	—	5,497	90.5

Appendix

Table 10: LLaDA-8B discrete-line control (semi-autoregressive decoding in blocks of 64, 16-token human prefixes, 512 tokens,

n{=}100

). The comparison is within the model: LLaDA is not trained on OpenWebText, so its bars are cross-domain. Opening the native sampling interface decides the outcome: random-order remasking at

T{=}1

reaches the human tail with zero repetition, while the semi-greedy schedules do not. Types per 100k fall with sample size, so in that column the human row (

1{,}000

documents) is not comparable to the LLaDA rows (

100

samples). The argmax arm decodes at temperature

0

, which shuts the sampling interface; the other rows name the remasking rule of Appendix D , low-confidence or random-order, and its temperature; and n.s. marks a

\Delta

whose

95\%

interval includes zero.

Configuration	tail $z<4$ (%)	gen. perplexity	4-gram rep. (%)
ELF-B, 16 steps, $g{=}3$	7.05	44.51	0.63
ELF-B, 32 steps, $g{=}1$	3.88	29.23	1.21
ELF-B, 32 steps, $g{=}3$ (default)	3.75	24.18	2.84
ELF-B, 32 steps, $g{=}5$	4.02	23.40	2.87
ELF-B, 64 steps, $g{=}3$	4.57	18.97	7.61
ELF-B, final-step sampling $T{=}1.0$	4.72	—	—

Appendix

Table 11: The sampler sweep on ELF, under the alphabetic rule. The ELF-B step-and-guidance rows have

300

samples each, and the ELF-M and ELF-L rows

1{,}000

at ELF’s 64-step preset (

\gamma{=}1.0

). The two final-step sampling rows come from a separate ELF-B batch, and the

1{,}000

-sample default batch of Table 1 reads a tail of

3.85

. The ELF-B 64-step row also changes the SDE churn from

\gamma{=}1.5

1.0

, and dashes mark quantities not scored for a configuration. Generative perplexity improves monotonically along the step axis while repetition rises

12\times

, and no configuration brings the tail near the human rate.

Model	Arm	tail@acc	foreign %	$\Delta$ raw	$\Delta$ discounted
human	—	11.17	0.8	—	—
ELF-B	stock	4.30	1.6	—	—
	fixed	9.67	13.8	$+5.37$	$+4.10$
	fixed, gated	9.54	5.0	$+5.24$	$\mathbf{+4.83}$
	tuned	12.53	24.5	$+8.23$	$+5.23$
	tuned, gated	11.65	2.1	$+7.35$	$\mathbf{+7.18}$

Appendix

Table 12: Foreign-dominant share of the accepted-set tail, and the gain with and without the foreign discount. All

\Delta

are against the same-model stock sampler, and every discounted

\Delta

shown is significant at

p{<}0.001

. Gated arms carry both gates of § 6 ; tuned is the

(8,5,3)

band quota at

q_{\min}{=}0.02

, and fixed is the fixed dose at

q_{\min}{=}0.08

. ELF rows are seed

31337

, except the ELF-B row fixed, gated , which averages three batch-16 runs with both gates; the LangFlow fixed row averages three seeds, and the LangFlow stock row is the seed-

42

run. The other ELF-B rows differ in their settings: the fixed row is a batch-8 run with the attestation gate and no language gate, and the tuned row is a batch-8 run with the attestation gate and the step-schedule shift. The tuned, gated row is the seed-

31337

run of the tuned arm of Appendix E.10 (batch

8

, both gates, default step schedule), so it differs from the tuned row in step schedule as well as in the language gate.

Sampler	accept %	tail@acc %	$\Delta$ tail@acc
ELF-B
stock	66.0	4.30	—
stock best-of-4	98.0	5.64	$+1.3$
argmax control	74.5	4.92	$+0.05$ (n.s.)
Graft	68.3	9.54	$+4.83$
Graft (tuned)	57.3	11.74	$+7.22$

Appendix

Table 13: Accept-then-measure protocol (

n{=}200

per arm; the human reference reads a tail@acc of

11.17

at an accept rate of

90.5\%

). In every row,

\Delta

is against the same-model stock sampler after the foreign discount (Appendix E.8 ). The Graft rows, the tuned rows, and the ELF-B ladder row average three seeds (

31337

55555

, and

77777

on ELF;

42

1042

, and

2042

on LangFlow), with each seed read against the same stock run. The stock, argmax-control, and LangFlow ladder rows are single runs, and each best-of-4 row selects among four stock runs. Every

\Delta

is significant at

p{<}0.001

on every seed, except those of the two argmax controls and of LangFlow’s tuned row (

p{=}0.29

0.017

, and

0.029

by seed). The ladder rows predate the gates and are ungated, and the ELF-B tuned row runs without the step-schedule shift (Appendix E.10 ). Among the Graft and ladder rows, bold marks an accept rate at or above the same-model stock rate. The ladder is a per-sample controller over ungated Graft variants, not Graft itself (Appendix A.3 ).

Model	Dose	selection	held-out 1	held-out 2
ELF-B	fixed	.0049	.0047	.0052
	tuned	.0012	.0012	.0012
ELF-M	fixed	.0082	.0083	.0072
	tuned	.0059	.0079	.0052
ELF-L	fixed	.0049	.0048	.0041
	tuned	.0047	.0045	.0033

Appendix

Table 14: Held-out dose selection: JSD to the human profile on unfiltered output. Fixed is the fixed dose,

m{=}8

q_{\min}{=}0.08

, and tuned is the band quota,

(8,5,3)

on ELF and

(0,1,1)

on LangFlow, both at

q_{\min}{=}0.02

. On LangFlow, the winner was chosen on the selection seed (

42

), and its held-out runs (

1042

and

2042

) played no part in the choice, although one losing candidate was ranked on seed

1042

(see text). On ELF, the columns are seeds

31337

55555

, and

77777

; the quota was chosen on ELF-B under an earlier configuration and reused on ELF-M and ELF-L, and on ELF-B the tuned arms also run without the step-schedule shift (see text). The tuned dose gives the lower JSD on every seed of every model.

Figure 4: JSD to the human profile against dose: ELF-B is U-shaped, LangFlow rises throughout, and the two optima sit

15\times

apart. Doses are commitments per

200

samples, read on unfiltered output from single-seed sweeps. On ELF-B, the minimum lies at

8{,}793

commitments, and every dose we ran improves on the stock sampler. On LangFlow, the JSD rises over the whole range we ran, so the lowest dose,

581

, is the best, and the fixed dose (

2{,}403

) lands above the stock sampler. Open rings mark the fixed dose of Table 2 and the tuned dose of Appendix E.10 , whose three-seed means are

.0049

and

.0012

on ELF-B and

.0058

and

.0002

on LangFlow, and dashed lines mark the stock samplers. The ELF-B points mix settings (see text).

Arm	commits	JSD $\,\downarrow$	$\|\Delta\zeta\|\,\downarrow$	accept %	$\Delta$ tail@acc	disc.
stock	—	.0025	.023	60.5	—	—
stock best-of-4	—	.0015	.002	99.0	$+1.45$	$+1.42$
quota $(0,1,1)$	581	.0001	.040	51.3	$+0.60$ (n.s.)	$+0.41$
quota $(0,1,1)$ ungated	781	.0003	.052	49.5	$+0.68$ (n.s.)	$+0.61$
quota $(1,1,0)$ ungated	1407	.0024	.079	42.4	$+2.49$	$+2.30$
quota $(1,1,1)$	1489	.0044	.091	39.2	$+2.89$	$+2.46$

Appendix

Table 15: LangFlow dose sweep on unfiltered output, ordered by commitments per

200

samples. Every row is a single run on seed

42

, except the best-of-4 row, which selects among four stock runs, and Graft , deeper, which is seed

1042

. Graft , deeper is the

z{<}3

band at

q_{\min}{=}0.02

; it and the rows marked ungated run without the attestation gate. The last two columns give the accepted-set gain against the LangFlow stock sampler, raw and after the foreign discount (Appendix E.8 ). The foreign share of the accepted-set tail is

1.2

–

1.4\%

for stock, best-of-4, and Graft , but

2.1

–

5.9\%

for the quotas and Graft , deeper, which reach deeper bands. For comparison, Table 2 reports three-seed means for stock (

|\Delta\zeta|

.011

), Graft (

.0058

.063

), and Graft (tuned) (

.0002

.045

Sampler	accept %	tail@acc	$\Delta$	$z{\geq}4$	$[3,4)$	$[2,3)$	$z{<}2$
Human reference	90.5	11.17	—	88.39	8.01	2.41	0.74
Stock	62.0	4.31	—	94.35	3.53	0.50	0.29
champion $+$ co-pinning ( $m{=}8$ )	68.0	9.65	$+5.34$	88.71	8.01	1.06	0.58
$+$ attestation gate	61.5	9.67	$+5.36$	88.68	8.15	1.01	0.51
Dose $m{=}16$ $+$ gate	74.0	14.39	$+10.08$	84.06	12.40	1.34	0.66
Band quota $(8,5,3)$	72.0	12.04	$+7.73$	86.50	8.54	2.40	1.10

Appendix

Table 16: Dose and placement on ELF-B, all against a batch-matched stock sampler, the stock run at the same batch size and seed (accept

62.0\%

, tail@acc

4.31

; see the batch control in the text). The four band columns are accepted-set shares with bootstrap CIs against human, and bold marks bands statistically indistinguishable from human. The band-quota and dose-16 arms without the attestation gate ran with an earlier German-only language gate, which their gated counterparts drop, so only the champion

+

co-pinning pair isolates the attestation gate (Table 18 ). The dose-16 and band-quota arms select at

q_{\min}{=}0.02

, against

0.08

for champion

+

co-pinning. The three samplers differ in where the tail lands, not in what it costs: the tail bought per point of clean-PPL is

0.25

–

0.26

for all of them.

ELF-B arm	$[4,5)$	$[3,4)$	$[2,3)$	$z{<}2$	non-word	JSD
Human	21.34	8.13	2.52	0.86	0.66	—
Stock	22.88	3.76	0.65	0.35	2.40	0.0124
Champion $+$ co-pinning ( $m{=}8$ )	20.24	8.23	1.12	0.64	2.30	0.0032
$m{=}8$ , $q_{\min}{=}0.02$	20.39	8.52	0.98	0.57	2.24	0.0038
$m{=}16$	18.03	12.79	1.23	0.74	2.01	0.0067
Band quota $(8,5,3)$	19.22	8.78	2.55	1.19	2.00	0.0011

Appendix

Table 17: Zipf profiles on unfiltered output, shown for the four rarer bands, with JSD to the human profile. As in Appendix A.2 , band shares are over lexicon words, with non-words removed and the shares renormalized; the non-word column is over all word tokens. Tables 1 and 6 keep non-words in the denominator and strip slightly more punctuation, so their band shares differ slightly (human

[4,5)

21.27

there,

21.34

here). The ELF-B rows are batch-8 runs on seed

31337

, and every JSD in this table is computed without the foreign discount. The stock, champion

+

co-pinning, and

(8,5,3)

rows are the runs of Table 16 without the attestation gate; the

m{=}16

row is the dose-16 run without that gate (Table 18 ), not the gated run of Table 16 , which reads

.0058

; and the

m{=}8

q_{\min}{=}0.02

and

(6,5,2)

rows appear only here. Every ELF-B row except stock and champion

+

co-pinning carries the earlier German-only language gate and selects at

q_{\min}{=}0.02

. The stock row reads

.0124

here against

.0134

in Table 2 mainly because the two stock runs differ (batch

8

here, batch

16

there): without the discount, the batch-16 run reads

.0135

, so the discount itself moves it by only

.0001

. On ELF-B, champion

+

co-pinning matches the common bands and

[3,4)

but leaves

[2,3)

at under half the human rate, a doubled dose overshoots

[3,4)

, and the two band quotas are the only arms that reach

[2,3)

, at the cost of depleting the common bands.

Model	Arm	commits	unattested	$\Delta$ tail@acc	with gate
ELF-B	champion $+$ co-pinning	6,220	$2.1\%$	$+5.34$	$+5.36$
ELF-B	band quota	9,517	$7.8\%$	$+7.73$	$+8.22$
ELF-B	dose $m{=}16$	12,800	—	$+9.91$	$+10.08$
LangFlow	champion $+$ co-pinning	2,429	$0.8\%$	$+5.33$	$+4.80$
LangFlow	quota $(1,2,3)$	2,640	$22.1\%$	$+8.41$	$+6.77$

Appendix

Table 18: Attestation audit. A commitment is unattested when the lowercased form of its token never appears in 30k OpenWebText documents (documents

1{,}000

–

30{,}999

of the reference shard, disjoint from the human reference and hence from the bar calibration). Contraction fragments ( wouldn , couldn ), an artifact of whitespace tokenization in the lexicon builder, make up

10

–

13\%

of the unattested counts and are excluded from the numerator of the percentages, whose denominator is all commitments. The gate drives the unattested count to zero in every arm. On ELF-B, only the champion

+

co-pinning pair isolates the gate, since the other ELF-B pairs also differ in a language gate (see text).

LangFlow arm	accept %	tail@acc	$\Delta$ tail@acc	$[2,3)$	$z{<}2$
Human	90.5	11.17	—	2.41	0.74
Stock	60.5	8.28	—	1.11	0.35
Quota $(1,1,2)$	35.7	13.24	$+4.96$	—	—
Quota $(1,2,3)$	37.0	16.69	$+8.41$	3.32	2.25
$+$ attestation gate	38.0	15.05	$+6.77$	4.10	0.87
Quota $(2,3,4)$	39.0	24.16	$+15.88$	—	—

Appendix

Table 19: LangFlow dose response of the band quota, with quotas given as commitments per round in each band. Every setting we tried overshoots. The realized commitment count is limited by supply rather than by the quota: the fill rate, the share of the quota’s commitment slots that are actually filled, is

54

–

55\%

across these arms and is set by

q_{\min}

, not by the quota, so narrowing the band while lowering

q_{\min}

raises commitments from

2{,}429

3{,}345

Value	Rule	Used in
Tail $11.88\%$	all-token, unfiltered, foreign discount on the numerator	Table 2
Tail $11.49\%$	alphabetic, unfiltered, no discount	Table 1 (three rare bands), Figure 1 b, Tables 5 , 6 , 10 , 11
Tail $11.17\%$ ( $11.2\%$ )	alphabetic, accepted set under the dual bar (accept $90.5\%$ )	every tail@acc reading, e.g. Tables 8 , 12 , 13 , 16 , 19
Tail $12.35\%$	top-5k tail, accepted set	Table 6 (right)
Non-words $1.14\%$ / $0.66\%$	all-token / alphabetic, unfiltered	Table 2 / Tables 1 , 5 , 17
$\zeta{=}0.971$ ; slope $-0.986$	all-token; regular-expression tokens	Table 2 ; § 4

Appendix

Table 20: Human reference values and the rule behind each; the rules are defined in Appendix A.2 .

Model	Row	seed	tail %	$\times$ human	non-word %	JSD	$\|\Delta\zeta\|$	types
ELF-B	Graft	31337	9.78	0.82	2.61	.0049	.138	13,757
		55555	9.91	0.83	2.55	.0047	.111	14,037
		77777	9.58	0.81	2.54	.0052	.141	13,693
ELF-M	Graft	31337	12.18	1.03	3.13	.0082	.061	15,832
		55555	12.14	1.02	2.96	.0083	.063	15,853
		77777	11.75	0.99	3.22	.0072	.075	15,400

Appendix

Table 21: Per-seed values behind the three-seed rows of Table 2 , scored as in that table (

n{=}200

per seed). Their means agree with Table 2 to within one unit in the last digit shown.

Setting	ELF-B / M / L	LangFlow
Checkpoint, vocabulary	official OpenWebText (105M / 342M / 652M), T5 ( $32$ k)	official OpenWebText (652M), GPT-2 BPE
Sampler	SDE, $32$ steps, churn $\gamma{=}1.5$ , self-conditioning guidance $3$ , logit-normal schedule a	deterministic Euler, $256$ steps, linear time grid
Length; samples, batch	$1{,}024$ ; $200$ , batch $16$ (ELF-B tuned: $8$ )	$1{,}024$ ; $200$ , batch $8$ (stock, seed $42$ : $16$ )
Seeds; read time	$31337$ , $55555$ , $77777$ (best-of-4 adds $99999$ ); $t_{\mathrm{read}}{=}0.20$	$42$ , $1042$ , $2042$ (best-of-4 adds $3042$ ); $0.72$
Selection	$R{=}4$ , $m{=}8$ ; candidates among the $512$ most likely words, rare words $0<z<4$ ; $q_{\min}{=}0.08$ ; spacing $s{=}4$ ; atomic commitment, splicing and co-pinning on
Attestation gate	on; attestation lexicon of documents $1{,}000$ – $30{,}999$ of the reference shard, minimum count $1$

Appendix

Table 22: Settings of the runs in Table 2 , as read from the arguments stored with each run, for the procedure of Algorithm 1 . A cell that spans both columns applies to both model families. The ELF-B tuned entries describe the arm of Appendix E.4 , and the LangFlow best-of-4 seed an arm that § 7.4 quotes; neither is a row of Table 2 .

Diffusion Language Model Decoding

Diffusion language model decoding generates text through iterative parallel refinement, balancing fewer denoising steps with accuracy, coherence, and control.

See all 250 papers

May 11, 2026

ELF: Embedded Language Flows

Diffusion and flow-based models have become the de facto approaches for generating continuous data, e.g., in domains such as images and videos. Their success has attracted growing interest in applying them to language modeling. Unlike their image-domain counterparts, today's leading diffusion language models (DLMs) primarily operate over discrete tokens. In this paper, we show that continuous DLMs can be made effective with minimal adaptation to the discrete domain. We propose Embedded Language Flows (ELF), a class of diffusion models in continuous embedding space based on continuous-time Flow Matching. Unlike existing DLMs, ELF predominantly stays within the continuous embedding space until the final time step, where it maps to discrete tokens using a shared-weight network. This formulation makes it straightforward to adapt established techniques from image-domain diffusion models, e.g., classifier-free guidance (CFG). Experiments show that ELF substantially outperforms leading discrete and continuous DLMs, achieving better generation quality with fewer sampling steps. These results suggest that ELF offers a promising path toward effective continuous DLMs.

Keya Hu, Linlu Qiu, Yiyang Lu +5

MIT

Flow-Based Generative Modeling Diffusion Language Models Continuous Diffusion Language Models

Ask PDF

Apr 13, 2026

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

Continuous diffusion has been the foundation of high-fidelity, controllable, and few-step generation of many data modalities such as images. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts due to the sparse data space and the underexplored design space. In this work, we close this gap with LangFlow, the first continuous DLM to rival discrete diffusion, by connecting embedding-space DLMs to Flow Matching via Bregman divergence, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, as we find it improves both likelihood and sample quality of embedding-space DLMs with effects substantially different from discrete diffusion. Putting everything together, LangFlow rivals top discrete DLMs on both the perplexity (PPL) and the generative perplexity (Gen. PPL), reaching a PPL of 30.0 on LM1B and 24.6 on OpenWebText. It even exceeds autoregressive baselines in zero-shot transfer on 4 out of 7 benchmarks. LangFlow provides the first clear evidence that continuous diffusion is a promising paradigm for language modeling. Homepage: https://github.com/nealchen2003/LangFlow

Yuxin Chen, Chumeng Liang, Hangke Sui +4

UIUC

Self-Conditioning Diffusion Language Models Continuous Diffusion Language Models Diffusion Language Model Training Flow Matching

Ask PDF

Oct 1, 2026

Clock Diffusion: Efficient Semi-Autoregressive Continuous Diffusion Language Models

Recent works on continuous diffusion for discrete data have demonstrated performance on par with comparable discrete diffusion models. However, these continuous counterparts lack key features that are essential to practical use as language models, namely variable-length generation and support for a key-value cache, and they still lag behind the frontier of autoregressive and discrete diffusion quality. In this work, we address these limitations. We do so by introducing a model parameterization that uses position-dependent noise schedules to define semi-autoregressive (SAR) continuous diffusion language models (DLMs). Together with efficient training and sampling algorithms, we call this framework Clock Diffusion, and we present two special cases of our method: block and sliding window generation. We then define ClockDLMs, a family of Gaussian DLMs based on sliding window Clock Diffusion that attain state-of-the-art diffusion likelihood bounds on OpenWebText, even beating the performant block SAR discrete diffusion models. ClockDLMs trained on TinyGSM also substantially outperform continuous baselines on the GSM8K benchmark and match and exceed comparable SAR discrete diffusion models. Finally, building on our parameterization, we propose more efficient samplers that we dub Cache Grab, which adapt techniques from accelerated inference in discrete diffusion, such as committing tokens whose probabilities exceed a confidence threshold and self-speculative decoding, further improving our models' quality and efficiency.

Yair Schiff, Omer Belhasin, Roy Uziel +7

NVIDIA · Cornell University

Diffusion Language Model Inference Language Modeling Diffusion Language Models Continuous Diffusion Language Models Diffusion Model Inference Acceleration Diffusion Language Model Training

Ask PDF

Abstract

Diffusion Language Model Decoding

Explore similar work

ELF: Embedded Language Flows

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

Clock Diffusion: Efficient Semi-Autoregressive Continuous Diffusion Language Models