PIG: Parallel Inconsistency Guidance for Accelerating Diffusion Language Models

Quentin Fruytier; Anton Xue; Sanjay Shakkottai; Sujay Sanghavi

cs.CLOct 12, 2026

PIG: Parallel Inconsistency Guidance for Accelerating Diffusion Language Models

Authors: Quentin Fruytier, Anton Xue, Sanjay Shakkottai, Sujay Sanghavi

Organizations: The University of Texas at Austin

Abstract

Diffusion language models accelerate generation by decoding multiple masked positions per forward pass. However, accuracy is known to drop when parallel decoding trajectories deviate from their sequential counterparts, which are slow and expensive "oracles" that adhere to the joint distribution of language. In this paper, we propose Parallel Inconsistency Guidance (PIG), a new paradigm that uses a trained auxiliary model to directly estimate the inconsistency between parallel and sequential generation from a single model forward pass. During inference, masked positions are traversed by decreasing confidence, and tokens are added to the parallel decode set one-at-a-time until the total estimated inconsistency between the parallel and sequential generation exceeds a threshold. Surprisingly, it suffices to train a lightweight ~2M parameter auxiliary model on a small mixture drawn from OpenCoder, GSM8K-train, MATH-train, and MBPP-train. For LLaDA-8B Instruct and NemotronDiffusion-3B Instruct, PIG extends the accuracy-TPF frontier on GSM8K, MATH500, MBPP, HumanEval, and MMLU, when compared to the industry standard Fast-dLLM and measures favorably on LLaDA-8B Instruct against DAPD, KLASS, EB-sampler, and other learned unmasking policies.

Figures & tables

Panel of Figure 1 — Figure 1: Parallel Inconsistency Guidance ( pig ). Left: from one forward pass, pig estimates how likely each parallel commit is to disagree with the unseen sequential decode and defers the risky ones (M3). Right: on Nemotron-3B HumanEval (64-token blocks), pig.int and pig.ext extend the accuracy–TPF frontier of the model’s own confidence decoder and baseline reported result.

Panel of Figure 2 — Figure 2: pig.int controls single-round regret. One forward pass ( nfe ) predicts every masked slot and ranks them by confidence; that ranking is the traversal order (I). Candidates are iteratively added to $S$ , each with its predicted incremental regret $b_{i}=\hat{p}_{\nu}(\nu_{i},\hat{\mu}_{i}(S))$ (II), and admitted only while the predicted regret $\hat{R}(S)$ fits the budget $\varphi$ (III); the first candidate that does not fit closes the round. We then greedily decode all tokens in $S$ in parallel, before starting the next round.

Panel of Figure 3 — Figure 3: Cross-round regret measures the value of waiting. On the canvas and confidence order of Figure 2 (I), pig.ext admits each candidate and then prices the next candidate $N(S)$ : $c_{i}$ is how much the mismatch risk of $i$ would drop after first committing $S$ (II). The round continues while $\hat{R}(N(S))-\hat{R}(N(S)\mid S)\leq\theta$ (III): committing “are” changes little for M3, so the walk widens; committing “blue” would turn M2 from “look” into “are”, so waiting is worth more than $\theta$ , the round closes and commits only “are” and “blue”. We implement with $\lvert N(S)\rvert=16$ look-ahead positions.

Panel of Figure 4 — Figure 4: Finite Differences Estimator (FDE). The frozen DLM’s pass feeds a learned adapter ( ${\approx}2.2$ M parameters), which returns in the same pass the full directed matrix $\hat{\mathbf{D}}$ of entropy finite differences of Equation 4 , $\hat{\mathbf{D}}_{ij}\approx H_{i}(\mathbf{x}_{t})-H_{i}(\mathbf{x}_{t}\oplus\hat{x}_{j})$ , the change in uncertainty at $i$ if $j$ were committed. Full implementation and training details in ( Appendix B ).

Figure 5: pig improves the accuracy-TPF frontier of parallel decoding On LLaDA-8B-Instruct (top) and Nemotron-Labs-Diffusion-3B-Instruct (bottom), both pig.int (red) and pig.ext (blue) outperform a strong confidence-based baseline, Fast-dLLM (gray). We report Wilson

95\%

error bars, and full numerical details in Appendices E and E.2 .

	our protocol (paired)			their protocol ( Kim et al., 2026a )
benchmark	Fast-dLLM	DAPD-Direct	DAPD-Staged	EB-Sampler	KLASS
PIG.INT	relative TPF increase ( $+$ or $-$ accuracy)
GSM8K	$1.15\times$ ( $+0.0$ )	$1.77\times$ ( $+2.0$ )	$2.62\times$ ( $+0.6$ )	$2.51\times$ ( $+0.5$ )	$2.71\times$ ( $+1.0$ )
MATH-500	$1.13\times$ ( $-0.2$ )	$1.65\times$ ( $+1.0$ )	$1.82\times$ ( $+0.6$ )	$3.38\times$ ( $+1.2$ )	$4.38\times$ ( $+1.4$ )
MBPP	$1.66\times$ ( $+0.0$ )	$0.83\times$ ( $+1.0$ )	$1.85\times$ ( $+0.4$ )	$2.60\times$ ( $+3.0$ )	$2.05\times$ ( $+4.0$ )
HumanEval	$1.29\times$ ( $-1.8$ )	$1.46\times$ ( $+2.4$ )	$4.28\times$ ( $+3.1$ )	$2.88\times$ ( $+0.0$ )	$3.90\times$ ( $-0.6$ )

Table 1: Matched Accuracy TPF Increase. Each cell: our TPF over that baseline’s best operating point, and in parentheses our accuracy minus theirs at the measured cell of ours that delivers it. Every cell is from Figure 16 . Fast-dLLM and both DAPD variants are our own runs; EB-Sampler and KLASS are the cells published by Kim et al. (2026a) . LLaDA-8B-Instruct, BL

=

64 fixed.

benchmark	Fast-dLLM	PIG.INT	PIG.EXT	Fast-dLLM	PIG.INT	PIG.EXT
LLaDA-8B-Instruct	interpolated accuracy @ TPF $=10$			interpolated accuracy @ TPF $=15$
GSM8K	$51.4$	$70.5$ ( $+19.1$ )	$56.0$ ( $+4.6$ )	$25.6$	$34.0$ ( $+8.5$ )	$39.7$ ( $+14.2$ )
MATH-500	$14.1$	$25.5$ ( $+11.4$ )	$20.0$ ( $+5.9$ )	$9.8$ @12.2	$15.4$ @13.4 (—)	$13.7$ (—)
MBPP	$21.0$	$29.3$ ( $+8.3$ )	$22.4$ ( $+1.4$ )	$7.8$	$13.2$ @15.0 ( $+5.3$ )	$17.5$ ( $+9.7$ )
HumanEval	$30.4$	$34.3$ ( $+3.8$ )	$28.7$ ( $-1.7$ )	$9.2$	$20.2$ ( $+11.0$ )	$19.7$ ( $+10.5$ )
Nemotron-3B-Instruct	interpolated accuracy @ TPF $=3$			interpolated accuracy @ TPF $=5$

Table 2: Accuracy at Matched TPF. Reported accuracies and gains relative to Fast-dLLM in parentheses, when above ( teal ) and below ( red ). Every lane is read at exactly the target throughput, linearly between the two measured cells that bracket it. A lane whose sweep ends before the target is not extrapolated: its last measured cell is given instead, as accuracy@TPF. All rows at BL

=

64 fixed.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Estimator quality on held-out test states. Pearson correlation between the exact quantity and its estimate, per benchmark, for the three quantities the decoders consume: the per-edge finite difference

\hat{\mathbf{D}}_{ij}

against

\mathbf{D}_{ij}

(left), the aggregate coupling

\hat{\mu}_{i}(S)

along the confidence walk (middle), and the local curvature

|H(S,N(S);\mathbf{x})|

over the next

32

candidates (right; median over set sizes of the per-size correlation). Green: the LLaDA-8B-Instruct estimator; orange: the attention-input U-Net it replaced (ablation); blue: the Nemotron-Diffusion-3B twin on block-

64

test states.

Panel of Figure 7 — Figure 7: The same evaluation in detail. Top: exact (x) versus estimate (y) on test states, one colour per benchmark, LLaDA-8B-Instruct (upper row) and Nemotron-Diffusion-3B (lower row), with the per-benchmark correlation printed in each panel; the curvature correlation here pools set sizes $|S|\in\{2,\dots,24\}$ and is therefore higher than the within-size value of Figure 6 . Bottom: on LLaDA test states, the exact mass $\sum|\mathbf{D}_{ij}|$ in each magnitude bin (grey) and the sign-consistent mass the estimator recovers (green; $49$ – $59\%$ of the total, the U-Net’s $26$ – $34\%$ in orange): the estimator under-shoots in magnitude but does so uniformly across bins.

Figure 8: The calibrated mismatch table $\hat{p}_{\nu}$ read by pig.int , one per backbone (left LLaDA-8B-Instruct, right Nemotron-Labs-Diffusion-3B). Rows: the committing position’s confidence $\nu_{i}$ at the fixed edges of Algorithm 5 ; columns: its aggregate coupling $\hat{\mu}_{i}$ to the higher-confidence members of the set, at the quantile edges of each model’s own fit. A cell is the empirical rate at which a parallel commit differed from its sequential replay, pooled over the GSM8K-train and OpenCoder probes; a cell holding fewer than $50$ commits is left blank and is read by the decoder at the pooled rate of its row.

Figure 9: Corpus transfer of $\hat{p}_{\nu}$ . The table of Figure 8 fitted on each training corpus alone, GSM8K-train (left) and OpenCoder (middle), at the pooled bin edges, and their difference (right; green where code mismatches less than math at matched coupling, magenta where more). Top row LLaDA-8B-Instruct, bottom row Nemotron-Labs-Diffusion-3B. A cell is drawn only where both corpora hold at least $50$ commits.

Figure 10: The entropy-indexed mismatch table $\hat{p}_{H}$ read by pig.ext , one per backbone (left LLaDA-8B-Instruct, right Nemotron-Labs-Diffusion-3B). As Figure 8 , with the committing position’s entropy $H_{i}$ in nats as the row feature, at the fixed edges given in the text; the empty top row ( $H_{i}\geq 6$ ) is not drawn. Fitted on the labelled commits of the same probe procedure, pooled over GSM8K-train and OpenCoder (Nemotron: the same $601{,}730$ commits as Figure 8 ; LLaDA: a separate probe of $395{,}854$ ); a cell holding fewer than $50$ commits is left blank.

Figure 11: Dynamic block decoding unlocks additional parallelism. Accuracy versus TPF on GSM8K and MATH-500 under fixed and dynamic

32

-candidate decoding. Filled markers and solid lines denote fixed blocks, while open markers and dashed lines denote dynamic blocks; every decoding rule is evaluated under both candidate geometries.

dataset	lane	$\mathrm{TPF}^{\star}$		acc @ $T_{1}$		acc @ $T_{2}$
		fixed	dynamic	fixed	dynamic	fixed	dynamic
GSM8K ( $a_{0}{=}0.786$ ; $T_{1}{=}8$ , $T_{2}{=}10$ )	Fast-dLLM	$5.80$	$\mathbf{6.58}$	$0.592$	$\mathbf{0.688}$	$0.343$	$\mathbf{0.535}$
	PIG.INT	$6.97$	$\mathbf{7.56}$	$0.723$	$\mathbf{0.745}$	$0.544^{\lx@sectionsign}$	$\mathbf{0.710}$
	PIG.EXT	$4.75$	$\mathbf{5.44}$	$0.563$	$\mathbf{0.604}$	$0.458$	$\mathbf{0.488}$
MATH-500 ( $a_{0}{=}0.364$ ; $T_{1}{=}7$ , $T_{2}{=}9$ )	Fast-dLLM	$4.23$	$\mathbf{4.75}$	$0.163$	$\mathbf{0.216}$	$0.101$	$\mathbf{0.122}$
	PIG.INT	$5.99$	$\mathbf{6.20}$	$0.281$	$\mathbf{0.311}$	$0.120^{\lx@sectionsign}$	$\mathbf{0.279}$

Appendix

Table 3: Fixed versus dynamic

32

-token blocks on LLaDA-8B-Instruct , the sweeps of Figure 11 .

\mathrm{TPF}^{\star}

: largest throughput at which the lane stays within one Wilson 95% half-width of the slowest fixed-block Fast-dLLM point’s accuracy (

a_{0}

; the matched-accuracy rule of Table 1 ). acc@

T

: accuracy linearly interpolated at

T

TPF; § the sweep ends short of

T

and is extended along its last segment. Best geometry per lane and column in bold.

Figure 12: PIG adapts its decoding width to information curvature, all four benchmarks. Diagnostic on LLaDA-8B-Instruct across GSM8K, MATH-500, MBPP, and HumanEval. The top row compares exact and estimated changes in the uncertainty of upcoming candidates, while the remaining rows show parallel widths induced by fixed regret budgets and live PIG decoding.

Figure 13: Domain transfer: Nemotron-3B-Instruct on MMLU , zero-shot,

500

questions stratified over all

57

subjects, at fixed blocks of

16

32

and

64

tokens (canvas

2{,}048

). Grey: Fast-dLLM (on Nemotron, the model’s own confidence-threshold decoder, the same rule); red: pig.int ; blue: pig.ext , whose two slowest operating points additionally floor admission at a confidence

\nu

. The estimator and tables are the ones fitted on math and code ( Appendix B ), used unchanged. Dashed line: the accuracy the model’s release reports at its default setting (BL

=

\nu=0.9

) ( Fu et al., 2026 ) . Error bars are Wilson

95\%

intervals.

Figure 14: The regret calibration on the decoder’s own trajectories: pig.int on LLaDA-8B-Instruct at the geometry of Figure 5 (GSM8K and MATH-500 at BL

=

32 fixed, MBPP and HumanEval at BL

=

64 fixed), every third round audited by replaying the committed set one token at a time from the round’s own canvas in walk order (Definition A.2 ). (a) Realized disagreement rate per table cell against the cell’s value, pooled over the

\varphi

ladder (Wilson

95\%

). (b) Per

\varphi

: realized regret per audited round

\mathbb{E}[R(S)]

(filled) and predicted

\mathbb{E}[\hat{R}(S)]

(hollow) against the budget line

y=\varphi

. (c) Realized (solid) and predicted (dashed) mismatch rate per decile of the decode, token-weighted.

Figure 15: The cross-round regret calibration on the decoder’s own trajectories: pig.ext on LLaDA-8B-Instruct at the geometry of Figure 5 (composite rule,

32

problems per cell, every sixth round audited by replaying

S

and then

N(S)

sequentially from the round’s own canvas, Definition A.2 ). (a) Realized deferred disagreement of the window tokens against the deferred estimate

\hat{p}_{H}(\hat{H}_{i}(\mathbf{x}\mid S),\hat{\mu}_{i})

(Wilson

95\%

); crosses: the current-round estimate of the same tokens. (b) Per

\theta

: realized cross-round regret

C_{\mathbf{x}}(S)

per audited round (filled) against the rule’s

\hat{C}_{\mathbf{x}}(S)

(hollow) and the budget line

y=\theta

. (c) The same per decile of the decode.

Figure 16: Our BL=64 frontier against every BL=64 operating point DAPD reports on LLaDA-8B-Instruct (math above, code below; flexible extraction for GSM8K/MATH-500, pass@1 for code with Wilson 95% bars. Lines: our Fast-dLLM, pig.int and pig.ext sweeps at BL=64 fixed. Hollow markers, all at BL=64 (four

64

-token blocks): Fast-dLLM ( Wu et al., 2025 ) , EB-Sampler ( Ben-Hamu et al., 2025 ) and KLASS ( Kim et al., 2025 ) as reported in Table 3 of Kim et al. (2026a) ; DAPD-Staged (star) and DAPD-Direct (pentagon) from the project README ( Kim et al., 2026b ) , absent from the paper. Filled black star and pentagon: DAPD-Staged and DAPD-Direct run by us under our protocol.

Figure 17: Our BL=32 frontier against the published BL=32 series of Jazbec et al. (2025b) on LLaDA-8B-Instruct (GSM8K flexible extraction, MATH-500 with Wilson 95% bars. Lines: our Fast-dLLM, pig.int and pig.ext sweeps at BL=32 fixed. Black diamonds: their RL policy, one per reward weight

\alpha

\pm

their three-seed spread; dotted grey: Fast-dLLM as run by them; both at BL=32 fixed, canvas

256

0

-shot. Same block geometry, canvas and NFE accounting.

Figure 18: The full LLaDA-8B-Instruct frontier (canvas

256

, greedy, no KV cache; flexible extraction for GSM8K/MATH-500, pass@1 for MBPP/HumanEval; bars: Wilson 95%). Rows: decode regime; columns: benchmark (shots and

n

in the title). Grey: Fast-dLLM confidence threshold, swept

0.95\to 0.1

; red: pig.int (

\varphi

); blue: pig.ext , with its two slowest operating points additionally flooring admission at a confidence

\nu

equal to

0.9

and

0.8

respectively.

Figure 19: Figure 18 under strict extraction (GSM8K #### , MATH-500 last \boxed{} ); MBPP and HumanEval panels are identical to the main text.

Figure 20: The full Nemotron-Diffusion-3B frontier (

0

-shot, canvas

2048

, greedy; flexible extraction for GSM8K, pass@1 for code; bars: Wilson 95%). Rows: the model’s own fixed

32

-token block, a sliding

32

-token window, and a

32

-token commit band inside a

64

-wide window; columns: benchmark. Grey: Fast-dLLM, the model’s own confidence-threshold decoder swept

0.9\to 0.3

under the same geometry; red: pig.int ; blue: pig.ext , the composite operating rule of every figure. Cells beyond

8.5

TPF are omitted; TPF is not comparable across the two models’ canvases.

Figure 21: The Nemotron-Diffusion-3B frontier at BL=16 (

0

-shot, canvas

2048

, greedy; flexible extraction for GSM8K, pass@1 for code; bars: Wilson 95%). Rows: fixed

16

-token block, sliding

16

-token window, and a

16

-token commit band inside a

32

-wide window; columns: benchmark. Grey: the model’s confidence-threshold decoder under the same geometry, swept

0.9\to 0.3

; red: pig.int ; blue: pig.ext , the composite operating rule of Figure 5 . Cells beyond

8.5

TPF are omitted.

Figure 22: The Nemotron-Diffusion-3B frontier at BL=64 , as Figure 21 with a fixed

64

-token block, a sliding

64

-token window, and a

64

-token commit band inside a

128

-wide window; MATH-500 was not run at this block size.

regime	Fast-dLLM				PIG.INT				PIG.EXT
	$\nu$	TPF	acc (flex)	acc (strict)	$\varphi$	TPF	acc (flex)	acc (strict)	$\theta$	TPF	acc (flex)	acc (strict)
BL=32 fixed	0.95	2.83	$0.786^{+0.034}_{-0.038}$	$0.488^{+0.044}_{-0.044}$	0.015	3.73	$0.796^{+0.033}_{-0.038}$	$0.500^{+0.044}_{-0.044}$	0.125 ( $\nu{\geq}0.9$ )	3.33	$0.796^{+0.033}_{-0.038}$	$0.500^{+0.044}_{-0.044}$
	0.85	3.85	$0.794^{+0.033}_{-0.038}$	$0.510^{+0.044}_{-0.044}$	0.03	4.36	$0.788^{+0.034}_{-0.038}$	$0.528^{+0.043}_{-0.044}$	0.25 ( $\nu{\geq}0.8$ )	4.13	$0.786^{+0.034}_{-0.038}$	$0.514^{+0.044}_{-0.044}$
	0.7	5.36	$0.772^{+0.035}_{-0.039}$	$0.492^{+0.044}_{-0.044}$	0.05	4.81	$0.798^{+0.033}_{-0.037}$	$0.532^{+0.043}_{-0.044}$	0.5	5.48	$0.708^{+0.038}_{-0.041}$	$0.416^{+0.044}_{-0.042}$
	0.5	7.50	$0.666^{+0.040}_{-0.042}$	$0.386^{+0.043}_{-0.042}$	0.1	5.44	$0.778^{+0.034}_{-0.038}$	$0.514^{+0.044}_{-0.044}$	1	6.82	$0.632^{+0.041}_{-0.043}$	$0.300^{+0.042}_{-0.039}$
	0.3	9.49	$0.372^{+0.043}_{-0.041}$	$0.184^{+0.036}_{-0.032}$	0.2	6.59	$0.764^{+0.035}_{-0.039}$	$0.490^{+0.044}_{-0.044}$	1.5	8.49	$0.534^{+0.043}_{-0.044}$	$0.260^{+0.040}_{-0.037}$

Appendix

Table 4: Frontier table, GSM8K, 5-shot. LLaDA-8B-Instruct, canvas 256, greedy, no KV cache; problems

[0,500)

(HumanEval all 164). Accuracy with its Wilson 95% interval; flexible and strict extraction. Every operating point behind Figure 18 and Figure 19 , including cells beyond the figures’ crops; PIG.EXT is the composite operating rule of every figure: its two slowest rows additionally floor admission at a confidence

\nu

(written

\theta

(

\nu{\geq}c

)), then the plain

\theta

ladder from

0.5

regime	Fast-dLLM				PIG.INT				PIG.EXT
	$\nu$	TPF	acc (flex)	acc (strict)	$\varphi$	TPF	acc (flex)	acc (strict)	$\theta$	TPF	acc (flex)	acc (strict)
BL=32 fixed	0.95	2.21	$0.364^{+0.043}_{-0.041}$	$0.356^{+0.043}_{-0.041}$	0.015	2.83	$0.360^{+0.043}_{-0.041}$	$0.354^{+0.043}_{-0.041}$	0.125 ( $\nu{\geq}0.9$ )	2.54	$0.362^{+0.043}_{-0.041}$	$0.352^{+0.043}_{-0.041}$
	0.85	2.88	$0.364^{+0.043}_{-0.041}$	$0.354^{+0.043}_{-0.041}$	0.03	3.23	$0.356^{+0.043}_{-0.041}$	$0.348^{+0.043}_{-0.040}$	0.25 ( $\nu{\geq}0.8$ )	3.09	$0.360^{+0.043}_{-0.041}$	$0.356^{+0.043}_{-0.041}$
	0.7	3.87	$0.332^{+0.042}_{-0.040}$	$0.330^{+0.042}_{-0.040}$	0.05	3.52	$0.342^{+0.043}_{-0.040}$	$0.334^{+0.042}_{-0.040}$	0.5	4.57	$0.334^{+0.042}_{-0.040}$	$0.322^{+0.042}_{-0.039}$
	0.5	5.39	$0.290^{+0.041}_{-0.038}$	$0.268^{+0.040}_{-0.037}$	0.1	4.00	$0.322^{+0.042}_{-0.039}$	$0.318^{+0.042}_{-0.039}$	1	6.23	$0.260^{+0.040}_{-0.037}$	$0.250^{+0.040}_{-0.036}$
	0.3	7.14	$0.152^{+0.034}_{-0.029}$	$0.112^{+0.031}_{-0.025}$	0.2	4.84	$0.344^{+0.043}_{-0.040}$	$0.342^{+0.043}_{-0.040}$	1.5	8.10	$0.184^{+0.036}_{-0.032}$	$0.172^{+0.036}_{-0.031}$

Appendix

Table 5: Frontier table, MATH-500, 4-shot. LLaDA-8B-Instruct, canvas 256, greedy, no KV cache; problems

[0,500)

\nu

(written

\theta

(

\nu{\geq}c

)), then the plain

\theta

ladder from

0.5

regime	Fast-dLLM			PIG.INT			PIG.EXT
	$\nu$	TPF	acc	$\varphi$	TPF	acc	$\theta$	TPF	acc
BL=32 fixed	0.95	3.29	$0.396^{+0.044}_{-0.042}$	0.015	4.21	$0.394^{+0.043}_{-0.042}$	0.125 ( $\nu{\geq}0.9$ )	3.78	$0.392^{+0.043}_{-0.042}$
	0.85	4.26	$0.388^{+0.043}_{-0.042}$	0.03	4.73	$0.394^{+0.043}_{-0.042}$	0.25 ( $\nu{\geq}0.8$ )	4.50	$0.398^{+0.044}_{-0.042}$
	0.7	5.53	$0.372^{+0.043}_{-0.041}$	0.05	5.16	$0.386^{+0.043}_{-0.042}$	0.5	7.15	$0.260^{+0.040}_{-0.037}$
	0.5	7.99	$0.230^{+0.039}_{-0.035}$	0.1	5.73	$0.366^{+0.043}_{-0.041}$	1	9.76	$0.214^{+0.038}_{-0.034}$
	0.3	12.10	$0.068^{+0.026}_{-0.019}$	0.2	6.87	$0.332^{+0.042}_{-0.040}$	1.5	13.50	$0.126^{+0.032}_{-0.026}$

Appendix

Table 6: Frontier table, MBPP, 3-shot. LLaDA-8B-Instruct, canvas 256, greedy, no KV cache; problems

[0,500)

(HumanEval all 164). Accuracy with its Wilson 95% interval; pass@1. Every operating point behind Figure 18 and Figure 19 , including cells beyond the figures’ crops; PIG.EXT is the composite operating rule of every figure: its two slowest rows additionally floor admission at a confidence

\nu

(written

\theta

(

\nu{\geq}c

)), then the plain

\theta

ladder from

0.5

regime	Fast-dLLM			PIG.INT			PIG.EXT
	$\nu$	TPF	acc	$\varphi$	TPF	acc	$\theta$	TPF	acc
BL=32 fixed	0.95	4.95	$0.396^{+0.076}_{-0.072}$	0.015	6.42	$0.378^{+0.076}_{-0.071}$	0.125 ( $\nu{\geq}0.9$ )	5.69	$0.390^{+0.076}_{-0.071}$
	0.85	6.76	$0.396^{+0.076}_{-0.072}$	0.03	7.37	$0.366^{+0.076}_{-0.070}$	0.25 ( $\nu{\geq}0.8$ )	6.77	$0.396^{+0.076}_{-0.072}$
	0.7	8.69	$0.342^{+0.075}_{-0.068}$	0.05	7.79	$0.372^{+0.076}_{-0.070}$	0.5	8.50	$0.274^{+0.073}_{-0.063}$
	0.5	10.64	$0.171^{+0.065}_{-0.050}$	0.1	8.67	$0.335^{+0.075}_{-0.068}$	1	10.46	$0.226^{+0.070}_{-0.057}$
	0.3	12.92	$0.018^{+0.034}_{-0.012}$	0.2	9.59	$0.317^{+0.075}_{-0.066}$	1.5	13.03	$0.207^{+0.068}_{-0.055}$

Appendix

Table 7: Frontier table, HumanEval, 0-shot. LLaDA-8B-Instruct, canvas 256, greedy, no KV cache; problems

[0,500)

\nu

(written

\theta

(

\nu{\geq}c

)), then the plain

\theta

ladder from

0.5

regime	Fast-dLLM			PIG.INT			PIG.EXT
	$\nu$	TPF	acc	$\varphi$	TPF	acc	$\theta$	TPF	acc
BL=32 fixed	0.9	2.56	$0.896^{+0.024}_{-0.030}$	0.03	2.52	$0.854^{+0.028}_{-0.034}$	0.125	2.05	$0.870^{+0.027}_{-0.032}$
	0.8	3.08	$0.870^{+0.027}_{-0.032}$	0.05	2.91	$0.868^{+0.027}_{-0.032}$	0.25	2.44	$0.856^{+0.028}_{-0.033}$
	0.7	3.66	$0.856^{+0.028}_{-0.033}$	0.1	3.31	$0.846^{+0.029}_{-0.034}$	0.5	3.66	$0.806^{+0.032}_{-0.037}$
	0.6	4.22	$0.808^{+0.032}_{-0.037}$	0.2	3.87	$0.844^{+0.029}_{-0.034}$	1	4.69	$0.742^{+0.036}_{-0.040}$
	0.5	4.91	$0.738^{+0.037}_{-0.040}$	0.3	4.21	$0.796^{+0.033}_{-0.038}$	1.5	5.70	$0.668^{+0.040}_{-0.042}$

Appendix

Table 8: Frontier table, Nemotron-3B, GSM8K, 0-shot (flex). Nemotron-Labs-Diffusion-3B, canvas 2048, 0-shot, greedy; the Fast-dLLM lane is the model’s own confidence-threshold decoder at the same geometry; PIG.EXT is the composite operating rule (the two slowest points additionally floor admission at a confidence

\nu

, written

\theta

(

\nu{\geq}c

)). Accuracy with its Wilson 95% interval; the full three-geometry grid is Figure 20 .

regime	Fast-dLLM			PIG.INT			PIG.EXT
	$\nu$	TPF	acc	$\varphi$	TPF	acc	$\theta$	TPF	acc
BL=32 fixed	0.9	2.80	$0.606^{+0.042}_{-0.043}$	0.03	2.72	$0.708^{+0.038}_{-0.041}$	0.125	2.24	$0.744^{+0.036}_{-0.040}$
	0.8	3.26	$0.598^{+0.042}_{-0.044}$	0.05	3.09	$0.686^{+0.039}_{-0.042}$	0.25	2.63	$0.708^{+0.038}_{-0.041}$
	0.7	3.78	$0.552^{+0.043}_{-0.044}$	0.1	3.45	$0.692^{+0.039}_{-0.042}$	0.5	3.96	$0.588^{+0.042}_{-0.044}$
	0.6	4.28	$0.498^{+0.044}_{-0.044}$	0.2	3.98	$0.614^{+0.042}_{-0.043}$	1	5.23	$0.492^{+0.044}_{-0.044}$
	0.5	4.95	$0.414^{+0.044}_{-0.042}$	0.3	4.37	$0.556^{+0.043}_{-0.044}$

Appendix

Table 9: Frontier table, Nemotron-3B, MATH-500, 0-shot. Nemotron-Labs-Diffusion-3B, canvas 2048, 0-shot, greedy; the Fast-dLLM lane is the model’s own confidence-threshold decoder at the same geometry; PIG.EXT is the composite operating rule (the two slowest points additionally floor admission at a confidence

\nu

, written

\theta

(

\nu{\geq}c

)). Accuracy with its Wilson 95% interval; the full three-geometry grid is Figure 20 .

regime	Fast-dLLM			PIG.INT			PIG.EXT
	$\nu$	TPF	acc	$\varphi$	TPF	acc	$\theta$	TPF	acc
BL=32 fixed	0.9	1.91	$0.553^{+0.043}_{-0.044}$	0.03	1.92	$0.548^{+0.043}_{-0.044}$	0.125	1.64	$0.574^{+0.043}_{-0.044}$
	0.8	2.26	$0.485^{+0.044}_{-0.044}$	0.05	2.18	$0.526^{+0.043}_{-0.044}$	0.25	1.89	$0.562^{+0.043}_{-0.044}$
	0.7	2.65	$0.433^{+0.044}_{-0.043}$	0.1	2.47	$0.504^{+0.044}_{-0.044}$	0.5	3.06	$0.412^{+0.044}_{-0.042}$
	0.6	3.11	$0.343^{+0.043}_{-0.040}$	0.2	2.87	$0.448^{+0.044}_{-0.043}$	1	5.20	$0.286^{+0.041}_{-0.038}$
	0.5	3.62	$0.222^{+0.039}_{-0.034}$	0.3	3.16	$0.408^{+0.044}_{-0.042}$

Appendix

Table 10: Frontier table, Nemotron-3B, MBPP, 0-shot. Nemotron-Labs-Diffusion-3B, canvas 2048, 0-shot, greedy; the Fast-dLLM lane is the model’s own confidence-threshold decoder at the same geometry; PIG.EXT is the composite operating rule (the two slowest points additionally floor admission at a confidence

\nu

, written

\theta

(

\nu{\geq}c

)). Accuracy with its Wilson 95% interval; the full three-geometry grid is Figure 20 .

regime	Fast-dLLM			PIG.INT			PIG.EXT
	$\nu$	TPF	acc	$\varphi$	TPF	acc	$\theta$	TPF	acc
BL=32 fixed	0.9	2.02	$0.750^{+0.060}_{-0.071}$	0.03	2.03	$0.756^{+0.059}_{-0.071}$	0.125	1.69	$0.793^{+0.055}_{-0.068}$
	0.8	2.41	$0.707^{+0.064}_{-0.074}$	0.05	2.30	$0.726^{+0.063}_{-0.073}$	0.25	2.00	$0.744^{+0.061}_{-0.072}$
	0.7	2.84	$0.543^{+0.074}_{-0.076}$	0.1	2.62	$0.707^{+0.064}_{-0.074}$	0.5	3.09	$0.512^{+0.075}_{-0.076}$
	0.6	3.34	$0.433^{+0.077}_{-0.073}$	0.2	3.06	$0.555^{+0.074}_{-0.076}$	1	4.59	$0.396^{+0.076}_{-0.072}$
	0.5	3.83	$0.329^{+0.075}_{-0.067}$	0.3	3.36	$0.427^{+0.077}_{-0.073}$

Appendix

Table 11: Frontier table, Nemotron-3B, HumanEval, 0-shot. Nemotron-Labs-Diffusion-3B, canvas 2048, 0-shot, greedy; the Fast-dLLM lane is the model’s own confidence-threshold decoder at the same geometry; PIG.EXT is the composite operating rule (the two slowest points additionally floor admission at a confidence

\nu

, written

\theta

(

\nu{\geq}c

)). Accuracy with its Wilson 95% interval; the full three-geometry grid is Figure 20 .

Parallel Decoding for Diffusion Language Models

See all 37 papers

May 22, 2026

DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling

Diffusion language models intrinsically fail to capture correlations between decoded tokens, which leads to a harsh trade-off between sampling quality and throughput. To solve this issue, we propose DiLaDiff, a variant of masked diffusion language models with three components: (1) a continuous latent space with semantic capabilities, learned by an auto-encoder fine-tuned from an existing masked diffusion language model; (2) a latent diffusion model learning the prior over the encoder distribution; (3) a consistency model distilling the learned prior into a few-step latent generative model. We show that, even without distillation, our latent-guided diffusion model outperforms the masked diffusion baseline while significantly accelerating inference. Consistency distillation further lowers the computational overhead of continuous diffusion, such that the latent is generated in negligible time compared to discrete decoding.

Jean-Marie Lemercier, Tomas Geffner, Karsten Kreis +3

NVIDIA

Diffusion Language Models Diffusion Language Model Distillation Diffusion Language Model Inference Masked Diffusion Models Continuous Diffusion Language Models Diffusion Model Inference Acceleration+1

Ask PDF

May 10, 2026

Edit-Based Refinement for Parallel Masked Diffusion Language Models

Masked diffusion language models enable parallel token generation and offer improved decoding efficiency over autoregressive models. However, their performance degrades significantly when generating multiple tokens simultaneously, due to a mismatch between token-level training objectives and joint sequence consistency. In this paper, we propose ME-DLM, an edit-based refinement framework that augments diffusion generation with lightweight post-editing steps. After producing an initial complete response, the model refines it through minimal edit operations, including replacement, deletion, and insertion, conditioned on the full sequence. Training supervision is derived from edit distance, providing a deterministic signal under a fixed canonicalization scheme for learning minimal corrections. This approach encourages sequence-level consistency through globally conditioned edits while preserving the efficiency benefits of parallel diffusion decoding. Extensive experiments demonstrate that ME-DLM improves the quality and robustness of multi-token parallel generation. In particular, when built upon LLaDA, our method achieves consistent gains of 11.6 points on HumanEval and 33.6 points on GSM8K while using one-eighth of the total diffusion steps. Code is available at https://github.com/renhouxing/ME-DLM.

Houxing Ren, Mingjie Zhan, Zimu Lu +5

CUHK MMLab · SenseTime Research · Shenzhen Loop Area Institute +1

Masked Diffusion Models Parallel Decoding Diffusion Language Model Decoding Diffusion Language Models Diffusion Model Inference Acceleration Text Generation+1

Ask PDF

Apr 9, 2026

DMax: Aggressive Parallel Decoding for dLLMs

We present DMax, a new paradigm for efficient diffusion language models (dLLMs). It mitigates error accumulation in parallel decoding, enabling aggressive decoding parallelism while preserving generation quality. Unlike conventional masked dLLMs that decode through a binary mask-to-token transition, DMax reformulates decoding as a progressive self-refinement from mask embeddings to token embeddings. At the core of our approach is On-Policy Uniform Training, a novel training strategy that efficiently unifies masked and uniform dLLMs, equipping the model to recover clean tokens from both masked inputs and its own erroneous predictions. Building on this foundation, we further propose Soft Parallel Decoding. We represent each intermediate decoding state as an interpolation between the predicted token embedding and the mask embedding, enabling iterative self-revising in embedding space. Extensive experiments across a variety of benchmarks demonstrate the effectiveness of DMax. Compared with the original LLaDA-2.0-mini, our method improves TPF on GSM8K from 2.04 to 5.47 while preserving accuracy. On MBPP, it increases TPF from 2.71 to 5.86 while maintaining comparable performance. On two H200 GPUs, our model achieves an average of 1,338 TPS at batch size 1. Code is available at: https://github.com/czg1225/DMax

Zigeng Chen, Gongfan Fang, Xinyin Ma +2

National University of Singapore

Diffusion Language Model Training Parallel Decoding for Diffusion Language Models Diffusion Language Model Decoding Diffusion Language Models Diffusion Model Inference Acceleration Masked Diffusion Models

Ask PDF

Abstract

Parallel Decoding for Diffusion Language Models

Explore similar work

DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling

Edit-Based Refinement for Parallel Masked Diffusion Language Models

DMax: Aggressive Parallel Decoding for dLLMs