Survival-Guided Length Control for Efficient Diffusion Language Models

Ivan Kobyzev; Abbas Ghaddar; Yufei Cui

cs.CLAug 26, 2026

Survival-Guided Length Control for Efficient Diffusion Language Models

Authors: Ivan Kobyzev, Abbas Ghaddar, Yufei Cui

Organizations: Huawei Noah’s Ark Lab, Montreal Research Center, Canada

Abstract

Diffusion language models (DLMs) generate text by iteratively denoising masked sequences, but standard decoding either fixes the sequence length or relies on ad hoc stopping rules, often leading to unnecessary denoising steps. We recast length selection as a discrete-time survival problem over the end-of-sequence token and propose a plug-in, training-free length predictor that can be added to any existing DLM. Across reasoning and code-generation benchmarks, survival-guided length decoding speeds up inference by up to 7 times while preserving task accuracy. We further find that predicted lengths vary widely even within the same dataset, making model performance sensitive to the chosen length.

Figures & tables

Task	LLaDA		Dream
Task	w/o	w/	w/o	w/
BBH	73	14 ( $5.2\times$ )	56.3	10.5 ( $5.4\times$ )
GSM8K	91	16 ( $5.2\times$ )	81.4	12.3 ( $6.6\times$ )
MATH	76	23 ( $3.3\times$ )	64.5	16.1 ( $4.0\times$ )
HumEval	54	9 ( $6.0\times$ )	53.0	16.6 ( $3.2\times$ )
MBPP	80	26 ( $3.1\times$ )	64.8	16.6 ( $3.9\times$ )

Table 1: Decoding speed (seconds per sample) for baseline ( w/o ) vs. length prediction ( w/ ). Parentheses report speedup relative to the baseline.

Task	LLaDA		Dream
Task	w/o	w/	w/o	w/
BBH	$49.5\!\pm\!0.6$	$51.8\!\pm\!0.6$	$63.7\!\pm\!0.5$	$63.8\!\pm\!0.5$
GSM8K	$69.9\!\pm\!1.3$	$70.0\!\pm\!1.3$	$74.5\!\pm\!1.2$	$74.6\!\pm\!1.2$
MATH	$31.6\!\pm\!0.6$	$31.0\!\pm\!0.6$	$39.6\!\pm\!0.7$	$39.5\!\pm\!0.7$
HumEval	$32.3\!\pm\!3.7$	$33.5\!\pm\!3.7$	$58.0\!\pm\!3.7$	$59.0\!\pm\!3.7$
MBPP	$40.0\!\pm\!2.0$	$40.2\!\pm\!2.1$	$58.0\!\pm\!2.0$	$58.0\!\pm\!2.0$

Table 2: Task performance and standard deviation for baseline ( w/o ) vs. length prediction ( w/ ).

Task	LLaDA		Dream
Task	$\bar{L}$	$\Delta$	$\bar{L}$	$\Delta$
BBH	$537\!\pm\!204$	$-3.2\!\pm\!0.7$	$482\!\pm\!176$	$-2.7\!\pm\!0.6$
GSM8K	$278\!\pm\!29$	$-2.8\!\pm\!1.8$	$270\!\pm\!52$	$-2.6\!\pm\!1.8$
MATH	$498\!\pm\!64$	$-0.6\!\pm\!0.8$	$390\!\pm\!62$	$-0.7\!\pm\!0.9$
HumEval	$382\!\pm\!57$	$-1.8\!\pm\!5.2$	$396\!\pm\!107$	$-2.3\!\pm\!5.2$
MBPP	$495\!\pm\!44$	$0.0\!\pm\!3.0$	$378\!\pm\!95$	$-2.0\!\pm\!3.0$

Table 3: We report the per-task mean predicted length (

\bar{L}

) along with its standard deviation, as well as the performance change (

\Delta

) and its standard deviation when using

\bar{L}

as a fixed horizon, compared to our per-sample survival-guided length selection.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

\mathbf{x}^{(0)}=[\mathbf{x}_{1:P},\texttt{[MASK]},\dots,\texttt{[MASK]}]

Appendix

Algorithm 1 Survival-guided length prediction for diffusion LMs

LLaDA (Instruct)
Task	w/o	DAEDAL	Ours
GSM8K	89 (1.0 $\times$ )	28 (3.2 $\times$ )	14 (6.4 $\times$ )
MATH	75 (1.0 $\times$ )	41 (1.8 $\times$ )	20 (3.8 $\times$ )
HumanEval	53 (1.0 $\times$ )	22 (2.4 $\times$ )	8 (6.6 $\times$ )
MBPP	78 (1.0 $\times$ )	68 (1.1 $\times$ )	23 (3.4 $\times$ )
Dream (Instruct)

Appendix

Table 4: Decoding speed (seconds per sample) on instruction-tuned models for DAEDAL and our length prediction method. Parentheses report speedup relative to the baseline decoder without (w/o) length prediction.

LLaDA (Instruct)
Task	w/o	DAEDAL	Ours
GSM8K	78.6 $\pm$ 1.4	79.2 $\pm$ 1.3	79.5 $\pm$ 1.5
MATH	26.6 $\pm$ 0.7	26.9 $\pm$ 0.8	27.1 $\pm$ 0.7
HumanEval	47.6 $\pm$ 3.6	48.3 $\pm$ 3.5	48.2 $\pm$ 3.7
MBPP	34.2 $\pm$ 2.1	34.8 $\pm$ 2.2	35.5 $\pm$ 2.0
Dream (Instruct)

Appendix

Table 5: Task performance on instruction-tuned models of baseline without length prediction compared with both DAEDAL and our survival-guided length selection method.

Task	Step 2	Step 5	Step 7	Step 10
BBH	$0.5\pm 2.3$	$8.2\pm 7.6$	$7.2\pm 8.6$	$8.4\pm 9.5$
GSM8K	$3.8\pm 4.2$	$4.1\pm 5.6$	$4.2\pm 5.8$	$4.2\pm 6.3$
MATH	$-1.9\pm 6.3$	$7.0\pm 6.7$	$7.4\pm 7.8$	$8.0\pm 9.3$
HumanEval	$2.7\pm 3.1$	$4.5\pm 8.3$	$9.4\pm 7.9$	$6.9\pm 7.9$
MBPP	$-0.8\pm 1.2$	$4.0\pm 6.2$	$8.2\pm 8.1$	$8.5\pm 9.2$

Appendix

Table 6: Absolute change in predicted length

\delta\hat{L}^{(s)}=\hat{L}^{(s)}-\hat{L}^{(0)}

when the estimator is recomputed from later denoising steps for LLaDA.

Diffusion Language Model Decoding

Diffusion language model decoding generates text through iterative parallel refinement, balancing fewer denoising steps with accuracy, coherence, and control.

See all 235 papers

Diffusion Language Models Are Natively Length-Aware
Mar 6, 2026Vittorio Rossi, Giacomo Cirò, Davide Beltrame +3Adaptive Computation for Diffusion Models Diffusion Language Models
Department of Computing Sciences, Bocconi University, Milan, Italy.
From Position Risks to Block Survival: Faster Generation for Diffusion Language Models
Sep 27, 2026Siwei Chen, Yuxiang Wan, Yifan Yu +1Speculative Decoding for Diffusion Language Models Diffusion Language Model Decoding
University of Illinois Urbana-Champaign
CARVE: Verified Expansion for Variable-Length Generation in Diffusion Language Models
Aug 31, 2026Wail Bouhedja, Amr Mohamed, Guokan ShangMasked Diffusion Models LLM Inference Acceleration
1MBZUAI · 2Sorbonne Université · 3Ecole Polytechnique

Mar 6, 2026J/K move · Enter open · S save

Diffusion Language Models Are Natively Length-Aware

Vittorio Rossi, Giacomo Cirò, Davide Beltrame, Luca Gandolfi+2

Department of Computing Sciences, Bocconi University, Milan, Italy.

Adaptive Computation for Diffusion Models Diffusion Language Models Diffusion Language Model Inference Efficient Language Model Inference Diffusion Language Model Decoding

Open paper AskPDF arXiv

Unlike autoregressive language models, which terminate variable-length generation upon predicting an End-of-Sequence (EoS) token, Diffusion Language Models (DLMs) operate over a fixed maximum-length context window for a predetermined number of denoising steps. However, this process is independent of the required response length, resulting in computational waste for the majority of short responses common in reasoning and chat tasks. To address this problem, we conjecture that the latent prompt representation contains sufficient information to estimate the required output length. We provide empirical evidence for this phenomenon and propose a zero-shot mechanism to dynamically crop the context window before generation begins, leading to fewer diffusion steps and substantial computational savings. We evaluate our approach on four benchmarks with diverse tasks -- GSM8K (reasoning), HumanEval (code generation), IfEval (instruction following), and LongFormQA (question answering) -- revealing massive efficiency gains at minimal performance impact. We report significant reductions in FLOPs across all tasks, with no statistically significant performance degradation, and significant performance improvements in 2 out of 4 tasks.

Abstract

Diffusion Language Model Decoding

Explore similar work

Diffusion Language Models Are Natively Length-Aware

From Position Risks to Block Survival: Faster Generation for Diffusion Language Models

CARVE: Verified Expansion for Variable-Length Generation in Diffusion Language Models