DyLLM: Efficient Diffusion LLM Inference via Saliency-based Token Selection and Partial Attention

Younjoo Lee; Seungkyun Dan; Junghoo Lee; Jaiyoung Park; Jung Ho Ahn

cs.CLMar 9, 2026

DyLLM: Efficient Diffusion LLM Inference via Saliency-based Token Selection and Partial Attention

Authors: Younjoo Lee, Seungkyun Dan, Junghoo Lee, Jaiyoung Park, Jung Ho Ahn

Organizations: Seoul National University, Seoul, Republic of Korea

Abstract

Masked diffusion language models enable parallel token decoding, providing a promising alternative to the sequential nature of autoregressive generation. However, their iterative denoising process remains computationally expensive because it repeatedly processes the entire sequence at every step. We observe that across these diffusion steps, most token representations remain stable; only a small subset, which we term salient tokens, contributes meaningfully to the next update. Leveraging this temporal sparsity, we present DyLLM, a training-free inference framework that accelerates decoding by selectively computing only these salient tokens. DyLLM identifies saliency by measuring the cosine similarity of attention contexts between adjacent denoising steps. It recomputes feed-forward and attention operations only for salient tokens while reusing cached activations for the remainder. Across diverse reasoning and code-generation benchmarks, DyLLM achieves up to 9.6x higher throughput while largely preserving the baseline accuracy of representative open-source diffusion LLMs, LLaDA, and Dream.

Figures & tables

Figure 1: Runtime breakdown of autoregressive decoding with vLLM ( Kwon et al., 2023 ) , original diffusion LLM implementation, and DyLLM on random GSM8K 5-shot prompts (

\text{batch size}=16

). Original diffusion LLM repeats the full steps, whereas DyLLM recomputes only salient tokens, reducing the dominant per-step overhead.

Figure 2: Distribution of temporal cosine similarity $s_{t,l}$ across denoising steps and layers ( $l\in\{8,16,24,32\}$ ), measured on LLaDA with GSM8K 5-shot prompts. Most token representations remain highly similar across consecutive steps, with the density concentrated near $s_{t,l}\approx 1$ . In deeper layers, the distribution shows a larger low-similarity tail, indicating that more tokens undergo meaningful updates.

Figure 3: Approximate attention in DyLLM. DyLLM computes exact attention rows for salient query tokens using the cached key/value states (row-sparse path). For non-salient query tokens, it approximates context updates using only the attention-score columns corresponding to salient token indices (column-sparse path).

Bench.	Model	Thres.	Orig.	Salient	Salient + Approx.
GSM8K	LLaDA	$0.995$	77.79	78.09	78.01
		$0.99$		78.85	79.08
		$0.985$		78.62	79.00
	Dream	$0.9975$	75.59	79.98	79.30
		$0.995$		79.98	78.09
		$0.99$		78.70	76.04

Table 1: Impact of Saliency-based FFN. The “Salient” column reports accuracy when FFN computation is restricted to salient tokens while using full attention. The “Salient+Approx.” column reports the full DyLLM configuration, where saliency-aware approximate attention is additionally applied.

Figure 4: Error of approximate attention compared to the exact attention measured by cosine similarity on LLaDA and Dream.

LLaDA 8B	Original	DyLLM ( $\tau$ =0.995)	DyLLM ( $\tau$ =0.99)	Fast-Prefix	Fast-Dual	dLLM-Cache
GSM8K	77.79	78.01	79.08	77.18	78.24	77.18
GSM8K	11.47 ( $\times 1.00$ )	80.15 ( $\times 6.99$ )	87.21 ( $\times 7.60$ )	51.05 ( $\times 4.45$ )	75.24 ( $\times 6.56$ )	36.77 ( $\times 3.21$ )
MBPP	29.20	28.20	30.00	25.40	25.40	29.00
MBPP	33.11 ( $\times 1.00$ )	151.27 ( $\times 4.57$ )	169.62 ( $\times 5.12$ )	116.27 ( $\times 3.51$ )	165.44 ( $\times 5.00$ )	93.04 ( $\times 2.81$ )
MATH	33.22	38.68	38.08	33.22	32.36	24.70
MATH	15.81 ( $\times 1.00$ )	96.98 ( $\times 6.13$ )	106.06 ( $\times 6.71$ )	62.12 ( $\times 3.93$ )	93.26 ( $\times 5.90$ )	36.56 ( $\times 2.31$ )

Table 2: Accuracy and throughput results across benchmarks with

n_{u}=1

. Accuracy is shown in black, throughput (tokens per second) in blue, and speedup relative to the original implementation in orange.

Figure 5: The average number of salient tokens per layer with 100 random GSM8K 5-shot prompts.

L_{R}

is set to 256, as in all the other experiments. The number of salient tokens (

L_{sal}

) increases for about 3/4 of layers and starts decreasing.

Figure 6: Accuracy results varying

\tau

across GSM8K and MBPP datasets. Accuracy generally declines as the threshold is lowered.

Figure 7: Accuracy and throughput of DyLLM, Fast-dLLM, and the original implementation across GSM8K, MBPP, MATH, and MMLU-Pro. Each curve contains three operating points corresponding to $n_{u}=1,2,4$ , where a larger $n_{u}$ yields higher throughput, but lower accuracy. DyLLM consistently preserves accuracy with high throughput compared to the strong baselines.

Model	Method	Avg. $n_{u}$	Acc.
LLaDA	Original	3.28	77.86
	Fast-dLLM Prefix	3.16	77.94
	Fast-dLLM Dual	2.98	78.39
	DyLLM ( $\tau$ = 0.99)	3.18	77.03
	DyLLM ( $\tau$ = 0.995)	3.20	78.47
Dream	Original	3.82	75.51

Table 3: Comparison of the average number of unmasked tokens per step and accuracy score on LLaDA and Dream with the GSM8K 5-shot dataset. Bold denotes the best accuracy among all methods, and underline denotes the highest Avg.

n_{u}

among acceleration methods.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Comparison of MDLM architecture and ARLM architecture.

Figure 9: Accuracy results varying

\tau

across MATH and MMLU-Pro benchmark.

LLaDA
Step	GSM8K				MBPP				MATH				MMLU-Pro
Step	L8	L16	L24	L32	L8	L16	L24	L32	L8	L16	L24	L32	L8	L16	L24	L32
0–15	33.4	50.2	65.4	23.4	52.2	63.4	107.4	54.3	36.3	45.0	64.4	28.6	40.1	56.6	82.8	34.8
64–79	25.5	48.5	72.8	43.3	22.3	49.5	77.5	46.7	26.6	46.2	67.6	40.2	23.3	58.4	92.7	56.9
128–143	21.6	43.0	59.4	40.9	22.8	46.3	68.9	46.7	23.8	44.1	63.2	42.1	20.8	45.9	71.8	44.4
192–207	17.8	36.3	51.3	41.8	19.2	41.6	66.3	50.2	20.3	38.4	59.7	48.5	17.3	39.2	68.0	49.4

Appendix

Table 4: Layer-wise average of

L_{sal}

across steps.

Figure 10: The average of

L_{sal}

per layer across MBPP, MATH, and MMLU-Pro. The same layer-wise trend observed on GSM8K also appears across these benchmarks.

	LLaDA		Dream
	Semi-AR	Non-Semi-AR	Semi-AR	Non-Semi-AR
$n_{u}=1$	77.79	58.07	75.59	42.30
$n_{u}=2$	74.68	54.36	67.12	40.41
$n_{u}=4$	66.41	50.19	48.45	39.12

Appendix

Table 5: Impact of Semi-AR decoding on accuracy score

Model	Interval	GSM8K	MBPP	MATH	MMLU-Pro
LLaDA 8B Instruct	$K_{P}$	50	100	50	51
LLaDA 8B Instruct	$K_{R}$	7	7	1	3
Dream 7B Instruct	$K_{P}$	25	10	50	5
Dream 7B Instruct	$K_{R}$	2	8	1	1

Appendix

Table 6: Interval steps for LLaDA Instruct and Dream Instruct of dLLM-Cache

LLaDA 8B	Original	DyLLM ( $\tau$ =0.995)	DyLLM ( $\tau$ =0.99)	Fast-Prefix	Fast-Dual
GSM8K	76.80	77.10	78.09	77.18	78.09
GSM8K	31.27 ( $\times 1.00$ )	184.81 ( $\times 5.91$ )	196.24 ( $\times 6.28$ )	94.85 ( $\times 3.03$ )	126.71 ( $\times 4.05$ )
MBPP	29.40	30.00	29.00	25.20	26.00
MBPP	43.11 ( $\times 1.00$ )	210.92 ( $\times 4.89$ )	227.29 ( $\times 5.27$ )	112.71 ( $\times 2.61$ )	143.69 ( $\times 3.33$ )
MATH	33.46	38.98	37.92	33.06	32.70
MATH	44.41 ( $\times 1.00$ )	226.55 ( $\times 5.10$ )	244.28 ( $\times 5.50$ )	124.38 ( $\times 2.80$ )	160.41 ( $\times 3.61$ )

Appendix

Table 7: Accuracy and throughput results across benchmarks with

n_{u}=1

on NVIDIA B200. Accuracy is shown in black, throughput (tokens per second) in blue, and speedup relative to the original implementation in orange.

Explore similar work

Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decoding
Jan 25, 2026Zhongyu Xiao, Zhiwei Hao, Jianyuan Guo +4Adaptive Inference Blockwise Diffusion Language Models
School of information and Electronics, Beijing Institute of Technology, Beijing · Department of Computer Science, City University of Hong Kong, Hong Kong · School of Computer Science, Wuhan University, Wuhan +2
Window-Diffusion: Accelerating Diffusion Language Model Inference with Windowed Token Pruning and Caching
Jan 28, 2026Fengrui Zuo, Zhiwei Ke, Yiming Liu +3Diffusion Language Model Inference Token Pruning
School of Computer Science and Technology, University of Science and Technology of China · Suzhou Institute of Advanced Research, University of Science and Technology of China
ES-dLLM: Efficient Inference for Diffusion Large Language Models by Early-Skipping
Mar 10, 2026Zijian Zhu, Fei Ren, Zhanhong Tan +1Diffusion Language Model Inference Diffusion Language Models
IIIS, Tsinghua University · Polar Bear Tech.

Jan 25, 2026J/K move · Enter open · S save

Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decoding

Zhongyu Xiao, Zhiwei Hao, Jianyuan Guo, Yong Luo+3

School of information and Electronics, Beijing Institute of Technology, Beijing · Department of Computer Science, City University of Hong Kong, Hong Kong · School of Computer Science, Wuhan University, Wuhan+2

Adaptive Inference Blockwise Diffusion Language Models Diffusion Language Model Inference Token Pruning Diffusion Language Model Decoding Diffusion Model Inference Acceleration

Open paper AskPDF arXiv

Diffusion Large Language Models (dLLMs) offer a compelling paradigm for natural language generation, leveraging parallel decoding and bidirectional attention to achieve superior global coherence compared to autoregressive models. While recent works have accelerated inference via KV cache reuse or heuristic decoding, they overlook the intrinsic inefficiencies within the block-wise diffusion process. Specifically, they suffer from spatial redundancy by modeling informative-sparse suffix regions uniformly and temporal inefficiency by applying fixed denoising schedules across all the decoding process. To address this, we propose Streaming-dLLM, a training-free framework that streamlines inference across both spatial and temporal dimensions. Spatially, we introduce attenuation guided suffix modeling to approximate the full context by pruning redundant mask tokens. Temporally, we employ a dynamic confidence aware strategy with an early exit mechanism, allowing the model to skip unnecessary iterations for converged tokens. Extensive experiments show that Streaming-dLLM achieves up to 68.2X speedup while maintaining generation quality, highlighting its effectiveness in diffusion decoding. The code is available at https://github.com/xiaoshideta/Streaming-dLLM.

Abstract

Explore similar work

Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decoding

Window-Diffusion: Accelerating Diffusion Language Model Inference with Windowed Token Pruning and Caching

ES-dLLM: Efficient Inference for Diffusion Large Language Models by Early-Skipping