Beyond Scattered Acceptance: Fast and Coherent Inference for DLMs via Longest Stable Prefixes

Pengxiang Li; Joey Tsai; Hongwei Xue; Kunyu Shi; Shilin Yan

cs.CVMar 5, 2026

Beyond Scattered Acceptance: Fast and Coherent Inference for DLMs via Longest Stable Prefixes

Authors: Pengxiang Li, Joey Tsai, Hongwei Xue, Kunyu Shi, Shilin Yan

Organizations: Accio Team, Alibaba Group · Tsinghua University

Abstract

Diffusion Language Models (DLMs) promise highly parallel text generation, yet their practical inference speed is often bottlenecked by suboptimal decoding schedulers. Standard approaches rely on 'scattered acceptance'-committing high confidence tokens at disjoint positions throughout the sequence. This approach inadvertently fractures the Key-Value (KV) cache, destroys memory locality, and forces the model into costly, repeated repairs across unstable token boundaries. To resolve this, we present the Longest Stable Prefix (LSP) scheduler, a training-free and model-agnostic inference paradigm based on monolithic prefix absorption. In each denoising step, LSP evaluates token stability via a single forward pass, dynamically identifies a contiguous left-aligned block of stable predictions, and snaps its boundary to natural linguistic or structural delimiters before an atomic commitment. This prefix-first topology yields dual benefits: systemically, it converts fragmented KV cache updates into efficient, contiguous appends; algorithmically, it preserves bidirectional lookahead over a geometrically shrinking active suffix, drastically reducing token flip rates and denoiser calls. Extensive evaluations on LLaDA-8B and Dream-7B demonstrate that LSP accelerates inference by up to 3.4x across rigorous benchmarks including mathematical reasoning, code generation, multilingual (CJK) tasks, and creative writing while matching or slightly improving output quality. By fundamentally restructuring the commitment topology, LSP bridges the gap between the theoretical parallelism of DLMs and practical hardware efficiency.

Figures & tables

Figure 1: The iterative process of the Longest Stable Prefix (LSP) scheduler. In each step, LSP performs a single forward pass to assess the stability of predictions for the current active suffix, measured by the logit margin (

\delta_{i}

). Instead of accepting scattered tokens, it atomically commits the longest contiguous prefix of tokens that meet an adaptively determined stability threshold (

\tau

). As shown, the frozen prefix (green) grows monolithically, causing the active suffix (white) to shrink.

Benchmark	LLaDA-8B		Dream-7B
Benchmark	Full	LSP (Ours)	Full	LSP (Ours)
General Tasks
MMLU	54.1	54.2 ( 2.32 $\times$ )	67.6	66.4 ( 2.12 $\times$ )
ARC-C	83.2	83.3 ( 1.91 $\times$ )	88.1	88.0 ( 2.28 $\times$ )
Hellaswag	68.7	70.7 ( 2.18 $\times$ )	81.2	82.1 ( 2.53 $\times$ )
TruthfulQA	34.4	45.8 ( 2.29 $\times$ )	55.6	53.5 ( 1.86 $\times$ )

Table 1: Main benchmark results on LLaDA-8B and Dream-7B. We report the task-specific score (%) and the inference speedup over the ‘Full‘ baseline. LSP delivers substantial speedups while maintaining or even improving task performance.

Strategy	GSM8K Score (%)	Total Steps (Avg.)
Fixed Prefix (1 tokens)	67.1	128
Fixed Prefix (2 tokens)	66.8	64
Fixed Prefix (4 tokens)	47.6	32
Fixed Prefix (8 tokens)	19.3	16
Adaptive (LSP)	69.9	$\sim$ 68

Table 2: Ablation on Sizing Strategy (GSM8K, LLaDA-8B). Fixed-size commitment strategies are brittle, forcing a trade-off between the number of inference steps (speed) and final accuracy. LSP’s adaptive sizing dynamically finds the most effective balance.

Method	Score (%)	Total Steps (Avg.)
Ablation on Structural Snapping
LSP w/o Snapping	67.8	$\sim$ 50
Full LSP (Ours)	69.9	$\sim$ 68
Ablation on Commitment Topology
Scattered-Margin	68.9	128
Full LSP (Ours)	69.9	$\sim$ 68

Table 3: Ablation studies on core LSP components (GSM8K, LLaDA-8B). Both structural snapping and the prefix-first topology are crucial for achieving high performance. Each component is compared against the full LSP method.

Task	LSP Generation Process (Committed Prefixes per Step)
Prompt:	Natalia sold clips to 4 of her friends. She sold 8 clips to each friend. Then she bought 15 more clips. How many clips does Natalia have now?
	Natalia sold 8 clips to each of her 4 friends. This means she sold a total of 4 * 8 = 32 clips. She then bought 15 more clips, so she now has 32 + 15 = 47 clips. The final answer is 47.
Prompt:	John has 12 apples. He gives half to Mary. Then Mary buys twice as many apples as she received from John. How many apples does Mary have now?
	John gives half of his 12 apples to Mary, so Mary receives 12 / 2 = 6 apples. Then Mary buys twice the number she received, which is 2 * 6 = 12 apples. Mary now has her original 6 plus the 12 she bought, for a total of 6 + 12 = 18 apples. Answer: 18.

Table 4: Visualization of LSP’s monolithic prefix absorption for math reasoning (Granular View). Each colored block represents the Longest Stable Prefix atomically committed in a single step. This more detailed breakdown shows how LSP iteratively extends the solution by committing shorter, coherent chunks. The strict left-to-right, contiguous growth (light to dark green) is maintained, and commit boundaries align with natural linguistic or mathematical units (e.g., clauses, calculations), demonstrating LSP’s structural snapping in action.

Figure 2: Quantifying Repair Costs via Token Flip Rate. We measure the percentage of tokens in the active suffix that change their top prediction between consecutive diffusion steps. While the scattered baseline forces the model to constantly reconcile a fragmented context (maintaining high flip rates), LSP locks in a coherent prefix early. This stabilizes the future generation context, drastically reducing token oscillations and repair costs in the mid-to-late stages (from 14.2% down to 4.3%).

Diffusion Language Model Inference

Diffusion language model inference studies how to generate, guide, and serve text efficiently through iterative denoising and parallel token updates.

See all 177 papers

Aug 24, 2026

Accelerating Diffusion Language Models via Structured Suffix Modeling

Diffusion Language Models (DLMs) exhibit strong parallel decoding capabilities by denoising multiple tokens in a single generation step. However, this parallelism comes with substantial computational overhead, as each step requires interactions with all suffix tokens. Existing methods typically reduce this cost by retaining only a local suffix window as a substitute for the full suffix. Despite their effectiveness, these methods overlook the structural heterogeneity across suffix regions and re-initialize suffix tokens with identical representations at each timestep. To this end, we propose a structured suffix modeling method for efficient DLM inference. Specifically, we divide the suffix into three regions, i.e., the local, middle, and tail regions, and retain different numbers of suffix tokens in each region according to their structural roles. Moreover, we incorporate the decoding results from the previous step into the suffix token representations at the current step, allowing them to carry evolving denoising information across generation steps. Notably, our method is training-free and orthogonal to several existing acceleration techniques, such as parallel decoding strategies and KV cache. Empirical results across multiple benchmarks on three DLMs demonstrate that our method can further accelerate DLM inference and improve performance in most cases. In particular, in long-sequence inference, our method achieves up to a

72.81\times

speedup when combined with other acceleration techniques. Our code is available at https://github.com/zifengcheng/SSM.

Zifeng Cheng, Keda Li, Zhiwei Jiang +3

State Key Laboratory for Novel Software Technology, Nanjing University · National University of Singapore

Diffusion Language Model Inference Diffusion Language Model Decoding Diffusion Language Models

Ask PDF

Jan 28, 2026

Window-Diffusion: Accelerating Diffusion Language Model Inference with Windowed Token Pruning and Caching

Diffusion language models (DLMs) generate text through iterative denoising, but inference requires full-sequence attention at every iteration, resulting in substantial redundant computation on masked tokens. Block-wise diffusion can reduce this cost, yet it typically relies on retraining and constrained update orders, limiting its direct applicability to pretrained DLMs. Our token-level analysis reveals pronounced structural locality in DLM inference. Decoding is driven by a small set of prefix-localized active tokens; the influence of distant undecoded context diminishes rapidly, and decoded tokens exhibit stage-wise temporal stability, enabling reuse of intermediate representations except for a brief post-decode transient. Motivated by these observations, we propose \placeholder\footnote{The source code is available at https://github.com/vhicrgit/Window-Diffusion.}, a window-based token pruning and caching method for inference. We maintain a local computation window that slides rightward as denoising progresses, and partition undecoded tokens into: (i) active tokens that are computed online, (ii) buffer tokens whose KV states are cached and periodically refreshed, and (iii) far-field tokens that are pruned outside the window. Computation is restricted to active and buffer tokens within the window, while far-field tokens are omitted at each stage. Experiments on LLaDA and Dream show that, under matched compute budgets, our method achieves up to

99\times

inference speedup while largely preserving generation performance.

Fengrui Zuo, Zhiwei Ke, Yiming Liu +3

School of Computer Science and Technology, University of Science and Technology of China · Suzhou Institute of Advanced Research, University of Science and Technology of China

Diffusion Language Model Inference Token Pruning Diffusion Model Caching Attention Mechanisms Diffusion Model Inference Acceleration

Ask PDF

May 15, 2026

PSD: Pushing the Pareto Frontier of Diffusion LLMs via Parallel Speculative Decoding

Diffusion large language models (dLLMs) generate text by iteratively denoising masked token sequences. Although dLLMs can predict all masked positions in parallel within each step, the large number of denoising iterations still makes inference expensive. This cost can be reduced spatially by unmasking multiple tokens per step, or temporally by collapsing multiple denoising steps into one verification call. We propose Parallel Speculative Decoding (PSD), a training-free framework that jointly improves inference along both axes. Using the confidence scores from a single forward pass, PSD selects positions to unmask via a configurable, adaptive unmasking policy and constructs multi-depth speculative drafts without extra model calls. A final batched verification pass then applies hierarchical acceptance, keeping the deepest draft that remains consistent with the updated predictions. Experiments on three dLLMs across reasoning and code generation tasks show that PSD achieves favorable trade-offs between inference efficiency and generation quality, reaching up to

5.5\times

tokens per forward pass with accuracy comparable to greedy decoding.

Shengyin Sun, Yiming Li, Renxi Liu +7

Department of Computer Science City University of Hong Kong · Huawei Technologies · City University of Hong Kong

Parallel Speculative Decoding Masked Diffusion Models Speculative Decoding for Diffusion Language Models Speculative Decoding Diffusion Language Model Inference Diffusion Language Model Decoding+3

Ask PDF

Abstract

Diffusion Language Model Inference

Explore similar work

Accelerating Diffusion Language Models via Structured Suffix Modeling

Window-Diffusion: Accelerating Diffusion Language Model Inference with Windowed Token Pruning and Caching

PSD: Pushing the Pareto Frontier of Diffusion LLMs via Parallel Speculative Decoding