Diffusion language models promise parallel generation, yet still lag behind autoregressive (AR) models in quality. We stem this gap to a failure of introspective consistency: AR models agree with their own generations, while DLMs often do not. We define the introspective acceptance rate, which measures whether a model accepts its previously generated tokens. This reveals why AR training has a structural advantage: causal masking and logit shifting implicitly enforce introspective consistency. Motivated by this observation, we introduce Introspective Diffusion Language Model (I-DLM), a paradigm that retains diffusion-style parallel decoding while inheriting the introspective consistency of AR training. I-DLM uses a novel introspective strided decoding (ISD) algorithm, which enables the model to verify previously generated tokens while advancing new ones in the same forward pass. From a systems standpoint, we build I-DLM inference engine on AR-inherited optimizations and further customize it with a stationary-batch scheduler. To the best of our knowledge, I-DLM is the first DLM to match the quality of its same-scale AR counterpart while outperforming prior DLMs in both model quality and practical serving efficiency across 15 benchmarks. It reaches 69.6 on AIME-24 and 45.7 on LiveCodeBench-v6, exceeding LLaDA-2.1-mini (16B) by more than 26 and 15 points, respectively. Beyond quality, I-DLM is designed for the growing demand of large-concurrency serving, delivering about 3x higher throughput than prior state-of-the-art DLMs.
Figures & tables
Figure 1: (a) Introspective consistency: standard DLMs generate tokens whose distributions q diverge from the model’s own next-step predictions p ; I-DLM trains generation and introspection to agree ( p≈q ). (b) Quality vs. throughput on MATH-500: I-DLM-8B matches Qwen3-8B (thinking) AR performance while achieving 3.1 × higher throughput and +11.8 points over LLaDA-2.1-mini (16B), and 4.0 × higher throughput over SDAR (8B).
Figure 2: Bottleneck analysis. Key gaps between DLMs and AR models: (a) existing DLMs exhibit a generation–introspection gap—they can generate tokens but cannot reliably introspect on their own output, as measured by the introspection rate; (b) DLM parallel decoding consumes far more compute per token, collapsing throughput under concurrency; (c) higher TPF does not translate to proportionally higher throughput for existing DLMs.
Figure 3: Compute efficiency (TPF/OH) vs. acceptance rate at N=4 . ISD is the only method above the break-even line.
Figure 4: Comparison of decoding paradigms. Our I-DLM uses strict causal attention with adaptive stride ( 1<stride<N ) and is a drop-in replacement within AR serving infrastructure. ISD produces a quality-guaranteed token xi+1 together with draft tokens x^i+2:i+N via introspective strided decoding, xi+1,x^i+2:i+N=πstrided(c1:i,m1:N−1) ; Residual ISD (R-ISD) additionally gates a LoRA residual for bit-for-bit lossless output: xi+1:i+N=πar(c1:i,m1:N−1)+πL(m1:N−1) .
LLaDA-2.1
LLaDA-2.0
LLaDA-2.1
SDAR
SDAR
I-DLM
Qwen3
I-DLM
Qwen3
-mini
-flash
-flash
8B
30B-A3B
8B
8B
32B
32B
Params
16B
100B
100B
8B
30B
8B
8B
32B
32B
Knowledge & Reasoning
ARC-C
90.2
—
—
91.9
93.2
95.8
95.8
96.8 †
97.2
MMLU
74.5
—
—
78.6
82.8
82.4
83.5
86.8 †
87.2
MMLU-Pro
64.8
74.8
76.6
56.9
61.5
73.1
75.1
79.7 †
80.1
Table 1: End-to-end quality. Accuracy (%) on 15 benchmarks. I-DLM results use ISD ( N=4 , sampling). Underline : best non-AR result under 30B. † : best non-AR result under 100B.
Method
GSM8K
MMLU
HumanEval
MBPP
IFEval
Qwen3-8B (AR)
96.0
83.5
95.1
93.4
84.7
NBDiff (7B)
91.0
82.9
89.0
87.6
60.8
Jacobi Forcing (7B)
91.4
—
83.5
70.4
—
WeDLM (8B)
90.2
75.5
75.0
67.0
—
LightningRL (8B)
90.3
—
72.6
58.3
—
TiDAR (8B)
80.4
76.6
57.9
65.4
—
Table 2: Extended comparison on benchmarks commonly reported across diffusion LLM methods. “—” indicates the result is not reported in the original paper.
Figure 6: Performance breakdown of the training design and systems optimizations.
Table 9
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Method
TPF
Overhead
Queries/fwd
ISD (var.)
2−pN−12+p+⋯+pN−2
2+p+⋯+pN−23N−1−NpN−1
N or 2N−1
ISD (fix.)
(same)
2+p+⋯+pN−2(2N−1)(2−pN−1)
2N−1
SDAR
E[S∣N]+1N
E[S∣N]+1
N
TiDAR
1−p1−pN
1−pNN(N+1)(1−p)
N(N+1)
Appendix
Table 5: TPF and overhead formulas (uniform p , stride N ).
Figure 7: Attention kernel forward latency at varying concurrency. I-DLM (Paged, single kernel) vs. Block DLLM (Cascade, three kernels). The cascade overhead grows from +4% at C=1 to +20% at C=64 .
Figure 8: Attention mask comparison for block size N=2 , sequence length L=6 . Input is [xt∣x0] (noisy ∥ clean). Rows are query positions; columns are key positions. Our I-DLM (left) uses strict causal attention everywhere, preserving AR compatibility. SDAR (right) uses bidirectional attention within noisy blocks and block-causal attention in the clean region. The three mask components— Mnoisy (noisy self-attention), Mcross (noisy ← clean cross-attention), Mclean (clean self-attention)—are color-coded.
Figure 9: Detailed ISD illustration at stride N=3 . Step 1 : Bootstrap—append 3 [MASK] tokens, producing x1 (exact) and proposals x^2,x^3,x^4 . Step 2 : Single forward pass that introspects on previous proposals (computing causal anchors pk ) while generating new proposals. (a) All accept : 4 tokens accepted + bonus x5 ; Step 3 introspects on new proposals. (b) Reject x^3 : x1,x2 accepted, x3′ resampled, rest discarded; Step 3 is a pure propose step.
Figure 10: Gated LoRA in Residual ISD (R-ISD). During a single forward pass, [MASK] (propose) positions compute Wx+ABx using base+LoRA weights, producing proposal distributions q . Clean and introspect positions compute Wx using base-only weights, producing the causal anchor distribution p —identical to a pure base AR forward pass. Because of causal attention, introspection positions never attend to [MASK] positions, so p is computed entirely from base-only KV entries. This makes the output bit-for-bit lossless with respect to the base AR model.
Position
Weights
Output
Role
x^k (introspect)
Wx (base only)
pk (causal anchor)
Produces p for p/q acceptance
x1 (exact)
Wx (base only)
p (AR-identical)
Quality-guaranteed token
[M] k (propose)
Wx+BAx (base+LoRA)
qk (proposal)
Strided proposal aligned with p
Appendix
Table 6: Per-position breakdown in R-ISD. Introspect and exact positions use base-only weights, producing the AR-identical causal anchor p . Propose positions add the LoRA residual to produce aligned proposals q .
Model
TP
#Servers
Dtype
DLLM Config
LoRA
8B models (1 GPU per server)
Qwen3-8B (AR)
1
8
bf16
—
—
I-DLM-8B ( N=3 )
1
8
bf16
blockN3
—
I-DLM-8B ( N=5 )
1
8
bf16
blockN5
—
I-DLM-8B (Lossless)
1
8
bf16
blockN3+LoRA
r=128
EAGLE-3
1
8
bf16
—
—
Appendix
Table 7: Serving configurations. All models are served with SGLang on H100 GPUs.
Config
N
block_size
temperature
top_ k
top_ p
spec_verify
blockN3
3
5
1.0
50
0.95
true
blockN5
5
9
1.0
50
0.95
true
blockN3+LoRA
3
5
1.0
50
0.95
true
Appendix
Table 8: ISD algorithm configurations. “blockN k ” denotes stride N=k with block_size=2k−1 .
Method
Draft Model
Draft Params
Notes
EAGLE-3
Tengyunw/qwen3_8b_eagle3
steps=3, topk=1, d=4
3 specs verified per step
Appendix
Table 9: Speculative decoding baseline configurations. “Steps” and “topk” are EAGLE-3 draft parameters.
Category
Benchmark
#Problems
max_tokens
Metric / Extraction
Knowledge
ARC-C
1,172
32,768
Accuracy; choice letter
MMLU
14,042
32,768
Accuracy; ABCD extraction
MMLU-Pro
12,032
32,768
Accuracy; ABCD extraction
GPQA-Diamond
198
32,768
Accuracy; ABCD extraction
GPQA
448
32,768
Accuracy; ABCD extraction
Math
GSM8K
1,319
32,768
Accuracy; \boxed{}
Appendix
Table 10: Evaluation configurations per benchmark. All benchmarks use the Qwen3 chat template with thinking mode enabled. “max_tokens” controls the maximum generation length including the <think> block.
Diffusion language models (DLMs) enable fast generation by predicting multiple tokens in parallel, but their practical adoption remains limited by a persistent quality gap relative to comparably sized autoregressive (AR) models. We attribute this gap to a computation-difficulty mismatch: within a partially observed sequence, some unknown tokens are easy to predict, while others require substantially more computation. Existing DLMs nevertheless apply uniform computational depth to all unknown positions at each denoising step. We introduce ALoDLM, which replaces uniform computation with token-adaptive latent recurrence. At each denoising step, ALoDLM iteratively refines latent representations and allocates computation according to token difficulty. Tokens ready to commit are fed back as discrete context, while unresolved tokens retain and further refine their latent states through additional recurrent passes. To learn token prediction and computation allocation jointly, we formulate token-wise computation schedules as latent variables and derive a conditional negative evidence lower bound (NELBO). We train ALoDLM at 1.7B and 8B parameter scales. Across eleven benchmarks, ALoDLM outperforms all evaluated DLMs and the corresponding AR baselines in average benchmark score at both scales. ALoDLM also retains fast parallel decoding, yielding a strong quality-efficiency trade-off among evaluated autoregressive and diffusion models under optimized inference engines.
Liancheng Fang, Zhuowei Li, Youngeun Kim +10
University of Illinois Chicago · Amazon AGI · Korea University
Diffusion Language Models (DLMs) are often advertised as enabling parallel token generation, yet practical fast DLMs frequently converge to left-to-right, autoregressive (AR)-like decoding dynamics. In contrast, genuinely non-AR generation is promising because it removes AR's sequential bottleneck, better exploiting parallel hardware to reduce synchronization/communication overhead and improve latency scaling with output length. We argue that a primary driver of AR-like decoding is a mismatch between DLM objectives and the highly sequential structure of widely used training data, including standard pretraining corpora and long chain-of-thought (CoT) supervision. Motivated by this diagnosis, we propose NAP (Non-Autoregressive Parallel DLMs), a proof-of-concept, data-centric approach that better aligns supervision with non-AR parallel decoding. NAP curates examples as multiple independent reasoning trajectories and couples them with a parallel-forced decoding strategy that encourages multi-token parallel updates. Across math reasoning benchmarks, NAP yields stronger performance under parallel decoding than DLMs trained on standard long CoT data, with gains growing as parallelism increases. Our results suggest that revisiting data and supervision is a principled direction for mitigating AR-like behavior and moving toward genuinely non-autoregressive parallel generation in DLMs. Our code is available at https://github.com/pixeli99/NAP.
Pengxiang Li, Dilxat Muhtar, Tianlong Chen +2
The Hong Kong Polytechnic University, Hong Kong, China · ELLIS Institute Tübingen, Tübingen, Germany · Max Planck Institute for Intelligent Systems, Tübingen, Germany +3
We present a controlled empirical comparison between autoregressive (AR) and masked diffusion (MDLM) language models. Both models are trained on identical data (50M tokens from TinyStories), identical compute budget (20,000 steps, batch size 32, sequence length 512), and identical hardware (NVIDIA H100 80GB), isolating the generation paradigm as the sole variable. We report three findings. First, both paradigms achieve comparable training throughput (~50K tokens/second), with MDLM requiring only 4.7% more wall-clock time. Second, AR converges faster and begins overfitting by step 14,000, while MDLM converges more slowly and is still improving at step 20,000, suggesting different compute-optimal training regimes. Third, quantitative diversity analysis over 1,000 generated samples reveals a structural diversity-fluency trade-off: AR produces fluent but repetitive outputs (99.8% begin with the same word), while MDLM generates more diverse narratives (93.4% unique 5-word openings, higher Distinct-n, lower Self-BLEU), at the cost of occasional grammatical inconsistencies. All code, trained checkpoints, and data pipelines are released for reproducibility.