Introspective Diffusion Language Models

Yifan Yu; Yuqing Jian; Junxiong Wang; Zhongzhu Zhou; Donglin Zhuang; Xinyu Fang; Sri Yanamandra; Xiaoxia Wu

cs.AIApr 13, 2026

Introspective Diffusion Language Models

Authors: Yifan Yu, Yuqing Jian, Junxiong Wang, Zhongzhu Zhou, Donglin Zhuang, Xinyu Fang, Sri Yanamandra, Xiaoxia Wu, +7 more

Organizations: Together AI · University of Illinois Urbana-Champaign · Princeton University · Stanford University · The University of Texas at Austin

Abstract

Diffusion language models promise parallel generation, yet still lag behind autoregressive (AR) models in quality. We stem this gap to a failure of introspective consistency: AR models agree with their own generations, while DLMs often do not. We define the introspective acceptance rate, which measures whether a model accepts its previously generated tokens. This reveals why AR training has a structural advantage: causal masking and logit shifting implicitly enforce introspective consistency. Motivated by this observation, we introduce Introspective Diffusion Language Model (I-DLM), a paradigm that retains diffusion-style parallel decoding while inheriting the introspective consistency of AR training. I-DLM uses a novel introspective strided decoding (ISD) algorithm, which enables the model to verify previously generated tokens while advancing new ones in the same forward pass. From a systems standpoint, we build I-DLM inference engine on AR-inherited optimizations and further customize it with a stationary-batch scheduler. To the best of our knowledge, I-DLM is the first DLM to match the quality of its same-scale AR counterpart while outperforming prior DLMs in both model quality and practical serving efficiency across 15 benchmarks. It reaches 69.6 on AIME-24 and 45.7 on LiveCodeBench-v6, exceeding LLaDA-2.1-mini (16B) by more than 26 and 15 points, respectively. Beyond quality, I-DLM is designed for the growing demand of large-concurrency serving, delivering about 3x higher throughput than prior state-of-the-art DLMs.

Figures & tables

Figure 1: (a) Introspective consistency: standard DLMs generate tokens whose distributions $q$ diverge from the model’s own next-step predictions $p$ ; I-DLM trains generation and introspection to agree ( $p\approx q$ ). (b) Quality vs. throughput on MATH-500: I-DLM-8B matches Qwen3-8B (thinking) AR performance while achieving 3.1 $\times$ higher throughput and +11.8 points over LLaDA-2.1-mini (16B), and 4.0 $\times$ higher throughput over SDAR (8B).

Panel of Figure 2 — Figure 2: Bottleneck analysis. Key gaps between DLMs and AR models: (a) existing DLMs exhibit a generation–introspection gap—they can generate tokens but cannot reliably introspect on their own output, as measured by the introspection rate; (b) DLM parallel decoding consumes far more compute per token, collapsing throughput under concurrency; (c) higher TPF does not translate to proportionally higher throughput for existing DLMs.

Figure 3: Compute efficiency (TPF/OH) vs. acceptance rate at

N{=}4

. ISD is the only method above the break-even line.

Figure 4: Comparison of decoding paradigms. Our I-DLM uses strict causal attention with adaptive stride ( $1<\text{stride}<N$ ) and is a drop-in replacement within AR serving infrastructure. ISD produces a quality-guaranteed token $\bm{x}_{i+1}$ together with draft tokens $\hat{\bm{x}}_{i+2:i+N}$ via introspective strided decoding, $\bm{x}_{i+1},\hat{\bm{x}}_{i+2:i+N}=\pi_{\text{strided}}(c_{1:i},m_{1:N-1})$ ; Residual ISD (R-ISD) additionally gates a LoRA residual for bit-for-bit lossless output: $\bm{x}_{i+1:i+N}=\pi_{\text{ar}}(c_{1:i},m_{1:N-1})+\pi_{\text{L}}(m_{1:N-1})$ .

	LLaDA-2.1	LLaDA-2.0	LLaDA-2.1	SDAR	SDAR	I-DLM	Qwen3	I-DLM	Qwen3
	-mini	-flash	-flash	8B	30B-A3B	8B	8B	32B	32B
Params	16B	100B	100B	8B	30B	8B	8B	32B	32B
Knowledge & Reasoning
ARC-C	90.2	—	—	91.9	93.2	95.8	95.8	96.8 †	97.2
MMLU	74.5	—	—	78.6	82.8	82.4	83.5	86.8 †	87.2
MMLU-Pro	64.8	74.8	76.6	56.9	61.5	73.1	75.1	79.7 †	80.1

Table 1: End-to-end quality. Accuracy (%) on 15 benchmarks. I-DLM results use ISD (

N{=}4

, sampling). Underline : best non-AR result under 30B. † : best non-AR result under 100B.

Method	GSM8K	MMLU	HumanEval	MBPP	IFEval
Qwen3-8B (AR)	96.0	83.5	95.1	93.4	84.7
NBDiff (7B)	91.0	82.9	89.0	87.6	60.8
Jacobi Forcing (7B)	91.4	—	83.5	70.4	—
WeDLM (8B)	90.2	75.5	75.0	67.0	—
LightningRL (8B)	90.3	—	72.6	58.3	—
TiDAR (8B)	80.4	76.6	57.9	65.4	—

Table 2: Extended comparison on benchmarks commonly reported across diffusion LLM methods. “—” indicates the result is not reported in the original paper.

Figure 5: Throughput–latency tradeoff across batch sizes (1, 4, 16, 64).

Panel of Figure 6 — Figure 6: Performance breakdown of the training design and systems optimizations.

Table 9

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Method	TPF	Overhead	Queries/fwd
ISD (var.)	$\frac{2+p+\cdots+p^{N-2}}{2-p^{N-1}}$	$\frac{3N-1-Np^{N-1}}{2+p+\cdots+p^{N-2}}$	$N$ or $2N{-}1$
ISD (fix.)	(same)	$\frac{(2N{-}1)(2-p^{N-1})}{2+p+\cdots+p^{N-2}}$	$2N{-}1$
SDAR	$\frac{N}{\mathbb{E}[S\|N]+1}$	$\mathbb{E}[S\|N]+1$	$N$
TiDAR	$\frac{1-p^{N}}{1-p}$	$\frac{N(N+1)(1-p)}{1-p^{N}}$	$N(N{+}1)$

Appendix

Table 5: TPF and overhead formulas (uniform

p

, stride

N

Figure 7: Attention kernel forward latency at varying concurrency. I-DLM (Paged, single kernel) vs. Block DLLM (Cascade, three kernels). The cascade overhead grows from

+4\%

C{=}1

+20\%

C{=}64

Panel of Figure 8 — Figure 8: Attention mask comparison for block size $N{=}2$ , sequence length $L{=}6$ . Input is $[x_{t}\,|\,x_{0}]$ (noisy $\|$ clean). Rows are query positions; columns are key positions. Our I-DLM (left) uses strict causal attention everywhere, preserving AR compatibility. SDAR (right) uses bidirectional attention within noisy blocks and block-causal attention in the clean region. The three mask components— $M_{\text{noisy}}$ (noisy self-attention), $M_{\text{cross}}$ (noisy $\leftarrow$ clean cross-attention), $M_{\text{clean}}$ (clean self-attention)—are color-coded.

Panel of Figure 9 — Figure 9: Detailed ISD illustration at stride $N{=}3$ . Step 1 : Bootstrap—append 3 [MASK] tokens, producing $x_{1}$ (exact) and proposals $\hat{x}_{2},\hat{x}_{3},\hat{x}_{4}$ . Step 2 : Single forward pass that introspects on previous proposals (computing causal anchors $p_{k}$ ) while generating new proposals. (a) All accept : 4 tokens accepted + bonus $x_{5}$ ; Step 3 introspects on new proposals. (b) Reject $\hat{x}_{3}$ : $x_{1},x_{2}$ accepted, $x^{\prime}_{3}$ resampled, rest discarded; Step 3 is a pure propose step.

Panel of Figure 10 — Figure 10: Gated LoRA in Residual ISD (R-ISD). During a single forward pass, [MASK] (propose) positions compute $Wx+ABx$ using base+LoRA weights, producing proposal distributions $q$ . Clean and introspect positions compute $Wx$ using base-only weights, producing the causal anchor distribution $p$ —identical to a pure base AR forward pass. Because of causal attention, introspection positions never attend to [MASK] positions, so $p$ is computed entirely from base-only KV entries. This makes the output bit-for-bit lossless with respect to the base AR model.

Position	Weights	Output	Role
$\hat{x}_{k}$ (introspect)	$Wx$ (base only)	$p_{k}$ (causal anchor)	Produces $p$ for $p/q$ acceptance
$x_{1}$ (exact)	$Wx$ (base only)	$p$ (AR-identical)	Quality-guaranteed token
[M] k (propose)	$Wx+BAx$ (base+LoRA)	$q_{k}$ (proposal)	Strided proposal aligned with $p$

Appendix

Table 6: Per-position breakdown in R-ISD. Introspect and exact positions use base-only weights, producing the AR-identical causal anchor

p

. Propose positions add the LoRA residual to produce aligned proposals

q

Model	TP	#Servers	Dtype	DLLM Config	LoRA
8B models (1 GPU per server)
Qwen3-8B (AR)	1	8	bf16	—	—
I-DLM-8B ( $N{=}3$ )	1	8	bf16	blockN3	—
I-DLM-8B ( $N{=}5$ )	1	8	bf16	blockN5	—
I-DLM-8B (Lossless)	1	8	bf16	blockN3+LoRA	r=128
EAGLE-3	1	8	bf16	—	—

Appendix

Table 7: Serving configurations. All models are served with SGLang on H100 GPUs.

Config	$N$	block_size	temperature	top_ $k$	top_ $p$	spec_verify
blockN3	3	5	1.0	50	0.95	true
blockN5	5	9	1.0	50	0.95	true
blockN3+LoRA	3	5	1.0	50	0.95	true

Appendix

Table 8: ISD algorithm configurations. “blockN

k

” denotes stride

N{=}k

with

\text{block\_size}=2k{-}1

Method	Draft Model	Draft Params	Notes
EAGLE-3	Tengyunw/qwen3_8b_eagle3	steps=3, topk=1, d=4	3 specs verified per step

Appendix

Table 9: Speculative decoding baseline configurations. “Steps” and “topk” are EAGLE-3 draft parameters.

Category	Benchmark	#Problems	max_tokens	Metric / Extraction
Knowledge	ARC-C	1,172	32,768	Accuracy; choice letter
	MMLU	14,042	32,768	Accuracy; ABCD extraction
	MMLU-Pro	12,032	32,768	Accuracy; ABCD extraction
	GPQA-Diamond	198	32,768	Accuracy; ABCD extraction
	GPQA	448	32,768	Accuracy; ABCD extraction
Math	GSM8K	1,319	32,768	Accuracy; \boxed{}

Appendix

Table 10: Evaluation configurations per benchmark. All benchmarks use the Qwen3 chat template with thinking mode enabled. “max_tokens” controls the maximum generation length including the <think> block.

Parameter	Value
Concurrency levels	$C\in\{1,2,4,8,16,32,48,64\}$
Request arrival	Burst mode (all requests submitted simultaneously)
Output length	2,048 tokens (fixed)
Measurement	Total tokens / wall-clock time (first to last completion)
Warmup	5 requests discarded before measurement
GPU	1 $\times$ H100 (TP=1) for 8B; 2 $\times$ H100 (TP=2) for 32B

Appendix

Table 11: Throughput benchmark configuration.

Toggle	Disables	Effect
--disable-cuda-graph	CUDA graph capture/replay	Forces eager model forward
NO_DECODE_LOOP=1	Stationary-batch decode loop	Falls back to full scheduler pipeline
NO_ARGMAX_PROPOSALS=1	Argmax for draft positions	Uses full sampling for specs
NO_FUSED_VERIFY=1	Fused Triton verify kernel	Uses 7+ separate kernel launches
NO_DEFERRED_STREAM=1	Deferred stream_output	Runs on critical path
DLLM_USE_RAGGED=1	Paged-only attention	Forces cascade (3 kernels/layer)

Appendix

Table 12: Infrastructure ablation toggles. Each toggle disables one optimization to measure its isolated contribution.

Hardware	TP	$N$	Model Variant	Benchmark	TPS
B200	2	8	I-DLM-8B (ext. training $N{=}8$ )	GSM8K	925
H100	2	8	I-DLM-8B (ext. training $N{=}8$ )	MBPP	685
H100	1	4	I-DLM-8B (ext. training $N{=}4$ )	MATH-500	341
H100	1	4	I-DLM-8B (ext. training $N{=}4$ )	AIME	314
H100	1	4	I-DLM-8B (ext. training $N{=}4$ )	ShareGPT	319
H100	1	3	I-DLM-8B	MATH-500	272

Appendix

Table 13: Peak per-request TPS at various stride, hardware, and model configurations. All models are 8B scale.

Explore similar work

ALoDLM: Adaptively Looped Diffusion Language Models
Oct 3, 2026Liancheng Fang, Zhuowei Li, Youngeun Kim +10Diffusion Language Models Diffusion Language Model Inference
University of Illinois Chicago · Amazon AGI · Korea University
Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding?
Feb 26, 2026Pengxiang Li, Dilxat Muhtar, Tianlong Chen +2Diffusion Language Model Decoding Diffusion Language Models
The Hong Kong Polytechnic University, Hong Kong, China · ELLIS Institute Tübingen, Tübingen, Germany · Max Planck Institute for Intelligent Systems, Tübingen, Germany +3
Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison
Mar 23, 2026Caio VicentinoDiffusion Language Models Autoregressive Language Modeling
Independent Researcher

Oct 3, 2026J/K move · Enter open · S save

ALoDLM: Adaptively Looped Diffusion Language Models

Liancheng Fang, Zhuowei Li, Youngeun Kim, Tianchen Zhao+9

University of Illinois Chicago · Amazon AGI · Korea University

Diffusion Language Models Diffusion Language Model Inference Diffusion Language Model Decoding

Open paper AskPDF arXiv

Diffusion language models (DLMs) enable fast generation by predicting multiple tokens in parallel, but their practical adoption remains limited by a persistent quality gap relative to comparably sized autoregressive (AR) models. We attribute this gap to a computation-difficulty mismatch: within a partially observed sequence, some unknown tokens are easy to predict, while others require substantially more computation. Existing DLMs nevertheless apply uniform computational depth to all unknown positions at each denoising step. We introduce ALoDLM, which replaces uniform computation with token-adaptive latent recurrence. At each denoising step, ALoDLM iteratively refines latent representations and allocates computation according to token difficulty. Tokens ready to commit are fed back as discrete context, while unresolved tokens retain and further refine their latent states through additional recurrent passes. To learn token prediction and computation allocation jointly, we formulate token-wise computation schedules as latent variables and derive a conditional negative evidence lower bound (NELBO). We train ALoDLM at 1.7B and 8B parameter scales. Across eleven benchmarks, ALoDLM outperforms all evaluated DLMs and the corresponding AR baselines in average benchmark score at both scales. ALoDLM also retains fast parallel decoding, yielding a strong quality-efficiency trade-off among evaluated autoregressive and diffusion models under optimized inference engines.

Abstract

Explore similar work

ALoDLM: Adaptively Looped Diffusion Language Models

Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding?

Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison