Diffusion language models (DLMs) enable fast generation by predicting multiple tokens in parallel, but their practical adoption remains limited by a persistent quality gap relative to comparably sized autoregressive (AR) models. We attribute this gap to a computation-difficulty mismatch: within a partially observed sequence, some unknown tokens are easy to predict, while others require substantially more computation. Existing DLMs nevertheless apply uniform computational depth to all unknown positions at each denoising step. We introduce ALoDLM, which replaces uniform computation with token-adaptive latent recurrence. At each denoising step, ALoDLM iteratively refines latent representations and allocates computation according to token difficulty. Tokens ready to commit are fed back as discrete context, while unresolved tokens retain and further refine their latent states through additional recurrent passes. To learn token prediction and computation allocation jointly, we formulate token-wise computation schedules as latent variables and derive a conditional negative evidence lower bound (NELBO). We train ALoDLM at 1.7B and 8B parameter scales. Across eleven benchmarks, ALoDLM outperforms all evaluated DLMs and the corresponding AR baselines in average benchmark score at both scales. ALoDLM also retains fast parallel decoding, yielding a strong quality-efficiency trade-off among evaluated autoregressive and diffusion models under optimized inference engines.
Figures & tables
Figure 1: ALoDLM combines strong performance with fast inference. Left: ALoDLM -8B outperforms all baselines on average across 11 reasoning, coding, and knowledge benchmarks. Right: On GSM8K, ALoDLM -8B delivers ∼2.7× the throughput of vLLM-served Qwen3-8B at comparable or higher accuracy, while outperforming the diffusion baselines shown in both accuracy and throughput.
Figure 2: Overview of ALoDLM . Standard DLMs apply a fixed-depth denoiser at each denoising step. ALoDLM introduces token-adaptive recurrence within each step: confident tokens commit early and provide discrete context, while unresolved tokens retain and refine their latent states through additional recurrent passes.
Algorithm 1 ALoDLM decoding
Benchmark
AR Baseline Instruct Model
DLM Baseline Instruct Model
Ours Instruct Model
AR Baseline Instruct Model
DLM Baseline Instruct Model
Ours Instruct Model
Qwen3 1.7B
SDAR 1.7B
ALoDLM 1.7B
Qwen3 8B
LLaDA 8B
Dream 7B
Fast-dLLM-v2 7B
SDAR 8B
WeDLM 8B
ALoDLM 8B
General Reasoning
ARC-C
82.4
74.9
77.7
93.9
85.6
84.3
77.2
90.0
91.9
94.4
ARC-E
90.0
86.5
88.7
96.1
92.6
93.0
83.4
93.4
97.5
98.1
MMLU
60.3
63.4
59.4
76.6
62.4
68.4
66.7
78.5
78.0
76.6
MMLU-Pro
39.0
37.0
39.6
56.8
35.6
42.0
40.6
56.3
58.3
63.9
Table 1: Main results. We compare ALoDLM against its corresponding AR baseline and recent diffusion language models (DLMs). Within each model scale, the overall best results are in bold , and the best non-AR results are underlined . Columns corresponding to our model are shaded in blue.
Figure 3: (a) Complementary roles of the entropy and halting gates. Increasing τ relaxes the commitment criterion and accelerates generation, with sharper accuracy losses at high thresholds (left). Increasing q permits deeper latent refinement, also yielding accuracy gains at lower throughput (right). These gates collectively balance parallel token commitment with adaptive refinement of unresolved tokens. (b) Adaptive recurrence improves the quality–efficiency frontier. Across most of the evaluated range, ALoDLM -8B achieves higher accuracy than WeDLM-8B at comparable throughput (left) or compute per generated token (right). Solid lines connect frontier configurations.
Figure 4: Left: Learning dynamics. After the initial training phase, deeper recurrent readouts exhibit lower training loss on their active tokens. Middle: Test-time scaling with q . Increasing q raises loops-per-token and the average score across eleven benchmarks. Right: Token-adaptive recurrence. Mean first-pass halting probabilities differ across token categories, with numerical tokens showing the lowest mean. These preferences are learned without explicit token-difficulty labels or predefined halting targets.
Figure 5: Left: At 8B, K=4 outperforms K=8 early in training and achieves comparable performance later, while K=2 plateaus at a lower score. The inset enlarges steps 10,000–17,000. Middle: Middle-layer recurrence outperforms last-layer recurrence later in training. The full-stack curve is a historical reference evaluated on a different benchmark subset. Right: Intermediate supervision reduces conditional gradient variance in ALoDLM -8B. Variance across sampled exit trajectories is normalized to the value with intermediate supervision at each checkpoint. Error bars show paired 95% bootstrap confidence intervals across eight fixed input packs.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
ALoDLM -1.7B
ALoDLM -8B
Transformer layers
28
36
Prelude layers
None
[0,10)
Recurrent core
[0,28)
[10,26)
Coda layers
None
[26,36)
Maximum recurrent depth
4
4
Packed sequence length
4,096
4,096
Appendix
Table 2: Model configurations and common optimization settings. A batch contains 256 packed examples across all workers and accumulation steps.
Figure 6: Token-wise recurrent depth in three GSM8K responses generated by ALoDLM . Each token is shaded by its recorded commitment pass; darker shades indicate later commitment within the corresponding denoising step. These examples use K=4 , q=0.5 , τ=0.4 , and a 16-token decoding window.
Diffusion Language Models (DLMs) generate text by iteratively denoising a masked sequence, independently predicting multiple tokens at each step. This conditional independence discards inter-token dependencies and degrades coherence-an issue that parallels the multi-modality problem in Non-Autoregressive Translation (NAT). Drawing on the Directed Acyclic Transformer (DAT), which tackles this problem in NAT via a Directed Acyclic Graph (DAG), we propose DA-DLM, a model that adapts DAG-based dependency modeling to DLMs' iterative setting through a position-oriented DAG design. The position-oriented DAG binds node groups to fixed output positions so that tokens fixed in earlier steps anchor neighboring predictions via learned transitions, and evolves with denoising to focus on remaining uncertainty as anchors accumulate. On language modeling, open-ended generation, and summarization, DA-DLM consistently outperforms Block Diffusion, especially under fewer denoising steps, and matches autoregressive models while preserving the parallel generation advantage. Our code is publicly available at https://github.com/jipy0222/DA-DLM.
Pengyu Ji, Zichen Zhang, Xiang Hu +1
‡School of Information Science and Technology, ShanghaiTech University · §Shanghai Engineering Research Center of Intelligent Vision and Imaging · ¶Tencent
Diffusion language models promise parallel generation, yet still lag behind autoregressive (AR) models in quality. We stem this gap to a failure of introspective consistency: AR models agree with their own generations, while DLMs often do not. We define the introspective acceptance rate, which measures whether a model accepts its previously generated tokens. This reveals why AR training has a structural advantage: causal masking and logit shifting implicitly enforce introspective consistency. Motivated by this observation, we introduce Introspective Diffusion Language Model (I-DLM), a paradigm that retains diffusion-style parallel decoding while inheriting the introspective consistency of AR training. I-DLM uses a novel introspective strided decoding (ISD) algorithm, which enables the model to verify previously generated tokens while advancing new ones in the same forward pass. From a systems standpoint, we build I-DLM inference engine on AR-inherited optimizations and further customize it with a stationary-batch scheduler. To the best of our knowledge, I-DLM is the first DLM to match the quality of its same-scale AR counterpart while outperforming prior DLMs in both model quality and practical serving efficiency across 15 benchmarks. It reaches 69.6 on AIME-24 and 45.7 on LiveCodeBench-v6, exceeding LLaDA-2.1-mini (16B) by more than 26 and 15 points, respectively. Beyond quality, I-DLM is designed for the growing demand of large-concurrency serving, delivering about 3x higher throughput than prior state-of-the-art DLMs.
Yifan Yu, Yuqing Jian, Junxiong Wang +12
Together AI · University of Illinois Urbana-Champaign · Princeton University +2
Discrete diffusion language models can generate multiple tokens in parallel, but reducing the number of denoising steps can lead to inconsistent predictions. Standard cross-entropy training fits conditional token marginals, whereas parallel generation requires consistent joint predictions. We introduce Alpha Diffusion Language Models (AlphaDLM), trained with a sequence-level alpha loss that recovers cross-entropy in the limit of vanishing alpha and has a joint-mode optimum at alpha one. Our analysis characterizes how the objective and factorization jointly determine the fitted distribution. We identify conditions under which intermediate alpha preserves multiple valid completions while excluding invalid token combinations. Trained on TinyGSM, our method achieves 34.6% accuracy on GSM8K with only four model evaluations. We further scale the method to SDAR-1.7B and evaluate it on code and mathematics benchmarks. These results show that changing the training objective can improve the accuracy-computation trade-off of factorized diffusion language models.
Nikita Gushchin, Dmitry Baranchuk, Alexander Korotin
Applied AI Institute, Moscow, Russia · AXXX, Russia · Yandex Research