Prefix-Denoising Consistency: Test-Time Verification for Diffusion Language Models

Yuki Ichihara; Naoto Iwase; Mohammad Atif Quamar; Junpei Komiyama

cs.LGAug 26, 2026

Prefix-Denoising Consistency: Test-Time Verification for Diffusion Language Models

Authors: Yuki Ichihara, Naoto Iwase, Mohammad Atif Quamar, Junpei Komiyama

Organizations: MBZUAI · Nagoya University · RIKEN AIP

Abstract

Diffusion Language Models (DLMs) have recently become increasingly competitive with autoregressive (AR) models, and even outperform them on certain tasks. Unlike AR models, DLMs produce output through iterative denoising without a left-to-right order. To further improve the performance of DLMs, we introduce PDC (Prefix-Denoising Consistency), a test-time self-verification method for DLMs. PDC exploits a distinctive test-time signal in DLMs under prefix conditioned regeneration, correct trajectories are more stable and reproducible than incorrect ones. Concretely, given an initially generated sample, PDC splits the sentence at an intermediate position and regenerates the remaining tokens conditioned on the fixed prefix. Across mathematical reasoning and commonsense reasoning benchmarks, PDC consistently improves upon the initial sample, outperforms independent generations under a computational constrained comparison, and is robust to different unmasking strategies and parameter settings. These results highlight prefix-conditioned regeneration as an effective DLM-specific primitive for test-time verification.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of Prefix-Denoising Consistency ( PDC ). Given an initial Diffusion Language Model output, PDC keeps exact prefixes at rates $\rho\in\{0.1,0.5,0.9\}$ , remasks the remaining positions within the same length- $L$ output, and regenerates them to obtain candidate answers. The final answer is selected by majority vote over the regenerated answers only; the initial answer is recorded for analysis but excluded from the vote.

Panel of Figure 2 — Figure 2: Reproduction rates under $\rho=0.5$ regeneration. Green denotes correct-answer preservation $R^{+}_{0.5}$ , while red denotes same-wrong-answer preservation $R^{-}_{0.5}$ , across generation lengths $L=128,256,512$ on three math benchmarks. The gap $R^{+}_{0.5}-R^{-}_{0.5}>0$ shows that correct answers are more reproducible than wrong answers. Full results of reproduction rates, including CSQA and SQA, are reported in Appendix Table 4 .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Method / Length	GSM8K			MATH-500			SVAMP			CSQA			SQA
Method / Length	128	256	512	128	256	512	128	256	512	128	256	512	128	256	512
Dream-7B
Init baseline	64.06	80.06	82.03	35.80	43.20	46.00	83.00	87.00	86.33	73.22	71.74	72.40	70.45	68.56	67.98
+ TiF Fixed	62.40	79.61	81.96	34.20	43.00	46.00	84.00	87.00	86.33	73.79	71.66	72.48	70.74	69.29	68.56
+ TiF Linear	64.52	80.14	82.03	34.80	43.40	46.00	83.67	87.00	86.33	73.96	71.66	72.48	70.74	69.29	68.56
+ TiF Exp. $\alpha=5$	65.81	80.36	82.03	35.60	44.00	46.20	84.00	87.00	86.33	73.55	71.74	72.48	70.74	69.29	68.56

Appendix

Table 5: Full accuracy results for Dream and LLaDA-family diffusion language models. PDC denotes majority voting over the three suffix-regenerated completions obtained with 0.1, 0.5, and 0.9 keep rates. Signed green/red values report accuracy-point changes relative to the Init baseline. For TiF , the reported change is averaged over Fixed, Linear, and Exp.

\alpha=5

variants. The table includes all completed lengths in the current full-diffusion sweep.

Method / Length	GSM8K			MATH-500			SVAMP			CSQA
Method / Length	128	256	512	128	256	512	128	256	512	128	256	512
LLaDA-8B
Init baseline	72.25	74.91	81.80	32.20	35.27	43.15	86.00	87.00	88.67	76.58	76.99	77.07
+ TiF Fixed	71.87	69.37	80.14	29.60	31.06	39.31	84.67	86.67	88.33	78.79	77.07	77.40
+ TiF Linear	73.54	74.75	81.65	30.80	34.67	41.53	87.67	86.33	88.33	77.31	77.07	77.40
+ TiF Exp. $\alpha=5$	73.62	76.27	81.96	31.80	35.27	42.54	88.00	86.67	88.33	76.99	77.31	77.40

Appendix

Table 7: Semi-AR LLaDA-family main accuracy results. These runs use block length 32 with generation lengths 128, 256, and 512.

Panel of Table 8 — Table 8: Two examples in which the wrong answer is corrected by regeneration. Red text highlights the incorrect continuation in the initial sample; green text highlights the corrected continuation produced by regeneration. When the wrong intermediate step is remasked, prefix-conditioned regeneration avoids the same mistake.

Diffusion Language Models

Diffusion language models generate and refine text through iterative denoising, enabling bidirectional context and parallel token updates.

See all 473 papers

Reducing Pretraining-Generation Mismatch in Diffusion Language Models
Aug 10, 2026Xiaocheng Lu, Huabin Liu, Song Guo +1Language Model Pretraining Diffusion Language Models
Inclusion AI · The Hong Kong University of Science and Technology
Introspective Diffusion Language Models
Apr 13, 2026Yifan Yu, Yuqing Jian, Junxiong Wang +12Diffusion Language Models Diffusion Language Model Decoding
Together AI · University of Illinois Urbana-Champaign · Princeton University +2
Learnability-Informed Fine-Tuning of Diffusion Language Models
May 21, 2026Shubham Parashar, Atharv Chagi, Jacob Helwig +5Supervised Fine-Tuning Diffusion Language Models
Department of Computer Science and Engineering, Texas A&M University, College Station, TX, USA · Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX, USA

Aug 10, 2026J/K move · Enter open · S save

Reducing Pretraining-Generation Mismatch in Diffusion Language Models

Xiaocheng Lu, Huabin Liu, Song Guo, Jianguo Li

Inclusion AI · The Hong Kong University of Science and Technology

Language Model Pretraining Diffusion Language Models

Open paper AskPDF arXiv

Diffusion language models (dLLMs) generate text through iterative denoising, allowing multiple tokens to be predicted in parallel. However, pretraining may mask tokens throughout a sequence, whereas prompt continuation conditions on an intact prefix. This difference remains in conversion pipelines that denoise entire sequences during the stable stage. We propose Prefix-Conditioned Diffusion (PCD), which samples a boundary, preserves the prefix, and denoises the suffix. The training recipe also applies autoregressive supervision to the prefix. We evaluate PCD in the stable stage of a warmup, stable, and decay conversion pipeline, with inference unchanged. Matched experiments across model families show improvements in reasoning and coding over native diffusion training. A matched continuation study further shows that the advantage persists after a shared decay stage. In a separate reconstruction diagnostic, the full PCD recipe's advantage over native diffusion training reverses as more evaluation prefix tokens are masked. Controlled experiments show lower suffix reconstruction loss with a clean training prefix, an intact evaluation prefix, and no autoregressive loss.

Abstract

Diffusion Language Models

Explore similar work

Reducing Pretraining-Generation Mismatch in Diffusion Language Models

Introspective Diffusion Language Models

Learnability-Informed Fine-Tuning of Diffusion Language Models