Diffusion Language Model Benchmarking

Latest papers

13 papers0 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Understanding Evaluation Illusion in Diffusion Large Language Models
Jun 28, 2026Hengxiang Zhang, Jiaxi Ren, Renchunzi Xie +1Prompt Sensitivity LLM Evaluation
Department of Statistics and Data Science, Southern University of Science and Technology · Intelligent Computing Research Center, Great Bay Institute for Advanced Study, Great Bay University
Speculative Refinement: A Hybrid Autoregressive Diffusion Decoding Strategy and Its Behavior Across Benchmarks
Jun 25, 2026Aditi Gupta, Neel Mishra, Kushagra Trivedi +1Masked Diffusion Models Language Model Generation Evaluation
IIIT Hyderabad, India · Microsoft
Diffusion Language Models: An Experimental Analysis
Jun 17, 2026Thomas Bertolani, Davide Bucciarelli, Leonardo Zini +2Diffusion Language Model Inference Diffusion Language Model Decoding
University of Modena and Reggio Emilia, Italy · University of Pisa, Italy
Continuous Language Diffusion as a Decoder-Interface Problem
Jun 7, 2026Zhicheng Du, Lan MaLatent Diffusion Models Diffusion Language Model Decoding
Tsinghua Shenzhen International Graduate School, Tsinghua University
Block-R1: Rethinking the Role of Block Size in Multi-domain Reinforcement Learning for Diffusion Large Language Models
May 12, 2026Yan Jiang, Ruihong Qiu, Zi HuangRL Benchmarks Reinforcement Learning
The University of Queensland
DiffScore: Text Evaluation Beyond Autoregressive Likelihood
May 12, 2026Wen Lai, Yingli Shen, Dingnan Jin +4LLM Evaluation Language Model Generation Evaluation
1Ant Group · 2Tsinghua University · 3Technical University of Munich
TrustLDM: Benchmarking Trustworthiness in Language Diffusion Models
Apr 15, 2026Yichuan Mo, Yukun Jiang, Yanbo Shi +4Diffusion Language Model Benchmarking LLM Safety Benchmarks
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University. · CISPA Helmholtz Center for Information Security · School of EECS, Peking University +1
Generative Frontiers: Why Evaluation Matters for Diffusion Language Models
Apr 3, 2026Patrick Pynadath, Jiaxin Shi, Ruqi ZhangDiffusion Language Model Benchmarking Language Model Generation Evaluation
Department of Computer Science, Purdue University West Lafayette, Indiana, 47906, USA
Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison
Mar 23, 2026Caio VicentinoDiffusion Language Models Autoregressive Language Modeling
Independent Researcher
dLLM: Simple Diffusion Language Modeling
Feb 26, 2026Zhanhui Zhou, Lingjie Chen, Hanghang Tong +1Diffusion Language Model Training Diffusion Language Model Benchmarking
UIUC
Is Your Diffusion Sampler Actually Correct? A Sampler-Centric Evaluation of Discrete Diffusion Language Models
Feb 23, 2026Luhan Tang, Longxuan Yu, Shaorong Zhang +1Diffusion Language Model Benchmarking Diffusion Sampling
University of California, Riverside
Scaling Beyond Masked Diffusion Language Models
Feb 16, 2026Subham Sekhar Sahoo, Jean-Marie Lemercier, Zhihan Yang +4Diffusion Language Models Diffusion Models
NVIDIA, Santa Clara, USA · Department of Computer Science, Cornell Tech, NYC, USA · Department of Computer Science, Cornell University, Ithaca, USA +2
ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMs
Oct 6, 2025Wonjun Kang, Kevin Galim, Seunghyuk Oh +8Diffusion Language Model Inference Parallel Decoding
FuriosaAI · Seoul National University · UW-Madison +4

Jun 28, 2026J/K move · Enter open · S save

Understanding Evaluation Illusion in Diffusion Large Language Models

Hengxiang Zhang, Jiaxi Ren, Renchunzi Xie, Hongxin Wei

Department of Statistics and Data Science, Southern University of Science and Technology · Intelligent Computing Research Center, Great Bay Institute for Advanced Study, Great Bay University

Prompt Sensitivity LLM Evaluation Language Model Generation Evaluation Diffusion Language Model Decoding Diffusion Language Model Benchmarking

Open paper AskPDF arXiv

Despite the capability of parallel decoding, diffusion large language models (dLLMs) require many denoising steps to maintain generation quality, motivating recent research on efficient decoding strategies. However, existing studies have reported inconsistent evaluation results even under seemingly identical evaluation settings, risking biased conclusions about dLLM decoding methods. To understand this evaluation concern, we conduct a rigorous evaluation of current decoding methods for dLLMs across diverse evaluation settings. Surprisingly, our analysis reveals that the ranking of decoding methods is highly sensitive to the choice of prompt templates. Single-template evaluation can lead to an illusion that decoding methods improve inference efficiency without performance degradation. Through comprehensive experiments, we find that current parallel decoding methods consistently underperform the single-token decoding baseline, failing to overcome the speed-quality trade-off. We further identify this evaluation inconsistency as the high sensitivity of parallel decoding methods to minor variations in prompt templates. Our experiments show that an effective prompt template can achieve strong evaluation results even with fewer denoising steps, markedly outperforming the marginal gain from increasing denoising steps. Beyond prompt templates, our experiments indicate that overlooked evaluation settings can also notably affect the assessment of decoding methods. Based on these findings, we propose practical guidelines for the reliable evaluation of decoding methods in dLLMs.