Language Model Generation Evaluation

Follow this topic Ask about this topic

Momentum

30 papers in the last four weeks, up 100% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 209

All topics

Understanding Evaluation Illusion in Diffusion Large Language Models
Jun 28, 2026Hengxiang Zhang, Jiaxi Ren, Renchunzi Xie +1Prompt Sensitivity LLM Evaluation
Department of Statistics and Data Science, Southern University of Science and Technology · Intelligent Computing Research Center, Great Bay Institute for Advanced Study, Great Bay University
Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA
Jun 26, 2026Sambaran BandyopadhyayIn-Context Retrieval LLM Evaluation
Adobe Research
Speculative Refinement: A Hybrid Autoregressive Diffusion Decoding Strategy and Its Behavior Across Benchmarks
Jun 25, 2026Aditi Gupta, Neel Mishra, Kushagra Trivedi +1Masked Diffusion Models Language Model Generation Evaluation
IIIT Hyderabad, India · Microsoft
Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement
Jun 25, 2026Sangwoo Cho, Kushal Chawla, Pengshan Cai +4LLM Evaluation LLM-as-a-Judge
Capital One, AI Foundations, McLean, VA 22102, USA.
The Capability Frontier: Benchmarks Miss 82% of Model Performance
Jun 25, 2026Bradley Fowler, Ryan Smith, Daniel Thi Graviet +8LLM Evaluation Language Model Generation Evaluation
1Martian · 3ThoughtWorks · University of Oxford
DualEval: Joint Model-Item Calibration for Unified LLM Evaluation
Jun 24, 2026Aaron J. Li, Hao Huang, Youngmin Park +6LLM Evaluation Language Model Generation Evaluation
University of California, Berkeley · Arena · University of California, Los Angeles
CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression
Jun 23, 2026Morayo Danielle Adeyemi, Ryan A. Rossi, Franck DernoncourtLLM Evaluation LLM Compression
Independent · Adobe Research
Do LLM Attribution Metrics Transfer? Auditing Retrieval-Augmented Generation Evaluation Across Datasets and Constructs
Jun 22, 2026Tianyu Ding, Aditya Nannapaneni, Juan Pablo De la Cruz WeinsteinLanguage Model Generation Evaluation LLM Auditing
Amazon Web Services
GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation
Jun 22, 2026Jeffrey FlyntAI Agent Reliability Tool-Use Evaluation
Independent Researcher
PRIME: Evaluating Prompt Resolution Under Incompatible Instructions in LLMs
Jun 21, 2026Tehreem Javed, Shumaim Fatimah, Masooma Bakhtiari +2LLM Evaluation Language Model Generation Evaluation
School of Electrical Engineering and Computer Science (SEECS), National University of Sciences and Technology (NUST), Islamabad, Pakistan
BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories
Jun 21, 2026Shreyas KCMultilingual Language Model Evaluation LLM-as-a-Judge
RECOM: A Validity Discrimination Tradeoff in Automatic Metrics for Open Ended Reddit Question Answering
Jun 17, 2026Pushwitha Krishnappa, Amit Das, Vinija Jain +2Open-Ended Generation LLM Evaluation
University of Alabama Huntsville · University of North Alabama · Stanford University +2
How Inference Compute Shapes Frontier LLM Evaluation
Jun 16, 2026Jessica McFadyen, Ole Jorgensen, Harry Coppock +2LLM Evaluation Language Model Generation Evaluation
AI Security Institute, London, UK · University of Oxford, Oxford, UK · Harvard University, Cambridge, MA, USA
Prompt Perturbation for Reliable LLM Evaluation over Comparison Graphs
Jun 16, 2026Dong Huang, Jianbo Sun, Pengkun YangLLM Evaluation Pairwise Comparison
Department of Statistics and Data Science, Tsinghua University
The Benchmark Illusion: Pruned LLMs Can Pass Multiple Choice but Fail to Answer
Jun 16, 2026Rui Wen, Lu Sun, Jiayang Liu +3Open-Ended Generation LLM Evaluation
Institute of Science Tokyo · Tohoku University · Nanyang Technological University +2
NarrativeWorldBench: A Frontier-Saturated Benchmark and a Latent World Model for Long-Horizon Co-Creative Audio Drama
Jun 16, 2026Logan Mann, Abdur Rahman, Mohammad Saifullah +2LLM World Models Multilingual Language Model Evaluation
University of California, Santa Barbara · Pocket FM
Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines
Jun 13, 2026Yitao LiLLM Evaluation LLM-as-a-Judge
Beyond Accuracy: Measuring Bias Acknowledgment in Chain-of-Thought Reasoning for Responsible AI Evaluation
Jun 13, 2026Xian Sun, Wei Gao, Yingshuo Wang +9Language Model Generation Evaluation CoT Evaluation
Duke University · Northeastern University · University of California, Berkeley +5
Metric Match: A Subset Selection Approach to Evaluating LLM Judge Reliability
Jun 12, 2026Alyssa Unell, Natalie Dullerud, Naomi Boneh +4LLM-as-a-Judge Language Model Generation Evaluation
Department of Computer Science Stanford University · Department of Medicine Stanford University
GENIE: A Fine-Grained Measure for Novelty
Jun 11, 2026Ramya Namuduri, Manya Wadhwa, Anshun Asher Zheng +2LLM Evaluation Language Model Generation Evaluation
The University of Texas at Austin · New York University
Correct Looks Better: Pairwise Comparisons Reveal Accuracy Rankings
Jun 8, 2026Mina Remeli, Moritz HardtPairwise Comparison LLM-as-a-Judge
Max Planck Institute for Intelligent Systems, Tübingen, Germany, Tübingen AI Center
Can LLMs understand LilyPond? A benchmark for symbolic music generation and understanding
Jun 7, 2026Matteo Spanio, Mohammad Torabi, Andrea Poltronieri +1Music Generation Evaluation LLM Evaluation
Centro di Sonologia Computazionale, University of Padova, Padova, Italy · Music Technology Group, Universitat Pompeu Fabra, Barcelona, Spain
Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics
Jun 7, 2026Antonio Franca, Alexander TongNon-Autoregressive Text Generation LLM Evaluation
AITHYRA Institute, Vienna, Austria.
SLMJury: Can Small Language Models Judge as Well as Large Ones?
Jun 5, 2026Anish Laddha, Nitesh Pradhan, Gaurav SrivastavaLLM-as-a-Judge Language Model Generation Evaluation
Department of Computer Science and Engineering, LNMIIT, Jaipur, India · Department of Computer Science, Virginia Tech, Blacksburg, VA, USA
Cutting LLM Evaluation Costs with SySRs: A Bandit Algorithm that Provably Exploits Model Similarity
Jun 5, 2026Zifan Lyu, Chahine Nejma, Tobias Wegel +2Model Selection Multi-Armed Bandits
1ETH Zurich · 2Centrale Sup´elec · 3ENS de Cachan +1
Explain Like I'm 5 or Whatever I Choose: Evaluating the Interactive Potential of Language Model Responses
Jun 5, 2026Indu Panigrahi, Tal AugustLLM Evaluation Language Model Generation Evaluation
Siebel School of Computing and Data Science University of Illinois Urbana-Champaign
UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs
Jun 4, 2026Amirhossein Abaskohi, Amirhossein Dabiriaghdam, Liang Luo +4LLM Evaluation Language Model Generation Evaluation
University of British Columbia · 2Independent Researcher
Asuka-Bench: Benchmarking Code Agents on Underspecified User Intent and Multi-Round Refinement
Jun 4, 2026Xin Wang, Liangtai Sun, Yaoming Zhu +8Computer-Use Agent Benchmarks Language Model Generation Evaluation
University of Science and Technology of China · 2Independent researchers
Elmes*: Automated Construction of Fine-Grained Evaluation Rubrics for Large Language Models in Long-Tail Educational Scenarios
Jun 4, 2026Tao Liu, Ye Lu, Ruohua Zhang +4LLM Evaluation LLM-as-a-Judge
1Shanghai Institute of AI for Education, East China Normal University, Shanghai 200062, China · School of Computer Science and Technology, East China Normal University, Shanghai 200062, China · 3Shanghai Innovation Institute, Shanghai 200231, China
Self-Evaluation Is Already There: Eliciting Latent Judge Calibration in Base LLMs with Minimal Data
Jun 3, 2026XiuYu Zhang, Yi Shan, Junfeng Fang +1LLM Evaluation LLM-as-a-Judge
National University of Singapore · Beijing University of Technology

Jun 28, 2026, cs.CLJ/K move · Enter open · S save

Understanding Evaluation Illusion in Diffusion Large Language Models

Hengxiang Zhang, Jiaxi Ren, Renchunzi Xie, Hongxin Wei

Department of Statistics and Data Science, Southern University of Science and Technology · Intelligent Computing Research Center, Great Bay Institute for Advanced Study, Great Bay University

Prompt Sensitivity LLM Evaluation Language Model Generation Evaluation Diffusion Language Model Decoding

Open paper AskPDF arXiv

Despite the capability of parallel decoding, diffusion large language models (dLLMs) require many denoising steps to maintain generation quality, motivating recent research on efficient decoding strategies. However, existing studies have reported inconsistent evaluation results even under seemingly identical evaluation settings, risking biased conclusions about dLLM decoding methods. To understand this evaluation concern, we conduct a rigorous evaluation of current decoding methods for dLLMs across diverse evaluation settings. Surprisingly, our analysis reveals that the ranking of decoding methods is highly sensitive to the choice of prompt templates. Single-template evaluation can lead to an illusion that decoding methods improve inference efficiency without performance degradation. Through comprehensive experiments, we find that current parallel decoding methods consistently underperform the single-token decoding baseline, failing to overcome the speed-quality trade-off. We further identify this evaluation inconsistency as the high sensitivity of parallel decoding methods to minor variations in prompt templates. Our experiments show that an effective prompt template can achieve strong evaluation results even with fewer denoising steps, markedly outperforming the marginal gain from increasing denoising steps. Beyond prompt templates, our experiments indicate that overlooked evaluation settings can also notably affect the assessment of decoding methods. Based on these findings, we propose practical guidelines for the reliable evaluation of decoding methods in dLLMs.

Momentum

Related topics

Latest papers 209