Language Model Generation Evaluation

Follow this topic Ask about this topic

The field studies how to tell whether generated text is correct, useful, faithful to its sources, well written, and appropriate for a task. Approaches range from lexical and embedding-based metrics to human ratings, rubric-based LLM judges, learned evaluators, and task-specific checks such as code execution or schema validation. Benchmarks increasingly test long-form and open-ended outputs, multiple samples, languages, and settings, while evaluation-cost methods aim to select informative examples efficiently. A central challenge is that scores can depend on prompts, judges, and task context: a fluent or highly rated answer may still omit facts, repeat familiar ideas, or fail to reflect human preferences.

Latest papers

396 papers56 papers in the last four weeks, up 17% on the four weeks before. 0.3% of all new papers.

Correct Looks Better: Pairwise Comparisons Reveal Accuracy Rankings
Jun 8, 2026Mina Remeli, Moritz HardtPairwise Comparison LLM-as-a-Judge
Max Planck Institute for Intelligent Systems, Tübingen, Germany, Tübingen AI Center
Can LLMs understand LilyPond? A benchmark for symbolic music generation and understanding
Jun 7, 2026Matteo Spanio, Mohammad Torabi, Andrea Poltronieri +1Music Generation Evaluation LLM Evaluation
Centro di Sonologia Computazionale, University of Padova, Padova, Italy · Music Technology Group, Universitat Pompeu Fabra, Barcelona, Spain
Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs New Metrics
Jun 7, 2026Antonio Franca, Alexander TongNon-Autoregressive Text Generation LLM Evaluation
AITHYRA, Vienna, Austria
SLMJury: Can Small Language Models Judge as Well as Large Ones?
Jun 5, 2026Anish Laddha, Nitesh Pradhan, Gaurav SrivastavaLLM-as-a-Judge Language Model Generation Evaluation
Department of Computer Science and Engineering, LNMIIT, Jaipur, India · Department of Computer Science, Virginia Tech, Blacksburg, VA, USA
Cutting LLM Evaluation Costs with SySRs: A Bandit Algorithm that Provably Exploits Model Similarity
Jun 5, 2026Zifan Lyu, Chahine Nejma, Tobias Wegel +2Model Selection Multi-Armed Bandits
1ETH Zurich · 2Centrale Sup´elec · 3ENS de Cachan +1
Explain Like I'm 5 or Whatever I Choose: Evaluating the Potential for Interactive Complexity in Language Model Responses
Jun 5, 2026Indu Panigrahi, Tal AugustLLM Evaluation Language Model Generation Evaluation
Siebel School of Computing and Data Science University of Illinois Urbana-Champaign
UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs
Jun 4, 2026Amirhossein Abaskohi, Amirhossein Dabiriaghdam, Liang Luo +4LLM Evaluation Language Model Generation Evaluation
University of British Columbia · 2Independent Researcher
Asuka-Bench: Benchmarking Code Agents on Underspecified User Intent and Multi-Round Refinement
Jun 4, 2026Xin Wang, Liangtai Sun, Yaoming Zhu +8Code Generation Language Model Generation Evaluation
University of Science and Technology of China · 2Independent researchers
Elmes*: Automated Construction of Fine-Grained Evaluation Rubrics for Large Language Models in Long-Tail Educational Scenarios
Jun 4, 2026Tao Liu, Ye Lu, Ruohua Zhang +4Automatic Rubric Generation LLM Evaluation
1Shanghai Institute of AI for Education, East China Normal University, Shanghai 200062, China · School of Computer Science and Technology, East China Normal University, Shanghai 200062, China · 3Shanghai Innovation Institute, Shanghai 200231, China
Self-Evaluation Is Already There: Eliciting Latent Judge Calibration in Base LLMs with Minimal Data
Jun 3, 2026XiuYu Zhang, Yi Shan, Junfeng Fang +1LLM Evaluation LLM-as-a-Judge
National University of Singapore · Beijing University of Technology
Benchmarking LLM-as-a-Judge for Long-Form Output Evaluation
Jun 1, 2026Junjie Chen, Yuxi Dong, Haitao Li +5LLM Evaluation LLM-as-a-Judge
Department of Computer Science and Technology, Tsinghua University · University of Science and Technology Beijing
Child-directed speech facilitates production, not comprehension, in BabyLMs
May 31, 2026Bastian Bunzeck, Sina ZarrießLanguage Model Pretraining LLM Evaluation
Computational Linguistics, Department of Linguistics Bielefeld University, Germany
How Much Can Repeated Running Buy? A Measured Variance Ceiling for LLM Code Evaluation
May 30, 2026Yongxi Zhou, Lai Yun Choi, Jiaxi Wen +2LLM Evaluation Language Model Generation Evaluation
Northeastern University, Massachusetts, USA · University of Southern California, California, USA
BenHalluEval: A Multi-Task Hallucination Evaluation Framework for Large Language Models on Bengali
May 29, 2026Shefayat E Shams Adib, Ahmed Alfey Sani, Ekramul Alam Esham +3LLM Reliability Multilingual Language Model Evaluation
Department of Computer Science and Engineering, Islamic University of Technology · Department of Computer Science and Engineering, University of California
TSM-Bench: Detecting LLM-Generated Text in Real-World Wikipedia Editing Practices
May 29, 2026Gerrit Quaremba, Elizabeth Black, Denny Vrandečić +1AI-Generated Text Detection Language Model Generation Evaluation
1King’s College London · 2Wikimedia Foundation
Fine-Tuning Improves Information Conveyance in Language Models
May 29, 2026Yuwei Cheng, Weiyi Tian, Haifeng XuLanguage Model Generation Evaluation LLM Fine-Tuning
Department of Data Science University of Chicago Chicago, IL 60637 · Department of Computer Science University of Chicago Chicago, IL 60637
Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge
May 28, 2026Zijie Wang, Eduardo BlancoAutomatic Rubric Generation LLM-as-a-Judge
University of Arizona Department of Computer Science
SEAL: Can Saturated Benchmarks Be Revived by LLM-as-a-Meta-Judge?
May 28, 2026Jiamin Chen, Yidi Wu, Qiexiang Wang +6LLM Evaluation LLM-as-a-Judge
1ByteDance Inc. · 2City University of Hong Kong
CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Models
May 28, 2026Vedant PadwalLanguage Model Generation Evaluation Code Generation
Independent
ExCAM: Explainable Cultural Awareness Metrics
May 28, 2026Christoph Leiter, Haiyue Song, Hour Kaing +4LLM Evaluation Language Model Generation Evaluation
University of Mannheim, Germany · National Institute of Information and Communications Technology · University of Technology Nuremberg, Germany
Personalized Turn-Level User Conversation Satisfaction Benchmark
May 28, 2026Zhefan Wang, Zhiqiang Guo, Weizhi Ma +3Human Preference Evaluation LLM Personalization
Department of Computer Science and Technology, Tsinghua University, Beijing, China. · Institute for AI Industry Research, Tsinghua University, Beijing, China. · Meituan.
TRACE: Toulmin-based Reasoning Assessment through Constructive Elements for LLM CoT Evaluation
May 28, 2026Yundong Kim, Heyoung YangStructured Reasoning Language Model Generation Evaluation
Applied Agent Research Center, Korea Institute of Science and Technology Information (KISTI), Republic of Korea · Department of Computer Science and Engineering, University of Seoul, Republic of Korea.
Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study
May 27, 2026Irune Zubiaga, Aitor Soroa, Rodrigo AgerriMultilingual Language Model Evaluation LLM-as-a-Judge
HiTZ Center - Ixa, University of the Basque Country EHU
Entropy Distribution as a Fingerprint for Hallucinations in Generative Models
May 27, 2026Mattia J. Villani, Pranav Deshpande, Akshay Seshadri +2Language Model Error Detection Language Model Generation Evaluation
Global Technology Applied Research, JPMorganChase, New York, NY 10001, USA
Auditing Stance Asymmetry in Generative Explanations
May 27, 2026Jiarui HanLanguage Model Generation Evaluation Social Bias in Language Models
MATCHA: Matching Text via Contrastive Semantic Alignment
May 26, 2026Siran Li, Ece Sena Etoglu, Carsten Eickhoff +1LLM Evaluation Semantic Textual Similarity
University of Tübingen
Plans for Evaluating Structured Generative Search Summaries
May 26, 2026Tetsuya Sakai, Jina Lee, Hanpei Fang +1Language Model Generation Evaluation Summarization Evaluation
Waseda University/Naver Corporation · Tokyo, Japan · Waseda University +2
QUIET: A Multi-Blank Cascaded Story Cloze Benchmark for LLM Creative Generation Capability
May 25, 2026Bo Zou, Chao XuOpen-Ended Generation Creativity Assessment
Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why
May 25, 2026Delip Rao, Chris Callison-BurchInter-Rater Reliability LLM Evaluation
University of Pennsylvania
SomaliBench Eval: Measuring English-to-Somali Refusal Gaps in Open-Weight Language Models
May 25, 2026Khalid Yusuf DahirLLM Safety Benchmarks Multilingual Language Model Evaluation
Independent researcher

Jun 8, 2026J/K move · Enter open · S save

Correct Looks Better: Pairwise Comparisons Reveal Accuracy Rankings

Mina Remeli, Moritz Hardt

Max Planck Institute for Intelligent Systems, Tübingen, Germany, Tübingen AI Center

Pairwise Comparison LLM-as-a-Judge Language Model Generation Evaluation Pairwise Preference Evaluation Pairwise Ranking Automated Evaluation+1

Open paper AskPDF arXiv

Pairwise comparisons combined with aggregation methods like Elo have become central to evaluating generative models, yet concerns remain that they reward superficial stylistic cues or display judge biases. In a more positive turn, we show that model rankings from pairwise comparisons strongly agree with ground-truth-based accuracy rankings when such ground truth is available for comparison. By converting five well-known benchmarks into free-form generative evaluations, we find that Elo rankings achieve a Spearman correlation above 0.9 with accuracy rankings and substantially outperform direct evaluation when the judge is weak. Furthermore, style and judge bias have only minor effects on model rankings, despite most judgments occurring on pairs where both candidate answers are correct (or incorrect). On such pairs, we find that repetition after the final answer (echo) is a causal driver of judge preference.