Language Model Generation Evaluation

Ask about this topic

Overview

The field studies how to tell whether generated text is correct, useful, faithful to its sources, well written, and appropriate for a task. Approaches range from lexical and embedding-based metrics to human ratings, rubric-based LLM judges, learned evaluators, and task-specific checks such as code execution or schema validation. Benchmarks increasingly test long-form and open-ended outputs, multiple samples, languages, and settings, while evaluation-cost methods aim to select informative examples efficiently. A central challenge is that scores can depend on prompts, judges, and task context: a fluent or highly rated answer may still omit facts, repeat familiar ideas, or fail to reflect human preferences.

Latest papers

CardsList

396 papers56 papers in the last four weeks, up 17% on the four weeks before. 0.3% of all new papers.

  1. A multilingual hallucination benchmark: MultiWikiQHalluA

    May 4, 2026Freja Thoresen, Dan Saattrup SmartMultilingual Language ModelsMultilingual Language Model Evaluation

    Alexandra Institute · Rued Langgaards Vej 7, 2300 København

  2. Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese

    May 2, 2026Roseval Malaquias Junior, Giovana Kerche Bonás, Thales Sales Almeida +6LLM EvaluationLLM-as-a-Judge

    Maritaca AI · Jusbrasil

  3. The Partial Testimony of Logs: Evaluation of Language Model Generation under Confounded Model Choice

    May 2, 2026Jikai Jin, Vasilis SyrgkanisLLM EvaluationOff-Policy Evaluation

  4. Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleading

    Apr 30, 2026Nicholas Sadjoli, Tim Siefken, Atin Ghosh +2LLM EvaluationLanguage Model Generation Evaluation

    SAP · Stanford University

  5. From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks

    Apr 30, 2026Qingyu Ren, Tianjun Pan, Xingzhou Chen +1Reward ModelingRL for Language Models

    Shanghai Key Laboratory of Data Science, College of Computer Science and Artificial Intelligence, Fudan University · 2Shanghai Artificial Intelligence Laboratory

  6. Instruction Complexity Induces Positional Collapse in Adversarial LLM Evaluation

    Apr 29, 2026Jon-Paul CacioliLLM EvaluationLanguage Model Generation Evaluation

    Independent Researcher, Melbourne, Australia

  7. Theory-Grounded Evaluation Exposes the Authorship Gap in LLM Personalization

    Apr 29, 2026Yash Ganpat SawantLLM EvaluationLLM Personalization

    Independent AI Researcher

  8. The Surprising Universality of LLM Outputs: A Real-Time Verification Primitive

    Apr 28, 2026Alex Bogdan, Adrian de Valois-FranklinLLM EvaluationLanguage Model Generation Evaluation

    Evolutionairy AI Toronto, Canada

  9. MGTEVAL: An Interactive Platform for Systemtic Evaluation of Machine-Generated Text Detectors

    Apr 28, 2026Yuanfan Li, Qi Zhou, Chengzhengxu Li +5AI-Generated Text DetectionLanguage Model Generation Evaluation

    Faculty of Electronic and Information Engineering, Xi’an Jiaotong University · Queen Mary University of London

  10. LongSumEval: Question-Answering Based Evaluation and Feedback-Driven Refinement for Long Document Summarization

    Apr 28, 2026Huyen Nguyen, Haoxuan Zhang, Yang Zhang +2Language Model Generation EvaluationFactual Consistency Evaluation

    Cigna Group Evernorth Health Services Austin, Texas, USA · dept. of Information Science University of North Texas Denton, Texas, USA · dept. of Data Science University of North Texas Denton, Texas, USA

  11. JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems

    Apr 26, 2026Rohith Reddy Bellibatlu, Edward Raff, Wenbin ZhangLLM EvaluationLLM-as-a-Judge

    Florida International University, Miami, FL · CrowdStrike, New York, USA · University of Maryland Baltimore County, Baltimore, USA

  12. Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines

    Apr 25, 2026Sadman Kabir SoumikDebiased MLLLM-as-a-Judge

    Independent Researcher

  13. BLAST: Benchmarking LLMs with ASP-based Structured Testing

    Apr 24, 2026Manuel Alejandro Borroto Santana, Erica Coppolillo, Francesco Calimeri +3Logic ProgrammingLLM Evaluation

    University of Calabria, Rende, Italy · ICAR-CNR, Rende, Italy · Artificial Intelligence LAB, DeMaCS, University of Calabria, Rende, Italy +1

  14. Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models

    Apr 23, 2026Mohammed Safi Ur Rahman Khan, Sanjay Suryanarayanan, Tushar Anand +1VLM EvaluationVLM Robustness

    ♠Nilekani Centre at AI4Bharat ♢Indian Institute of Technology Madras · ♠Nilekani Centre at AI4Bharat · ♣BITS Pilani, Hyderabad

  15. HoWToBench: Holistic Evaluation for LLM's Capability in Human-level Writing using Tree of Writing

    Apr 21, 2026Andrew Zhuoer Feng, Cunxiang Wang, Yu Luo +7LLM EvaluationLLM-as-a-Judge

    Department of Computer Science and Technology, Tsinghua University

  16. Beyond One Output: Visualizing and Comparing Distributions of Language Model Generations

    Apr 20, 2026Emily Reif, Claire Yang, Jared Hwang +3Open-Ended GenerationData Visualization

    University of Washington, Seattle, WA, USA · Allen Institute for AI

  17. Benchmarking System Dynamics AI Assistants: Cloud Versus Local LLMs on CLD Extraction and Discussion

    Apr 20, 2026Terry LeitchLLM QuantizationDynamical Systems

    Ruxton AI

  18. VIBE: Voice-Induced open-ended Bias Evaluation for Large Audio-Language Models via Real-World Speech

    Apr 19, 2026Yi-Cheng Lin, Yusuke Hirota, Sung-Feng Huang +1Audio-Language Model EvaluationGender Bias in Language Models

    Graduate Institute of Communication Engineering National Taiwan University Taipei, Taiwan · NVIDIA Research NVIDIA Taipei, Taiwan · AI Center of Research Excellence National Taiwan University Taipei, Taiwan

  19. Configuration Over Selection: Hyperparameter Sensitivity Exceeds Model Differences in Open-Source LLMs for RTL Generation

    Apr 18, 2026Minghao Shao, Zeng Wang, Weimin Fu +5LLM EvaluationLanguage Model Generation Evaluation

    NYU Tandon School of Engineering, USA · Kansas State University, USA · NYU Abu Dhabi, UAE

  20. Spotlights and Blindspots: Evaluating Machine-Generated Text Detection

    Apr 17, 2026Kevin Stowe, Kailash PatilAI-Generated Text DetectionLanguage Model Generation Evaluation

    Pindrop

  21. Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations

    Apr 16, 2026Manan Gupta, Dhruv KumarLLM EvaluationLLM-as-a-Judge

    1BITS Pilani, Pilani Campus, India

  22. IUQ: Interrogative Uncertainty Quantification for Long-Form Large Language Model Generation

    Apr 16, 2026Haozhi Fan, Jinhao Duan, Kaidi XuUncertainty QuantificationLanguage Model Generation Evaluation

    University of Pennsylvania · 2UNC Chapel Hill · 3City University of Hong Kong

  23. Interpretable Stylistic Variation in Human and LLM Writing Across Genres, Models, and Decoding Strategies

    Apr 15, 2026Swati Rallapalli, Shannon Gallagher, Ronald Yurko +4StylometryLanguage Model Generation Evaluation

    Software Engineering Institute, AI Division, Carnegie Mellon University · Department of Statistics & Data Science, Carnegie Mellon University

  24. One Token Away from Collapse: The Fragility of Instruction-Tuned Helpfulness

    Apr 14, 2026Erfan Baghaei Potraghloo, Seyedarmin Azizi, Souvik Kundu +1Language Model RobustnessInstruction Tuning

    University of Southern California, Los Angeles, USA · Intel AI, USA

  25. Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Miss

    Apr 14, 2026Ronald Skorobogat, Ameya Prabhu, Matthias BethgeMultilingual Language Model EvaluationLanguage Model Generation Evaluation

    Tübingen AI Center, University of Tübingen

  26. lmfaoooo at SemEval-2026 Task 1: Humor Is an Audience. Preference Modeling for Constrained Humor Generation

    Apr 14, 2026Alexey Tikhonov, Alexey IvanovHumor Explanation GenerationText Generation

    Inworld.AI Berlin, Germany · OpenAI Mountain View, CA

  27. Temporal Flattening in LLM-Generated Text: Comparing Human and LLM Writing Trajectories

    Apr 13, 2026Zhanwei Cao, YeoJin Go, Yifan Hu +1Language Model Generation EvaluationAI-Generated Text Detection

    Northeastern University, Seattle, WA, USA

  28. Filtered Reasoning Score: Evaluating Reasoning Quality on a Model's Most-Confident Traces

    Apr 13, 2026Manas Pathak, Xingyao Chen, Shuozhe Li +2LLM EvaluationLanguage Model Generation Evaluation

    University of Texas at Austin

  29. GoodPoint: Learning Constructive Scientific Paper Feedback from Author Responses

    Apr 13, 2026Jimin Mun, Chani Jung, Xuhui Zhou +2LLM Post-TrainingAutomated Peer Review

    Carnegie Mellon University · Independent Researcher · NVIDIA

  30. Please Make it Sound like Human: Encoder-Decoder vs. Decoder-Only Transformers for AI-to-Human Text Style Transfer

    Apr 13, 2026Utsav PaneruText Style TransferText Generation

    Department of Computer Science, Kathmandu Engineering College, Kalimati, Kathmandu, Nepal