Language Model Generation Evaluation

Ask about this topic

Overview

The field studies how to tell whether generated text is correct, useful, faithful to its sources, well written, and appropriate for a task. Approaches range from lexical and embedding-based metrics to human ratings, rubric-based LLM judges, learned evaluators, and task-specific checks such as code execution or schema validation. Benchmarks increasingly test long-form and open-ended outputs, multiple samples, languages, and settings, while evaluation-cost methods aim to select informative examples efficiently. A central challenge is that scores can depend on prompts, judges, and task context: a fluent or highly rated answer may still omit facts, repeat familiar ideas, or fail to reflect human preferences.

Latest papers

CardsList

396 papers56 papers in the last four weeks, up 17% on the four weeks before. 0.3% of all new papers.

  1. Atomic and Holistic LLM Judges for Reference-Grounded Support Labels: A Prompt-Controlled Comparison

    Mar 30, 2026Xinran ZhangLLM-as-a-JudgeLanguage Model Generation Evaluation

    University of California, Berkeley

  2. ViviDoc: Generating Interactive Documents through Human-Agent Collaboration

    Mar 30, 2026Yinghao Tang, Yupeng Xie, Yingchaojie Feng +4Human-AI Co-CreationLanguage Model Generation Evaluation

    State Key Lab of CAD&CG, Zhejiang University · HKUST(GZ) · National University of Singapore +2

  3. When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models

    Mar 27, 2026Juan Gabriel Kostelec, Qinghai GuoLLM Inference EfficiencyLLM Evaluation

    Huawei Switzerland · ACS Lab, Huawei Technologies

  4. ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs

    Mar 27, 2026Inês Vieira, Inês Calvo, Iago Paulo +6LLM EvaluationLanguage Model Generation Evaluation

    1NOVA University of Lisbon, Portugal · 2NOVA LINCS

  5. Pingala: Prosody-Aware Decoding for Sanskrit Poetry Generation

    Mar 25, 2026Manoj Balaji Jagadeeshan, Atul Singh, Nallani Chakravartula Sahith +2Text GenerationLanguage Model Decoding

    Indian Institute of Technology, Kharagpur · Oriflow

  6. Decoding AI Authorship: Can LLMs Truly Mimic Human Style Across Literature and Politics?

    Mar 24, 2026Nasser A AlsadhanLanguage Model Generation EvaluationAI-Generated Text Detection

    College of Computer and Information Sciences, King Saud University, Riyadh, Saudi Arabia

  7. Generating and Evaluating Sustainable Procurement Criteria for the Swiss Public Sector using In-Context Prompting with Large Language Models

    Mar 23, 2026Yingqiang Gao, Veton Matoshi, Luca Rolshoven +5Language Model Generation Evaluation

    University of Zurich Zurich, Switzerland · Bern University of Applied Sciences Bern, Switzerland · University of Bern Bern, Switzerland

  8. Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison

    Mar 23, 2026Caio VicentinoDiffusion Language ModelsAutoregressive Language Modeling

    Independent Researcher

  9. DATASHI: A Parallel English-Tashlhiyt Corpus for Orthography Normalization and Low-Resource Language Processing

    Mar 23, 2026Nasser-Eddine Monir, Zakaria BaouLow-Resource Language ProcessingLanguage Model Generation Evaluation

    Université de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France · ISIMA, Université Clermont Auvergne, 63178 Aubière, France

  10. LLM-Based Intelligent Notification Composition: From Static Personalization to Context-Aware Persuasive Messaging

    Mar 22, 2026Nilesh AgrawalPersonalized Text GenerationLLM Personalization

    Independent Researcher, Seattle, WA, USA

  11. Semantic Delta: An Interpretable Signal Differentiating Human and LLMs Dialogue

    Mar 20, 2026Riccardo Scantamburlo, Mauro Mezzanzana, Giacomo Buonanno +1AI-Generated Text DetectionLanguage Model Generation Evaluation

    School of Industrial Engineering Intelligence, Complexity and Technology Lab (ICT Lab) LIUC - Università Cattaneo Castellanza (Italy), 21053

  12. Evaluating 5W3H Structured Prompting for Intent Alignment in Human-AI Interaction

    Mar 19, 2026Peng GangStructured PromptingLLM Prompting

    Huizhou Lateni AI Technology Co., Ltd., Huizhou, China · Huizhou University, Huizhou, China

  13. Modeling the human lexicon under temperature variations: linguistic factors, diversity and typicality in LLM word associations

    Mar 18, 2026Maria Andueza Rodriguez, Marie Candito, Richard HuygheLanguage Model Generation Evaluation

    University of Fribourg · Lucerne University of Applied Sciences and Arts · LLF (Université Paris Cité / CNRS)

  14. Text-to-Stage: Spatial Layouts from Long-form Narratives

    Mar 18, 2026Jefferson Hernandez, Swarnadeep Saha, Chenxi Whitehouse +6Spatial Reasoning in Language ModelsSpatial Reasoning

    Rice University · Meta FAIR · Meta Reality Labs Research

  15. When Stability Fails: Hidden Failure Modes Of LLMS in Data-Constrained Scientific Decision-Making

    Mar 16, 2026Nazia RiasatLLM EvaluationLLM Reliability

    North Dakota State University

  16. Writer-R1: Dynamic Rubric Mining for Reinforcement Learning of Open-Ended Writing

    Mar 16, 2026Jihao Zhao, Shuaishuai Zu, Zhiyuan Ji +2Memory-Augmented RLRL for Language Models

    School of Information, Renmin University of China, Beijing, China · School of Information, BRAIN, Renmin University of China, Beijing, China

  17. Creative Convergence or Imitation? Genre-Specific Homogeneity in LLM-Generated Chinese Literature

    Mar 15, 2026Yuanchi Ma, Kaize Shi, Hui He +5Text GenerationComputational Literary Studies

  18. The Chronicles of RiDiC: Generating Datasets with Controlled Popularity Distribution for Long-form Factuality Evaluation

    Mar 11, 2026Pavel Braslavski, Dmitrii Iarosh, Nikita Sushko +4Language Model Generation EvaluationFactual Knowledge in Language Models

    HSE University · Ural Federal University · ITMO University +3

  19. Generating Hierarchical JSON Representations of Scientific Sentences Using LLMs

    Mar 8, 2026Satya Sri Rajiteswari Nimmagadda, Ethan Young, Niladri Sengupta +2Structured Output GenerationHierarchical Representation Learning

    Marshall University · Fractal Analytics Inc · West Virginia State University

  20. Moral Mazes in the Era of LLMs

    Mar 6, 2026Dang Nguyen, Harvey Yiyun Fu, Peter West +2Pragmatic Reasoning in Language ModelsLanguage Model Generation Evaluation

    University of Chicago · University of British Columbia

  21. Building an Ensemble LLM Semantic Tagger for UN Security Council Resolutions

    Mar 6, 2026Hussein GhalyLLM-Assisted AnnotationLanguage Model Ensembles

    Independent Researcher, Rego Park, New York

  22. AI-Assisted Moot Courts: Simulating Justice-Specific Questioning in Oral Arguments

    Mar 5, 2026Kylie Zhang, Nimra Nadeem, Lucia Zheng +2Large Language Model-Based Role-Play SimulationLanguage Model Generation Evaluation

    Princeton University, Princeton, NJ, USA · Stanford University, Stanford, USA

  23. The Arrival of AGI? When Expert Personas Exceed Expert Benchmarks

    Mar 4, 2026Drake Mullens, Stella ShenLLM PromptingLanguage Model Generation Evaluation

    Tarleton State University · University of Texas at Tyler

  24. FINEST: Improving LLM Responses to Sensitive Topics Through Fine-Grained Evaluation

    Mar 4, 2026Juhyun Oh, Nayeon Lee, Chani Jung +5Language Model Generation EvaluationLanguage Model Safety Evaluation

    School of Computing, KAIST, Daejeon, Republic of Korea · Samsung, Suwon, Republic of Korea

  25. A Multi-Dimensional Quality Scoring Framework for Decentralized LLM Inference with Proof of Quality

    Mar 4, 2026Arther Tian, Alex Ding, Frank Chen +2Language Model Generation EvaluationLLM Evaluation

    DGrid AI

  26. Markovian ODE-guided scoring can assess the quality of offline reasoning traces in language models

    Mar 2, 2026Arghodeep Nandi, Ojasva Saxena, Tanmoy ChakrabortyOrdinary Differential EquationsLanguage Model Reasoning Evaluation

    Department of Electrical Engineering, Indian Institute of Technology Delhi, New Delhi, India · Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi, New Delhi, India · Indian Institute of Technology Delhi, Abu Dhabi, UAE

  27. CollabEval: Enhancing LLM-as-a-Judge via Multi-Agent Collaboration

    Mar 1, 2026Yiyue Qian, Shinan Zhang, Yun Zhou +3LLM EvaluationLLM-as-a-Judge

    Amazon AWS Generative AI Innovation Center · Amazon AWS Bedrock

  28. RAVEL: Reasoning Agents for Validating and Evaluating LLM Text Synthesis

    Feb 28, 2026Andrew Zhuoer Feng, Cunxiang Wang, Yu Luo +9Language Model Generation EvaluationLLM Agent Evaluation

    Tsinghua University · Peking University

  29. How Large Language Models Get Stuck: Early structure with persistent errors

    Feb 27, 2026Alokesh Manna, William Snyder, Whitney TaborLanguage ModelingLLM Training

    Department of Statistics University of Connecticut, Storrs, USA · Department of Linguistics University of Connecticut, Storrs, USA · Department of Psychological Sciences University of Connecticut, Storrs, USA

  30. Generative Active Testing: Efficient LLM Evaluation via Proxy Task Adaptation

    Feb 26, 2026Aashish Anantha Ramakrishnan, Ardavan Saeedi, Hamid Reza Hassanzadeh +2LLM EvaluationUncertainty Sampling

    The Pennsylvania State University · Optum AI