Language Model Generation Evaluation

Ask about this topic

Overview

The field studies how to tell whether generated text is correct, useful, faithful to its sources, well written, and appropriate for a task. Approaches range from lexical and embedding-based metrics to human ratings, rubric-based LLM judges, learned evaluators, and task-specific checks such as code execution or schema validation. Benchmarks increasingly test long-form and open-ended outputs, multiple samples, languages, and settings, while evaluation-cost methods aim to select informative examples efficiently. A central challenge is that scores can depend on prompts, judges, and task context: a fluent or highly rated answer may still omit facts, repeat familiar ideas, or fail to reflect human preferences.

Latest papers

CardsList

396 papers56 papers in the last four weeks, up 17% on the four weeks before. 0.3% of all new papers.

  1. Homebrewed SLMs: Autoregressive and Masked Diffusion Language Models Built at Home for Portuguese

    Oct 29, 2026Pedro Carvalho BromLanguage ModelingSmall Language Models

    Department of Mathematics Federal Institute of Brasília Brasília, Brazil

  2. Offline Language-Based Support System for Tactical Decision-Making in Air Combat Operations

    Oct 26, 2026Joao P. A. DantasAI-Assisted Decision MakingLanguage Model Generation Evaluation

    Decision Support Systems Subdivision Institute for Advanced Studies São José dos Campos, São Paulo, Brazil, 12228-001

  3. Useful Diversity or Hallucinated Variety? Persona-Conditioned Synthetic Query Generation for E-Commerce Search

    Oct 26, 2026Filippos Ventirozos, Tom Armitage, Mahmoud Oshagh +2E-Commerce SearchPersona-Conditioned Generation

    Manchester Metropolitan University, Manchester, UK · Autotrader UK, Manchester, UK

  4. Shorter Is Not Cheaper: A Controlled Comparison of Declarative Animation Specifications and Free-Form Code for LLM-Generated Explanations

    Oct 23, 2026Jiantang HuangStructured Output GenerationCode Generation

  5. DraMaze: A Multi-Agent Pipeline with Future-Aware Memory for Diverse Microdrama Script Generation

    Oct 22, 2026Siyang He, Qiqi Wang, Fan Liu +5Narrative GenerationText Generation

    Fudan University · Shanghai Innovation Institute · Solar System of OVB, Tencent, China

  6. An Empirical Study of Recent Large Language Models for Knowledge Graph to Text Generation on Standard Benchmarks

    Oct 22, 2026Priyanka, Renaud Delbru, Souhail Bakkali +2Knowledge GraphsLLM Evaluation

    La Rochelle Université, La Rochelle, France · Siren, Bordeaux, France · Université de Rennes, Rennes, France

  7. Abstract Priorities, Concrete Rewards: How Stated Preferences Relate to What People Reward in AI Responses

    Oct 22, 2026Zihan Chen, Lei Nico Zheng, Di ZhuUser Preference ModelingPreference Elicitation

    Stevens Institute of Technology, Hoboken, NJ, USA · University of Massachusetts Boston, Boston, MA, USA

  8. How Prompt Style Affects AI Accuracy and Speed in Competitive Programming

    Oct 21, 2026Saad B. Alarifi, Mohammed A. AlamLLM PromptingPrompt Sensitivity

    Riyadh Schools Riyadh, Saudi Arabia · Department of Computer Science Northwestern University Evanston, IL, USA

  9. LLMs as survey respondents: Comparison to simple semantic baselines

    Oct 21, 2026Esko Lehtonen, Tommi Buder-GröndahlLarge Language Model-Based Data AugmentationLanguage Model Generation Evaluation

    VTT Technical Research Centre of Finland Ltd Espoo, Finland · University of Helsinki Helsinki, Finland

  10. Why is AI-generated language so hard to understand?

    Oct 21, 2026Thomas Hikaru Clark, Matias B. Martin, Kyle MahowaldLanguage Model Generation Evaluation

    1New York University · 2The University of Texas at Austin

  11. Value-Rich, Differently Organized: Measuring Value Expression and Organization in AI-Generated Debates

    Oct 20, 2026Maoyuan Li, Haoyuan Li, Zhongsheng Wang +1Language Model Generation EvaluationMulti-Agent Debate

    Wuhan College of Communication · University of Auckland

  12. Speak vs. Know: Balancing Behavior and Knowledge Distributions of Large Language Models

    Oct 20, 2026Yang Liu, Masahiro Kaneko, Chenhui ChuLLM EvaluationLanguage Model Generation Evaluation

    Kyoto University · MBZUAI

  13. Target-Aware Curation of Hallucination Stress Tests for Large Language Models

    Oct 19, 2026Jiho Lee, Bingquan Zhang, Lin Ting Liang +2LLM EvaluationLanguage Model Generation Evaluation

    Dartmouth College · Oakland University

  14. When Empathy Misses the Goal: A Benchmark for Goal Displacement in LLM Advice

    Oct 18, 2026Dean Ariel, Guy LabanLLM EvaluationLanguage Model Generation Evaluation

    Department of Family Medicine, Clalit Health Services, Tel Aviv District, Israel · Faculty of Medicine and Health Sciences, Tel Aviv University, Tel Aviv, Israel · Stein Faculty of Computer and Information Science, Ben-Gurion University of the Negev, Beer Sheva, Israel +3

  15. Beyond Factual Accuracy: A Benchmark for Evaluating Perspective Alignment in LLMs

    Oct 18, 2026Sunghoon Kim, Sangyeop KimLanguage Model Generation EvaluationLLM Alignment

    Coxwave Seoul, Republic of Korea · Ewha Womans University Seoul, Republic of Korea

  16. How Well Can LLMs Verbalize Mathematical Formulas? A Multi-Domain Evaluation

    Oct 17, 2026Peter Volbach, Tim SchlippeLanguage Model Generation EvaluationLLM Evaluation

    IU International University of Applied Sciences Germany

  17. fxBench: Evaluating and Understanding Formula Suggestions in Spreadsheets

    Oct 17, 2026Sanket Mhatre, Sumit Gulwani, Vu Le +2Spreadsheet AutomationBenchmark Design

    Proximal · Microsoft, USA · Microsoft, India +1

  18. The Cognitive Cost of LLM Writing: A Reading-Time Comparison with Human Text

    Oct 16, 2026Kamer Cekini, Elisabetta Biondi, Chiara Boldrini +2Computational PsycholinguisticsLanguage Model Generation Evaluation

    IIT-CNR, via Moruzzi 1, Pisa, Italy · University of Pisa, Pisa, Italy

  19. Not X, but When: Diagnosing a Mandarin LLM Writing Tic through Contextual Appropriateness

    Oct 15, 2026Yihang Peng, Yi Jing, Dewen Liu +2Pragmatic Reasoning in Language ModelsLanguage Model Generation Evaluation

    Tsinghua University · Fudan University

  20. Change the Number, Change the Answer: Assessing Rating Scale Sensitivity in LLM-Based Text Scoring

    Oct 15, 2026Wonjeong Jo, Gyuwan Kim, Ming Shan HeeLLM ReliabilityLanguage Model Robustness

    UC Davis · UC Santa Barbara · MBZUAI

  21. Can LLMs Tell Stories Out of Order? A Corpus and Benchmark for Nonlinear Narrative Generation

    Oct 15, 2026Syed Rifat Raiyan, Md. Tariquzzaman, Md Farhan Ishmam +1Narrative GenerationText Generation

    Islamic University of Technology, Bangladesh · University of Utah, USA · University of Southern California, USA

  22. Rubric Judgments Lack Uncertainty: Conformal Weighting for LLM Evaluation

    Oct 15, 2026Licheng Pan, Haocheng Yang, Haoxuan Li +7Language Model Generation EvaluationRubric-Based Evaluation

    Zhejiang University · Xiaohongshu Inc · National University of Singapore +3

  23. APRiL: An Open-ended Benchmark for Chinese Social Awareness in Large Language Models

    Oct 14, 2026Tian Lan, Han Zhang, Shanshan Wang +3LLM EvaluationSocial Reasoning

    Kyoto University · Shanghai Jiao Tong University · University of Macau +2

  24. CSWBench: Evaluating Context-Sensitive Code-Switching in Creative Writing

    Oct 14, 2026Zewen Jiang, Yahan Li, Chenlu Guo +3Code-SwitchingMultilingual Language Model Evaluation

    School of Artificial Intelligence, Jilin University, China · College of Software, Jilin University, China · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd., China +4

  25. Gap Detection: Mitigating Homogenization through Demand and Defensibility

    Oct 14, 2026Xucheng Yu, Haibo Jin, Xiaoqi Wang +2Diverse Text GenerationLanguage Model Generation Evaluation

    University of Illinois Urbana-Champaign · Fudan University · University of Michigan, Ann Arbor

  26. Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation

    Oct 14, 2026Haocheng Yang, Licheng Pan, Yiming Dong +7Language Model Generation EvaluationRubric-Based Evaluation

    National University of Singapore · Zhejiang University · Alibaba Group +6

  27. RCCM: Diagnosing Gains and Harms in Reasoning-Prompt Transfer

    Oct 14, 2026Yuqi Zhang, Di ZhangLanguage Model Reasoning EvaluationLLM Reasoning

  28. KoAGE: Do Large Language Models Know, Learn, and Coin Korean Neologisms?

    Oct 14, 2026SungHo Kim, Eojin Jeon, SangKeun LeeLanguage ModelingLanguage Model Generation Evaluation

    Department of Artificial Intelligence, Korea University, Seoul, South Korea · Department of Computer Science and Engineering, Korea University, Seoul, South Korea

  29. Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation

    Oct 13, 2026Anubhav Jangra, Adrian de WynterLanguage Model Generation EvaluationGenerative AI Evaluation

    Columbia University, USA · The University of York, UK

  30. Beyond Aggregate Accuracy: Item-Level Transitions in Prompt Optimization

    Oct 13, 2026Chinmoy Mitra, Tarek MahmudAutomatic Prompt OptimizationLanguage Model Generation Evaluation

    Department of Computer Science & Engineering, Rajshahi University of Engineering & Technology, Rajshahi, Bangladesh · Texas A&M University–Kingsville, Kingsville, Texas, USA