Language Model Generation Evaluation

Overview

The field studies how to tell whether generated text is correct, useful, faithful to its sources, well written, and appropriate for a task. Approaches range from lexical and embedding-based metrics to human ratings, rubric-based LLM judges, learned evaluators, and task-specific checks such as code execution or schema validation. Benchmarks increasingly test long-form and open-ended outputs, multiple samples, languages, and settings, while evaluation-cost methods aim to select informative examples efficiently. A central challenge is that scores can depend on prompts, judges, and task context: a fluent or highly rated answer may still omit facts, repeat familiar ideas, or fail to reflect human preferences.

Latest papers

CardsList

422 papers32 papers in the last four weeks, down 50% on the four weeks before. 0.2% of all new papers.

  1. PosteriorBench: Evaluating Behavior Posterior Sampling in Language Models

    Nov 19, 2026Zhiyang Xun, Daniel D. Johnson, Sarah Schwettmann +1Language Model Generation EvaluationLanguage Model Post-Training

  2. Reversing the text is not reversing the path: a fixed-text measure without a zero, and an assembly-order comparison that vanishes identically

    Nov 17, 2026Y. -H. TaguchiLanguage Model Generation EvaluationInput Order Sensitivity in Language Models

  3. Poetry 'resurrection' from LLMs

    Nov 12, 2026Vincent Cellucci, Peter van der PuttenComputational CreativityText Generation

  4. Leaderboard Hacking: Preference-Based Model Evaluations are Vulnerable to Manipulation

    Nov 10, 2026Eve Fleisig, Joachim Baumann, Dylan Hadfield-Menell +1LLM EvaluationLanguage Model Generation Evaluation

  5. Adapting Small Language Models to Romanized Nepali: Tokenizer Fertility, Reference Likelihood, and Generation Metrics

    Nov 9, 2026Pujan Pant, Sagar Newpane, Hari Sitaula +1Language ModelingSmall Language Models

  6. AlienBench: A Benchmark of Structural Novelty in Language Model Outputs

    Nov 6, 2026Jonas Oppenlaender, Johanna M. SilvennoinenLLM EvaluationLanguage Model Generation Evaluation

  7. The Invisible Gap: Romanized Indic Generation Fails Where Automatic Metrics Succeed

    Nov 6, 2026Madhu Sudhan Rao Beesetty, Surbhi Sharma, Devesh Maheshwari +1Language Model Generation EvaluationGenerative AI Evaluation

  8. Does Reasoning Improve Psychological Depth in Large Language Models?

    Nov 6, 2026Ruichen Zheng, Yihe Wang, Fabrice Harel-Canada +4Text GenerationLanguage Model Reasoning Evaluation

  9. Frame: Form-Restricted Authoring with Model-Based Evaluation

    Nov 6, 2026Nan Jiang, Mohammed Mohsin Hussain, Onkar Tiwari +4Evidence-Grounded GenerationLong-Form Document Generation

  10. Signals, Evidence, and Context: User Scrutiny in Real-World LLM Conversations

    Nov 5, 2026Ahana Biswas, Yu-Ru LinLLM Answer VerificationLLM Reliability

  11. How Well Can Large Language Models Impersonate Individual Authors? A Comparative Evaluation

    Nov 4, 2026Harshit Gupta, Shunichi IshiharaLanguage Model Generation EvaluationStylometry

  12. IdeaTaste: Do LLMs Share the Research Taste of Human Researchers?

    Nov 2, 2026Shanshan Zhong, Chen Xu, Jingjie Ning +3Research Idea GenerationLLM Evaluation

  13. Character-Name Concentration in LLM Fiction: Measuring Displacement and Mitigating Repetition

    Oct 31, 2026Alejandro Mozo Quesada, Sergio Gálvez Rojas, Francisco López ValverdeRepetition in Language GenerationControllable Text Generation

  14. Spotting the Artificial: Dialogic Humor Improvisation in Humans and LLMs

    Oct 31, 2026Lizeth Palacios Patiño, Ximena Gutierrez-Vasques, Daniel RodriguezLanguage Model Generation EvaluationComputational Creativity

  15. Homebrewed SLMs: Autoregressive and Masked Diffusion Language Models Built at Home for Portuguese

    Oct 29, 2026Pedro Carvalho BromLanguage ModelingSmall Language Models

  16. Offline Language-Based Support System for Tactical Decision-Making in Air Combat Operations

    Oct 26, 2026Joao P. A. DantasAI-Assisted Decision MakingLanguage Model Generation Evaluation

  17. Useful Diversity or Hallucinated Variety? Persona-Conditioned Synthetic Query Generation for E-Commerce Search

    Oct 26, 2026Filippos Ventirozos, Tom Armitage, Mahmoud Oshagh +2E-Commerce SearchPersona-Conditioned Generation

  18. DraMaze: A Multi-Agent Pipeline with Future-Aware Memory for Diverse Microdrama Script Generation

    Oct 22, 2026Siyang He, Qiqi Wang, Fan Liu +5Narrative GenerationText Generation

  19. An Empirical Study of Recent Large Language Models for Knowledge Graph to Text Generation on Standard Benchmarks

    Oct 22, 2026Priyanka, Renaud Delbru, Souhail Bakkali +2Knowledge GraphsLLM Evaluation

  20. Abstract Priorities, Concrete Rewards: How Stated Preferences Relate to What People Reward in AI Responses

    Oct 22, 2026Zihan Chen, Lei Nico Zheng, Di ZhuUser Preference ModelingPreference Elicitation

  21. How Prompt Style Affects AI Accuracy and Speed in Competitive Programming

    Oct 21, 2026Saad B. Alarifi, Mohammed A. AlamLLM PromptingPrompt Sensitivity