Language Model Generation Evaluation

Ask about this topic

Momentum

30 papers in the last four weeks, up 100% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 209

All topics
CardsList

  1. CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation

    Jul 31, 2026Mengting Chen, Yanshu Sun, Wanting Liang +5Open-Ended GenerationLLM Evaluation

    FinStep · StepFun

  2. Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B

    Jul 30, 2026Iliya MirzaeiLLM Self-RefinementLanguage Model Generation Evaluation

    Department of Computer Science Stony Brook University

  3. Beyond a Single Judge: The Evidence-Grounded, Social-Weighted Persona Panel for Generative UI Evaluation

    Jul 30, 2026Zheng Wu, Yibo Luo, Pu Zhang +2Human Preference EvaluationLLM-as-a-Judge

    School of Computer Science, Shanghai Jiao Tong University · 2ByteDance Inc

  4. IFHierBench: Hierarchical Instruction Following for Large Language Models

    Jul 30, 2026Yuetian Mao, Chunyang ChenLLM EvaluationLanguage Model Generation Evaluation

    Technical University of Munich

  5. LayerRAG-Bench: A Cross-Layer Reliability Benchmark for Agentic Retrieval-Augmented Generation

    Jul 29, 2026Musa ShamsAI Agent ReliabilityRetrieval-Augmented Generation

    Independent Researcher

  6. Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgets

    Jul 27, 2026Zongyou Yang, Yinghan HouLLM EvaluationLLM Answer Verification

    Dyson School of Design Engineering Imperial College London London, United Kingdom · Department of Electrical and Electronic Engineering Imperial College London London, United Kingdom

  7. QuantiBias: Benchmarking Quantization-Induced Bias in LLMs

    Jul 23, 2026Emilio FerraraOpen-Ended GenerationLLM Quantization

    Thomas Lord Department of Computer Science University of Southern California

  8. Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness

    Jul 21, 2026Xilun Chen, Zhaleh Feizollahi, Ross Goodwin +5Open-Ended GenerationLLM-as-a-Judge

    Meta AI

  9. Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models

    Jul 21, 2026Netanel EliavLLM EvaluationLanguage Model Generation Evaluation

    Machine Human Intelligence Lab (MHIL)

  10. StabilityBench: Benchmarking Instability in LLMs

    Jul 17, 2026Emma Kondrup, Zachary Yang, Anne Imouza +1LLM EvaluationLanguage Model Generation Evaluation

    1Mila — Quebec AI Institute · 2McGill University · 3Ubisoft La Forge

  11. The One-Word Census: Answer-Choice Conformity Across 44 Language Models

    Jul 14, 2026Tapan ParikhLLM EvaluationLanguage Model Generation Evaluation

    Cornell Tech

  12. Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking

    Jul 13, 2026Niranjan Kumar M, Balaji Nagarajan, Karthik Nair +2Language Model Generation EvaluationCustomer Support Automation

    Lowe's

  13. BackendForge: Benchmarking Agentic End-to-End Code Generation with Backend Services

    Jul 13, 2026Yuzhe Guo, Mengzhou Wu, Yuan Cao +4Tool-Augmented Language Model AgentsLanguage Model Generation Evaluation

    1Key Lab of HCST (PKU), MOE; SCS, Peking University · 2Beijing Tongming Lake Information Technology Application Innovation Center (TLAIC) · 3Fudan University Institute of Systems for Advanced Computing +1

  14. Eval-Pair Matrix: Answer-Paired Meta-Evaluation of LLM Judges for Grounded RAG

    Jul 12, 2026Sriram Selvam, Anneswa GhoshLLM-as-a-JudgeLanguage Model Generation Evaluation

  15. Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging

    Jul 11, 2026Jinglan Gong, Jiefan Lu, Hewei Guo +5Language Model Generation EvaluationPersona Consistency

    SenseTime Research · University of Science and Technology of China · Tsinghua University

  16. Prompt Robustness Is Task-Dependent: Comparing Objective and Belief-Style Questions in LLM Evaluation

    Jul 6, 2026Sadia Kamal, Arefa Patwary, Anthony Marchiafava +2LLM EvaluationLanguage Model Generation Evaluation

    Oklahoma State University · University of North Texas

  17. SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models

    Jul 6, 2026Thomas Thebaud, Yuzhe Wang, Hao Zhang +5Audio-Language Model EvaluationLanguage Model Generation Evaluation

    ∗Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA · †Amazon Inc., Seattle, WA, USA

  18. Rating the Pitch, Not the Product: User Evaluations of LLMs Reflect Expectations More Than Performance

    Jul 6, 2026Robert Morabito, Tyler McDonald, Charitra Viswanath +4Human Preference EvaluationLLM Evaluation

    Brock University · Emory University · University of Southern California +2

  19. CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion

    Jul 6, 2026Adam Fisch, Daniel Deutsch, Joshua Maynez +5Language Model Generation EvaluationMatrix Completion

    Google DeepMind · Google Research

  20. Telescope: Improving Zero Shot Detection of LLM Generated Content By Measuring Token Repetition Probability

    Jul 5, 2026Christopher Nassif, Josh F. CooperAI-Generated Text DetectionLanguage Model Generation Evaluation

    Virginia Tech Department of Electrical and Computer Engineering · Virginia Tech Department of Industrial and Systems Engineering

  21. Object Aligner: A Configurable JSON Schema Similarity Score for Graphs, Applied to LLM Prompt Optimization

    Jul 2, 2026Jan DrchalLLM EvaluationLanguage Model Generation Evaluation

    Artificial Intelligence Center, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic

  22. Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling

    Jul 2, 2026Dazhi Fu, Jiuding Yang, Yiwen Guo +1Reward ModelingLLM-as-a-Judge

    School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · LIGHTSPEED · Independent Researcher

  23. LitReview Arena: Evaluating Literature Review Agents with Battle-Style Peer Review Platform

    Jul 1, 2026Ruotong Zhao, Zhiyu Chen, Xurui Liu +7Human Preference EvaluationLLM-as-a-Judge

    Tsinghua University, Beijing, China · Zhongguancun Academy, Beijing, China · Zhongguancun Institute of AI, Beijing, China +2

  24. Persona Non Grata: LLM Persona-Driven Generations in MCQA are Unstable in Distinct Dimensions

    Jul 1, 2026César Guerra-Solano, Xiang Lorraine LiLLM EvaluationLanguage Model Generation Evaluation

    Department of Computer Science, University of Pittsburgh

  25. "Don't Say It!": Constraints, Compliance, and Communication when Language Models Play Taboo

    Jul 1, 2026Sara Candussio, Francesca Padovani, Daniel Scalena +1LLM GroundingLanguage Model Generation Evaluation

    AILab, MIGe, Univeristy of Trieste, Italy · Center for Language and Cognition (CLCG), University of Groningen, The Netherlands · University of Milano - Bicocca, Italy

  26. Low Perplexity is Repetition: A One-Dimensional Self-Conditioning Attractor in Continuous Diffusion LMs

    Jul 1, 2026Shuai Zhang, Zijie Chen, Hongliang He +2Language Model Generation EvaluationDiffusion Language Model Inference

    Zhejiang University · Westlake University · Ant Group

  27. SEFORA: Student Essays with Feedback Corpus and LLM Feedback Evaluation Framework

    Jun 30, 2026Shayan Peyghambari Oskoui, Norah Almousa, Zhaoyi Joey Hou +5LLM EvaluationLanguage Model Generation Evaluation

    University of Pittsburgh

  28. Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truth

    Jun 30, 2026Sajjad Abdoli, Ghassan Al-Sumaidaee, Ahmad ElShiekh +2Multilingual Language Model EvaluationLLM-as-a-Judge

    1Perle AI

  29. Little Brains, Big Feats: Exploring Compact Language Models

    Jun 29, 2026Dari Baturova, Elena Bruches, Ivan Chernov +3Retrieval-Augmented GenerationOn-Device Language Model Inference

    Siberian Neuronets LLC, Novosibirsk, Russia

  30. Can MLLMs Critique Like Humans? Evaluating Open-Ended Aesthetic Reasoning in Multimodal Large Language Models

    Jun 29, 2026Sajjad Ghiasvand, Maryam Amirizaniani, Haniyeh Ehsani Oskouie +2Language Model Generation EvaluationMultimodal Reasoning

    UCSB · University of Washington · UCLA

  31. Understanding Evaluation Illusion in Diffusion Large Language Models

    Jun 28, 2026Hengxiang Zhang, Jiaxi Ren, Renchunzi Xie +1Prompt SensitivityLLM Evaluation

    Department of Statistics and Data Science, Southern University of Science and Technology · Intelligent Computing Research Center, Great Bay Institute for Advanced Study, Great Bay University

  32. Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA

    Jun 26, 2026Sambaran BandyopadhyayIn-Context RetrievalLLM Evaluation

    Adobe Research

  33. Speculative Refinement: A Hybrid Autoregressive Diffusion Decoding Strategy and Its Behavior Across Benchmarks

    Jun 25, 2026Aditi Gupta, Neel Mishra, Kushagra Trivedi +1Masked Diffusion ModelsLanguage Model Generation Evaluation

    IIIT Hyderabad, India · Microsoft

  34. Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement

    Jun 25, 2026Sangwoo Cho, Kushal Chawla, Pengshan Cai +4LLM EvaluationLLM-as-a-Judge

    Capital One, AI Foundations, McLean, VA 22102, USA.

  35. The Capability Frontier: Benchmarks Miss 82% of Model Performance

    Jun 25, 2026Bradley Fowler, Ryan Smith, Daniel Thi Graviet +8LLM EvaluationLanguage Model Generation Evaluation

    1Martian · 3ThoughtWorks · University of Oxford

  36. DualEval: Joint Model-Item Calibration for Unified LLM Evaluation

    Jun 24, 2026Aaron J. Li, Hao Huang, Youngmin Park +6LLM EvaluationLanguage Model Generation Evaluation

    University of California, Berkeley · Arena · University of California, Los Angeles

  37. CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression

    Jun 23, 2026Morayo Danielle Adeyemi, Ryan A. Rossi, Franck DernoncourtLLM EvaluationLLM Compression

    Independent · Adobe Research

  38. Do LLM Attribution Metrics Transfer? Auditing Retrieval-Augmented Generation Evaluation Across Datasets and Constructs

    Jun 22, 2026Tianyu Ding, Aditya Nannapaneni, Juan Pablo De la Cruz WeinsteinLanguage Model Generation EvaluationLLM Auditing

    Amazon Web Services

  39. GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation

    Jun 22, 2026Jeffrey FlyntAI Agent ReliabilityTool-Use Evaluation

    Independent Researcher

  40. PRIME: Evaluating Prompt Resolution Under Incompatible Instructions in LLMs

    Jun 21, 2026Tehreem Javed, Shumaim Fatimah, Masooma Bakhtiari +2LLM EvaluationLanguage Model Generation Evaluation

    School of Electrical Engineering and Computer Science (SEECS), National University of Sciences and Technology (NUST), Islamabad, Pakistan

  41. BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories

    Jun 21, 2026Shreyas KCMultilingual Language Model EvaluationLLM-as-a-Judge

  42. RECOM: A Validity Discrimination Tradeoff in Automatic Metrics for Open Ended Reddit Question Answering

    Jun 17, 2026Pushwitha Krishnappa, Amit Das, Vinija Jain +2Open-Ended GenerationLLM Evaluation

    University of Alabama Huntsville · University of North Alabama · Stanford University +2

  43. How Inference Compute Shapes Frontier LLM Evaluation

    Jun 16, 2026Jessica McFadyen, Ole Jorgensen, Harry Coppock +2LLM EvaluationLanguage Model Generation Evaluation

    AI Security Institute, London, UK · University of Oxford, Oxford, UK · Harvard University, Cambridge, MA, USA

  44. Prompt Perturbation for Reliable LLM Evaluation over Comparison Graphs

    Jun 16, 2026Dong Huang, Jianbo Sun, Pengkun YangLLM EvaluationPairwise Comparison

    Department of Statistics and Data Science, Tsinghua University

  45. The Benchmark Illusion: Pruned LLMs Can Pass Multiple Choice but Fail to Answer

    Jun 16, 2026Rui Wen, Lu Sun, Jiayang Liu +3Open-Ended GenerationLLM Evaluation

    Institute of Science Tokyo · Tohoku University · Nanyang Technological University +2

  46. NarrativeWorldBench: A Frontier-Saturated Benchmark and a Latent World Model for Long-Horizon Co-Creative Audio Drama

    Jun 16, 2026Logan Mann, Abdur Rahman, Mohammad Saifullah +2LLM World ModelsMultilingual Language Model Evaluation

    University of California, Santa Barbara · Pocket FM

  47. Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines

    Jun 13, 2026Yitao LiLLM EvaluationLLM-as-a-Judge

  48. Beyond Accuracy: Measuring Bias Acknowledgment in Chain-of-Thought Reasoning for Responsible AI Evaluation

    Jun 13, 2026Xian Sun, Wei Gao, Yingshuo Wang +9Language Model Generation EvaluationCoT Evaluation

    Duke University · Northeastern University · University of California, Berkeley +5

  49. Metric Match: A Subset Selection Approach to Evaluating LLM Judge Reliability

    Jun 12, 2026Alyssa Unell, Natalie Dullerud, Naomi Boneh +4LLM-as-a-JudgeLanguage Model Generation Evaluation

    Department of Computer Science Stanford University · Department of Medicine Stanford University

  50. GENIE: A Fine-Grained Measure for Novelty

    Jun 11, 2026Ramya Namuduri, Manya Wadhwa, Anshun Asher Zheng +2LLM EvaluationLanguage Model Generation Evaluation

    The University of Texas at Austin · New York University

  51. Correct Looks Better: Pairwise Comparisons Reveal Accuracy Rankings

    Jun 8, 2026Mina Remeli, Moritz HardtPairwise ComparisonLLM-as-a-Judge

    Max Planck Institute for Intelligent Systems, Tübingen, Germany, Tübingen AI Center

  52. Can LLMs understand LilyPond? A benchmark for symbolic music generation and understanding

    Jun 7, 2026Matteo Spanio, Mohammad Torabi, Andrea Poltronieri +1Music Generation EvaluationLLM Evaluation

    Centro di Sonologia Computazionale, University of Padova, Padova, Italy · Music Technology Group, Universitat Pompeu Fabra, Barcelona, Spain

  53. Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics

    Jun 7, 2026Antonio Franca, Alexander TongNon-Autoregressive Text GenerationLLM Evaluation

    AITHYRA Institute, Vienna, Austria.

  54. SLMJury: Can Small Language Models Judge as Well as Large Ones?

    Jun 5, 2026Anish Laddha, Nitesh Pradhan, Gaurav SrivastavaLLM-as-a-JudgeLanguage Model Generation Evaluation

    Department of Computer Science and Engineering, LNMIIT, Jaipur, India · Department of Computer Science, Virginia Tech, Blacksburg, VA, USA

  55. Cutting LLM Evaluation Costs with SySRs: A Bandit Algorithm that Provably Exploits Model Similarity

    Jun 5, 2026Zifan Lyu, Chahine Nejma, Tobias Wegel +2Model SelectionMulti-Armed Bandits

    1ETH Zurich · 2Centrale Sup´elec · 3ENS de Cachan +1

  56. Explain Like I'm 5 or Whatever I Choose: Evaluating the Interactive Potential of Language Model Responses

    Jun 5, 2026Indu Panigrahi, Tal AugustLLM EvaluationLanguage Model Generation Evaluation

    Siebel School of Computing and Data Science University of Illinois Urbana-Champaign

  57. UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs

    Jun 4, 2026Amirhossein Abaskohi, Amirhossein Dabiriaghdam, Liang Luo +4LLM EvaluationLanguage Model Generation Evaluation

    University of British Columbia · 2Independent Researcher

  58. Asuka-Bench: Benchmarking Code Agents on Underspecified User Intent and Multi-Round Refinement

    Jun 4, 2026Xin Wang, Liangtai Sun, Yaoming Zhu +8Computer-Use Agent BenchmarksLanguage Model Generation Evaluation

    University of Science and Technology of China · 2Independent researchers

  59. Elmes*: Automated Construction of Fine-Grained Evaluation Rubrics for Large Language Models in Long-Tail Educational Scenarios

    Jun 4, 2026Tao Liu, Ye Lu, Ruohua Zhang +4LLM EvaluationLLM-as-a-Judge

    1Shanghai Institute of AI for Education, East China Normal University, Shanghai 200062, China · School of Computer Science and Technology, East China Normal University, Shanghai 200062, China · 3Shanghai Innovation Institute, Shanghai 200231, China

  60. Self-Evaluation Is Already There: Eliciting Latent Judge Calibration in Base LLMs with Minimal Data

    Jun 3, 2026XiuYu Zhang, Yi Shan, Junfeng Fang +1LLM EvaluationLLM-as-a-Judge

    National University of Singapore · Beijing University of Technology