Language Model Generation Evaluation

Follow this topic Ask about this topic

Momentum

30 papers in the last four weeks, up 100% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 209

All topics

CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation
Jul 31, 2026Mengting Chen, Yanshu Sun, Wanting Liang +5Open-Ended Generation LLM Evaluation
FinStep · StepFun
Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B
Jul 30, 2026Iliya MirzaeiLLM Self-Refinement Language Model Generation Evaluation
Department of Computer Science Stony Brook University
Beyond a Single Judge: The Evidence-Grounded, Social-Weighted Persona Panel for Generative UI Evaluation
Jul 30, 2026Zheng Wu, Yibo Luo, Pu Zhang +2Human Preference Evaluation LLM-as-a-Judge
School of Computer Science, Shanghai Jiao Tong University · 2ByteDance Inc
IFHierBench: Hierarchical Instruction Following for Large Language Models
Jul 30, 2026Yuetian Mao, Chunyang ChenLLM Evaluation Language Model Generation Evaluation
Technical University of Munich
LayerRAG-Bench: A Cross-Layer Reliability Benchmark for Agentic Retrieval-Augmented Generation
Jul 29, 2026Musa ShamsAI Agent Reliability Retrieval-Augmented Generation
Independent Researcher
Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgets
Jul 27, 2026Zongyou Yang, Yinghan HouLLM Evaluation LLM Answer Verification
Dyson School of Design Engineering Imperial College London London, United Kingdom · Department of Electrical and Electronic Engineering Imperial College London London, United Kingdom
QuantiBias: Benchmarking Quantization-Induced Bias in LLMs
Jul 23, 2026Emilio FerraraOpen-Ended Generation LLM Quantization
Thomas Lord Department of Computer Science University of Southern California
Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness
Jul 21, 2026Xilun Chen, Zhaleh Feizollahi, Ross Goodwin +5Open-Ended Generation LLM-as-a-Judge
Meta AI
Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models
Jul 21, 2026Netanel EliavLLM Evaluation Language Model Generation Evaluation
Machine Human Intelligence Lab (MHIL)
StabilityBench: Benchmarking Instability in LLMs
Jul 17, 2026Emma Kondrup, Zachary Yang, Anne Imouza +1LLM Evaluation Language Model Generation Evaluation
1Mila — Quebec AI Institute · 2McGill University · 3Ubisoft La Forge
The One-Word Census: Answer-Choice Conformity Across 44 Language Models
Jul 14, 2026Tapan ParikhLLM Evaluation Language Model Generation Evaluation
Cornell Tech
Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking
Jul 13, 2026Niranjan Kumar M, Balaji Nagarajan, Karthik Nair +2Language Model Generation Evaluation Customer Support Automation
Lowe's
BackendForge: Benchmarking Agentic End-to-End Code Generation with Backend Services
Jul 13, 2026Yuzhe Guo, Mengzhou Wu, Yuan Cao +4Tool-Augmented Language Model Agents Language Model Generation Evaluation
1Key Lab of HCST (PKU), MOE; SCS, Peking University · 2Beijing Tongming Lake Information Technology Application Innovation Center (TLAIC) · 3Fudan University Institute of Systems for Advanced Computing +1
Eval-Pair Matrix: Answer-Paired Meta-Evaluation of LLM Judges for Grounded RAG
Jul 12, 2026Sriram Selvam, Anneswa GhoshLLM-as-a-Judge Language Model Generation Evaluation
Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging
Jul 11, 2026Jinglan Gong, Jiefan Lu, Hewei Guo +5Language Model Generation Evaluation Persona Consistency
SenseTime Research · University of Science and Technology of China · Tsinghua University
Prompt Robustness Is Task-Dependent: Comparing Objective and Belief-Style Questions in LLM Evaluation
Jul 6, 2026Sadia Kamal, Arefa Patwary, Anthony Marchiafava +2LLM Evaluation Language Model Generation Evaluation
Oklahoma State University · University of North Texas
SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models
Jul 6, 2026Thomas Thebaud, Yuzhe Wang, Hao Zhang +5Audio-Language Model Evaluation Language Model Generation Evaluation
∗Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA · †Amazon Inc., Seattle, WA, USA
Rating the Pitch, Not the Product: User Evaluations of LLMs Reflect Expectations More Than Performance
Jul 6, 2026Robert Morabito, Tyler McDonald, Charitra Viswanath +4Human Preference Evaluation LLM Evaluation
Brock University · Emory University · University of Southern California +2
CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion
Jul 6, 2026Adam Fisch, Daniel Deutsch, Joshua Maynez +5Language Model Generation Evaluation Matrix Completion
Google DeepMind · Google Research
Telescope: Improving Zero Shot Detection of LLM Generated Content By Measuring Token Repetition Probability
Jul 5, 2026Christopher Nassif, Josh F. CooperAI-Generated Text Detection Language Model Generation Evaluation
Virginia Tech Department of Electrical and Computer Engineering · Virginia Tech Department of Industrial and Systems Engineering
Object Aligner: A Configurable JSON Schema Similarity Score for Graphs, Applied to LLM Prompt Optimization
Jul 2, 2026Jan DrchalLLM Evaluation Language Model Generation Evaluation
Artificial Intelligence Center, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic
Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling
Jul 2, 2026Dazhi Fu, Jiuding Yang, Yiwen Guo +1Reward Modeling LLM-as-a-Judge
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · LIGHTSPEED · Independent Researcher
LitReview Arena: Evaluating Literature Review Agents with Battle-Style Peer Review Platform
Jul 1, 2026Ruotong Zhao, Zhiyu Chen, Xurui Liu +7Human Preference Evaluation LLM-as-a-Judge
Tsinghua University, Beijing, China · Zhongguancun Academy, Beijing, China · Zhongguancun Institute of AI, Beijing, China +2
Persona Non Grata: LLM Persona-Driven Generations in MCQA are Unstable in Distinct Dimensions
Jul 1, 2026César Guerra-Solano, Xiang Lorraine LiLLM Evaluation Language Model Generation Evaluation
Department of Computer Science, University of Pittsburgh
"Don't Say It!": Constraints, Compliance, and Communication when Language Models Play Taboo
Jul 1, 2026Sara Candussio, Francesca Padovani, Daniel Scalena +1LLM Grounding Language Model Generation Evaluation
AILab, MIGe, Univeristy of Trieste, Italy · Center for Language and Cognition (CLCG), University of Groningen, The Netherlands · University of Milano - Bicocca, Italy
Low Perplexity is Repetition: A One-Dimensional Self-Conditioning Attractor in Continuous Diffusion LMs
Jul 1, 2026Shuai Zhang, Zijie Chen, Hongliang He +2Language Model Generation Evaluation Diffusion Language Model Inference
Zhejiang University · Westlake University · Ant Group
SEFORA: Student Essays with Feedback Corpus and LLM Feedback Evaluation Framework
Jun 30, 2026Shayan Peyghambari Oskoui, Norah Almousa, Zhaoyi Joey Hou +5LLM Evaluation Language Model Generation Evaluation
University of Pittsburgh
Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truth
Jun 30, 2026Sajjad Abdoli, Ghassan Al-Sumaidaee, Ahmad ElShiekh +2Multilingual Language Model Evaluation LLM-as-a-Judge
1Perle AI
Little Brains, Big Feats: Exploring Compact Language Models
Jun 29, 2026Dari Baturova, Elena Bruches, Ivan Chernov +3Retrieval-Augmented Generation On-Device Language Model Inference
Siberian Neuronets LLC, Novosibirsk, Russia
Can MLLMs Critique Like Humans? Evaluating Open-Ended Aesthetic Reasoning in Multimodal Large Language Models
Jun 29, 2026Sajjad Ghiasvand, Maryam Amirizaniani, Haniyeh Ehsani Oskouie +2Language Model Generation Evaluation Multimodal Reasoning
UCSB · University of Washington · UCLA

Jul 31, 2026, cs.CLJ/K move · Enter open · S save

CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation

Mengting Chen, Yanshu Sun, Wanting Liang, Beidi Luan+4

FinStep · StepFun

Open-Ended Generation LLM Evaluation Language Model Generation Evaluation Rubric-Based Evaluation Automated Evaluation

Open paper AskPDF arXiv

Reliable evaluation of open-ended LLM outputs requires fine-grained rubrics, yet expert curation is costly and difficult to scale. Existing automated pipelines rely on strict judge unanimity and binary variance filters, which cannot distinguish measurable rubrics from informative ones. We introduce CalibratedRubric, a task-adaptive framework that combines type-specific scoring, Bayesian rubric-measurability filtering, and item response theory (IRT)-based bank assembly. CalibratedRubric estimates each rubric's measurability with a Beta--Bernoulli agreement posterior and uses a submodular information-coverage objective to construct compact rubric banks over the observed capability range. Across financial, healthcare, general, and legal benchmarks, measurability filtering improves human-gold agreement on JudgmentBench from

κ=0.604

0.743

. IRT-based greedy selection improves cross-fitted rank fidelity over random selection across all six evaluated response blocks and requires only 49 rather than 131 rubrics to reach the target correlation on FinResearchBench decision-support tasks. Task-label perturbations further reduce system separation, confirming the practical relevance of task-adaptive scoring. These results support CalibratedRubric as an efficient, uncertainty-aware approach to open-ended LLM evaluation, with calibration gains depending on sufficient judge redundancy.

Momentum

Related topics

Latest papers 209