Rubric-Based Evaluation

Ask about this topic

Overview

Rubric-based evaluation turns broad quality judgments into explicit dimensions, score levels, or checkable criteria, often tied to evidence in a response or task trajectory. Researchers use expert-written, retrieved, or automatically generated rubrics, then apply LLM judges, specialized scoring models, or executable checks across settings such as education, clinical reasoning, research agents, and multimodal generation. Rubric scores also serve as reward signals for training, where criterion weighting, calibration, and task-specific adaptation can make feedback more useful. A central challenge is that detailed rubrics do not guarantee valid judgments: criteria can be incomplete or biased, judges can disagree with people, and models may exploit what rubrics reward.

Latest papers

CardsList

201 papers29 papers in the last four weeks, up 26% on the four weeks before. 0.2% of all new papers.

  1. Quantifying the Statistical Effect of Rubric Modifications on Human-Autorater Agreement

    May 7, 2026Jessica Huynh, Alfredo Gomez, Athiya Deviyani +3Inter-Rater ReliabilityLLM-as-a-Judge

    Carnegie Mellon University · Google Research

  2. Principles and Guidelines for Randomized Controlled Trials in AI Evaluation

    May 3, 2026Christopher Kelly, Angelica Chowdhury, Alexandra Campili +5Human-in-the-Loop EvaluationRubric-Based Evaluation

    Independent · Dalhousie University · Northeastern University London +4

  3. Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese

    May 2, 2026Roseval Malaquias Junior, Giovana Kerche Bonás, Thales Sales Almeida +6LLM EvaluationLLM-as-a-Judge

    Maritaca AI · Jusbrasil

  4. Measurement Risk in Supervised Financial NLP: Rubric and Metric Sensitivity on JF-ICR

    Apr 30, 2026Sidi Chang, Peiying Zhu, Yuxiao Chen +1LLM EvaluationRubric-Based Evaluation

    Blossom AI Labs, Tokyo, Japan · Blossom AI Labs, San Francisco, USA

  5. Human-in-the-Loop Benchmarking of Heterogeneous LLMs for Automated Competency Assessment in Secondary Level Mathematics

    Apr 29, 2026Jatin Bhusal, Nancy Mahatha, Aayush Acharya +1LLM EvaluationEducational Assessment

    Research and Incubation Center (RAIN), Sunway College Kathmandu

  6. SciEval: A Benchmark for Automatic Evaluation of K-12 Science Instructional Materials

    Apr 28, 2026Zhaohui Li, Peng He, Zhiyuan Chen +4LLM EvaluationAI in Education

    Department of Computer Science and Engineering, University at Buffalo, USA · Department of Teaching and Learning, Washington State University, USA · College of Chemistry, Beijing Normal University, China +1

  7. Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks

    Apr 27, 2026Lawrence Keunho Jang, Jing Yu Koh, Daniel Fried +1Computer-Use Agent BenchmarksWeb Agent Benchmarks

    Carnegie Mellon University

  8. Case-Specific Rubrics for Clinical AI Evaluation: Methodology, Validation, and LLM-Clinician Agreement Across 823 Encounters

    Apr 27, 2026Aaryan Shah, Andrew Hines, Alexia Downs +6LLM-as-a-JudgeHuman-AI Agreement

    1Canvas Medical, San Francisco, CA, USA · Department of Biomedical Data Science, Stanford University, Stanford, CA, USA · 3XPC (X Primary Care) +2

  9. REC-CBM: Rubric-Aware Error-Correction Concept Bottleneck Models for Trustworthy Open-Ended Grading

    Apr 24, 2026Chengshuai Zhao, Fan Zhang, Kumar Satvik Chaudhary +4Concept Bottleneck ModelsAutomated Essay Scoring

    School of Computing and Augmented Intelligence, Arizona State University, USA · Mary Lou Fulton Teachers College, Arizona State University, USA · Department of Computer Science, National Yang Ming Chiao Tung University, TW

  10. Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews

    Apr 21, 2026Bowen Li, Haochen Ma, Yuxin Wang +5LLM EvaluationRubric-Based Evaluation

    College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China

  11. TactileEval: A Step Towards Automated Fine-Grained Evaluation and Editing of Tactile Graphics

    Apr 20, 2026Adnan Khan, Abbas Akkasi, Majid KomeiliRubric-Based EvaluationAutomated Evaluation

    School of Computer Science, Carleton University, Ottawa, Canada

  12. From Scoring to Explanations: Evaluating SHAP and LLM Rationales for Rubric-based Teaching Quality Assessment

    Apr 18, 2026Ivo Bueno, Babette Bühler, Philipp Stark +5Educational AssessmentShapley Value Attribution

    1Technical University of Munich · 2Munich Center for Machine Learning (MCML) · 3Lund University +3

  13. AsymmetryZero: A Framework for Operationalizing Human Expert Preferences as Semantic Evals

    Apr 15, 2026Tadhg Looram, Lucas Nuzzi, Kyle Waters +1Automated EvaluationHuman Preference Evaluation

    PortexAI · Stanford University

  14. ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents

    Apr 15, 2026Zhuofeng Li, Yi Lu, Dongfu Jiang +7Automated Peer ReviewScholarly Peer Review

    Texas A&M University · University of Waterloo · UC San Diego +2

  15. C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences

    Apr 15, 2026Akira Kawabata, Saku SugawaraReward ModelingRubric-Based Evaluation

    The Graduate University for Advanced Studies (SOKENDAI) · National Institute of Informatics · The Asahi Shimbun Company +1

  16. Designing Reliable LLM-Assisted Rubric Scoring for Constructed Responses: Evidence from Physics Exams

    Apr 14, 2026Xiuxiu Tang, G. Alex Ambrose, Ying ChengAutomated GradingRubric-Based Evaluation

    Department of Psychology, University of Notre Dame, Notre Dame, IN, 46556, USA · Notre Dame Learning, University of Notre Dame, Notre Dame, IN, 46556, USA

  17. RPA-Check: A Multi-Stage Automated Framework for Evaluating Dynamic LLM-based Role-Playing Agents

    Apr 13, 2026Riccardo Rosati, Edoardo Colucci, Massimiliano Bolognini +2Role-Playing Agent EvaluationLLM Agent Evaluation

    Department of Political Sciences, Communication and International Relations, University of Macerata, Via Don Minzoni 22/A, Macerata, 62100, Italy · Deparment of Information Engineering, Università Politecnica delle Marche, Via Brecce Bianche 12, Ancona, 60131, Italy · Department of Law, University of Macerata, Piaggia dell’Università 2, Macerata, 62100, Italy

  18. Judge Like Human Examiners: A Weighted Importance Multi-Point Evaluation Framework for Generative Tasks with Long-form Answers

    Apr 13, 2026Guoxin Yu, Chulun Zhou, Lemao Liu +7Language Model Generation EvaluationAutomated Evaluation

    Pengcheng Laboratory, Shenzhen, China. · The Chinese University of Hong Kong. · Fudan University. +2

  19. Self-Preference Bias in Rubric-Based Evaluation of Large Language Models

    Apr 8, 2026José Pombal, Ricardo Rei, André F. T. MartinsBias in LLM-as-a-JudgeLLM Evaluation

    Sword Health · Instituto de Telecomunicações · Instituto Superior Técnico, Universidade de Lisboa +2

  20. The Hitchhikers Guide to Rubric Quality Understanding and Enrichment

    Apr 1, 2026Ankit Aich, Zhengyang Qi, Charles Dickens +7Agent EvaluationPsychometric Validation

    Snorkel AI · Department of Computer Sciences, University of Wisconsin–Madison

  21. LLM Essay Scoring Under Holistic and Analytic Rubrics: Prompt Effects and Bias

    Mar 31, 2026Filip J. Kucia, Anirban Chakraborty, Anna WróblewskaAutomated Essay ScoringLanguage Model Bias Evaluation

    Faculty of Mathematics and Information Science, Warsaw University of Technology, 00-662 Warsaw, Poland · University of Wolverhampton, UK

  22. PeopleSearchBench: Evaluating AI-Powered People Search Platforms with Criteria-Grounded Verification

    Mar 29, 2026Tianyu Shi, Wei Wang, Zequn Xie +10Web Search AgentsInformation Retrieval

  23. Xpertbench: Expert Level Tasks with Rubrics-Based Evaluation

    Mar 27, 2026Xue Liu, Xin Ma, Yuxin Ma +36LLM EvaluationRubric-Based Evaluation

    ByteDance Seed

  24. RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following

    Mar 26, 2026Tianjun Pan, Xuan Lin, Wenyan Yang +7Rubric-Based EvaluationLLM-as-a-Judge

    College of Computer Science and Artificial Intelligence, Fudan University · Ant Group · Donghua University

  25. AdaRubric: Task-Adaptive Rubrics for Reliable LLM Agent Evaluation and Reward Learning

    Mar 22, 2026Liang DingLLM Agent EvaluationRubric-Based Evaluation

    The University of Sydney

  26. RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation

    Mar 21, 2026Kaustubh D. Dhole, Eugene AgichteinAutomatic Rubric GenerationLLM Evaluation

    Department of Computer Science, Emory University, Atlanta, GA, USA

  27. Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge

    Mar 11, 2026Mingyang Song, Mao Zheng, Chenning XuLLM-as-a-JudgeRubric-Based Evaluation

    Large Language Model Department, Tencent, China

  28. Criterion Validity of LLM-as-Judge for Business Outcomes in Conversational Commerce

    Mar 11, 2026Liang Chen, Qi Liu, Wenhuan Lin +1Rubric-Based EvaluationMulti-Turn Dialogue Evaluation

  29. CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling

    Mar 9, 2026Dengcan Liu, Fengkai Yang, Xiaohan Wang +7Preference ModelingReward Modeling

    University of Science and Technology of China · Peking University · Meituan +1

  30. PresentBench: A Fine-Grained Rubric-Based Benchmark for Slide Generation

    Mar 7, 2026Xin-Sheng Chen, Jiayu Zhu, Pei-lin Li +3Benchmark DesignGenerative AI Evaluation

    Tsinghua University