Overview

Rubric-based evaluation turns broad quality judgments into explicit dimensions, score levels, or checkable criteria, often tied to evidence in a response or task trajectory. Researchers use expert-written, retrieved, or automatically generated rubrics, then apply LLM judges, specialized scoring models, or executable checks across settings such as education, clinical reasoning, research agents, and multimodal generation. Rubric scores also serve as reward signals for training, where criterion weighting, calibration, and task-specific adaptation can make feedback more useful. A central challenge is that detailed rubrics do not guarantee valid judgments: criteria can be incomplete or biased, judges can disagree with people, and models may exploit what rubrics reward.

Latest papers

CardsList

208 papers15 papers in the last four weeks, down 46% on the four weeks before. 0.1% of all new papers.

  1. Rubric-Based AES with LLM Ensembles: Scoring of Student Responses in Critical Online Reasoning Using Ensembles of LLMs

    Nov 16, 2026Daniel Baumartz, Alexander Mehler, Patrick Schrottenbacher +2Automated Essay ScoringEducational Assessment

  2. Objectify, Don't Delete: Complaint-Derived Evaluation of Ambient Clinical Notes

    Nov 14, 2026Nikita Gupta, Narayanan Unny Edakunni, Vikram Okaly +4Clinical Language Model EvaluationRubric-Based Evaluation

  3. Toward Cross-Rubric Surgical Skill Assessment via Joint Embedding Prediction

    Nov 11, 2026Jiajun Cheng, Haidar M. Abdul-Muhsin, Juha Choi +1Rubric-Based EvaluationSurgical Video Understanding

  4. GRAP: Gender Rubric-Aware Probing for Analyzing Gender Bias in LLMs

    Nov 9, 2026Seona Seo, Geonhee Ahn, Hyunsoo Cho +1Gender Bias in Language ModelsLanguage Model Bias Evaluation

  5. CARTBench: Rubric-Based Evaluation of Shopping Advice for Complex and Ambiguous Requests with Timelines

    Nov 4, 2026Yuqin Dai, Zhanwei Zhang, Honghui Yang +9Large Language Model-Based RecommendationLLM Evaluation

  6. RubricWorld: World Model-Guided Rubric Optimization for LLM Graders

    Nov 1, 2026Hongji Pu, Hang Li, Yucheng Chu +1Rubric-Based EvaluationAutomated Essay Scoring

  7. Rubric-Based Assessment of Student Reports Using Small Language Models

    Nov 1, 2026Sandesh Kumar Kafle, Jan Vykopal, Valdemar ŠvábenskýAutomated GradingRubric-Based Evaluation

  8. What Makes Medical Examination Items Difficult? Using Large Language Models to Explore Patterns Associated With Empirical Item Difficulty

    Oct 29, 2026Xiaomeng Xiong, Marcus Walker, Yanlin JiangItem Difficulty EstimationMedical Education

  9. RubricAtlas: Mining Type-Level Rubrics from Expert Reports to Evaluate Deep Research Agents

    Oct 27, 2026Delin Wang, Sinuo Wang, Rui Sun +5Agent EvaluationRubric-Based Evaluation

  10. RubricCraft: Where Solutions Diverge, Criteria Emerge

    Oct 21, 2026Hosein Hasanbeig, Priyanshu Gupta, Ashish Tiwari +2Automatic Rubric GenerationRubric-Based Evaluation

  11. SciDiagBench: A Type-Specific Benchmark for Human-Aligned Evaluation of Scientific Diagrams

    Oct 20, 2026Hao Wang, Zixuan Li, Long Bai +4VLM EvaluationImage Generation Evaluation

  12. SMART: Semantic Segment Mapping and Assessment of Recognition Transcripts with Contextual Rubrics

    Oct 20, 2026Xiaoyu Gu, Chencheng Wang, Fuyang Li +4ASR EvaluationAutomatic Speech Recognition

  13. Inside the Scorer: Mechanistic Evidence on Rubric Control in LLM-Based Automated Essay Scoring from 25 Base Models

    Oct 19, 2026Chungche Huang, Po-Ya Angela Wang, Shu-Kai HsiehAutomated Essay ScoringRubric-Based Evaluation

  14. MindEval2: Clinically Grounded Patients and Rubrics for Evaluating Language Models in Mental Health Support

    Oct 15, 2026John Mendonça, Daniel Tschernutter, José Pombal +6Clinical Language Model EvaluationMulti-Turn Dialogue Evaluation

  15. Rubric Judgments Lack Uncertainty: Conformal Weighting for LLM Evaluation

    Oct 15, 2026Licheng Pan, Haocheng Yang, Haoxuan Li +7Language Model Generation EvaluationRubric-Based Evaluation

  16. Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation

    Oct 14, 2026Haocheng Yang, Licheng Pan, Yiming Dong +7Language Model Generation EvaluationRubric-Based Evaluation

  17. TeachGrade: Training-Free Short Answer Scoring with Teacher Evidence

    Oct 10, 2026Jinhao Yan, Zhihong Shen, Shengjie Wang +4Automated GradingLLM-as-a-Judge

  18. How to Build a Benchmark That Actually Measures Taste

    Oct 10, 2026Xiaofan Wu, Xi Zeng, Miaoxia Chen +8Benchmark DesignBenchmark Validity

  19. From Structure to Validity: Mechanism-Guided Generation and Refinement of LLM Rubrics

    Oct 9, 2026Feifei Qian, Xuan Chen, Lixin Cui +6Automatic Rubric GenerationRubric-Based Evaluation

  20. Evaluating Rubric Generation with Interventional Transfer

    Oct 7, 2026Erik Skalnes, Layne C. Price, Raviteja Anantha +1Rubric-Based EvaluationGenerative AI Evaluation

  21. Alice: A Large-Scale German Benchmark for Rubric-Based Multi-Dimensional Automatic Short Answer Scoring

    Oct 7, 2026Zhifan Sun, Sebastian Gombert, Jannik Lossjew +6Automated GradingEducational Assessment

  22. Rubric Spans are Label Representations: Joint LLM Encoding for Short Answer Scoring

    Oct 7, 2026Zhifan Sun, Sebastian Gombert, Fabian Zehner +3Automated GradingRubric-Based Evaluation

  23. RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generation

    Oct 4, 2026Haocheng Yang, Yuchao Zhang, Licheng Pan +8Reward HackingRubric-Based RL