Rubric-Based Evaluation

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 64% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 136

All topics

Learnable Assessment Skills for LLM-based Automated Scoring: Rubric Construction via Iterative Optimization
May 28, 2026Yun Wang, Xin Xia, Xuansheng Wu +2Automated Essay Scoring Rubric-Based Evaluation
School of Computing, University of Georgia, Athens, GA, USA · AI4STEM Education Center, University of Georgia, Athens, GA, USA · The Hong Kong Polytechnic University, Hong Kong, China
GrowLoop: Self-Evolving Conversation Evaluation Seeded by Human
May 26, 2026Yihang Lin, Yunze Gao, Zeyang Lin +3Benchmark Design Human-in-the-Loop Evaluation
Amap, Alibaba Group · The Chinese University of Hong Kong, Shenzhen
Focal Reward: Balanced Reinforcement Learning under Rubric-Based Rewards
May 26, 2026Yu Huang, Zihua Zhao, Zhaoxin Huan +9Multi-Objective Reinforcement Learning RL for Language Models
Shanghai Jiao Tong University · Ant Group
A Two-Phase Stability Study of LLM Judges and Bar Council Examiners on Thai Bar-Exam Free-Form Essays
May 25, 2026Pawitsapak Akarajaradwong, Wuttikrai Lertprasertphakorn, Chompakorn Chaksangchaichot +1Inter-Rater Reliability Annotator Disagreement
VISAI AI · Independent Researcher · Vidyasirimedhi Institute of Science and Technology
JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment
May 24, 2026Russell Yang, Ruishi Chen, Pierce Kelaita +6Human Preference Evaluation LLM Evaluation
Stanford University · Snorkel AI · Harvey
ARES: Automated Rubric Synthesis for Scalable LLM Reinforcement Learning
May 22, 2026Xiaoyuan Li, Keqin Bao, Moxin Li +5RL for Language Models Reinforcement Learning
University of Science and Technology of China · Alibaba Group · National University of Singapore
Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR
May 19, 2026Utkarsh Tyagi, Xingang Guo, MohammadHossein Rezaei +5Reward Modeling Rubric-Based RL
Scale AI · Persona
CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging
May 19, 2026Zhenyu Li, Aleksandar Cvejic, Zehui Chen +1Pairwise Preference Learning Human Preference Evaluation
KAUST · USTC
Automated Grading of Handwritten Mathematics Using Vision-Capable LLMs
May 18, 2026Jacob Levine, Miguel Aenlle, Craig Zilles +2VLM Evaluation Rubric-Based Evaluation
University of Illinois Urbana-Champaign
AMARIS: A Memory-Augmented Rubric Improvement System for Rubric-Based Reinforcement Learning
May 18, 2026Peilin Wu, Xinlu Zhang, Kun Wan +4RL for Language Models Reinforcement Learning
The University of Texas at Dallas · Department of Computer Science, University of California, Santa Barbara · Adobe Inc.
AutoRubric-T2I: Robust Rule-Based Reward Model for Text-to-Image Alignment
May 17, 2026Kuei-Chun Kao, Daixuan Huo, Yuanhao Ban +1VLM Evaluation Rubric-Based Evaluation
University of California, Los Angeles · Arena
Evaluating Deep Research Agents on Expert Consulting Work: A Benchmark with Verifiers, Rubrics, and Cognitive Traps
May 17, 2026Tanmay Asthana, Aman Saksena, Divyansh SahuAI Agent Reliability Deep Research Agents
Deccan AI Research
QQJ: Quantifying Qualitative Judgment for Scalable and Human-Aligned Evaluation of Generative AI
May 17, 2026Marjan Veysi, Pirooz Shamsinejadbabaki, Mohammad Zare +1LLM-as-a-Judge Human-in-the-Loop Evaluation
AI Lab, Arioobarzan Engineering Team, Shiraz, Iran · Department of Computer Engineering and Information Technology, Shiraz University of Technology, Shiraz, Iran · Department of Informatics, Bioengineering, Robotics and Systems Engineering, University of Genoa, Genoa, Italy
Reward Hacking in Rubric-Based Reinforcement Learning
May 12, 2026Anas Mahmoud, MohammadHossein Rezaei, Zihao Wang +3Reward Hacking Reinforcement Learning
Scale AI
RETUYT-INCO at BEA 2026 Shared Task 2: Meta-prompting in Rubric-based Scoring for German
May 11, 2026Ignacio Sastre, Ignacio Remersaro, Facundo Díaz +4LLM Prompting Rubric-Based Evaluation
Instituto de Computación, Facultad de Ingeniería, Universidad de la República Montevideo, Uruguay
RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards
May 11, 2026Gaotang Li, Bhavana Dalvi Mishra, Zifeng Wang +9Reinforcement Learning Rubric-Based RL
University of Illinois Urbana-Champaign · Google Cloud AI Research
RubricRefine: Improving Tool-Use Agent Reliability with Training-Free Pre-Execution Refinement
May 10, 2026Will LeVine, Brendan Evers, Sam Saltwick +1LLM Self-Refinement Rubric-Based Evaluation
Anduril Industries
DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification
May 10, 2026Rui Liu, Dian Yu, Zhenwen Liang +6Multimodal Large Language Models Multimodal Reward Modeling
1Tencent Hunyuan · University of Maryland, College Park · University of North Carolina, Chapel Hill
Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria
May 8, 2026Juanxi Tian, Fengyuan Liu, Jiaming Han +6Policy Optimization Multimodal Reward Modeling
1Nanyang Technological University · 2Ant Group · 3MMLab, The Chinese University of Hong Kong +1
Creating and Evaluating K-12 GenAI Assessment Graders Through Context Engineering
May 8, 2026Zewei Tian, Alex Liu, Lief Esbenshade +6Educational Assessment Generative AI in Education
University of Washington · Colleague AI
Think-with-Rubrics: From External Evaluator to Internal Reasoning Guidance
May 8, 2026Jiachen Yu, Zhihao Xu, Junjie Wang +1Instruction Following Rubric-Based Evaluation
1Tsinghua University · 2Renmin University of China
SCRuB: Social Concept Reasoning under Rubric-Based Evaluation
May 7, 2026Jamelle Watson-Daniels, Himaghna Bhattacharjee, Skyler Wang +11LLM Evaluation Rubric-Based Evaluation
1Meta · 2McGill University · 3Handshake AI +1
Quantifying the Statistical Effect of Rubric Modifications on Human-Autorater Agreement
May 7, 2026Jessica Huynh, Alfredo Gomez, Athiya Deviyani +3Inter-Rater Reliability LLM-as-a-Judge
Carnegie Mellon University · Google Research
Principles and Guidelines for Randomized Controlled Trials in AI Evaluation
May 3, 2026Christopher Kelly, Angelica Chowdhury, Alexandra Campili +5Human-in-the-Loop Evaluation Rubric-Based Evaluation
Independent · Dalhousie University · Northeastern University London +4
Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese
May 2, 2026Roseval Malaquias Junior, Giovana Kerche Bonás, Thales Sales Almeida +6LLM Evaluation LLM-as-a-Judge
Maritaca AI · Jusbrasil
Measurement Risk in Supervised Financial NLP: Rubric and Metric Sensitivity on JF-ICR
Apr 30, 2026Sidi Chang, Peiying Zhu, Yuxiao Chen +1LLM Evaluation Rubric-Based Evaluation
Blossom AI Labs, Tokyo, Japan · Blossom AI Labs, San Francisco, USA
Human-in-the-Loop Benchmarking of Heterogeneous LLMs for Automated Competency Assessment in Secondary Level Mathematics
Apr 29, 2026Jatin Bhusal, Nancy Mahatha, Aayush Acharya +1LLM Evaluation Educational Assessment
Research and Incubation Center (RAIN), Sunway College Kathmandu
SciEval: A Benchmark for Automatic Evaluation of K-12 Science Instructional Materials
Apr 28, 2026Zhaohui Li, Peng He, Zhiyuan Chen +4LLM Evaluation AI in Education
Department of Computer Science and Engineering, University at Buffalo, USA · Department of Teaching and Learning, Washington State University, USA · College of Chemistry, Beijing Normal University, China +1
Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks
Apr 27, 2026Lawrence Keunho Jang, Jing Yu Koh, Daniel Fried +1Computer-Use Agent Benchmarks Web Agent Benchmarks
Carnegie Mellon University
Case-Specific Rubrics for Clinical AI Evaluation: Methodology, Validation, and LLM-Clinician Agreement Across 823 Encounters
Apr 27, 2026Aaryan Shah, Andrew Hines, Alexia Downs +6LLM-as-a-Judge Human-AI Agreement
1Canvas Medical, San Francisco, CA, USA · Department of Biomedical Data Science, Stanford University, Stanford, CA, USA · 3XPC (X Primary Care) +2

May 28, 2026, cs.CLJ/K move · Enter open · S save

Learnable Assessment Skills for LLM-based Automated Scoring: Rubric Construction via Iterative Optimization

Yun Wang, Xin Xia, Xuansheng Wu, Xiaoming Zhai+1

School of Computing, University of Georgia, Athens, GA, USA · AI4STEM Education Center, University of Georgia, Athens, GA, USA · The Hong Kong Polytechnic University, Hong Kong, China

Automated Essay Scoring Rubric-Based Evaluation

Open paper AskPDF arXiv

LLM-based automated scoring approaches near-human performance, but scaling to new tasks remains bottlenecked by the per-item human configuration of upstream stages such as rubric construction. Human experts bypass this bottleneck through evaluation heuristics developed over extensive practice. We ask whether LLMs can learn similar heuristics directly from scoring experience, and formalize this as the concept of assessment skills: item-independent natural-language procedural knowledge that guides LLMs through specific stages of the scoring workflow. Focusing on rubric construction as a first instantiation, we propose an iterative framework that decomposes a skill into a fixed scaffold and learnable item-agnostic rules, refining the rules through LLM-driven diagnosis of scoring errors and validation-gated selection. The framework requires no expert-written rubric. On all ten ASAP-SAS items, optimized skills substantially improve LLM-based scoring and frequently surpass the dataset-provided expert rubric. Cross-item transfer experiments further reveal that learned skills capture both generalizable and item-specific patterns.

Momentum

Related topics

Latest papers 136