Rubric-Based Evaluation

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 64% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 136

All topics

A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks
Jul 2, 2026Samiha A. Ismail, Fan X. Chen, Ali MeraliClinical Decision-Making Clinical Reasoning
Prime Analytics Consulting Limited · Talbert House; Thomas More University · MAKZ
SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use
Jul 2, 2026Jiayin Zhu, Kelong Mao, Yudong Guo +4LLM Agent Evaluation LLM Agent Skill Learning
1HKUST(GZ), 2JD.COM · ∗Yutao Yue is also affiliated with Institute of Deep Perception Technology, JITRI, Wuxi, China.
Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling
Jul 2, 2026Dazhi Fu, Jiuding Yang, Yiwen Guo +1Reward Modeling LLM-as-a-Judge
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · LIGHTSPEED · Independent Researcher
PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Tasks
Jun 30, 2026Apurva Gandhi, Vishwas Suryanarayanan, Raja Hasnain Anwar +6Computer-Use Agents Rubric-Based Evaluation
Carnegie Mellon University · Microsoft · UMass Amherst +3
Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?
Jun 29, 2026Yangda Peng, Yunjia Qi, Haotian Xia +8LLM-as-a-Judge AI Agent Benchmarks
Department of Computer Science and Technology, Tsinghua University · Tencent Hunyuan
PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception
Jun 26, 2026Yana Wei, Hongbo Peng, Yanlin Lai +14Rubric-Based Evaluation Automated Evaluation
Johns Hopkins University · StepFun · Tsinghua University +1
EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning
Jun 22, 2026Hongxin Ding, Baixiang Huang, Yue Fang +6RL for Language Models Rubric-Based RL
National Engineering Research Center of Software Engineering, Peking University, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education +2
Rubric-as-Experts: Case-Specific MQM Rubrics for Translation Quality Evaluation
Jun 19, 2026Weilu Xu, Yunzhi Shen, Xinye Wang +2LLM Evaluation Rubric-Based Evaluation
National Key Laboratory for Novel Software Technology, Nanjing University
RubricsTree: Scalable and Evolving Open-Ended Evaluation of Personal Health Agents across Health Memory and Medical Skills
Jun 16, 2026Weizhi Zhang, Zechen Li, Hamid Palangi +16AI Agent Evaluation Human-in-the-Loop Evaluation
1Google Research, 2University of Illinois Chicago, *Work done during an internship at Google, †Corresponding Author
LLM-as-Judge in Education: A Curriculum-Grounded Marking Pipeline
Jun 16, 2026Xiwei Xu, Chen Wang, Jacky Jiang +5LLM-as-a-Judge Educational Assessment
CSIRO, Australia · UNSW, Sydney, Australia · Studitory, Australia +1
LexRubric: A Rubric-Guided Diagnostic Benchmark for Open-Ended Legal Tasks
Jun 8, 2026Yifan Chen, Haitao Li, Yiran Hu +6LLM Evaluation Rubric-Based Evaluation
1Beijing University of Posts and Telecommunications · 2Tsinghua University · University of Waterloo +1
Reliable to Expressive: A Curriculum for Rubric-Following Safety Judges
Jun 8, 2026Yongtaek Lim, Hyeji Choi, Minwoo KimLLM-as-a-Judge LLM Safety Evaluation
AI Safety Team, DATUMO.INC, Seoul, South Korea.
ComplexConstraints and Beyond: Expert Rubrics for RLVR
Jun 8, 2026Sushant Mehta, Liudas Panavas, Suhaas Garre +1LLM-as-a-Judge Rubric-Based RL
Surge AI
From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape
Jun 7, 2026Hao Chen, Ziyu Han, Yukun Yan +3LLM Evaluation Rubric-Based Evaluation
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology · Department of Computer Science and Technology, Institute for AI, Tsinghua University
Support Vector Rubrics: Closing the Gap Between Self-Generated and Human Rubrics
Jun 6, 2026Mengyuan Sun, Yu Li, Zhuohao Yu +2Reward Modeling Rubric-Based Evaluation
National Engineering Research Center for Software Engineering, Peking University · University of Science and Technology of China
DEFINED: A Data-Efficient Computational Framework for Fine-Grained Creativity Assessment in Debate Scenarios
Jun 5, 2026Tongzhou Yu, Mingjia Li, Hong Qian +6Creativity Assessment Computational Creativity
Nanjing University Nanjing, China · Shanghai Innovation Institute Shanghai, China · East China Normal University Shanghai, China
MADRAG: Multi-Agent Debate with Retrieval-Augmented Generation for Training-Free Analytic Essay Scoring
Jun 4, 2026Ali Keramati, Shiyuan Zhou, Sharad Mehrotra +1Multi-Agent LLM Systems LLM-as-a-Judge
University of California, Irvine
EDIT: Evidence-Diagnosed Intervention Training for Rule-Faithful LLM Grading
Jun 4, 2026Zhihao Wu, Linhai Zhang, Taiyi Wang +4LLM-as-a-Judge Rubric-Based Evaluation
1King’s College London · University of Cambridge · 3AQA +1
Elmes*: Automated Construction of Fine-Grained Evaluation Rubrics for Large Language Models in Long-Tail Educational Scenarios
Jun 4, 2026Tao Liu, Ye Lu, Ruohua Zhang +4LLM Evaluation LLM-as-a-Judge
1Shanghai Institute of AI for Education, East China Normal University, Shanghai 200062, China · School of Computer Science and Technology, East China Normal University, Shanghai 200062, China · 3Shanghai Innovation Institute, Shanghai 200231, China
QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards
Jun 2, 2026Rongzhi Zhang, Rui Feng, Zhihan Zhang +8Reinforcement Learning RL for Language Model Reasoning
Amazon · Georgia Institute of Technology
BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents
Jun 2, 2026Alex Wang, Georg Meinhardt, Jacob Katz +4Benchmark Design Financial QA
1Rogo · 2OpenAI · †Work done while at Rogo.
Leveraging BART to Assess CS1 C++ Programming Assignments using Rubric-based Criteria
Jun 2, 2026Kelsey Rainey, Jesse RobertsComputer Science Education Multi-Task Learning
Department of Computer Science Tennessee Technological University Cookeville, TN
AI Rater Discrimination Depends on Scoring Protocol in Complex Clinical Decision-Making
Jun 2, 2026Sangwon Baek, Kyu Yeon Hur, Kyunga KimLLM-as-a-Judge Clinical Decision Support
Asclep Korea Inc., Changwon, Republic of Korea · Center for Data Science, New York University, New York, USA · Division of Endocrinology and Metabolism, Department of Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Korea +1
AnyAudio-Judge: A Dynamic Rubric-Based Benchmark and Evaluator for Audio Instruction Following
Jun 2, 2026Haitao Li, Tian Tan, Yuguang Yang +2Audio-Language Model Evaluation LLM-as-a-Judge
Zhejiang University · Shanghai Innovation Institute · Shanghai Jiao Tong University +1
LLMs as Teaching Assistants for Mathematics Exam Grading: Reliability, and Practical Usability
Jun 1, 2026Aastha Sapkota, M. G. Sarwar MurshedLLM Evaluation LLM-as-a-Judge
Computer Science Department University of Wisconsin - Green Bay, Green Bay, WI, USA
Preference-Aware Rubric Learning for Personalized Evaluation
May 29, 2026Yilun Qiu, Xiaoyan Zhao, Yang Zhang +7LLM Evaluation LLM Alignment
National University of Singapore · Xiaohongshu Inc. · The University of Tokyo
PReMISE: Policy Rubrics as Measurement Specifications for LLM Judges
May 29, 2026Swastik Roy, Rajkumar Pujari, Tharindu Kumarage +5LLM-as-a-Judge LLM Auditing
Amazon AGI
Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge
May 28, 2026Zijie Wang, Eduardo BlancoLLM-as-a-Judge Language Model Generation Evaluation
University of Arizona Department of Computer Science
Feedback-to-Rubrics: Can We Learn Expert Criteria from Inline Comments?
May 28, 2026Kotaro Yoshida, So Kuroki, Yuki Imajuku +4Rubric-Based Evaluation Automated Peer Review
Sakana AI · Institute of Science Tokyo
EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generation
May 28, 2026Xin Guan, Xiaomeng Hu, Shen Huang +6Open-Ended Generation RL for Language Models
Tongyi Lab, Alibaba Group

Jul 2, 2026, cs.AIJ/K move · Enter open · S save

A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks

Samiha A. Ismail, Fan X. Chen, Ali Merali

Prime Analytics Consulting Limited · Talbert House; Thomas More University · MAKZ

Clinical Decision-Making Clinical Reasoning Rubric-Based Evaluation Clinical Language Model Evaluation

Open paper AskPDF arXiv

Multiple-choice medical benchmarks are increasingly saturated, and recent rubric-based evaluations such as HealthBench have shown that open-ended clinical performance is far from solved - its "Hard" subset top score remains 32%. We present a small, deliberately difficult evaluation dataset of five clinician-authored clinical scenarios spanning four specialties (anaesthesia, internal/family medicine, emergency medicine, and obstetrics), each accompanied by an atomic, weighted, MECE rubric (25-62 criteria per task; 184 criteria total) authored from a clinician-drafted golden answer. We evaluate three frontier models: GPT 5.4, Claude Opus 4.7, and Gemini 3.1 Pro. Mean rubric pass rates were 0.47 (Claude), 0.38 (GPT), and 0.37 (Gemini). The central finding is an inversion of clinical priority: the highest-weighted (weight-5, critical) criteria passed at only 32.4-41.7%, while low-stakes weight-1 criteria passed at 80-90%. 55 of 108 critical (weight-5) criteria (51%) were satisfied by no model. Three LLM autoraters reproduced expert met/not-met labels on 92.8-94.6% of 552 graded criteria. We position this as a methods-and-preliminary-findings contribution: the five tasks demonstrate a scalable, defensible pipeline ready to develop into a large-scale benchmark.

Momentum

Related topics

Latest papers 136