Rubric-Based Evaluation

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 64% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 136

All topics

Alice: A Large-Scale German Benchmark for Rubric-Based Multi-Dimensional Automatic Short Answer Scoring
Oct 7, 2026Zhifan Sun, Sebastian Gombert, Jannik Lossjew +6Automated Grading Educational Assessment
DIPF | Leibniz Institute for Research and Information in Education · IPN | Leibniz Institute for Science and Mathematics Education · Umeå University +1
Rubric Spans are Label Representations: Joint LLM Encoding for Short Answer Scoring
Oct 7, 2026Zhifan Sun, Sebastian Gombert, Fabian Zehner +3Automated Grading Rubric-Based Evaluation
DIPF | Leibniz Institute for Research and Information in Education · Centre for International Student Assessment (ZIB) · Computer Science Department & Studiumdigitale, Goethe University Frankfurt
RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generation
Oct 4, 2026Haocheng Yang, Yuchao Zhang, Licheng Pan +8Reward Hacking Rubric-Based RL
National University of Singapore · Beijing University of Chemical Technology · Zhejiang University +7
MatrixReward: Reward from Rubric Matrix for Open-Ended Generation
Sep 30, 2026Zihan Shen, Qi Liu, Zixuan Yang +4Open-Ended Generation Reward Modeling
Zhejiang University · Qwen Business Unit of Alibaba · Renmin University of China
A Proposed Rubric for Evaluating Expressed Clinical Reasoning in Large Language Model Responses
Sep 29, 2026Zhangshu Joshua Jiang, Zina Ibrahim, James T. TeoClinical Reasoning Rubric-Based Evaluation
DRIVE-Health CDT Department of Biostatistics and Health Informatics Institute of Psychiatry, Psychology and Neuroscience King’s College London, London, UK · Neurological Institute, Cleveland Clinic London, London, UK · King’s College London, London, UK +1
Mubric: Mutation Testing-Guided Rubric Generation for LLM Evaluation
Sep 29, 2026Jiayuxuan Yang, Jie M. Zhang, Yiling Lou +1Language Model Generation Evaluation Rubric-Based Evaluation
Tsinghua University · King’s College London · University of Illinois Urbana-Champaign
VISTA-Bench: Benchmarking Multilingual Image Translation with Image-Specific Rubrics
Sep 29, 2026Bo Lv, Mao Zheng, Zheng Li +3Multilingual VLM Evaluation Rubric-Based Evaluation
Foundation Model Department, Tencent
FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agents
Sep 28, 2026Hoyoung Lee, Suyeol Yun, Jack Haverty +17LLM Agent Evaluation Rubric-Based Evaluation
LinqAlpha · UNIST · MassMutual Life Insurance +9
SkillRubric: Co-Evolving Actor Guidance and Evaluator Rubrics for Multimodal Agents
Sep 28, 2026Bingqing Jiang, Guoxi Zhang, Jasper Wang +7Multimodal Reward Modeling Rubric-Based RL
The University of Hong Kong · WeChat, Tencent · Peking University +2
JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places
Sep 24, 2026Delip Rao, Chris Callison-BurchLLM Inference Efficiency LLM Evaluation
University of Pennsylvania
Feed the Panel Dimensions, Not Verdicts: Rubric-Decomposed Fusion of Vision-Language Aesthetic Judges
Sep 22, 2026Amit Jadhav, Shaurya Beriwala, Beomjin KimVLM Evaluation Vision-Language Models
Purdue University Fort Wayne
RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling
Sep 19, 2026Zhenchen Tang, Yang Li, Songlin Yang +6Reward Modeling Multimodal Reward Modeling
New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Tencent +1
The Complexity Kink: A Prompt-Side Structural Complexity Index for Code-Generation Reliability
Sep 17, 2026Michael Hernandez, Tian ZhaoRubric-Based Evaluation Code Generation Evaluation
University of Wisconsin–Milwaukee
ImpossibleRubrics: Stress-Testing Generated Rubrics as Reward Signals
Sep 15, 2026Bowen Qin, Yi Xie, Yesheng Liu +1Reward Hacking Language Model Generation Evaluation
National University of Singapore · Peking University · Institute of Automation, Chinese Academy of Sciences
Don't Count the Edits, Judge by the Outcome Alone: Reward-Based Evaluation for Grammatical Error Correction
Sep 14, 2026Hayeong Ryu, Sunhee Jo, Seunguk Yu +1Reward Modeling Reference-Free Evaluation
Department of Artificial Intelligence, Chung-Ang University · Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University
When Rubrics Fail: Hallucinations Reveal Blind Spots in Medical AI Evaluation
Sep 14, 2026Griffin Farrow, Lily Sijia Li, Jack Johnson +4LLM Hallucination Detection Rubric-Based Evaluation
University of Oxford
Vision-Language Models for Criterion-Level Grading of Handwritten Examinations in Outcome-Based Education
Sep 13, 2026Asif Hasan Tonmoy, Saad Ahmed, Md Khalid Syfullah +1VLM Evaluation Rubric-Based Evaluation
Rubric-Aligned Disentangled Evaluation of Human Simultaneous Interpreting
Sep 11, 2026Ziyu Zhang, Satoshi NakamuraRubric-Based Evaluation Automated Evaluation
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · Shenzhen Loop Area Institute, China
Vision-language models know more about agriculture than they show and rubric-grounded verifications close the gap
Sep 8, 2026Earl Ranario, Jared Smith, Lars Lundqvist +2Agricultural Image Analysis Plant Disease Classification
University of California, Davis
Bridging Network Psychometrics and Artificial Intelligence: An Ising-Potts Model with LLM-Derived Weights
Sep 8, 2026Matthias von DavierInter-Rater Reliability Psychometric Validation
Efficient Test-Time Adaptation through Human-AI Interaction
Sep 3, 2026Zora Zhiruo Wang, Apurva Gandhi, Rulin Shao +22Test-Time Adaptation Rubric-Based Evaluation
Carnegie Mellon University · University of Washington · Handshake AI +3
PhoenixNest-Video: Evidence-Grounded Multimodal Agent Framework for Automated Video Interview Assessment
Sep 2, 2026Fan Yuxuan, Huang Miaojun, Zhang Haimei +2Automated Evaluation Rubric-Based Evaluation
The Hong Kong University of Science and Technology (Guangzhou)
Towards a Reliable and Practical Eval Pipeline
Sep 1, 2026Emma Thuong Nguyen, Abhishek GhoseLLM Evaluation LLM-as-a-Judge
Salesforce
PaperGym: Rubric-Centered Evolution for Research-Plan Generation
Aug 31, 2026Yuhan Wang, Zhengxi Lu, Yuchen Yan +6AI Agents for Scientific Discovery Rubric-Based RL
Zhejiang University · Apple
Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents
Aug 31, 2026Xuehai Wang, Haowei Qin, Tongxin Liu +6AI Agent Evaluation Autonomous Scientific Discovery
1Zhejiang University · University of Electronic Science and Technology of China · 3Beijing University of Posts and Telecommunications +1
Small Language Models as Judges for Rubric-Based Reinforcement Learning
Aug 30, 2026Fengyu Xie, Yilun Zhao, Bingsen Chen +2LLM-as-a-Judge Language Model Generation Evaluation
New York University · Yale University
GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation
Aug 30, 2026Yifan Chen, Haitao Li, Qingyao Ai +4LLM-as-a-Judge Language Model Generation Evaluation
Department of Computer Science and Technology, Tsinghua University · National University of Singapore
ExecRubrics: Executable Tool-Augmented Rubrics for Verifiable and Efficient Long-Form Evaluation
Aug 23, 2026Kaustubh D. Dhole, Charles L. A. Clarke, Eugene Y. AgichteinLLM Evaluation Rubric-Based Evaluation
Department of Computer Science Emory University Atlanta, USA · School of Computer Science University of Waterloo Waterloo, Canada
Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judges
Aug 12, 2026Xi Chen, Jie Mu, Mo Xuan +1Human Preference Evaluation LLM-as-a-Judge
Ant Group
Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL
Aug 12, 2026Minglai Yang, Xinyu Guo, Utkarsh Tyagi +6Reward Hacking Rubric-Based RL
Scale AI · University of Arizona · University of Texas at Dallas

Oct 7, 2026, cs.CLJ/K move · Enter open · S save

Alice: A Large-Scale German Benchmark for Rubric-Based Multi-Dimensional Automatic Short Answer Scoring

Zhifan Sun, Sebastian Gombert, Jannik Lossjew, Tobias Wyrwich+5

DIPF | Leibniz Institute for Research and Information in Education · IPN | Leibniz Institute for Science and Mathematics Education · Umeå University+1

Automated Grading Educational Assessment Rubric-Based Evaluation LLM Evaluation

Open paper AskPDF arXiv

Automatic Short Answer Scoring (ASAS) is central to NLP for Education. However, openly available benchmarks remain scarce, and existing datasets largely address how well students answer a question directly rather than how well they master underlying concepts (knowledge elements) such as thermal energy or epistemic activities (skills) such as reasoning or claim. To address this gap, we introduce Alice, a large-scale, rubric-based German ASAS dataset that is pedagogically aligned and comprises three subtasks: (i) learning performance (Alice-LP), (ii) knowledge elements (Alice-KE), and (iii) skills (Alice-SK). We further formulate rubric-based ASAS as a rubric-retrieval task and benchmark the dataset with a range of language models, from encoder-only models to lightweight LLMs. We also benchmark the dataset with zero-shot prompting via LLMs and a standard classification baseline. The experiments show that LLMs, in particular, struggle to score knowledge elements and skills in the zero-shot setting. They also indicate that rubric text is often useful, especially for Alice-KE and Alice-SK, while on Alice-LP gains over sample-solution-focused inputs are more modest and vary by model and input format.

Momentum

Related topics

Latest papers 136