Automatic Rubric Generation

Latest papers

26 papers5 papers in the last four weeks, up 67% on the four weeks before. 0.0% of all new papers.

From Structure to Validity: Mechanism-Guided Generation and Refinement of LLM Rubrics
Oct 9, 2026Feifei Qian, Xuan Chen, Lixin Cui +6Automatic Rubric Generation Rubric-Based Evaluation
School of Artificial Intelligence, Beijing Normal University · KE Holdings Inc. · School of Information, Central University of Finance and Economics
Evaluating Rubric Generation with Interventional Transfer
Oct 7, 2026Erik Skalnes, Layne C. Price, Raviteja Anantha +1Rubric-Based Evaluation Generative AI Evaluation
Johns Hopkins Computer Science · Amazon
RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generation
Oct 4, 2026Haocheng Yang, Yuchao Zhang, Licheng Pan +8Reward Hacking Rubric-Based RL
National University of Singapore · Beijing University of Chemical Technology · Zhejiang University +7
Mubric: Mutation Testing-Guided Rubric Generation for LLM Evaluation
Sep 29, 2026Jiayuxuan Yang, Jie M. Zhang, Yiling Lou +1Language Model Generation Evaluation Rubric-Based Evaluation
Tsinghua University · King’s College London · University of Illinois Urbana-Champaign
FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agents
Sep 28, 2026Hoyoung Lee, Suyeol Yun, Jack Haverty +17Automatic Rubric Generation LLM Agent Evaluation
LinqAlpha · UNIST · MassMutual Life Insurance +9
ImpossibleRubrics: Stress-Testing Generated Rubrics as Reward Signals
Sep 15, 2026Bowen Qin, Yi Xie, Yesheng Liu +1Reward Hacking Language Model Generation Evaluation
National University of Singapore · Peking University · Institute of Automation, Chinese Academy of Sciences
Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents
Aug 31, 2026Xuehai Wang, Haowei Qin, Tongxin Liu +6Automatic Rubric Generation AI Agent Evaluation
1Zhejiang University · University of Electronic Science and Technology of China · 3Beijing University of Posts and Telecommunications +1
GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation
Aug 30, 2026Yifan Chen, Haitao Li, Qingyao Ai +4LLM-as-a-Judge Language Model Generation Evaluation
Department of Computer Science and Technology, Tsinghua University · National University of Singapore
DecoEvo: Score-Decoupled Co-Evolution of Solver and Rubric-Generator Skills in Text Space
Jul 28, 2026Jiangwang Chen, Zixin Song, Junlin Liu +10Automatic Rubric Generation Large Language Model-Guided Optimization
1Tsinghua University · 4Qwen Business Unit of Alibaba · University of Chinese Academy of Sciences +1
Co-Evolving LLM Evaluators and Policies via DynamicRubric
Jul 22, 2026Beining Wang, Weihang Su, Hongtao Tian +8Automatic Rubric Generation RL for Language Models
Department of Computer Science and Technology, Tsinghua University · 2WeChat, Tencent
Rubrics on Trial: Evolving Rubrics from a Single Query via Synthetic Pairwise Evidence
Jul 16, 2026Haocheng Yang, Licheng Pan, Yuan Lu +7Rubric-Based Evaluation Automatic Rubric Generation
School of Computing, National University of Singapore · Xiaohongshu Inc. · School of Cyber Science and Technology, Zhejiang University +3
Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction
Jul 14, 2026Hanhua Hong, Yizhi Li, Jiaoyan Chen +4LLM Evaluation Rubric-Based Evaluation
The University of Manchester · IQuest Research · University of Information Technology, VNU +2
Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling
Jul 2, 2026Dazhi Fu, Jiuding Yang, Yiwen Guo +1Reward Modeling LLM-as-a-Judge
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · LIGHTSPEED · Independent Researcher
EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning
Jun 22, 2026Hongxin Ding, Baixiang Huang, Yue Fang +6Automatic Rubric Generation RL for Language Models
National Engineering Research Center of Software Engineering, Peking University, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education +2
Rubric-as-Experts: Case-Specific MQM Rubrics for Translation Quality Evaluation
Jun 19, 2026Weilu Xu, Yunzhi Shen, Xinye Wang +2Automatic Rubric Generation LLM Evaluation
National Key Laboratory for Novel Software Technology, Nanjing University
Are LLMs Bad at Moral Reasoning?
Jun 10, 2026Menghang Zhu, Seth LazarAutomatic Rubric Generation LLM Evaluation
School of Philosophy (Political Philosophy) Renmin University of China · School of Government and Policy Johns Hopkins University
Support Vector Rubrics: Closing the Gap Between Self-Generated and Human Rubrics
Jun 6, 2026Mengyuan Sun, Yu Li, Zhuohao Yu +2Reward Modeling Preference Modeling
National Engineering Research Center for Software Engineering, Peking University · University of Science and Technology of China
Elmes*: Automated Construction of Fine-Grained Evaluation Rubrics for Large Language Models in Long-Tail Educational Scenarios
Jun 4, 2026Tao Liu, Ye Lu, Ruohua Zhang +4Automatic Rubric Generation LLM Evaluation
1Shanghai Institute of AI for Education, East China Normal University, Shanghai 200062, China · School of Computer Science and Technology, East China Normal University, Shanghai 200062, China · 3Shanghai Innovation Institute, Shanghai 200231, China
Deep Research as Rubric for Reinforcement Learning
May 31, 2026Wangyi Mei, Zhouhong Gu, Zhenhan Bai +9Reward Design for RL Open-Ended Generation
Fudan University · Xiaohongshu Inc. · Beijing University of Posts and Telecommunications
Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge
May 28, 2026Zijie Wang, Eduardo BlancoAutomatic Rubric Generation LLM-as-a-Judge
University of Arizona Department of Computer Science
Feedback-to-Rubrics: Can We Learn Expert Criteria from Inline Comments?
May 28, 2026Kotaro Yoshida, So Kuroki, Yuki Imajuku +4Rubric-Based Evaluation Automated Peer Review
Sakana AI · Institute of Science Tokyo
Learnable Assessment Skills for LLM-based Automated Scoring: Rubric Construction via Iterative Optimization
May 28, 2026Yun Wang, Xin Xia, Xuansheng Wu +2Automatic Rubric Generation Automated Essay Scoring
School of Computing, University of Georgia, Athens, GA, USA · AI4STEM Education Center, University of Georgia, Athens, GA, USA · The Hong Kong Polytechnic University, Hong Kong, China
ARES: Automated Rubric Synthesis for Scalable LLM Reinforcement Learning
May 22, 2026Xiaoyuan Li, Keqin Bao, Moxin Li +5Automatic Rubric Generation RL for Language Models
University of Science and Technology of China · Alibaba Group · National University of Singapore
AutoRubric-T2I: Robust Rule-Based Reward Model for Text-to-Image Alignment
May 17, 2026Kuei-Chun Kao, Daixuan Huo, Yuanhao Ban +1VLM Evaluation Rubric-Based Evaluation
University of California, Los Angeles · Arena
RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation
Mar 21, 2026Kaustubh D. Dhole, Eugene AgichteinAutomatic Rubric Generation LLM Evaluation
Department of Computer Science, Emory University, Atlanta, GA, USA
RubricBench: Aligning Model-Generated Rubrics with Human Standards
Mar 2, 2026Qiyuan Zhang, Junyi Zhou, Yufei Wang +8LLM Evaluation Rubric-Based Evaluation
City University of Hong Kong · Tencent Hunyuan · McGill - Mila & Quebec AI Institute +2

Oct 9, 2026J/K move · Enter open · S save

From Structure to Validity: Mechanism-Guided Generation and Refinement of LLM Rubrics

Feifei Qian, Xuan Chen, Lixin Cui, Chaoyang Mei+5

School of Artificial Intelligence, Beijing Normal University · KE Holdings Inc. · School of Information, Central University of Finance and Economics

Automatic Rubric Generation Rubric-Based Evaluation LLM Evaluation Rubric-Guided Generation

Open paper AskPDF arXiv

Rubrics have been widely used to evaluate and optimize language models on open-ended tasks. However, existing work primarily focuses on generating better rubrics, with limited understanding of what makes rubric design effective. In this work, we conduct a systematic mechanistic analysis of rubric design across criterion organization, criterion operationalization, and score aggregation. Our analysis shows that structural organization enables more comprehensive criterion construction, semantic specification alone does not ensure effective criterion-level judgments, and sophisticated score aggregation provides limited additional gains. Guided by these findings, we introduce Adaptive Rubric Tree Search with Hybrid refinement (ARTS-Hybrid), which combines adaptive tree construction with semantic-first, response-calibrated refinement. Experiments across benchmarks show that ARTS-Hybrid improves both end-to-end evaluation performance and intrinsic rubric quality. Further experiments suggest that, beyond evaluation, rubrics generated by ARTS-Hybrid provide effective signals for model optimization.