Rubric-Based RL

RL: Reinforcement Learning

Rubric-based reinforcement learning trains language and multimodal models using explicit criteria—such as factual accuracy, instruction coverage, or safe tool use—instead of relying only on a final answer or a single scalar score. Common approaches generate or adapt rubrics for each task, have language-model judges or deterministic verifiers score them, and optimize policies with methods such as GRPO; some methods also assign credit to reasoning steps, tokens, or agent actions. This provides more detailed supervision for open-ended tasks, including writing, research, medical reasoning, image generation, and tool use. The central tension is that rubrics can make rewards more informative and interpretable, but incomplete criteria, biased judges, and reward aggregation can still encourage gaming or behavior that scores well without improving overall quality.

Latest papers

104 papers30 papers in the last four weeks, up 131% on the four weeks before. 0.2% of all new papers.

SetGRPO: Calibrating Grader-Induced Policy Optimization
Oct 27, 2026Jiale Huang, Zixu Li, Zhiwei Chen +3Reinforcement Learning Group Relative Policy Optimization
Shandong University · National University of Singapore
Eat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-based RL
Oct 24, 2026Hyun Ryu, Jong-Heon Kim, Yiqing Xie +3Rubric-Based RL Reward Design for RL
KAIST AI · Carnegie Mellon University Language Technologies Institute
Intra-Rubric Reward Divergence: Mitigating Rubric Hacking through Latent Dimension Recovery and Balancing
Oct 19, 2026Jianqiang Xia, Han Xu, Dianxi ShiReward Hacking Rubric-Based RL
Shanghai Jiao Tong University · Baidu, Inc. · Intelligent Game and Decision Lab
HarnessRL: Co-Evolving Policies with a Just-in-Time Training Harness for Open-Ended RL
Oct 19, 2026Xiaoqiang Wang, Ting-En Lin, Yuchuan Wu +7Rubric-Based RL Reward Design for RL
Alibaba Token Hub, Alibaba Group · Université de Montréal · Mila - Quebec AI Institute
Rubrics Beyond Evaluation: Exploiting Rubric Structure in Reinforcement Learning
Oct 19, 2026Yuxun Huang, Yunyao Zhang, ZeLiang Chen +3Rubric-Based RL Credit Assignment in RL
Huazhong University of Science and Technology, Wuhan, China · La Trobe University, Melbourne, Australia
RARE: Reinforcement Learning with Adaptive Rubric Evolution for Open-Ended Generation
Oct 18, 2026Xueyun Tian, Bingbing Xu, Chuyue Li +9Rubric-Based RL Reward Design for RL
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · Qwen Business Unit of Alibaba +1
Expert-Guided Rubrics as Dense Reward Signals for Reinforcement Learning in Non-Verifiable Domains
Oct 17, 2026Ashish Bansal, Hossam Amer, Keerat Kaur Guliani +3RL for LLM Agents Reward Design for RL
The Vanguard Group Malvern, Pennsylvania, USA · The Vanguard Group Toronto, Ontario, Canada
Rubric Judgments Lack Uncertainty: Conformal Weighting for LLM Evaluation
Oct 15, 2026Licheng Pan, Haocheng Yang, Haoxuan Li +7Language Model Generation Evaluation Rubric-Based Evaluation
Zhejiang University · Xiaohongshu Inc · National University of Singapore +3
Judging the Judges: Confidence and Disagreement in Rubrics as Rewards
Oct 15, 2026Michael Tian-Yue Liu, Anne Harrington, Trevor Darrell +1Rubric-Based RL Reward Design for RL
UC Berkeley
Recursive Reward Improvement via Adversarial Meta-Prompt Evolution
Oct 15, 2026Wei Dai, Qianchu Liu, Guanghui Qin +4Recursive Self-Improvement Rubric-Based RL
Massachusetts Institute of Technology · Microsoft Research · Johns Hopkins University
PrismaCoder: Pushing the Boundaries of Open Multimodal Code Intelligence through Decomposed Rubric Synthesis
Oct 14, 2026Qiushi Sun, Jingyang Gong, Yang Liu +14Multimodal Large Language Models Code Generation
The University of Hong Kong · Nanjing University · Fudan University +3
CoCA: Criterion-Aware Credit Assignment for Rubric-Based Reinforcement Learning
Oct 13, 2026Guangcheng Zhu, Yingfan Ma, Xing Zheng +7RL for Language Models Rubric-Based RL
Zhejiang University · Ant Group
RELAY: Selective Self-Distillation with Adaptive Hand-off for Rubric-Based Reinforcement Learning
Oct 13, 2026Kunwang Zheng, Xinyu Sun, Yu Cao +7Adaptive Knowledge Distillation Rubric-Based RL
University of Science and Technology of China · Fudan University · Alibaba Group
AdaptEvo: Learning under Imperfect Supervision through Confidence-Adaptive Training with Knowledge Evolution
Oct 8, 2026AllSpark TeamMultimodal Content Moderation Reinforcement Learning
RewardWeaver: Long-Horizon Interactive Learning for Language Agents via Self-Evolving Reward Adaptation
Oct 7, 2026Hengbo Xiao, Boyao Zhang, Purui Liu +4RL for LLM Agents Rubric-Based RL
EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans
Oct 4, 2026Xuancheng Li, Beining Wang, Haitao Li +7Reward Modeling Process Reward Models
Department of Computer Science and Technology, Tsinghua University, Beijing, China · Tencent, Beijing, China
RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generation
Oct 4, 2026Haocheng Yang, Yuchao Zhang, Licheng Pan +8Reward Hacking Rubric-Based RL
National University of Singapore · Beijing University of Chemical Technology · Zhejiang University +7
Self-Evaluating Recursive Agents
Oct 4, 2026TianYi Lyu, Xiaozhe Li, Yang Li +7LLM Agent Training Recursive Language Models
Tongji University · Independent · Shanghai Artificial Intelligence Laboratory +3
Rubric-Based Optimization for Text-to-Music Generation
Oct 2, 2026Ping Wang, Guang Yang, Shao-Rong Su +3Text-to-Music Generation Rubric-Based RL
Paul G. Allen School of Computer Science & Engineering, University of Washington · Department of Electrical and Computer Engineering, University of Washington · Allen Institute for AI
MetaRubric: Learning to Reward for Rubric-Based Reinforcement Learning
Oct 2, 2026Yuxuan Fan, Jaehong YoonRubric-Based RL Reward Design for RL
NTU Singapore
OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillation
Oct 2, 2026Xinpeng Wang, Wei Shi, Yu-Chia Chen +3RL for Language Models Rubric-Based RL
New York University · Work done at Meta · Meta +1
RISED: RubrIcs for agentic multi-environment Selection and sElf-Distillation
Oct 1, 2026Jingtan Wang, Sirajul Salekin, Young mok Jung +3Reinforcement Learning Rubric-Based RL
National University of Singapore · Work done during an internship at Apple · Apple
MatrixReward: Reward from Rubric Matrix for Open-Ended Generation
Sep 30, 2026Zihan Shen, Qi Liu, Zixuan Yang +4Open-Ended Generation Reward Modeling
Zhejiang University · Qwen Business Unit of Alibaba · Renmin University of China
LexReward: A Taxonomy-Driven Reward Framework for Legal Language Models
Sep 30, 2026Yida Cai, Xin Dai, Bingxiang He +5Reward Modeling Legal NLP
Peking University · Tsinghua University · Northeastern University
Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubrics
Sep 30, 2026Maoqi Liu, Junwei He, Bowen Zhang +5Reward Hacking Rubric-Based RL
Beijing University of Posts and Telecommunications · ByteDance
Momentum-Coupled Rubric Adaptation for Detailed Image Captioning
Sep 29, 2026Zhenwen Ji, Lei Jin, Shanyong Wang +6Image Captioning Rubric-Based RL
Xiaohongshu Inc. · the Joint SDU-NTU Centre for Artificial Intelligence Research (C-FAIR), Shandong University
Rubric Rewards from Item Response Theory
Sep 28, 2026Milad Yazdani, Yaser Souri, Xiren Zhou +4Reward Modeling Rubric-Based RL
Department of Electrical and Computer Engineering, University of British Columbia · Microsoft · School of Biomedical Engineering, University of British Columbia
ARISE: Adapting to Evolving Capability Gaps in Agentic Reinforcement Learning
Sep 28, 2026Kun Feng, Yuchen Fang, Yiyang Tan +6Agentic RL Long-Horizon Agent Evaluation
ShanghaiTech University · Ant Group
SkillRubric: Co-Evolving Actor Guidance and Evaluator Rubrics for Multimodal Agents
Sep 28, 2026Bingqing Jiang, Guoxi Zhang, Jasper Wang +7Multimodal Reward Modeling Rubric-Based RL
The University of Hong Kong · WeChat, Tencent · Peking University +2
Dr.Credit: Rubric-Grounded Process Credit Assignment for Deep Research Agents
Sep 28, 2026Yingjian Zhu, Zhenyi Wang, Jiaxin Guo +6Deep Research Agents Step-Level Credit Assignment in RL
School of Artificial Intelligence, University of Chinese Academy of Sciences · State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences · Alibaba Token Hub, Alibaba Group +1

Oct 27, 2026J/K move · Enter open · S save

SetGRPO: Calibrating Grader-Induced Policy Optimization

Jiale Huang, Zixu Li, Zhiwei Chen, Zhiheng Fu+2

Shandong University · National University of Singapore

Reinforcement Learning Group Relative Policy Optimization RL Post-Training Rubric-Based RL

Open paper AskPDF arXiv

Rubric-based reinforcement learning uses fine-grained grader feedback to train large language models on open-ended tasks, but grader randomness can alter GRPO updates in ways that score discrepancies alone cannot characterize. Because GRPO jointly normalizes rewards within each response group and combines the resulting training coefficients with response-level gradients, a scoring discrepancy can shift the coefficients of the entire group and induce policy-dependent update changes. Selective execution further affects the direction-reversal risk of retained updates. To address this, we propose Set-Valued Group Relative Policy Optimization (SetGRPO). It comprises three modules: (a) Joint Reward Set Calibration constructs a joint uncertainty set for the training coefficients; (b) Geometry-Guided Safe Recovery maps candidate coefficients into the current policy's local update geometry and determines the largest safe recovery ratio for each response group; and (c) Selective Risk-Gated Optimization selects a global threshold through independent risk calibration to control the conditional direction-reversal risk of accepted updates while preserving the original GRPO objective. Under exchangeability, SetGRPO provides finite-sample marginal coverage for the joint coefficient set and simultaneous risk control across candidate thresholds. On ten open-ended benchmarks spanning three domains, SetGRPO achieves the highest macro average among all compared training methods on both Qwen3.5 and Llama3.2 model families.