Rubric-Based RL

RL: Reinforcement Learning

Rubric-based reinforcement learning trains language and multimodal models using explicit criteria—such as factual accuracy, instruction coverage, or safe tool use—instead of relying only on a final answer or a single scalar score. Common approaches generate or adapt rubrics for each task, have language-model judges or deterministic verifiers score them, and optimize policies with methods such as GRPO; some methods also assign credit to reasoning steps, tokens, or agent actions. This provides more detailed supervision for open-ended tasks, including writing, research, medical reasoning, image generation, and tool use. The central tension is that rubrics can make rewards more informative and interpretable, but incomplete criteria, biased judges, and reward aggregation can still encourage gaming or behavior that scores well without improving overall quality.

Latest papers

104 papers30 papers in the last four weeks, up 131% on the four weeks before. 0.2% of all new papers.

Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria
May 8, 2026Juanxi Tian, Fengyuan Liu, Jiaming Han +6Policy Optimization Multimodal Reward Modeling
1Nanyang Technological University · 2Ant Group · 3MMLab, The Chinese University of Hong Kong +1
Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning
May 8, 2026Manish Bhattarai, Ismael Boureima, Nishath Rajiv Ranasinghe +2Reinforcement Learning RL for Language Model Reasoning
Los Alamos National Laboratory, Los Alamos, NM, USA.
Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks
Apr 3, 2026Tianze Xu, Yanzhao Zheng, Pengrui Lu +11RL for Language Models Rubric-Based RL
Shanghai Jiao Tong University · GAIR · Alibaba Group +2
REM-CTX: Automated Peer Review via Reinforcement Learning with Auxiliary Context
Mar 31, 2026Pawin Taechoyotin, Daniel E. AcunaAutomated Peer Review RL for Language Models
Department of Computer Science, University of Colorado Boulder, Boulder, CO 80309
PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalization
Mar 27, 2026Zelin Tan, Zhouliang Yu, Bohan Lin +9RL for Language Model Reasoning Process Reward Models
University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · The Chinese University of Hong Kong +2
Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMs
Mar 20, 2026Wenjian Zhang, Kongcheng Zhang, Jiaxin Qi +2Hindsight Experience Replay RL for Language Model Reasoning
Dalian University of Technology · Zhejiang University · Chinese Academy of Sciences
Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLM Reward Models
Mar 17, 2026Weijie Qiu, Dai Guan, Junxin Wang +6Multimodal Reward Modeling VLM Evaluation
Qwen Large Model Application Team, Alibaba · Institute of Automation, Chinese Academy of Sciences
Writer-R1: Dynamic Rubric Mining for Reinforcement Learning of Open-Ended Writing
Mar 16, 2026Jihao Zhao, Shuaishuai Zu, Zhiyuan Ji +2Memory-Augmented RL RL for Language Models
School of Information, Renmin University of China, Beijing, China · School of Information, BRAIN, Renmin University of China, Beijing, China
Beyond Verifiable Rewards: Rubric-Based GRM for Reinforced Fine-Tuning SWE Agents
Mar 13, 2026Jiawei Huang, Qingping Yang, Renjie Zheng +1Software Engineering Agents Reward Design for RL
ETH Zurich · ByteDance Seed
Scaling Agentic Capabilities, Not Context: Efficient Reinforcement Finetuning for Large Toolspaces
Mar 5, 2026Karan Gupta, Pranav Vajreshwari, Yash Pandya +3RL for Tool Use Agentic RL
Microsoft Research
Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategy
Mar 4, 2026Guangchen Lan, Lian Xiong, Xin Zhou +7Reinforcement Learning Rubric-Based RL
Amazon · University of California, San Diego
RuCL: Stratified Rubric-Based Curriculum Learning for Multimodal Large Language Model Reasoning
Feb 25, 2026Yukun Chen, Jiaming Li, Longze Chen +10VLM Reasoning Rubric-Based RL
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Alibaba Group +1
SibylSense: Adaptive Rubric Learning via Memory Tuning and Adversarial Probing
Feb 24, 2026Yifei Xu, Guilherme Potje, Shivam Shandilya +9Rubric-Based RL Reward Design for RL
Microsoft · University of California, Los Angeles
RubricRL: Simple Generalizable Rewards for Text-to-Image Generation
Nov 25, 2025Xuelu Feng, Yunsheng Li, Ziyu Wan +4Reward Modeling T2I Generation
University at Buffalo · Microsoft AI · Nikola Tesla STEM High School

May 8, 2026J/K move · Enter open · S save

Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria

Juanxi Tian, Fengyuan Liu, Jiaming Han, Yilei Jiang+5

1Nanyang Technological University · 2Ant Group · 3MMLab, The Chinese University of Hong Kong+1

Policy Optimization Multimodal Reward Modeling Rubric-Based RL Multimodal Generation RL for Image Generation Rubric-Based Evaluation

Open paper AskPDF arXiv

Aligning multimodal generative models with human preferences demands reward signals that respect the compositional, multi-dimensional structure of human judgment. Prevailing RLHF approaches reduce this structure to scalar or pairwise labels, collapsing nuanced preferences into opaque parametric proxies and exposing vulnerabilities to reward hacking. While recent Rubrics-as-Reward (RaR) methods attempt to recover this structure through explicit criteria, generating rubrics that are simultaneously reliable, scalable, and data-efficient remains an open problem. We introduce Auto-Rubric as Reward (ARR), a framework that reframes reward modeling from implicit weight optimization to explicit, criteria-based decomposition. Before any pairwise comparison, ARR externalizes a VLM's internalized preference knowledge as prompt-specific rubrics, translating holistic intent into independently verifiable quality dimensions. This conversion of implicit preference structure into inspectable, interpretable constraints substantially suppresses evaluation biases including positional bias, enabling both zero-shot deployment and few-shot conditioning on minimal supervision. To extend these gains into generative training, we propose Rubric Policy Optimization (RPO), which distills ARR's structured multi-dimensional evaluation into a robust binary reward, replacing opaque scalar regression with rubric-conditioned preference decisions that stabilize policy gradients. On text-to-image generation and image editing benchmarks, ARR-RPO outperforms pairwise reward models and VLM judges, demonstrating that explicitly externalizing implicit preference knowledge into structured rubrics achieves more reliable, data-efficient multimodal alignment, revealing that the bottleneck is the absence of a factorized interface, not a deficit of knowledge.