Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLM Reward Models

Weijie Qiu; Dai Guan; Junxin Wang; Zhihang Li; Yongbo Gai; Mengyu Zhou; Erchao Zhao; Xiaoxi Jiang

cs.CVMar 17, 2026

Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLM Reward Models

Authors: Weijie Qiu, Dai Guan, Junxin Wang, Zhihang Li, Yongbo Gai, Mengyu Zhou, Erchao Zhao, Xiaoxi Jiang,

Organizations: Qwen Large Model Application Team, Alibaba · Institute of Automation, Chinese Academy of Sciences

Abstract

Generative reward models (GRMs) for vision-language models (VLMs) often evaluate outputs via a three-stage pipeline: rubric generation, criterion-based scoring, and a final verdict. However, the intermediate rubric is rarely optimized directly. Prior work typically either treats rubrics as incidental or relies on expensive LLM-as-judge checks that provide no differentiable signal and limited training-time guidance. We propose Proxy-GRM, which introduces proxy-guided rubric verification into Reinforcement Learning (RL) to explicitly enhance rubric quality. Concretely, we train lightweight proxy agents (Proxy-SFT and Proxy-RL) that take a candidate rubric together with the original query and preference pair, and then predict the preference ordering using only the rubric as evidence. The proxy's prediction accuracy serves as a rubric-quality reward, incentivizing the model to produce rubrics that are internally consistent and transferable. With ~50k data samples, Proxy-GRM reaches state-of-the-art results on the VL-Reward Bench, Multimodal Reward Bench, and MM-RLHF-Reward Bench, outperforming the methods trained on four times the data. Ablations show Proxy-SFT is a stronger verifier than Proxy-RL, and implicit reward aggregation performs best. Crucially, the learned rubrics transfer to unseen evaluators, improving reward accuracy at test time without additional training. Our code is available at https://github.com/Qwen-Applications/Proxy-GRM.

Figures & tables

Figure 1: Overview of rubric-quality reward modeling via a proxy. Top-left: Traditional GRM training uses reward only from the final answer; rubric quality is implicit/latent, making the supervision a black box and allowing the model to guess correct answers without reliable guidance. Top-right: Our framework explicitly generates a transferable rubric alongside the final answer and optimizes the GRM with a rubric-quality reward computed by a proxy model that predicts preference given the question and generated rubric; correct proxy predictions provide strong guidance to improve rubric quality. Bottom: Data-efficiency comparison on VL-RewardBench, MultiModal Reward Bench and MM-RLHF-Reward Bench shows our method (1 $\times$ data) outperforming baselines trained with $\geq$ 4 $\times$ more data.

Figure 2: Training pipeline for Proxy-GRM with transferable rubrics. (A) Data & distillation. We merge multiple VLM preference datasets and apply automated filters to obtain ${\sim}60\text{k}$ samples. A teacher then separates the data into Correct and Hard subsets, which are allocated to train the proxy agent and to cold-start / RL-train Proxy-GRM. (B) Proxy agent training. Starting from Qwen2.5-VL-7B-Instruct, we train a proxy agent via Proxy-SFT followed by Proxy-RL, producing an evaluator that provides rubric-quality assessment signals. (C) Proxy-GRM training. We first cold-start the GRM policy with SFT. We then perform reinforcement learning with accuracy, proxy (rubric-quality), and format rewards, and optimize with GRPO updates. (D) Transferable rubric. A rubric is transferable if an independent agent can follow it to select the correct answer; otherwise, it is non-transferable and may lead to wrong decisions despite appearing plausible.

Model	Proxy Agent	Data	VL-RewardBench		Multimodal	MM-RLHF-Reward Bench
Model	Proxy Agent	Size	Overall Acc	Macro Acc	Reward Bench	Acc	Acc+
Proprietary Models
GPT-4o-(2024-08-06)	None	-	$65.80$	$62.40$	$70.80$	$58.23$	$26.01$
Claude-3.5-Sonnet-(2024-06-22)	None	-	$55.30$	$53.60$	$71.50$	$62.94$	$26.11$
Claude-3.7-Sonnet	None	-	$66.31$	$66.53$	$71.90$	82.35	65.22
Open-source Models

Table 1: Combined Results on VL-RewardBench, Multimodal Reward Bench, and MM-RLHF-Reward Bench. Best results are marked with a green background and second best with a yellow background.

Figure 3: Qualitative comparison of rubrics generated under different proxy agent configurations. Given the same multimodal input with a pairwise preference (r1 is preferred), we compare four variants: RL (no proxy) produces rubrics with redundant criteria and an incorrect verdict; RL+Qwen2.5-VL-3B fails to correct repetition and distorts weight allocation, also yielding an incorrect answer; RL+Qwen2.5-VL-32B introduces a more informative criterion ( Contextual Awareness ) and arrives at the correct verdict; RL+Proxy-SFT generates the most discriminative rubric with Contextual Relevance , balanced weights, and a correct prediction. Stronger proxy agents guide the policy toward more specific and non-redundant rubrics.

Proxy Agents	Avg Score	VL-Reward Bench	Multimodal Reward Bench	MM-RLHF- Reward Bench
Proxy-SFT	81.26	75.22	85.62	82.94
Proxy-RL	$79.97$	$73.38$	$84.78$	$81.76$
Unified-Reward-SFT	$81.18$	$74.98$	85.62	82.94
Unified-Reward-Think	$79.77$	$73.14$	$85.00$	$81.18$
R1-Reward	$80.59$	$73.54$	$85.28$	82.94
Qwen2.5-VL-3B-Instruct	$77.87$	$72.25$	$84.88$	$76.48$

Table 2: Proxy Agent Selection Results using Proxy-GRM-RL as the base model. Best results are marked with a green background .

Config	$r_{acc}=1$		$r_{acc}\neq 1$		Zero-out $r_{proxy}$
Config	$r_{proxy}=1$	$r_{proxy}\neq 1$	$r_{proxy}=1$	$r_{proxy}\neq 1$	Zero-out $r_{proxy}$
fb 1	$1.5$	$0.5$	$-1.0$	$-1.0$	X
fb 2	$1.5$	$0.5$	$-1.5$	$-1.0$	X
fb 3	$1.5$	$0.5$	$-1.5$	$-2.0$	X
fb 4	$1.0$	$0.5$	$-1.0$	$-1.0$	X
fb 5	$1.5$	$0.5$	$-1.0$	$-1.0$	✓

Table 3: Reward feedback configurations. Each row specifies the composite reward value for the four possible combinations of accuracy reward (

r_{acc}

) and proxy reward (

r_{proxy}

) outcomes. The rightmost column indicates whether

r_{proxy}

is ultimately zeroed out in the final reward.

Figure 4: Comparison of average scores across different Proxy Reward Feedback on three benchmarks. Each subplot reports the mean score for each variant, with numeric annotations above bars. Triangular markers indicate the best and worst performing variants within each benchmark. Overall, the base model achieves the highest scores on VL-RewardBench and Instructability, while performance on MM-RLHF-RewardBench remains consistently high with smaller variation across variants.

Evaluator	Rubric Source	VL-Reward Bench	Multimodal Reward Bench	MM-RLHF- Reward Bench
Qwen2.5-VL-7B-Instruct	Qwen2.5-VL-7B-Instruct	36.89	70.98	40.00
	Qwen2.5-VL-32B-Instruct	41.06	71.08	41.18
	UnifiedReward-SFT	42.02	72.36	41.76
	Proxy-GRM-RL	52.33	79.46	71.03
	Proxy-GRM-RL + UnifiedReward-SFT	53.87	79.82	73.79
	Proxy-GRM-RL + Proxy-SFT	54.13	80.00	73.79

Table 4: Rubric Transferability Results on VL-Reward-Bench, Multimodal Reward Bench, MM-RLHF-Reward Bench.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: System Prompt for Distillation and RL.

Figure 6: System Prompt for Rubric Evaluation.

Split	Size	Purpose
Proxy-SFT training	5k	Supervised fine-tuning of the proxy agent
Proxy-RL training	10k	Reinforcement learning of the proxy agent
Proxy-GRM-SFT	10k	Supervised fine-tuning initialization of the policy model

Appendix

Table 5: Data allocation for Proxy model training and Proxy-GRM Training.

Hyperparameter	Value
Training framework	ms-swift
Epochs	1
Learning rate	$1\times 10^{-5}$
LR schedule	Cosine decay
Training samples	10k

Appendix

Table 12

Hyperparameter	Value
Training framework	verl
Epochs	1
Learning rate	$5\times 10^{-6}$
Rollout group size $G$	7
Training batch size	256
Mini-batch size	128

Appendix

Table 13

Model	Proxy Agent	$\textbf{Overall}_{\text{Acc}}$	$\textbf{Macro}_{\text{Acc}}$	General	Hallucination	Reasoning
Proprietary Models
GPT-4o (2024-08-06)	None	65.80	62.40	49.10	67.60	70.50
Claude-3.5-Sonnet (2024-06-22)	None	55.30	53.60	43.40	55.00	62.30
Claude-3.7-Sonnet	None	66.31	66.53	68.08	70.70	60.81
Gemini-1.5-Flash (2024-09-24)	None	57.60	55.30	47.80	59.60	58.40
Gemini-1.5-Pro (2024-09-24)	None	67.20	62.50	50.80	72.50	64.20

Appendix

Table 6: Results on VL-RewardBench. Best results are marked with a green background and second best with a yellow background.

Model	Proxy Agent	Overall	Correctness	Preference	Knowledge	Math	Coding	Safety	VQA
Proprietary Models
GPT-4o (2024-08-06)	None	70.8	62.6	69.0	72.0	67.6	62.1	74.8	87.2
Gemini-1.5-Pro (2024-09-24)	None	71.9	63.5	67.7	66.3	68.9	55.5	94.5	87.2
Claude-3.5-Sonnet (2024-06-22)	None	71.5	62.6	67.8	73.9	68.6	65.1	76.8	85.6
Claude-3.7-Sonnet	None	71.9	58.4	60.7	78.1	76.3	71.3	72.0	86.8
Open-source Reward Models

Appendix

Table 7: Results on Multimodal Reward Bench. Best results are marked with a green background and second best with a yellow background.

Model	Proxy Agent	Acc	Acc+	MCQ	Long	Short	Safety	Video
Proprietary Models
Gemini-2.0-Flash-Exp	None	44.71	13.04	33.33	45.94	67.64	43.75	32.00
GPT-4o (2024-08-06)	None	58.23	26.01	64.28	78.37	44.11	56.25	40.00
Claude-3.5-Sonnet (2024-06-22)	None	62.94	26.11	64.28	67.56	55.88	65.62	60.00
Claude-3.7-Sonnet	None	82.35	65.22	66.67	91.89	91.18	87.50	76.00
Open-source Reward Models

Appendix

Table 8: Results on MM-RLHF-Reward Bench. Best results are marked with a green background and second best with a yellow background.

Figure 7: Left. With Proxy-SFT, Unified-Reward-SFT, Qwen2.5-VL-7B-Instruct, or Qwen2.5-VL-32B-Instruct as the sole proxy model, early $r_{proxy}$ optimization shows high variance and occasional oscillations across updates. This suggests rubric-quality feedback is proxy-sensitive at the start of RL, and the proxy signal can be noisy or poorly calibrated before the policy produces consistently interpretable rubrics. Right. We evaluate Proxy-GRM-RL with different proxy-model combinations across three benchmarks. Contrary to the expectation that ensembling improves verification, performance consistently degrades when combining multiple proxy reward models. This implies heterogeneous proxies may be misaligned; aggregating them can introduce conflicting gradients and hinder learning, making a single well-matched proxy preferable.

Data Size	$\textbf{VL-RB}\penalty\ \textbf{Overall}_{\text{Acc}}$	$\textbf{VL-RB}\penalty\ \textbf{Macro}_{\text{Acc}}$	MM-RB Overall	MM-RLHF-RB Acc	MM-RLHF-RB Acc+
2.5k	74.58	73.91	85.36	81.76	56.52
5k	75.22	73.93	85.62	82.94	56.52
7.5k	72.81	72.15	85.48	81.18	52.17
10k	73.22	72.57	85.20	80.59	52.17

Appendix

Table 9: Ablation on Proxy-SFT training data size. Results reported for Proxy-GRM-RL with each Proxy-SFT variant as the proxy agent. Best results are marked with a green background .

Figure 8: Qualitative comparison of rubrics generated with different proxy configurations. Given the same multimodal input and a pairwise preference signal (r1 is preferred), we compare four variants: RL (no proxy) produces a rubric with duplicated criteria (e.g., repeated Accuracy of Identification ), yielding a weakly discriminative evaluation and an incorrect verdict; RL+Qwen2.5-VL-3B still fails to remove repetition, introduces an irrelevant criterion (e.g., Contextual Awareness ), and misallocates weights, also leading to an incorrect prediction; RL+Qwen2.5-VL-32B replaces redundant/irrelevant items with a more task-aligned and informative criterion ( Descriptive Precision ), improving discriminability and producing the correct verdict; RL+Proxy-SFT generates the most discriminative, non-redundant rubric with better-balanced weights and a correct prediction. Overall, stronger proxy agents guide the policy toward more specific, task-relevant, and non-redundant rubrics.

Figure 9: Qualitative comparison of rubrics generated under different proxy agent configurations (action understanding example). Given the same multimodal input and a pairwise preference signal (r1 is preferred), we compare four variants: RL (no proxy) produces a generic rubric that does not sufficiently penalize hallucinated actions, resulting in an incorrect preference judgment; RL+Qwen2.5-VL-3B over-emphasizes Contextual Awareness and shifts weights away from fabrication control, failing to correct the key error and still predicting the wrong winner; RL+Qwen2.5-VL-32B uses a more task-aligned rubric that prioritizes Avoidance of Fabrication alongside observation accuracy, correctly favoring r1 over r2’s invented “unloading/moving items” details; RL+Proxy-SFT yields the most robust and balanced rubric, explicitly discouraging invented actions and consistently producing the correct verdict. Overall, stronger proxy agents guide the policy toward more hallucination-sensitive and discriminative rubrics.

Figure 10: Qualitative comparison of rubrics generated under different proxy agent configurations (game-state reasoning example). Given the same multimodal input and a pairwise preference signal (r1 is preferred), we compare four variants: RL (no proxy) generates a rubric with an ordering/structure issue and a redundant criterion (repeated Contextual Awareness ), which weakens the evaluation and leads to an incorrect preference decision; RL+Qwen2.5-VL-3B inherits the same redundancy and fails to emphasize game-state grounding, again producing a wrong verdict; RL+Qwen2.5-VL-32B improves task alignment by focusing on Accuracy of Game State Interpretation and Avoidance of Misinformation , making the rubric more discriminative and correctly preferring r1 over r2’s incorrect “middle cell” suggestion; RL+Proxy-SFT yields the most balanced and informative rubric, refining instruction quality and explicitly penalizing misleading moves, and thus outputs the correct prediction. Overall, stronger proxy agents steer the policy toward more game-aware, non-redundant rubrics that better detect incorrect strategic advice.

Multimodal Reward Modeling

Multimodal reward modeling builds evaluators that score images, video, audio, and agent behavior against human preferences or task goals.

See all 92 papers

Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling
Jul 2, 2026Dazhi Fu, Jiuding Yang, Yiwen Guo +1Reward Modeling LLM-as-a-Judge
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · LIGHTSPEED · Independent Researcher
Support Vector Rubrics: Closing the Gap Between Self-Generated and Human Rubrics
Jun 6, 2026Mengyuan Sun, Yu Li, Zhuohao Yu +2Reward Modeling Preference Modeling
National Engineering Research Center for Software Engineering, Peking University · University of Science and Technology of China
Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR
May 19, 2026Utkarsh Tyagi, Xingang Guo, MohammadHossein Rezaei +5Reward Modeling Rubric-Based RL
Scale AI · Persona

Jul 2, 2026J/K move · Enter open · S save

Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling

Dazhi Fu, Jiuding Yang, Yiwen Guo, Jicong Fan

School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · LIGHTSPEED · Independent Researcher

Reward Modeling LLM-as-a-Judge Language Model Generation Evaluation Rubric-Based Evaluation Automatic Rubric Generation

Open paper AskPDF arXiv

Reliable reward and preference signals are critical for evaluating and optimizing large language models on open-ended tasks. Rubric-based judges offer a transparent way to decompose such judgments into explicit evaluation criteria, but existing annotation-free rubric generators typically rely on a single generic evaluator. As a result, they may overlook important dimensions of human preference, a failure mode we term dimensional blind spots. To address this limitation, we propose Multi-Role Rubric Generation (MRRG), a training-free and reference-free framework that elicits evaluation criteria from multiple complementary roles and consolidates them into an auditable rubric-based scorer. This scorer can be used both to validate pairwise preferences and to provide rewards for GRPO-style Reinforcement Learning with Verifiable Rewards (RLVR). Experiments on preference validation benchmarks show that MRRG consistently outperforms single-role rubric generation baselines across multiple backbone models. Further RLVR experiments demonstrate that MRRG yields a stronger reward signal for improving open-ended generation.

Abstract

Multimodal Reward Modeling

Explore similar work

Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling

Support Vector Rubrics: Closing the Gap Between Self-Generated and Human Rubrics

Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR