Generative reward models (GRMs) for vision-language models (VLMs) often evaluate outputs via a three-stage pipeline: rubric generation, criterion-based scoring, and a final verdict. However, the intermediate rubric is rarely optimized directly. Prior work typically either treats rubrics as incidental or relies on expensive LLM-as-judge checks that provide no differentiable signal and limited training-time guidance. We propose Proxy-GRM, which introduces proxy-guided rubric verification into Reinforcement Learning (RL) to explicitly enhance rubric quality. Concretely, we train lightweight proxy agents (Proxy-SFT and Proxy-RL) that take a candidate rubric together with the original query and preference pair, and then predict the preference ordering using only the rubric as evidence. The proxy's prediction accuracy serves as a rubric-quality reward, incentivizing the model to produce rubrics that are internally consistent and transferable. With ~50k data samples, Proxy-GRM reaches state-of-the-art results on the VL-Reward Bench, Multimodal Reward Bench, and MM-RLHF-Reward Bench, outperforming the methods trained on four times the data. Ablations show Proxy-SFT is a stronger verifier than Proxy-RL, and implicit reward aggregation performs best. Crucially, the learned rubrics transfer to unseen evaluators, improving reward accuracy at test time without additional training. Our code is available at https://github.com/Qwen-Applications/Proxy-GRM.
Figures & tables
Figure 1: Overview of rubric-quality reward modeling via a proxy. Top-left: Traditional GRM training uses reward only from the final answer; rubric quality is implicit/latent, making the supervision a black box and allowing the model to guess correct answers without reliable guidance. Top-right: Our framework explicitly generates a transferable rubric alongside the final answer and optimizes the GRM with a rubric-quality reward computed by a proxy model that predicts preference given the question and generated rubric; correct proxy predictions provide strong guidance to improve rubric quality. Bottom: Data-efficiency comparison on VL-RewardBench, MultiModal Reward Bench and MM-RLHF-Reward Bench shows our method (1 × data) outperforming baselines trained with ≥ 4 × more data.
Figure 2: Training pipeline for Proxy-GRM with transferable rubrics. (A) Data & distillation. We merge multiple VLM preference datasets and apply automated filters to obtain ∼60k samples. A teacher then separates the data into Correct and Hard subsets, which are allocated to train the proxy agent and to cold-start / RL-train Proxy-GRM. (B) Proxy agent training. Starting from Qwen2.5-VL-7B-Instruct, we train a proxy agent via Proxy-SFT followed by Proxy-RL, producing an evaluator that provides rubric-quality assessment signals. (C) Proxy-GRM training. We first cold-start the GRM policy with SFT. We then perform reinforcement learning with accuracy, proxy (rubric-quality), and format rewards, and optimize with GRPO updates. (D) Transferable rubric. A rubric is transferable if an independent agent can follow it to select the correct answer; otherwise, it is non-transferable and may lead to wrong decisions despite appearing plausible.
Model
Proxy Agent
Data
VL-RewardBench
Multimodal
MM-RLHF-Reward Bench
Size
Overall Acc
Macro Acc
Reward Bench
Acc
Acc+
Proprietary Models
GPT-4o-(2024-08-06)
None
-
65.80
62.40
70.80
58.23
26.01
Claude-3.5-Sonnet-(2024-06-22)
None
-
55.30
53.60
71.50
62.94
26.11
Claude-3.7-Sonnet
None
-
66.31
66.53
71.90
82.35
65.22
Open-source Models
Table 1: Combined Results on VL-RewardBench, Multimodal Reward Bench, and MM-RLHF-Reward Bench. Best results are marked with a green background and second best with a yellow background.
Figure 3: Qualitative comparison of rubrics generated under different proxy agent configurations. Given the same multimodal input with a pairwise preference (r1 is preferred), we compare four variants: RL (no proxy) produces rubrics with redundant criteria and an incorrect verdict; RL+Qwen2.5-VL-3B fails to correct repetition and distorts weight allocation, also yielding an incorrect answer; RL+Qwen2.5-VL-32B introduces a more informative criterion ( Contextual Awareness ) and arrives at the correct verdict; RL+Proxy-SFT generates the most discriminative rubric with Contextual Relevance , balanced weights, and a correct prediction. Stronger proxy agents guide the policy toward more specific and non-redundant rubrics.
Proxy Agents
Avg Score
VL-Reward Bench
Multimodal Reward Bench
MM-RLHF- Reward Bench
Proxy-SFT
81.26
75.22
85.62
82.94
Proxy-RL
79.97
73.38
84.78
81.76
Unified-Reward-SFT
81.18
74.98
85.62
82.94
Unified-Reward-Think
79.77
73.14
85.00
81.18
R1-Reward
80.59
73.54
85.28
82.94
Qwen2.5-VL-3B-Instruct
77.87
72.25
84.88
76.48
Table 2: Proxy Agent Selection Results using Proxy-GRM-RL as the base model. Best results are marked with a green background .
Config
racc=1
racc=1
Zero-out rproxy
rproxy=1
rproxy=1
rproxy=1
rproxy=1
fb 1
1.5
0.5
−1.0
−1.0
X
fb 2
1.5
0.5
−1.5
−1.0
X
fb 3
1.5
0.5
−1.5
−2.0
X
fb 4
1.0
0.5
−1.0
−1.0
X
fb 5
1.5
0.5
−1.0
−1.0
✓
Table 3: Reward feedback configurations. Each row specifies the composite reward value for the four possible combinations of accuracy reward ( racc ) and proxy reward ( rproxy ) outcomes. The rightmost column indicates whether rproxy is ultimately zeroed out in the final reward.
Figure 4: Comparison of average scores across different Proxy Reward Feedback on three benchmarks. Each subplot reports the mean score for each variant, with numeric annotations above bars. Triangular markers indicate the best and worst performing variants within each benchmark. Overall, the base model achieves the highest scores on VL-RewardBench and Instructability, while performance on MM-RLHF-RewardBench remains consistently high with smaller variation across variants.
Supervised fine-tuning initialization of the policy model
Appendix
Table 5: Data allocation for Proxy model training and Proxy-GRM Training.
Hyperparameter
Value
Training framework
ms-swift
Epochs
1
Learning rate
1×10−5
LR schedule
Cosine decay
Training samples
10k
Appendix
Table 12
Hyperparameter
Value
Training framework
verl
Epochs
1
Learning rate
5×10−6
Rollout group size G
7
Training batch size
256
Mini-batch size
128
Appendix
Table 13
Model
Proxy Agent
OverallAcc
MacroAcc
General
Hallucination
Reasoning
Proprietary Models
GPT-4o (2024-08-06)
None
65.80
62.40
49.10
67.60
70.50
Claude-3.5-Sonnet (2024-06-22)
None
55.30
53.60
43.40
55.00
62.30
Claude-3.7-Sonnet
None
66.31
66.53
68.08
70.70
60.81
Gemini-1.5-Flash (2024-09-24)
None
57.60
55.30
47.80
59.60
58.40
Gemini-1.5-Pro (2024-09-24)
None
67.20
62.50
50.80
72.50
64.20
Appendix
Table 6: Results on VL-RewardBench. Best results are marked with a green background and second best with a yellow background.
Model
Proxy Agent
Overall
Correctness
Preference
Knowledge
Math
Coding
Safety
VQA
Proprietary Models
GPT-4o (2024-08-06)
None
70.8
62.6
69.0
72.0
67.6
62.1
74.8
87.2
Gemini-1.5-Pro (2024-09-24)
None
71.9
63.5
67.7
66.3
68.9
55.5
94.5
87.2
Claude-3.5-Sonnet (2024-06-22)
None
71.5
62.6
67.8
73.9
68.6
65.1
76.8
85.6
Claude-3.7-Sonnet
None
71.9
58.4
60.7
78.1
76.3
71.3
72.0
86.8
Open-source Reward Models
Appendix
Table 7: Results on Multimodal Reward Bench. Best results are marked with a green background and second best with a yellow background.
Model
Proxy Agent
Acc
Acc+
MCQ
Long
Short
Safety
Video
Proprietary Models
Gemini-2.0-Flash-Exp
None
44.71
13.04
33.33
45.94
67.64
43.75
32.00
GPT-4o (2024-08-06)
None
58.23
26.01
64.28
78.37
44.11
56.25
40.00
Claude-3.5-Sonnet (2024-06-22)
None
62.94
26.11
64.28
67.56
55.88
65.62
60.00
Claude-3.7-Sonnet
None
82.35
65.22
66.67
91.89
91.18
87.50
76.00
Open-source Reward Models
Appendix
Table 8: Results on MM-RLHF-Reward Bench. Best results are marked with a green background and second best with a yellow background.
Figure 7: Left. With Proxy-SFT, Unified-Reward-SFT, Qwen2.5-VL-7B-Instruct, or Qwen2.5-VL-32B-Instruct as the sole proxy model, early rproxy optimization shows high variance and occasional oscillations across updates. This suggests rubric-quality feedback is proxy-sensitive at the start of RL, and the proxy signal can be noisy or poorly calibrated before the policy produces consistently interpretable rubrics. Right. We evaluate Proxy-GRM-RL with different proxy-model combinations across three benchmarks. Contrary to the expectation that ensembling improves verification, performance consistently degrades when combining multiple proxy reward models. This implies heterogeneous proxies may be misaligned; aggregating them can introduce conflicting gradients and hinder learning, making a single well-matched proxy preferable.
Data Size
VL-RB\penaltyOverallAcc
VL-RB\penaltyMacroAcc
MM-RB Overall
MM-RLHF-RB Acc
MM-RLHF-RB Acc+
2.5k
74.58
73.91
85.36
81.76
56.52
5k
75.22
73.93
85.62
82.94
56.52
7.5k
72.81
72.15
85.48
81.18
52.17
10k
73.22
72.57
85.20
80.59
52.17
Appendix
Table 9: Ablation on Proxy-SFT training data size. Results reported for Proxy-GRM-RL with each Proxy-SFT variant as the proxy agent. Best results are marked with a green background .
Figure 8: Qualitative comparison of rubrics generated with different proxy configurations. Given the same multimodal input and a pairwise preference signal (r1 is preferred), we compare four variants: RL (no proxy) produces a rubric with duplicated criteria (e.g., repeated Accuracy of Identification ), yielding a weakly discriminative evaluation and an incorrect verdict; RL+Qwen2.5-VL-3B still fails to remove repetition, introduces an irrelevant criterion (e.g., Contextual Awareness ), and misallocates weights, also leading to an incorrect prediction; RL+Qwen2.5-VL-32B replaces redundant/irrelevant items with a more task-aligned and informative criterion ( Descriptive Precision ), improving discriminability and producing the correct verdict; RL+Proxy-SFT generates the most discriminative, non-redundant rubric with better-balanced weights and a correct prediction. Overall, stronger proxy agents guide the policy toward more specific, task-relevant, and non-redundant rubrics.
Figure 9: Qualitative comparison of rubrics generated under different proxy agent configurations (action understanding example). Given the same multimodal input and a pairwise preference signal (r1 is preferred), we compare four variants: RL (no proxy) produces a generic rubric that does not sufficiently penalize hallucinated actions, resulting in an incorrect preference judgment; RL+Qwen2.5-VL-3B over-emphasizes Contextual Awareness and shifts weights away from fabrication control, failing to correct the key error and still predicting the wrong winner; RL+Qwen2.5-VL-32B uses a more task-aligned rubric that prioritizes Avoidance of Fabrication alongside observation accuracy, correctly favoring r1 over r2’s invented “unloading/moving items” details; RL+Proxy-SFT yields the most robust and balanced rubric, explicitly discouraging invented actions and consistently producing the correct verdict. Overall, stronger proxy agents guide the policy toward more hallucination-sensitive and discriminative rubrics.
Figure 10: Qualitative comparison of rubrics generated under different proxy agent configurations (game-state reasoning example). Given the same multimodal input and a pairwise preference signal (r1 is preferred), we compare four variants: RL (no proxy) generates a rubric with an ordering/structure issue and a redundant criterion (repeated Contextual Awareness ), which weakens the evaluation and leads to an incorrect preference decision; RL+Qwen2.5-VL-3B inherits the same redundancy and fails to emphasize game-state grounding, again producing a wrong verdict; RL+Qwen2.5-VL-32B improves task alignment by focusing on Accuracy of Game State Interpretation and Avoidance of Misinformation , making the rubric more discriminative and correctly preferring r1 over r2’s incorrect “middle cell” suggestion; RL+Proxy-SFT yields the most balanced and informative rubric, refining instruction quality and explicitly penalizing misleading moves, and thus outputs the correct prediction. Overall, stronger proxy agents steer the policy toward more game-aware, non-redundant rubrics that better detect incorrect strategic advice.
Reliable reward and preference signals are critical for evaluating and optimizing large language models on open-ended tasks. Rubric-based judges offer a transparent way to decompose such judgments into explicit evaluation criteria, but existing annotation-free rubric generators typically rely on a single generic evaluator. As a result, they may overlook important dimensions of human preference, a failure mode we term dimensional blind spots. To address this limitation, we propose Multi-Role Rubric Generation (MRRG), a training-free and reference-free framework that elicits evaluation criteria from multiple complementary roles and consolidates them into an auditable rubric-based scorer. This scorer can be used both to validate pairwise preferences and to provide rewards for GRPO-style Reinforcement Learning with Verifiable Rewards (RLVR). Experiments on preference validation benchmarks show that MRRG consistently outperforms single-role rubric generation baselines across multiple backbone models. Further RLVR experiments demonstrate that MRRG yields a stronger reward signal for improving open-ended generation.
Dazhi Fu, Jiuding Yang, Yiwen Guo +1
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · LIGHTSPEED · Independent Researcher
Rubric-based evaluation is a promising paradigm for judging large language model (LLM) outputs, yet self-generated rubrics lag human-annotated criteria on hard instances. We argue this discriminative gap reflects an objective mismatch: self-generated rubrics describe good responses, whereas effective criteria must discriminate between close candidates. To close this gap, we introduce SVR (Support Vector Rubrics), a framework that recasts rubric construction as max-margin boundary learning over preference data. SVR mines contrastive features from preference pairs into a rubric bank, learns a prompt-conditioned selector together with global rubric weights, and iteratively refines the bank through support-pair selection and adversarial probing of hard negatives. At inference, given only the prompt, SVR retrieves the top-rubrics from the bank and scores responses. On RubricBench, SVR narrows the gap to human reference rubrics from 24.1 to 0.3 points and outperforms strong self-rubric and judge baselines, and the learned bank transfers across judges without retraining. On RewardBench 1&2, and RM-Bench, it remains competitive with dedicated reward models, demonstrating broader reward modeling capability. Overall, boundary-defining rubrics offer a principled route to closing the discriminative gap in LLM evaluation.
Mengyuan Sun, Yu Li, Zhuohao Yu +2
National Engineering Research Center for Software Engineering, Peking University · University of Science and Technology of China
Reinforcement learning with verifiable rewards has made post-training highly effective when correctness can be checked automatically. However, many important model behaviors require satisfying several qualitative criteria at once. Rubric-based rewards address this setting by grading prompt-specific criteria and aggregating them into a scalar reward. Yet standard static aggregations conflate a criterion's human-assigned importance with its current usefulness as an optimization signal. We show that this assumption breaks down in rubric RL: many important criteria are already saturated or currently unreachable, while criteria that distinguish rollouts are not necessarily those with the largest human weights. We introduce POW3R, a policy-aware rubric reward framework that preserves human weights and category balance as the rubric objective while adapting criterion-level reward weights during training. POW3R uses rollout-level contrast to emphasize criteria that currently separate the policy's outputs, making the GRPO reward more informative without changing the underlying evaluation target. Across three base policies on two datasets spanning multimodal and text-only settings, POW3R wins 24 of 30 base-policy/metric comparisons, improving both mean rubric reward and strict completion (the fraction of prompts whose response satisfies every required rubric criterion) over vanilla GRPO with rubric rewards, and reaches the same plateau in 2.5--4× fewer training steps. Rubric rewards should therefore distinguish what should matter in the final answer from what can teach the current policy.