Vision-language models can produce fluent answers that are insufficiently grounded in the visual evidence: a single unsupported object, chart value, or intermediate inference can undermine an otherwise plausible response. We argue that this is a credit-assignment failure in multimodal post-training. Scalar outcome rewards indicate whether an answer is acceptable, but do not identify which visual facts are grounded, which reasoning steps are valid, or which instruction constraints are missed. We introduce Visual Rubrics-Based Reinforcement Learning, which decomposes reference responses into atomic propositions and scores generated answers along Visual Faithfulness (VF), Reasoning Consistency (RC), and Instruction Following (IF). The resulting rubric items provide structured partial credit and localize rubric credit when supporting evidence spans are available. We first obtain an SFT checkpoint by fine-tuning Qwen3-VL-8B-Instruct on the public OpenMMReasoner-SFT-874K corpus, adapting OpenMMReasoner's cold-start data recipe. We construct V-Rubrics 50K, a 50,248-example training set from 17 visually grounded sources, by applying rule-based filters before deriving example difficulty from rejection-sampling scores and then annotating every example with Gemini-3-Pro under the same structured prompt and protocol. We train our model based on the same SFT checkpoint using component-wise, prefix-localized rubric credit. Experiments show that our rubricbased GRPO improves over both the shared SFT baseline and answer-only GRPO, with the largest gains on knowledge-oriented and visually grounded reasoning benchmarks. The results show rubrics as a useful reward abstraction for visual post-training.
Figures & tables
Figure 1: Rubric-grounded visual reasoning. A visual question answering example is decomposed into rubric items that check visual faithfulness, reasoning consistency, and instruction following.
Figure 2: Overview of Visual Rubrics-Based Reinforcement Learning. V-Rubrics 50K expands VQA to atomic VF/RC/IF rubric items, which provide fine-grained scores and prefix-localized credit for GRPO-based VLM post-training.
Figure 3: Data Distribution of V-Rubrics 50K. The inner ring shows the VF/RC/IF composition of 352,938 rubric items, while the thin outer ring shows the distribution of 50,248 examples across 17 canonical sources. Outer-ring arc length is proportional to each source’s percentage, with colors mapped to source names in the legend.
Model
SFT Data
RL Data
General VLM
Knowledge & Academic Reasoning
Overall Avg.
MMBench-Dev
MMMU Val
MMMU-Pro
MMMU-Pro V
Avg.
Closed-source models
GPT-4o ( OpenAI, 2024b )
–
–
88.4
69.1
54.0
49.7
57.60
65.30
GPT-4o mini ( OpenAI, 2024a )
–
–
83.8
59.4
39.9
35.2
44.83
54.58
Open-source Instruct models
LLaVA-OneVision-7B ( Li et al., 2024a )
4.8M
–
80.8
48.8
29.5
18.7
32.33
44.45
Table 1: Performance on general VLM and knowledge benchmarks. Within each model group, green and blue mark the best and second-best results; Knowledge Avg. is the mean of the three knowledge metrics, while Overall Avg. additionally includes MMBench-Dev.
Model
SFT Data
RL Data
Visual Math & Reasoning
Chart & Logic
Overall Avg.
MathVista mini
MathVision test
MathVerse V/O
DynaMath Worst
WeMath Loose
Math Avg.
Logic Vista
CharXiv Reas.
Chart Avg.
Closed-source models
GPT-4o ( OpenAI, 2024b )
–
–
63.8
31.1
40.6
34.5
62.8
46.56
64.4
–
–
–
GPT-4o mini ( OpenAI, 2024a )
–
–
55.1
27.3
30.0
31.6
48.8
38.56
41.4
34.1
37.75
38.33
Open-source Instruct models
LLaVA-OneVision-7B ( Li et al., 2024a )
4.8M
–
62.6
17.6
17.6
9.0
43.5
30.06
32.0
23.6
27.80
29.41
Table 2: Performance on visual math, chart, and logic benchmarks. Within each model group, green and blue mark the best and second-best results; Math Avg. is the mean of the five visual-math metrics, Chart Avg. the mean of LogicVista and CharXiv, and Overall Avg. the mean of all seven metrics.
Signal
Credit
Overall Avg.
Answer only
Sequence-level
66.25
Answer + rubrics
Sequence-level
67.74
Answer + rubrics
Component + prefix
68.04
Table 3: Reward and credit-assignment ablation.
Figure 4: Qualitative comparison between answer-only GRPO and rubric-based checkpoints. Top: rubric training improves reasoning consistency by making the intermediate inference explicit. Bottom: rubric training improves visual faithfulness by tracing the graph before answering.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Source
Final examples
Share
AI2D
1,758
3.5%
ChartGalaxy
1,289
2.6%
ChartNet
681
1.4%
ChartQA
687
1.4%
ChartQA-X
1,314
2.6%
ChartX
1,203
2.4%
Appendix
Table 4: Canonical source inventory of V-Rubrics 50K. Counts are computed from the 50,248 records in the current release; source shares are rounded independently to one decimal place.
Setting
Configuration
Initialization and data
Backbone before SFT
Qwen3-VL-8B-Instruct
SFT corpus
OpenMMReasoner-SFT-874K
RL corpus
V-Rubrics 50K
Actor initialization
πθ(0)=πSFT ; πθ is trainable
Reference policy
πref=πSFT at initialization and frozen thereafter
Appendix
Table 6: GRPO training configuration. Settings are shared by the rubric-based run and the answer-level baseline except where a row lists both variants separately.
Reinforcement Learning (RL) is an important paradigm for improving the reasoning capabilities of Vision-Language Models (VLMs). However, directly applying RL to rollout multimodal reasoning can lead to instability, due to the exploitation of language priors, the neglect of visual evidence, and the generation of reasoning traces that are fluent yet not visually grounded. The question arises: Can initially steer the policy toward visually faithful reasoning regime before applying reinforcement learning? To this end, we propose a Faithful Warm-Start (FWS) strategy that first curates samples with explicit vision-language causal relationships from six general VQA benchmarks to construct the FaithfulQA dataset, where each of the image-question pairs gains a certain degree of visual observations, question requirements, commonsense knowledge, domain knowledge, and the final answer. Subsequently, a VLM-based judge is employed to further purify the dataset, ensuring strong causal consistency and visual faithfulness. This warm-start stage equips the model with the capability to understand causally grounded vision-language patterns before subsequent RL optimization under sparse answer-level rewards. Experimental results show that such faithful supervision improves answer accuracy, stabilizes RL training, and reduces visually unsupported reasoning.
Peng, Lee, Yin Zhang +9
GMLab · Hong Kong Polytechnic University · XPENG Robotics
Generative reward models (GRMs) for vision-language models (VLMs) often evaluate outputs via a three-stage pipeline: rubric generation, criterion-based scoring, and a final verdict. However, the intermediate rubric is rarely optimized directly. Prior work typically either treats rubrics as incidental or relies on expensive LLM-as-judge checks that provide no differentiable signal and limited training-time guidance. We propose Proxy-GRM, which introduces proxy-guided rubric verification into Reinforcement Learning (RL) to explicitly enhance rubric quality. Concretely, we train lightweight proxy agents (Proxy-SFT and Proxy-RL) that take a candidate rubric together with the original query and preference pair, and then predict the preference ordering using only the rubric as evidence. The proxy's prediction accuracy serves as a rubric-quality reward, incentivizing the model to produce rubrics that are internally consistent and transferable. With ~50k data samples, Proxy-GRM reaches state-of-the-art results on the VL-Reward Bench, Multimodal Reward Bench, and MM-RLHF-Reward Bench, outperforming the methods trained on four times the data. Ablations show Proxy-SFT is a stronger verifier than Proxy-RL, and implicit reward aggregation performs best. Crucially, the learned rubrics transfer to unseen evaluators, improving reward accuracy at test time without additional training. Our code is available at https://github.com/Qwen-Applications/Proxy-GRM.
Weijie Qiu, Dai Guan, Junxin Wang +6
Qwen Large Model Application Team, Alibaba · Institute of Automation, Chinese Academy of Sciences
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a prevailing paradigm for enhancing reasoning in Multimodal Large Language Models (MLLMs). However, relying solely on outcome supervision risks reward hacking, where models learn spurious reasoning patterns to satisfy final answer checks. While recent rubric-based approaches offer fine-grained supervision signals, they suffer from high computational costs of instance-level generation and inefficient training dynamics caused by treating all rubrics as equally learnable. In this paper, we propose Stratified Rubric-based Curriculum Learning (RuCL), a novel framework that reformulates curriculum learning by shifting the focus from data selection to reward design. RuCL generates generalized rubrics for broad applicability and stratifies them based on the model's competence. By dynamically adjusting rubric weights during training, RuCL guides the model from mastering foundational perception to tackling advanced logical reasoning. Extensive experiments on various visual reasoning benchmarks show that RuCL yields a remarkable +7.83% average improvement over the Qwen2.5-VL-7B model, achieving a state-of-the-art accuracy of 60.06%.
Yukun Chen, Jiaming Li, Longze Chen +10
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Alibaba Group +1