Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Shanghai Jiao Tong University · Shadoweave · University of Electronic Science and Technology of China · The Hong Kong University of Science and Technology
Reinforcement learning with verifiable rewards (RLVR) improves multimodal reasoning, but uniformly assigning a single response-level advantage to all tokens overlooks the distinct roles of visual observations and subsequent reasoning. Visual affinity alone is insufficient: predictive states may align with both image and prompt-context representations, and upweighting valid visual observations can amplify their negative-advantage updates when later reasoning fails. To address this issue, we propose Outcome-Conditioned Visual Credit (OVC), a plug-in framework that separates visual-association measurement from token-level credit assignment and is adaptable to policy-optimization methods with uniform response-level advantage allocation. Relative Visual Encoding constructs a context-relative visual coordinate by contrasting each predictive state's affinity to image and prompt-context representations. Outcome-conditioned Evidence Calibration combines this coordinate with uncertainty-scaled predictive features and calibrates their contributions to outcome-discriminative evidence through mean contrasts between positive- and negative-advantage responses, scaled by regularized within-side variation. Visual association thus informs credit through observed outcome differences rather than directly determining update priority. Normalized Credit Allocation conditions the calibrated evidence on the response's advantage sign to produce positive token weights with unit global mean. These detached weights preserve advantage signs while modulating individual token-gradient contributions. OVC reuses existing rollouts without auxiliary reward models, process annotations, or inference-time modules. Instantiated through GSPO-token, OVC retains sequence-level clipping and consistently improves six-benchmark mean accuracy over GSPO across three models, with gains of 1.86-2.39 percentage points.
Figures & tables
Figure 1: Affinity and response correctness. We analyze 8 valid responses per randomly sampled question from a frozen Qwen3.5-9B trained with GSPO on Math77K, without correctness filtering. Affinity is siu=∣Ri∣−1∑t∈Ricos(hit,giu) , where hit predicts token t , giu is the mean-pooled image ( u=img ) or non-image prompt ( u=ctx ) prototype from the same intermediate layer, and Ri indexes reasoning tokens excluding special tokens and the final answer. No coordinate rescaling or image–context subtraction is applied. (a) Mean accuracy over four within-question image-affinity rank groups (two responses each). Each faint point is a group mean from a question-level bootstrap resample, with shared indices across groups. Violins show the distributions; dark markers are original-sample means. (b) Joint correctness of within-question pairs roughly matched on image affinity. Rows and columns represent lower- and higher-context-affinity response outcomes, respectively; cells report percentages of all matched pairs.
Figure 2: Overview of Outcome-Conditioned Visual Credit (OVC). An image–question pair produces a group of responses whose rewards yield group-relative advantages. For each response token, OVC combines relative image–context affinity with a prediction-aware representation, calibrates this evidence across advantage groups, and assigns normalized token weights to the policy update.
Method
MMMU
MathVision
We-Math
MathVista
MMK12-val
MathVerse
Avg.
Qwen3-VL-4B
Base
46.56
21.71
75.52
73.83
54.73
63.12
55.91
GSPO [ 55 ]
44.67
24.58
76.52
74.83
67.14
64.29
58.67 +2.76
DelTA [ 52 ]
44.33
25.00
76.00
74.00
74.58
64.67
59.76 +3.85
VPPO [ 11 ]
45.21
23.12
76.43
74.19
69.23
64.23
58.74 +2.83
AT-RL [ 13 ]
43.52
22.78
76.33
74.22
70.22
63.45
58.42 +2.51
Table 1: Main results on six multimodal benchmarks. Scores (%) are averaged over four evaluation sampling seeds at fixed checkpoints. Bold and underlined values indicate the best and second-best distinct scores within each model group. Avg. is the six-benchmark mean; red superscripts denote gains over Base. Qwen3.5-9B uses larger-scale Math77K training data. OOM denotes out of memory.
Configuration
MathVision
MathVista
MathVerse
MMK12-val
Avg.
Δ
Base
24.67
74.40
62.84
64.46
56.59
−6.13
GSPO (uniform credit)
27.30
76.10
63.53
74.73
60.42
−2.30
OVC (full)
29.93
76.97
65.05
78.94
62.72
—
(a) Relative visual encoding
No visual coordinate
28.21
74.97
63.46
75.43
60.52
−2.20
Absolute image affinity
28.03
75.63
63.98
76.31
60.99
−1.73
Table 2: Ablation study of OVC on Qwen3-VL-8B. Base and GSPO are from Table 1 . Scores are percentages; Avg. denotes the four-benchmark mean. Δ indicates the change from full OVC in percentage points. Best scores are bold. See Appendix A.2 for details.
Method
Chart QA-Pro
Char Xiv
Chart Museum
ChartX- QA
Evo Chart-QA
Chart Bench
Avg.
Base
37.69
62.66
36.83
63.69
47.39
62.08
51.72
GSPO
40.56
63.47
41.57
66.55
47.60
65.03
54.13 +2.41
OVC (Ours)
44.18
64.77
43.42
66.30
48.28
65.18
55.36 +3.64
Table 3: Chart-centric reasoning on Qwen3-VL-4B. Scores (%) are averaged over three evaluation sampling seeds. Avg. is the six-benchmark mean. Bold and underlined values denote the best and second-best results. Red superscripts indicate gains over Base.
Method
MMMU
Math Vision
We- Math
Math Vista
MMK12 -val
Math Verse
Avg.
DAPO
50.00
14.80
69.54
70.67
76.88
45.58
54.58
DAPO+OVC
52.78
20.39
70.92
71.30
76.58
45.89
56.31
GRPO
51.11
14.80
69.37
70.20
74.88
45.00
54.23
GRPO+OVC
52.22
15.13
70.63
70.27
75.50
46.17
54.99
Table 4: Extending OVC to alternative policy optimizers. Results on Qwen2.5-VL-7B trained on ViRL39K. Avg. is the mean across six benchmarks. Bold marks the better result in each pair.
Figure 3: Zero-shot transfer gains over Base. Bars show score changes relative to the corresponding instruction-tuned Base model. Avg. is the mean over four VQA benchmarks.
Figure 4: Qualitative visualization of within-response token-credit patterns. OVC correctly converts the bearing to the 65∘ triangle angle and obtains d=11.8 km, whereas DelTA first errs by using 25∘ as the triangle angle and obtains 25.3 km. Colors indicate the final normalized token-credit weights wit , recomputed offline from the corresponding checkpoints. Red boxes mark the key angle conversions and final answers. The OVC and DelTA responses have positive and negative group-relative advantages, respectively.
Science & Domain-Specific
Spatial & Embodied AI
Method
ScienceQA
RxnBench
SUPERChem
EmbSpatial
ERQA
Avg.
Qwen3.5-9B
92.88
87.08
10.20
73.19
44.50
61.57
GSPO
93.15
88.32
12.30
75.80
45.20
62.95
OVC (Ours)
93.89
89.05
13.40
77.45
45.75
63.91
Table 5: Zero-shot transfer to scientific and spatial benchmarks. GSPO and OVC are trained on Math77K using Qwen3.5-9B. Avg. denotes the mean score across five benchmarks. The best results are highlighted in bold.
Figure 5: Training dynamics on Qwen3.5-9B.
Figure 6: Training-time cost breakdown of OVC.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Split
Samples
Prompting protocol
Scoring protocol
MMMU
validation
900
multiple-choice prompting
deterministic accuracy
MathVerse
testmini
3,940
zero-shot direct query
DeepSeek V4 correctness judge
MathVision
testmini
304
original question
DeepSeek V4 correctness judge
We-Math
testmini
1,740
reasoning with explicit final answer
deterministic Loose accuracy
MathVista
testmini
1,000
three zero-shot prompting settings
answer extraction + deterministic scoring; mean over settings
MMK12-val
test
2,000
training validation prompt
validation accuracy
Appendix
Table 6: Benchmark protocol used for external evaluation.
Method
ChartQA
CharXiv
A-OKVQA
SimpleVQA
Avg.
Qwen3-VL-4B
Base
83.84
58.04
84.89
25.93
63.18
GSPO [ 55 ]
84.12
59.68
85.41
25.98
63.80
GSPO+OVC (Ours)
84.16
60.12
86.03
25.43
63.94
Qwen3-VL-8B
Base
85.24
65.68
87.95
27.70
66.64
Appendix
Table 7: Per-task zero-shot transfer results. CharXiv is the mean of its reasoning and descriptive scores. The corresponding gains over Base are visualized in Figure 3 .
Benchmark / task
Split or set
Questions
Scoring rule
ChartQA-Pro [ 23 ]
Evaluation set
1,948
Native scorer, including numerical tolerance and textual ANLS where applicable.
CharXiv-RQ [ 40 ]
Validation
1,000
Benchmark-specific reasoning rubric.
CharXiv-DQ [ 40 ]
Validation
4,000
Benchmark-specific descriptive rubric.
ChartMuseum [ 32 ]
Test
1,000
Benchmark-specific answer-comparison rubric.
ChartX-QA [ 43 ]
Test
1,152
QA scoring rubric, including 5% numerical tolerance.
EvoChart-QA [ 10 ]
Evaluation set
1,250
Exact matching or 5% numerical tolerance, depending on question type.
Appendix
Table 8: Chart-centric evaluation protocol. The six benchmarks comprise seven evaluated tasks.
Model
Training data
Prompt batch
Rollouts G
Learning rate
Max prompt
Max response
Checkpoint
Qwen2.5-VL-7B
ViRL39K
512
16
1×10−6
2,048
16384
step 76
Qwen3-VL-4B
ViRL39K
512
16
1×10−6
2,048
20,480
step 76
Qwen3-VL-8B
ViRL39K
512
16
1×10−6
2,048
8,192
step 76
Qwen3.5-9B
Math77K
256
8
1×10−6
4,096
16,384
step 300
Qwen3-VL-4B (Chart)
ChartVerse (18k)
256
16
1×10−6
16,384
16,384
step 72
Appendix
Table 9: Training recipes and evaluated checkpoints.
Quantity
Setting
Predictive layer and hidden width
ℓ=16 , H=4096
Responses per input
G=8
Global actor mini-batch
512 responses
Forward micro-batch per GPU
1 response
Reference refinements
K=1
Raw coefficient interval
[0.8,1.2]
Appendix
Table 10: Defaults of the supplied Qwen3.5-9B recipe.
Reinforcement learning with verifiable rewards scales multimodal reasoning, but an outcome reward says how much a trajectory is worth, not how that value should be spread over the decisions that produced it. We introduce Residual Visual Credit Optimization (RVCO), which treats token credit as a conserved routing problem. A controlled visual intervention yields a per-token evidence response; robust within-trajectory coordinates remove incidental scale; and a budgeted entropic router distributes a fixed amount of sequence utility according to perceptual dependence. A residual support path guarantees positive credit at every valid position, and an analytic correction restores the prescribed credit mass exactly. The resulting field is selective, bounded, full-support, and invariant to response-local score shifts, and recovers hard token selection as a limiting case. Across four model families and seven reasoning benchmarks, RVCO improves accuracy over strong RLVR baselines while maintaining late-stage optimization stability, corruption robustness, and competitive training cost. Rewards, rollouts, and the group-relative advantage estimator are unchanged; only the geometry of token-level credit differs.
Lin Qiu, Yao Liu, Diyi Hu +6
Meta Superintelligence Labs · University of Southern California · Meta Recommendation System +1
Reinforcement Learning with Verifiable Rewards (RLVR) has been extended to Large Vision-Language Models (LVLMs), and perception-aware methods further encourage policies to rely on visual evidence. Yet relying on the image does not guarantee that visual claims are supported by it. Before RL training, 27.81% of the correctly answered responses of Qwen2.5-VL-7B on four multimodal reasoning benchmarks contain at least one direct visual claim that the image does not support. Since outcome-level RL rewards each response as a whole, these claims inherit the positive credit of the correct answer. We introduce a fixed-rollout counterfactual diagnostic that re-scores the same response under an intervened image to separate Evidence-Function Sensitivity (EFS), how strongly the model's predictions change, from claim persistence, whether the model keeps supporting the same claim rather than retracting it. The diagnostic reveals Sensitivity-Persistence Decoupling (SPD): under DAPO and VPPO, EFS increases and claims become more retractable overall, yet unsupported claims become significantly more persistent, whereas GRPO raises EFS without this deterioration. We therefore propose Persistence-Aware Credit Gating (PACG), which attenuates positive credit for unusually persistent visual claims and leaves all other credit unchanged. It requires no supported/unsupported labels and adds no inference cost. On Qwen2.5-VL-7B, PACG raises the nine-benchmark average over three seeds from 58.1% to 59.9% with DAPO and from 59.8% to 60.9% with VPPO, while making unsupported claims more retractable. The gains extend to a larger model, a newer backbone, and the accuracy of HallusionBench also improves consistently. These results suggest that visual sensitivity and claim retractability are complementary dimensions of multimodal credit assignment.
Zhongan Bi, Kepeng Lin, Xuanang Gao +2
Zhejiang University · Huazhong University of Science and Technology · Shanghai Jiao Tong University
Reinforcement learning from verifiable rewards (RLVR), especially with Group Relative Policy Optimization (GRPO), has shown strong potential for improving the reasoning capabilities of large vision-language models (LVLMs). However, in multimodal reasoning, final-answer rewards are typically assigned at the sequence level and do not distinguish the functional roles of different tokens, making it difficult to determine whether a correct answer is supported by task-relevant visual evidence. In this paper, we revisit multimodal RLVR from the perspective of role-aware token-level credit assignment, where structured responses are decomposed into perception tokens for extracting visual evidence and reasoning tokens for deriving answers from that evidence. Based on this perspective, we propose Structured Role-aware Policy Optimization (SRPO), which refines the sequence-level GRPO advantage into role-aware token-level advantages without changing the reward function. Specifically, SRPO assigns role-specific credit by using self-distilled on-policy contrasts: perception tokens are emphasized according to their visual dependency under original versus corrupted visual inputs, while reasoning tokens are emphasized according to their consistency with the generated perception. These role-specific signals are further unified through a shared trajectory-level baseline, yielding positive token weights that adjust relative update magnitudes while preserving the original GRPO reward and optimization direction, without requiring external reward models or separate teachers. Experiments across diverse multimodal reasoning benchmarks show that SRPO improves evidence-grounded reasoning, highlighting the importance of moving beyond uniform sequence-level credit toward role-aware optimization for reliable multimodal reasoning.
Bingqing Jiang, Difan Zou
School of Computing & Data Science, The University of Hong Kong · School of Computing & Data Science and Institute of Data Science, The University of Hong Kong