From Visual Affinity to Learning Credit in Multimodal Reinforcement Learning

Yuhang Han; Dingning Liu; Yujie Chen; Yaojie Zhang; Jingwei Song; Zequn Yang; Xuanming Shang; Li Ding

cs.AIOct 23, 2026

From Visual Affinity to Learning Credit in Multimodal Reinforcement Learning

Authors: Yuhang Han, Dingning Liu, Yujie Chen, Yaojie Zhang, Jingwei Song, Zequn Yang, Xuanming Shang, Li Ding,

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Shanghai Jiao Tong University · Shadoweave · University of Electronic Science and Technology of China ·

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves multimodal reasoning, but uniformly assigning a single response-level advantage to all tokens overlooks the distinct roles of visual observations and subsequent reasoning. Visual affinity alone is insufficient: predictive states may align with both image and prompt-context representations, and upweighting valid visual observations can amplify their negative-advantage updates when later reasoning fails. To address this issue, we propose Outcome-Conditioned Visual Credit (OVC), a plug-in framework that separates visual-association measurement from token-level credit assignment and is adaptable to policy-optimization methods with uniform response-level advantage allocation. Relative Visual Encoding constructs a context-relative visual coordinate by contrasting each predictive state's affinity to image and prompt-context representations. Outcome-conditioned Evidence Calibration combines this coordinate with uncertainty-scaled predictive features and calibrates their contributions to outcome-discriminative evidence through mean contrasts between positive- and negative-advantage responses, scaled by regularized within-side variation. Visual association thus informs credit through observed outcome differences rather than directly determining update priority. Normalized Credit Allocation conditions the calibrated evidence on the response's advantage sign to produce positive token weights with unit global mean. These detached weights preserve advantage signs while modulating individual token-gradient contributions. OVC reuses existing rollouts without auxiliary reward models, process annotations, or inference-time modules. Instantiated through GSPO-token, OVC retains sequence-level clipping and consistently improves six-benchmark mean accuracy over GSPO across three models, with gains of 1.86-2.39 percentage points.

Figures & tables

Figure 1: Affinity and response correctness. We analyze 8 valid responses per randomly sampled question from a frozen Qwen3.5-9B trained with GSPO on Math77K, without correctness filtering. Affinity is $s_{i}^{u}=|\mathcal{R}_{i}|^{-1}\sum_{t\in\mathcal{R}_{i}}\cos(h_{it},g_{i}^{u})$ , where $h_{it}$ predicts token $t$ , $g_{i}^{u}$ is the mean-pooled image ( $u=\mathrm{img}$ ) or non-image prompt ( $u=\mathrm{ctx}$ ) prototype from the same intermediate layer, and $\mathcal{R}_{i}$ indexes reasoning tokens excluding special tokens and the final answer. No coordinate rescaling or image–context subtraction is applied. (a) Mean accuracy over four within-question image-affinity rank groups (two responses each). Each faint point is a group mean from a question-level bootstrap resample, with shared indices across groups. Violins show the distributions; dark markers are original-sample means. (b) Joint correctness of within-question pairs roughly matched on image affinity. Rows and columns represent lower- and higher-context-affinity response outcomes, respectively; cells report percentages of all matched pairs.

Figure 2: Overview of Outcome-Conditioned Visual Credit (OVC). An image–question pair produces a group of responses whose rewards yield group-relative advantages. For each response token, OVC combines relative image–context affinity with a prediction-aware representation, calibrates this evidence across advantage groups, and assigns normalized token weights to the policy update.

Qwen3-VL-4B
Method	MMMU	MathVision	We-Math	MathVista	MMK12-val	MathVerse	Avg.
Base	46.56	21.71	75.52	73.83	54.73	63.12	55.91
GSPO [ 55 ]	44.67	24.58	76.52	74.83	67.14	64.29	58.67 +2.76
DelTA [ 52 ]	44.33	25.00	76.00	74.00	74.58	64.67	59.76 +3.85
VPPO [ 11 ]	45.21	23.12	76.43	74.19	69.23	64.23	58.74 +2.83
AT-RL [ 13 ]	43.52	22.78	76.33	74.22	70.22	63.45	58.42 +2.51

Table 1: Main results on six multimodal benchmarks. Scores (%) are averaged over four evaluation sampling seeds at fixed checkpoints. Bold and underlined values indicate the best and second-best distinct scores within each model group. Avg. is the six-benchmark mean; red superscripts denote gains over Base. Qwen3.5-9B uses larger-scale Math77K training data. OOM denotes out of memory.

Configuration	MathVision	MathVista	MathVerse	MMK12-val	Avg.	$\Delta$
Base	24.67	74.40	62.84	64.46	56.59	$-6.13$
GSPO (uniform credit)	27.30	76.10	63.53	74.73	60.42	$-2.30$
OVC (full)	29.93	76.97	65.05	78.94	62.72	—
(a) Relative visual encoding
No visual coordinate	28.21	74.97	63.46	75.43	60.52	$-2.20$
Absolute image affinity	28.03	75.63	63.98	76.31	60.99	$-1.73$

Table 2: Ablation study of OVC on Qwen3-VL-8B. Base and GSPO are from Table 1 . Scores are percentages; Avg. denotes the four-benchmark mean.

\Delta

indicates the change from full OVC in percentage points. Best scores are bold. See Appendix A.2 for details.

Method	Chart QA-Pro	Char Xiv	Chart Museum	ChartX- QA	Evo Chart-QA	Chart Bench	Avg.
Base	37.69	62.66	36.83	63.69	47.39	62.08	51.72
GSPO	40.56	63.47	41.57	66.55	47.60	65.03	54.13 +2.41
OVC (Ours)	44.18	64.77	43.42	66.30	48.28	65.18	55.36 +3.64

Table 3: Chart-centric reasoning on Qwen3-VL-4B. Scores (%) are averaged over three evaluation sampling seeds. Avg. is the six-benchmark mean. Bold and underlined values denote the best and second-best results. Red superscripts indicate gains over Base.

Method	MMMU	Math Vision	We- Math	Math Vista	MMK12 -val	Math Verse	Avg.
DAPO	50.00	14.80	69.54	70.67	76.88	45.58	54.58
DAPO+OVC	52.78	20.39	70.92	71.30	76.58	45.89	56.31
GRPO	51.11	14.80	69.37	70.20	74.88	45.00	54.23
GRPO+OVC	52.22	15.13	70.63	70.27	75.50	46.17	54.99

Table 4: Extending OVC to alternative policy optimizers. Results on Qwen2.5-VL-7B trained on ViRL39K. Avg. is the mean across six benchmarks. Bold marks the better result in each pair.

Figure 3: Zero-shot transfer gains over Base. Bars show score changes relative to the corresponding instruction-tuned Base model. Avg. is the mean over four VQA benchmarks.

Figure 4: Qualitative visualization of within-response token-credit patterns. OVC correctly converts the bearing to the

65^{\circ}

triangle angle and obtains

d=11.8

km, whereas DelTA first errs by using

25^{\circ}

as the triangle angle and obtains

25.3

km. Colors indicate the final normalized token-credit weights

w_{it}

, recomputed offline from the corresponding checkpoints. Red boxes mark the key angle conversions and final answers. The OVC and DelTA responses have positive and negative group-relative advantages, respectively.

Method	ScienceQA	RxnBench	SUPERChem	EmbSpatial	ERQA	Avg.
	Science & Domain-Specific			Spatial & Embodied AI
Qwen3.5-9B	92.88	87.08	10.20	73.19	44.50	61.57
GSPO	93.15	88.32	12.30	75.80	45.20	62.95
OVC (Ours)	93.89	89.05	13.40	77.45	45.75	63.91

Table 5: Zero-shot transfer to scientific and spatial benchmarks. GSPO and OVC are trained on Math77K using Qwen3.5-9B. Avg. denotes the mean score across five benchmarks. The best results are highlighted in bold.

Figure 5: Training dynamics on Qwen3.5-9B.

Figure 6: Training-time cost breakdown of OVC.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Split	Samples	Prompting protocol	Scoring protocol
MMMU	validation	900	multiple-choice prompting	deterministic accuracy
MathVerse	testmini	3,940	zero-shot direct query	DeepSeek V4 correctness judge
MathVision	testmini	304	original question	DeepSeek V4 correctness judge
We-Math	testmini	1,740	reasoning with explicit final answer	deterministic Loose accuracy
MathVista	testmini	1,000	three zero-shot prompting settings	answer extraction + deterministic scoring; mean over settings
MMK12-val	test	2,000	training validation prompt	validation accuracy

Appendix

Table 6: Benchmark protocol used for external evaluation.

Qwen3-VL-4B
Method	ChartQA	CharXiv	A-OKVQA	SimpleVQA	Avg.
Base	83.84	58.04	84.89	25.93	63.18
GSPO [ 55 ]	84.12	59.68	85.41	25.98	63.80
GSPO+OVC (Ours)	84.16	60.12	86.03	25.43	63.94
Qwen3-VL-8B
Base	85.24	65.68	87.95	27.70	66.64

Appendix

Table 7: Per-task zero-shot transfer results. CharXiv is the mean of its reasoning and descriptive scores. The corresponding gains over Base are visualized in Figure 3 .

Benchmark / task	Split or set	Questions	Scoring rule
ChartQA-Pro [ 23 ]	Evaluation set	1,948	Native scorer, including numerical tolerance and textual ANLS where applicable.
CharXiv-RQ [ 40 ]	Validation	1,000	Benchmark-specific reasoning rubric.
CharXiv-DQ [ 40 ]	Validation	4,000	Benchmark-specific descriptive rubric.
ChartMuseum [ 32 ]	Test	1,000	Benchmark-specific answer-comparison rubric.
ChartX-QA [ 43 ]	Test	1,152	QA scoring rubric, including 5% numerical tolerance.
EvoChart-QA [ 10 ]	Evaluation set	1,250	Exact matching or 5% numerical tolerance, depending on question type.

Appendix

Table 8: Chart-centric evaluation protocol. The six benchmarks comprise seven evaluated tasks.

Model	Training data	Prompt batch	Rollouts $G$	Learning rate	Max prompt	Max response	Checkpoint
Qwen2.5-VL-7B	ViRL39K	512	16	$1\times 10^{-6}$	2,048	16384	step 76
Qwen3-VL-4B	ViRL39K	512	16	$1\times 10^{-6}$	2,048	20,480	step 76
Qwen3-VL-8B	ViRL39K	512	16	$1\times 10^{-6}$	2,048	8,192	step 76
Qwen3.5-9B	Math77K	256	8	$1\times 10^{-6}$	4,096	16,384	step 300
Qwen3-VL-4B (Chart)	ChartVerse (18k)	256	16	$1\times 10^{-6}$	16,384	16,384	step 72

Appendix

Table 9: Training recipes and evaluated checkpoints.

Quantity	Setting
Predictive layer and hidden width	$\ell=16$ , $H=4096$
Responses per input	$G=8$
Global actor mini-batch	$512$ responses
Forward micro-batch per GPU	$1$ response
Reference refinements	$K=1$
Raw coefficient interval	$[0.8,1.2]$

Appendix

Table 10: Defaults of the supplied Qwen3.5-9B recipe.

RL for VLMs

Reinforcement learning for vision-language models aligns multimodal reasoning, tool use, and action with task-specific feedback.

See all 472 papers

Residual Visual Credit Optimization: Conserved Evidence Routing for Multimodal Reinforcement Learning
Oct 4, 2026Lin Qiu, Yao Liu, Diyi Hu +6Credit Assignment in RL Step-Level Credit Assignment in RL
Meta Superintelligence Labs · University of Southern California · Meta Recommendation System +1
Visual sensitivity is not claim retractability: persistence-aware credit assignment for multimodal reinforcement learning
Sep 29, 2026Zhongan Bi, Kepeng Lin, Xuanang Gao +2Counterfactual Evaluation of VLMs Credit Assignment in RL
Zhejiang University · Huazhong University of Science and Technology · Shanghai Jiao Tong University
Structured Role-Aware Policy Optimization for Multimodal Reasoning
May 8, 2026Bingqing Jiang, Difan ZouToken-Level Credit Assignment Multimodal Reasoning
School of Computing & Data Science, The University of Hong Kong · School of Computing & Data Science and Institute of Data Science, The University of Hong Kong

Oct 4, 2026J/K move · Enter open · S save

Residual Visual Credit Optimization: Conserved Evidence Routing for Multimodal Reinforcement Learning

Lin Qiu, Yao Liu, Diyi Hu, Hanqing Zeng+5

Meta Superintelligence Labs · University of Southern California · Meta Recommendation System+1

Credit Assignment in RL Step-Level Credit Assignment in RL RL for VLMs Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards scales multimodal reasoning, but an outcome reward says how much a trajectory is worth, not how that value should be spread over the decisions that produced it. We introduce Residual Visual Credit Optimization (RVCO), which treats token credit as a conserved routing problem. A controlled visual intervention yields a per-token evidence response; robust within-trajectory coordinates remove incidental scale; and a budgeted entropic router distributes a fixed amount of sequence utility according to perceptual dependence. A residual support path guarantees positive credit at every valid position, and an analytic correction restores the prescribed credit mass exactly. The resulting field is selective, bounded, full-support, and invariant to response-local score shifts, and recovers hard token selection as a limiting case. Across four model families and seven reasoning benchmarks, RVCO improves accuracy over strong RLVR baselines while maintaining late-stage optimization stability, corruption robustness, and competitive training cost. Rewards, rollouts, and the group-relative advantage estimator are unchanged; only the geometry of token-level credit differs.

Abstract

RL for VLMs

Explore similar work

Residual Visual Credit Optimization: Conserved Evidence Routing for Multimodal Reinforcement Learning

Visual sensitivity is not claim retractability: persistence-aware credit assignment for multimodal reinforcement learning

Structured Role-Aware Policy Optimization for Multimodal Reasoning