Residual Visual Credit Optimization: Conserved Evidence Routing for Multimodal Reinforcement Learning

Lin Qiu; Yao Liu; Diyi Hu; Hanqing Zeng; Onur Gungor; Chujie Chen; Jiayi Liu; Jianyu Wang

cs.LGOct 4, 2026

Residual Visual Credit Optimization: Conserved Evidence Routing for Multimodal Reinforcement Learning

Authors: Lin Qiu, Yao Liu, Diyi Hu, Hanqing Zeng, Onur Gungor, Chujie Chen, Jiayi Liu, Jianyu Wang,

Organizations: Meta Superintelligence Labs · University of Southern California · Meta Recommendation System · Meta

Abstract

Reinforcement learning with verifiable rewards scales multimodal reasoning, but an outcome reward says how much a trajectory is worth, not how that value should be spread over the decisions that produced it. We introduce Residual Visual Credit Optimization (RVCO), which treats token credit as a conserved routing problem. A controlled visual intervention yields a per-token evidence response; robust within-trajectory coordinates remove incidental scale; and a budgeted entropic router distributes a fixed amount of sequence utility according to perceptual dependence. A residual support path guarantees positive credit at every valid position, and an analytic correction restores the prescribed credit mass exactly. The resulting field is selective, bounded, full-support, and invariant to response-local score shifts, and recovers hard token selection as a limiting case. Across four model families and seven reasoning benchmarks, RVCO improves accuracy over strong RLVR baselines while maintaining late-stage optimization stability, corruption robustness, and competitive training cost. Rewards, rollouts, and the group-relative advantage estimator are unchanged; only the geometry of token-level credit differs.

Figures & tables

Figure 1: Conserved evidence routing in RVCO . Controlled visual replay produces a sampled-token evidence response without changing the generated trajectory. Response-local calibration removes incidental scale, the entropic router concentrates a fixed mass $m_{i}$ , and residual mixing preserves positive support before analytic correction restores the exact budget. Only the detached token weights enter policy optimization; rewards and advantages remain unchanged.

Figure 2: Main architecture of RVCO . Multiple controlled visual interventions are evaluated by a shared behavior policy while the visual question and generated response state are held fixed. The resulting original/intervened token responses define a decision-relative evidence coordinate, which is transformed into a residual credit field and routed under the exact budget constraint $\sum_{t}g_{i,t}=m_{i}$ . The conserved field then modulates the ordinary policy update, emphasizing visually grounded decisions while retaining full-support credit across the response.

Backbone	Method	MathV.	Dyna	MMK12	Geo3K	VisNum	Logic	MMMU-P	Avg.
Qwen3-VL-4B	GRPO	66.5	65.8	72.3	65.0	23.1	57.8	39.2	55.7
	Visual-RFT	80.7	75.6	76.3	67.6	25.4	59.1	49.3	62.0
	VPPO	80.9	75.3	76.2	67.7	24.8	59.2	49.5	61.9
	RVCO	82.3	78.2	78.0	69.7	27.2	61.5	51.1	64.0
Qwen3-VL-8B	GRPO	76.5	75.0	71.2	64.4	28.7	56.0	42.8	59.2
	Visual-RFT	81.0	78.1	77.2	69.1	29.6	63.0	49.3	63.9

Table 1: Benchmark-wise comparison across the Qwen3-VL family. All values are accuracy@8 (%); Avg. is the macro-average over the seven benchmarks with reported results. The same rollout budget, verifier, and decoding protocol are used for all methods within each backbone. Bold and underlined values denote the best and second-best results within each backbone.

Backbone	Method	MathV.	Dyna	MMK12	Geo3K	VisNum	Logic	MMMU-P	Avg.
Qwen2.5-VL-7B	GRPO	75.0	72.9	69.5	62.0	26.4	54.0	40.7	57.2
	Visual-RFT	79.5	78.2	75.5	66.7	28.1	58.0	46.8	61.8
	VPPO	82.4	78.0	78.6	69.6	28.6	60.4	50.4	64.0
	RVCO	83.1	78.5	79.1	70.3	29.0	62.3	53.4	65.1
InternVL3-8B	GRPO	73.1	72.3	68.0	61.5	25.9	53.1	39.6	56.2
	Visual-RFT	77.4	75.2	73.8	66.0	27.3	56.9	45.5	60.3

Table 2: Benchmark-wise comparison across vision-language model families. All values are accuracy@8 (%); Avg. is the macro-average over the seven benchmarks. Each backbone retains its native visual encoder and tokenizer. Bold and underlined values denote the best and second-best results within each backbone.

Variant	MathV.	Dyna	MMK12	Geo3K	VisNum	Logic	MMMU-P	Avg.
w/o Robust Coordinate	83.2	78.7	79.3	71.3	29.7	62.9	53.2	65.5
w/o Budgeted Routing	81.7	77.0	78.5	71.6	28.9	61.5	52.1	64.5
w/o Residual Support	80.9	76.4	77.2	69.0	27.8	60.6	49.3	63.0
w/o Exact Conservation	81.9	77.6	78.1	70.5	28.6	61.2	51.7	64.2
RVCO	84.4	80.4	80.6	72.5	30.5	64.1	55.4	66.8

Table 3: Ablation study on Qwen3-VL-8B. Each variant changes one component while preserving the same reward, rollout budget, optimizer, and decoding protocol.

Figure 3: Hyperparameter sensitivity on Qwen3-VL-8B. We vary the residual ratio

\beta

, visual budget

\rho

, and gate temperature

\tau

while holding the other two parameters fixed. Dashed lines mark the default configuration.

Corruption	GRPO	VPPO	RVCO
Patch dropout (15%)	58.9	64.2	66.8
Gaussian blur	57.4	61.9	64.1
Low contrast	55.8	60.5	62.9
Unseen crop	52.6	58.1	60.7
Gaussian noise	54.9	59.8	62.5

Table 4: Visual corruption robustness and end-to-end training cost on Qwen3-VL-8B. Robustness reports exact-match accuracy (%).

Method	Flicker $\downarrow$	Continuity $\uparrow$	Ent. Var. $\downarrow$	Grad. CV $\downarrow$	Reward $\uparrow$
Hard Top- $\rho$	0.219	0.184	0.047	0.384	0.751
PGPO	0.143	0.231	0.031	0.319	0.766
VPPO	0.126	0.246	0.024	0.298	0.758
Entropic routing ( $\beta=0$ )	0.073	0.267	0.018	0.251	0.775
RVCO	0.041	0.281	0.012	0.214	0.784

Table 5: Optimization stability diagnostics over the final 20% of Qwen3-VL-8B updates. Lower support flicker, entropy variance, and gradient CV are better.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value	Setting	Value
Rollouts / prompt	8	Global batch	128
Rollout temperature	1.0	Clip lower / upper	0.20 / 0.28
Top- $p$	0.99	Weight decay	$10^{-2}$
Prompt length	4096	Grad. norm	1.0
Response length	8192	Precision	BF16
Vision tower	Trainable	Sharding	Full FSDP

Appendix

Table 6: Additional reinforcement-learning hyperparameters.

Method	MathV.	Dyna	MMK12	Geo3K	VisNum	Logic	MMMU-P	Avg.
Instruction	67.7	72.2	62.0	57.5	25.0	50.1	33.7	52.6
GRPO	76.5	75.0	71.2	64.4	28.7	56.0	42.8	59.2
DAPO	78.2	76.1	73.0	66.0	28.4	60.2	46.1	61.1
Vision-R1	79.0	76.8	74.1	66.8	28.9	58.0	50.7	62.0
MM-Eureka	80.1	78.5	76.0	72.3	29.5	61.1	52.1	64.2
PAPO	82.1	79.0	78.7	70.8	29.4	61.4	51.0	64.6

Appendix

Table 7: Extended comparison on Qwen3-VL-8B. Values are accuracy@8 (%); Avg. denotes the macro-average over all seven benchmarks. Bold and underlined values denote the best and second-best results, respectively.

Method	Continuity $\uparrow$	Support overlap $\uparrow$	Weight Gini $\downarrow$	Mass error $\downarrow$
Hard Top- $\rho$	0.184	0.621	0.604	0.000
Independent sigmoid	0.251	0.652	0.421	0.087
Entropic gate ( $\beta=0$ )	0.267	0.638	0.468	0.000
RVCO	0.281	0.807	0.392	0.000

Appendix

Table 8: Evidence-retention diagnostics on held-out trajectories. Credit continuity is cosine similarity between allocation fields under paired mild interventions; support overlap uses matched top-

\rho

sets.

Figure 4: Support overlap during training under matched perturbations. The residual route maintains a larger stable intersection of credited positions as evidence becomes sharper late in training.

Figure 5: Held-out reward across training for representative perception-aware methods. The vertical marker denotes the late-stage window used for stability diagnostics.

Figure 6: Training dynamics over a 16

\times

range of routing temperatures on Qwen3-VL-4B. Moderate temperatures preserve evidence contrast without abrupt late-stage changes.

Figure 7: Sensitivity to residual ratio

\beta

, visual budget

\rho

,and gate temperature

\tau

on Qwen3-VL-8B. The dashed line marks the default.

Backbone	Method	A-OKVQA	SimpleVQA	ChartQA	InfoVQA	DocVQA	Avg.
Qwen3-VL-8B	Instruction	87.5	43.9	82.4	55.6	89.7	71.8
	GRPO	87.3	42.7	82.8	55.1	89.1	71.4
	RVCO	88.7	47.8	84.9	59.4	91.0	74.4
Qwen3-VL-4B	Instruction	86.3	38.4	79.6	51.2	87.1	68.5
	GRPO	86.2	37.1	79.9	50.8	86.5	68.1
	RVCO	87.3	42.0	82.1	55.0	88.4	71.0

Appendix

Table 9: Out-of-domain visual question answering and document understanding. Values use each benchmark’s official primary metric (%).

Length bucket	GRPO	VPPO	RVCO	Gain over VPPO
$\leq 512$ tokens	64.9	69.1	70.4	+1.3
513–2048 tokens	59.5	66.0	67.6	+1.6
$>2048$ tokens	53.2	60.6	63.0	+2.4

Appendix

Table 10: Accuracy by generated-response length on the seven-task reasoning suite. Length buckets are measured after answer extraction.

Figure 8: Sensitivity to rollout-group size under approximately matched generated-token budgets.

Figure 9: Sensitivity to controlled verifier label noise. All methods use the same corrupted reward labels at each flip rate.

Intervention	Avg. acc. $\uparrow$	Continuity $\uparrow$	Overhead
Random patch mask (default)	66.8	0.281	4.3%
Gaussian noise	66.3	0.275	4.1%
Gaussian blur	66.0	0.270	4.0%
Random cutout	66.1	0.272	4.2%
Grayscale attenuation	65.4	0.264	3.9%

Appendix

Table 11: Sensitivity to the visual intervention used to construct evidence coordinates on Qwen3-VL-8B. Overhead is relative to GRPO wall-clock time.

Figure 10: Accuracy under increasing patch corruption severity. The full-support field degrades more gradually as the visual signal becomes incomplete.

Figure 11: Reasoning accuracy as the ViRL39K training fraction increases. The proposed field retains an advantage throughout the data range.

Figure 12: Representative token-credit allocation for a visual comparison response under the same 40% total budget. Hard routing alternates between unit and zero credit, whereas RVCO preserves graded support around visually grounded and connective tokens.

Credit Assignment in RL

Credit assignment in reinforcement learning studies how to trace rewards and failures back to the actions, states, or agents that caused them.

See all 219 papers

From Visual Affinity to Learning Credit in Multimodal Reinforcement Learning
Oct 23, 2026Yuhang Han, Dingning Liu, Yujie Chen +7RL for VLMs Step-Level Credit Assignment in RL
The Hong Kong University of Science and Technology (Guangzhou) · Shanghai Jiao Tong University · Shadoweave +2
Entropy Is Not Enough: Unlocking Effective Reinforcement Learning for Visual Reasoning via Vision-Anchored Token Selection
Jun 2, 2026Senjie Jin, Peixin Wang, Boyang Liu +8Token-Level Credit Assignment Visual Reasoning
College of Computer Science and Artificial Intelligence, Fudan University
Structured Role-Aware Policy Optimization for Multimodal Reasoning
May 8, 2026Bingqing Jiang, Difan ZouToken-Level Credit Assignment Multimodal Reasoning
School of Computing & Data Science, The University of Hong Kong · School of Computing & Data Science and Institute of Data Science, The University of Hong Kong

Oct 23, 2026J/K move · Enter open · S save

From Visual Affinity to Learning Credit in Multimodal Reinforcement Learning

Yuhang Han, Dingning Liu, Yujie Chen, Yaojie Zhang+6

The Hong Kong University of Science and Technology (Guangzhou) · Shanghai Jiao Tong University · Shadoweave+2

RL for VLMs Step-Level Credit Assignment in RL Reinforcement Learning with Verifiable Rewards Reasoning for VLMs

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) improves multimodal reasoning, but uniformly assigning a single response-level advantage to all tokens overlooks the distinct roles of visual observations and subsequent reasoning. Visual affinity alone is insufficient: predictive states may align with both image and prompt-context representations, and upweighting valid visual observations can amplify their negative-advantage updates when later reasoning fails. To address this issue, we propose Outcome-Conditioned Visual Credit (OVC), a plug-in framework that separates visual-association measurement from token-level credit assignment and is adaptable to policy-optimization methods with uniform response-level advantage allocation. Relative Visual Encoding constructs a context-relative visual coordinate by contrasting each predictive state's affinity to image and prompt-context representations. Outcome-conditioned Evidence Calibration combines this coordinate with uncertainty-scaled predictive features and calibrates their contributions to outcome-discriminative evidence through mean contrasts between positive- and negative-advantage responses, scaled by regularized within-side variation. Visual association thus informs credit through observed outcome differences rather than directly determining update priority. Normalized Credit Allocation conditions the calibrated evidence on the response's advantage sign to produce positive token weights with unit global mean. These detached weights preserve advantage signs while modulating individual token-gradient contributions. OVC reuses existing rollouts without auxiliary reward models, process annotations, or inference-time modules. Instantiated through GSPO-token, OVC retains sequence-level clipping and consistently improves six-benchmark mean accuracy over GSPO across three models, with gains of 1.86-2.39 percentage points.

Abstract

Credit Assignment in RL

Explore similar work

From Visual Affinity to Learning Credit in Multimodal Reinforcement Learning

Entropy Is Not Enough: Unlocking Effective Reinforcement Learning for Visual Reasoning via Vision-Anchored Token Selection

Structured Role-Aware Policy Optimization for Multimodal Reasoning