Selective Transfer of RL Updates for Visual Reasoning

Suxin Ji; Hungtao Wan; Mingjun Liu; An Zhang

cs.CVOct 6, 2026

Selective Transfer of RL Updates for Visual Reasoning

Authors: Suxin Ji, Hungtao Wan, Mingjun Liu, An Zhang

Organizations: University of Pennsylvania · Independent Researcher · University of Electronic Science and Technology of China · Chongqing University

Abstract

Model merging provides a training-free way to transfer reasoning capabilities from language models to vision-language models (VLMs), but endpoint-based transfer can conflate pre-existing model differences with changes acquired during reasoning post-training. We instead formulate capability transfer around the training-stage update, isolating the parameter changes induced by reinforcement learning (RL). Yet transferring this update in full remains suboptimal: we find that its components differ substantially in cross-model transferability, with dominant directions transferring more effectively than the complete update. Based on this finding, we introduce Selective-RL, which isolates the RL-stage update, retains its dominant matrix-wise directions with magnitude preservation, and transfers them to the language modules of a VLM. Across three model families and five visual-reasoning benchmarks, Selective-RL improves full-update interpolation in 12 of 15 comparisons, including an 8.55 percentage-point MathVision gain on the Qwen recipient. Matched controls show that update magnitude or arbitrary low rank alone does not reproduce these gains. These results highlight a distinction between what is acquired during post-training and what remains transferable across models, providing a training-stage perspective on cross-model capability transfer. Code is available at https://anonymous.4open.science/r/selective-rl.

Figures & tables

Figure 1: Selective RL-update transfer. Four stages isolate $R-B$ , select a matrixwise direction, restore its Frobenius norm, and compose it with the recipient. Interpolation includes initialization displacement; direct addition omits it. Matrix patterns are schematic.

H1: Qwen3-VL-8B-Instruct
Method	Math Vista	Math Verse	Math Vision	DynaMath	MMStar
Receiver	$75.00$	$42.00$	$37.50$	$65.03$	$54.80$
Task Arithmetic	$76.10_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 1.10}}$	$44.50_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 2.50}}$	$39.47_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 1.97}}$	$65.67_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 0.64}}$	$54.67_{{\color[rgb]{0,0.5547,0.6133}\downarrow 0.13}}$
Full interpolation	$76.70_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 1.70}}$	$45.30_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 3.30}}$	$40.46_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 2.96}}$	$66.07_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 1.04}}$	$55.93_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 1.13}}$
TIES	$76.40_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 1.40}}$	$44.80_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 2.80}}$	$39.80_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 2.30}}$	$65.77_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 0.74}}$	$55.53_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 0.73}}$
DARE	$76.30_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 1.30}}$	$45.00_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 3.00}}$	$40.13_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 2.63}}$	$65.97_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 0.94}}$	$55.73_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 0.93}}$

Table 1: Cross-backbone transfer and paired H1 outcomes. Scores (%) for methods evaluated on all five benchmarks; extended comparisons: Table 6 . Bold/underline: best/second within displayed groups; arrows: changes from Receiver.

Figure 2: Coefficient sensitivity on H1. Full and rank-one interpolation for

0\leq\lambda\leq 0.2

; dashed lines mark Receiver accuracy. Validation selects

0.05

; full sweeps: Table 9 .

(a) Composition: initialization displacement and RL update
Construction	$\alpha$	$\beta$	Math Vista	Math Verse	DynaMath	MMStar
Receiver	0.00	0.00	75.00	42.00	65.03	54.80
Initialization only	0.05	0.00	74.70	41.50	64.65	54.07
Direct full	0.00	0.05	76.10	44.50	65.67	54.67
Direct rank-1	0.00	0.05	79.20	50.20	69.06	58.80
Interpolation full	0.05	0.05	76.70	45.30	66.07	55.93

Table 2: Composition, direction controls, and generation behavior on H1. Panels (a,b): four tasks with complete controls; five-task results: Table 7 . Panel (c): 1,000 MathVista examples.

Figure 3: Direction, scale, and spectral concentration. Left: gains over Full under the H1 controls; only random-direction means carry sample-SD bars from ten runs. Right: module-wise median retained energy

\rho_{1}

and interquartile ranges across eligible matrices.

Figure 4: Transfer scope and its limits. Rank-1 minus Full in percentage points: three receiver backbones (left) and four post-training sources on H1 (right). Negative cells retain regressions; the H1/RL setting is shared across panels.

Figure 5: Rank and output efficiency. Left: gains over Full for finite ranks on three H1 tasks. Right: MathVista accuracy versus mean output tokens; bubble area encodes mean tokens. Points show measurements without fitted trends.

Figure 6: Six Full errors corrected by Selective-RL. Receiver also fails except in (b). Cases are outcome-selected; (f) uses left-to-right post-order. Records were retained during qualitative analysis.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

1	Input: recipient $V$ , actual pre-RL checkpoint $B$ , donor $R$ , matrix support $\mathcal{S}$ , coefficients $\alpha,\beta$ .
2	Verify aligned tensor coordinates and consistent support and updates for tied parameters.
3	$W\leftarrow\operatorname{copy}(V)$ . // copy all tensors
4	for each $j\in\mathcal{S}$ do
5	$\Gamma_{j}\leftarrow B_{j}-V_{j}$ ; $\Delta_{j}\leftarrow R_{j}-B_{j}$ . // isolate RL
6	$\eta_{j}\leftarrow\sqrt{\langle\Delta_{j},\Delta_{j}\rangle_{F}}=\\|\Delta_{j}\\|_{F}$ .
7	if $\eta_{j}=0$ then $Z_{j}\leftarrow 0$ // zero RL term

Appendix

Algorithm 1 Fixed Matrixwise Rank-One Transfer

Configuration	H1: primary	H2: cross-family	H3: cross-backbone
Receiver	Qwen3-VL-8B-Instruct	LLaVA-NeXT-LLaMA3-8B	Idefics2-8B
LM family	Qwen3	LLaMA3	Mistral
Layers / matrices	36 / 252	32 / 224	32 / 224
Donor seed	42	42	42
Selected step	120	100	110
Training budget	150 steps	150 steps	150 steps

Appendix

Table 3: Model groups and donor selection. All donors use matched DAPO-style training on DAPO-Math-17K.

Benchmark	$N$	Metric
MathVista	1,000	Accuracy (%)
MathVerse	1,000	Accuracy (%)
MathVision	304	Accuracy (%)
DynaMath	5,010	Accuracy (%)
MMStar	1,500	Accuracy (%)
TextVQA	1,000	VQA soft score

Appendix

Table 4: Evaluation tasks and sample sizes. Main results use the first five tasks; the remaining four assess broader capabilities.

$\lambda$	MathVista-val	DynaMath-val
0.01	76.90	66.10
0.025	79.00	68.35
0.05	80.20	69.40
0.075	79.90	69.10
0.1	79.50	68.75

Appendix

Table 5: H1 held-out coefficient selection.

\lambda=0.05

leads both validation metrics and is then frozen for H2 and H3.

H1: Qwen3-VL-8B-Instruct
Method	Math Vista	Math Verse	Math Vision	DynaMath	MMStar
Receiver	$75.00$	$42.00$	$37.50$	$65.03$	$54.80$
Task Arithmetic	$76.10_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 1.10}}$	$44.50_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 2.50}}$	$39.47_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 1.97}}$	$65.67_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 0.64}}$	$54.67_{{\color[rgb]{0,0.5547,0.6133}\downarrow 0.13}}$
Full interpolation	$76.70_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 1.70}}$	$45.30_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 3.30}}$	$40.46_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 2.96}}$	$66.07_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 1.04}}$	$55.93_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 1.13}}$
TIES	$76.40_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 1.40}}$	$44.80_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 2.80}}$	$39.80_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 2.30}}$	$65.77_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 0.74}}$	$55.53_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 0.73}}$
DARE	$76.30_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 1.30}}$	$45.00_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 3.00}}$	$40.13_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 2.63}}$	$65.97_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 0.94}}$	$55.73_{{\color[rgb]{0.832,0.3633,0.1758}\uparrow 0.93}}$

Appendix

Table 6: Extended comparison across all reported merging methods. Accuracy (%). Colored subscripts show changes from the Receiver within each group; changes use displayed scores. Bold/underline: best/second-best available score per group and task.

(a) Composition: initialization displacement and RL update
Construction	$\alpha$	$\beta$	Math Vista	Math Verse	Math Vision	DynaMath	MMStar
Receiver	0.00	0.00	75.00	42.00	37.50	65.03	54.80
Initialization only	0.05	0.00	74.70	41.50	37.17	64.65	54.07
Direct full	0.00	0.05	76.10	44.50	39.47	65.67	54.67
Direct rank-1	0.00	0.05	79.20	50.20	47.04	69.06	58.80
Interpolation full	0.05	0.05	76.70	45.30	40.46	66.07	55.93

Appendix

Table 7: Five-benchmark composition and direction controls. The same five benchmarks are used across controls. The ten-seed random row reports mean

\pm

sample standard deviation.

Rank	Math Vista	Math Verse	Math Vision	DynaMath	MMStar
1	81.00	52.00	49.01	70.06	59.67
2	80.70	51.60	—	69.66	59.27
4	80.30	50.80	47.70	69.06	58.67
8	79.50	49.40	—	68.26	58.00
16	78.30	47.60	—	67.27	57.00
Full	76.70	45.30	40.46	66.07	55.93

Appendix

Table 8: Rank sensitivity on H1. Frobenius matching is applied at every finite rank.

$\lambda$	MathVista Full	MathVista Rank-1	DynaMath Full	DynaMath Rank-1	MMStar Full	MMStar Rank-1
0	75.00	75.00	65.03	65.03	54.80	54.80
0.01	75.40	77.40	65.27	66.67	55.33	56.60
0.025	76.20	79.90	65.71	69.26	55.67	58.60
0.05	76.70	81.00	66.07	70.06	55.93	59.67
0.1	76.00	80.50	65.47	69.66	54.93	59.27
0.2	74.40	78.90	63.87	68.26	53.00	57.73

Appendix

Table 9: Complete test-set coefficient sweep. The main coefficient

0.05

is selected using H1 validation, not this test sweep.

Support	Matrices	Math Vista	Math Verse	Math Vision	DynaMath	MMStar
QKV only	108	78.10	47.60	—	—	57.47
Attention only	144	79.40	49.30	—	68.52	58.47
MLP only	108	78.80	48.60	—	68.88	57.93
O + Down only	72	77.90	46.90	—	67.54	57.20
Attention + MLP	252	81.00	52.00	49.01	70.06	59.67

Appendix

Table 10: Eligible support on H1. The remaining recipient parameters stay fixed.

Configuration	Attention median	MLP median	Overall	Q1	Q3	Median amplification
H1 Qwen RL	0.24	0.15	0.18	0.11	0.31	2.36
H2 LLaMA RL	0.17	0.11	0.13	0.08	0.24	2.77
H3 Mistral RL	0.12	0.08	0.09	0.05	0.17	3.33

Appendix

Table 11: Spectral concentration by model configuration.

\rho_{1}

is retained squared Frobenius energy; amplification is

\rho_{1}^{-1/2}

Module	Mean $\rho_{1}$	Median $\rho_{1}$	Q1	Q3
q_proj	0.27	0.25	0.17	0.35
k_proj	0.24	0.22	0.15	0.31
v_proj	0.25	0.23	0.16	0.33
o_proj	0.21	0.19	0.12	0.28
gate_proj	0.16	0.14	0.09	0.22
up_proj	0.15	0.13	0.08	0.21

Appendix

Table 12: H1 spectral energy by module. Quartiles summarize the distribution across eligible matrices.

Layer region	Median $\rho_{1}$	Median amplification
Early 1-12	0.13	2.77
Middle 13-24	0.22	2.13
Late 25-36	0.19	2.29

Appendix

Table 13: H1 spectral energy by depth.

Normalization	Math Vista	Math Verse	Math Vision	DynaMath	MMStar
None	75.20	42.60	38.16	65.27	54.80
Nuclear-norm match	80.00	50.10	—	69.16	58.67
Frobenius match	81.00	52.00	49.01	70.06	59.67

Appendix

Table 14: Normalization ablation on H1. Leading rank-one updates are compared at the same composition coefficients.

Reference control	Math Vista	Math Verse	Math Vision	DynaMath	MMStar
Full, actual pre-RL	76.70	45.30	40.46	66.07	55.93
Rank-1, actual pre-RL	81.00	52.00	49.01	70.06	59.67
Full, earlier reference	75.60	43.80	—	65.41	54.93
Rank-1, earlier reference	77.80	46.90	—	67.11	56.40

Appendix

Table 15: Stage-reference ablation. The actual donor initialization is compared with an earlier reference checkpoint.

$\beta$	MathVista	DynaMath	MMStar
0.05	75.20	65.27	54.80
0.1	76.40	65.87	55.47
0.2	77.60	66.91	56.33
0.5	78.50	67.52	56.87
1	77.80	66.95	56.40
2	74.90	63.95	53.87

Appendix

Table 16: Unscaled leading rank-one coefficient sweep. Initialization displacement remains fixed at

\alpha=0.05

Control	MathVista	DynaMath	MMStar
Full, global beta	76.70	66.07	55.93
Full, rank-1 matrixwise gains	77.60	66.93	56.33
Rank-1, tuned global beta	78.50	67.52	56.87
Rank-1, Frobenius matched	81.00	70.06	59.67

Appendix

Table 17: Separating direction selection from matrixwise amplification.

Direction seed	MathVista	DynaMath	MMStar
0	74.60	64.43	53.27
1	73.80	63.87	52.87
2	74.10	64.07	53.13
3	73.40	63.55	52.47
4	74.70	64.51	53.60
5	73.90	63.91	52.93

Appendix

Table 18: Random-direction robustness. Ten matched rank-one direction draws at

\alpha=\beta=0.05

Reconstruction	MATH	GSM8K	Accuracy retention (%)	Gain retention (%)
Pre-RL	62.00	84.99	—	—
Donor	76.20	90.98	100.00	100.00
rank-1	75.80	90.74	99.48	97.18
rank-2	75.40	90.60	98.95	94.37
rank-4	76.00	90.82	99.74	98.59
rank-8	75.60	90.67	99.21	95.77

Appendix

Table 19: Source-side reconstruction. MATH ( Hendrycks et al., 2021 ) and GSM8K ( Cobbe et al., 2021 ) accuracy; retention ratios use the donor accuracy or its gain over the pre-RL model.

Stage	Source MATH	MathVista Full	MathVista Rank-1	DynaMath Full	DynaMath Rank-1	MMStar Full	MMStar Rank-1
Early	69.60	75.80	77.90	65.47	66.91	55.33	56.47
Middle	75.60	76.60	79.90	65.87	68.72	55.80	58.33
Late	76.20	76.70	81.00	66.07	70.06	55.93	59.67

Appendix

Table 20: H1 donor progression. Stage labels refer to checkpoints within one RL training trajectory.

Source	Transfer	MathVista	DynaMath	MMStar	Median $\rho_{1}$
RL	Full	76.70	66.07	55.93	—
RL	Rank-1	81.00	70.06	59.67	0.18
Offline SFT	Full	77.10	66.67	56.27	—
Offline SFT	Rank-1	78.00	67.15	56.13	0.10
Offline distillation	Full	77.60	67.27	56.67	—
Offline distillation	Rank-1	78.80	68.18	56.80	0.12

Appendix

Table 21: Transfer from different post-training sources. Within each source, Full and Rank-1 share the donor and support.

Seed	Full MathVista	Selective MathVista	MathVista gain	DynaMath gain	MMStar gain
17	76.60	80.70	$\uparrow\!4.10$	$\uparrow\!3.75$	$\uparrow\!3.47$
42	76.70	81.00	$\uparrow\!4.30$	$\uparrow\!3.99$	$\uparrow\!3.73$
73	76.50	80.50	$\uparrow\!4.00$	$\uparrow\!3.81$	$\uparrow\!3.33$
Mean $\pm$ SD	$76.60\pm 0.10$	$80.73\pm 0.25$	$4.13\pm 0.15$	$3.85\pm 0.12$	$3.51\pm 0.20$

Appendix

Table 22: Donor-training seed robustness. Each seed compares Full and Selective with the same trained donor.

Task	$a$	$b$	$c$	$d$	$N$	Gain (pp)	95% CI
MathVista	727	40	83	150	1000	$\uparrow\!4.30$	[2.14, 6.46]
MathVerse	410	43	110	437	1000	$\uparrow\!6.70$	[4.31, 9.09]
MathVision	108	15	41	140	304	$\uparrow\!8.55$	[3.82, 13.29]
DynaMath	3197	113	313	1387	5010	$\uparrow\!3.99$	[2.10, 5.86]
MMStar	800	39	95	566	1500	$\uparrow\!3.73$	[2.23, 5.23]

Appendix

Table 23: Paired H1 outcomes and 95% confidence intervals. Full is the baseline.

Statistic	Full	Selective-RL
All-ten-correct (%)	29.1	34.3
Median correct variants	7.0	8.0

Appendix

Table 24: DynaMath robustness across variants. Each of the 501 seed questions has ten variants.

Method	Mean tokens	P95 tokens	Repeat	Extract fail	At cap
Receiver	460	1550	8	5	6
Full	575	2200	15	8	14
IP-Merging	590	2050	11	6	10
FRISM	640	2200	7	4	8
STAR	520	1820	8	4	7
AdaRank	495	1740	6	4	6

Appendix

Table 25: Generation diagnostics on H1 MathVista. Token counts and events over 1,000 examples.

Full error category	Count	Share (%)
Repetition / loop	8	9.6
Token-cap truncation	6	7.2
Extraction pathology	4	4.8
Reasoning / perception	65	78.3

Appendix

Table 26: Error categories among 83 Selective-only MathVista successes.

Statistic	Receiver	Full	Selective-RL
Audited examples	200.0	200.0	200.0
Answer/extraction disagreement	6.0	7.0	5.0
False-positive extraction	2.0	3.0	2.0
False-negative extraction	4.0	4.0	3.0
Disagreement (%)	3.0	3.5	2.5

Appendix

Table 27: Final-answer/evaluator audit on H1 MathVista. Each model has 200 manually audited examples.

Benchmark	Receiver	Full	Selective-RL	$\Delta$ vs Full
TextVQA	82.03	81.06	83.64	$\uparrow\!2.58$
MMMU	60.00	59.20	62.80	$\uparrow\!3.60$
OCRBench	78.60	78.00	78.70	$\uparrow\!0.70$
POPE	86.90	86.30	86.80	$\uparrow\!0.50$

Appendix

Table 28: Broader H1 capabilities. TextVQA uses the VQA soft score; other metrics follow Table 4 .

Method	SVD	Other	Write	Total	Search (GPU-h)
Receiver	0	0	0	0	0.0
Full	0	0	35	35	3.0
TIES	0	40	35	75	3.0
DARE	0	27	35	62	3.0
TSV	260	90	35	385	3.0
STAR	285	75	35	395	3.0

Appendix

Table 29: Offline construction costs. Stage times and totals are in seconds; validation search is separate in GPU-hours.

Method	Dense ratio	Added module	VRAM (GB)
Receiver	1.00	No	24.60
Full	1.00	No	24.60
STAR	1.00	No	24.60
AdaRank	1.00	No	24.60
FRISM	1.00	No	24.60
SELECTIVE-RL	1.00	No	24.60

Appendix

Table 30: Deployment footprint. The merged model retains the recipient architecture.

Figure 7: An edge-coloring correction (DynaMath 5234; question 234, variant 6). The graph is a five-vertex cycle, whose chromatic index is 3. The receiver and Full each answer 5. Rank-one merging correctly identifies that the odd cycle requires three edge colors.

Figure 8: Correct integration of a velocity graph (DynaMath 7268; question 268, variant 8). The three segment areas give $4+1+0.5=5.5$ m. Full treats $0$ – $9$ s as a single rising segment and answers $5$ m. Rank-one merging and the receiver both account for the constant-velocity interval and correctly answer $5.5$ m.

Explore similar work

May 2, 2026cs.CV

MIRL: Mutual Information-Guided Reinforcement Learning for Vision-Language Models

Vision-Language Models (VLMs) frequently suffer from visual perception errors and hallucinations that compromise answer accuracy in complex reasoning tasks. Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising solution by optimizing policies using answer correctness signals. Despite their effectiveness, prevailing RLVR methods face two critical limitations. First, much of the sampling budget is wasted on trajectories doomed to fail due to early visual description errors. Second, sparse rewards cannot distinguish whether failures stem from visual perception or reasoning stages. We introduce MIRL, a decoupled framework that addresses both limitations by leveraging mutual information (MI) between generated descriptions and visual inputs as a cheap pre-screening signal. This enables intelligent budget allocation toward high-potential trajectories via forking, while decoupled training provides independent MI-based rewards for visual perception optimization, resolving reward blindness. Experiments on six vision-language reasoning benchmarks demonstrate that MIRL achieves 70.22% average accuracy and successfully surpasses the performance of sampling 16 complete trajectories using only 10 pre-samples with top-6 selection (25% fewer complete trajectories). Our code is available at: https://anonymous.4open.science/r/mirl-main/.

Yin Zhang, Jiaxuan Zhao, Zonghan Wu +5

School of Mathematics, Tianjin University, Tianjin, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · Shanghai Advanced Institute of Finance (SAIFS), East China Normal University, Shanghai, China +4

Recent Vision-Language Models Reinforcement Learning With Verifiable Reward Multimodal Reasoning Benchmarks Visual Perception Spatial Blindness

Ask PDF

May 19, 2026cs.CL

From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models

Recent advances in vision-language models (VLMs) emphasize long chain-of-thought reasoning; yet, we find that their performance on visual tasks is primarily limited by a lack of visual perception as opposed to reasoning itself. In this work, we systematically study the interplay between perception and reasoning in VLM post-training by decomposing their capabilities into three separate training stages: visual perception, visual reasoning, and textual reasoning, incorporating specialized training data. We demonstrate that visual perception (a) requires targeted optimization with specialized data; (b) serves as a fundamental scaffold that should be solidified through staged training before refining visual reasoning; and (c) is more effectively learned via RL than caption-based SFT. Our experiments across multiple VLMs demonstrate that staged training consistently improves both visual perception and reasoning performance over merged training. Notably, models trained with our approach achieve 1.5% higher reasoning accuracy with 20.8% shorter reasoning traces, suggesting that superior perception reduces the need for excessive reasoning. Furthermore, we show that this capability-based staging represents a new curriculum dimension orthogonal to traditional difficulty-based curricula, and combining both yields further additive gains. Our staged-training models achieve superior performance among open-weight VLMs, establishing advanced results on several visual math and perception (e.g., +5.2% on WeMath and +3.7% on RealWorldQA) tasks compared with the base counterpart.

Juncheng Wu, Hardy Chen, Haoqin Tu +6

Amazon · UC Santa Cruz · University of Waterloo +1

Recent Vision-Language Models Visual Reasoning Visual Perception Perception Post-Training Model Training

Ask PDF

Jun 29, 2026cs.AI

Be Faithful When Response: Returning Fluent and Grounded Answers for Vision-Language Models Reinforcement Learning

Reinforcement Learning (RL) is an important paradigm for improving the reasoning capabilities of Vision-Language Models (VLMs). However, directly applying RL to rollout multimodal reasoning can lead to instability, due to the exploitation of language priors, the neglect of visual evidence, and the generation of reasoning traces that are fluent yet not visually grounded. The question arises: Can initially steer the policy toward visually faithful reasoning regime before applying reinforcement learning? To this end, we propose a Faithful Warm-Start (FWS) strategy that first curates samples with explicit vision-language causal relationships from six general VQA benchmarks to construct the FaithfulQA dataset, where each of the image-question pairs gains a certain degree of visual observations, question requirements, commonsense knowledge, domain knowledge, and the final answer. Subsequently, a VLM-based judge is employed to further purify the dataset, ensuring strong causal consistency and visual faithfulness. This warm-start stage equips the model with the capability to understand causally grounded vision-language patterns before subsequent RL optimization under sparse answer-level rewards. Experimental results show that such faithful supervision improves answer accuracy, stabilizes RL training, and reduces visually unsupported reasoning.

Peng, Lee, Yin Zhang +9

GMLab · Hong Kong Polytechnic University · XPENG Robotics

Recent Vision-Language Models Multimodal Reasoning Explanation Faithfulness Responses

Ask PDF