Although reinforcement learning (RL) effectively trains Vision-Language Model (VLM) agents, the mechanisms behind these capability gains are poorly understood. Across four diverse agent environments, we investigate whether the task-success gains from multi-turn RL are accompanied by improved grounded visual question answering about the environment. To assess this, we design environment-specific grounded VQA tasks that reveal improvements in grounded visual understanding. Representational probing shows that vision-encoder probe accuracy remains relatively stable, while LLM-layer probe accuracy improves. Attention analysis further reveals that LLM layers pay more attention to visual tokens during RL training. To test whether increased attention to visual tokens contributes to VQA gains, we introduce a lightweight plug-and-play intervention using a fixed attention bias. Without updating model parameters or computing gradients, this intervention partially restores VQA gains on the untrained model. These results reveal better utilization of visual information as a key mechanism behind the gains from RL.
Figures & tables
Figure 1: Overview of our evaluation framework and attention bias intervention. Left: RL progressively improves multi-turn task performance in FrozenLake. Top right: VQA evaluation with layer-wise probing reveals that RL enhances visual utilization in deep LLM layers. Bottom right: RL training deepens the attention weights on visual tokens (darker colors) in deep LLM layers; injecting an additive attention bias into the base model partially replicates this shift and improves VQA accuracy.
Figure 2: Representative visual observations from the four environments. A. FrozenLake. B. EzPoints. C. Sokoban. D. Navigation.
Figure 3: Comparison of VQA accuracy for the RL agent and SFT fine-tuning in FrozenLake, EzPoints, Sokoban, and Navigation environments. Compared to the SFT model (red), the VQA accuracy of the RL agent (blue) achieves greater improvement as the number of training steps increases across all four environments.
Figure 4: Probe accuracy of the visual encoder and two LLM layers across RL training checkpoints in FrozenLake, EzPoints, Sokoban, and Navigation. The visual encoder (green) remains flat throughout training, while LLM Layer 17 (blue) and LLM Layer 35 (red) improve steadily, with most gains concentrated in the early checkpoints.
Figure 5: Evolution of image RAPT (blue, left axis) and text RAPT (red, right axis) across RL training checkpoints. Image RAPT rises steadily while text RAPT declines, indicating that RL systematically shifts attention from text toward visual tokens.
Configuration
FrozenLake
EzPoints
Sokoban
Navigation
No intervention
28.89
82.22
25.00
78.75
Image token + deep layers
33.89
83.33
27.86
80.50
Text token + deep layers
23.33
77.78
24.29
78.75
All prompt tokens + deep layers
30.56
81.67
26.43
78.88
Image token + shallow layers
31.67
81.67
26.43
79.00
Image token + all layers
33.89
82.22
26.43
80.63
Table 1: Ablation study of the fixed-bias attention intervention on the base model. Accuracy (%) is reported for six configurations across four environments; the best result in each column is in bold.
Figure 6: Attention heatmaps for shallow (left) and deep (right) layers in FrozenLake. Shallow layers distribute attention diffusely across the grid, while deep layers concentrate on task-relevant objects (the hole), consistent with the layer-wise attention shift observed during RL training.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Accuracy gain of the trained plugin (blue) and fixed-bias variant (red) over the RL baseline across checkpoints in FrozenLake, EzPoints, and Sokoban. The trained plugin consistently achieves larger gains, while the fixed-bias variant shows higher variance and occasionally falls below the RL baseline, confirming that the learned bias values are key to reliable improvement.
Figure 8: Image RAPT (A) and text RAPT (B) before and after plugin training across three environments. Plugin optimization consistently increases image RAPT while decreasing text RAPT in all three environments, mirroring the attention redistribution direction induced by full RL fine-tuning.
Hyperparameter
FrozenLake
EzPoints
Sokoban
Total training steps
150
150
150
Actor learning rate
1×10−6
1×10−6
1×10−6
Train batch size
16
16
32
PPO mini-batch size
32
32
32
Max prompt length (tokens)
1,024
1,024
1,024
Max response length (tokens)
200
256
512
Appendix
Table 2: RL (GRPO) hyperparameters.
Architecture
FrozenLake
EzPoints
Sokoban
Qwen3VL-O
42.22
90.56
37.86
Qwen3VL-P
53.33
92.22
39.29
LLaVA-O
41.67
73.33
22.86
LLaVA-P
44.44
73.89
24.29
Gemma-O
24.44
90.00
10.71
Gemma-P
27.22
91.67
13.57
Appendix
Table 3: VQA accuracy (%) of the attention bias plugin on Qwen3-VL, LLaVA, and Gemma across FrozenLake, EzPoints, and Sokoban.
Hyperparameter
Value
Training epochs
4
Learning rate
1×10−5
LR schedule
cosine
Warmup ratio
0.1
Gradient accumulation
2
Max sequence length
4,096
Appendix
Table 4: SFT hyperparameters.
Checkpoint
VQA
Vision encoder
LLM L17
LLM L35
0
42.22
72.22
78.89
88.89
1
44.56
72.78
79.44
89.44
2
46.11
73.89
80.56
90.00
3
45.89
71.67
80.00
89.44
4
47.44
72.22
81.11
90.00
5
48.00
74.44
81.67
91.11
Appendix
Table 5: VQA and probe accuracy (%) of Qwen3-VL-4B across RL checkpoints in FrozenLake.
Reinforcement learning has brought strong gains to language model reasoning, but its improvements on visual tasks remain less consistent. One difficulty is that visual task failures can arise from different sources: the same benchmark may be limited by perception for one model and by reasoning for another. We introduce Perception and Reasoning Attribution (PARA), which measures the remaining perceptual and reasoning headroom of each model--benchmark pair by comparing how many current errors are recovered with more accurate visual information versus a stronger reasoner. Across 4 vision-language models and 17 multimodal benchmarks, PARA broadly agrees with the expected characteristics of existing benchmarks while revealing substantial model-specific variation. More importantly, it shows that RL gains become smaller as perceptual headroom increases, but larger when reasoning explains more of the recoverable errors, suggesting that current RL exploits reasoning headroom more effectively than perceptual headroom. Motivated by this gap, we analyze how existing RL objectives affect visual perception and find that reward- and token-level signals influence visual attention only indirectly. We therefore propose GRAD, a group-relative attention distillation objective that uses the best rollout in each group to directly guide the visual attention of weaker rollouts. GRAD consistently improves performance across two model scales and seven benchmarks, and further complements existing RL methods.
Reinforcement Learning (RL) is an important paradigm for improving the reasoning capabilities of Vision-Language Models (VLMs). However, directly applying RL to rollout multimodal reasoning can lead to instability, due to the exploitation of language priors, the neglect of visual evidence, and the generation of reasoning traces that are fluent yet not visually grounded. The question arises: Can initially steer the policy toward visually faithful reasoning regime before applying reinforcement learning? To this end, we propose a Faithful Warm-Start (FWS) strategy that first curates samples with explicit vision-language causal relationships from six general VQA benchmarks to construct the FaithfulQA dataset, where each of the image-question pairs gains a certain degree of visual observations, question requirements, commonsense knowledge, domain knowledge, and the final answer. Subsequently, a VLM-based judge is employed to further purify the dataset, ensuring strong causal consistency and visual faithfulness. This warm-start stage equips the model with the capability to understand causally grounded vision-language patterns before subsequent RL optimization under sparse answer-level rewards. Experimental results show that such faithful supervision improves answer accuracy, stabilizes RL training, and reduces visually unsupported reasoning.
Peng, Lee, Yin Zhang +9
GMLab · Hong Kong Polytechnic University · XPENG Robotics
Vision-Language Models (VLMs) frequently suffer from visual perception errors and hallucinations that compromise answer accuracy in complex reasoning tasks. Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising solution by optimizing policies using answer correctness signals. Despite their effectiveness, prevailing RLVR methods face two critical limitations. First, much of the sampling budget is wasted on trajectories doomed to fail due to early visual description errors. Second, sparse rewards cannot distinguish whether failures stem from visual perception or reasoning stages. We introduce MIRL, a decoupled framework that addresses both limitations by leveraging mutual information (MI) between generated descriptions and visual inputs as a cheap pre-screening signal. This enables intelligent budget allocation toward high-potential trajectories via forking, while decoupled training provides independent MI-based rewards for visual perception optimization, resolving reward blindness. Experiments on six vision-language reasoning benchmarks demonstrate that MIRL achieves 70.22% average accuracy and successfully surpasses the performance of sampling 16 complete trajectories using only 10 pre-samples with top-6 selection (25% fewer complete trajectories). Our code is available at: https://anonymous.4open.science/r/mirl-main/.
Yin Zhang, Jiaxuan Zhao, Zonghan Wu +5
School of Mathematics, Tianjin University, Tianjin, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · Shanghai Advanced Institute of Finance (SAIFS), East China Normal University, Shanghai, China +4