cs.CVOct 14, 2026

RL Teaches VLMs Where to Attend but Not How to Perceive

Authors: Ziqian Zhang, Junjie Liao, Zhining Zhang, Wentao Zhu

Organizations: National University of Singapore · Beijing Normal University · Peking University · Eastern Institute of Technology, Ningbo

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Although reinforcement learning (RL) effectively trains Vision-Language Model (VLM) agents, the mechanisms behind these capability gains are poorly understood. Across four diverse agent environments, we investigate whether the task-success gains from multi-turn RL are accompanied by improved grounded visual question answering about the environment. To assess this, we design environment-specific grounded VQA tasks that reveal improvements in grounded visual understanding. Representational probing shows that vision-encoder probe accuracy remains relatively stable, while LLM-layer probe accuracy improves. Attention analysis further reveals that LLM layers pay more attention to visual tokens during RL training. To test whether increased attention to visual tokens contributes to VQA gains, we introduce a lightweight plug-and-play intervention using a fixed attention bias. Without updating model parameters or computing gradients, this intervention partially restores VQA gains on the untrained model. These results reveal better utilization of visual information as a key mechanism behind the gains from RL.

Figures & tables

Figure 1: Overview of our evaluation framework and attention bias intervention. Left: RL progressively improves multi-turn task performance in FrozenLake. Top right: VQA evaluation with layer-wise probing reveals that RL enhances visual utilization in deep LLM layers. Bottom right: RL training deepens the attention weights on visual tokens (darker colors) in deep LLM layers; injecting an additive attention bias into the base model partially replicates this shift and improves VQA accuracy.
Figure 2: Representative visual observations from the four environments. A. FrozenLake. B. EzPoints. C. Sokoban. D. Navigation.
Figure 3: Comparison of VQA accuracy for the RL agent and SFT fine-tuning in FrozenLake, EzPoints, Sokoban, and Navigation environments. Compared to the SFT model (red), the VQA accuracy of the RL agent (blue) achieves greater improvement as the number of training steps increases across all four environments.
Figure 4: Probe accuracy of the visual encoder and two LLM layers across RL training checkpoints in FrozenLake, EzPoints, Sokoban, and Navigation. The visual encoder (green) remains flat throughout training, while LLM Layer 17 (blue) and LLM Layer 35 (red) improve steadily, with most gains concentrated in the early checkpoints.
Figure 5: Evolution of image RAPT (blue, left axis) and text RAPT (red, right axis) across RL training checkpoints. Image RAPT rises steadily while text RAPT declines, indicating that RL systematically shifts attention from text toward visual tokens.
ConfigurationFrozenLakeEzPointsSokobanNavigation
No intervention28.8982.2225.0078.75
Image token + deep layers33.8983.3327.8680.50
Text token + deep layers23.3377.7824.2978.75
All prompt tokens + deep layers30.5681.6726.4378.88
Image token + shallow layers31.6781.6726.4379.00
Image token + all layers33.8982.2226.4380.63
Table 1: Ablation study of the fixed-bias attention intervention on the base model. Accuracy (%) is reported for six configurations across four environments; the best result in each column is in bold.
Figure 6: Attention heatmaps for shallow (left) and deep (right) layers in FrozenLake. Shallow layers distribute attention diffusely across the grid, while deep layers concentrate on task-relevant objects (the hole), consistent with the layer-wise attention shift observed during RL training.
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Accuracy gain of the trained plugin (blue) and fixed-bias variant (red) over the RL baseline across checkpoints in FrozenLake, EzPoints, and Sokoban. The trained plugin consistently achieves larger gains, while the fixed-bias variant shows higher variance and occasionally falls below the RL baseline, confirming that the learned bias values are key to reliable improvement.
Figure 8: Image RAPT (A) and text RAPT (B) before and after plugin training across three environments. Plugin optimization consistently increases image RAPT while decreasing text RAPT in all three environments, mirroring the attention redistribution direction induced by full RL fine-tuning.
HyperparameterFrozenLakeEzPointsSokoban
Total training steps150150150
Actor learning rate1×10−61\times 10^{-6}1×10−61\times 10^{-6}1×10−61\times 10^{-6}
Train batch size161632
PPO mini-batch size323232
Max prompt length (tokens)1,0241,0241,024
Max response length (tokens)200256512
Appendix
Table 2: RL (GRPO) hyperparameters.
ArchitectureFrozenLakeEzPointsSokoban
Qwen3VL-O42.2290.5637.86
Qwen3VL-P53.3392.2239.29
LLaVA-O41.6773.3322.86
LLaVA-P44.4473.8924.29
Gemma-O24.4490.0010.71
Gemma-P27.2291.6713.57
Appendix
Table 3: VQA accuracy (%) of the attention bias plugin on Qwen3-VL, LLaVA, and Gemma across FrozenLake, EzPoints, and Sokoban.
HyperparameterValue
Training epochs4
Learning rate1×10−51\times 10^{-5}
LR schedulecosine
Warmup ratio0.1
Gradient accumulation2
Max sequence length4,096
Appendix
Table 4: SFT hyperparameters.
CheckpointVQAVision encoderLLM L17LLM L35
042.2272.2278.8988.89
144.5672.7879.4489.44
246.1173.8980.5690.00
345.8971.6780.0089.44
447.4472.2281.1190.00
548.0074.4481.6791.11
Appendix
Table 5: VQA and probe accuracy (%) of Qwen3-VL-4B across RL checkpoints in FrozenLake.
More in this topic

RL for VLMs

Reinforcement learning for vision-language models aligns multimodal reasoning, tool use, and action with task-specific feedback.

See all 452 papers

Explore similar work

CardsList

  1. Perception or Reasoning? Diagnosing and Tackling Reinforcement Learning Bottlenecks in VLMs

    Oct 15, 2026Zeyuan Yang, Zhuohao Chen, Jiaben Chen +3Attention-Guided DistillationRL for VLMs

    UMass Amherst · UIUC · MIT

  2. Be Faithful When Response: Returning Fluent and Grounded Answers for Vision-Language Models Reinforcement Learning

    Jun 29, 2026Peng, Lee, Yin Zhang +9RL for VLMsVision-Language Grounding

    GMLab · Hong Kong Polytechnic University · XPENG Robotics

  3. MIRL: Mutual Information-Guided Reinforcement Learning for Vision-Language Models

    May 2, 2026Yin Zhang, Jiaxuan Zhao, Zonghan Wu +5VLM ReasoningVision-Language Models

    School of Mathematics, Tianjin University, Tianjin, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · Shanghai Advanced Institute of Finance (SAIFS), East China Normal University, Shanghai, China +4