Multimodal post-training has moved from broad instruction tuning toward reinforcement learning for image and video reasoning, where outcome rewards can improve final-answer accuracy. Yet these rewards are assigned to complete generated trajectories, while visual evidence is used only at sparse and uneven positions inside the response. Recent work has begun to make multimodal RL more token-sensitive through grounding signals or external token-level weighting. We ask a more specific question: can the allocation of the same trajectory-level reward be inferred from the model's own multimodal computation, rather than from an auxiliary annotator, perception module, or process label? We propose Gaze-RL, a representation-driven policy optimization method that redistributes completion-level advantage through a latent reward landing field. The field is computed from internal cross-modal geometry and local predictive uncertainty, assigning larger update weight to positions that are both visually anchored and decision-bearing while remaining mean-one, so the total credit of each rollout is preserved and only its positionwise allocation changes. Gaze-RL requires no process supervision, auxiliary perception module, or architectural modification, and improves both Qwen2.5-VL and InternVL3.5 across a broad range of image and video benchmarks.
Figures & tables
Figure 2: Average response length.
Figure 3: Response-side reward landing on diagnostic rollouts. Red and blue heatmaps show positive- and negative-return examples; darker cells indicate larger landing weight φt .
Figure 4: Visual evidence sensitivity. Guided drop masks visual cells ranked by the projected landing field; random drop masks the same budget uniformly. Larger accuracy drops indicate more necessary evidence. (c) Projected landing field of the GRPO and Gaze-RL checkpoints on two examples.
Figure 5: Ablation and sweep on Qwen2.5-VL-7B. Bars report Image Avg and Video Avg; the line reports their mean. Bars with error bars give the mean and standard deviation over three matched seeds: entropy without visual support and the default Gaze-RL configuration. The other bars are single runs. Image Avg averages MMVP, Geometry3K, MathVista, and LogicVista; Video Avg averages Video-MME, TempCompass, Video-MMMU, LongVideoBench, and VSI-Bench.
Figure 6: Image-editing transfer probe. We replace the VLM inside Qwen-Image-Edit with the Gaze-RL -trained Qwen2.5-VL-7B-Instruct while keeping the editing backbone fixed.
VLM in the editor
Overall
Qwen2.5-VL-7B (original)
7.54
+ GRPO
7.55
+ Gaze-RL (ours)
7.60
Table 4: GEdit-Bench-EN with a fixed editor. Only the VLM inside Qwen-Image-Edit is replaced.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Asset
Type
Usage in this paper
License / terms
Qwen2.5-VL-3B / Qwen2.5-VL-7B [ 4 ]
Model backbone
Main multimodal policy backbones
Apache-2.0 official release
InternVL3.5-8B [ 69 ]
Model backbone
Main multimodal policy backbone
Apache-2.0 official release
ViRL39K [ 67 ]
Training dataset
Image-side RL curriculum
MIT official dataset release
Video-R1 [ 14 ]
Training dataset / baseline
Video-side RL curriculum and video RL baseline
Apache-2.0 official dataset release
Qwen-Image-Edit [ 72 ]
Image-editing model
Image-editing transfer probe
Apache-2.0 official release
GEdit-Bench [ 39 ]
Evaluation benchmark
Quantitative image-editing transfer probe
Public benchmark terms
Appendix
Table 9: Existing assets used in this work. Licenses and terms are taken from the official model, dataset, or project release pages when available.
Reinforcement learning from verifiable rewards (RLVR), especially with Group Relative Policy Optimization (GRPO), has shown strong potential for improving the reasoning capabilities of large vision-language models (LVLMs). However, in multimodal reasoning, final-answer rewards are typically assigned at the sequence level and do not distinguish the functional roles of different tokens, making it difficult to determine whether a correct answer is supported by task-relevant visual evidence. In this paper, we revisit multimodal RLVR from the perspective of role-aware token-level credit assignment, where structured responses are decomposed into perception tokens for extracting visual evidence and reasoning tokens for deriving answers from that evidence. Based on this perspective, we propose Structured Role-aware Policy Optimization (SRPO), which refines the sequence-level GRPO advantage into role-aware token-level advantages without changing the reward function. Specifically, SRPO assigns role-specific credit by using self-distilled on-policy contrasts: perception tokens are emphasized according to their visual dependency under original versus corrupted visual inputs, while reasoning tokens are emphasized according to their consistency with the generated perception. These role-specific signals are further unified through a shared trajectory-level baseline, yielding positive token weights that adjust relative update magnitudes while preserving the original GRPO reward and optimization direction, without requiring external reward models or separate teachers. Experiments across diverse multimodal reasoning benchmarks show that SRPO improves evidence-grounded reasoning, highlighting the importance of moving beyond uniform sequence-level credit toward role-aware optimization for reliable multimodal reasoning.
Bingqing Jiang, Difan Zou
School of Computing & Data Science, The University of Hong Kong · School of Computing & Data Science and Institute of Data Science, The University of Hong Kong
Recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) for multimodal large language models (MLLMs) have mainly focused on improving final answer correctness and strengthening visual grounding. However, a critical bottleneck remains: although models can attend to relevant visual regions, they often fail to effectively incorporate visual evidence into subsequent reasoning, leading to reasoning chains that are weakly grounded in visual facts. To address this issue, we propose Trajectory-Guided Reinforcement Learning (TGRL), which guides the policy model to integrate visual evidence into fine-grained reasoning processes using expert reasoning trajectories from stronger models. We further introduce token-level reweighting and trajectory filtering to ensure stable and effective policy optimization. Extensive experiments on multiple multimodal reasoning benchmarks demonstrate that TGRL consistently improves reasoning performance and effectively bridges the gap between visual perception and logical reasoning.
Jinda Lu, Junkang Wu, Jinghan Li +6
University of Science and Technology of China · Pecking University · Institute of Software, Chinese Academy of Sciences
Reinforcement learning with verifiable rewards (RLVR) improves vision-language benchmark scores even without visual information during training. With images at test, blind-trained models recover roughly half of the real-image gain at 3B and nearly four fifths at 7B. Prolonged real-image training can erode grounding while benchmark gains persist. Both findings expose the same gap: an image in the prompt is not an image in the learning signal. Our design rule, visual resolvability, asks that visual evidence be necessary for a correct answer and that the task remain learnable. We test it on counterfactual coordinate scenes in which the question stays fixed and the target is never named, so a correct answer requires finding the target in the image. With standard GRPO and correctness-and-format rewards, a 7B model raises its accuracy at finding the target (discovery) from 0.425 to 0.875 on held-out scenes denser than any it trained on, and it improves on question types it never trained on. Two controls locate the source of the gain. Replacing test images with gray canvases drops discovery to zero; training on gray canvases instead, at matched step 30 and in each of four seeds, yields essentially none of the gain even when the model is then tested with real images. The learned skill carries over to grounding tasks built independently of the training corpus. A caption that answers the training question, added to the same images, reward and budget, cuts the gain by nearly two thirds. Changing what reward requires changes what RL learns.
Haocun Ye, Xinlong Jiang, Qile Chen +6
University of the Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences · Independent Researcher