Recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) for multimodal large language models (MLLMs) have mainly focused on improving final answer correctness and strengthening visual grounding. However, a critical bottleneck remains: although models can attend to relevant visual regions, they often fail to effectively incorporate visual evidence into subsequent reasoning, leading to reasoning chains that are weakly grounded in visual facts. To address this issue, we propose Trajectory-Guided Reinforcement Learning (TGRL), which guides the policy model to integrate visual evidence into fine-grained reasoning processes using expert reasoning trajectories from stronger models. We further introduce token-level reweighting and trajectory filtering to ensure stable and effective policy optimization. Extensive experiments on multiple multimodal reasoning benchmarks demonstrate that TGRL consistently improves reasoning performance and effectively bridges the gap between visual perception and logical reasoning.
Figures & tables
Figure 1: Comparison of different supervision strategies in multimodal RLVR. (A) Outcome-only supervision may lead to perception failures, where incorrect visual evidence is used for reasoning. (B) Adding perception supervision improves visual grounding but does not guarantee correct reasoning transitions. (C) Our trajectory supervision aligns the perception–reasoning process, enabling reasoning grounded in correct visual evidence and leading to correct answers.
Figure 2: Overview of T rajectory G uided R einforcement L earning (TGRL). Given an image-question pair, we construct a rollout group by integrating on-policy and expert trajectories. Expert trajectories are then reweighted at the token level and filtered by correctness, enabling trajectory-level alignment within RLVR framework.
Method
WeMath
MathVista
MathVerse
MathVision
HalluBench
Avg.
TGRL-GRPO
70.29
71.50
51.08
27.41
72.02
58.46
w/o Filter
68.74
70.80
50.65
26.68
71.04
57.50
w/o Reweight
67.58
70.40
50.14
27.11
69.90
57.03
w/o Expert (GRPO)
67.40
70.50
50.80
27.30
69.80
57.16
TGRL-DAPO
71.05
71.40
55.21
28.61
71.29
59.51
w/o Filter
69.80
70.20
51.74
27.57
70.35
57.93
Table 1: Component ablation of TGRL on Qwen2.5-VL-7B under GRPO and DAPO. Removing trajectory reweighting or filtering consistently degrades performance, demonstrating the importance of properly utilizing expert trajectories. We set β=5 for GRPO and β=14 for DAPO.
Method
WeMath
MathVista
MathVerse
MathVision
HalluBench
Avg.
GRPO
67.40
70.50
50.80
27.30
69.80
57.16
β=1
68.70
70.60
51.03
27.68
71.40
57.88
β=4
69.37
70.80
51.14
27.60
71.71
58.12
β=5
70.29
71.50
51.08
27.41
72.02
58.46
β=7
68.10
70.54
50.53
27.50
70.14
57.36
β=10
68.36
70.24
50.81
27.83
69.62
57.37
Table 2: Sensitivity analysis of the token reweighting coefficient β on the Qwen2.5-VL-7B under GRPO and DAPO. Baseline denotes vanilla GRPO/DAPO training without trajectories. Best results within each block are highlighted in bold .
Method
Data
WeMath
MathVista
MathVerse
MathVision
HalluBench
Avg.
VLAA-Thinker [ 1 ]
25K
67.90
68.80
49.90
26.90
68.60
56.42
Perception-R1 [ 33 ]
1.4K
72.00
74.20
54.30
28.60
-
-
ThinkLite [ 29 ]
11K
69.20
72.70
50.20
27.60
71.00
58.14
NoisyRollout [ 12 ]
6.4K
70.30
74.50
53.0
30.60
72.20
60.12
R1-Onevision [ 36 ]
165K
62.10
63.90
46.10
22.50
65.60
52.04
OpenVLThinker [ 3 ]
50K
67.80
71.50
48.00
25.00
70.80
56.62
Table 3: Comparison with state-of-the-art multimodal reasoning methods on five benchmarks using Qwen-2.5-VL-7B as base model. TGRL consistently improves both GRPO and DAPO across various training scales and achieves the best performance.
Hangzhou International Innovation Institute, Beihang University · Shenzhen Institute of Advanced Integration Technology, Shenzhen · Kuaishou Technology +1