Video reasoning requires models to locate and track question-relevant evidence across frames. While reinforcement learning (RL) with verifiable rewards improves accuracy, it still struggles to achieve reliable spatio-temporal grounding during the reasoning process. Moreover, improving grounding typically relies on scaled training data or inference-time perception tools, which increases annotation cost or computational cost. To address this challenge, we propose VisonCoach, an input-adaptive RL framework that improves spatio-temporal grounding through visual prompting as training-time guidance. During RL training, visual prompts are selectively applied to challenging inputs to amplify question-relevant evidence and suppress distractors. The model then internalizes these improvements through self-distillation, enabling grounded reasoning directly on raw videos without visual prompting at inference. VisonCoach consists of two components: (1) Visual Prompt Selector, which predicts appropriate prompt types conditioned on the video and question, and (2) Spatio-Temporal Reasoner, optimized with RL under visual prompt guidance and object-aware grounding rewards that enforce object identity consistency and multi-region bounding-box overlap. Extensive experiments demonstrate that VisonCoach achieves state-of-the-art performance under comparable settings, across diverse video reasoning, video understanding, and temporal grounding benchmarks (V-STAR, VideoMME, World-Sense, VideoMMMU, PerceptionTest, and Charades-STA), while maintaining a single efficient inference pathway without external tools. Our results show that visual prompting during training improves grounded video reasoning, while self-distillation enables the model to internalize this ability without requiring prompts at inference time.
Figures & tables
Figure 1: Comparison with previous video reasoning approaches and VisionCoach . Comparing with other video reasoning models, VisionCoach leverages visual-prompt-guided RL with self-distillation to internalize improved spatio-temporal grounding behaviors induced by visual guidance. At inference time, it maintains a single forward-pass reasoning while achieving enhanced grounding performance.
Figure 2: Detailed Architecture of VisionCoach . We introduce VisionCoach , a visual-prompt-guided RL framework for training a spatio-temporally grounded reasoner ( Section 3.2 ). The framework includes Visual Prompt Selector ( VP-Selector ) that predicts optimal visual prompts for challenging inputs ( Section 3.3 ), and object-aware spatial grounding rewards that enforce object identity consistency and multiple predicted bounding boxes ( Section 3.4 ).
Figure 3
Model
What
When (Temporal IoU)
Where (Spatial IoU)
Overall
Acc
Chain1
Chain2
Chain1
Chain2
mAM
mLGM
Proprietary Models
Gemini-2-Flash [ 23 ]
53.0
24.5
23.8
4.6
2.2
26.9
35.6
GPT-4o [ 32 ]
60.8
16.7
12.8
6.5
3.0
26.8
38.2
Tool-calling Methods
VideoZoomer [ 17 ]
47.7
11.1
3.7
2.0
0.5
18.8
24.6
Table 2: Performance on the V-STAR benchmark. VisionCoach achieves strong spatio-temporal reasoning across overall dimensions.
Model
VideoMME
WorldSense
VideoMMMU
PerceptionTest
Overall
Long
Overall
Recognition
Overall
Perception
Overall
Proprietary Models
GPT-4o
71.9
-
42.6
-
61.2
66.0
-
Tool-calling Methods
EgoR1
-
64.9
-
-
-
-
-
EgoR1 (w/o RAG)
58.7
50.8
42.0
40.3
34.4
37.6
65.8
Table 3: Performance across different general video understanding benchmarks. * indicates our implementation performance.
Table 5: Zero-shot spatio-temporal grounding performance on HCSTVG.
Table 8
Figure 4: Additional analysis of VisionCoach . We provide analysis including (a) statistics of visual prompting, (b) inference latency, and (c) the effect of visual prompting on spatial grounding reward.
Figure 5: Spatio-temporal attention map. Visual prompting (VP) improves grounding in both temporal and spatial dimensions. The green box indicates the key frame, while the red box highlights the corresponding spatial region. Temporally, VP increases attention on the correct key frame containing the cowboy. Spatially, VP concentrates attention on the region corresponding to the queried visual attributes (e.g., the cowboy wearing specific clothing), while suppressing irrelevant regions.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: VP-Selector training data generation pipeline. Given a video-question pair, we construct multiple visual prompting candidates, generate grounded reasoning with proxy reasoners, and derive pseudo-labels for selecting the most effective visual prompt.
Figure 7: Distribution selected visual prompting of VP-Selector training datasets.
Table 13Table 14Table 15
Acc
Where (Chain 1)
Where (Chain 2)
mAM
mLGM
VideoMME
Open-o3-video ( rspamax )
58.4
25.4
4.6
30.3
41.4
61.9
+ Average IoU ( rspaavg )
58.7
28.0
5.0
30.1
41.3
61.9
+ Identity matching ( rspa )
59.9
27.5
5.4
31.3
43.1
62.1
Appendix
Table 16: Ablation of object-aware spatial reward components on V-STAR and VideoMME.
Avg.obj
Avg.box
Open-o3-video ( rspamax )
0.25
0.38
+ Average IoU ( rspaavg )
0.70
1.18
+ Identity matching ( rspa )
0.83
1.40
Appendix
Table 17: Average number of predicted objects and bounding boxes per generated reasoning.
Gating method
Acc
mAM
mLGM
VideoMME
-
59.6
30.4
41.6
61.9
Fixed (darken)
58.3
29.7
40.6
62.4
Fixed (circle)
57.7
29.3
39.9
62.0
Gating (darken, frame#)
59.6
30.6
42.7
62.6
VP-Selector
60.7
31.3
43.1
62.8
Appendix
Table 18: Ablation of visual prompting assignment to hard samples on V-STAR and VideoMME.
Figure 8: Comparison of grounding with Open-o3-video and VisionCoach .
Figure 9: Prompt to proxy reasoner.
Figure 10: Prompt for extracting key object from question and answer pair.
School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · OPPO AI Center, OPPO Inc.