Reliable video reasoning requires models to produce correct answers and expose supporting visual evidence. Open-o3-Video enables verifiable reasoning by embedding timestamps and object-level bounding boxes into reasoning traces, but is restricted to frames from an initial sparse sampler. We introduce Open-o3-Video-v2, which formulates video reasoning as a unified agentic evidencegrounded process. The model identifies missing information, acquires additional observations, grounds the moments and objects supporting its reasoning, and writes this evidence into the trajectory before continuing or answering. To support this behavior, we construct STGR-v2-CoT135K, including 52K multi-turn agentic grounded reasoning trajectories, and STGR-v2-RL-13K for reinforcement learning. We further propose Clue-Seek GRPO (CS-GRPO), which tracks how tool calls change accumulated evidence, rewards clue coverage and step-wise gains, and penalizes excessive, repetitive, or uninformative retrieval. On V-STaR, Open-o3-Video-v2-4B achieves 40.4% mAM and 56.3% mLGM, outperforming its Qwen3-VL-4B backbone by 8.5% and 12.6%, respectively. It also delivers consistent gains across general video understanding, long-video reasoning, and temporal grounding benchmarks. These results show that active evidence acquisition and verifiable object-level grounding are complementary for reliable video reasoning. The code and models are released at https://msalab-pku.github.io/projects/Open-o3-Video-v2.
Figures & tables
Figure 1: From single-pass spatio-temporal grounded reasoning to agentic evidence seeking and grounded reasoning. Left: Open-o3-Video-v1 reasons over fixed sampled frames and grounds visible evidence, yielding verifiable spatio-temporal support on short videos. Center: Open-o3-Video-v2 performs adaptive temporal zoom-in through iterative crop_video calls, following a seek–ground–reason process to find evidence beyond the initial frames for long video understanding. Right: Comparison of five models on V-STaR and six general video understanding and reasoning benchmarks.
Method
Adaptive temp. res.
Temp. ev.
Obj. ev.
Native agent
Open-o3-Video [ 38 ]
✗
✓
✓
✗
VideoZoomer [ 15 ]
✓
✗
✗
✓
Video-o3 [ 67 ]
✓
✗
✗
✓
VideoTemp-o3 [ 36 ]
✓
✗
✗
✓
VideoSeeker [ 76 ]
✓
✗
✗
✓
Open-o3-Video-v2
✓
✓
✓
✓
Table 1: Capability comparison with closely related video reasoning methods. “Adaptive temp. res.” denotes adaptive temporal resampling, “Temp. ev.” temporal evidence explicitly cited in reasoning, “Obj. ev.” object-level evidence explicitly cited in reasoning, and “Native agent” multi-turn interaction with interleaved observations.
Figure 2: Overview of the data construction pipelines and STGR-v2 data statistics. (a) Open-o3-Video constructs single-turn spatio-temporal grounded reasoning data through initial annotation, bounding-box filtering, and self-consistency checking. (b) Open-o3-Video-v2 extends this pipeline to agentic multi-turn reasoning through strict data filtering, spatio-temporal evidence annotation, CoT (chain-of-thought) rewriting with tool observations, and quality checking. (c) STGR-v2-CoT-135K contains 135K SFT samples across seven task groups, including 59K spatio-temporal grounded reasoning samples, while STGR-v2-RL-13K contains 13K RL samples selected through strict filtering.
Figure 3: Overview of the reinforcement learning designs in Open-o3-Video and Open-o3-Video-v2. (a) Open-o3-Video produces a single-turn spatio-temporal grounded reasoning trace and optimizes it with format, answer-accuracy, and grounded thinking rewards; the thinking reward jointly evaluates temporal proximity and spatial overlap with the target evidence. (b) Open-o3-Video-v2 extends this formulation to multi-turn agentic reasoning with CS-GRPO. The model iteratively calls crop_video to acquire denser observations, while CS-GRPO tracks how the accumulated evidence evolves across calls and assigns credit according to whether each interaction contributes useful new evidence.
Model
What
When (Temporal IoU)
Where (Visual IoU)
Overall
Acc
Chain1
Chain2
Chain1
Chain2
mAM
mLGM
GPT-4o [ 29 ]
60.8
16.7
12.8
6.5
3.0
26.8
38.2
Gemini-2-Flash [ 13 ]
53.0
24.5
23.8
4.6
2.2
26.9
35.6
Gemini-3-Pro [ 13 ]
59.1
24.2
23.8
7.2
4.8
29.9
41.2
Video-LLaMA3-7B [ 69 ]
41.9
23.0
23.1
0.9
0.2
21.7
27.0
LLaVA-Video-7B [ 75 ]
49.5
10.5
12.2
1.9
1.3
20.8
27.3
Table 2: Performance on V-STaR (%). Chain1 denotes what–when–where , whereas Chain2 denotes what–where–when . mAM and mLGM jointly summarize answer accuracy and spatio-temporal alignment. Bold and underlined numbers denote the best and second-best results, respectively.
Model
General Video Understanding and Reasoning
Temporal Grounding
Video- Holmes
LV Bench
Video- MMMU
WorldSense
Video MME
Video MME-v2
Charades- TimeLens
TVG Bench
Acc
Acc
Acc
Overall
Recog.
Acc
Acc
mIoU
mIoU
Qwen2.5-VL-7B [ 3 ]
34.7
45.3
51.2
36.1
33.7
62.4
25.3
39.3
16.3
Qwen3-VL-4B [ 2 ]
37.3
46.0
54.4
41.7
42.0
65.3
26.2
45.6
27.1
Qwen3-VL-8B [ 2 ]
44.2
47.3
57.3
45.7
44.0
67.4
27.9
48.3
29.4
VideoRFT-7B [ 53 ]
40.3
40.4
51.1
38.2
36.6
59.8
22.4
17.0
14.3
Table 3: Performance on general video understanding and temporal grounding benchmarks (%). For WorldSense, we report overall and recognition accuracy. The last two columns report mIoU. All other columns report accuracy. Bold and underlined numbers denote the best and second-best results, respectively.
Figure 4: Qualitative comparison with Video-o3. Left: Video-o3 searches an incorrect segment, while Open-o3-Video-v2 inspects the relevant 1170–1340 s interval and grounds the pigs at 1310 s, providing direct evidence for answer D. Right: although the question explicitly specifies 07:48–07:50, Video-o3 queries 478–480 s and incorrectly predicts bowing. Open-o3-Video-v2 inspects the correct 468–470 s interval, grounds the three girls at 469 s, and identifies their group handshake. In both cases, the acquired observation is converted into explicit spatio-temporal evidence and used directly in the subsequent reasoning process.
Model
V-STaR
VideoMME
mAM
mLGM
Acc
Qwen3-VL-4B (baseline)
31.9
43.7
65.3
Open-o3-Video-v2-SFT-4B (SFT)
38.2
53.4
63.7
Open-o3-Video-v2-4B (SFT+RL)
40.4
56.3
66.6
Table 4: Stage-wise ablation of Open-o3-Video-v2-4B (%). V-STaR is evaluated with mAM and mLGM, while VideoMME is evaluated with accuracy.
Reasoning mechanism
Video-Holmes
LVBench
V-STaR mLGM
Qwen3-VL-4B (base) [ 2 ]
37.3
46.0
43.7
Single-turn, no evidence
45.0
48.8
47.8
Multi-turn, no evidence
47.1
51.9
52.3
Multi-turn with grounded evidence (ours)
48.0
53.2
56.3
Table 5: Comparison of reasoning mechanisms. All variants use the same backbone, source training data, and evaluation setting. Only the reasoning mechanism is changed.
Method
Video-Holmes
LVBench
VideoMME
GRPO
46.7
52.0
65.1
CS-GRPO w/o gain bonus
47.0
52.2
66.2
CS-GRPO w/o penalties
46.9
50.4
65.5
CS-GRPO (ours)
48.0
53.2
66.6
Table 6: Comparison of GRPO and CS-GRPO variants across three video understanding benchmarks (%).
Setting
SFT
CS-GRPO
Training framework
ms-swift
verl AgentLoop
Model initialization
Qwen3-VL-4B-Instruct
SFT checkpoint
Training corpus
STGR-v2-CoT-135K
STGR-v2-RL-13K
Training epochs
1
1
Global batch size
16
32
Learning rate
5×10−6
1×10−6
Table 7: Training and optimization configurations of Open-o3-Video-v2-4B.
Stage
Initial Observation
Coarse Crop
Medium Crop
Fine Crop
Interaction Limit
SFT
1 FPS, ≤96 frames
1 FPS, ≤96 frames
2 FPS, ≤96 frames
4 FPS, ≤96 frames
Given by annotated trajectory
CS-GRPO
1 FPS, ≤32 uniform frames
≤8 frames
≤16 frames
≤32 frames
≤5 assistant turns
Inference
1 FPS, ≤96 frames
Model-selected crop_video observations
≤5 assistant turns
Table 8: Video observation and interaction settings of Open-o3-Video-v2. SFT frame caps are applied to each observation. For CS-GRPO, crop strategies control the maximum number of uniformly sampled frames within the requested interval rather than fixed sampling rates.
Table 13Table 14
Setting
WorldSense
VideoMMMU
Single response ( N=1 )
37.5
52.3
Majority voting ( N=8 )
37.3
53.1
Confidence-aware voting ( N=8 )
38.5
54.1
Table 13: Test-time scaling of Open-o3-Video on WorldSense and VideoMMMU (%). Confidence-aware voting uses grounded evidence to weight eight sampled responses.
Model
QA
S
T
S&T
QA&S
QA&T
QA&S&T
Qwen3-VL-4B
59.6
52.7
25.9
17.8
40.1
15.6
11.7
Open-o3-Video-v2-4B
62.8
68.0
41.5
32.8
50.1
26.4
21.7
Table 14: Consistency between answer correctness and spatio-temporal localization on V-STaR Chain1 (%). Spatial correctness uses vIoU >0.1 , and temporal correctness uses tIoU >0.3 .
Figure 5: Effect of initial input-frame count and maximum interaction budget on LVBench and Video-Holmes. Results are evaluated on randomly sampled 30% subsets of each benchmark using the same model and decoding configuration.
Reward level
Coefficient
Low / default / high
Video-Holmes
V-STaR mLGM
Final reward R
wacc
0.5/1.0/1.5
47.6 / 48.0 / 48.0
55.8 / 56.3 / 56.2
wfmt
0.05/0.1/0.2
48.1 / 48.0 / 48.2
56.1 / 56.3 / 55.7
wthk
0.05/0.1/0.2
48.4 / 48.0 / 47.6
55.9 / 56.3 / 56.6
wtool
0.5/0.8/1.0
47.3 / 48.0 / 47.9
56.0 / 56.3 / 56.3
Tool reward rtool
λclue
0.5/1.0/1.5
47.6 / 48.0 / 48.1
56.0 / 56.3 / 56.6
λgain
0.05/0.1/0.2
47.3 / 48.0 / 47.8
55.4 / 56.3 / 57.1
Table 15: Sensitivity of CS-GRPO to reward weights. Each coefficient is varied independently while all others remain at their default values. Results follow the low/default/high order of the tested weights.
Setting
VideoMME
Video-Holmes
LVBench
Uniform sampling (120 frames, no tool)
65.4
47.4
45.5
Length-matched random intervals
64.6
47.1
46.2
Open-o3-Video-v2
66.6
48.0
53.2
Table 16: Analysis of adaptive evidence acquisition under comparable visual observation budgets (%). The uniform baseline uses the same model with tool use disabled and uniformly samples 120 frames. For random retrieval, each tool-returned clip is replaced by a length-matched random interval from the same video while preserving the number of tool calls and returned frames.
Model
Accuracy (%)
Qwen3-VL-4B
37.6
Video-o3
38.3
Open-o3-Video-v2
48.3
Table 17: Accuracy on a 149-question long-horizon subset of LVBench, covering long videos with sparse evidence and questions involving event chains, ordinal localization, or repeated-instance tracking.
Figure 6: Qualitative comparisons on fine-grained visual reasoning. Top: For action-change perception, Open-o3-Video-v2 inspects the relevant 134–143 s interval and grounds the man’s hand, sea urchin, and fish at 138.5 s. The denser observation reveals the fish’s blue body and yellow fins after the interaction, enabling the correct answer. Bottom: For fine-grained event understanding, Open-o3-Video-v2 grounds the metal ring cutter and pineapple slice and further inspects the 33–46 s interval using denser temporal sampling. The resulting observation reveals the repeated removal of the fibrous core and provides the missing evidence for answer B. In both cases, targeted temporal observation together with explicit object-level grounding exposes the decisive visual evidence and yields a human-verifiable spatio-temporal reasoning trace.
Figure 7: Qualitative comparisons on fine-grained visual counting. Top: For counting food pictures on the menu at 17:49, Open-o3-Video-v2 inspects the relevant 1065–1075 s interval and grounds the menu at 1069.5 s, providing a clearer view of the individual food images and enabling the correct count of 11. In contrast, Video-o3 searches an incorrect temporal interval and produces an incorrect count. Bottom: For counting the people in the car, both methods inspect the relevant 10–12 s interval, but Open-o3-Video-v2 further grounds the car and its occupants, allowing it to distinguish three individuals rather than two. In both cases, explicit instance-level grounding complements temporal evidence acquisition by resolving the individual visual entities required for accurate counting.
Figure 8: Representative failure cases of Open-o3-Video-v2. Top: Insufficient identity knowledge leads to an incorrect answer even after a relevant face is retrieved. Middle: During a camera pan, inaccurate instance discrimination and association across changing views lead to an incorrect count of the women appearing offstage. Bottom: The model anchors its reasoning to the wrong scoring event and misidentifies the player who ends the game. These examples expose remaining limitations in world knowledge, fine-grained spatio-temporal instance reasoning, and temporal event selection, respectively.
Recent video reasoning models increasingly produce spatio-temporal evidence chains that localize objects at specific timestamps. While these traces improve interpretability by grounding where and when evidence appears, they often leave the motion connecting observations, the how, implicit. This makes dynamic and trajectory-dependent claims difficult to supervise, verify, or penalize when unsupported by the video. We formalize this missing component as Spatial-Temporal-Trajectory (STT) reasoning and introduce Motion-o, a motion-centric extension to vision-language models (VLMs) that makes trajectories explicit and verifiable. Motion-o augments evidence chains with Motion Chain of Thought (MCoT), a structured pathway that represents object motion through a discrete <motion/> tag summarizing direction, speed, and scale change. To supervise MCoT, we densify sparse spatio-temporal annotations into object tracks and derive motion descriptors from centroid displacement and box-area change. We then train with complementary rewards for trajectory consistency and visual grounding, including a perturbation-based signal that penalizes motion descriptions that remain unchanged when temporal evidence is removed. Across multiple video understanding benchmarks, Motion-o consistently improves trajectory-faithful reasoning without architectural modifications. These results suggest that an explicit motion interface can complement existing VLM pipelines by converting implicit dynamics into verifiable evidence. Code is available at~\faGithub\ ostadabbas/Motion-o.
Video MLLMs often struggle with fine-grained spatio-temporal reasoning, sometimes generating correct answers based on irrelevant frames or objects. Although outputting spatio-temporal evidence during reasoning is a promising direction, existing RL frameworks typically rely on geometry-only (IoU) rewards, which can be sensitive to boundary perturbations and overlook semantic alignment. To address this, we propose Semantic Evidence Reward (SER), which reformulates spatio-temporal evidence grounding as a constrained verification task. Instead of computing pixel-level overlap, SER uses a referee VLM as a local checker to evaluate model-generated evidence claims across two dimensions: relevance and localization quality, combined with a temporal penalty. This design reduces the reliance on dense box annotations and enables training directly on standard video QA data. On the V-STAR benchmark, SER achieves 49.6% mLGM, improving by 3.0 points over the strong evidence-grounded baseline Open-o3-Video, demonstrating its potential in enhancing both answer accuracy and evidence grounding.
Sheng Xia, Zhengqin Lai, Tianxiang Jiang +4
Nanjing University · Shanghai Innovation Institute · Harbin Institute of Technology, Shenzhen +2
Current Video Large Language Models (Video LLMs) excel in question answering (QA) but largely operate as black boxes, providing textual answers without verifiable visual grounding. Existing explainability efforts rely on textual rationales or sparse bounding boxes, which struggle to capture complex video dynamics such as occlusions and non-rigid deformations. We propose Evidence-Backed Video Question Answering (E-VQA), a novel task requiring models to jointly output a semantic answer and precise spatio-temporal evidence: temporal segments and dense, tracked object segmentation masklets. To support this, we introduce ST-Evidence, the first human-verified benchmark for both discriminative and generative pixel-level grounding. Evaluations of state-of-the-art models reveal a critical decoupling between QA accuracy and true visual perception that scaling alone fails to bridge. To address this, we develop scalable, automated generation pipelines to create ST-Evidence-Instruct, a 160k-scale dataset bridging high-level reasoning with fine-grained grounding. Fine-tuning grounded Video LLMs on this data yields substantial gains over the corresponding size-matched UniPixel baselines (e.g., +27.2 t-mean and +13.8 J&F on a 7B model), establishing a robust baseline for explainable, evidence-backed video understanding. Code and data are available at https://github.com/SalesforceAIResearch/EVQA.
Shijie Wang, Honglu Zhou, Ziyang Wang +5
Salesforce, Palo Alto, CA, USA · Brown University, Providence, RI, USA