cs.CVOct 15, 2026
SaveEvent-STVG: Spatio-Temporal Video Grounding for Long-Form Event Reasoning
Organizations: Tsinghua University · Beijing University of Posts and Telecommunications
Abstract
Spatio-Temporal Video Grounding (STVG) aims to jointly localize a target in space and time given a natural-language query. Despite continued advances, existing STVG benchmarks remain largely centered on localizing targets through instance-specific descriptions, leaving them largely decoupled from long-form reasoning increasingly emphasized in multimodal large language models (MLLMs). In this paper, we introduce Event-STVG, a task that shifts STVG from locating what is described toward event-level reasoning by unifying long-form context, repeated-event reasoning, event localization, role inference, and multi-target grounding. To facilitate research on Event-STVG, we construct a large-scale benchmark comprising 4,016 annotated event occurrences across soccer and seven additional domains. Experiments with state-of-the-art MLLMs reveal substantial difficulty, with the strongest model achieving only 5.4% m-vIoU at the 10-minute horizon. Across domains, we consistently observe confusion among repeated same-type events, spatial grounding bottlenecks, and rare success in jointly grounding all required roles. Event-STVG establishes a new challenge for visual understanding in MLLMs, calling for future models that can jointly reason about complex events across space and time. Code and annotations are available at https://github.com/THU-KEG/EventSTVG
Figures & tables
Figure 1: Model performance on Event-STVG across different video durations.
Figure 2: Comparison between Event-STVG and existing STVG. Existing STVG primarily performs description-guided localization, whereas Event-STVG requires complex event-level reasoning.
| Benchmark | Videos | Duration (s) | Anno | TG | SG | Joint Grounding | Event–Role Grounding |
|---|---|---|---|---|---|---|---|
| Video-MME | 900 | 1,017.9 | M | ✗ | ✗ | ✗ | ✗ |
| LVBench | 103 | 4,101 | M | ✗ | ✗ | ✗ | ✗ |
| VideoZeroBench | 138 | 667.14 | M | ✓ | ✓ | ✗ | ✗ |
| CaST-Bench | 1,015 | 13.68 | A&M | ✓ | ✓ | ✓ | ✗ |
| ST-Evidence | 2,896 | NR | A&M | ✓ | ✓ | ✓ | ✗ |
| VidEvent | 1,110 | 82 | M | ✓ | ✗ | ✗ | ✗ |
Table 1: Comparison of Event-STVG with related benchmarks. TG/SG: temporal/spatial grounding; Joint Grounding: joint spatio-temporal grounding; Event–Role Grounding: spatial grounding of event roles. M: manual annotation; A&M: automatic and manual annotation; NR: not reported. Duration reports mean video length. Videos count all evaluation settings.
Figure 3: Overview of Event-STVG benchmark construction and composition. We construct Event-STVG through carefully designed human annotation with automated tracking assistance and multi-stage quality control. The resulting benchmark exhibits substantial diversity in event types and semantic roles, covering major soccer events and seven additional domains.
| Oracle | 1 minute | ||||||
| Model | m-vIoU | [email protected] | [email protected] | m-tIoU | m-vIoU | [email protected] | [email protected] |
| General-Purpose VLMs | |||||||
| Gemini-3.7 Flash | 24.18 | 38.40 | 11.35 | 43.39 | 10.50 | 12.66 | 3.56 |
| GPT-5.6 Sol | 13.38 | 13.99 | 5.49 | 48.23 | 2.27 | 0.43 | 0.11 |
| Qwen-3.8 Max | 11.42 | 12.43 | 2.84 | 46.46 | 3.95 | 1.34 | 0.11 |
| Qwen-3.7 Plus | 2.73 | 0.71 | 0.09 | 38.47 | 0.42 | 0.00 | 0.00 |
Table 2: Performance on the Event-STVG benchmark (3,516 events in soccer; %) under the Oracle and 1-minute settings. Bold indicates the best result in each column.
| 3 minutes | 5 minutes | 10 minutes | ||||
|---|---|---|---|---|---|---|
| Model | m-tIoU | m-vIoU | m-tIoU | m-vIoU | m-tIoU | m-vIoU |
| Gemini-3.7 Flash | 33.21 | 11.36 | 25.14 | 7.37 | 21.54 | 5.40 |
| GPT-5.6 Sol | 43.16 | 1.55 | 41.22 | 1.53 | 36.82 | 1.29 |
| Qwen-3.8 Max | 39.00 | 2.00 | 32.67 | 0.94 | 24.57 | 0.80 |
Table 3: Performance on the fixed 300-query Event-STVG analysis subset (300 events in soccer; %) under the 3-, 5-, and 10-minute settings. Bold indicates the best result in each column.
| Gemini-3.7 Flash | GPT-5.6 Sol | Qwen-3.8 Max | Mean | |||||
|---|---|---|---|---|---|---|---|---|
| Domain | Proportion (%) | m-tIoU | m-vIoU | m-tIoU | m-vIoU | m-tIoU | m-vIoU | m-vIoU |
| Daily work | 20 | 53.97 | 21.84 | 53.04 | 28.35 | 45.29 | 13.71 | 21.30 |
| Entertainment | 20 | 25.07 | 16.82 | 27.27 | 19.90 | 15.36 | 9.21 | 15.31 |
| Animal activity | 3 | 20.65 | 7.83 | 36.14 | 8.06 | 19.30 | 5.12 | 7.00 |
| Volleyball | 18 | 34.75 | 8.47 | 44.95 | 3.57 | 31.82 | 5.23 | 5.76 |
| Esports | 10 | 42.87 | 7.24 | 47.65 | 2.97 | 35.24 | 0.48 | 3.56 |
Table 4: Performance on Event-STVG (500 events in multiple domains; %). Domains are ranked by mean m-vIoU across the three models.
Figure 4: Bottleneck I: temporal instance resolution across video durations, with and without same-type distractors. The panels report target hit rate, instance confusion rate, and temporal intersection over union (tIoU) on predictions that consistently hit the target across all durations in soccer videos.
Figure 5: Bottleneck II and III: spatial and multi-target grounding bottlenecks. (a) Relative Spatial Grounding Gap among temporally overlapping predictions. (b–d) Proportions of multi-target events for which each model localizes all, some, or none of the required role fillers under Oracle and different video-duration settings, using per-role full-trajectory as the localization criterion.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Cross-scenario prompt examples in Event-STVG. Each row shows an event-specific query and sampled frames with role bounding boxes, spanning soccer, basketball, ice hockey, volleyball, esports, wildlife hunting, food packaging, and entertainment.
| Model / interface | Video input | Inference setting |
|---|---|---|
| GPT-5.6 Sol | 2 FPS JPEG frames, , quality 80 | detail=low ; provider generation defaults |
| Gemini-3.7 Flash (1 min) | 2 FPS JPEG frames, , quality 85 | Google GenAI interface; provider generation defaults |
| Qwen / GLM / MiniMax API models | Native MP4 video input | Provider generation defaults |
| Qwen3.5-9B API baseline | Native MP4 video input | Fixed OpenRouter/SiliconFlow route; provider generation defaults |
| PTD (Qwen3-VL-4B) | Uniformly sampled 64 frames | FP16, SDPA, greedy decoding |
Table C.1: Main video input and inference protocols. Provider-native video models receive the original MP4 rather than a client-side reconstruction of the 2 FPS frame sequence.
| Oracle m-vIoU | 1-min m-tIoU | 1-min m-vIoU | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | R0 | R1 | R2 | R0 | R1 | R2 | R0 | R1 | R2 |
| GLM-4.6V | 10.889 | 7.592 | 6.481 | 7.550 | 4.658 | 4.190 | 0.159 | 0.119 | 0.071 |
| MiniMax-M3 | 0.262 | 0.269 | 0.198 | 3.552 | 5.154 | 3.998 | 0.019 | 0.026 | 0.003 |
| Qwen3.7-Plus | 2.518 | 2.654 | 2.235 | 38.095 | 38.346 | 41.125 | 0.383 | 0.418 | 0.616 |
| Qwen3.8-Max | 11.548 | 12.208 | 11.389 | 47.726 | 46.905 | 44.702 | 3.384 | 3.057 | 3.756 |
| PTD-64f | 13.736 | 13.736 | 13.736 | 15.719 | 15.719 | 15.719 | 1.659 | 1.659 | 1.659 |
Table C.2: Repeated evaluation on a fixed 352-query subset. R0 denotes the original run and R1–R2 denote two additional runs under the same inference configuration. All values are percentages and failures remain in the denominator.
| Setting | Condition | N | m-tIoU | m-vIoU |
|---|---|---|---|---|
| Oracle | Qwen3.8, Event-STVG | 100 | – | 9.448 |
| Qwen3.8, PTD-style | 100 | – | 8.532 | |
| PTD | 100 | – | 13.040 | |
| 1 minute | Qwen3.8, Event-STVG | 100 | 53.022 | 4.214 |
| Qwen3.8, PTD-style | 100 | 31.341 | 2.184 | |
| PTD | 100 | 16.165 | 2.104 |
Table C.3: PTD-style interface control on the same 100 target queries.
| Detail | N | m-tIoU | m-vIoU | m-vIoU-Int |
|---|---|---|---|---|
| low | 300 | 45.141 | 2.211 | 3.914 |
| omit | 300 | 45.318 | 2.021 | 3.349 |
Table C.4: Paired comparison of GPT visual detail settings on 300 1-minute queries. Both conditions achieve 300/300 parseable responses.
| Event | Roles | Start | End |
|---|---|---|---|
| Throw-in | throw-in player | When the standard view returns and the thrower and ball are visible | When another player first touches or controls the thrown ball |
| Corner Won | corner winner, corner conceder | When the attacking player creating the corner gains possession | When the defender makes the final touch/block and the ball goes out for a corner |
| Corner Kick | corner taker | When the corner kick begins in the standard broadcast view | When another player first touches the ball, the ball is cleared/goes out, the view changes, or the role becomes invisible |
| Goal Kick | goalkeeper | When the goalkeeper begins the goal kick in the standard view | When another player first touches or controls the ball |
| Offside | offside passer, offsider, last defender | When the passer first gains possession | When the pass to the offside player is completed and the offside decision becomes clear |
| Indirect Free Kick after Offside | indirect free-kick taker | When the restart begins in the standard view | When another player first touches or controls the ball |
Table E.1: Soccer event-specific roles and temporal-boundary guidelines.
| Domain | Event | Semantic roles |
|---|---|---|
| Basketball | Shot | last passer, shooter |
| Ice hockey | Shot | last passer, shooter, goalkeeper |
| Volleyball | Serve | server, receiver |
| Volleyball | Spike | setter, attacker, blocker |
| Esports | Kill | killer, victim |
| Animal activity | Initial Attack | predator, prey |
Table E.2: Event types and semantic roles in the multidomain evaluation.
| Metric | Result |
|---|---|
| Paired temporal annotations | 355 |
| Mean temporal IoU | 0.904 |
| Median temporal IoU | 1.000 |
| Temporal IoU 0.5 | 96.06% |
| Temporal IoU 0.7 | 85.07% |
| Mean start-time difference | 0.194 s |
Table E.3: Soccer paired-annotation agreement.
More in this topic
Video Reasoning
Video reasoning studies how AI systems use video evidence to infer events, actions, temporal relations, causes, and future states.
See all 211 papers