Spatio-Temporal Video Grounding requires jointly localizing target objects across both temporal and spatial dimensions based on natural language queries, posing fundamental challenges for existing Multimodal Large Language Models (MLLMs). We identify two core challenges: entangled spatio-temporal alignment, arising from coupling two heterogeneous sub-tasks within the same autoregressive output space, and dual-domain visual token redundancy, where target objects exhibit simultaneous temporal and spatial sparsity, rendering the overwhelming majority of visual tokens irrelevant to the grounding query. To address these, we propose Bridge-STG, an end-to-end framework that decouples temporal and spatial localization while maintaining semantic coherence. While decoupling is the natural solution to this entanglement, it risks creating a semantic gap between the temporal MLLM and the spatial decoder. Bridge-STG resolves this through two pivotal designs: the Spatio-Temporal Semantic Bridging (STSB) mechanism with Explicit Temporal Alignment (ETA) distills the MLLM's temporal reasoning context into enriched bridging queries as a robust semantic interface; and the Query-Guided Spatial Localization (QGSL) module leverages these queries to drive a purpose-built spatial decoder with multi-layer interactive queries and positive/negative frame sampling, jointly eliminating dual-domain visual token redundancy. Extensive experiments across multiple benchmarks demonstrate that Bridge-STG achieves state-of-the-art performance among MLLM-based methods. Bridge-STG improves average m_vIoU from 26.4 to 34.3 on VidSTG and demonstrates strong cross-task transfer across various fine-grained video understanding tasks under a unified multi-task training regime.
Figures & tables
Figure 1. Comparison of MLLM-based models on the Spatio-Temporal Video Grounding (STVG) task. Current MLLM-based methods struggle with complex scenarios, specifically through inaccurate temporal grounding, confusion with similar distractors, and spatial-semantic misalignment. Content in green and red represents correct and wrong answers, respectively.
Figure 2. Overall architecture of Bridge-STG. The model first predicts the event’s temporal window with the ETA strategy. Triggered by the [DET] token, the Spatio-Temporal Semantic Bridging mechanism distills the MLLM’s reasoning context into bridging queries ( Qbridge ). Finally, the QGSL module utilizes Qbridge to perform precise spatial grounding on the located frames.
Figure 3. System-level comparison of MLLM-based grounding architectures. Left: Baseline MLLMs generate an entangled response of text and spatial coordinates, which decreases grounding precision. Right: Our Bridge-STG explicitly decouples this process. The STSB distills temporal reasoning context to drive a customized spatial decoder, achieving precise localization.
Figure 4. The comparison of visual query selection. Left: Standard methods extract visual features ( Eimgn ) only from the final encoder layer. Right: Our Multi-Layer QGSL aggregates features ( {Eimgl}l=1n ) from all n layers. Conditioned directly by Qbridge , it produces semantically aligned spatial queries ( {Qsell} ) to robustly capture grounded semantics.
Figure 5. The positive/negative frame sampling strategy.
Training Stage
Task
Data Source
# of Samples
Multi-Task Instruction Tuning
Spatial-Temporal Video Grounding
HCSTVG-v1&v2 ( Tang et al., 2021 ) , VidSTG ( Zhang et al., 2020b ) , Self-collected
127K
Video Temporal Grounding
Charades-STA ( Gao et al., 2017 )
12K
Video Object Tracking
GOT-10k ( Huang et al., 2019 )
10K
Video Question Answering
NextQA ( Xiao et al., 2021 ) , Clevrer ( Yi et al., 2019 )
90K
Referring Expression Comprehension
RefCOCO ( Kazemzadeh et al., 2014 ) , RefCOCO+ ( Kazemzadeh et al., 2014 ) , RefCOCOg ( Mao et al., 2016 )
Table 5. Results on GOT-10K ( Huang et al., 2019 ) for VOT task.
Model
RefCOCO
RefCOCO+
RefCOCOg
val
test-A
test-B
val
test-A
test-B
val
test
Non-generative and task-specific models
G-DINO ( Liu et al., 2024 )
90.6
93.2
88.2
88.2
89.0
75.9
86.1
87
GLEE ( Wu et al., 2024 )
91.0
-
-
86.4
-
-
82.6
-
7B-based MLLMs (only-Spatial)
Elysium ( Wang et al., 2024c )
89.1
92.1
85.0
82.9
88.9
75.6
82.9
83.6
Table 6. The [email protected] performance on RefCOCO ( Kazemzadeh et al., 2014 ) , RefCOCO+ ( Kazemzadeh et al., 2014 ) and RefCOCOg ( Mao et al., 2016 ) for REC task.
Model
w/o subs
w/ subs
7B-based MLLMs
Video-LLaVA ( Lin et al., 2024 )
39.9
41.6
Videollama2.1 ( Cheng et al., 2024 )
54.9
56.4
LLaVA-OV ( Li et al., 2024 )
58.2
61.5
SpaceVLLM ( Wang et al., 2025d )
60.0
65.6
Bridge-STG
67.9
74.8
Table 7. Results on VideoMME ( Fu et al., 2025 ) for VQA task.
Event Duration (Avg. Len = Avg. Time of Event Duration)
Short event ( < 1s)
366
0.7s
36.7
29.9
39.8
20.1
Appendix
Table 16. Performance on edge cases from the VidSTG declarative test set. “Overall” refers to the full test set result in main paper.
Figure 6. Visualization of good case on VidSTG interrogative test subset between different model for STVG task. As the box in the video, green represents ground-truth bounding box, blue is the CG-STVG, orange is LLaVA-ST and red is our Bridge-STG.
Figure 7. Visualization of bad case on VidSTG interrogative test subset between different model for STVG task.
Figure 8. Visualization of good case on VidSTG declarative test subset between different model for STVG task.
Figure 9. Visualization of bad case on VidSTG declarative test subset between different model for STVG task.
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · School of Cyberspace Security, Hainan University, Haikou, China