Spatio-Temporal Video Grounding requires jointly localizing target objects across both temporal and spatial dimensions based on natural language queries, posing fundamental challenges for existing Multimodal Large Language Models (MLLMs). We identify two core challenges: entangled spatio-temporal alignment, arising from coupling two heterogeneous sub-tasks within the same autoregressive output space, and dual-domain visual token redundancy, where target objects exhibit simultaneous temporal and spatial sparsity, rendering the overwhelming majority of visual tokens irrelevant to the grounding query. To address these, we propose Bridge-STG, an end-to-end framework that decouples temporal and spatial localization while maintaining semantic coherence. While decoupling is the natural solution to this entanglement, it risks creating a semantic gap between the temporal MLLM and the spatial decoder. Bridge-STG resolves this through two pivotal designs: the Spatio-Temporal Semantic Bridging (STSB) mechanism with Explicit Temporal Alignment (ETA) distills the MLLM's temporal reasoning context into enriched bridging queries as a robust semantic interface; and the Query-Guided Spatial Localization (QGSL) module leverages these queries to drive a purpose-built spatial decoder with multi-layer interactive queries and positive/negative frame sampling, jointly eliminating dual-domain visual token redundancy. Extensive experiments across multiple benchmarks demonstrate that Bridge-STG achieves state-of-the-art performance among MLLM-based methods. Bridge-STG improves average m_vIoU from 26.4 to 34.3 on VidSTG and demonstrates strong cross-task transfer across various fine-grained video understanding tasks under a unified multi-task training regime.
Figures & tables
Figure 1. Comparison of MLLM-based models on the Spatio-Temporal Video Grounding (STVG) task. Current MLLM-based methods struggle with complex scenarios, specifically through inaccurate temporal grounding, confusion with similar distractors, and spatial-semantic misalignment. Content in green and red represents correct and wrong answers, respectively.
Figure 2. Overall architecture of Bridge-STG. The model first predicts the event’s temporal window with the ETA strategy. Triggered by the [DET] token, the Spatio-Temporal Semantic Bridging mechanism distills the MLLM’s reasoning context into bridging queries ( Qbridge ). Finally, the QGSL module utilizes Qbridge to perform precise spatial grounding on the located frames.
Figure 3. System-level comparison of MLLM-based grounding architectures. Left: Baseline MLLMs generate an entangled response of text and spatial coordinates, which decreases grounding precision. Right: Our Bridge-STG explicitly decouples this process. The STSB distills temporal reasoning context to drive a customized spatial decoder, achieving precise localization.
Figure 4. The comparison of visual query selection. Left: Standard methods extract visual features ( Eimgn ) only from the final encoder layer. Right: Our Multi-Layer QGSL aggregates features ( {Eimgl}l=1n ) from all n layers. Conditioned directly by Qbridge , it produces semantically aligned spatial queries ( {Qsell} ) to robustly capture grounded semantics.
Figure 5. The positive/negative frame sampling strategy.
Training Stage
Task
Data Source
# of Samples
Multi-Task Instruction Tuning
Spatial-Temporal Video Grounding
HCSTVG-v1&v2 ( Tang et al., 2021 ) , VidSTG ( Zhang et al., 2020b ) , Self-collected
127K
Video Temporal Grounding
Charades-STA ( Gao et al., 2017 )
12K
Video Object Tracking
GOT-10k ( Huang et al., 2019 )
10K
Video Question Answering
NextQA ( Xiao et al., 2021 ) , Clevrer ( Yi et al., 2019 )
90K
Referring Expression Comprehension
RefCOCO ( Kazemzadeh et al., 2014 ) , RefCOCO+ ( Kazemzadeh et al., 2014 ) , RefCOCOg ( Mao et al., 2016 )
Table 5. Results on GOT-10K ( Huang et al., 2019 ) for VOT task.
Model
RefCOCO
RefCOCO+
RefCOCOg
val
test-A
test-B
val
test-A
test-B
val
test
Non-generative and task-specific models
G-DINO ( Liu et al., 2024 )
90.6
93.2
88.2
88.2
89.0
75.9
86.1
87
GLEE ( Wu et al., 2024 )
91.0
-
-
86.4
-
-
82.6
-
7B-based MLLMs (only-Spatial)
Elysium ( Wang et al., 2024c )
89.1
92.1
85.0
82.9
88.9
75.6
82.9
83.6
Table 6. The [email protected] performance on RefCOCO ( Kazemzadeh et al., 2014 ) , RefCOCO+ ( Kazemzadeh et al., 2014 ) and RefCOCOg ( Mao et al., 2016 ) for REC task.
Model
w/o subs
w/ subs
7B-based MLLMs
Video-LLaVA ( Lin et al., 2024 )
39.9
41.6
Videollama2.1 ( Cheng et al., 2024 )
54.9
56.4
LLaVA-OV ( Li et al., 2024 )
58.2
61.5
SpaceVLLM ( Wang et al., 2025d )
60.0
65.6
Bridge-STG
67.9
74.8
Table 7. Results on VideoMME ( Fu et al., 2025 ) for VQA task.
Event Duration (Avg. Len = Avg. Time of Event Duration)
Short event ( < 1s)
366
0.7s
36.7
29.9
39.8
20.1
Appendix
Table 16. Performance on edge cases from the VidSTG declarative test set. “Overall” refers to the full test set result in main paper.
Figure 6. Visualization of good case on VidSTG interrogative test subset between different model for STVG task. As the box in the video, green represents ground-truth bounding box, blue is the CG-STVG, orange is LLaVA-ST and red is our Bridge-STG.
Figure 7. Visualization of bad case on VidSTG interrogative test subset between different model for STVG task.
Figure 8. Visualization of good case on VidSTG declarative test subset between different model for STVG task.
Figure 9. Visualization of bad case on VidSTG declarative test subset between different model for STVG task.
Spatio-temporal video grounding (STVG) requires models to identify when a referred event occurs and localize the target entity throughout that interval. Existing multimodal large language models typically serialize dense localization trajectories autoregressively, causing decoding latency to grow with tube length and allowing localization errors to propagate across time. We introduce Parallel Tube Decoding (PTD), a generative formulation that decomposes grounding into a temporal block followed by time-conditioned spatial blocks decoded simultaneously. This removes both token-level and trajectory-level dependencies, reducing the sequential decoding depth to a fixed 1+1 rounds, independent of tube length. To enable parallel spatial generation, we introduce Decoupled Block Attention, which preserves access to shared video-query context while eliminating cross-box dependencies, together with localization-aware policy optimization for temporal boundaries and spatial geometry. On VidSTG, PTD reduces Tube Completion Latency by 79x and increases spatial decoding throughput by 92x over standard autoregressive decoding, while also improving grounding accuracy. With a compact 4B backbone, our model performs favorably well on VidSTG and HC-STVG, and generalizes zero-shot to temporal grounding, grounded VideoQA, and referring video object tracking. Our results show parallel tube generation is an efficient and effective alternative to autoregressive localization in videos.
Hanoona Rasheed, Haania Siddiqui, Ming-Hsuan Yang +2
Mohamed bin Zayed University of Artificial Intelligence · University of California, Merced · Apertix
Video temporal grounding (VTG) is a critical task in video understanding and a key capability for extending video large language models (Vid-LLMs) to broader applications. However, existing Vid-LLMs rely on uniform frame sampling to extract video information, resulting in a sparse distribution of key frames and the loss of crucial temporal cues. To address this limitation, we propose Grounded Visual Token Sampling (GroundVTS), a Vid-LLM architecture that focuses on the most informative temporal segments. GroundVTS employs a fine-grained, query-guided mechanism to filter visual tokens before feeding them into the LLM, thereby preserving essential spatio-temporal information and maintaining temporal coherence. Futhermore, we introduce a progressive optimization strategy that enables the LLM to effectively adapt to the non-uniform distribution of visual features, enhancing its ability to model temporal dependencies and achieve precise video localization. We comprehensively evaluate GroundVTS on three standard VTG benchmarks, where it outperforms existing methods, achieving a 7.7-point improvement in mIoU for moment retrieval and 12.0-point improvement in mAP for highlight detection. Code is available at https://github.com/Florence365/GroundVTS.
Rong Fan, Kaiyan Xiao, Minghao Zhu +3
Newcapec AI Research · Fudan University · Tongji University
Video Temporal Grounding (VTG) faces a cross-modal semantic gap that often leads to background features being incorrectly aligned with the query, while directly matching the query to moments results in insufficient discriminability and consistency of temporal semantics. To address this issue, we propose MLLM-Assisted Semantic-Relational Consistent Alignment (MASRA), a training-time MLLM-based optimization framework for VTG. MASRA leverages an MLLM during training to produce two forms of textual priors, namely event-level descriptions with temporal spans and clip-level captions, and instantiates two MLLM-assisted alignments. Event Semantic Temporal Alignment (ESTA) aligns temporal context with event semantics to explicitly strengthen the correspondence between semantics and temporal events and improve span-level separability. Local Relational Consistency Alignment (LRCA) constructs a textual relation matrix derived from clip-level captions and aligns it with the temporal feature similarity matrix in the model, enhancing temporal consistency while capturing local structural information. MASRA includes two simple supporting modules, semantic-guided enhancement and second-order relational attention, to better utilize the learned semantic context and relational structure. Moreover, we introduce Decoupled Alignment Interaction (DAI) with a context-aware codebook to adaptively absorb query-irrelevant semantics and alleviate the cross-modal gap. The MLLM is only invoked during training and is not used at inference. Extensive experiments show that MASRA outperforms existing methods, and ablation studies validate its effectiveness.
Ran Ran, Jiwei Wei, Shuchang Zhou +5
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · School of Cyberspace Security, Hainan University, Haikou, China