ID-VTG: Image-Disambiguated Video Temporal Grounding
Authors: Minghang Zheng, Jingli Wei, Hongyi Yang, Yang Liu
Organizations: Wangxuan Institute of Computer Technology, Peking University Beijing, China · Wangxuan Institute of Computer Technology, Peking University State Key Laboratory of General Artificial Intelligence, Peking University Beijing, China
Video Temporal Grounding (VTG) faces significant challenges when natural language queries must distinguish between multiple events involving visually similar entities, particularly when relying on fine-grained visual attributes that are difficult to describe accurately in words alone. To address this, we introduce Image-Disambiguated Video Temporal Grounding (ID-VTG), a task that leverages multimodal queries combining a reference image and a text description to precisely localize segments where a specific instance performs a described action. To facilitate research, we construct two benchmarks: IDVTG-Gym, focusing on fine-grained, compositionally ordered gymnastics actions with athletes in similar uniforms; and IDVTG-InternVid, an open-world dataset featuring diverse entities (e.g., humans, animals, fictional characters) and significant temporal distractors. Methodologically, we propose the Visually-Guided Disambiguation Aggregation (VGD-Agg) framework based on a dual-branch fast-slow architecture. The fast branch efficiently generates preliminary event proposals, while the slow branch performs fine-grained frame-level matching between video frames and the reference image. We enhance discriminability via two learnable tokens: a Compare Token, which represents hard negatives to probe for the presence of the target instance (as referred to by the query image), and a Depress Value, which represents text-irrelevant events. Proposals that the Compare Token identifies as lacking the target instance are pushed toward the Depress Value, thus easing disambiguation via the text query. Extensive experiments validate our approach, which achieves state-of-the-art results on the proposed benchmarks. Code is available at https://github.com/oceanflowlab/ID-VTG.
Figures & tables
Figure 1. Illustration of the proposed Image-Disambiguated Video Temporal Grounding (ID-VTG) task. Illustration of the proposed Image-Disambiguated Video Temporal Grounding (ID-VTG) task
Figure 2. Overview of the ID-VTG datasets. (a) IDVTG-Gym focuses on fine-grained, ordered gymnastics actions with visually similar athletes. (b) IDVTG-InternVid focuses on open-world videos with diverse entities and strong temporal distractors. (c) The semantic category distribution of IDVTG-InternVid. (d) The lexical distribution of IDVTG-Gym and IDVTG-InternVid. Overview of the ID-VTG datasets
Figure 3. Overview of the ID-VTG dataset construction pipeline. Stage 1 generates text descriptions and timestamps of ambiguous events for IDVTG-Gym and IDVTG-InternVid. Stage 2 selects disambiguating image queries by grounding target instances and verifying candidates with an MLLM. ID-VTG dataset construction pipeline
Dataset
Domain
Duration
Queries
Query Type
Ambiguity
TACoS ( Regneri et al., 2013 )
Cooking
10.1 h
18.2K
Text
No
Charades-STA ( Gao et al., 2017 )
Activity
57.1 h
16.1K
Text
No
DiDeMo ( Hendricks et al., 2017 )
Flickr
88.7 h
41.2K
Text
No
QVHighlights ( Lei et al., 2021 )
Vlog / News
425 h
10.3K
Text
No
ANet-Captions ( Krishna et al., 2017 )
Activity
487.6 h
72.0K
Text
No
ICQ-Highlight ( Zhang et al., 2025 )
Vlog / News
65 h
6.19K
Text + Image
No
Table 1. Comparison of our ID-VTG dataset with existing VTG datasets.
Figure 4. Overview of the Visually-Guided Disambiguation Aggregation (VGD-Agg) framework. The architecture consists of a dual-branch design: (1) The Fast Branch (left) efficiently generates generic, image-agnostic proposal features from the video stream. (2) The Slow Branch (middle) performs fine-grained matching between video frames and the image query to compute similarity scores and learn a video-specific Compare Token. (3) The Vision-Assisted Disambiguation module (top) leverages the Compare Token as a dynamic decision boundary. It acts as a soft gate that suppresses proposal features dominated by visual distractors ( Image-Irrelevant ) while enhancing those containing the target instance ( Image-Relevant ). (4) Finally, the Text-Guided Grounding head (right) aligns the disambiguated features with the text to regress the precise temporal boundaries. Overview of the Visually-Guided Disambiguation Aggregation (VGD-Agg) framework.
Method
Overall
Sub-act.
Holistic
R0.51
R0.71
mIoU
R0.51
R0.71
mIoU
R0.51
R0.71
mIoU
RaTSG ( Dong et al., 2024 )
18.45
15.99
16.96
6.64
3.98
5.54
26.28
23.50
27.64
UVCOM ( Xiao et al., 2023 )
18.07
15.00
17.32
5.20
2.43
6.86
25.07
22.54
22.99
CG-DETR ( Moon et al., 2024 )
27.96
23.36
26.99
6.64
2.77
10.14
44.47
39.86
40.29
ICQ ( Zhang et al., 2025 )
46.12
42.24
40.24
43.92
35.73
34.10
47.87
47.17
45.06
SnAG ( Mu et al., 2024 )
53.64
48.82
46.73
51.66
41.37
40.32
54.83
54.05
51.46
Table 2. Performance comparison on the IDVTG-Gym dataset.
Method
IDVTG-InternVid
Web
R0.51
R0.71
mIoU
R0.51
R0.71
mIoU
RaTSG ( Dong et al., 2024 )
30.58
19.71
30.91
7.96
5.79
8.53
UVCOM ( Xiao et al., 2023 )
30.11
18.89
30.07
7.70
5.62
8.10
CG-DETR ( Moon et al., 2024 )
44.81
31.62
43.80
13.86
5.91
18604
ICQ ( Zhang et al., 2025 )
31.04
25.39
29.90
7.84
4.53
8.31
SnAG ( Mu et al., 2024 )
44.26
35.46
41.70
16.48
9.06
16.53
Table 3. Performance comparison on the IDVTG-InternVid and Web datasets.
Model Variant
R0.51
R0.71
mIoU
w/o Slow Branch
53.64
48.82
46.73
w/o Fast Branch
59.41
54.40
51.32
w/o Aggregation
61.12
56.76
53.50
VGD-Agg (Full)
61.83
56.53
54.17
Table 4. Ablation on branch-level architecture.
Model Variant
R0.51
R0.71
mIoU
w/o Compare Token
55.63
50.76
48.49
Global Learnable Token
60.36
55.53
52.62
w/o Similarity Loss
60.50
55.58
52.67
w/o Image in Compare Token Generation
60.64
56.20
53.26
w/o Text in Depress Value Generation
59.34
54.26
51.94
VGD-Agg (Full)
61.83
56.53
54.17
Table 5. Ablation on compare token and depress value.
Input Modality
R0.51
R0.71
mIoU
Text-only
46.78
41.96
40.74
Attention Fusion
53.64
48.82
46.73
BLIP-2 Fusion ( Li et al., 2023 )
54.07
48.86
46.75
ICQ Fusion ( Zhang et al., 2025 )
46.12
42.24
40.24
Text + Image + VGD-Agg (Ours)
61.83
56.53
54.17
Table 6. Ablation on different input fusion methods.
Method
Full Amb.
Text Amb. Only
Img Amb. Only
R0.51
R0.71
mIoU
R0.51
R0.71
mIoU
R0.51
R0.71
mIoU
RaTSG ( Dong et al., 2024 )
19.61
14.96
20.87
23.47
20.88
24.15
26.95
21.18
27.03
UVCOM ( Xiao et al., 2023 )
19.49
14.79
20.55
23.13
19.97
23.67
26.73
20.89
26.01
CG-DETR ( Moon et al., 2024 )
30.29
19.61
31.06
37.88
26.67
38.60
37.26
23.83
37.88
ICQ ( Zhang et al., 2025 )
20.54
15.77
20.50
25.70
21.45
25.15
27.48
21.46
27.29
SnAG ( Mu et al., 2024 )
30.39
23.34
30.34
41.94
34.24
39.59
38.19
27.53
36.40
Table 7. Ablation study on different ambiguity types on IDVTG-InternVid.
Method
Clean
Brightness
Low Resolution
R0.51
R0.71
mIoU
R0.51
R0.71
mIoU
R0.51
R0.71
mIoU
RaTSG ( Dong et al., 2024 )
30.58
19.71
30.91
29.04
18.87
28.99
29.98
18.35
29.31
UVCOM ( Xiao et al., 2023 )
30.11
18.89
30.07
28.76
18.16
28.90
29.83
18.51
29.84
CG-DETR ( Moon et al., 2024 )
44.81
31.62
43.80
44.35
31.60
43.36
44.31
31.54
43.27
ICQ ( Zhang et al., 2025 )
31.04
25.39
29.90
29.00
23.29
28.02
30.77
24.66
29.54
SnAG ( Mu et al., 2024 )
44.26
35.46
41.70
43.57
35.05
41.18
43.48
35.13
41.15
Table 8. Ablations on the robustness to image queries on IDVTG-InternVid.
Figure 5. Visualization of the learned attention weights. qualitative results
Video Temporal Grounding (VTG) localizes the video segment that matches a natural-language query. Many queries describe an action performed by a particular entity. Existing methods often encode the query as a whole or use general video-text interactions, without explicitly checking whether the action and entity occur together. They may therefore select a segment that contains both concepts but not the event described by the query. We propose Interaction Aligned Action-Entity Video Temporal Grounding (IAE-VTG), which models this rela?tionship at both the representation and training assignment levels. First, the Fine-grained Disentangled Interaction Module (FDIM) separates action and entity related query information and aligns it with complementary motion and appearance features. It then combines token-level interactions to build representations that capture the relationship between the action and entity. Second, Interaction-Sensitive Assignment (ISA) adds this interaction evidence to bipartite matching, so training targets are selected using both temporal overlap and semantic compatibility. This reduces supervision from temporally plausible but semantically incorrect proposals. Experiments on QVHighlights, Charades?STA, and TACoS show that IAE-VTG consistently improves strong baselines and achieves competitive or state-of-the-art performance on standard grounding metrics. Additional analyses show that the method is especially effective when similar actions or entities appear at multiple times and produces more reliable assignments for complex events.
Shiwen Zhao, Qi Zhang, Sezer Karaoglu +2
School of Computer Science, The University of Sydney, Sydney, NSW 2006, Australia · Informatics Institute at University of Amsterdam, 1098 XH Amsterdam, the Netherlands
Text-guided Video Temporal Grounding (VTG) aims to localize the relevant segments in an untrimmed video based on text queries, yet collecting dense temporal annotations and training task-specific models remain costly and brittle under distribution shift. Recent training-free VTG approaches mitigate this issue by directly matching pretrained vision-language representations, but they still face two fundamental information bottlenecks: frame-wise visual encoding overlooks temporal dynamics, while fixed query embeddings cannot resolve query ambiguity. To address these issues, we propose DSE-VTG, a Dual-Side Enhancement framework that addresses both without any task-specific training. On the visual side, Multi-scale Similarity Fusion (MSF) combines frame- and clip-level similarities into a unified, temporally aware similarity profile. On the textual side, Query-level Test-Time Adaptation (Q-TTA) optimizes a lightweight additive offset to adapt the query embedding to the video at test time, without finetuning the backbone or calling external large language models. Extensive experiments on three standard and two OOD benchmarks show that DSE-VTG achieves state-of-the-art performance among training-free methods. On Charades-STA, it improves mIoU over the strongest prior training-free method by 5.61 points. Under distribution shift, DSE-VTG reaches 50.86 mIoU on Charades-CG Novel-Word, surpassing the strongest supervised baseline by 2.76 mIoU. Our code will be released upon acceptance.
Video temporal grounding (VTG) is a critical task in video understanding and a key capability for extending video large language models (Vid-LLMs) to broader applications. However, existing Vid-LLMs rely on uniform frame sampling to extract video information, resulting in a sparse distribution of key frames and the loss of crucial temporal cues. To address this limitation, we propose Grounded Visual Token Sampling (GroundVTS), a Vid-LLM architecture that focuses on the most informative temporal segments. GroundVTS employs a fine-grained, query-guided mechanism to filter visual tokens before feeding them into the LLM, thereby preserving essential spatio-temporal information and maintaining temporal coherence. Futhermore, we introduce a progressive optimization strategy that enables the LLM to effectively adapt to the non-uniform distribution of visual features, enhancing its ability to model temporal dependencies and achieve precise video localization. We comprehensively evaluate GroundVTS on three standard VTG benchmarks, where it outperforms existing methods, achieving a 7.7-point improvement in mIoU for moment retrieval and 12.0-point improvement in mAP for highlight detection. Code is available at https://github.com/Florence365/GroundVTS.
Rong Fan, Kaiyan Xiao, Minghao Zhu +3
Newcapec AI Research · Fudan University · Tongji University