ID-VTG: Image-Disambiguated Video Temporal Grounding

Minghang Zheng; Jingli Wei; Hongyi Yang; Yang Liu

cs.CVAug 20, 2026

ID-VTG: Image-Disambiguated Video Temporal Grounding

Authors: Minghang Zheng, Jingli Wei, Hongyi Yang, Yang Liu

Organizations: Wangxuan Institute of Computer Technology, Peking University Beijing, China · Wangxuan Institute of Computer Technology, Peking University State Key Laboratory of General Artificial Intelligence, Peking University Beijing, China

Abstract

Video Temporal Grounding (VTG) faces significant challenges when natural language queries must distinguish between multiple events involving visually similar entities, particularly when relying on fine-grained visual attributes that are difficult to describe accurately in words alone. To address this, we introduce Image-Disambiguated Video Temporal Grounding (ID-VTG), a task that leverages multimodal queries combining a reference image and a text description to precisely localize segments where a specific instance performs a described action. To facilitate research, we construct two benchmarks: IDVTG-Gym, focusing on fine-grained, compositionally ordered gymnastics actions with athletes in similar uniforms; and IDVTG-InternVid, an open-world dataset featuring diverse entities (e.g., humans, animals, fictional characters) and significant temporal distractors. Methodologically, we propose the Visually-Guided Disambiguation Aggregation (VGD-Agg) framework based on a dual-branch fast-slow architecture. The fast branch efficiently generates preliminary event proposals, while the slow branch performs fine-grained frame-level matching between video frames and the reference image. We enhance discriminability via two learnable tokens: a Compare Token, which represents hard negatives to probe for the presence of the target instance (as referred to by the query image), and a Depress Value, which represents text-irrelevant events. Proposals that the Compare Token identifies as lacking the target instance are pushed toward the Depress Value, thus easing disambiguation via the text query. Extensive experiments validate our approach, which achieves state-of-the-art results on the proposed benchmarks. Code is available at https://github.com/oceanflowlab/ID-VTG.

Figures & tables

Figure 1. Illustration of the proposed Image-Disambiguated Video Temporal Grounding (ID-VTG) task. Illustration of the proposed Image-Disambiguated Video Temporal Grounding (ID-VTG) task

Figure 2. Overview of the ID-VTG datasets. (a) IDVTG-Gym focuses on fine-grained, ordered gymnastics actions with visually similar athletes. (b) IDVTG-InternVid focuses on open-world videos with diverse entities and strong temporal distractors. (c) The semantic category distribution of IDVTG-InternVid. (d) The lexical distribution of IDVTG-Gym and IDVTG-InternVid. Overview of the ID-VTG datasets

Figure 3. Overview of the ID-VTG dataset construction pipeline. Stage 1 generates text descriptions and timestamps of ambiguous events for IDVTG-Gym and IDVTG-InternVid. Stage 2 selects disambiguating image queries by grounding target instances and verifying candidates with an MLLM. ID-VTG dataset construction pipeline

Dataset	Domain	Duration	Queries	Query Type	Ambiguity
TACoS ( Regneri et al., 2013 )	Cooking	10.1 h	18.2K	Text	No
Charades-STA ( Gao et al., 2017 )	Activity	57.1 h	16.1K	Text	No
DiDeMo ( Hendricks et al., 2017 )	Flickr	88.7 h	41.2K	Text	No
QVHighlights ( Lei et al., 2021 )	Vlog / News	425 h	10.3K	Text	No
ANet-Captions ( Krishna et al., 2017 )	Activity	487.6 h	72.0K	Text	No
ICQ-Highlight ( Zhang et al., 2025 )	Vlog / News	65 h	6.19K	Text + Image	No

Table 1. Comparison of our ID-VTG dataset with existing VTG datasets.

Figure 4. Overview of the Visually-Guided Disambiguation Aggregation (VGD-Agg) framework. The architecture consists of a dual-branch design: (1) The Fast Branch (left) efficiently generates generic, image-agnostic proposal features from the video stream. (2) The Slow Branch (middle) performs fine-grained matching between video frames and the image query to compute similarity scores and learn a video-specific Compare Token. (3) The Vision-Assisted Disambiguation module (top) leverages the Compare Token as a dynamic decision boundary. It acts as a soft gate that suppresses proposal features dominated by visual distractors ( Image-Irrelevant ) while enhancing those containing the target instance ( Image-Relevant ). (4) Finally, the Text-Guided Grounding head (right) aligns the disambiguated features with the text to regress the precise temporal boundaries. Overview of the Visually-Guided Disambiguation Aggregation (VGD-Agg) framework.

Method	Overall			Sub-act.			Holistic
Method	$R_{0.5}^{1}$	$R_{0.7}^{1}$	mIoU	$R_{0.5}^{1}$	$R_{0.7}^{1}$	mIoU	$R_{0.5}^{1}$	$R_{0.7}^{1}$	mIoU
RaTSG ( Dong et al., 2024 )	18.45	15.99	16.96	6.64	3.98	5.54	26.28	23.50	27.64
UVCOM ( Xiao et al., 2023 )	18.07	15.00	17.32	5.20	2.43	6.86	25.07	22.54	22.99
CG-DETR ( Moon et al., 2024 )	27.96	23.36	26.99	6.64	2.77	10.14	44.47	39.86	40.29
ICQ ( Zhang et al., 2025 )	46.12	42.24	40.24	43.92	35.73	34.10	47.87	47.17	45.06
SnAG ( Mu et al., 2024 )	53.64	48.82	46.73	51.66	41.37	40.32	54.83	54.05	51.46

Table 2. Performance comparison on the IDVTG-Gym dataset.

Method	IDVTG-InternVid			Web
Method	$R_{0.5}^{1}$	$R_{0.7}^{1}$	mIoU	$R_{0.5}^{1}$	$R_{0.7}^{1}$	mIoU
RaTSG ( Dong et al., 2024 )	30.58	19.71	30.91	7.96	5.79	8.53
UVCOM ( Xiao et al., 2023 )	30.11	18.89	30.07	7.70	5.62	8.10
CG-DETR ( Moon et al., 2024 )	44.81	31.62	43.80	13.86	5.91	18604
ICQ ( Zhang et al., 2025 )	31.04	25.39	29.90	7.84	4.53	8.31
SnAG ( Mu et al., 2024 )	44.26	35.46	41.70	16.48	9.06	16.53

Table 3. Performance comparison on the IDVTG-InternVid and Web datasets.

Model Variant	$R_{0.5}^{1}$	$R_{0.7}^{1}$	mIoU
w/o Slow Branch	53.64	48.82	46.73
w/o Fast Branch	59.41	54.40	51.32
w/o Aggregation	61.12	56.76	53.50
VGD-Agg (Full)	61.83	56.53	54.17

Table 4. Ablation on branch-level architecture.

Model Variant	$R_{0.5}^{1}$	$R_{0.7}^{1}$	mIoU
w/o Compare Token	55.63	50.76	48.49
Global Learnable Token	60.36	55.53	52.62
w/o Similarity Loss	60.50	55.58	52.67
w/o Image in Compare Token Generation	60.64	56.20	53.26
w/o Text in Depress Value Generation	59.34	54.26	51.94
VGD-Agg (Full)	61.83	56.53	54.17

Table 5. Ablation on compare token and depress value.

Input Modality	$R_{0.5}^{1}$	$R_{0.7}^{1}$	mIoU
Text-only	46.78	41.96	40.74
Attention Fusion	53.64	48.82	46.73
BLIP-2 Fusion ( Li et al., 2023 )	54.07	48.86	46.75
ICQ Fusion ( Zhang et al., 2025 )	46.12	42.24	40.24
Text + Image + VGD-Agg (Ours)	61.83	56.53	54.17

Table 6. Ablation on different input fusion methods.

Method	Full Amb.			Text Amb. Only			Img Amb. Only
	$R_{0.5}^{1}$	$R_{0.7}^{1}$	mIoU	$R_{0.5}^{1}$	$R_{0.7}^{1}$	mIoU	$R_{0.5}^{1}$	$R_{0.7}^{1}$	mIoU
RaTSG ( Dong et al., 2024 )	19.61	14.96	20.87	23.47	20.88	24.15	26.95	21.18	27.03
UVCOM ( Xiao et al., 2023 )	19.49	14.79	20.55	23.13	19.97	23.67	26.73	20.89	26.01
CG-DETR ( Moon et al., 2024 )	30.29	19.61	31.06	37.88	26.67	38.60	37.26	23.83	37.88
ICQ ( Zhang et al., 2025 )	20.54	15.77	20.50	25.70	21.45	25.15	27.48	21.46	27.29
SnAG ( Mu et al., 2024 )	30.39	23.34	30.34	41.94	34.24	39.59	38.19	27.53	36.40

Table 7. Ablation study on different ambiguity types on IDVTG-InternVid.

Method	Clean			Brightness			Low Resolution
Method	$R_{0.5}^{1}$	$R_{0.7}^{1}$	mIoU	$R_{0.5}^{1}$	$R_{0.7}^{1}$	mIoU	$R_{0.5}^{1}$	$R_{0.7}^{1}$	mIoU
RaTSG ( Dong et al., 2024 )	30.58	19.71	30.91	29.04	18.87	28.99	29.98	18.35	29.31
UVCOM ( Xiao et al., 2023 )	30.11	18.89	30.07	28.76	18.16	28.90	29.83	18.51	29.84
CG-DETR ( Moon et al., 2024 )	44.81	31.62	43.80	44.35	31.60	43.36	44.31	31.54	43.27
ICQ ( Zhang et al., 2025 )	31.04	25.39	29.90	29.00	23.29	28.02	30.77	24.66	29.54
SnAG ( Mu et al., 2024 )	44.26	35.46	41.70	43.57	35.05	41.18	43.48	35.13	41.15

Table 8. Ablations on the robustness to image queries on IDVTG-InternVid.

Figure 5. Visualization of the learned attention weights. qualitative results

Temporal Video Grounding

Temporal video grounding locates the moments in a video that match a natural-language query, enabling precise search and evidence-based understanding.

See all 166 papers

IAE-VTG: Interaction-Aligned Action-Entity Video Temporal Grounding
Sep 9, 2026Shiwen Zhao, Qi Zhang, Sezer Karaoglu +2Temporal Video Grounding Vision-Language Alignment
School of Computer Science, The University of Sydney, Sydney, NSW 2006, Australia · Informatics Institute at University of Amsterdam, 1098 XH Amsterdam, the Netherlands
DSE-VTG: Dual-Side Enhancement for Training-Free Video Temporal Grounding
Sep 8, 2026Zhuo Cao, Bingqing Zhang, Sen Wang +1Temporal Video Grounding Test-Time Adaptation of VLMs
The University of Queensland, Australia
GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding
Apr 2, 2026Rong Fan, Kaiyan Xiao, Minghao Zhu +3Temporal Video Grounding Video-Language Models
Newcapec AI Research · Fudan University · Tongji University

Sep 9, 2026J/K move · Enter open · S save

IAE-VTG: Interaction-Aligned Action-Entity Video Temporal Grounding

Shiwen Zhao, Qi Zhang, Sezer Karaoglu, Theo Gevers+1

School of Computer Science, The University of Sydney, Sydney, NSW 2006, Australia · Informatics Institute at University of Amsterdam, 1098 XH Amsterdam, the Netherlands

Temporal Video Grounding Vision-Language Alignment

Open paper AskPDF arXiv

Video Temporal Grounding (VTG) localizes the video segment that matches a natural-language query. Many queries describe an action performed by a particular entity. Existing methods often encode the query as a whole or use general video-text interactions, without explicitly checking whether the action and entity occur together. They may therefore select a segment that contains both concepts but not the event described by the query. We propose Interaction Aligned Action-Entity Video Temporal Grounding (IAE-VTG), which models this rela?tionship at both the representation and training assignment levels. First, the Fine-grained Disentangled Interaction Module (FDIM) separates action and entity related query information and aligns it with complementary motion and appearance features. It then combines token-level interactions to build representations that capture the relationship between the action and entity. Second, Interaction-Sensitive Assignment (ISA) adds this interaction evidence to bipartite matching, so training targets are selected using both temporal overlap and semantic compatibility. This reduces supervision from temporally plausible but semantically incorrect proposals. Experiments on QVHighlights, Charades?STA, and TACoS show that IAE-VTG consistently improves strong baselines and achieves competitive or state-of-the-art performance on standard grounding metrics. Additional analyses show that the method is especially effective when similar actions or entities appear at multiple times and produces more reliable assignments for complex events.

Abstract

Temporal Video Grounding

Explore similar work

IAE-VTG: Interaction-Aligned Action-Entity Video Temporal Grounding

DSE-VTG: Dual-Side Enhancement for Training-Free Video Temporal Grounding

GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding