Where, How, and When: Query-Guided Grounding for Video Understanding

Yogesh Kumar

cs.CVOct 15, 2026

Where, How, and When: Query-Guided Grounding for Video Understanding

Authors: Yogesh Kumar

Organizations: Indian Institute of Technology Jodhpur Department of Computer Science and Engineering

Abstract

Understanding video content requires reasoning at multiple levels of granularity, from identifying objects to interpreting their interactions and localizing when specific actions occur. This thesis studies three query-guided grounding tasks that answer where key objects are (object-level grounding), how they interact (relationship-level grounding), and when specific moments occur (moment-level grounding). For object-level grounding, we present QDETRv, a query-guided extension of the Detection Transformer for one-shot localization of unseen objects in videos, which combines query-context fusion, a memory module for temporal consistency, and unsupervised pretraining on synthetic object trajectories. We further propose the Temporal Object-Aware Vision Transformer (TOViT) for few-shot video object detection, which builds on a language-aligned open-set vision encoder and propagates high-confidence object features across frames. For relationship-level grounding, we introduce few-shot referring relationships in videos: given a (subject, predicate, object) query and a few support videos sharing the predicate, the entities are localized by optimizing a T-partite random field using a meta-learned relation network with query-conditioned translational embeddings, Global Semantic Aggregation, Local Localization Aggregation, and belief-propagation-based message passing. For moment-level grounding, we propose the Moment Alignment Transformer (MATR) for video-to-video moment retrieval, which uses dual-stage Dynamic Time Warping for pre- and post-fusion alignment, learnable queries for boundary prediction, and self-supervised pretraining with temporal augmentations; MATR also extends to text-to-video moment retrieval with performance comparable to, or better than, the state of the art. These contributions advance query-guided video understanding across spatial, relational, and temporal grounding.

Figures & tables

Figure 1.1 : Thesis Objectives . This work aims to advance query-guided video understanding by addressing three core questions: where objects are present, how they interact, and when specific actions occur. Consider a football match: understanding the play requires (i) identifying key entities such as the ball, goalposts, and the players (where important objects are), (ii) recognizing interactions like “player kicking the ball to a teammate" (how objects interact), and (iii) localizing key events such “goa" moment (when critical actions occur). Together, these tasks enable structured interpretation of complex, fast-paced sports scenes, moving beyond frame-level recognition toward temporally grounded, interaction-aware understanding.

Figure 1.2 : Given an unseen query image and a target video, the model is expected to localize all instances of objects present in the query image on the target video.

Figure 1.3 : Given a query visual relationship as

<

subject, predicate, object

>

and a test video, our goal is to localize the subject and object on the test video with the help of a support set containing a few videos with the same predicate. In this example, the goal is to spatio-temporally localize the plane (subject) and person (object) that are connected via fly above (predicate), with the help of a support set containing only four videos sharing the predicate fly above . It should be noted here that fly above is unseen during training. We refer to this problem as few-shot referring relationship in videos .

Figure 1.4 : Given a query video depicting an unseen moment or action, the task is to temporally localize the corresponding moment or action within a target video.

Model	Modality / Task	Key Contributions and Innovations
Transformer [ 28 ]	Text (Seq2Seq)	Introduced self-attention and encoder-decoder architecture for sequence modeling.
BERT [ 35 ] , GPT [ 36 ]	Text (Pretraining)	Masked LM (BERT) and autoregressive LM (GPT); fine-tuning for downstream tasks.
ViT [ 29 ]	Image (Classification)	Treats image as patch sequence; shows Transformers rival CNNs with large-scale pretraining.
DeiT [ 40 ]	Image (Classification)	Data-efficient ViT training via distillation and augmentation.
Swin Transformer [ 41 ]	Image (Dense Tasks)	Hierarchical features; shifted window attention for scalable vision tasks.
DETR [ 10 ]	Image (Detection)	Set prediction via object queries; removes anchors and NMS.

Table 2.1 : Representative Transformer architectures across modalities and tasks. We summarize their core innovations and modality scope.

Approach	Core Principle	Representative Methods	Strengths / Limitations
Metric-based	Learn a shared embedding space where classification is performed via similarity to class prototypes or exemplars. Often trained episodically.	Prototypical Nets [ 80 ] , Matching Nets [ 76 ] , Siamese Nets [ 77 ] , RelationNet [ 78 ]	+ Simple, fast inference; non-parametric adaptation. – Sensitive to embedding quality; vulnerable to domain shift.
Optimization-based	Meta-learn an initialization or update rule that enables rapid adaptation to new tasks with few gradient steps.	MAML [ 81 ] , Reptile [ 82 ] , Meta-SGD [ 83 ] , ANIL [ 84 ] , ProtoMAML [ 85 ]	+ Strong task-level generalization. – Computationally heavy; unstable in high dimensions.
Data Augmentation / Generative	Enrich support data via synthetic samples or feature-level perturbations using generative models or semantic mixing.	FeatGen [ 86 ] , GAN-based [ 87 ] , VAE-based [ 88 ] , Diffusion-based [ 89 ] , CLIP-based [ 90 ]	+ Mitigates scarcity; improves diversity. – Risk of unrealistic or biased samples.
Self-Supervised / Contrastive	Learn transferable representations via contrastive or self-supervised pretraining, then adapt to few-shot tasks.	MoCo [ 91 ] , SimCLR [ 92 ] , BYOL [ 93 ] , DINO [ 94 ] , SwAV [ 95 ]	+ Strong embeddings; reduces label reliance. – Pretraining requires massive unlabeled data.
Transformer-based / Prompting	Use attention-based architectures or prompts to model task structure and leverage foundation models.	Few-shot Transformers [ 96 ] , Prompt-tuning [ 97 ] , GPT-4V-assisted FSL [ 98 ] , Flamingo [ 43 ]	+ Scales with LLMs; interpretable task conditioning. – Prompt sensitivity; high compute demand.

Table 2.2 : Major approaches in few-shot learning (FSL), with principles, representative methods, and trade-offs.

Figure 3.1 : Given an unseen query image and a target video, the model is expected to localize all instances of objects present in the query image on the target video. [Best viewed in color].

Figure 3.2 : Illustration of the proposed QDETRv. The process begins with the feature extraction of a query image and video frames using a CNN encoder. A cross-attention mechanism and dot-product attention are used to create an attention map, transforming target frame features. The output is integrated into DETR’s encoder, and predictions for bounding boxes are generated using the DETR decoder. [Best viewed in color].

Figure 3.3 : Unsupervised Video Pretraining: Each frame, along with its associated query patch and context frame, is fed into QDETRv. The model detects the patch on every frame using decoded object queries responsible for predicting the bounding box. These decoded queries are then forwarded to the next frame as object queries for the decoder. Containing the semantic information extracted from previous frames for patch objects, they facilitate the detection in subsequent frames. During this phase, an unsupervised query reconstruction loss is also applied as an extra supervisory mechanism. The model has been pre-trained on a synthetically created dataset to enhance its performance. [Best viewed in color].

Split	Vid-OR		VidVRD
	Seen	Unseen	Seen	Unseen
#Train Videos	4,795	6,164	574	758
#Train Object Queries	53,277	57,599	3,413	4,395
#Train Object Categories	75	75	30	30
#Test Videos	1,319	32	184	42
#Test Object Queries	29,108	407	1,088	112

Table 3.1: Overview of the Vid-OR [ 113 ] and VidVRD [ 114 ] datasets, summarizing the distribution of training and testing videos, object queries, and object categories for both seen and unseen splits.

Method	Pretraining	Vid-OR [ 113 ]		VidVRD [ 114 ]
		Seen	Unseen	Seen	Unseen
CoAE [ 11 ]	-	35.3	32.8	29.2	27.1
Retrieval-Based [ 12 ]	-	3.11	2.47	2.08	1.48
OWL-ViT [ 6 ]	-	-	28.6	-	22.5
UP-DETR [ 126 ]	✗	37.4	35.4	33.7	28.5
UP-DETR [ 126 ]	✓	41.7	38.1	39.4	37.1

Table 3.2: Comparative performance of different baselines and our method on Vid-OR [ 113 ] and VidVRD [ 114 ] datasets. The metric for evaluation is mAP. QDETRv methods outperform other baselines across both datasets and splits, with pertaining significantly boosting the performance.

Panel of Figure 3 — Figure 3.4 : Ablation study on the temporal context window size using the VidOR dataset. A positive correlation is observed between the window size and the mAP.

Recurrent Query	VidVRD	VidOR
✗	37.9	40.3
✓	38.5	41.6

Table 3.3: Ablation studies on recurrent queries during pretraining showed improved performance. This highlights the importance of adding temporal context in the pretraining phase.

Figure 3.5 : Cross-attention visualization on test frame with varying window size. Attention improves on objects of interest with an increase in window size.

Method	Bear	Carrot	Coconut	Frisbee	Kangaroo	Lemon	Leopard	Melon	Orange	Stop sign
CoAE	37.8	19.2	33.6	25.4	42.6	21.3	46.9	22.8	28.1	30.6
OWL-ViT	35.7	20.3	28.5	21.7	38.6	22.8	42.8	20.3	25.3	27.4
UP-DETR	47.5	32.6	36.1	32.8	50.1	30.2	50.7	31.4	30.2	35.3
QDETRv	47.9	35.3	38.6	33.4	51.4	30.9	52.3	33.2	30.8	35.8

Table 3.4: Class-wise performance comparison. Each column indicates performance for distinct object categories. Larger objects like bears, kangaroos, and leopards are better detected than smaller ones like coconuts and lemons.

Figure 3.6 : Qualitative results. In the green box results, QDETRv accurately localizes objects from the query image on the left. The red box highlights the model’s limitations, with missed localization in videos where the object is only partially visible in the query image.

Figure 3.7 : Failure Cases. We observed the following major failure types: (i) The occluded object is missed, (ii) A visually similar object is detected as dear on the top left, (iii) a few small instances of the hammer are missed, (iv) an overlapping instance is commonly detected as a single object.

Figure 4.1 : Given a support set containing novel objects and a target video, the goal of FSVOD [ 110 ] is to detect all the instances of novel objects on the target video. We propose object-aware temporally consistent few-shot object detection framework that significantly improves state of the art for this task. [ Best viewed in color ].

Figure 4.2 : Object-aware Temporal Consistency. We utilize the OWL-ViT encoder output tokens as queries but, critically, only forward the matched object tokens (

c^{p}>\tau

) as key-value pairs to the next frame’s decoder. This mechanism selectively propagates high-confidence object features across frames. This enables efficient feature progression, reducing noise accumulation. We empirically found a threshold (

\tau

) and compared it with class probability (

c^{p}

) scores for selective propagation. This selective propagation mechanism significantly reduces noise accumulation across frames and maintains focused, consistent visual representations of detected objects throughout the video sequence.

Figure 4.3 : Few-shot Classification and Detection Heads. Our architecture processes object queries through parallel projection heads for classification and localization. Classification embeddings are compared against support embeddings via cosine similarity, while the detection head predicts bounding box coordinates. The temporal consistency is maintained by propagating matched object queries across frames.

Split	Vid-OR	VidVRD	FSVOD	FSYTV
#Train Videos	6,164	758	2553	1627
#Train Object Queries	57,599	4,395	-	-
#Train Object Categories	75	30	320	30
#Test Videos	32	42	949	608
#Test Object Queries	407	112	-	-
#Test Object Categories	9	5	100	10

Table 4.1 : Overview of the datasets used in this work.

	Method	FSVOD-500			FSYTV-40			VidOR			VidVRD
	Method	$AP$	$AP_{50}$	$AP_{75}$	$AP$	$AP_{50}$	$AP_{75}$	$AP$	$AP_{50}$	$AP_{75}$	$AP$	$AP_{50}$	$AP_{75}$
Non-temporal	FR-CNN	18.2	26.4	19.6	9.3	15.4	9.6	21.6	29.1	23.2	14.3	18.6	14.2
	FSOD	21.1	31.3	22.6	12.5	20.9	13.0	-	-	-	-	-	-
	RDN	18.2	27.9	19.7	8.1	13.4	8.6	-	-	-	-	-	-
	CoAE	18.4	29.3	18.9	9.8	18.4	10.1	35.8	43.7	36.1	30.2	38.1	31.8
	Retrieval-Based	1.3	2.4	1.6	0.6	1.1	0.8	1.4	2.6	1.3	1.8	2.5	2.0

Table 4.2 : Performance comparison on FSVOD-500, FSYTV-40, VidOR, and VidVRd in 5-shot set-up. Our Object-aware Video OWL-ViT consistently surpasses both state-of-the-art and baseline methods across all datasets and metrics.

	Method	FSVOD-500			FSYTV-40			VidOR			VidVRD
	Method	$AP$	$AP_{50}$	$AP_{75}$	$AP$	$AP_{50}$	$AP_{75}$	$AP$	$AP_{50}$	$AP_{75}$	$AP$	$AP_{50}$	$AP_{75}$
1-Shot	CoAE	17.2	27.8	17.2	6.7	17.3	9.8	33.2	42.5	35.2	29.6	37.6	31.5
	UP-DETR	17.3	24.3	18.4	9.7	16.9	9.9	35.6	45.2	37.2	8.5	15.2	9.3
	DeepSort+YOLOv8 + CLIP	12.2	20.1	12.8	4.1	7.9	4.7	21.4	28.4	22.7	21.9	31.7	22.8
	FSVOD	20.7	29.4	21.4	11.4	18.6	12.1	36.8	45.3	36.3	34.5	45.2	35.2
	QDETRv	25.2	34.9	25.8	13.6	24.1	14.2	41.8	50.3	42.2	40.3	49.7	40.1

Table 4.3 : Few-shot performance comparison. Our method consistently outperforms SoTA and implemented baselines across all settings and datasets, demonstrating its effectiveness across diverse settings and datasets.

	FSVOD-500			FSYTV-40
Temopral Fusion	$AP$	$AP_{50}$	$AP_{75}$	$AP$	$AP_{50}$	$AP_{75}$
✗	25.4	37.1	26.9	19.2	26.8	21.4
✓	30.6	42.9	32.1	21.2	29.8	23.5

Table 4.4 : Advantage of Object Aware Temporal-fusion. Object-Aware Temporal Fusion improves detection performance across all metrics on FSVOD-500 and FSYTV-40 datasets. Removing the Temporal fusion (✗) significantly reduces the performance shows the efficacy of object aware decoder.

Figure 4.4 : Threshold effect (

c^{p}>\tau

) during the object-aware temporal consistency.

Figure 4.5 : . Qualitative comparison of detection results across SoTA methods for two test videos from FSVOD-500. Our approach accurately detects multiple instances of the Bactrian camel (top) having a different visual appearance and precisely localizes the Autogyro (bottom), even as it moves farther from the camera in the subsequent frames. Notably, competing methods miss some camel instances, while Video OWL-ViT fails to detect all instances in the first two frames.

Figure 4.6 : Error type analysis on FSVOD val set. A lower AP score is better.

Method	FPS	Memory (GB)	AP (FSVOD-500)
Ours	14	8.1	30.6
QDETRv	11	7.6	26.1
Video OWL-ViT	10	8.4	25.8
FSVOD	16	5.8	25.1
UP-DETR	13	7.4	20.1
CenterTrack	12	6.2	20.6

Table 4.5 : Inference cost and accuracy on FSVOD-500. Our method is both faster and more accurate than QDETRv and Video OWL-ViT; the only trade-off is against FSVOD, which is leaner but substantially less accurate.

Figure 5.2 : Illustration of

T

-partite random field. Here, we illustrate graph construction using three frames (

T=3

) and three object bounding boxes (

M=3

). The ordered pair of bounding boxes are represented as a node. For this example, each frame will result in

2\times{3\choose 2}=6

nodes. Each node corresponds to a binary random variable (

u_{jk}^{i}

) that takes one of the two labels, i.e. {select (

1

), reject (

0

)}. The green- and red-filled nodes correspond to those random variables that the optimization framework aims to assign

1

and

0

, respectively, and the dotted blue line indicates the ground truth tracklet. [Best viewed in color] .

		VidVRD					VidOR
Approach	Support Set Size	$A_{s-t}^{sub}$	$mIoU_{s}$	$Acc_{s-t}^{obj}$	$mIoU_{o}$	$A_{s-t}^{r}$	$A_{s-t}^{sub}$	$mIoU_{s}$	$A_{s-t}^{obj}$	$mIoU_{o}$	$Acc_{s-t}^{r}$
tracklet-based	2	12.3	12.1	12.2	11.7	8.8	10.9	10.7	10.1	10.3	7.3
tracklet-based	4	12.6	12.9	12.8	12.0	9.6	11.1	11.0	10.9	10.4	8.2
tracklet-based	8	13.4	13.2	13.7	13.3	10.4	12.8	12.6	13.2	13.0	9.7
w/o vis.sim. potential, w/o GSA, w/o LLA	2	19.2	18.2	17.8	19.9	15.9	19.3	18.1	18.4	19.8	16.7
w/o vis.sim. potential, w/o GSA, w/o LLA	4	19.6	19.1	18.2	20.8	16.3	19.8	18.7	18.5	20.4	17.3

Table 5.1 : Comparison of baseline and proposed method on different support set sizes. For every method (baseline and all variants of our method) the performance is increased with an increase in support set size.

Method	$A^{sub}_{s-t}$	$A^{sub}_{s}$	$mIoU^{sub}_{s}$	$A^{obj}_{s-t}$	$A^{obj}_{s}$	$mIoU^{obj}_{s}$	$A^{r}_{s-t}$	$A^{sub}_{s-t}$	$A^{sub}_{s}$	$mIoU^{sub}_{s}$	$A^{obj}_{s-t}$	$A^{obj}_{s}$	$mIoU^{obj}_{s}$	$A^{r}_{s-t}$
	ImageNet-VidVRD						VidOR
VRC [ 169 ]	8.4	10.7	11.9	8.6	10.3	11.2	5.8	7.2	9.9	10.3	8.0	10.1	11.1	5.6
vRGV-fs [ 4 ]	8.2	11.1	12.4	8.3	11.6	12.9	6.1	7.5	10.3	10.9	7.3	10.7	11.4	6.3
Tracklet-based	12.6	15.1	15.9	12.8	14.8	15.3	9.6	11.1	13.7	14.2	10.9	12.6	13.8	8.2
Ours
w/o vis.sim. potential, w/o GSA, w/o LLA	19.6	21.4	19.1	18.2	18.7	20.8	16.3	19.8	20.7	18.7	18.5	19.2	20.4	17.3

Table 5.2 : Performance of few-shot referring relationship in videos. Each method is trained on the same split of train and test datasets with a support size of four videos. Ours (full model) represents the proposed method where global semantic aggregation (GSA) and local localization aggregation (LLA) are performed to enrich the query-conditioned relationship representation, a translational relation embedding is learned, and an objective function containing frame-level and visual similarity potential is optimized using belief propagation on the

T

-partite random field. For more details, refer Section 5.4 .

GSA	LLA	$A_{s-t}^{sub}$	$A_{s-t}^{obj}$	$A_{s-t}^{r}$
✗	✗	21.3	22.1	20.8
✓	✗	24.7	23.5	22.8
✗	✓	22.3	21.7	21.1
✓	✓	26.8	26.0	25.1

Table 5.3 : Ablation study to demonstrate the importance of global semantic aggregation (GSA) and local localization aggregation (LLA) on the ImageNet-VidVRD dataset. Our full model (the last row) performs better as compared to the settings where one or both of these are removed.

Relationship Embedding	$A_{s-t}^{sub}$	$A_{s-t}^{obj}$	$A_{s-t}^{r}$
Concatenation	18.3	20.1	18.7
Translation Embedding	26.8	26.0	25.1

Table 5.5 : Performance comparison of translational and Concat relationship embedding on ImageNet-VidVRD dataset. In this study, both embeddings are fused with the global and local context.

Figure 5.3 : Spatiotemporal localization accuracy of the proposed method with different support sizes on the ImageNet-VidVRD.

Figure 5.4 : Qualitative result on a selection of test videos. Each subject and object of the query relationship is spatiotemporally localized on the test videos. Red and blue color bounding boxes represent the subject and object, respectively. [Best viewed in color] .

subject	object	$A_{s-t}^{sub}$	$A_{s-t}^{obj}$	$A_{s-t}^{r}$
✓	✓	28.3	27.6	27.3
✓	✗	23.7	21.4	18.6
✗	✓	19.4	20.5	17.2

Table 5.6 : Performance comparison when subject or object is unseen during training on ImageNet-VidVRD dataset.

Figure 6.1 : ( Top ): Activity where text falls short to explain complex action or events, such as dumbbell side raises and bicycle kick , necessitating the need for a more intuitive query modality for localizing semantically matching moments. ( Bottom ): Shows the Vid2Vid moment retrieval setting, which is the goal of this work.

Figure 6.2 : M oment A lignment TR ansformer (MATR) Architecture. We represent the target video as a query-aligned representation capturing the required correlation and dependencies with query video at the abstract level (output of encoder) and finer level (output of decoder). These features are used to extract foreground target frames, as well as localize the relevant moment in terms of start and end times via two prediction heads. The alignment is computed using representations both before and after the encoder (shown using blue dotted and red dashed lines, respectively). [Best viewed in color] .

Figure 6.3 : Proposed self-supervised pre-training strategy: Given a target video, a query clip is randomly sampled from it and processed by the model. The model then predicts the boundaries of the selected clip within the target video, highlighting the corresponding frames with a green border. (Best viewed in color)

Dataset	Split	# Pairs	# Classes
SportsMoments	Train	750,393	16
	Val	10,000	4
	Test	10,000	9
ActivityNet-VRL [ 173 ]	Train	462,872	160
	Val	829	20
	Test	978	20

Table 6.1 : Statistical Comparison Between ActivityNet-VRL and the Proposed SportsMoments Dataset. While ActivityNet-VRL contains greater class diversity (160 training classes), SportsMoments offers substantially larger validation and test sets (10,000 pairs each versus 829/978), enabling more comprehensive evaluation of model generalization capabilities.

Dataset	Type	Mean	Median	Max.	Min.
SportsMoment	Target Video	98.3	98.2	208.8	35.4
SportsMoment	Query Video	6.0	5.0	66.0	1.0
ActivityNet-VRL	Target Video	38.2	29.3	191.5	1.3
ActivityNet-VRL	Query Video	8.2	7.2	57.2	0.5

Table 6.2 : Temporal Characteristics of Videos in SportsMoments and ActivityNet-VRL Datasets. SportsMoment contains significantly longer target videos (98.3s mean vs 38.2s) while maintaining shorter query segments (6.0s vs 8.2s).

Figure 6.4 : We visualize the distribution of a) target video durations and b) query video durations in our proposed dataset, SportsMoments.

Figure 6.5 : Frequency distribution of action classes in the SportsMoments dataset. The dataset comprises a wide range of actions across the two sports, including frequent activities such as "header," “four," and “throw," as well as less common ones like “red card" and “uppercut." The figure highlights the variation in class occurrences, reflecting the diversity and imbalance in real-world sports scenarios.

4a5Oou5u3io	GrIOmtbyHNc	5j2tQfj2Osk	mo3H_qusvxI	RpmSeHG8SJ0	cpU0GSxliSU
EOvpUWZC6ng	Txq8RoclyKM	-vYVBUyEuAA	IvjBlpb8GTA	ukuNlWxcR2k	LmqBbPwCy8o
zuPuDzEWI-A	OFbyNU6UQQs	iPXz5XWfj1k	IA5BNsNu6fw	9sRb61qsoLs	X0we8220k74
MbdWVX2jUhA	04vWaGaqw-4	92s9HCGG4vQ	BMXfphiHj1s	1yHWGw8DH4A	shyBGceNASI
YqKYpgZ9FWU	GxO1tlZKa8E	qElsVosGiXA	zUTUA0TKvfQ	G4vxbWF79f4	v9Akttif8gY
Mtnmvuhs2RA	3K90O0Y9NZc	9gyv2xh7qQw	1-SCubr2yc4	ezQUPi74IIQ	cTqY53zWypk
qQfuOCns4P4	iGGElV40TcU	X6VNTk-Vl5w	iFYm_ogzmdA	6j1Oj5N7kwE	WkEvdVImkec

Table 6.3 : YouTube IDs used for SportsMoments dataset creation. We used 80 videos for dataset curation.

				ActivityNet-VRL [ 214 ]		SportsMoments
	Methods	Variants	Query Modality	mIoU	R@1	mIoU	R@1
Fully-supervised	Random	-	Video	7.3	16.2	-	-
	Video Match	-	Video	12.4	24.3	-	-
	SST	-	Video	17.1	33.2	-	-
	CGBM	-	Video	25.7	43.5	-	-
	GDP	-	Video	27.8	44.0	-	-

Table 6.4 : Comparing MATR with fully-supervised Vid2VidMR methods, VLMs, image-VMR methods, and variants of Text-VMR methods. Trained implies random initialization. Finetuned implies initialization using a pre-trained (respective to each architecture) checkpoint. Query video can be represented using text captions or as video itself. Since MATR is inherently designed to take video query as input, experimenting with text captions is not needed. For more details of baselines and their variants, please refer to Section 6.6.1 . Results for the overall best, best among implemented baselines and previously reported SOTA are highlighted in bold, underline and box, respectively.

	Recall		mAP
Method	[email protected]	[email protected]	[email protected]	[email protected]	Avg.
Moment-DETR	53.9	34.8	-	-	32.2
QD-DETR	62.7	46.7	62.2	41.8	41.2
UniVTG	59.7	45.8	61.9	41.9	38.8
BM-DETR	63.6	46.8	63.7	43.5	44.2
CG-DETR	67.4	52.1	65.6	45.7	44.9

Table 6.5 : Comparative Analysis of MATR versus State-of-the-Art Text-VMR Models on the QVHighlights Dataset. While primarily designed for video queries, our approach demonstrates competitive performance on text queries.

		ActivityNet-VRL		SportsMoments
Pre-Fusion	Post-Fusion	mIoU	R@1	mIoU	R@1
✗	✗	49.7	50.2	49.1	46.6
✗	\checkmark	52.3	52.9	53.1	48.2
\checkmark	✗	50.4	51.2	52.4	46.7
\checkmark	\checkmark	53.2	54.8	56.2	52.7

Table 6.6 : Impact of Pre-Fusion and Post-Fusion Alignment Strategies on MATR Performance. This study is done with CLIP features, soft-DTW alignment and without pre-training.

		ActivityNet-VRL		SportsMoments
Pre-training	Augmentation	mIoU	R@1	mIoU	R@1
✗	-	53.2	54.8	56.2	52.7
\checkmark	✗	54.1	55.6	56.9	53.6
\checkmark	\checkmark	56.8	57.1	59.2	56.5

Table 6.7 : Impact of Pre-training and Data Augmentation on MATR Performance Across both Datasets. This study is done with CLIP features and soft-DTW alignment.

	ActivityNet-VRL		SportsMoments
	mIoU	R@1	mIoU	R@1
Pre-fusion alignment ( $\mathrm{A^{\text{pre}}}$ )	34.1	32.6	31.6	29.8
Post-fusion alignment ( $\mathrm{A^{\text{post}}}$ )	38.3	35.2	36.4	33.5
Prediction-heads	56.8	57.1	59.2	56.5

Table 6.8 : Advantage of predicting using prediction heads on the decoder. This study is done with CLIP features, soft-DTW alignment and with pre-training.

	ActivityNet-VRL		SportsMoments
Alignment	mIoU	R@1	mIoU	R@1
None	50.4	51.8	52.1	48.6
TCC [ 196 ]	52.4	52.7	53.9	52.8
Needleman-Wunsch [ 227 ]	50.8	51.2	52.6	49.5
LCSS [ 228 ]	50.9	52.6	53.7	49.6
Drop-DTW [ 175 ]	54.2	54.7	55.6	54.7

Table 6.9 : Performance Comparison of Sequence Alignment Methods. This comparison is done with CLIP features.

Figure 6.6 : Visualization of Vid2VidMR on four query-target samples. Our proposed MATR model shows improved generalization capabilities over the best-performing baseline methods (variant (e), i.e., finetuned with video query input). The visualization demonstrates how our approach more accurately aligns with ground truth temporal boundaries across diverse activity types and durations. Start and end times for the ground truth and predicted moments are shown in the brackets.

	ActivityNet-VRL		SportsMoments
Feature	mIoU	R@1	mIoU	R@1
ResNet [ 74 ]	42.3	42.8	50.6	48.1
I3D [ 165 ]	43.7	43.4	52.2	50.6
S3D [ 188 ]	46.2	47.5	53.9	52.1
C(2+1)D [ 229 ]	46.6	47.1	53.3	50.9
CLIP [ 50 ]	53.2	54.8	56.2	52.7

Table 6.10 : Comparison of Video Feature Extraction Backbones. Slow-Fast networks achieve superior performance across both datasets. This study was done with soft-DTW alignment and without pretraining.

	ActivityNet-VRL		SportsMoments
Noise Type	mIoU	R@1	mIoU	R@1
Frame Reverse	42.7	43.1	43.1	44.6
Speed Variation	53.3	55.9	57.4	55.3
Gaussian Noise	55.8	56.3	58.5	56.1
None	56.8	57.1	59.2	56.5

Table 6.11 : Robustness Analysis of MATR against Different Types of Perturbations. This study is done with CLIP features, soft-DTW alignment and with pre-training.

Method	Params	FLOPs	Inference	Memory	Throughput
	(M)	(G)	Time (ms)	(GB)	(FPS)
Moment-DETR	45.2	12.3	48.5 $\pm$ 2.1	2.1	20.6
QD-DETR	62.8	16.7	52.3 $\pm$ 2.4	2.6	19.1
UniVTG	38.4	10.2	45.8 $\pm$ 2.0	2.0	21.8
CG-DETR	69.5	18.1	58.4 $\pm$ 2.6	3.1	17.1
BM-DETR	81.3	23.8	71.5 $\pm$ 3.5	3.8	14.0

Table 6.12 : Efficiency analysis of MATR and baselines. All measurements conducted on an NVIDIA A6000 GPU with batch size 1, averaged over 100 inference runs.

Sampling Rate	ActivityNet-VRL		SportsMoments		Inference	Memory
Sampling Rate	mIoU	R@1	mIoU	R@1	Time (ms)	(GB)
1 frame/4s	51.3	52.1	53.8	50.2	30.1	1.7
1 frame/3s	54.2	55.4	56.4	53.6	41.2	2.1
1 frame/2s (default)	56.8	57.1	59.2	56.5	53.8	2.5
1 frame/1s	56.2	56.8	59.5	56.1	95.6	4.4
1.5 frames/s	55.8	56.3	58.1	55.4	137.4	6.3

Table 6.13 : Impact of frame sampling rate on MATR performance and cost. Inference time and memory measured on an NVIDIA A6000 GPU.

Video Moment Retrieval

See all 16 papers

Aug 16, 2026

What You Ask is What You Ground: Bridging Question Intent to Temporal Evidence for Grounded VideoQA

We study a critical yet overlooked failure mode in Grounded Video Question Answering: question-invariant grounding, where models predict nearly identical temporal segments for different questions about the same video. We trace this behavior to two structural limitations in prior common designs: (i) modality isolation that fixes video representations before they receive question semantics, and (ii) weak question injection inside the grounding module. To address this, we propose GroundFormer, which conditions video features on question intent before localization via learnable communication tokens that mediate directed visuo-lingual interaction. On top of the question-conditioned features, a factorized MIL cross-attention couples answer selection with temporal evidence under candidate-level supervision, while Gaussian smoothing converts peaked attention into temporally coherent segments. We further introduce a hierarchical multi-modal contrastive loss that aligns video, question, and answer embeddings across a two-pass training pipeline. GroundFormer achieves state-of-the-art grounded VideoQA performance on NExT-GQA and STAR, substantially improving question-discriminative temporal grounding.

Jinhwan Seo, Kyubeom Han, Jumin Lee +2

KAIST · Ewha Womans University

Video QA Temporal Video Grounding

Ask PDF

Oct 18, 2026

DiffGround: Semantic Difference-Guided Zero-Shot Video Question Grounding

Video Question Grounding (VideoQG) requires both accurate question answering and precise temporal localization. Existing supervised methods rely on costly temporal annotations, while existing zero-shot approaches are primarily designed for Video Question Answering (VQA) and are not optimized for temporal grounding. We propose DiffGround, an action-driven zero-shot framework built upon frozen Multimodal Large Language Models (MLLMs). Our framework first predicts an answer-relevant action cue that transforms the input question into an explicit semantic target for grounding. Action Cue converts a reasoningoriented question into an action-centric retrieval query. Guided by the action cue, a Vision-Language Model (VLM) efficiently selects informative keyframes for reasoning. Finally, we introduce Semantic Diff Refinement, a training-free temporal refinement method that exploits semantic differences between neighboring SigLIP2 embeddings to recover complete action segments from sparse predictions. Extensive experiments on NExT-GQA and STAR show that DiffGround consistently outperforms existing zero-shot and supervised methods, establishing a new state of the art for Video Question Grounding.

Cheng-Han Lee, Amir Salimi, Hsiang-Wei Huang +3

The University of Texas at Austin, United States · Amazon , United States

Temporal Video Grounding Video QA Zero-Shot Learning

Ask PDF

May 13, 2026

EvoGround: Self-Evolving Video Agents for Video Temporal Grounding

Video temporal grounding (VTG) takes an untrimmed video and a natural-language query as input and localizes the temporal moment that best matches the query. Existing methods rely on large, task-specific datasets requiring costly manual annotation. We introduce EvoGround, a framework of two coupled self-evolving agents, a proposer and a solver, that learn temporal grounding from raw videos without any task-specific annotations. The proposer generates query--moment pairs from raw videos, while the solver learns to ground them and feeds back signals that improve the proposer in return. Through this self-reinforcing reinforcement-learning loop, the two agents are initialized from the same backbone and mutually improve across iterations. Trained on 2.5K unlabeled videos, EvoGround matches or surpasses most fully supervised models across five VTG benchmarks, while emerging as a state-of-the-art fine-grained video captioner. We publicly release all resources at https://minjoong507.github.io/projects/EvoGround/.

Minjoon Jung, Byoung-Tak Zhang, Lorenzo Torresani

Seoul National University · Northeastern University

Temporal Video Grounding Self-Supervised Learning

Ask PDF

Abstract

Video Moment Retrieval

Explore similar work

What You Ask is What You Ground: Bridging Question Intent to Temporal Evidence for Grounded VideoQA

DiffGround: Semantic Difference-Guided Zero-Shot Video Question Grounding

EvoGround: Self-Evolving Video Agents for Video Temporal Grounding