Investigating Query-Insensitive Behavior in Spatio-Temporal Video Grounding

Eryk Kołodziejczyk; Alberto Presta; Karol Szurkowski; Michal Byra

cs.CVOct 5, 2026

Investigating Query-Insensitive Behavior in Spatio-Temporal Video Grounding

Authors: Eryk Kołodziejczyk, Alberto Presta, Karol Szurkowski, Michal Byra

Organizations: Samsung AI Center, Warsaw, Poland · Institute of Fundamental Technological Research, Polish Academy of Sciences, Warsaw, Poland

Abstract

Spatio-temporal video grounding (STVG) aims to localize objects or events described by natural language queries in both space and time. Existing STVG models are typically trained and evaluated under the assumption that each query is relevant to the input video. In this work, we challenge this assumption by studying the behavior of state-of-the-art STVG models under irrelevant queries and missing textual input. Our experiments show that current models can still produce plausible spatio-temporal predictions even when the query is unrelated to the video or removed entirely. We further analyze HCSTVG-v2 and VidSTG to identify dataset regularities that may encourage such query-insensitive behavior. Our study highlights an underexplored limitation of STVG models and motivates negative-aware evaluation protocols and architectures that explicitly assess query relevance.

Figures & tables

Panel of Figure 1 — Figure 1: Temporal analysis of HCSTVG-v2 and VidSTG: moment duration and its time center. Values are decimal units normalized to the video duration.

Panel of Figure 2 — Figure 2: Comparison of spatial annotations between HCSTVG-v2 and VidSTG datasets. Datapoints show average bounding box centers on X and Y axes, normalized to video frame size. Scatter plot indicates frequency of centers at the same spot.

Panel of Figure 3 — Figure 3: Comparison of words distribution in datasets.

	HCSTVG-v2				VidSTG declarative				VidSTG interrogative
	TubeDETR		TA-STVG		TubeDETR		TA-STVG		TubeDETR		TA-STVG
Query type	m_vIoU	m_tIoU	m_vIoU	m_tIoU	m_vIoU	m_tIoU	m_vIoU	m_tIoU	m_vIoU	m_tIoU	m_vIoU	m_tIoU
Positive	0.343	0.525	0.335	0.548	0.283	0.468	0.342	0.517	0.244	0.461	0.291	0.501
Negative in-domain	0.170	0.416	0.206	0.476	0.083	0.298	0.097	0.314	0.092	0.318	0.101	0.323
	-50%	-21%	-39%	-13%	-71%	-36%	-72%	-39%	-62%	-31%	-65%	-36%
Negative out-of-domain	0.161	0.398	0.186	0.457	0.078	0.273	0.106	0.274	0.074	0.276	0.085	0.279

Table 1: Inference results: we evaluate TubeDETR and TA-STVG on VidSTG and HCSTVG-v2 datasets, using m_vIoU and m_tIoU. Three query types are compared: positive, negative in-domain, and negative out-of-domain. For negative queries, we report relative IoU changes compared to positive ones. A smaller decrease in the IoU value indicates a behavior that is less sensitive to changes in the query.

	HCSTVG-v2		VidSTG
Query	m_vIoU	m_tIoU	m_vIoU	m_tIoU
Original	${0.357}_{\pm{0.01}}$	${0.535}_{\pm{0.00}}$	${0.267}_{\pm{0.02}}$	${0.468}_{\pm{0.01}}$
Empty	${0.262}_{\pm{0.01}}$	${0.485}_{\pm{0.01}}$	${0.151}_{\pm{0.02}}$	${0.399}_{\pm{0.00}}$
	-27%	-9%	-43%	-15%

Table 2: Training validation results of TA-STVG model on HCSTVG-v2 and VidSTG datasets. Comparison of trainings with original and empty query. For empty query, we report IoU changes relative to original one.

Model version	m_vIoU	m_tIoU
TA-STVG, baseline	${0.357}_{\pm{0.01}}$	${0.535}_{\pm{0.00}}$
TA-STVG, text-processing modules removed	${0.238}_{\pm{0.01}}$	${0.479}_{\pm{0.00}}$

Table 3: Results for TA-STVG model trained on HCSTVG-v2 without the text-processing modules.

	HCSTVG-v2		VidSTG
	m_vIoU	m_tIoU	m_vIoU	m_tIoU
Center-prior baseline	0.063	0.248	0.014	0.089

Table 4: Results from testing evaluation with constant spatio-temporal annotations taken from data distribution charts.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Comparison of averaged spatial heatmaps between HCSTVG-v2 and VidSTG dataset. Heatmaps illustrate mean bounding boxes of each video in the whole dataset.

Panel of Figure 5 — Figure 5: Qualitative results of spatio-temporal grounding prediction of TA-STVG model (weights shared by its authors) on the example HCSTVG-v2 videos. The green border marks temporal prediction frame, the yellow box stands for spatial prediction. Input queries are in order: (1) positive (2) negative in-domain (3) negative out-of-domain.

Explore similar work

Learning Compositional Spatio-Temporal Video Grounding with Synthetic Curriculum
Aug 31, 2026Xingjian Wang, Shijian Wang, Yibo Wang +4Video Temporal Grounding Synthetic Training Data
1Monash University · 2Southeast University · 3Shanghai University of Electric Power
DSE-VTG: Dual-Side Enhancement for Training-Free Video Temporal Grounding
Sep 8, 2026Zhuo Cao, Bingqing Zhang, Sen Wang +1Video Temporal Grounding Dense Temporal Annotation
The University of Queensland, Australia
DART: Difficulty-Adaptive Routing for Zero-Shot Video Temporal Grounding
Jul 1, 2026Zhengbo Zhang, Mark He Huang, Zhigang Tu +1Video Temporal Grounding Temporal Grounding
Singapore University of Technology and Design · Wuhan University · University of California, Merced

Aug 31, 2026, cs.CVJ/K move · Enter open · S save

Learning Compositional Spatio-Temporal Video Grounding with Synthetic Curriculum

Xingjian Wang, Shijian Wang, Yibo Wang, Zihao Yu+3

1Monash University · 2Southeast University · 3Shanghai University of Electric Power

Video Temporal Grounding Synthetic Training Data Curriculum Reinforcement Learning Curriculum

Open paper AskPDF arXiv

Despite the impressive progress of recent MLLMs on spatio-temporal video grounding (STVG), existing evaluations and training data focus primarily on simple queries. They largely overlook the compositional queries prevalent in real-world scenarios, where a target must be disambiguated by jointly reasoning about its attributes and relations to other entities. To bridge this gap, we propose Compositional Spatio-Temporal Video Grounding (CompSTVG), a task that requires models to process complex textual queries where every intertwined attribute and relational cue is essential for disambiguation. To facilitate this task at scale, we build a synthetic data engine that leverages a spatio-temporal scene graph as a difficulty measure and casts difficulty-controlled query synthesis as a constraint programming problem, producing difficulty-graded data for both evaluation and training. Built on this engine, we introduce STVG-CompBench, a benchmark stratified by explicit difficulty levels that jointly capture temporal complexity and spatial interference. Evaluating 11 representative STVG models on STVG-CompBench reveals that current models perform poorly on compositional queries, exhibiting a sharp performance drop that is typically obscured by overall dataset-level averages. We further construct synthetic training data and propose CurrSTVG, a curriculum reinforcement learning framework that delivers consistent gains, with the largest improvements observed on the most challenging compositional queries.

Abstract

Explore similar work

Learning Compositional Spatio-Temporal Video Grounding with Synthetic Curriculum

DSE-VTG: Dual-Side Enhancement for Training-Free Video Temporal Grounding

DART: Difficulty-Adaptive Routing for Zero-Shot Video Temporal Grounding