Pocket-STVG: lightweight architecture for Spatio-Temporal Video Grounding

Alberto Presta; Michal Byra; Grzegorz Stefański; Karol Szurkowski; Eryk Kołodziejczyk; Krzysztof Arendt

cs.CVSep 25, 2026

Pocket-STVG: lightweight architecture for Spatio-Temporal Video Grounding

Authors: Alberto Presta, Michal Byra, Grzegorz Stefański, Karol Szurkowski, Eryk Kołodziejczyk, Krzysztof Arendt

Organizations: Samsung AI Center, Warsaw, Poland · IFTR, Polish Academy of Sciences, Warsaw, Poland

Abstract

Spatio-Temporal Video Grounding (STVG) aims to localize the spatio-temporal tube in a video corresponding to a natural language query. While recent methods achieve strong performance in fully supervised, weakly supervised, and zero-shot settings, they typically rely on computationally expensive architectures, complex training pipelines, or multimodal large language models. We present Pocket-STVG (P-STVG), a lightweight cascade architecture that addresses STVG by combining efficient pre-trained components instead of large end-to-end models. P-STVG integrates a temporal-aware video encoder based on MobileViCLIP, a spatial encoder-decoder derived from MDETR, and a shared aligned text encoder. Temporal localization is performed through either a lightweight 1D U-Net or a simple thresholding strategy, enabling the same framework to operate in both weakly supervised and zero-shot settings. Furthermore, video representations are precomputed independently of the query, yielding an indexing-friendly pipeline for efficient inference and large-scale video collections. Despite requiring fewer than 90M parameters, P-STVG performs on par with weakly supervised methods and improves on earlier zero-shot approaches at a fraction of their memory and computational cost, establishing a favorable performance-efficiency trade-off for STVG.

Figures & tables

Figure 1: Performance–efficiency comparison on HC-STVG v2. P-STVG occupies a compact operating point, retaining useful grounding accuracy with substantially fewer parameters than recent weakly supervised and zero-shot methods. Marker shapes indicate the supervision setting.

Figure 2: Blueprint of our proposed P-STVG architecture. The horizontal line separates indexing and inference phase, while red boxes represents modules we fine-tuned or trained, while gray boxes denote pre-trained modules that were kept frozen.

Figure 3: Diagram of the T-Unet . Values inside boxes are input dimension, output dimension, kernel size, and padding. On square brackets there is embedding dimensions at the specific stage.

Sup.	Method	STVG-sup. Pars	Total Pars.	HCSTVG-v1				HCSTVG-v2
		(M)	(M)	m_t	m_v	[email protected]	[email protected]	m_t	m_v	[email protected]	[email protected]
Full	Bridge-STG [ 36 ]	$\sim$ 250–400	$\sim$ 8.98k	-	-	-	-	64.1	41.5	67.5	38.6
Full	TA-STVG [ 10 ]	206	234	53	39.1	63.1	36.8	60.4	40.2	65.8	36.7
	ART-STVG [ 8 ]	207	235	-	-	-	-	59.2	39.2	64.4	33.2
	TubeDETR [ 47 ]	185	185	43.7	32.4	49.8	23.5	53.9	36.4	58.8	30.6
	CG-STVG [ 9 ]	203	231	52.8	38.4	61.5	36.3	60	39.5	64.5	36.3

Table 1: Quantitative comparison on HCSTVG-v1/v2, between our methods and SOTA algorithms. Here, m_t and v_t stand for m_tIoU and m_vIoU, respectively. Light red indicates our method. STVG-supervised parameters count only modules exposed to STVG annotations

Method	m_tIoU	m_vIoU	[email protected]	[email protected]
Bridge-STG	52.6	37.2	52.4	37.4
TA-STVG	51.7	34.4	48.2	33.5
STPro	-	15.5	19.4	12.7
COSPAL	41.10	16	20.1	13.1
Tube-RMC	-	15.93	25.16	9.09
RedCircle	-	8.6	7.6	0.9

Table 2: Spatio-temporal performance on VidSTG declarative.

Method	Peak VRAM (GB)	Query Time (s)	TFLOPs
TA-STVG	4.36	0.90	2.87
UNLEASH	14.71	3.35	8.09
P-STVG T-Unet	3.94	0.71	1.11

Table 3: Efficiency analysis on HC-STVG. We report peak VRAM, query time, and TFLOPs under the same inference protocol on an RTX 4060 GPU with 8 GB of VRAM. Blue indicates methods that could not be executed within this hardware budget and were measured on an H100 instead.

	architecture configuration				training details		performance
ID	MC	CC	L	Total pars	LR	BS	m_tIoU	tIoU@3	tIoU@5
$TVD_{1}$	128	8	1	351625	1e-4	16	50.11	78.45	54.95
$TVD_{2}$	512	8	1	4749833	1e-4	16	51.66	80.75	58.10
$TVD_{3}$	512	8	2	5536265	5e-5	16	52.14	81.75	59.95
$TVD_{4}$	512	8	3	7896585	5e-5	16	52.47	81.75	61.60
$TVD_{5}$	512	8	4	10256905	1e-4	16	53.12	82.65	61.15

Table 4: Results in terms of m_tIoU,tIoU@3, and tIoU@5 on HCSTVG-v2 considering different T-Unet configurations. In light red the one chosen for the main architecture

Figure 4: m_tIoU with respect to MC, CC, and L. Each blue dot is one trained configuration from the full grid search. The red line is the linear trend line that best fits them, the light red zone represents the 95% confidence interval around the regression line, and $r$ is the Pearson correlation.

NC	m_tIoU T-Unet	m_tIoU Thresholding	MF (Kb)	IT (s)
40	51.87	43.71	82	3.29
24	52.40	44.31	49.3	1.95
16	48.56	44.01	32.9	1.29
12	45.67	43.83	24.7	0.99
20	53.11	45.19	44	1.61

Table 5: Evaluation of our pipeline in terms of m_tIoU considering different number of chunks (NC), on hcstvg-v2. We also report memory footprint for a single video (MF) and time for indexing (IT). Light red indicates main architecture configuration.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Starting Net	$\lambda$	R@1	R@5	R@10
Scratch	1	72.27	86.11	90.84
Scratch	0	72.09	85.55	89.66
Teacher	1	79.41	91.84	94.48
Teacher	0	79.28	91.65	94.29
-	-	82.9	93.8	95.6

Appendix

Table 6: Results of MDETR fine-tuning for text encoder alignment in terms of Recall@{1,5,10}. Light blue line represents performance of the teacher.

threshold $\tau$	m_tIoU	tIoU@3	tIoU@5
0.65	45.44	71.55	44.55
0.60	45.56	71.55	44.55
0.55	45.57	70.9	46
0.5	45.19	70.15	46.2
0.45	44.81	69.65	45.75
0.4	44.76	69.5	44.7

Appendix

Table 7: Performance on HCSTVG-v2 considering our zero-shot methods with different threshold. Light blue indicates configuration chosen for the main paper.

Method	Component	Model used	Parameters	State
Unleash [ 50 ]	Vision Encoder	CLIP-ViT-L/14	$\sim$ 300M	Frozen
	Language Model	LLaVA-v1.5-7B	$\sim$ 6.7B	Frozen
	Projection Layer	MLP Adapter	$\sim$ 20M	Frozen
	TOTAL	-	$\sim$ 7B	Frozen
E3M [ 2 ]	Vision Encoder	CLIP-ViT-L/14	$\sim$ 300M	Frozen
	Language Model	Video-LLaVA-7B	$\sim$ 7B	Frozen

Appendix

Table 8: Parameter analysis of state-of-the-art method for zero-shot and weak-shot STVG.

Panel of Figure 5 — Figure 5: Qualitative samples taken from HCSTV-v2 using our method with thresholding temporal grounding (top) and Unet temporal grounding (bottom).

Panel of Figure 6 — Figure 6: Qualitative samples taken from HCSTV-v2 using our method with thresholding temporal grounding (top) and Unet temporal grounding (bottom).

Temporal Video Grounding

Temporal video grounding locates the moments in a video that match a natural-language query, enabling precise search and evidence-based understanding.

See all 159 papers

Towards Long-Form Spatio-Temporal Video Grounding
Feb 26, 2026Xin Gu, Bing Fan, Jiali Yao +5Temporal Video Grounding Long-Context Modeling
University of Chinese Academy of Sciences · 2UNT · Institute of Software Chinese Academy of Sciences +2
Unlocking the Potential of Grounding DINO in Videos: Parameter-Efficient Adaptation for Limited-Data Spatial-Temporal Localization
Apr 14, 2026Zanyi Wang, Fan Li, Dengyang Jiang +4Temporal Video Grounding VLM Adaptation
SGIT AI Lab, State Grid Corporation of China · University of HongKong · Zhejiang University of Technology
Investigating Query-Insensitive Behavior in Spatio-Temporal Video Grounding
Oct 5, 2026Eryk Kołodziejczyk, Alberto Presta, Karol Szurkowski +1Temporal Video Grounding
Samsung AI Center, Warsaw, Poland · Institute of Fundamental Technological Research, Polish Academy of Sciences, Warsaw, Poland

Feb 26, 2026J/K move · Enter open · S save

Towards Long-Form Spatio-Temporal Video Grounding

Xin Gu, Bing Fan, Jiali Yao, Zhipeng Zhang+4

University of Chinese Academy of Sciences · 2UNT · Institute of Software Chinese Academy of Sciences+2

Temporal Video Grounding Long-Context Modeling Vision-Language Grounding

Open paper AskPDF arXiv

In real scenarios, videos can span several minutes or even hours. However, existing research on spatio-temporal video grounding (STVG), given a textual query, mainly focuses on localizing targets in short videos of tens of seconds, typically less than one minute, which limits real-world applications. In this paper, we explore Long-Form STVG (LF-STVG), which aims to locate targets in long-term videos. Compared with short videos, long-term videos contain much longer temporal spans and more irrelevant information, making it difficult for existing STVG methods that process all frames at once. To address this challenge, we propose an AutoRegressive Transformer architecture for LF-STVG, termed ART-STVG. Unlike conventional STVG methods that require the entire video sequence to make predictions at once, ART-STVG treats the video as streaming input and processes frames sequentially, enabling efficient handling of long videos. To model spatio-temporal context, we design spatial and temporal memory banks and apply them to the decoders. Since memories from different moments are not always relevant to the current frame, we introduce simple yet effective memory selection strategies to provide more relevant information to the decoders, significantly improving performance. Furthermore, instead of parallel spatial and temporal localization, we propose a cascaded spatio-temporal design that connects the spatial decoder to the temporal decoder, allowing fine-grained spatial cues to assist complex temporal localization in long videos. Experiments on newly extended LF-STVG datasets show that ART-STVG significantly outperforms state-of-the-art methods, while achieving competitive performance on conventional short-form STVG. Our code is at: https://github.com/HengLan/ART-STVG.

Abstract

Temporal Video Grounding

Explore similar work

Towards Long-Form Spatio-Temporal Video Grounding

Unlocking the Potential of Grounding DINO in Videos: Parameter-Efficient Adaptation for Limited-Data Spatial-Temporal Localization

Investigating Query-Insensitive Behavior in Spatio-Temporal Video Grounding