UniversalVTG: A Universal and Lightweight Foundation Model for Video Temporal Grounding

Joungbin An; Agrim Jain; Kristen Grauman

cs.CVApr 9, 2026

UniversalVTG: A Universal and Lightweight Foundation Model for Video Temporal Grounding

Authors: Joungbin An, Agrim Jain, Kristen Grauman

Organizations: The University of Texas at Austin

Abstract

Video temporal grounding (VTG) is typically tackled with dataset-specific models that transfer poorly across domains and query styles. Recent efforts to overcome this limitation have adapted large multimodal language models (MLLMs) to VTG, but their high compute cost and limited video context still hinder long-video grounding. We instead scale unified supervision while keeping the model lightweight. We present UniversalVTG, a single VTG model trained with large-scale cross-dataset pretraining. An offline Query Unifier canonicalizes heterogeneous query formats into a shared declarative space, reducing linguistic mismatch and preventing the negative transfer observed under naïve joint training. Combined with an efficient grounding head, UniversalVTG scales to long, untrimmed videos. Across diverse benchmarks-GoalStep-StepGrounding, Ego4D-NLQ, TACoS, Charades-STA, and ActivityNet-Captions-one UniversalVTG checkpoint achieves state-of-the-art performance versus dedicated VTG models. Moreover, despite being $>100\times$ smaller than recent MLLM-based approaches, UniversalVTG matches or exceeds their accuracy on multiple benchmarks, offering a practical alternative to parameter-heavy MLLMs.

Figures & tables

Figure 2: UniversalVTG Framework. A single, lightweight model generalizes across heterogeneous video domains and query styles. (Left) Diverse videos are mapped to a Shared Representation via an efficient backbone, while a Query Unifier canonicalizes multi-style text inputs into a Unified Semantic Space . (Right) With both modalities standardized, one grounding head localizes events across varied viewpoints (ego/exo), durations (short/long), and linguistic forms (e.g., questions, declarations). UniversalVTG achieves real-time inference ( $\sim$ 10 ms/query) suitable for long-form deployment.

Backbone	Feature FPS	Avg. R@1&5 ( $\uparrow$ )	TFLOPs / min. video ( $\downarrow$ )
Video-based backbones
SlowFast [ 10 ]	1.88	16.3	7.40
EgoVLP [ 27 ]	1.88	25.7	83.1
InternVideo [ 52 ]	1.88	29.0	161.0
Image-based backbones
CLIP-ViT-L14 [ 39 ]	2.00	20.0	21.0

Table 1: Backbone comparison on Ego4D-NLQ. We report feature extraction rate (FPS), mean of R@1&5 (evaluated at IoU 0.3/0.5), and feature-extraction compute (TFLOPs/min). We follow prior work [ 1 ] and use the same extraction settings per backbone for a fair comparison: video-based encoders process 32-frame clips with stride 16 on 30 FPS videos, whereas image-based encoders operate at 2 frames per second.

Training Regime	GoalStep		NLQ		TACoS		Charades-STA		ANet-Cap.
Training Regime	Perf.	$\Delta$	Perf.	$\Delta$	Perf.	$\Delta$	Perf.	$\Delta$	Perf.	$\Delta$
Dataset-Specific	22.92	–	19.55	–	56.70	–	62.35	–	39.52	–
Joint (Naïve)	22.22	-3.05%	20.16	+3.09%	55.16	-2.72%	61.68	-1.07%	38.11	-3.56%
Joint (Unified)	23.10	+3.94%	21.14	+4.86%	58.76	+6.53%	63.25	+2.55%	40.14	+5.31%

Table 3: Effect of joint training and language unification. All settings use the same visual encoder and grounding head (HieraMamba); only the training regime differs. Performance is reported as Avg. R@1 (higher is better).

\Delta

denotes relative change compared to dataset-specific training.

Benchmark	Original Query (Input)	Unified Query (Canonical Space)	Resolved Friction
GoalStep	“Add onion to the pan”	“A person added onion to the pan”	Imperative, No Subject
Ego4D-NLQ	“What did I put in the bucket?”	“I placed an object into the bucket.”	Interrogative
TACoS	“Takes a cup out of the cabinet.”	“A person took a cup out of the cabinet.”	Missing Subject
Charades-STA	“person takes a cup out the fridge.”	“A person took a cup out of the fridge.”	Missing Preposition
ActivityNet	“A woman is walking along a track.”	“A woman walked along a track.”	Present Continuous

Table 4: Qualitative examples of semantic canonicalization. The Query Unifier maps dataset-specific query styles (questions, fragments, tense/aspect mismatches) into a standardized declarative format used for joint training.

Dataset	# Videos	# Queries	Avg. Video Length (s)	Avg. Seg. Length (s)	Domain
Stage-I: Pretraining Datasets
NaQ [ 40 ]	5,018	858,350	416	9.7	Open (Ego)
Momentor [ 38 ]	9,697	250,738	408	3.64	Open
COIN [ 48 ]	7,938	31,251	146	14.9	Open
YouCook2 [ 65 ]	1,560	12,066	316	19.6	Cooking
HiREST [ 57 ]	583	4,414	268	19.2	Open

Table 5: Statistics of the video temporal grounding datasets used in this work. We partition the datasets into our large-scale Stage-I pretraining corpus (top) and the five Stage-II target benchmarks used for joint fine-tuning (bottom). Notably, our Stage-I corpus aggregates over 1M query–segment pairs, providing an order of magnitude more temporal supervision than the combined Stage-II training datasets.

Method	Shared Weights	Long-Form Videos									Short-Form Videos
		GoalStep			Ego4D-NLQ			TACoS			Cha.-STA			ANet-Cap.
		0.3	0.5	Avg.	0.3	0.5	Avg.	0.3	0.5	Avg.	0.5	0.7	Avg.	0.5	0.7	Avg.
Dedicated VTG Models
2D-TAN [ 63 ]	$\square$	–	–	–	5.04	2.05	3.55	45.61	35.77	40.69	56.64	36.21	46.43	46.16	29.21	37.69
M-DETR [ 25 ]	$\square$	–	–	–	8.23	5.01	6.62	37.97	24.67	31.32	52.07	30.59	41.33	–	–	–
BAM-DETR [ 24 ]	$\square$	–	–	–	–	–	–	56.69	41.54	49.12	59.95	39.38	49.67	–	–	–

Table 6: Comparison across five VTG benchmarks. R@1 is reported at multiple IoU thresholds with the average shown in each block. Models above UniversalVTG are dedicated VTG architectures; models below are MLLM-based approaches. “–” indicates results not reported by the original authors (thus unavailable for direct comparison). Models marked with

\blacksquare

utilize shared weights across all benchmarks, whereas

\square

denotes models requiring separate weights per dataset.

Method	Params (B)	TFLOPs	Runtime (s)
UniTime-7B [ 26 ]	7.61	579	37.6
UniversalVTG	0.06	0.0865	0.0886

Table 7: Grounding-head efficiency comparison. We report parameter count, computational cost (TFLOPs), and inference runtime for the grounding module, measured on a 500 s video (average Ego4D-NLQ length) with a single NVIDIA A40 GPU. 3 3 3 Query unification is a one-time offline step excluded from VTG inference. Using Qwen3-4B [ 55 ] on the same A40, unifying a query takes 0.34 s (0.136 TFLOPs). See Supp. for efficiency trade-offs.

Unifier		Pre- train	GoalStep- StepGrounding	Ego4D- NLQ	TACoS	Charades- STA	ActivityNet- Captions
T	I	Pre- train	GoalStep- StepGrounding	Ego4D- NLQ	TACoS	Charades- STA	ActivityNet- Captions
$\times$	$\times$	$\times$	22.22	20.16	55.16	61.68	38.11
$\times$	✓	$\times$	20.39	17.21	54.54	59.91	35.57
✓	✓	$\times$	23.10	21.14	58.76	63.25	40.14
✓	✓	✓	24.44	24.81	58.91	65.15	39.39

Table 8: Ablation study of the Query Unifier and Large-Scale Pretraining. We evaluate the impact of mapping queries to a unified semantic space during training (T) and inference (I). All values are reported as Avg. R@1.

Method	Unifier	GoalStep	NLQ	TACoS	Cha.- STA	Anet- Cap.	Unifier Compute
							Runtime (s)		TFLOPs
							Prefill	Decode	TFLOPs
Naive Joint Training	–	22.22	20.16	55.16	61.68	38.11	–	–	–
Naive Joint Training	GPT5	20.39	17.21	54.54	59.91	35.57	–	–	–
UniversalVTG	$\times$	22.69	21.05	55.55	63.03	37.95	–	–	–
	GPT5	24.44	24.81	58.91	65.15	39.39	–	–	–

Table S1 : Performance and unifier compute metrics across multiple datasets. Consistent with the main manuscript and following established conventions, performance is reported as Average Recall@1 at IoU thresholds of 0.3 and 0.5 for long-form video datasets (GoalStep [ 47 ] , NLQ [ 16 ] , TACoS [ 42 ] ), and at 0.5 and 0.7 for short-form datasets (Charades [ 44 ] , Activitynet [ 23 ] ).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Parameter Count (B) ( $\downarrow$ )		TFLOPs ( $\downarrow$ )			Runtime (s) ( $\downarrow$ )
Method	Vis. Enc.	Grounder	Feat. Ext.	Grounding	Total	Feat. Ext.	Grounding	Total
8.3-Minute Video
UniTime [ 26 ]	0.68	7.61	2184	579	2763	72.1	35.4	107.6
UniversalVTG	0.32	0.06	175	0.0865	175	11.1	0.0886	11.2
15.0-Minute Video
UniTime [ 26 ]	0.68	7.61	5384	588	5972	154.6	80.2	234.8

Appendix

Table S2 : Efficiency comparison of parameter count, TFLOPs, and runtime across different video lengths. Vis. Enc.: Visual Encoder; Feat. Ext.: Feature Extraction.

Method

Shared Weights

Long-Form Videos

Short-Form Videos

GoalStep

Ego4D-NLQ

TACoS

Cha.-STA

ANet-Cap.

0.3

0.5

Avg.

0.3

0.5

Avg.

0.3

0.5

Avg.

0.5

0.7

Avg.

0.5

0.7

Avg.

2D-TAN [ 63 ]

\square

–

12.89

5.88

9.39

69.11

57.31

63.21

89.14

61.13

75.14

78.80

60.85

69.83

M-DETR [ 25 ]

\square

–

23.23

13.37

18.30

–

CONE [ 19 ]

\square

–

30.33

18.02

24.18

–

UnLoc-L [ 54 ]

\square

–

88.20

61.10

74.65

79.20

61.30

70.25

Appendix

Table S4 : Comparison across five VTG benchmarks for Recall@5 (R@5). R@5 is reported at multiple IoU thresholds with the average shown in each block. “–” indicates results not available or not applicable. Models marked with

\blacksquare

utilize shared weights across all benchmarks (a single foundational model), whereas

\square

denotes models requiring separate weights fine-tuned per dataset.

Panel of Figure 14 — Figure S2 : Qualitative results of failure cases.

Figure S3 : Qualitative grounding results across five diverse VTG benchmarks. UniversalVTG accurately localizes temporal segments across varying camera perspectives, video durations, and linguistic query styles using a single unified model. Ground-truth (GT) and predicted segments (Ours) are denoted for each video.

Temporal Video Grounding

Temporal video grounding locates the moments in a video that match a natural-language query, enabling precise search and evidence-based understanding.

See all 159 papers

OmniVTG: A Large-Scale Dataset and Training Paradigm for Open-World Video Temporal Grounding
Apr 28, 2026Minghang Zheng, Zihao Yin, Yi Yang +2Temporal Video Grounding LLM Self-Correction
Wangxuan Institute of Computer Technology, Peking University · Central Media Technology Institute, Huawei Technologies Ltd. · State Key Laboratory of General Artificial Intelligence, Peking University +1
GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding
Apr 2, 2026Rong Fan, Kaiyan Xiao, Minghao Zhu +3Temporal Video Grounding Video-Language Models
Newcapec AI Research · Fudan University · Tongji University
DSE-VTG: Dual-Side Enhancement for Training-Free Video Temporal Grounding
Sep 8, 2026Zhuo Cao, Bingqing Zhang, Sen Wang +1Temporal Video Grounding Test-Time Adaptation of VLMs
The University of Queensland, Australia

Apr 28, 2026J/K move · Enter open · S save

OmniVTG: A Large-Scale Dataset and Training Paradigm for Open-World Video Temporal Grounding

Minghang Zheng, Zihao Yin, Yi Yang, Yuxin Peng+1

Wangxuan Institute of Computer Technology, Peking University · Central Media Technology Institute, Huawei Technologies Ltd. · State Key Laboratory of General Artificial Intelligence, Peking University+1

Temporal Video Grounding LLM Self-Correction LLM Grounding

Open paper AskPDF arXiv

Video Temporal Grounding (VTG), the task of localizing video segments from text queries, struggles in open-world settings due to limited dataset scale and semantic diversity, causing performance gaps between common and rare concepts. To overcome these limitations, we introduce OmniVTG, a new large-scale dataset for open-world VTG, coupled with a Self-Correction Chain-of-Thought (CoT) training paradigm designed to enhance the grounding capabilities of Multimodal Large Language Models (MLLMs). Our OmniVTG is constructed via a novel Semantic Coverage Iterative Expansion pipeline, which first identifies gaps in the vocabulary of existing datasets and collects videos that are highly likely to contain these target concepts. For high-quality annotation, we leverage the insight that modern MLLMs excel at dense captioning more than direct grounding and design a caption-centric data engine to prompt MLLMs to generate dense, timestamped descriptions. Beyond the dataset, we observe that simple supervised finetuning (SFT) is insufficient, as a performance gap between rare and common concepts still persists. We find that MLLMs' video understanding ability significantly surpasses their direct grounding ability. Based on this, we propose a Self-Correction Chain-of-Thought (CoT) training paradigm. We train the MLLM to first predict, then use its understanding capabilities to reflect on and refine its own predictions. This capability is instilled via a three-stage pipeline of SFT, CoT finetuning, and reinforcement learning. Extensive experiments show our approach not only excels at open-world grounding in our OmniVTG dataset but also achieves state-of-the-art zero-shot performance on four existing VTG benchmarks. Code is available at https://github.com/oceanflowlab/OmniVTG.

Abstract

Temporal Video Grounding

Explore similar work

OmniVTG: A Large-Scale Dataset and Training Paradigm for Open-World Video Temporal Grounding

GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding

DSE-VTG: Dual-Side Enhancement for Training-Free Video Temporal Grounding