How Should Video LLMs Output Time? An Analysis of Efficient Temporal Grounding Paradigms

Shengji Jin; Yuanhao Zou; Victor Zhu; Zhengping Ji; Chen Chen

cs.CVApr 10, 2026

How Should Video LLMs Output Time? An Analysis of Efficient Temporal Grounding Paradigms

Authors: Shengji Jin, Yuanhao Zou, Victor Zhu, Zhengping Ji, Chen Chen

Organizations: The University of Central Florida · Axon

Abstract

While Multimodal Large Language Models (MLLMs) have advanced Video Temporal Grounding (VTG), existing methods often couple output paradigms with different backbones, datasets, and training protocols. This makes it challenging to isolate the specific impact of the output design. Additionally, as VTG systems are increasingly considered for resource-constrained edge deployment, the trade-off between output formulation and system-level efficiency requires systematic investigation. In this paper, we present a controlled empirical study comparing three dominant VTG output paradigms: Text Numeral Generation, Temporal Token Generation, and Continuous Temporal Decoding. We evaluate these paradigms across identical compact VLMs (SmolVLM2, FastVLM, and Molmo2) using consistent datasets and LoRA fine-tuning protocols. Evaluations on Charades-STA, QVHighlights, and YouCook2 measure both localization accuracy and system efficiency, including inference latency, training throughput, and parameter overhead. Our results demonstrate that the choice of output formulation significantly affects both grounding accuracy and computational cost, independent of model scale. Specifically, the continuous distribution paradigm consistently achieves the most favorable efficiency-accuracy trade-off on the Pareto frontier, delivering robust localization with minimal latency overhead. These findings provide objective empirical guidelines for designing efficient, deployment-ready VTG systems.

Figures & tables

Figure 1: Overview of three output formulation paradigms for temporal grounding in Video LLMs. After extracting features through shared M-LLM Layers, the architecture diverges into three distinct designs: (a) Text Numeral Generation utilizes the standard M-LLM head to directly generate timestamps as sequences of regular text tokens. (b) Temporal Token Generation introduces specific temporal tokens and a dedicated temporal head, utilizing a restricted vocabulary library for controlled, sequential boundary generation. (c) Continuous Temporal Decoding maps hidden states through a lightweight MLP head to predict continuous temporal distributions, directly decoding boundary coordinates without auto-regressive generation.

Paradigm	Method	LLM Backbone	Visual Encoder	Data Size	Stages	Architectural Additions / Key Designs
Text Numeral	HawkEye [ 29 ]	Vicuna v1.5 (7B)	CLIP ViT-L/14	1.1M	3-S	Recursive search space narrowing
	SlowFocus [ 20 ]	Vicuna v1.5 (7B)	CLIP ViT-L/14	1.0M	3-S	Mixed-frequency sampling mechanism
	VTimeLLM [ 8 ]	Vicuna v1.5 (7B)	CLIP ViT-L/14	170K	3-S	None (Textual boundary format)
	TimeChat [ 23 ]	LLaMA-2 (7B)	EVA-CLIP ViT-G/14	125K	1-S	Sliding video Q-Former
	GroundingGPT [ 14 ]	Vicuna v1.5 (7B)	CLIP ViT-L/14	1.4M	3-S	Modality adapters
	Chrono [ 18 ]	BLIP-2 (OPT-2.7B)	BLIP-2 ViT-g/14	$\sim$ 12K	1-S	None (Input sequence blueprint)

Table 1: Detailed comparison of existing VTG-MLLM systems. To highlight the extreme heterogeneity, we categorize the setups by LLM backbone, visual encoder, data scale, and optimization pipeline. 1-S , 2-S , and 3-S denote 1-stage, 2-stage, and 3-stage training protocols, respectively.

		Charades-STA			QVHighlights					YouCook2
		MR			MR			HD		DVC
Backbone	Para.	[email protected]	[email protected]	mIoU	[email protected]	[email protected]	mIoU	mAP	HIT@1	CIDEr	SODA_c	F1
SmolVLM2-0.5B	Text	15.1	6.8	16.2	9.0	5.6	12.9	–	–	0.9	0.5	5.9
	Cont.	41.8	22.5	43.4	46.9	31.0	50.4	–	–	1.4	1.0	9.0
	Gen.	20.1	9.5	21.9	14.7	9.6	18.6	20.0	21.8	1.2	0.9	7.9
FastVLM-1.5B	Text	19.5	8.9	19.9	11.7	7.6	16.5	–	–	1.5	0.8	7.5

Table 2: Quantitative comparison of three temporal output paradigms across diverse video grounding benchmarks. Task abbreviations: MR (Moment Retrieval), HD (Highlight Detection), and DVC (Dense Video Captioning). Both mAP and HIT@1 metrics on QVHighlights require native confidence or saliency score outputs for ranking. Because the Text and Cont. paradigms inherently lack such scoring mechanisms, these metrics are exclusively evaluated on the Gen. paradigm via its dedicated ScoreTower. Variance across 3 random seeds for SmolVLM2-2.2B and Molmo2-4B is reported in Table S5 (Appendix).

Paradigm	Method (Backbone)	[email protected]	mIoU
Text Numeral	VTimeLLM (Vicuna-7B)	27.5	31.2
	GroundingGPT (Vicuna-7B)	29.6	–
	TimeChat (LLaMA-2-7B)	32.2	–
	Chrono-GPT (GPT-4o)	28.8	33.0
	HawkEye (Vicuna-7B)	31.4	33.7
Temporal Token	Momentor (LLaMA-7B)	26.6	28.5

Table 3: Reference comparison with published zero-shot results on Charades-STA. † Our controlled setting on FastVLM-1.5B.

Figure 2: Qualitative comparison on Charades-STA (SmolVLM2-2.2B) across three difficulty levels. Each column shows sampled video frames and temporal predictions from all paradigms against the ground truth. (a) Simple action; (b) Temporally ambiguous query; (c) Compound action leading to collective failure.

Figure 3: (a) Scaling behavior: mIoU vs. backbone size for each paradigm. (b) Efficiency-accuracy Pareto frontier across all paradigm-backbone combinations. Both evaluated on Charades-STA.

Backbone	Paradigm	Extra Params	Throu. (samp/s) $\uparrow$	Mem. (GB) $\downarrow$
SmolVLM2-0.5B	Text	0	4.07	1.8
	Cont.	3.8M	4.09	1.8
	Gen.	52K	4.12	1.9
SmolVLM2-2.2B	Text	0	4.03	5.5
	Cont.	17.1M	4.36	5.8
	Gen.	111K	4.41	5.9

Table 4: Training efficiency profile (single H200 GPU, batch size 1). Extra Params accounts only for paradigm-specific heads.

Figure 4: Ablation studies evaluated on Charades-STA. (a) Context length robustness evaluated with varying sampled video frames (from 8 to 64). (b) Data efficiency evaluated under varying percentages of the training corpus (25% to 100%). Both studies use the SmolVLM2-2.2B backbone.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Samples	Videos
InternVid [ 27 ]	608K	87K
YTTemporal [ 36 ]	278K	21K
Valley [ 17 ]	229K	229K
DiDeMo [ 1 ]	33K	8K
ShareGPT4Video [ 3 ]	24K	24K
ViTT [ 10 ]	8K	5K

Appendix

Table S1 : Training data composition by source.

Backbone	Vision Enc.	LLM	$d$
SmolVLM2-0.5B	SigLIP-B/16	SmolLM2-360M	960
SmolVLM2-2.2B	SigLIP2-SO400M	SmolLM2-1.7B	2048
FastVLM-1.5B	FastViTHD	Qwen2-1.5B	1536
Molmo2-4B	SigLIP2-SO400M	Qwen3-4B	2560
Molmo2-8B	SigLIP2-SO400M	Qwen3-8B	4096

Appendix

Table S2 : Backbone-level specifications.

Shared annotation: video: split_video_7w/RP6nWBaB2K8.mp4 ground truth:

[52.0,\,63.0]

(a) Text Numeral Generation

Input: <video>\n With the input: ‘The worker cleans up excess plaster from the ground, ensuring a tidy workspace.’, identify the precise second in the video where this event occurs. Output: The event happens from 52.0 to 63.0 seconds. Supervision: standard next-token cross-entropy (no additional temporal modules)

(b) Temporal Token Generation

\underbrace{\texttt{<time>}\,\cdots\,\texttt{<time>}}_{14}

<score> The worker cleans up excess plaster from the ground, ensuring a tidy workspace. Supervision: times: [[52.0, 63.0]], scores: [[]]

\;\rightarrow\;

dedicated 13-token vocabulary

Appendix

Table S3: Training data format comparison across the three paradigms for the same moment retrieval annotation. All paradigms share identical video inputs and queries; only the output format and temporal supervision differ.

Task (Dataset)	Inference Prompt Template
Moment Retrieval (Charades-STA)	Localize the visual content described by the given textual query ’{query}’ in the video, and output the start and end timestamps in seconds.
Highlight Detection (QVHighlights)	Please find the highlight contents in the video described by a sentence query, determining the highlight timestamps and its saliency score on a scale from 1 to 5. Now I will give you the sentence query: ’{query}’. Please return the query-based highlight timestamps and salient scores.
Dense Video Captioning (YouCook2)	Scrutinize the video and determine multiple occurrences, providing their initial and final timestamps as well as a summary of each action.

Appendix

Table S4: Exact inference prompt templates utilized for evaluating the models across three temporal grounding tasks. All paradigms receive the identical text prompt alongside the visual input to ensure fair comparison.

		Charades-STA	QVHighlights
Backbone	Para.	mIoU	mIoU
SmolVLM2-2.2B	Text	20.6 $\pm$ 0.4	16.3 $\pm$ 0.3
	Cont.	46.8 $\pm$ 0.3	54.2 $\pm$ 0.4
	Gen.	27.9 $\pm$ 0.5	23.7 $\pm$ 0.4
Molmo2-4B	Text	24.1 $\pm$ 0.3	18.8 $\pm$ 0.4
	Cont.	56.1 $\pm$ 0.2	59.8 $\pm$ 0.3

Appendix

Table S5 : Variance analysis across 3 random seeds on two representative backbones. We report mean

\pm

std of mIoU. The consistently small standard deviations confirm that the observed paradigm gaps are robust and not attributable to stochastic training variation.

Method	Backbone	Charades-STA			QVHighlights		YouCook2
		(Moment Retrieval)			(Highlight Detection)		(Dense Video Captioning)
		[email protected]	[email protected]	mIoU	mAP	HIT@1	CIDEr	SODA c	F1
Text Numeral Generation
TimeChat [ 23 ]	LLaMA-2-7B	32.2	13.4	–	14.5	23.9	1.2	3.4	12.6
VTimeLLM [ 8 ]	Vicuna-7B	27.5	11.4	31.2	–	–	3.4	0.9	–
GroundingGPT [ 14 ]	Vicuna-7B	29.6	11.9	–	–	–	–	–	–

Appendix

Table S6: Comprehensive performance leaderboard of existing VTG-MLLMs. Metrics represent zero-shot performance reported in their respective papers or official repositories. ”-” indicates the absolute metric was not reported or evaluated by the original authors. Methods are grouped by their temporal output paradigm to facilitate direct comparison.

Figure S1 : Quantitative error taxonomy across representative paradigms on Charades-STA. The composition explicitly breaks down the failure cases (where IoU

<0.5

) into three types. The Continuous paradigm drastically shifts the error mode from severe hallucinations to minor boundary jitters.

Figure S2 : Qualitative comparisons focusing on Type A (Temporal Hallucination) errors. Green bars denote Ground Truth. (Top) DisTime successfully localizes the event, whereas both discrete paradigms (TRACE and VTimeLLM) completely hallucinate early timeframes, demonstrating the continuous paradigm’s robustness against blind guessing. (Middle) DisTime and TRACE accurately capture the action, leaving VTimeLLM as the sole model exhibiting a severe Type A hallucination (predicting the very end of the video). This underscores the text-numeral paradigm’s extreme susceptibility to temporal hallucinations. (Bottom) A universally challenging query where all paradigms fail via hallucination, predicting disjoint early segments. This highlights that while continuous decoding mitigates hallucinations, extreme causal complexity can still trigger Type A errors across all architectures.

Figure S3 : Qualitative comparisons focusing on Type B (Boundary Jitter) errors. (Top) DisTime accurately covers the true temporal window. TRACE, however, exhibits massive “Token Bloat” (a severe Type B error spanning almost the entire video), while VTimeLLM hallucinates entirely (Type A). (Middle) Both DisTime and TRACE successfully localize the action. VTimeLLM identifies the general vicinity but severely truncates the boundary (7.0-8.0s vs. GT 0.7-9.8s), yielding a failing IoU of 0.11. (Bottom) A scenario characterized by ambiguous action boundaries where all models suffer from Type B errors. Notably, DisTime and TRACE regress overly bloated durations (over-extension), whereas the text-based VTimeLLM predicts an extremely narrow window (truncation), reflecting their distinct mechanisms for handling temporal uncertainty.

Figure S4 : Qualitative comparisons focusing on Type C (Semantic Confusion) errors. (Top) DisTime perfectly localizes “opening a cabinet”. In contrast, both TRACE and VTimeLLM confuse this action with a semantically similar but chronologically later event, resulting in disjoint predictions (Type C). (Middle) A highly deceptive case where all three paradigms are misled by earlier visual cues of a person eating, completely missing the ground-truth timeframe. This represents a universal semantic failure across current VTG-MLLMs. (Bottom) A complex failure case demonstrating diverse error manifestations. DisTime successfully captures the event but fails due to benign boundary jitter ( Type B , IoU=0.21). Meanwhile, TRACE predicts a semantically incorrect early action ( Type C ), and VTimeLLM hallucinates entirely off-target ( Type A ). This perfectly visually encapsulates our taxonomy findings.

Temporal Video Grounding

Temporal video grounding locates the moments in a video that match a natural-language query, enabling precise search and evidence-based understanding.

See all 166 papers

GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding
Apr 2, 2026Rong Fan, Kaiyan Xiao, Minghao Zhu +3Temporal Video Grounding Video-Language Models
Newcapec AI Research · Fudan University · Tongji University
MLLMs Know When Before Speaking: Revealing and Recovering Temporal Grounding via Attention Cues
May 21, 2026Dazhao Du, Liao Duan, Jian Liu +5Temporal Video Grounding Multimodal Large Language Models
Hong Kong University of Science and Technology · Tencent · Xi’an Jiaotong University
Foresee-to-Ground: From Predictive Temporal Perception to Evidence-Driven Reasoning for Video Temporal Grounding
May 21, 2026Zelin Zheng, Xinyan Liu, Ruixin Li +4Temporal Video Grounding Temporal Representation Learning
School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China · Beijing Key Laboratory of Embodied Intelligence Computing, Beijing, China · Faculty of Computing, Harbin Institute of Technology, Weihai, China +1

Apr 2, 2026J/K move · Enter open · S save

GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding

Rong Fan, Kaiyan Xiao, Minghao Zhu, Liuyi Wang+2

Newcapec AI Research · Fudan University · Tongji University

Temporal Video Grounding Video-Language Models Query-Aware Video Frame Selection Visual Token Pruning

Open paper AskPDF arXiv

Video temporal grounding (VTG) is a critical task in video understanding and a key capability for extending video large language models (Vid-LLMs) to broader applications. However, existing Vid-LLMs rely on uniform frame sampling to extract video information, resulting in a sparse distribution of key frames and the loss of crucial temporal cues. To address this limitation, we propose Grounded Visual Token Sampling (GroundVTS), a Vid-LLM architecture that focuses on the most informative temporal segments. GroundVTS employs a fine-grained, query-guided mechanism to filter visual tokens before feeding them into the LLM, thereby preserving essential spatio-temporal information and maintaining temporal coherence. Futhermore, we introduce a progressive optimization strategy that enables the LLM to effectively adapt to the non-uniform distribution of visual features, enhancing its ability to model temporal dependencies and achieve precise video localization. We comprehensively evaluate GroundVTS on three standard VTG benchmarks, where it outperforms existing methods, achieving a 7.7-point improvement in mIoU for moment retrieval and 12.0-point improvement in mAP for highlight detection. Code is available at https://github.com/Florence365/GroundVTS.

Abstract

Temporal Video Grounding

Explore similar work

GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding

MLLMs Know When Before Speaking: Revealing and Recovering Temporal Grounding via Attention Cues

Foresee-to-Ground: From Predictive Temporal Perception to Evidence-Driven Reasoning for Video Temporal Grounding