SuperClip Pyramid for Video Temporal Grounding

Sanghyeok Lee; Juyeon Ko; Joonmyung Choi; Jongha Kim; Dohwan Ko; Ji Soo Lee; Hyunwoo J. Kim

cs.CVOct 17, 2026

SuperClip Pyramid for Video Temporal Grounding

Authors: Sanghyeok Lee, Juyeon Ko, Joonmyung Choi, Jongha Kim, Dohwan Ko, Ji Soo Lee, Hyunwoo J. Kim

Organizations: KAIST · TwelveLabs · KT · Korea Univertsity

Abstract

With the rapid expansion of online video content, video temporal grounding has become essential for localizing moments based on user-specific queries, comprising two subtasks: moment retrieval (MR) and highlight detection (HD). While recent approaches adopt DETR-style decoders or temporal pyramid architectures, they rely on single-scale feature maps or uniformly downsampled representations, limiting their ability to capture semantic boundaries and variable-length events. In this paper, we present SuperClip Video Temporal Grounding (SCVTG) built on a multi-scale temporal hierarchy that better captures distinctive events at various granularities. SCVTG constructs a SuperClip pyramid by grouping adjacent clips into semantically coherent segments, termed SuperClips. Further, to aggregate these SuperClips having variable duration and temporal location, we introduce a temporally adaptive kernel that aggregates SuperClips through temporally conditioned weights, enabling flexible aggregation. On top of this, we design relative saliency losses to guide the model in learning inter-clip importance, enhancing its ability to distinguish relevant events. Extensive experiments show that SCVTG achieves the best performance across five video temporal grounding benchmarks with promising gains.

Figures & tables

Figure 1: Performance comparison. SCVTG outperforms all baseline models, demonstrating its effectiveness in video temporal grounding. For moment retrieval (MR) and highlight detection (HD), we use R1@0.7 and mAP@Avg, respectively. — Figure 1: Performance comparison. SCVTG outperforms all baseline models, demonstrating its effectiveness in video temporal grounding. For moment retrieval (MR) and highlight detection (HD), we use [email protected] and mAP@Avg, respectively.

Figure 2: Comparison between (a) conventional temporal pyramid and (b) SuperClip pyramid. Unlike conventional temporal pyramids that segment videos at uniform intervals, our SuperClip pyramid adaptively groups adjacent clips into semantically coherent segments.

Figure 3: Overall architecture of SCVTG. After multi-modal fusion, video clip features are progressively grouped and aggregated to form a SuperClip pyramid with varying temporal granularity. Given this multi-level hierarchy capturing distinctive events, task-specific heads predict both foreground moments and clip-wise saliency scores.

Figure 4: Illustration of SuperClip grouping and aggregation. (a) Given the input clips, a non-anchor clip is assigned to one of its adjacent anchor clips based on semantic similarity, forming coherent SuperClips. (b) Next, each clip is projected using its corresponding weight generated by the temporally adaptive kernel ( TA-Kernel ), followed by pooling and a linear layer. (c) Unlike fixed convolutional kernels, which cannot handle varying lengths or temporal characteristics, we generate TA-kernel weights for SuperClips, considering the temporal information ( $\phi(f_{i})$ ) of its constituent clips.

Method	MR Val					HD Val		MR Test					HD Test
	R1		mAP			Hit	mAP	R1		mAP			Hit	mAP
	@0.5	@0.7	@0.5	@0.75	Avg	@1	Avg	@0.5	@0.7	@0.5	@0.75	Avg	@1	Avg
MomentDETR [ 18 ]	53.94	34.84	-	-	32.20	55.55	35.65	52.89	33.02	54.82	29.40	30.73	55.60	35.69
UniVTG [ 20 ]	59.74	-	-	-	36.13	61.81	38.83	58.86	40.86	57.60	35.59	35.47	60.96	38.20
UMT [ 30 ]	60.26	44.26	56.70	39.90	38.59	64.19	39.85	56.23	41.18	53.83	37.01	36.12	59.99	38.18
MomentDiff [ 19 ]	-	-	-	-	-	-	-	58.21	41.48	54.57	37.21	36.84	–	–

Table 1: Video moment retrieval and highlight detection results on QVHighlights [ 18 ] .

Method	TVSum [ 43 ]											YouTube-HL [ 45 ]
Method	VT	VU	GA	MS	PK	PR	FM	BK	BT	DS	Avg.	Dog	Gym.	Par.	Ska.	Ski.	Sur.	Avg.
LIM-S [ 53 ]	55.9	42.9	61.2	54.0	60.4	47.5	43.2	66.3	69.1	62.6	56.3	57.9	41.7	67.0	57.8	48.6	65.1	56.4
SL-Module [ 55 ]	86.5	68.7	74.9	86.2	79.0	63.2	58.9	72.6	78.9	64.0	73.3	70.8	53.2	77.2	72.5	66.1	76.2	69.3
Joint-VA [ 2 ]	83.7	57.3	78.5	86.1	80.1	69.2	70.0	73.0	97.4	67.5	76.3	64.5	71.9	80.8	62.0	73.2	78.3	71.8
UniVTG [ 20 ]	83.9	85.1	89.0	80.1	84.6	81.4	70.9	91.7	73.5	69.3	81.0	71.8	76.5	73.9	73.3	73.2	82.2	75.2
UMT [ 30 ]	87.5	81.5	88.2	78.8	81.4	87.0	76.0	86.9	84.4	79.6	83.1	65.9	75.2	81.6	71.8	72.3	82.7	74.9

Table 2: Highlight detection on TVSum [ 43 ] and YouTube-HL [ 45 ] .

Figure 5: Qualitative results of SuperClip Pyramid. The interval indicated by the red arrow denotes the ground-truth moments. The SuperClip highlighted with a red hatched region perfectly aligns with the ground truth, serving as an ideal candidate for temporal grounding.

Method	MR		HD
Method	[email protected]	mAP	Hit@1	mAP
(a) Grouping method
Pairwise	58.58	53.83	74.00	44.71
Similarity	52.00	46.43	72.26	44.25
SuperClip	59.23	54.50	74.32	45.33
(b) Temporally adaptive kernel

Table 4: Ablation studies on SCVTG. The highlighted row indicates the default settings.

Method	MR		HD		Latency	Memory
Method	[email protected]	mAP	Hit@1	mAP	ms	M
(g) Low-rank approximation for weight basis
Original $W_{b}$	59.87	54.07	74.65	45.04	0.188	8222
Low-rank $U_{b},V_{b}$	59.23	54.50	74.32	45.33	0.112	3978

Table 5: Ablation studies on low-rank approximation.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Short	Middle	Long	All
UMT [ 30 ]	5.02	42.22	42.84	38.59
UniVTG [ 20 ]	8.64	39.90	37.82	36.13
CG-DETR [ 32 ]	10.58	48.30	49.36	44.93
LLMEPET [ 15 ]	10.58	48.30	49.36	44.93
$R^{2}$ -Tuning [ 29 ]	12.62	48.56	55.42	47.86
FlashVTG [ 3 ]	15.65	56.54	56.19	52.84

Appendix

Table H: Results according to the various moment lengths. Results are reported using mAP@Avg of moment retrieval.

# $B$	2	4	8	16	32
mAP	54.14	54.50	54.48	54.57	54.24

Appendix

Table I: Sensitivity analysis on the number of weights basis

B

and the dimension of low rank

r

in positional parameterization.

Method	SLIC [ 1 ]	SEEDS [ 47 ]	SuperClip
Latency(ms)	0.355	3.069	0.112

Appendix

Table J: Runtime comparison of grouping method.

Panel of Figure 13 — Figure F : Routing scores from different center positions across pyramid levels.

Figure H : Qualitative results of SuperClip Pyramid. The red arrows indicate the ground truth moments.

Temporal Video Grounding

Temporal video grounding locates the moments in a video that match a natural-language query, enabling precise search and evidence-based understanding.

See all 166 papers

GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding
Apr 2, 2026Rong Fan, Kaiyan Xiao, Minghao Zhu +3Temporal Video Grounding Video-Language Models
Newcapec AI Research · Fudan University · Tongji University
TimePLE: Rethinking Temporal Representation for Video Temporal Grounding
Jul 27, 2026Yuhui Zeng, Xinyu Mao, Xiaokun Liu +4Temporal Video Grounding Representation Learning
Media Analytics and Computing Lab, Xiamen University, Xiamen, China · Kling Team, Kuaishou Technology, China · The Chinese University of Hong Kong, HKSAR, China +1
Summarize Before Grounding: Query-Guided Chunk Condensation for Long-Video Temporal Grounding
Sep 28, 2026Nanxing Hu, Xiaoyue Duan, Qiwei Yan +3Temporal Video Grounding Video Token Compression
Beihang University · University of the Chinese Academy of Sciences · Institute of automation, Chinese academy of science

Apr 2, 2026J/K move · Enter open · S save

GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding

Rong Fan, Kaiyan Xiao, Minghao Zhu, Liuyi Wang+2

Newcapec AI Research · Fudan University · Tongji University

Temporal Video Grounding Video-Language Models Query-Aware Video Frame Selection Visual Token Pruning

Open paper AskPDF arXiv

Video temporal grounding (VTG) is a critical task in video understanding and a key capability for extending video large language models (Vid-LLMs) to broader applications. However, existing Vid-LLMs rely on uniform frame sampling to extract video information, resulting in a sparse distribution of key frames and the loss of crucial temporal cues. To address this limitation, we propose Grounded Visual Token Sampling (GroundVTS), a Vid-LLM architecture that focuses on the most informative temporal segments. GroundVTS employs a fine-grained, query-guided mechanism to filter visual tokens before feeding them into the LLM, thereby preserving essential spatio-temporal information and maintaining temporal coherence. Futhermore, we introduce a progressive optimization strategy that enables the LLM to effectively adapt to the non-uniform distribution of visual features, enhancing its ability to model temporal dependencies and achieve precise video localization. We comprehensively evaluate GroundVTS on three standard VTG benchmarks, where it outperforms existing methods, achieving a 7.7-point improvement in mIoU for moment retrieval and 12.0-point improvement in mAP for highlight detection. Code is available at https://github.com/Florence365/GroundVTS.

Abstract

Temporal Video Grounding

Explore similar work

GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding

TimePLE: Rethinking Temporal Representation for Video Temporal Grounding

Summarize Before Grounding: Query-Guided Chunk Condensation for Long-Video Temporal Grounding