SlotVTG: Object-Centric Adapter for Generalizable Video Temporal Grounding

Jiwook Han; Geo Ahn; Youngrae Kim; Jinwoo Choi

cs.CVMar 26, 2026

SlotVTG: Object-Centric Adapter for Generalizable Video Temporal Grounding

Authors: Jiwook Han, Geo Ahn, Youngrae Kim, Jinwoo Choi

Organizations: Kyung Hee University · University of Southern California

Abstract

Multimodal Large Language Models (MLLMs) have shown strong performance on Video Temporal Grounding (VTG). However, their coarse recognition capabilities are insufficient for fine-grained temporal understanding, making task-specific fine-tuning indispensable. This fine-tuning causes models to memorize dataset-specific shortcuts rather than faithfully grounding in the actual visual content, leading to poor Out-of-Domain (OOD) generalization. Object-centric learning offers a promising remedy by decomposing scenes into entity-level representations, but existing approaches require re-running the entire multi-stage training pipeline from scratch. We propose SlotVTG, a framework that steers MLLMs toward object-centric, input-grounded visual reasoning at minimal cost. SlotVTG introduces a lightweight slot adapter that decomposes visual tokens into abstract slots via slot attention and reconstructs the original sequence, where objectness priors from a self-supervised vision model encourage semantically coherent slot formation. Cross-domain evaluation on standard VTG benchmarks demonstrates that our approach significantly improves OOD robustness while maintaining competitive In-Domain (ID) performance with minimal overhead.

Figures & tables

Figure 2: Observations. We naively fine-tune Qwen2.5-VL-3B [ 2 ] on Charades-STA (Cha.) [ 11 ] (source) and evaluate on QVHighlights (QVH) [ 25 ] (target). (a) ID vs. OOD performance. The model achieves 63.4 [email protected] on ID but drops to 43.6 on OOD, confirming severe overfitting to dataset-specific patterns. (b) Visual similarity analysis. We extract visual features from the vision encoder and compute cosine similarity between ID and OOD samples; performance on the most similar 20% of OOD samples (52.8) far exceeds the most dissimilar 20% (39.1), indicating that the model fails when visual distribution shifts. (c) Noise perturbation. We report [email protected] for a stricter localization threshold. On ID, ground-truth perturbation causes a 17.4% drop while random perturbation causes only 9.6%, a significant gap confirming the model attends to the target moment. On OOD, however, the two cause nearly identical drops (12.6% vs. 12.1%), revealing that the model does not attend to the actual visual content under distribution shift. (d) Domain gap. MMD distance [ 12 ] of our slot-based representations (0.097) is substantially lower than the baseline (0.192), showing that object-centric decomposition reduces the domain gap between source and target distributions.

Figure 3: (a) Overview of SlotVTG. Video frames are encoded into visual tokens and projected into the LLM decoder. In the early decoder layers, a lightweight Slot Adapter decomposes visual tokens into entity-level slots via iterative slot attention, then reconstructs the token sequence. The resulting tokens carry disentangled, entity-aware representations before entering the later layers, which are fine-tuned with LoRA for temporal reasoning and answer generation. Text tokens bypass the Slot Adapter throughout. (b) Slot Alignment Loss. Token-pair similarity derived from slot attention weights is aligned with that from a pre-trained DINOv2 model, encouraging semantically coherent tokens to be grouped into the same slot.

Source dataset	Method	LLM size	Target dataset
			Charades-STA				ActivityNet-Captions				QVHighlights
			[email protected]	[email protected]	[email protected]	mIoU	[email protected]	[email protected]	[email protected]	mIoU	[email protected]	[email protected]	[email protected]	mIoU
Zero-Shot	HawkEye [ 46 ]	7B	50.6	31.4	14.5	33.7	49.1	29.3	10.7	32.7	-	-	-	-
	TimeSuite [ 55 ]	7B	69.9	48.7	24.0	-	-	16.6	9.3	22.0	-	12.3	9.2	21.3
	UniTime [ 29 ]	7B	-	59.1	31.9	52.2	-	22.8	14.1	27.3	-	41.0	31.5	43.7
	VideoMind [ 31 ]	2B	67.6	51.1	26.0	45.2	44.0	26.5	12.6	30.1	-	-	-	-

Table 1: Performance comparison on video temporal grounding benchmarks. We evaluate SlotVTG against state-of-the-art models on Charades-STA [ 11 ] , QVHighlights [ 25 ] , and ActivityNet Captions [ 22 ] . We report both In-Domain (ID) settings, where the source and target datasets are the same, and Out-of-Distribution (OOD) settings, where they differ. DETR-based methods (EATR [ 18 ] and CG-DETR [ 35 ] ) are reproduced using pre-extracted CLIP [ 41 ] + SlowFast [ 10 ] features at 0.5 fps, following their original implementation. The performance of zero-shot VTG models is reported for reference. Our results are highlighted in green. The best results under the same cross-domain evaluation setting (source

\rightarrow

target, LLM size) are highlighted in bold .

Figure 4: Slot attention visualization. We visualize the slot assignments on samples from Cha. (ID), QVH. (OOD), and ANet (OOD) by masking each frame with its highest-attending slot. Each column corresponds to one of the four learned slots.

Adapter Type	Cha. (ID)		ANet (OOD)
Adapter Type	[email protected]	[email protected]	[email protected]	[email protected]
LoRA [ 16 ]	63.4	40.3	26.3	13.1
Adapter w/ self attention	63.5	40.9	26.5	13.7
Slot Adapter	64.0	41.2	28.7	14.4

Table 2: Ablation study. To validate the effect of each component in SlotVTG, we show the results on the Cha.

\rightarrow

ANet setting. ‘Cha.’ and ‘ANet’ denote Charades-STA [ 11 ] and ActivityNet Captions [ 22 ] . We use Qwen2.5-VL [ 2 ] 3B as a backbone MLLM. We report [email protected] and [email protected] scores on both ID and OOD settings. The best numbers are highlighted .

Temporal Video Grounding

Temporal video grounding locates the moments in a video that match a natural-language query, enabling precise search and evidence-based understanding.

See all 166 papers

EVIDENT: Routing MLLM Adaptation through Entity-Grounded Visual Evidence for Cross-Domain Video Temporal Grounding
May 25, 2026Geo Ahn, Jiwook Han, Youngrae Kim +2Temporal Video Grounding VLM Robustness
Seoul National University
OmniVTG: A Large-Scale Dataset and Training Paradigm for Open-World Video Temporal Grounding
Apr 28, 2026Minghang Zheng, Zihao Yin, Yi Yang +2Temporal Video Grounding LLM Self-Correction
Wangxuan Institute of Computer Technology, Peking University · Central Media Technology Institute, Huawei Technologies Ltd. · State Key Laboratory of General Artificial Intelligence, Peking University +1
GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding
Apr 2, 2026Rong Fan, Kaiyan Xiao, Minghao Zhu +3Temporal Video Grounding Video-Language Models
Newcapec AI Research · Fudan University · Tongji University

May 25, 2026J/K move · Enter open · S save

EVIDENT: Routing MLLM Adaptation through Entity-Grounded Visual Evidence for Cross-Domain Video Temporal Grounding

Geo Ahn, Jiwook Han, Youngrae Kim, Joonseok Lee+1

Seoul National University

Temporal Video Grounding VLM Robustness VLM Adaptation

Open paper AskPDF arXiv

Fine-tuning MLLMs for Video Temporal Grounding (VTG) often improves in-domain performance but degrades sharply under domain shift. In this work, we find that this failure is primarily driven not just by unseen query concepts, but by visual domain shift, which prevents the model from coupling its learned temporal localization knowledge with its inherent entity-attention capability. To address this, we introduce EVIDENT, a parameter-efficient adaptation framework that anchors temporal grounding in the inherent entity-attention of pre-trained MLLMs by routing VTG adaptation through explicit visual entity evidence. EVIDENT consists of three components: (i) an Entity Bottleneck Adapter that transforms dense visual tokens into compact entity-level slots, (ii) an Entity-Binding Distillation loss that instills objectness priors into the semantically unstructured MLLM visual space, guiding each slot to bind to a coherent entity, and (iii) an Entity-to-eVidence gating mechanism that leverages the captured entities as evidence, steering the model to localize moments containing query-relevant entities. Together, these components enable VTG fine-tuning to rely on entity-grounded evidence rather than brittle dataset shortcuts. Experiments on cross-domain VTG benchmarks show that EVIDENT consistently improves out-of-domain robustness while preserving competitive in-domain performance with modest parameter overhead. These results suggest that entity-level grounding is an effective inductive bias for generalizable temporal localization.

Abstract

Temporal Video Grounding

Explore similar work

EVIDENT: Routing MLLM Adaptation through Entity-Grounded Visual Evidence for Cross-Domain Video Temporal Grounding

OmniVTG: A Large-Scale Dataset and Training Paradigm for Open-World Video Temporal Grounding

GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding