T2SGrid: Temporal-to-Spatial Gridification for Video Temporal Grounding

Chaohong Guo; Yihan He; Yongwei Nie; Fei Ma; Xuemiao Xu; Chengjiang Long

cs.CVMar 7, 2026

T2SGrid: Temporal-to-Spatial Gridification for Video Temporal Grounding

Authors: Chaohong Guo, Yihan He, Yongwei Nie, Fei Ma, Xuemiao Xu, Chengjiang Long

Organizations: South China University of Technology. · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ). · Bytedance Inc.

Abstract

Video Temporal Grounding (VTG) aims to localize the video segment that corresponds to a natural language query, which requires a comprehensive understanding of complex temporal dynamics. Existing Vision-LMMs typically perceive temporal dynamics via positional encoding, text-based timestamps, or visual frame numbering. However, these approaches exhibit notable limitations: assigning each frame a text-based timestamp token introduces additional computational overhead and leads to sparsity in visual attention, positional encoding struggles to capture absolute temporal information, and visual frame numbering often compromises spatial detail. To address these issues, we propose Temporal to Spatial Gridification (T2SGrid), a novel framework that reformulates video temporal understanding as a spatial understanding task. The core idea of T2SGrid is to process video content in clips rather than individual frames. we employ a overlapping sliding windows mechanism to segment the video into temporal clips. Within each window, frames are arranged chronologically in a row-major order into a composite grid image, effectively transforming temporal sequences into structured 2D layouts. The gridification not only encodes temporal information but also enhances local attention within each grid. Furthermore, T2SGrid enables the use of composite text timestamps to establish global temporal awareness. Experiments on standard VTG benchmarks demonstrate that T2SGrid achieves superior performance.

Figures & tables

Figure 1: Comparison between T2SGrid and previous Vision-LLMs. (a) Traditional methods process frames sequentially or apply temporal pooling to capture information at multiple scales. However, sequential processing and pooling can cause information loss, obscure local temporal details within a window, and often require additional time encoding/label for temporal modeling. (b) Our T2SGrid method folds multiple frames within a temporal window into the spatial dimension via gridification, allowing direct processing a temporal window by the standard 2D ViT. This leverages the model’s strong spatial reasoning capability for temporal understanding.

Figure 2: Illustration of Qwen2-VL temporal reasoning on 2D grid layouts. The model correctly infers the temporal order of events (before and after) and accurately identifies the biting action in frame 6.

Figure 3: Overview of our T2SGrid framework. The original video frames are first arranged into a 2D grid in a row-major order ( gridication ) to enable spatialized temporal reasoning. A lightweight composite textual timestamp is incorporated to provide global temporal awareness. Our framework can operate in a training-free manner or be further enhanced via LoRA fine-tuning.

Figure 4: Top : spatial and temporal attention with sequential frame input. Bottom : spatial and temporal attention with grid-based input. The grid method captures dynamic actions and maintain focus on the correct temporal intervals.

VTG-Tuned Vision-LLMs
Model	Charades-STA				ActivityNet
Model	[email protected]	[email protected]	[email protected]	mIoU	[email protected]	[email protected]	[email protected]	mIoU
GroundingGPT [ 32 ]	-	29.6	11.9	-	-	-	-	-
LITA [ 20 ]	-	-	-	-	-	25.9	-	28.6
VTG-LLM [ 16 ]	52.0	33.8	15.7	-	-	-	-	-
TimeChat [ 44 ]	47.7	22.9	12.5	30.6	30.2	16.9	8.2	21.8

Table 1: Performance comparison on the video temporal grounding task with prior state-of-the-art methods. T2SGrid denotes the use of spatiotemporal gridification with global time awareness, while T2SGrid-FT indicates fine-tuning with the instruction dataset augmented using our T2SGrid method.

Model	Charades-STA				ActivityNet
Model	[email protected]	[email protected]	[email protected]	mIoU	[email protected]	[email protected]	[email protected]	mIoU
LLaVA-OneVision-1.5-8B [ 28 ]	19.8	6.7	2.3	14.5	9.2	4.9	2.1	7.5
+FT	70.6	52.9	29.8	48.4	50.1	35.1	18.5	33.4
+T2SGrid-FT	$73.9_{\hskip 1.42271pt{\color[rgb]{0.1953,0.7969,0}\textbf{(+3.3)}}}$	$57.7_{\hskip 1.42271pt{\color[rgb]{0.1953,0.7969,0}\textbf{(+4.8)}}}$	$32.7_{\hskip 1.42271pt{\color[rgb]{0.1953,0.7969,0}\textbf{(+2.9)}}}$	$50.8_{\hskip 1.42271pt{\color[rgb]{0.1953,0.7969,0}\textbf{(+2.4)}}}$	$63.5_{\hskip 1.42271pt{\color[rgb]{0.1953,0.7969,0}\textbf{(+13.4)}}}$	$45.4_{\hskip 1.42271pt{\color[rgb]{0.1953,0.7969,0}\textbf{(+10.3)}}}$	$26.2_{\hskip 1.42271pt{\color[rgb]{0.1953,0.7969,0}\textbf{(+7.7)}}}$	$44.6_{\hskip 1.42271pt{\color[rgb]{0.1953,0.7969,0}\textbf{(+10.2)}}}$
Qwen2-VL-7B [ 53 ]	8.7	5.4	2.4	7.9	17.0	9.4	3.9	12.5
+FT	73.1	56.8	32.7	50.4	54.9	37.2	19.2	37.7

Table 2: Training effectiveness of our method. T2SGrid-FT consistently outperforms standard fine-tuning (FT) when trained on the same data with identical training settings, demonstrating the benefits of our approach.

Model	Video MME			MVbench					VideoInstruct
Model	Temporal Perception	Temporal Reasoning	All	Action Sequence	Fine-grained Action	State Change	Scene Transition	All	Temporal Understanding	Detail Orientation
Qwen2-VL-7B	60.0	41.7	63.3	60.5	46.5	44.0	79.5	51.7	2.47	2.57
+T2SGrid	74.5	50.2	64.1	71.5	49.5	51.5	89.0	58.3	2.52	2.69

Table 3: Performance comparison on Video-MME, MVBench, and VideoInstruct benchmarks . Our T2SGrid method consistently improves temporal perception and action understanding, demonstrating strong generalization QA tasks.

Figure 5: Qualitative Comparison with State-of-the-Art. Our method outperforms TRACE [ 17 ] and Numpro [ 61 ] on Charades by accurately identifying event boundaries in challenging scenes.

ComTextNum	Sliding Window	Grid	[email protected]	[email protected]	[email protected]	mIoU
✗	✗	✗	8.7	5.4	2.4	7.9
✓	✗	✗	53.5	23.2	7.9	32.9
✓	✓	✗	58.3	35.1	13.6	36.5
✓	✓	✓	70.1	46.7	20.1	44.3

Table 4: Ablation study of key components: composite textual timestamps (ComTextNum), sliding window, and grid layout. Each component contributes to improved temporal modeling and overall grounding performance.

Temporal Modeling	[email protected]	[email protected]	[email protected]	mIoU	mToken	mTime/s
PE	53.1	28.4	10.6	33.8	5760.4	1.69
TextNum	53.5	23.2	7.9	32.5	5791.2	1.45
VisualNum	60.7	36.8	15.9	38.5	5760.4	2.17
Ours (wo/ overlap)	64.5	42.9	18.1	41.2	5766.1	1.43
Ours (w/ overlap)	70.2	46.7	20.1	44.3	7909.7	2.31

Table 5: Comparison of different temporal modeling strategies applied to the Qwen2-VL-7B baseline, including position encoding (PE), textual timestamping (TextNum), visual frame-level numeric labeling (VisualNum), and our proposed T2SGrid framework. mToken and mTime/s denote the average token count and inference time per sample, respectively.

Config	[email protected]	[email protected]	[email protected]	mIoU	mToken	mTime/s
Stage 1: Exploring Grid size and stride
g11_s1	53.5	23.2	7.9	32.9	5791.2	1.45
g22_s4	56.5	26.9	8.7	34.2	5789.0	1.43
g23_s6	61.6	38.7	16.2	38.8	5784.3	1.42
g32_s6	61.8	38.8	16.9	39.0	5784.3	1.43
g33_s9	63.7	42.8	18.4	41.1	5782.1	1.43

Table 6: Ablation results of grid configurations on Qwen2-VL-7B. g{col}{row}_s{stride} denoting grid size and stride. Stage 1 varies size and stride, and Stage 2 refines stride for g43.

Temporal Video Grounding

Temporal video grounding locates the moments in a video that match a natural-language query, enabling precise search and evidence-based understanding.

See all 166 papers

GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding
Apr 2, 2026Rong Fan, Kaiyan Xiao, Minghao Zhu +3Temporal Video Grounding Video-Language Models
Newcapec AI Research · Fudan University · Tongji University
Foresee-to-Ground: From Predictive Temporal Perception to Evidence-Driven Reasoning for Video Temporal Grounding
May 21, 2026Zelin Zheng, Xinyan Liu, Ruixin Li +4Temporal Video Grounding Temporal Representation Learning
School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China · Beijing Key Laboratory of Embodied Intelligence Computing, Beijing, China · Faculty of Computing, Harbin Institute of Technology, Weihai, China +1
Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding
Apr 9, 2026Xuezhen Tu, Jingyu Wu, Fangyu Kang +4Vision-Language Grounding Temporal Video Grounding
Shanghai Jiao Tong University China · ZTE Corporation China

Apr 2, 2026J/K move · Enter open · S save

GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding

Rong Fan, Kaiyan Xiao, Minghao Zhu, Liuyi Wang+2

Newcapec AI Research · Fudan University · Tongji University

Temporal Video Grounding Video-Language Models Query-Aware Video Frame Selection Visual Token Pruning

Open paper AskPDF arXiv

Video temporal grounding (VTG) is a critical task in video understanding and a key capability for extending video large language models (Vid-LLMs) to broader applications. However, existing Vid-LLMs rely on uniform frame sampling to extract video information, resulting in a sparse distribution of key frames and the loss of crucial temporal cues. To address this limitation, we propose Grounded Visual Token Sampling (GroundVTS), a Vid-LLM architecture that focuses on the most informative temporal segments. GroundVTS employs a fine-grained, query-guided mechanism to filter visual tokens before feeding them into the LLM, thereby preserving essential spatio-temporal information and maintaining temporal coherence. Futhermore, we introduce a progressive optimization strategy that enables the LLM to effectively adapt to the non-uniform distribution of visual features, enhancing its ability to model temporal dependencies and achieve precise video localization. We comprehensively evaluate GroundVTS on three standard VTG benchmarks, where it outperforms existing methods, achieving a 7.7-point improvement in mIoU for moment retrieval and 12.0-point improvement in mAP for highlight detection. Code is available at https://github.com/Florence365/GroundVTS.

Abstract

Temporal Video Grounding

Explore similar work

GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding

Foresee-to-Ground: From Predictive Temporal Perception to Evidence-Driven Reasoning for Video Temporal Grounding

Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding