Temporal Gains, Spatial Costs: Revisiting Video Fine-Tuning in Multimodal Large Language Models

Linghao Zhang; Jungang Li; Yonghua Hei; Sicheng Tao; Song Dai; Yibo Yan; Zihao Dongfang; Weiting Liu

cs.CVMar 18, 2026

Temporal Gains, Spatial Costs: Revisiting Video Fine-Tuning in Multimodal Large Language Models

Authors: Linghao Zhang, Jungang Li, Yonghua Hei, Sicheng Tao, Song Dai, Yibo Yan, Zihao Dongfang, Weiting Liu,

Organizations: SJTU · CityU · HKUST(GZ) · HKUST ·

Abstract

Multimodal large language models (MLLMs) are typically trained in multiple stages, with video-based supervised fine-tuning (Video-SFT) serving as a key step for improving visual understanding. Yet its effect on the fine-grained evolution of visual capabilities, particularly the balance between spatial and temporal understanding, remains poorly understood. In this paper, we systematically study how Video-SFT reshapes visual capabilities in MLLMs. Across architectures, parameter scales, and frame sampling settings, we observe a consistent pattern: Video-SFT reliably improves video performance, but often yields limited gains or even degradation on static image benchmarks. We further show that this trade-off is closely tied to temporal budget: increasing the number of sampled frames generally improves video performance, but does not reliably improve static image performance. Motivated by this finding, we study an instruction-aware Hybrid-Frame strategy that adaptively allocates frame counts and partially mitigates the image-video trade-off. Our results indicate that Video-SFT is not a free lunch for MLLMs, and preserving spatial understanding remains a central challenge in joint image-video training.

Figures & tables

Figure 1: Overview of the temporal trap and our study. (A) Illustration of the temporal trap : Video-SFT improves video performance but can weaken spatial capability on static images. (B) Image and video benchmarks used in our experiments. (C) Main study dimensions: architecture, scale, frame budget, and task. (D) Hybrid-Frame Strategy , which adaptively assigns a suitable frame budget to each training sample and partially mitigates the image–video trade-off.

Figure 3: Cross-scale attention visualizations before and after Video-SFT on Qwen2.5-VL models (7B, 32B, 72B). For the query “Is there a bird in the image?”, attention becomes more dispersed in smaller models after Video-SFT, while larger models retain more localized focus on the target object, suggesting improved robustness to the temporal trap.

Figure 4: Comparison of image and video benchmark performance before and after Video-SFT across different scale Qwen2.5-VL models, including 3B, 7B, 32B and 72B parameters.

Figure 5: Comparison of performance on image and video benchmarks before and after Video-SFT across 8/16/32/64 training frames on Qwen2.5-VL-7B model.

Setting	Image		Video
	MMStar	POPE	Video-MME	Video-MMMU
Base	62.07	88.03	51.19	44.22
DINOv2-based	61.40 $\downarrow$	87.91 $\downarrow$	53.96 $\uparrow$	47.13 $\uparrow$
Qwen2.5-VL-3B-based	62.19 $\uparrow$	88.19 $\uparrow$	53.47 $\uparrow$	47.91 $\uparrow$
Qwen3-VL-8B-based	62.33 $\uparrow$	88.20 $\uparrow$	54.96 $\uparrow$	47.22 $\uparrow$

Table 1: Comparison of frame allocation strategies for Video-SFT on the Qwen2.5-VL-7B, training and inference using 8 frames.

Qwen2.5-VL-7B
Setting	Image		Video
	MMStar	POPE	Video-MME	Video-MMMU
Video-SFT (8F)	61.67	87.82	54.41	47.56
Hybrid (8F)	62.33	88.20	54.96	47.22
Video-SFT (16F)	62.00	87.93	58.74	50.44
Hybrid (16F)	62.33	88.20	58.15	51.78

Table 2: Unified comparison of Video-SFT strategies across model architectures and frame budgets. For Qwen2.5-VL-7B, training and inference frame counts follow the settings in parentheses, while LLaVA-1.5-7B are evaluated with a fixed 8-frame inference.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Modality	Task	Samples	Dataset
Video-Text	Caption	2k	VidOR, YouCook2, Charades, ActivityNet, Kinetics-700
	Caption	2k	Sthsth2, Ego4D, InternVid-10M, HD-VILA-100M, VIDAL
	Open-Ended	4k	VidOR, YouCook2, Charades, ActivityNet
	Open-Ended	4k	Ego4D, InternVid-10M, HD-VILA-100M, VIDAL
	Multiple-Choice	14k	VidOR, YouCook2, Charades, ActivityNet
	Multiple-Choice	14k	Ego4D, InternVid-10M, HD-VILA-100M, VIDAL

Appendix

Table A: Detailed breakdown of training data statistics. The dataset is categorized into three distinct tasks: Caption, Open-Ended, and Multiple-Choice. For each task, we specify the number of samples and the provenance of the source datasets used to construct the SFT corpus.

Figure A: Comparison of Qwen2.5-VL-7B model’s performance on MME, MMStar and MMBench before and after 8, 16, 32 and 64 frames of Video-SFT, where the number of input images is the same as the number of frames used in Video-SFT.

\blacktriangledown

indicates performance degradation after Video-SFT. Key Insight: (1) SFT models generally lag behind Base models for any given number of frame. (2) A greater number of redundant image inputs hurt visual performance for MLLMs, justifying our Hybrid-Frame Strategy .

Figure B: Performance impact of Video-SFT on Qwen2.5-VL-7B across Fine-Grained Perception, General Understanding, and Visual Reasoning tasks on MME, MMStar, and MMBench. Each column corresponds to a specific capability dimension in image understanding.

\blacktriangledown

indicates performance degradation after Video-SFT, while

\blacktriangle

denotes improvement. The results reveal a non-uniform influence of Video-SFT across visual tasks, with Fine-Grained Perception exhibiting the most pronounced performance drop.

Task	Start Model	Training Frames	Eval Frames	Benchmark
				Image	Video
Model Architecture	LLaVA-1.5-7B LLaVA-NeXT-Video-7B Qwen2.5-VL-7B	8	8	MME	Video-MME
				MMStar	Video-MMMU
				MMBench	MVBench
				POPE	TempCompass
Model Size	Qwen2.5-VL-3B/7B/32B/72B	8	8	MME	Video-MME

Appendix

Table B: Experimental setup for assessing Video-SFT’s effect on the visual capability of MLLMs. We designed experiments from three aspects: model size, model architecture, and training frame count to explore the influence of Video-SFT on the visual capability of MLLMs.

Setting	Video-MME	Video-MMMU	MVBench
Inference with 8 Frames
Qwen2.5-VL-7B (Base)	51.19	44.22	62.68
+ Video-SFT-8F	54.41 (↑3.22)	47.56 (↑3.34)	64.62 (↑1.94)
+ Video-SFT-16F	54.52 (↑3.33)	48.11 (↑3.89)	63.05 (↑0.37)
+ Video-SFT-32F	55.67 (↑4.48)	47.44 (↑3.22)	63.17 (↑0.49)
+ Video-SFT-64F	55.26 (↑4.07)	47.44 (↑3.22)	63.05 (↑0.37)

Appendix

Table C: Performance comparison of Qwen2.5-VL-7B under different Video-SFT strategies across multiple frame settings (8, 16, 32, 64, and our Hybrid Strategy) on Video-MME, Video-MMMU, and MVBench. All models are evaluated using a fixed input of 8 frames. Numbers in

\uparrow

indicate improvements relative to the base model.

Perception (Coarse-grained)
Task Categories	Models
	Qwen2.5-VL-7B		LLaVA-Next-Video-7B		LLaVA-1.5-7B
	Base	SFT	Base	SFT	Base	SFT
Existence	195.00	190.00 (↓5.00)	200.00	190.00 (↓10.00)	195.00	135.00 (↓60.00)
Count	165.00	170.00 (↑5.00)	135.00	153.33 (↑18.33)	148.33	105.00 (↓43.33)
Position	165.00	160.00 (↓5.00)	121.67	130.00 (↑8.33)	128.33	51.67 (↓76.66)

Appendix

Table D: Performance comparison of Qwen2.5-VL-7B, LLaVA-Next Video-7B and LLaVA-1.5-7B before and after Video-SFT in different tasks on the MME Benchmark. Numbers in

\uparrow

indicate improvements, while numbers in

\downarrow

indicate degradations relative to the base model.

Coarse Perception
Task Capabilities	Models
	Qwen2.5-VL-7B		LLaVA-Next-Video-7B		LLaVA-1.5-7B
	Base	SFT	Base	SFT	Base	SFT
Image Scene & Topic	63.83	60.99 (↓2.84)	51.77	48.23 (↓3.54)	49.65	32.62 (↓17.03)
Image Emotion	70.97	67.74 (↓3.23)	80.65	67.74 (↓12.91)	80.65	54.84 (↓25.81)
Image Style & Quality	88.46	87.18 (↓1.28)	69.23	75.64 (↑6.41)	69.23	38.46 (↓30.77)

Appendix

Table E: Performance comparison of Qwen2.5-VL-7B, LLaVA-Next Video-7B and LLaVA-1.5-7B before and after Video-SFT in different tasks on the MMStar Benchmark. Numbers in

\uparrow

indicate improvements, while numbers in

\downarrow

indicate degradations relative to the base model.

Coarse Perception
Task Categories	Models
	Qwen2.5-VL-7B		LLaVA-Next-Video-7B		LLaVA-1.5-7B
	Base	SFT	Base	SFT	Base	SFT
Image Scene	98.28	98.28	79.12	73.96 (↓5.16)	78.38	47.17 (↓31.21)
Image Emotion	83.00	84.50 (↑1.50)	54.50	70.00 (↑15.50)	74.00	43.50 (↓30.50)
Image Quality	58.67	62.67 (↑4.00)	36.67	40.67 (↑4.00)	37.33	34.67 (↓2.66)

Appendix

Table F: Performance comparison of Qwen2.5-VL-7B, LLaVA-Next Video-7B and LLaVA-1.5-7B before and after Video-SFT in different tasks on the MMBench Benchmark. Numbers in

\uparrow

indicate improvements, while numbers in

\downarrow

indicate degradations relative to the base model.

Panel of Figure 15 — Figure C: The prompt used for Hybrid-Frame Strategy. It comprehensively evaluates aspects such as event duration, motion continuity, causality, object interaction, and fine-grained visual attributes, aiming to adaptively determine the optimal number of frames required to resolve each instruction.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,091 papers

The Visual Bottleneck: Sparse-Frame Adaptation of MLLMs for Joint Spatial-Temporal Video Grounding
Jul 27, 2026Jiameng Zhang, Srikanth MadikeriTemporal Video Grounding VLM Adaptation
University of Zurich
VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs
Sep 15, 2026Haoyu Guo, Yuan Feng, Junlin Lv +3Efficient VLM Inference Long-Video Understanding
School of Biomedical Engineering, University of Science and Technology of China · Data Darkness Lab, MIRACLE Center, Suzhou Institute for Advanced Research · China +1
Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events
Jun 1, 2026Xiaolin Liu, Yilun Zhu, Xiangyu Zhao +9VLM Evaluation Video QA
Shandong University · Southeast University · Shanghai Jiao Tong University +1

Jul 27, 2026J/K move · Enter open · S save

The Visual Bottleneck: Sparse-Frame Adaptation of MLLMs for Joint Spatial-Temporal Video Grounding

Jiameng Zhang, Srikanth Madikeri

University of Zurich

Temporal Video Grounding VLM Adaptation Vision-Language Grounding Video Frame Selection

Open paper AskPDF arXiv

Large-scale video platforms process millions of uploads hourly, requiring moderation systems that can localize when and where policy violations occur within each video. Processing every frame is infeasible at scale, so systems are constrained to sparse inputs of 8 to 16 frames per video. Yet state-of-the-art multimodal large language models (MLLMs) are pretrained on dense sequences of hundreds of frames, creating a fundamental mismatch between training and deployment conditions. This mismatch causes severe performance collapse: the Qwen3-VL 8B model drops from 56.0% to 22.3% temporal mIoU when frames are reduced to 16, a 60.2% relative degradation. We present a systematic empirical study of training strategies to close this gap for spatial-temporal video grounding. Our results suggest that visual feature extraction is the dominant bottleneck under sparse-frame inputs. Adapting only the final three ViT layers, 4% of total parameters, achieves 68.8% temporal mIoU and surpasses a zero-shot 8B model using dense inputs by 12.8 points. Language model fine-tuning, by contrast, offers negligible or negative returns. A boundary-aware sampling strategy, Hybrid16, further improves temporal mIoU by 26 points over uniform sampling when temporal boundaries are available. We conclude that for sparse-frame video grounding, training strategy dominates model scale: a fine-tuned 2B model consistently outperforms a zero-shot 8B model, with or without dense frame access.

Abstract

Multimodal Large Language Models

Explore similar work

The Visual Bottleneck: Sparse-Frame Adaptation of MLLMs for Joint Spatial-Temporal Video Grounding

VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs

Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events