LVSum: A Benchmark for Timestamp-Aware Long Video Summarization

Alkesh Patel; Melis Ozyildirim; Ying-Chang Cheng; Ganesh Nagarajan

cs.CVApr 11, 2026

LVSum: A Benchmark for Timestamp-Aware Long Video Summarization

Authors: Alkesh Patel, Melis Ozyildirim, Ying-Chang Cheng, Ganesh Nagarajan

Organizations: Apple

Abstract

Long video summarization presents significant challenges for multimodal large language models (MLLMs), particularly in maintaining temporal fidelity over extended durations and producing summaries that are both semantically and temporally grounded. We introduce LVSum, a human-annotated benchmark for evaluating long-form video summarization with fine-grained temporal alignment. LVSum comprises 72 diverse videos spanning 13 domains with an average duration of 16 minutes, each annotated with up to 10 human-generated summaries containing temporal references. We conduct a comprehensive evaluation of leading proprietary and open-source MLLMs using newly introduced LLM-based metrics for content relevance and modality coherence, alongside standard automatic metrics. Our experiments reveal three key findings: (1) transcripts contribute substantially more to summarization quality than visual frames alone, (2) a significant performance gap persists between model-generated and human-written summaries, and (3) current MLLMs exhibit systematic weaknesses in temporal grounding, instruction adherence, and cross-modal coherence. We release the dataset and code.

Figures & tables

Dataset	Domain	# Videos	Len. (avg)	Ann./ video	Imp. Score
SumMe	3 cat.	25	1–3 (2.6)m	15–18	$\checkmark$
TVSum	10 cat.	50	2–10 (4.2)m	20	$\checkmark$
OVP	5 cat.	50	1–4 (3)m	5	$\times$
YouTube	5 cat.	50	1–10 (3.5)m	5	$\times$
EDUVSUM	Lectures	98	up to 15 (6.5)m	1	$\checkmark$
VideoXum	open	14K	up to 12.5 (2)m	10	$\times$

Table 1: Comparison of LVSum with other Video Summarization datasets

Panel of Figure 1 — Figure 1: Distribution of video categories in the LVSum dataset.

Model	Model Input	$\tau$	$\rho$	CR	MC	Sum. Len.
Human	-	0.13388	0.13635	2.77	3.88	$\approx 10\%$
Random	-	0.01310	0.01357	2.14	3.91	$\approx 18\%$
Even	-	0.00258	0.00263	2.30	3.87	$\approx 18\%$
Opus-4.5	V + T	0.10093	0.10513	3.03	3.90	$\approx 30\%$
Gemini-2.5-Pro	V + T	0.08902	0.09071	3.28	4.32	$\approx 12\%$
Qwen3-VL-235B	V + T	0.06389	0.06706	2.40	3.90	$\approx 56\%$

Table 2: Evaluation of representative models on the LVSum benchmark. Higher values indicate better performance for Kendall’s

\tau

, Spearman’s

\rho

, Content Relevance (CR), and Modality Coherence (MC). The Summary Length column reports the average percentage of video duration selected by each method. V=Video, T=Transcript

Panel of Figure 2 — Figure 2: Correlation vs. summary length for different models. Solid lines denote Kendall’s $\tau$ , dashed lines denote Spearman’s $\rho$ .

Model	Input	$\tau$	$\rho$	Sum. Len.
Opus-4.5	V	0.06182	0.06425	$\approx 27\%$
	T	0.08879	0.09259	$\approx 27\%$
	V + T	0.10093	0.10513	$\approx 42\%$
Gemini-2.5-Pro	V	0.06106	0.06252	$\approx 14\%$
	T	0.07742	0.07904	$\approx 12\%$
	V + T	0.08902	0.09071	$\approx 13\%$

Table 3: Ablations of input modality. V=Video, T=Transcript

Model	# Frames	$\tau$	$\rho$	Sum. Len.
Gemini-2.5-Pro	24	0.03645	0.03740	$\approx 14\%$
	32	0.04221	0.04298	$\approx 12\%$
	64	0.05580	0.05721	$\approx 14\%$
	96	0.05580	0.05714	$\approx 14\%$
	128	0.05734	0.05854	$\approx 12\%$
Qwen3-VL-235B	24	0.03559	0.03674	$\approx 25\%$

Table 4: Ablations on number of input frames for video-only evaluation

Model	Category	$\tau$	$\rho$	CR	MC	Sum. Len.
Opus-4.5	Lectures	0.11163	0.11633	3.22	4.21	$\approx 31\%$
	Podcasts	0.10354	0.10820	2.94	4.13	$\approx 30\%$
	Vlogs	0.10852	0.11220	3.05	3.00	$\approx 24\%$
	Explainer Videos	0.10855	0.11340	2.90	3.75	$\approx 31\%$
	News Reports	0.07502	0.07813	2.77	3.75	$\approx 33\%$
Gemini-2.5-Pro	Lectures	0.08215	0.08399	3.58	4.46	$\approx 13\%$

Table 5: Ablations by category on Video + Transcript input

Panel of Figure 3 — Figure 3: Video summarization comparison for a selected video from LVSum.

Figure 4: Failure cases illustrating distinct evaluation modes. (a) Low Content Relevance (CR): summary omits salient events. (b) Low Modality Coherence (MC): textual descriptions contradict visual evidence within the predicted interval.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$\tau$	$\rho$	Sum. Len.
O (In this paper)	0.06389	0.06706	$\approx 56\%$
A (Length Constraint First)	0.06389	0.06698	$\approx 58\%$
B (Length Constraint Last)	0.07396	0.07703	$\approx 45\%$
C (Sandwich)	0.07198	0.07549	$\approx 50\%$

Appendix

Table 7: Ablations by category, Qwen3-VL-235B on Video + Transcript input

Panel of Figure 5 — Figure 5: Comparison of video summarization results across different models and videos

Model	# Frames	$\tau$	$\rho$	Sum. Len.
Opus-4.5	24	0.04582	0.04759	$\approx 27\%$
	32	0.06170	0.06408	$\approx 26\%$
	64	0.05404	0.05614	$\approx 28\%$
	96	0.06329	0.06569	$\approx 26\%$
	100	0.06401	0.06639	$\approx 25\%$

Appendix

Table 8: Ablations on number of input frames for video-only evaluation

Video Summarization

Video summarization selects or generates concise representations of video while preserving important events, details, and temporal structure.

See all 22 papers

Long-form Video-to-Text Summarization using Recursive Language Models
Oct 19, 2026Alessia Intini, Lorenzo Vaiani, Enrico Chen +1Video Summarization Video-Language Models
Politecnico di Torino
Summarize Before Grounding: Query-Guided Chunk Condensation for Long-Video Temporal Grounding
Sep 28, 2026Nanxing Hu, Xiaoyue Duan, Qiwei Yan +3Temporal Video Grounding Video Token Compression
Beihang University · University of the Chinese Academy of Sciences · Institute of automation, Chinese academy of science
Enhancing Localized Reasoning for Long Video Understanding via Efficient Segment-to-Video Supervision
Aug 21, 2026Beibei Zhang, Chao Xu, Jun Lan +4Long-Video Understanding VLM Reasoning
State Key Laboratory for Novel Software Technology, Nanjing University · Ant Group · School of Computer Science, Shanghai Jiao Tong University

Oct 19, 2026J/K move · Enter open · S save

Long-form Video-to-Text Summarization using Recursive Language Models

Alessia Intini, Lorenzo Vaiani, Enrico Chen, Luca Cagliero

Politecnico di Torino

Video Summarization Video-Language Models Recursive Language Models

Open paper AskPDF arXiv

While VideoLLMs have made significant progress in understanding short video benchmarks, they often struggle to summarize long-form ones. This is mainly due to performance degradation caused by long-context prompts, the inherent difficulty of preserving temporal continuity and a lack of semantically rich concepts in event descriptions. To tackle these issues, this paper introduces the first recursive Video Language Summarization Model (RVLSM) designed for video processing and tailored to long-form video-to-text summarization. RVLSM recursively decomposes videos into semantically coherent segments and delegates chunk-level reasoning to VideoLLM sub-models and external tools. It employs (1) Semantic Temporal Decomposition (STD), which preserves temporal coherence and continuity through a scene-aware chunking strategy that dynamically partitions videos according to semantic boundaries, and (2) Adaptive Multimodal Tool Calling (AMTC), which ensures semantic richness of the summary via a routing mechanism that selectively invokes auxiliary modules such as action recognition, optical character recognition, automatic speech recognition, and object detection. Experiments on the VideoXum and MixTyVSum benchmarks show that RVLSM consistently improves over zero-shot VideoLLM baselines across multiple model settings and performance metrics, and is competitive with a state-of-the-art fine-tuned model for video-to-text summarization. Importantly, the named entities in the RVLSM's summaries align more closely with those detected in the ground-truth summaries than those detected by baseline and state-of-the-art methods, confirming the saliency of the event descriptions it generates.

Abstract

Video Summarization

Explore similar work

Long-form Video-to-Text Summarization using Recursive Language Models

Summarize Before Grounding: Query-Guided Chunk Condensation for Long-Video Temporal Grounding

Enhancing Localized Reasoning for Long Video Understanding via Efficient Segment-to-Video Supervision