TR-EduVSum: A Turkish-Focused Dataset and Consensus Framework for Educational Video Summarization
Authors: Figen Eğin, Aytuğ Onan
Organizations: Department of Computer Engineering, Izmir Katip Celebi University, Cigli, Izmir, Türkiye · Department of Computer Engineering, Izmir Institute of Technology, Urla, Izmir, Türkiye
This study presents a framework for generating the gold-standard summary fully automatically and reproducibly based on multiple human summaries of Turkish educational videos. Within the scope of the study, a new dataset called TR-EduVSum was created, encompassing 82 Turkish course videos in the field of "Data Structures and Algorithms" and containing a total of 3281 independent human summaries. Inspired by existing pyramid-based evaluation approaches, the AutoMUP (Automatic Meaning Unit Pyramid) method is proposed, which extracts consensus-based content from multiple human summaries. AutoMUP clusters the meaning units extracted from human summaries using embedding, statistically models inter-participant agreement, and generates graded summaries based on consensus weight. In this framework, the gold summary corresponds to the highest-consensus AutoMUP configuration, constructed from the most frequently supported meaning units across human summaries. Experimental results show that AutoMUP summaries exhibit high semantic overlap with robust LLM (Large Language Model) summaries such as Flash 2.5 and GPT-5.1. Furthermore, ablation studies clearly demonstrate the decisive role of consensus weight and clustering in determining summary quality. The proposed approach can be generalized to other Turkic languages at low cost.
Figures & tables
Statistic
Value
Total video duration
26 h 23 min 24 sec
Average video duration
19 min 18 sec
Median video duration
15 min 24 sec
Minimum video duration
3 min 06 sec
Maximum video duration
48 min 12 sec
Total transcript word count
161,464
Table 1: Descriptive statistics of the TR-EduVSum dataset.
Metric
A1
A2
A3
Flash 2.5
BERTScore-F1
0.872
0.860
0.849
SBERT
0.720
0.634
0.614
SimCSE
0.975
0.973
0.969
USE
0.711
0.660
0.630
ROUGE-L
0.246
0.166
0.144
Table 2: Similarity between AutoMUP summaries and LLM-generated summaries. Higher scores indicate greater similarity.
Figure 1: Distribution of mean SBERT similarity across human summaries.
Metric
AutoMUP-1
AutoMUP-2
AutoMUP-3
No-Clustering
No-Consensus
SimCSE
0.742 ± 0.049
0.562 ± 0.055
0.526 ± 0.086
0.625 ± 0.126
0.488 ± 0.104
USE
0.626 ± 0.068
0.433 ± 0.079
0.381 ± 0.093
0.521 ± 0.122
0.353 ± 0.105
SBERT
0.740 ± 0.059
0.609 ± 0.068
0.596 ± 0.087
0.625 ± 0.126
0.488 ± 0.104
BERTScore-F1
0.574 ± 0.028
0.518 ± 0.027
0.493 ± 0.037
0.538 ± 0.043
0.478 ± 0.037
ROUGE-L
0.217 ± 0.035
0.157 ± 0.022
0.137 ± 0.027
0.200 ± 0.047
0.116 ± 0.028
Table 3: Comparison of AutoMUP and its ablation variants in terms of their alignment with human summaries (mean ± std). Higher scores indicate stronger alignment with human summaries. AutoMUP-1 corresponds to the gold summary in the proposed framework; other variants are included for ablation and comparison purposes.
Figure 2: Semantic similarity between AutoMUP summaries and human summaries across different consensus levels and ablation settings.
Video-to-text summarization remains underexplored in terms of comprehensive evaluation methods. Traditional n-gram overlap-based metrics and recent large language model (LLM)-based approaches depend heavily on human-written reference summaries, limiting their practicality and sensitivity to nuanced semantic aspects. In this paper, we propose QEVA, a reference-free metric evaluating candidate summaries directly against source videos through multimodal question answering. QEVA assesses summaries along three clear dimensions: Coverage, Factuality, and Chronology. We also introduce MLVU(VS)-Eval, a new annotated benchmark derived from the MLVU dataset, comprising 800 summaries generated from 200 videos using state-of-the-art video-language multimodal models. This dataset establishes a transparent and consistent framework for evaluation. Experimental results demonstrate that QEVA shows higher correlation with human judgments compared to existing approaches, as measured by Kendall's τb, τc, and Spearman's ρ. We hope that our benchmark and metric will facilitate meaningful progress in video-to-text summarization research and provide valuable insights for the development of future evaluation methods.
Woojun Jung, Junyeong Kim
Department of Artificial Intelligence, Chung-Ang University
Long video summarization presents significant challenges for multimodal large language models (MLLMs), particularly in maintaining temporal fidelity over extended durations and producing summaries that are both semantically and temporally grounded. We introduce LVSum, a human-annotated benchmark for evaluating long-form video summarization with fine-grained temporal alignment. LVSum comprises 72 diverse videos spanning 13 domains with an average duration of 16 minutes, each annotated with up to 10 human-generated summaries containing temporal references. We conduct a comprehensive evaluation of leading proprietary and open-source MLLMs using newly introduced LLM-based metrics for content relevance and modality coherence, alongside standard automatic metrics. Our experiments reveal three key findings: (1) transcripts contribute substantially more to summarization quality than visual frames alone, (2) a significant performance gap persists between model-generated and human-written summaries, and (3) current MLLMs exhibit systematic weaknesses in temporal grounding, instruction adherence, and cross-modal coherence. We release the dataset and code.
Multimodal video summarization requires visual features that align semantically with language generation. Traditional approaches rely on CNN features trained for object classification, which represent visual concepts as discrete categories not aligned with natural language. We propose ClipSum, a framework that leverages frozen CLIP vision-language features with explicit temporal modeling and dimension-adaptive fusion for instructional video summarization. CLIP's contrastive pre-training on 400M image-text pairs yields visual features semantically aligned with the linguistic concepts that text decoders generate, bridging the vision-language gap at the representation level. On YouCook2, ClipSum achieves 33.0% ROUGE-1 versus 30.5% for ResNet-152 with 4x lower dimensionality (512 vs. 2048), demonstrating that semantic alignment matters more than feature capacity. Frozen CLIP (33.0%) surpasses fine-tuned CLIP (32.3%), showing that preserving pre-trained alignment is more valuable than task-specific adaptation. https://github.com/aqeeelmirza/clipsum
Maham Nazir, Muhammad Aqeel, Richong Zhang +1
Beihang University, Beijing, China · University of Verona, Italy