TR-EduVSum: A Turkish-Focused Dataset and Consensus Framework for Educational Video Summarization

Figen Eğin; Aytuğ Onan

cs.CLApr 8, 2026

TR-EduVSum: A Turkish-Focused Dataset and Consensus Framework for Educational Video Summarization

Authors: Figen Eğin, Aytuğ Onan

Organizations: Department of Computer Engineering, Izmir Katip Celebi University, Cigli, Izmir, Türkiye · Department of Computer Engineering, Izmir Institute of Technology, Urla, Izmir, Türkiye

Abstract

This study presents a framework for generating the gold-standard summary fully automatically and reproducibly based on multiple human summaries of Turkish educational videos. Within the scope of the study, a new dataset called TR-EduVSum was created, encompassing 82 Turkish course videos in the field of "Data Structures and Algorithms" and containing a total of 3281 independent human summaries. Inspired by existing pyramid-based evaluation approaches, the AutoMUP (Automatic Meaning Unit Pyramid) method is proposed, which extracts consensus-based content from multiple human summaries. AutoMUP clusters the meaning units extracted from human summaries using embedding, statistically models inter-participant agreement, and generates graded summaries based on consensus weight. In this framework, the gold summary corresponds to the highest-consensus AutoMUP configuration, constructed from the most frequently supported meaning units across human summaries. Experimental results show that AutoMUP summaries exhibit high semantic overlap with robust LLM (Large Language Model) summaries such as Flash 2.5 and GPT-5.1. Furthermore, ablation studies clearly demonstrate the decisive role of consensus weight and clustering in determining summary quality. The proposed approach can be generalized to other Turkic languages at low cost.

Figures & tables

Statistic	Value
Total video duration	26 h 23 min 24 sec
Average video duration	19 min 18 sec
Median video duration	15 min 24 sec
Minimum video duration	3 min 06 sec
Maximum video duration	48 min 12 sec
Total transcript word count	161,464

Table 1: Descriptive statistics of the TR-EduVSum dataset.

Metric	A1	A2	A3
Flash 2.5
BERTScore-F1	0.872	0.860	0.849
SBERT	0.720	0.634	0.614
SimCSE	0.975	0.973	0.969
USE	0.711	0.660	0.630
ROUGE-L	0.246	0.166	0.144

Table 2: Similarity between AutoMUP summaries and LLM-generated summaries. Higher scores indicate greater similarity.

Figure 1: Distribution of mean SBERT similarity across human summaries.

Metric	AutoMUP-1	AutoMUP-2	AutoMUP-3	No-Clustering	No-Consensus
SimCSE	0.742 $\pm$ 0.049	0.562 $\pm$ 0.055	0.526 $\pm$ 0.086	0.625 $\pm$ 0.126	0.488 $\pm$ 0.104
USE	0.626 $\pm$ 0.068	0.433 $\pm$ 0.079	0.381 $\pm$ 0.093	0.521 $\pm$ 0.122	0.353 $\pm$ 0.105
SBERT	0.740 $\pm$ 0.059	0.609 $\pm$ 0.068	0.596 $\pm$ 0.087	0.625 $\pm$ 0.126	0.488 $\pm$ 0.104
BERTScore-F1	0.574 $\pm$ 0.028	0.518 $\pm$ 0.027	0.493 $\pm$ 0.037	0.538 $\pm$ 0.043	0.478 $\pm$ 0.037
ROUGE-L	0.217 $\pm$ 0.035	0.157 $\pm$ 0.022	0.137 $\pm$ 0.027	0.200 $\pm$ 0.047	0.116 $\pm$ 0.028

Table 3: Comparison of AutoMUP and its ablation variants in terms of their alignment with human summaries (mean

\pm

std). Higher scores indicate stronger alignment with human summaries. AutoMUP-1 corresponds to the gold summary in the proposed framework; other variants are included for ablation and comparison purposes.

Figure 2: Semantic similarity between AutoMUP summaries and human summaries across different consensus levels and ablation settings.

Video Summarization

Video summarization selects or generates concise representations of video while preserving important events, details, and temporal structure.

See all 22 papers

QEVA: A Reference-Free Evaluation Metric for Narrative Video Summarization with Multimodal Question Answering
Apr 27, 2026Woojun Jung, Junyeong KimVideo Summarization Summarization Evaluation
Department of Artificial Intelligence, Chung-Ang University
LVSum: A Benchmark for Timestamp-Aware Long Video Summarization
Apr 11, 2026Alkesh Patel, Melis Ozyildirim, Ying-Chang Cheng +1Video Summarization Video-Language Model Evaluation
Apple
Multimodal Abstractive Summarization of Instructional Videos with Vision-Language Models
May 12, 2026Maham Nazir, Muhammad Aqeel, Richong Zhang +1Video Summarization Video-Language Models
Beihang University, Beijing, China · University of Verona, Italy

Apr 27, 2026J/K move · Enter open · S save

QEVA: A Reference-Free Evaluation Metric for Narrative Video Summarization with Multimodal Question Answering

Woojun Jung, Junyeong Kim

Department of Artificial Intelligence, Chung-Ang University

Video Summarization Summarization Evaluation Reference-Free Evaluation

Open paper AskPDF arXiv

Video-to-text summarization remains underexplored in terms of comprehensive evaluation methods. Traditional n-gram overlap-based metrics and recent large language model (LLM)-based approaches depend heavily on human-written reference summaries, limiting their practicality and sensitivity to nuanced semantic aspects. In this paper, we propose QEVA, a reference-free metric evaluating candidate summaries directly against source videos through multimodal question answering. QEVA assesses summaries along three clear dimensions: Coverage, Factuality, and Chronology. We also introduce MLVU(VS)-Eval, a new annotated benchmark derived from the MLVU dataset, comprising 800 summaries generated from 200 videos using state-of-the-art video-language multimodal models. This dataset establishes a transparent and consistent framework for evaluation. Experimental results demonstrate that QEVA shows higher correlation with human judgments compared to existing approaches, as measured by Kendall's

τ_b

τ_c

, and Spearman's

ρ

. We hope that our benchmark and metric will facilitate meaningful progress in video-to-text summarization research and provide valuable insights for the development of future evaluation methods.

Abstract

Video Summarization

Explore similar work

QEVA: A Reference-Free Evaluation Metric for Narrative Video Summarization with Multimodal Question Answering

LVSum: A Benchmark for Timestamp-Aware Long Video Summarization

Multimodal Abstractive Summarization of Instructional Videos with Vision-Language Models