SD-MVSum: Script-Driven Multimodal Video Summarization Method and Datasets
Organizations: CERTH-ITI Thermi, Greece
Abstract
In this work, we present a method and two large-scale datasets for Script-Driven Multimodal Video Summarization. The proposed method, SD-MVSum, builds on our earlier SD-VSum method for script-driven video summarization. SD-VSum considered the relevance of a provided script, which outlines the desired content of the summary, only with the visual content of the video. SD-MVSum takes into account, in addition to the visual modality, the relevance of the script with the spoken content (i.e., the audio transcript) of the video. The dependence between each considered pair of data modalities, i.e., script-video and script-transcript, is modeled using a new weighted cross-modal attention mechanism. This mechanism explicitly exploits the semantic similarity between the paired modalities in order to promote the parts of the full-length video with the highest relevance to the script. Furthermore, we extend two large-scale datasets for script-driven (S-VideoXum) and generic (MrHiSum) video summarization, to make them suitable for training and evaluation of script-driven multimodal video summarization methods. Experimental comparisons document the competitiveness of the proposed SD-MVSum method against other SotA approaches for script-driven and generic video summarization. Our new method and extended datasets are available at: https://github.com/IDT-ITI/SD-MVSum.
Figures & tables
| Dataset | Domains | Videos | Data modalities | Annotations per video | Type of annotations | Task |
|---|---|---|---|---|---|---|
| VideoXum [ 18 ] (TMM’23) | open domain | 14,001 | video, text (video description) | 10 | ground-truth video summaries, text description of the video | video summarization with multimodal output |
| MrHiSum [ 7 ] (NeurIPS’23) | 3,509 | 31,892 | video | 1 | frame-level importance scores | video summarization & highlight detection |
| MMSum [ 19 ] (CVPR’24) | 17 | 5,100 | video, text, transcripts video metadata | 1 | ground-truth video and text summary | multimodal summarization with multimodal output |
| S-VideoXum [ 6 ] (ACM MM’25) | open domain | 11,908 | video, text (summary script) | 10 | ground-truth video summaries, text descriptions of the summaries | script-driven video summarization |
| SM-VideoXum (Ours) | open domain | 11,908 | video, text (summary script), transcripts | 10 | ground-truth video summaries, text descriptions of the summaries | script-driven multimodal video summarization |
| SM-MrHiSum (Ours) | 3,509 | 29,917 | video, text (summary script), transcripts | 1 | ground-truth video summaries, text descriptions of the summaries | script-driven multimodal video summarization |
| Min | Max | Avg | Std | |
| SM-VideoXum | ||||
| # words per script | 41 | 187 | 147.0 | 26.4 |
| # sentences per script | 3 | 25 | 8.5 | 1.8 |
| SM-MrHiSum | ||||
| # words per script | 32 | 191 | 148.0 | 25.4 |
| # sentences per script | 1 | 23 | 8.6 | 1.8 |
| Spoken content statistics | ||||
| Dataset | # videos total | # videos w/o speech | % videos w/o speech | |
| SM-VideoXum | 11,908 | 3,893 | 32.7% | |
| SM-MrHiSum | 29,917 | 5,638 | 18.8% | |
| For videos with speech, # sentences per transcript | ||||
| Dataset | Min | Max | Avg | Std |
| SM-VideoXum | 1 | 226 | 21.6 | 20.1 |
| Data modalities | SM-VideoXum | SM-MrHiSum | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Task | Model | Script | Visual | Transcript | F1 | F1 | ||||
| Script- driven summ. | SD-MVSum (proposed) | ✓ | ✓ | ✓ | 27.3 | N/A | N/A | 59.3 | 0.204 | 0 .273 |
| SD-VSum [ 6 ] (ACM MM’25) | ✓ | ✓ | X | 2 4.4 | N/A | N/A | 57.9 | 0.175 | 0.236 | |
| CLIP-It [ 3 ] (NeurIPS’21) | ✓ | ✓ | X | 22.8 | N/A | N/A | 56.3 | 0.120 | 0.169 | |
| Generic summ. | A2Summ [ 26 ] (CVPR’23) | X | ✓ | ✓ | 21.3 | 0.145 | 0.193 | 5 8.0 | 0.169 | 0.239 |
| CSTA [ 64 ] (CVPR’24) | X | ✓ | X | 23.5 | 0.176 | 0.233 | 57.7 | 0.193 | 0.274 | |
| Data modalities | SM-VideoXum | SM-MrHiSum | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Task | Model | Script | Visual | Transcript | Scaling | F1 | F1 | ||
| Script- driven summ. | SD-MVSum (proposed) | ✓ | ✓ | ✓ | ✓ | 27.3 | 59.3 | 0.204 | 0.273 |
| Variant #1 | ✓ | ✓ | X | ✓ | 26.6 | 58.3 | 0.169 | 0.230 | |
| Variant #2 | ✓ | ✓ | ✓ | X | 25.1 | 58.4 | 0.195 | 0.265 | |
| Variant #3 | ✓ | ✓ | ✓ | convent. | 24.9 | 58.5 | 0.196 | 0.264 | |
| SM-VideoXum | SM-MrHiSum | |||
|---|---|---|---|---|
| Att. heads | F1 | F1 | ||
| 4 | 25.7 | 58.2 | 0.181 | 0.243 |
| 8 (prop.) | 27.3 | 59.3 | 0.204 | 0.273 |
| 16 | 27.2 | 58.4 | 0.185 | 0.251 |
| SM-VideoXum | |
|---|---|
| Batch size | F1 |
| 2 | 26.7 |
| 4 (prop.) | 27.3 |
| 8 | 26.4 |
| SM-MrHiSum | |||
|---|---|---|---|
| Batch size | F1 | ||
| 32 | 58.4 | 0.176 | 0.237 |
| 64 (prop.) | 59.3 | 0.204 | 0.273 |
| 128 | 58.7 | 0.195 | 0.263 |
| SM-VideoXum | SM-MrHiSum | |||
|---|---|---|---|---|
| Data fusion | F1 | F1 | ||
| Mean pooling | 26.4 | 58.5 | 0.182 | 0.247 |
| Max pooling | 26.2 | 59.0 | 0.184 | 0.248 |
| Concat. (prop.) | 27.3 | 59.3 | 0.204 | 0.273 |
| Missing data representation | SM-VideoXum | SM-MrHiSum | ||
|---|---|---|---|---|
| F1 | F1 | |||
| Zero vectors (prop.) | 27.3 | 59.3 | 0.204 | 0.273 |
| Learned embeddings | 26.0 | 58.6 | 0.199 | 0.268 |
| SM-VideoXum | SM-MrHiSum | |||
|---|---|---|---|---|
| F1 | F1 | |||
| SD-MVSum (prop.) | 27.3 | 59.3 | 0.204 | 0.273 |
| SD-MVSum ext | 26.7 | 58.7 | 0.207 | 0.279 |
| Visual-transcript correl. (#samples per dataset) | SM-VideoXum | SM-MrHiSum | ||
|---|---|---|---|---|
| F1 | F1 | |||
| Low (3,490 / 35) | 26.8 | 55.1 | 0.070 | 0.101 |
| Medium (13,460 / 560) | 27.1 | 59.0 | 0.175 | 0.238 |
| High (6,120 / 956) | 28.0 | 59.7 | 0.232 | 0.307 |
| Script-transcript correl. (#samples per dataset) | SM-VideoXum | SM-MrHiSum | ||
|---|---|---|---|---|
| F1 | F1 | |||
| Low (1,599 / 40) | 27.7 | 54.3 | 0.023 | 0.036 |
| Medium (10,692 / 582) | 27.1 | 59.2 | 0.206 | 0.278 |
| High (10,779 / 929) | 27.4 | 59.7 | 0.216 | 0.288 |
| Transcript availability (#samples per dataset) | SM-VideoXum | SM-MrHiSum | ||
|---|---|---|---|---|
| F1 | F1 | |||
| No (11,120 / 313) | 27.2 | 58.7 | 0.185 | 0.250 |
| Yes (23,070 / 1,551) | 27.3 | 59.4 | 0.208 | 0.278 |
| SM-VideoXum | SM-MrHiSum | |||
|---|---|---|---|---|
| Script-generation | F1 | F1 | ||
| LLaVA-Next-Video-7B & prompt #1 | 25.2 | 58.8 | 0.187 | 0.253 |
| Qwen3-VL-8B-Instruct & prompt #1 | 27.0 | 58.9 | 0.199 | 0.269 |
| Qwen3-VL-8B-Instruct & prompt #2 (proposed) | 27.3 | 59.3 | 0.204 | 0.273 |
Video Summarization
Video summarization selects or generates concise representations of video while preserving important events, details, and temporal structure.
See all 23 papers