In this work, we present a method and two large-scale datasets for Script-Driven Multimodal Video Summarization. The proposed method, SD-MVSum, builds on our earlier SD-VSum method for script-driven video summarization. SD-VSum considered the relevance of a provided script, which outlines the desired content of the summary, only with the visual content of the video. SD-MVSum takes into account, in addition to the visual modality, the relevance of the script with the spoken content (i.e., the audio transcript) of the video. The dependence between each considered pair of data modalities, i.e., script-video and script-transcript, is modeled using a new weighted cross-modal attention mechanism. This mechanism explicitly exploits the semantic similarity between the paired modalities in order to promote the parts of the full-length video with the highest relevance to the script. Furthermore, we extend two large-scale datasets for script-driven (S-VideoXum) and generic (MrHiSum) video summarization, to make them suitable for training and evaluation of script-driven multimodal video summarization methods. Experimental comparisons document the competitiveness of the proposed SD-MVSum method against other SotA approaches for script-driven and generic video summarization. Our new method and extended datasets are available at: https://github.com/IDT-ITI/SD-MVSum.
Figures & tables
Dataset
Domains
Videos
Data modalities
Annotations per video
Type of annotations
Task
VideoXum [ 18 ] (TMM’23)
open domain
14,001
video, text (video description)
10
ground-truth video summaries, text description of the video
video summarization with multimodal output
MrHiSum [ 7 ] (NeurIPS’23)
3,509
31,892
video
1
frame-level importance scores
video summarization & highlight detection
MMSum [ 19 ] (CVPR’24)
17
5,100
video, text, transcripts video metadata
1
ground-truth video and text summary
multimodal summarization with multimodal output
S-VideoXum [ 6 ] (ACM MM’25)
open domain
11,908
video, text (summary script)
10
ground-truth video summaries, text descriptions of the summaries
script-driven video summarization
SM-VideoXum (Ours)
open domain
11,908
video, text (summary script), transcripts
10
ground-truth video summaries, text descriptions of the summaries
script-driven multimodal video summarization
SM-MrHiSum (Ours)
3,509
29,917
video, text (summary script), transcripts
1
ground-truth video summaries, text descriptions of the summaries
script-driven multimodal video summarization
TABLE I: Overview of large-scale datasets for generic (top three) and script-driven (bottom three) video summarization in the literature.
Fig. 1: Overview of the SD-MVSum network architecture. Given an input video, a script outlining the desired content of the summary, and a set of audio transcripts, SD-MVSum produces a video summary by finding associations of the script with both the visual and the spoken content in the video, using two weighted cross-modal attention mechanisms. The outputs of these mechanisms are concatenated and forwarded to a trainable Transformer-based scorer which computes frame-level importance scores. These scores are used by a frame/fragment selection component that forms the video summary given a video fragmentation and a time-budget about the summary duration.
Fig. 2: The processing pipeline in the weighted cross-modal attention mechanism when fusing the visual and the script embeddings. The dynamic scaling of attention scores is performed based on the computed cosine similarity matrix S of the input embeddings.
Fig. 3: Overview of the processing pipeline for creating the SM-VideoXum and SM-MrHiSum datasets for script-driven multimodal video summarization.
Fig. 4: A keyframe-based representation of the video “v_d3crFny-e3E”, and examples of the generated scripts for three ground-truth summaries of this video from the SM-VideoXum dataset.
Min
Max
Avg
Std
SM-VideoXum
# words per script
41
187
147.0
26.4
# sentences per script
3
25
8.5
1.8
SM-MrHiSum
# words per script
32
191
148.0
25.4
# sentences per script
1
23
8.6
1.8
TABLE II: Statistics about the scripts in the extended SM-VideoXum and SM-MrHiSum datasets.
Spoken content statistics
Dataset
# videos total
# videos w/o speech
% videos w/o speech
SM-VideoXum
11,908
3,893
32.7%
SM-MrHiSum
29,917
5,638
18.8%
For videos with speech, # sentences per transcript
Dataset
Min
Max
Avg
Std
SM-VideoXum
1
226
21.6
20.1
TABLE III: Statistics about the audio transcripts in the extended SM-VideoXum and SM-MrHiSum datasets.
Data modalities
SM-VideoXum
SM-MrHiSum
Task
Model
Script
Visual
Transcript
F1
τ
ρ
F1
τ
ρ
Script- driven summ.
SD-MVSum (proposed)
✓
✓
✓
27.3
N/A
N/A
59.3
0.204
0 .273
SD-VSum [ 6 ] (ACM MM’25)
✓
✓
X
2 4.4
N/A
N/A
57.9
0.175
0.236
CLIP-It [ 3 ] (NeurIPS’21)
✓
✓
X
22.8
N/A
N/A
56.3
0.120
0.169
Generic summ.
A2Summ [ 26 ] (CVPR’23)
X
✓
✓
21.3
0.145
0.193
5 8.0
0.169
0.239
CSTA [ 64 ] (CVPR’24)
X
✓
X
23.5
0.176
0.233
57.7
0.193
0.274
TABLE IV: Performance comparisons with SotA methods for script-driven (upper part) and generic (lower part) video summarization on SM-VideoXum and SM-MrHiSum, in terms of F-Score (%, denoted “F1”) and Kendall’s τ and Spearman’s ρ rank correlation coefficients. Best scores in bold, second-best scores underlined. Scores whose difference from those of SD-MVSum is not statistically significant are shown in italics.
Data modalities
SM-VideoXum
SM-MrHiSum
Task
Model
Script
Visual
Transcript
Scaling
F1
F1
τ
ρ
Script- driven summ.
SD-MVSum (proposed)
✓
✓
✓
✓
27.3
59.3
0.204
0.273
Variant #1
✓
✓
X
✓
26.6
58.3
0.169
0.230
Variant #2
✓
✓
✓
X
25.1
58.4
0.195
0.265
Variant #3
✓
✓
✓
convent.
24.9
58.5
0.196
0.264
TABLE V: Performance comparison with variants of SD-MVSum on SM-VideoXum and SM-MrHiSum, in terms of F-Score (%, denoted “F1”) and Kendall’s τ and Spearman’s ρ rank correlation coefficients. Best scores in bold. Scores whose difference from those of SD-MVSum is not statistically significant are shown in italics.
SM-VideoXum
SM-MrHiSum
Att. heads
F1
F1
τ
ρ
4
25.7
58.2
0.181
0.243
8 (prop.)
27.3
59.3
0.204
0.273
16
27.2
58.4
0.185
0.251
TABLE VI: Performance of SD-MVSum on SM-VideoXum and SM-MrHiSum in terms of F-Score (%) and Kendall’s τ and Spearman’s ρ rank correlation coefficients, for different numbers of attention heads. Best scores in bold. The differences between the scores attained using the proposed number of attention heads and all other examined options are statistically significant.
SM-VideoXum
Batch size
F1
2
26.7
4 (prop.)
27.3
8
26.4
TABLE VII: Performance of SD-MVSum on SM-VideoXum in terms of F-Score (%), for different training data batch sizes. Best scores in bold. The differences between the scores attained using the proposed batch size and all other examined batch sizes are statistically significant.
SM-MrHiSum
Batch size
F1
τ
ρ
32
58.4
0.176
0.237
64 (prop.)
59.3
0.204
0.273
128
58.7
0.195
0.263
TABLE VIII: Performance of SD-MVSum on SM-MrHiSum in terms of F-Score (%) and Kendall’s τ and Spearman’s ρ rank correlation coefficients, for different training data batch sizes. Best scores in bold. Scores, whose difference from those attained using the proposed batch size is not statistically significant, are shown in italics.
SM-VideoXum
SM-MrHiSum
Data fusion
F1
F1
τ
ρ
Mean pooling
26.4
58.5
0.182
0.247
Max pooling
26.2
59.0
0.184
0.248
Concat. (prop.)
27.3
59.3
0.204
0.273
TABLE IX: Performance of SD-MVSum on SM-VideoXum and SM-MrHiSum in terms of F-Score (%) and Kendall’s τ and Spearman’s ρ rank correlation coefficients, for different data fusion approaches. Best scores in bold. Scores, whose difference from those attained using the proposed fusion approach is not statistically significant, are shown in italics.
Missing data representation
SM-VideoXum
SM-MrHiSum
F1
F1
τ
ρ
Zero vectors (prop.)
27.3
59.3
0.204
0.273
Learned embeddings
26.0
58.6
0.199
0.268
TABLE X: Performance comparison with an SD-MVSum variant that employs learned embeddings when there is no spoken content (audio transcripts) in the videos of SM-VideoXum and SM-MrHiSum datasets, in terms of F-Score (%, denoted “F1”) and Kendall’s τ and Spearman’s ρ rank correlation coefficients. Best scores in bold.
SM-VideoXum
SM-MrHiSum
F1
F1
τ
ρ
SD-MVSum (prop.)
27.3
59.3
0.204
0.273
SD-MVSum ext
26.7
58.7
0.207
0.279
TABLE XI: Performance comparison with an extended variant of SD-MVSum that integrates a third weighted cross-modal attention mechanism to model the relationship between the visual content and the audio transcripts, on the SM-VideoXum and SM-MrHiSum datasets, in terms of F-Score (%, denoted “F1”) and Kendall’s τ and Spearman’s ρ rank correlation coefficients. Best scores in bold.
Visual-transcript correl. (#samples per dataset)
SM-VideoXum
SM-MrHiSum
F1
F1
τ
ρ
Low (3,490 / 35)
26.8
55.1
0.070
0.101
Medium (13,460 / 560)
27.1
59.0
0.175
0.238
High (6,120 / 956)
28.0
59.7
0.232
0.307
TABLE XII: Performance evaluation of SD-MVSum on different subsets of samples (i.e., “video-script” pairs) of the SM-VideoXum and SM-MrHiSum test sets, that exhibit different levels of correlation (low, medium, high) between the visual content and the audio transcripts, in terms of F-Score (%, denoted “F1”) and Kendall’s τ and Spearman’s ρ rank correlation coefficients. Best scores in bold.
Script-transcript correl. (#samples per dataset)
SM-VideoXum
SM-MrHiSum
F1
F1
τ
ρ
Low (1,599 / 40)
27.7
54.3
0.023
0.036
Medium (10,692 / 582)
27.1
59.2
0.206
0.278
High (10,779 / 929)
27.4
59.7
0.216
0.288
TABLE XIII: Performance evaluation of SD-MVSum on different subsets of samples (i.e., “video-script” pairs) of the SM-VideoXum and SM-MrHiSum test sets, that exhibit different levels of correlation (low, medium, high) between the script and the audio transcripts, in terms of F-Score (%, denoted “F1”) and Kendall’s τ and Spearman’s ρ rank correlation coefficients. Best scores in bold.
Transcript availability (#samples per dataset)
SM-VideoXum
SM-MrHiSum
F1
F1
τ
ρ
No (11,120 / 313)
27.2
58.7
0.185
0.250
Yes (23,070 / 1,551)
27.3
59.4
0.208
0.278
TABLE XIV: Performance evaluation of SD-MVSum on two different subsets of samples (i.e., “video-script” pairs) of the SM-VideoXum and SM-MrHiSum test set, with and without spoken content, in terms of F-Score (%, denoted “F1”) and Kendall’s τ and Spearman’s ρ rank correlation coefficients. Best scores in bold.
SM-VideoXum
SM-MrHiSum
Script-generation
F1
F1
τ
ρ
LLaVA-Next-Video-7B & prompt #1
25.2
58.8
0.187
0.253
Qwen3-VL-8B-Instruct & prompt #1
27.0
58.9
0.199
0.269
Qwen3-VL-8B-Instruct & prompt #2 (proposed)
27.3
59.3
0.204
0.273
TABLE XV: Performance of SD-MVSum on SM-VideoXum and SM-MrHiSum datasets in terms of F-Score (%) and Kendall’s τ and Spearman’s ρ rank correlation coefficients, for different script-generation approaches. Best scores in bold. Scores, whose difference from those attained using the proposed script generation model and prompt is not statistically significant, are shown in italics.
Fig. 5: An indicative example from our qualitative analysis. The upper part provides a keyframe-based representation of the content of the full-length video, and the tabular structure beneath shows the utilized input data and the generated video summary by each method.
Fig. 6: An indicative example from our qualitative analysis. The upper part provides a keyframe-based representation of the content of the full-length video, and the tabular structure beneath shows the utilized input data and the generated video summary by the SD-MVSum and SD-VSum methods.
Fig. 7: An indicative sample from our qualitative analysis. The upper part provides a keyframe-based representation of the content of the full-length video, and the tabular structure beneath shows the utilized input data, the generated video summary by the proposed SD-MVSum method, and the ground-truth summary for this video.
School of Computer Science, Hangzhou Dianzi University · School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen) · School of Information Science and Technology, University of Science and Technology of China