Organizations: School of Computer Science, Hangzhou Dianzi University · School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen) · School of Information Science and Technology, University of Science and Technology of China
Multimodal Summarization (MMS) aims to generate concise textual summaries by understanding and integrating information across videos, transcripts, and images. However, existing approaches still suffer from three main challenges: (1) reliance on domain-specific supervision, (2) implicit fusion with weak cross-modal grounding, and (3) flat temporal modeling without event transitions. To address these issues, we introduce CoE, a training-free MMS framework that performs structured reasoning through a Chain-of-Events guided by a Hierarchical Event Graph (HEG). The HEG encodes textual semantics into an explicit event hierarchy that scaffolds cross-modal grounding and temporal reasoning. Guided by this structure, CoE localizes key visual cues, models event evolution and causal transitions, and refines outputs via lightweight style adaptation for domain alignment. Extensive experiments on eight diverse datasets demonstrate that CoE consistently outperforms state-of-the-art video CoT baselines, achieving average gains of +3.04 ROUGE, +9.51 CIDEr, and +1.88 BERTScore, highlighting its robustness, interpretability, and cross-domain generalization. Our code is available at https://github.com/youxiaoxing/CoE.
Figures & tables
Figure 1: Motivating Experiments. Existing MMS models (e.g., MLASK [ 23 ] and MMSum [ 38 ] ) achieve strong in-domain results when trained on VIEWS [ 4 ] , but their performance drops sharply under domain shift. In contrast, our training-free CoE framework generalizes effectively across diverse datasets, maintaining stable zero-shot performance without task-specific training or adaptation.
Figure 2: Framework. CoE is a training-free CoT framework for MMS with text-only output. Given a video-text pair (V,T) , Module 1 constructs a Hierarchical Event Graph (HEG) that organizes global events, sub-events, and entity-relation structures. Guided by the HEG, Module 2 grounds video clips to sub-events and their visual entity-relation graphs. Module 3 models event evolution by aggregating temporally coherent clips, and finally, Module 4 generates a domain-adaptive summary s^ from the resulting event trajectories.
Figure 3: Video Clip Aggregation. CoE merges adjacent clips grounded to the same sub-event and sharing identical entity-relation graphs into longer temporal segments. Red circles indicate new entities or sub-event changes that trigger a new segment.
Metric
Method
Dataset
VIEWS
MM-AVS
XMSMO
TIB
VISTA
BLiSS
SoccerNet
Summ
BLEU-4 ↑
TCoT
1.78
5.03
1.67
1.17
2.65
0.44
0.00
4.27
CoF
4.09
4.98
0.51
1.66
3.69
0.42
0.42
1.04
ViTCoT
3.50
4.26
1.67
1.68
4.28
0.45
1.46
0.80
CoS
3.50
4.10
0.51
2.24
4.03
0.31
1.27
5.69
CoE
5.36
6.02
4.08
2.13
4.80
0.53
15.44
4.59
Table 1: Comparison with Video CoT Baselines. Quantitative results of CoE and competing video CoT methods on eight MMS datasets, all evaluated in a zero-shot, training-free setting without dataset-specific fine-tuning. For each dataset-metric pair, bold and underline denote the best and second-best results, respectively.
Method
Dataset
VIEWS
MM-AVS
XMSMO
TIB
VISTA
BLiSS
SoccerNet
Summ
CoE – HEG
18.68
16.68
19.82
12.80
16.77
10.44
22.48
16.25
CoE – CSG
18.56
16.35
20.79
13.16
16.56
9.91
23.16
16.20
CoE – EER
18.66
19.44
20.88
13.04
16.53
9.99
22.44
15.90
CoE – DSG
18.76
18.70
21.17
13.38
16.75
10.80
22.55
15.67
CoE
19.28
19.53
21.28
13.75
17.00
10.83
23.50
16.15
Table 2: Ablation Study. ROUGE scores of CoE and its ablated variants across eight benchmark datasets.
Method
Dataset
VIEWS
MM-AVS
XMSMO
TIB
VISTA
BLiSS
SoccerNet
Summ
CoE – HEG
17.20
17.01
44.09
1.33
10.16
8.02
5.46
1.89
CoE – CSG
17.78
16.27
40.47
1.44
9.98
7.03
4.60
3.91
CoE – EER
18.54
13.83
43.58
1.60
9.37
7.21
4.23
4.54
CoE – DSG
9.23
12.85
40.56
1.51
9.55
3.70
6.00
2.14
CoE
19.73
17.89
56.55
3.85
11.50
7.19
6.46
4.72
Table 3: Ablation Study. CIDEr scores of CoE and its ablated variants across eight benchmark datasets.
Figure 4: Backbone Generalization. Performance across different backbones in terms of ROUGE score on eight datasets. For each backbone, we report the vanilla model and its CoE-based variant, denoted as CoE( · ) . The consistent gains of CoE( · ) over the corresponding vanilla backbones demonstrate the strong generalization ability and robustness of our approach across diverse architectures.
Figure 5: Effect of Model Size. Performance of CoE with different MLLM backbones. We report results for Qwen2.5-VL models of increasing size (3B, 7B, 32B) and a proprietary GPT-5 model, showing a clear performance gain as model capacity grows across the four evaluation datasets.
Figure 6: Case Study . (a) Guided by the HEG, CoE accurately assigns sub-events to segments and grounds both visible and script-referenced entities at a fine-grained level. (b) CoE captures event evolution and adapts to the TV-script style, whereas the baseline mainly describes local scenes, missing long-range transitions and stylistic alignment.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1 : Motivating Experiments on Summ. Existing MMS models (e.g., MLASK [ 23 ] and MMSum [ 38 ] ) achieve strong in-domain results when trained on Summ [ 35 ] , but their performance drops sharply under domain shift. In contrast, our training-free CoE framework generalizes effectively across diverse datasets, maintaining stable zero-shot performance without task-specific training or adaptation.
Figure A2 : Motivating Experiments on MM-AVS. Existing MMS models (e.g., MLASK [ 23 ] and MMSum [ 38 ] ) achieve strong in-domain results when trained on MM-AVS [ 14 ] , but their performance drops sharply under domain shift. In contrast, our training-free CoE framework generalizes effectively across diverse datasets, maintaining stable zero-shot performance without task-specific training or adaptation.
Figure A3 : Motivating Experiments on XMSMO. Existing MMS models (e.g., MLASK [ 23 ] and MMSum [ 38 ] ) achieve strong in-domain results when trained on XMSMO [ 44 ] , but their performance drops sharply under domain shift. In contrast, our training-free CoE framework generalizes effectively across diverse datasets, maintaining stable zero-shot performance without task-specific training or adaptation.
Figure A4 : Motivating Experiments on TIB. Existing MMS models (e.g., MLASK [ 23 ] and MMSum [ 38 ] ) achieve strong in-domain results when trained on TIB [ 16 ] , but their performance drops sharply under domain shift. In contrast, our training-free CoE framework generalizes effectively across diverse datasets, maintaining stable zero-shot performance without task-specific training or adaptation.
Figure A5 : Motivating Experiments on VISTA. Existing MMS models (e.g., MLASK [ 23 ] and MMSum [ 38 ] ) achieve strong in-domain results when trained on VISTA [ 30 ] , but their performance drops sharply under domain shift. In contrast, our training-free CoE framework generalizes effectively across diverse datasets, maintaining stable zero-shot performance without task-specific training or adaptation.
Figure A6 : Motivating Experiments on BLiSS. Existing MMS models (e.g., MLASK [ 23 ] and MMSum [ 38 ] ) achieve strong in-domain results when trained on BLiSS [ 18 ] , but their performance drops sharply under domain shift. In contrast, our training-free CoE framework generalizes effectively across diverse datasets, maintaining stable zero-shot performance without task-specific training or adaptation.
Figure A7 : Motivating Experiments on SoccerNet. Existing MMS models (e.g., MLASK [ 23 ] and MMSum [ 38 ] ) achieve strong in-domain results when trained on SoccerNet [ 34 ] , but their performance drops sharply under domain shift. In contrast, our training-free CoE framework generalizes effectively across diverse datasets, maintaining stable zero-shot performance without task-specific training or adaptation.
Dataset
Domain
# Train
# Val
# Test
VIEWS
News
141.0 K
1.6 K
1.6 K
MM-AVS
News
1.8 K
–
0.4 K
XMSMO
News
4.4 K
0.3 K
0.3 K
TIB
Video Lectures
7.3 K
0.9 K
0.9 K
VISTA
Academic
14.9 K
1.8 K
1.8 K
BLiSS
Social Media
11.0 K
2.5 K
2.5 K
Appendix
Table A1: Summary of datasets used in our experiments, grouped by domain, along with the number of samples in the training, validation, and test splits.
Figure A8 : Case study on VIEWS . (a) Guided by the HEG, CoE correctly links speeches by public figures to the corresponding entities and locations, aligning visual evidence with article-referenced mentions. (b) CoE follows the evolution of the news story across scenes and generates compact news-style summaries, while the baseline mainly enumerates local visual details without capturing the full narrative.
Figure A9 : Case study on MM-AVS . (a) CoE uses the event graph as a scaffold to attach each news clip to the correct sub-event and to ground entities such as locations, organizations, and key objects in the scene. (b) By aggregating grounded clips along the event trajectories, CoE produces summaries that mirror how the news article develops, whereas the baseline remains close to scene-level description and often ignores the structure of the story.
Figure A10 : Case study on XMSMO . (a) For the Suez Canal blockage example, CoE compresses the multi-shot sequence into a short headline-style summary that closely follows the reference, while the baseline, which cannot exploit style exemplars, outputs a long verbose caption that reads like a literal description of the scene. (b) For the “Mark Zuckerberg” interview, CoE uses the hierarchical event graph and transcript to recover the correct identity and role of the speaker and to summarize the topic of the interview, whereas the baseline never names “Zuckerberg” and only provides a generic description of a news video.
Figure A11 : Case study on TIB . (a) In technical lectures about topics such as Rust packaging or knowledge management with Tiki, CoE aligns clips with the corresponding sub-events and grounds abstract entities like tools, libraries, and platforms using the slide and speaker context. (b) CoE tracks how the speaker introduces the problem, presents solutions, and discusses future work, which leads to summaries that match the logical flow of the talk, whereas the baseline mainly reports isolated visual scenes without reconstructing the full argument.
Figure A12 : Case study on VISTA . (a) For a talk on isometry and layer normalization, CoE uses the hierarchical event graph to segment the presentation into coherent conceptual blocks and to ground mathematical notions such as Gram matrix and stochastic gradient descent in the corresponding slide regions. (b) In the BOUNCE example, CoE tracks how the speaker introduces the problem, presents the algorithm based on random embeddings for mixed variables, and summarizes empirical observations, which leads to a concise summary that highlights the main scientific message. CoS also produces reasonable descriptions on VISTA, but its outputs are typically more generic and rely less on the detailed terminology that appears on the slides.
Figure A13 : Effect of Temporal Granularity. We report the ROUGE scores across eight datasets with varying video clip sizes ( K∈{2,4,6,8} ). While performance varies slightly across domains, a clip size of K=6 yields the most robust and consistent improvements.
Method
TCoT
CoF
ViTCoT
CoS
CoE
Time (s)
36.90
29.01
17.05
39.06
28.51
Appendix
Table A2: Mean runtime comparison between CoE and video CoT baselines.
The exponential growth of video content necessitates effective video summarization to efficiently extract key information from long videos. However, current approaches struggle to fully comprehend complex videos, primarily because they employ static or modality-agnostic fusion strategies. These methods fail to account for the dynamic, frame-dependent variations in modality saliency inherent in video data. To overcome these limitations, we propose TripleSumm, a novel architecture that adaptively weights and fuses the contributions of visual, text, and audio modalities at the frame level. Furthermore, a significant bottleneck for research into multimodal video summarization has been the lack of comprehensive benchmarks. Addressing this bottleneck, we introduce MoSu (Most Replayed Multimodal Video Summarization), the first large-scale benchmark that provides all three modalities. Extensive experiments demonstrate that TripleSumm achieves state-of-the-art performance, outperforming existing methods by a significant margin on four benchmarks, including MoSu. Our code and dataset are available at https://github.com/smkim37/TripleSumm.
While Multimodal Large Language Models (MLLMs) exhibit strong performance on standard video tasks, their ability to faithfully summarize and reason over complex narratives remains poorly evaluated. Existing summarization benchmarks fragment supervision across isolated granularities, such as keyframes, key shots, or disjointed text summaries, failing to capture the inherently hierarchical structure of cross-modal alignment. To address this critical gap, we introduce HAVEN, a hierarchically aligned multimodal benchmark for unified video understanding. HAVEN pioneers a fully granular (frame, shot, and video levels) and fully multimodal (video and text) dataset architecture, complete with explicit, continuous alignment between modalities. Built upon this unified annotation paradigm, we propose a comprehensive evaluation suite spanning summarization, temporal reasoning, multimodal grounding, and saliency ranking. Extensive benchmarking of state-of-the-art MLLMs exposes a persistent gap between surface-level textual fluency and grounded multimodal understanding. Ultimately, HAVEN advances the evaluation of multimodal systems beyond traditional QA formats, offering a rigorous, standardized testbed to drive future research in interpretable, hierarchical video understanding. We publicly release the dataset, benchmark suite, and evaluation protocols.
Mengqi Shi, Haopeng Zhang
Department of Information and Computer Sciences · University of Hawaii at Manoa
Retrieving events from large-scale video datasets is challenging due to complex temporal, spatial, and multimodal information. This paper presents U-CESE, our solution for the AI Challenge HCMC 2025, a Unified Clip-based Event Search Engine for multimodal event retrieval across diverse video sources. Building on CESE, U-CESE integrates its three modules into a single cohesive framework, ensuring consistent processing and retrieval across query types. A core component is the Unified Clipping Algorithm, which merges separate clipping algorithms into one efficient pipeline. To handle large-scale data, we propose DAKE, a lightweight, training-free keyframe extraction method using JPEG file size variations to identify significant scene changes. Finally, we introduce ReCap, a temporally consistent captioning framework inspired by Recurrent Neural Network, generating detailed and context-aware textual descriptions. Experiments show that U-CESE delivers robust, consistent, and efficient performance in large-scale multimodal event retrieval.
Duc-Nhuan Le, Hoang-Phuc Nguyen, Thanh-Duy Lam +2
Faculty of Information Technology, University of Science, VNU-HCM · Vietnam National University, Ho Chi Minh City, Vietnam