cs.CVMar 6, 2026
SaveCut to the Chase: Training-free Multimodal Summarization via Chain-of-Events
Organizations: School of Computer Science, Hangzhou Dianzi University · School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen) · School of Information Science and Technology, University of Science and Technology of China
Abstract
Multimodal Summarization (MMS) aims to generate concise textual summaries by understanding and integrating information across videos, transcripts, and images. However, existing approaches still suffer from three main challenges: (1) reliance on domain-specific supervision, (2) implicit fusion with weak cross-modal grounding, and (3) flat temporal modeling without event transitions. To address these issues, we introduce CoE, a training-free MMS framework that performs structured reasoning through a Chain-of-Events guided by a Hierarchical Event Graph (HEG). The HEG encodes textual semantics into an explicit event hierarchy that scaffolds cross-modal grounding and temporal reasoning. Guided by this structure, CoE localizes key visual cues, models event evolution and causal transitions, and refines outputs via lightweight style adaptation for domain alignment. Extensive experiments on eight diverse datasets demonstrate that CoE consistently outperforms state-of-the-art video CoT baselines, achieving average gains of +3.04 ROUGE, +9.51 CIDEr, and +1.88 BERTScore, highlighting its robustness, interpretability, and cross-domain generalization. Our code is available at https://github.com/youxiaoxing/CoE.
Figures & tables
Figure 1: Motivating Experiments. Existing MMS models (e.g., MLASK [ 23 ] and MMSum [ 38 ] ) achieve strong in-domain results when trained on VIEWS [ 4 ] , but their performance drops sharply under domain shift. In contrast, our training-free CoE framework generalizes effectively across diverse datasets, maintaining stable zero-shot performance without task-specific training or adaptation.
Figure 2: Framework. CoE is a training-free CoT framework for MMS with text-only output. Given a video-text pair , Module 1 constructs a Hierarchical Event Graph (HEG) that organizes global events, sub-events, and entity-relation structures. Guided by the HEG, Module 2 grounds video clips to sub-events and their visual entity-relation graphs. Module 3 models event evolution by aggregating temporally coherent clips, and finally, Module 4 generates a domain-adaptive summary from the resulting event trajectories.
Figure 3: Video Clip Aggregation. CoE merges adjacent clips grounded to the same sub-event and sharing identical entity-relation graphs into longer temporal segments. Red circles indicate new entities or sub-event changes that trigger a new segment.
| Metric | Method | Dataset | |||||||
|---|---|---|---|---|---|---|---|---|---|
| VIEWS | MM-AVS | XMSMO | TIB | VISTA | BLiSS | SoccerNet | Summ | ||
| BLEU-4 | TCoT | 1.78 | 5.03 | 1.67 | 1.17 | 2.65 | 0.44 | 0.00 | 4.27 |
| CoF | 4.09 | 4.98 | 0.51 | 1.66 | 3.69 | 0.42 | 0.42 | 1.04 | |
| ViTCoT | 3.50 | 4.26 | 1.67 | 1.68 | 4.28 | 0.45 | 1.46 | 0.80 | |
| CoS | 3.50 | 4.10 | 0.51 | 2.24 | 4.03 | 0.31 | 1.27 | 5.69 | |
| CoE | 5.36 | 6.02 | 4.08 | 2.13 | 4.80 | 0.53 | 15.44 | 4.59 | |
Table 1: Comparison with Video CoT Baselines. Quantitative results of CoE and competing video CoT methods on eight MMS datasets, all evaluated in a zero-shot, training-free setting without dataset-specific fine-tuning. For each dataset-metric pair, bold and underline denote the best and second-best results, respectively.
| Method | Dataset | |||||||
|---|---|---|---|---|---|---|---|---|
| VIEWS | MM-AVS | XMSMO | TIB | VISTA | BLiSS | SoccerNet | Summ | |
| CoE – HEG | 18.68 | 16.68 | 19.82 | 12.80 | 16.77 | 10.44 | 22.48 | 16.25 |
| CoE – CSG | 18.56 | 16.35 | 20.79 | 13.16 | 16.56 | 9.91 | 23.16 | 16.20 |
| CoE – EER | 18.66 | 19.44 | 20.88 | 13.04 | 16.53 | 9.99 | 22.44 | 15.90 |
| CoE – DSG | 18.76 | 18.70 | 21.17 | 13.38 | 16.75 | 10.80 | 22.55 | 15.67 |
| CoE | 19.28 | 19.53 | 21.28 | 13.75 | 17.00 | 10.83 | 23.50 | 16.15 |
Table 2: Ablation Study. ROUGE scores of CoE and its ablated variants across eight benchmark datasets.
| Method | Dataset | |||||||
|---|---|---|---|---|---|---|---|---|
| VIEWS | MM-AVS | XMSMO | TIB | VISTA | BLiSS | SoccerNet | Summ | |
| CoE – HEG | 17.20 | 17.01 | 44.09 | 1.33 | 10.16 | 8.02 | 5.46 | 1.89 |
| CoE – CSG | 17.78 | 16.27 | 40.47 | 1.44 | 9.98 | 7.03 | 4.60 | 3.91 |
| CoE – EER | 18.54 | 13.83 | 43.58 | 1.60 | 9.37 | 7.21 | 4.23 | 4.54 |
| CoE – DSG | 9.23 | 12.85 | 40.56 | 1.51 | 9.55 | 3.70 | 6.00 | 2.14 |
| CoE | 19.73 | 17.89 | 56.55 | 3.85 | 11.50 | 7.19 | 6.46 | 4.72 |
Table 3: Ablation Study. CIDEr scores of CoE and its ablated variants across eight benchmark datasets.
Figure 4: Backbone Generalization. Performance across different backbones in terms of ROUGE score on eight datasets. For each backbone, we report the vanilla model and its CoE-based variant, denoted as CoE( · ) . The consistent gains of CoE( · ) over the corresponding vanilla backbones demonstrate the strong generalization ability and robustness of our approach across diverse architectures.
Figure 5: Effect of Model Size. Performance of CoE with different MLLM backbones. We report results for Qwen2.5-VL models of increasing size (3B, 7B, 32B) and a proprietary GPT-5 model, showing a clear performance gain as model capacity grows across the four evaluation datasets.
Figure 6: Case Study . (a) Guided by the HEG, CoE accurately assigns sub-events to segments and grounds both visible and script-referenced entities at a fine-grained level. (b) CoE captures event evolution and adapts to the TV-script style, whereas the baseline mainly describes local scenes, missing long-range transitions and stylistic alignment.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1 : Motivating Experiments on Summ. Existing MMS models (e.g., MLASK [ 23 ] and MMSum [ 38 ] ) achieve strong in-domain results when trained on Summ [ 35 ] , but their performance drops sharply under domain shift. In contrast, our training-free CoE framework generalizes effectively across diverse datasets, maintaining stable zero-shot performance without task-specific training or adaptation.
Figure A2 : Motivating Experiments on MM-AVS. Existing MMS models (e.g., MLASK [ 23 ] and MMSum [ 38 ] ) achieve strong in-domain results when trained on MM-AVS [ 14 ] , but their performance drops sharply under domain shift. In contrast, our training-free CoE framework generalizes effectively across diverse datasets, maintaining stable zero-shot performance without task-specific training or adaptation.
Figure A3 : Motivating Experiments on XMSMO. Existing MMS models (e.g., MLASK [ 23 ] and MMSum [ 38 ] ) achieve strong in-domain results when trained on XMSMO [ 44 ] , but their performance drops sharply under domain shift. In contrast, our training-free CoE framework generalizes effectively across diverse datasets, maintaining stable zero-shot performance without task-specific training or adaptation.
Figure A4 : Motivating Experiments on TIB. Existing MMS models (e.g., MLASK [ 23 ] and MMSum [ 38 ] ) achieve strong in-domain results when trained on TIB [ 16 ] , but their performance drops sharply under domain shift. In contrast, our training-free CoE framework generalizes effectively across diverse datasets, maintaining stable zero-shot performance without task-specific training or adaptation.
Figure A5 : Motivating Experiments on VISTA. Existing MMS models (e.g., MLASK [ 23 ] and MMSum [ 38 ] ) achieve strong in-domain results when trained on VISTA [ 30 ] , but their performance drops sharply under domain shift. In contrast, our training-free CoE framework generalizes effectively across diverse datasets, maintaining stable zero-shot performance without task-specific training or adaptation.
Figure A6 : Motivating Experiments on BLiSS. Existing MMS models (e.g., MLASK [ 23 ] and MMSum [ 38 ] ) achieve strong in-domain results when trained on BLiSS [ 18 ] , but their performance drops sharply under domain shift. In contrast, our training-free CoE framework generalizes effectively across diverse datasets, maintaining stable zero-shot performance without task-specific training or adaptation.
Figure A7 : Motivating Experiments on SoccerNet. Existing MMS models (e.g., MLASK [ 23 ] and MMSum [ 38 ] ) achieve strong in-domain results when trained on SoccerNet [ 34 ] , but their performance drops sharply under domain shift. In contrast, our training-free CoE framework generalizes effectively across diverse datasets, maintaining stable zero-shot performance without task-specific training or adaptation.
| Dataset | Domain | # Train | # Val | # Test |
|---|---|---|---|---|
| VIEWS | News | 141.0 K | 1.6 K | 1.6 K |
| MM-AVS | News | 1.8 K | – | 0.4 K |
| XMSMO | News | 4.4 K | 0.3 K | 0.3 K |
| TIB | Video Lectures | 7.3 K | 0.9 K | 0.9 K |
| VISTA | Academic | 14.9 K | 1.8 K | 1.8 K |
| BLiSS | Social Media | 11.0 K | 2.5 K | 2.5 K |
Table A1: Summary of datasets used in our experiments, grouped by domain, along with the number of samples in the training, validation, and test splits.
Figure A8 : Case study on VIEWS . (a) Guided by the HEG, CoE correctly links speeches by public figures to the corresponding entities and locations, aligning visual evidence with article-referenced mentions. (b) CoE follows the evolution of the news story across scenes and generates compact news-style summaries, while the baseline mainly enumerates local visual details without capturing the full narrative.
Figure A9 : Case study on MM-AVS . (a) CoE uses the event graph as a scaffold to attach each news clip to the correct sub-event and to ground entities such as locations, organizations, and key objects in the scene. (b) By aggregating grounded clips along the event trajectories, CoE produces summaries that mirror how the news article develops, whereas the baseline remains close to scene-level description and often ignores the structure of the story.
Figure A10 : Case study on XMSMO . (a) For the Suez Canal blockage example, CoE compresses the multi-shot sequence into a short headline-style summary that closely follows the reference, while the baseline, which cannot exploit style exemplars, outputs a long verbose caption that reads like a literal description of the scene. (b) For the “Mark Zuckerberg” interview, CoE uses the hierarchical event graph and transcript to recover the correct identity and role of the speaker and to summarize the topic of the interview, whereas the baseline never names “Zuckerberg” and only provides a generic description of a news video.
Figure A11 : Case study on TIB . (a) In technical lectures about topics such as Rust packaging or knowledge management with Tiki, CoE aligns clips with the corresponding sub-events and grounds abstract entities like tools, libraries, and platforms using the slide and speaker context. (b) CoE tracks how the speaker introduces the problem, presents solutions, and discusses future work, which leads to summaries that match the logical flow of the talk, whereas the baseline mainly reports isolated visual scenes without reconstructing the full argument.
Figure A12 : Case study on VISTA . (a) For a talk on isometry and layer normalization, CoE uses the hierarchical event graph to segment the presentation into coherent conceptual blocks and to ground mathematical notions such as Gram matrix and stochastic gradient descent in the corresponding slide regions. (b) In the BOUNCE example, CoE tracks how the speaker introduces the problem, presents the algorithm based on random embeddings for mixed variables, and summarizes empirical observations, which leads to a concise summary that highlights the main scientific message. CoS also produces reasonable descriptions on VISTA, but its outputs are typically more generic and rely less on the detailed terminology that appears on the slides.
Figure A13 : Effect of Temporal Granularity. We report the ROUGE scores across eight datasets with varying video clip sizes ( ). While performance varies slightly across domains, a clip size of yields the most robust and consistent improvements.
| Method | TCoT | CoF | ViTCoT | CoS | CoE |
| Time (s) | 36.90 | 29.01 | 17.05 | 39.06 | 28.51 |
Table A2: Mean runtime comparison between CoE and video CoT baselines.
More in this topic
Video Summarization
Video summarization selects or generates concise representations of video while preserving important events, details, and temporal structure.
See all 22 papers