Clips2Story: Training-Free Video Storyboarding and Editing Using Multimodal Retrieval-Embedded Generation
Organizations: New York University · University of Washington · University of Michigan
Abstract
Video editors often work with large collections of fragmented footage and must select and organize relevant clips into a coherent story. However, many existing video editing and summarization methods assume a temporally ordered source video and aim to preserve its original narrative and temporal structure. In this paper, we introduce Clips2Story, a training-free, LLM-based pipeline that enables users to create keyword-specified short videos from a pool of video clips. Given a pool of clips and a user-specified keyword, we first extract text-based multimodal descriptors for each clip and retrieve the clips most relevant to the keyword. We then provide the textual descriptors of these candidate clips to an LLM, which selects and arranges them into an editable storyboard with connective narrations. Finally, we assemble the selected clips and retrieve accompanying visuals for the narrations to compose the output video. To evaluate the proposed pipeline, we construct a synthetic evaluation dataset from 100 long-form videos across five genres by segmenting each video into short clips using a scene-change detection model and shuffling the resulting clips to remove their temporal order. Through objective evaluation and qualitative examples, we demonstrate that Clips2Story can effectively retrieve keyword-relevant clips and compose coherent storyboards based on user-specified keywords. Human evaluation further shows that Clips2Story outperforms a strong baseline in terms of narrative coherence, keyword alignment, and overall realism.
Figures & tables
| Genre | Video duration (min) (mean std. dev.) | Word density (# words/min) | Clip-level shuffling | Scene-level shuffling | ||
|---|---|---|---|---|---|---|
| Count | Duration (sec) | Count | Duration (sec) | |||
| Film | 152.43 | 656.55 | 9.66 | 164.95 | 38.33 | |
| Documentary | 161.27 | 730.75 | 4.13 | 183.55 | 16.42 | |
| Lecture | 168.50 | 99.05 | 165.22 | 25.70 | 300.32 | |
| Vlog | 186.46 | 52.65 | 74.36 | 14.05 | 198.39 | |
| News | 187.95 | 198.65 | 8.44 | 50.50 | 32.99 | |
| Model | MCD (sec) | REP (%) | CLIPScore | VTGHLS | OPC | KR-Narration | KR-Frame | KR-Caption | |
|---|---|---|---|---|---|---|---|---|---|
| Scene-level shuffling | A2Summ ( He et al., 2023 ) | ||||||||
| TeaserGen ( Xu et al., 2024 ) | |||||||||
| REGen ( Xu et al., 2025 ) | |||||||||
| Clips2Story-NF | |||||||||
| Clips2Story-ND | |||||||||
| Clip-level shuffling | A2Summ ( He et al., 2023 ) |
| Generated videos | Generated textual storyboards | |||||||
|---|---|---|---|---|---|---|---|---|
| Model | Coherence | Keyword alignment | Realness | Narration effectiveness | Clip-selection effectiveness | Organization | Keyword informativeness | Engagingness |
| REGen ( Xu et al., 2025 ) | ||||||||
| Clips2Story-NF | – | – | – | – | ||||
| Clips2Story-ND | ||||||||
| Model | MCD (sec) | REP (%) | CLIPScore | VTGHLS | KR-Narration | KR-Frame | KR-Caption | OPC |
|---|---|---|---|---|---|---|---|---|
| Clips2Story-NF (full model) | ||||||||
| w/o visual descriptors | ||||||||
| w/o transcript | ||||||||
| w/o keyword-based clip retrieval | ||||||||
| w/ video captioning | ||||||||
| w/ Qwen3-32B ( Yang et al., 2025 ) |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Task type | Inputs | Capabilities | ||||||
|---|---|---|---|---|---|---|---|---|
| Model | Summ. / Edit. | Gen. | Ord. | Unord. | Keyword guided | Clip retr. | Narr. ord. | Conn. narr. |
| MovieClip [ Bose et al., 2023 ] | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| A2Summ [ He et al., 2023 ] | ✓ | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ |
| LfVS [ Argaw et al., 2024 ] | ✓ | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ |
| TaleSumm [ Singh et al., 2024 ] | ✓ | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ |
| VideoXUM [ Lin et al., 2023b ] | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ |
| Model | MCD (sec) | REP (%) | CLIPScore | VTGHLS | OPC | KR-Narration | KR-Frame | KR-Caption |
|---|---|---|---|---|---|---|---|---|
| Documentary | ||||||||
| Input video clips | ||||||||
| A2Summ | ||||||||
| TeaserGen [ Xu et al., 2024 ] | ||||||||
| REGen [ Xu et al., 2025 ] | ||||||||
| Clips2Story-NF | ||||||||
| Model | MCD (sec) | REP (%) | CLIPScore | VTGHLS | OPC | KR-Narration | KR-Frame | KR-Caption |
|---|---|---|---|---|---|---|---|---|
| Documentary | ||||||||
| Input video clips | ||||||||
| A2Summ | ||||||||
| TeaserGen [ Xu et al., 2024 ] | ||||||||
| REGen [ Xu et al., 2025 ] | ||||||||
| Clips2Story-NF | ||||||||
Video Editing
Video editing uses AI to alter, compose, relight, or reorganize moving footage while controlling how changes persist across frames.
See all 188 papers