Organizations: UC Davis Davis, California, USA · Fudan University Shanghai, China · Stanford University Palo Alto, California, USA · University of Rochester Rochester, New York, USA · Meta AI Menlo Park, California, USA
Music-grounded mashup video creation is a challenging form of video non-linear editing, where a system must compose a coherent timeline from large collections of source videos while aligning with music rhythm, user intent, story completeness, and long-range structural constraints. Existing approaches typically rely on fixed pipelines or simplified retrieval-and-concatenation paradigms, limiting their ability to adapt to diverse prompts and heterogeneous source materials. In this paper, we present GLANCE, a global-local coordination multi-agent framework for music-grounded nonlinear video editing. GLANCE adopts a bi-loop architecture for better editing practice: an outer loop performs long-horizon planning and task-graph construction, and an inner loop adopts the "Observe-Think-Act-Verify" flow for segment-wise editing tasks and their refinements. To address the cross-segment and global conflict emerging after subtimelines composition, we introduce a dedicated global-local coordination mechanism with both preventive and corrective components, which includes a novelly designed context controller, conflict region decomposition module, and a bottom-up dynamic negotiation mechanism. To support rigorous evaluation, we construct MVEBench, a new benchmark that factorizes editing difficulty along task type, prompt specificity, and music length, and propose an agent-as-a-judge evaluation framework for scalable multi-dimensional assessment. Experimental results show that GLANCE consistently outperforms prior research baselines and open-source product baselines under the same backbone models. With GPT-4o-mini as the backbone, GLANCE improves over the strongest baseline by 33.2% and 15.6% on two task settings, respectively. Human evaluation further confirms the quality of the generated videos and validates the effectiveness of the proposed evaluation framework.
Figures & tables
Figure 1. The illustration of MVEBench.
Figure 2. Overview of the GLANCE framework.
On-beat Video
Story-driven Video
Method
Rhythm Alignment ↑
Emotion Alignment ↑
Instruction Following ↑
Overall Quality ↑
Story Completeness ↑
Character Continuity ↑
Instruction Following ↑
Overall Quality ↑
LLM-based Agentic Framework
CoT
2.06
1.95
1.98
1.47
2.12
2.05
2.04
1.65
GPTSwarm
2.31
2.22
2.24
1.64
2.34
2.29
2.28
1.95
Multi-modal Video Editing Agentic Framework
TeaserGen
2.01
1.90
1.94
1.76
2.43
2.33
2.34
2.10
Table 1. Overall comparison on the full MVEBench. Judge-based metrics are reported on a 1–5 scale. Best results are in bold ; second-best are underlined. All baselines utilize GPT-4o-mini as the backbone model, except for specific demonstrations for GLANCE . The results are the average score of all types of configuration as described in Section 4 .
Config
#. Samples
Avg. Music Len.
Avg. Source Video Len.
O-Sh-GP
45
27.9s
1.7h
O-Me-GP
35
83.7s
3.1h
O-Me-DP
35
83.7s
3.1h
O-Lo-DP
44
154.2s
5.5h
S-Me-GP
37
85.6s
2.9h
S-Lo-GP
45
182.1s
5.2h
Table 2. Statistics of MVEBench under different task configurations. The unit of music length is second (s), while the unit of video source length is hour (h). “#. Samples” represent the number of tasks for each configuration.
Variant
Preventive
Region Decomp.
Bottom-up Negot.
Rhythm Alignment ↑
Emotion Alignment ↑
Instruction Following ↑
Overall Quality ↑
Efficiency ↑
Only Preventive Controller
✓
3.18
3.07
3.12
3.05
1.18
w/o Bottom-up Negotiation
✓
✓
3.36
3.25
3.30
3.24
1.10
w/o Region Decomposition
✓
✓
3.46
3.35
3.39
3.33
0.76
w/o Preventive Controller
✓
✓
3.58
3.46
3.52
3.40
0.96
Full GLANCE
✓
✓
✓
3.72
3.54
3.66
3.45
1.00
Table 3. Ablation of global–local coordination modules on on-beat mashup video creation. Efficiency is reported as normalized processing efficiency measured by token usage, with higher being better.
Figure 3. Overall quality across different task configurations.
Task Type
Metric
Gemini
Qwen-30B
EditDuet
On-Beat
Rhythm Alignment
3.43
2.91
2.57
On-Beat
Emotion Alignment
3.31
2.78
2.45
On-Beat
Instruction Following
3.38
2.83
2.48
On-Beat
Overall Quality
3.11
2.54
2.29
Story-Dri.
Story Completeness
3.49
2.97
2.68
Story-Dri.
Character Continuity
3.41
2.88
2.53
Table 4. Human evaluation results on sampled On-Beat and Story-Driven subsets.
Task Type
Metric
Spearman ρ↑
Kendall τ↑
On-Beat
Rhythm Alignment
0.64
0.48
On-Beat
Emotion Alignment
0.62
0.46
On-Beat
Instruction Following
0.60
0.47
On-Beat
Overall Quality
0.66
0.50
Story-Driven
Story Completeness
0.71
0.55
Story-Driven
Character Continuity
0.68
0.52
Table 5. Agreement between the proposed judge and human evaluation.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Variant
Preventive
Region Decomp.
Bottom-up Negot.
Story Completeness ↑
Character Continuity ↑
Instruction Following ↑
Overall Quality ↑
Efficiency ↑
Only Preventive Controller
✓
3.20
3.10
3.38
3.07
1.18
w/o Bottom-up Negotiation
✓
✓
3.45
3.37
3.63
3.28
1.10
w/o Region Decomposition
✓
✓
3.56
3.48
3.72
3.34
0.76
w/o Preventive Controller
✓
✓
3.69
3.59
3.82
3.38
0.95
Full GLANCE
✓
✓
✓
3.75
3.68
3.95
3.42
1.00
Appendix
Table 6. Ablation of global–local coordination modules on story-driven mashup video creation. Efficiency is reported as normalized processing efficiency, with higher being better.
Figure 4. Evaluation scores across different task configurations..
Video mashup creation represents a complex video editing paradigm that recomposes existing footage to craft engaging audio-visual experiences, demanding intricate orchestration across semantic, visual, and auditory dimensions and multiple levels. However, existing automated editing frameworks often overlook the cross-level multimodal orchestration to achieve professional-grade fluidity, resulting in disjointed sequences with abrupt visual transitions and musical misalignment. To address this, we formulate video mashup creation as a Multimodal Coherency Satisfaction Problem (MMCSP) and propose the DIRECT framework. Simulating a professional production pipeline, our hierarchical multi-agent framework decomposes the challenge into three cascade levels: the Screenwriter for source-aware global structural anchoring, the Director for instantiating adaptive editing intent and guidance, and the Editor for intent-guided shot sequence editing with fine-grained optimization. We further introduce Mashup-Bench, a comprehensive benchmark with tailored metrics for visual continuity and auditory alignment. Extensive experiments demonstrate that DIRECT significantly outperforms state-of-the-art baselines in both objective metrics and human subjective evaluation. Project page and code: https://github.com/AK-DREAM/DIRECT
Ke Li, Maoliang Li, Jialiang Chen +4
School of Electronics Engineering and Computer Science, Peking University · School of Computer Science, Peking University · Huazhong University of Science and Technology
Editing the video content with audio alignment forms a digital human-made art in current social media. However, the time-consuming and repetitive nature of manual video editing has long been a challenge for filmmakers and professional content creators alike. In this paper, we introduce CutClaw, an autonomous multi-agent framework designed to edit hours-long raw footage into meaningful short videos that leverages the capabilities of multiple Multimodal Language Models~(MLLMs) as an agent system. It produces videos with synchronized music, followed by instructions, and a visually appealing appearance. In detail, our approach begins by employing a hierarchical multimodal decomposition that captures both fine-grained details and global structures across visual and audio footage. Then, to ensure narrative consistency, a Playwriter Agent orchestrates the whole storytelling flow and structures the long-term narrative, anchoring visual scenes to musical shifts. Finally, to construct a short edited video, Editor and Reviewer Agents collaboratively optimize the final cut via selecting fine-grained visual content based on rigorous aesthetic and semantic criteria. We conduct detailed experiments to demonstrate that CutClaw significantly outperforms state-of-the-art baselines in generating high-quality, rhythm-aligned videos. The code is available at: https://github.com/GVCLab/CutClaw.
Shifang Zhao, Yihan Hu, Ying Shan +2
Beijing Jiaotong University · GVC Lab, Great Bay University · ARC Lab, Tencent
While instruction-based video editing has advanced rapidly, real-world videos contain tightly coupled audio and visual signals, and editing one modality often requires coordinated changes in the other. Existing benchmarks primarily evaluate visual transformations on silent clips or isolated audio editing, leaving complex audio-visual editing and cross-modal consistency underexplored. We introduce AVE-Compass, a comprehensive benchmark with 145 curated source videos, 196 audio-visually coupled editing instructions, and 2,688 fine-grained checklist items. It evaluates Instruction Following, Fidelity Preserving, Realism, and Editing Intent through checklist-based MLLM judging and a dedicated realism rubric, complemented by automated cross-modal, video, and audio metrics. Extensive evaluation shows that state-of-the-art models still struggle to execute cross-modal instructions while preserving non-target content. We further propose AVE-Agent, a modular agent framework that decomposes complex instructions into dependent subtasks and iteratively improves editing results through self-reflection and evaluator feedback. AVE-Agent improves instruction execution, Fidelity Preserving, and audio-visual alignment in joint editing while maintaining competitive perceptual quality.
Yuqing Wen, Yukai Huang, Qianqian Xie +6
National University of Singapore · Beijing University of Posts and Telecommunications · NJU-LINK Team, Nanjing University +3