DIRECT: Video Mashup Creation via Hierarchical Multi-Agent Planning and Intent-Guided Editing

Ke Li; Maoliang Li; Jialiang Chen; Jiayu Chen; Zihao Zheng; Shaoqi Wang; Xiang Chen

cs.CVApr 6, 2026

DIRECT: Video Mashup Creation via Hierarchical Multi-Agent Planning and Intent-Guided Editing

Authors: Ke Li, Maoliang Li, Jialiang Chen, Jiayu Chen, Zihao Zheng, Shaoqi Wang, Xiang Chen

Organizations: School of Electronics Engineering and Computer Science, Peking University · School of Computer Science, Peking University · Huazhong University of Science and Technology

Abstract

Video mashup creation represents a complex video editing paradigm that recomposes existing footage to craft engaging audio-visual experiences, demanding intricate orchestration across semantic, visual, and auditory dimensions and multiple levels. However, existing automated editing frameworks often overlook the cross-level multimodal orchestration to achieve professional-grade fluidity, resulting in disjointed sequences with abrupt visual transitions and musical misalignment. To address this, we formulate video mashup creation as a Multimodal Coherency Satisfaction Problem (MMCSP) and propose the DIRECT framework. Simulating a professional production pipeline, our hierarchical multi-agent framework decomposes the challenge into three cascade levels: the Screenwriter for source-aware global structural anchoring, the Director for instantiating adaptive editing intent and guidance, and the Editor for intent-guided shot sequence editing with fine-grained optimization. We further introduce Mashup-Bench, a comprehensive benchmark with tailored metrics for visual continuity and auditory alignment. Extensive experiments demonstrate that DIRECT significantly outperforms state-of-the-art baselines in both objective metrics and human subjective evaluation. Project page and code: https://github.com/AK-DREAM/DIRECT

Figures & tables

Figure 1. Hierarchical constraints of multimodal coherency in video mashup. (1) Global Structural Alignment of narrative flow, visual elements and musical progression; (2) Local Segment Cohesion through adaptive editing intents that embody multimodal synergy; (3) Low-Level Coherency from fluid visual transitions and precise auditory-visual alignment.

Figure 2. Overview of DIRECT. We decompose video mashup creation into three collaborative modules: the Screenwriter anchors the global structure to align the multimodal content; the Director instantiates segment-level guidance (query, heuristic, pacing); and the Editor executes shot retrieval and orchestration following the editing guidance with closed-loop validation.

Figure 3. Visualization of the hierarchical planning workflow in DIRECT. The Screenwriter leverages multimodal source analysis to generate a section-wise global structural plan, and the Director expands it into segment-level editing guidance.

Figure 4. Intent-Guided Shot Sequence Editing. The Editor uses a tailored beam search algorithm with dynamic sliding-window trimming to find optimal shot sequences.

	Semantic Rele.		Visual Cont.		Auditory Align.		Human Eval.
Method	Prom. ( $m_{1}$ ) $\uparrow$	Seg. ( $m_{2}$ ) $\uparrow$	Mot. ( $m_{3}$ ) $\uparrow$	Fram. ( $m_{4}$ ) $\uparrow$	Sync. ( $m_{5}$ ) $\uparrow$	Ener. ( $m_{6}$ ) $\uparrow$	Glob. $\uparrow$	Loc. $\uparrow$	Qual. $\uparrow$
T2V* ( Xiong et al., 2022 )	25.89	81.80	61.72	76.21	91.46	52.67	3.6	2.6	2.7
MMSC ( Zhu et al., 2025a )	25.63	90.24	61.64	76.43	88.25	48.23	3.3	3.5	3.1
VideoAgent ( HKUDS, 2025 )	25.09	82.59	62.24	78.85	88.43	51.88	5.1	3.8	4.6
w/o Screenwriter	25.96	84.80	76.93	83.35	98.62	82.41	4.4	4.8	5.0
w/o Director	25.30	84.73	75.38	83.20	98.34	82.29	5.3	3.5	4.4

Table 1. Main Results on Baseline Comparison and Ablation Study. Results are averaged over all test cases for quantitative metrics

m_{1}

–

m_{6}

(scaled by 100) and a 10-cases subset for human evaluation. Best in bold, second best underlined .

Figure 5. Qualitative Comparison of Low-Level Coherency. While baseline (top row) only ensures semantic relevance, our method achieves superior visual continuity (matched subject position and motion flow across transitions) and auditory alignment (visual cut points synchronized with musical beats indicated by green crests).

Figure 6. Case study of Footage Summarization. It deconstructs the expansive footage library by clustering and captioning semantically related shots into distinct groups.

Figure 7. Effectiveness of Closed-Loop Validation. The validator detects editing failures (31% rejection rate) and prompts query adjustment to ensure sequence coherence.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S8. Illustration of Visual Continuity Metrics ( $m_{3},m_{4}$ ).

Figure S9. Visualization of Dataset Statistics.

Genre	Movie Example	Key Characteristics
Action	Mission Impossible	Dynamic motion, fast-paced
Sci-Fi	Interstellar	Grand scale, futuristic
Fantasy	The Lord of the Rings	Epic narrative, sweeping
Thriller	Joker	Psychological tension, moody
Animation	Zootopia	Vibrant visuals, expressive

Appendix

Table S2. Movie Genres of Mashup-Bench.

Count ( $B$ )	$m_{1}\uparrow$	$m_{2}\uparrow$	$m_{3}\uparrow$	$m_{4}\uparrow$	$m_{5}\uparrow$	$m_{6}\uparrow$	Latency (s) $\downarrow$
$B=1$ (Greedy)	25.22	84.78	75.10	82.36	98.73	80.24	380
$B=3$ (Ours)	25.49	84.97	77.26	83.37	98.69	82.40	646
$B=5$	25.54	85.36	77.95	83.61	98.70	83.15	891

Appendix

Table S3. Ablation on Candidate Shot Sequence Count (

B

Figure S10. Latency Breakdown and Average Time Cost.

Template Name	Suitable Scenarios	Prioritization
<Semantic_Priority>	Narrative focus or specific subjects	stricter shot pool, $m_{2}$
<Motion_Continuity_Priority>	Fluid action sequences (e.g., chases, racing)	$m_{3},m_{6}$
<Composition_Similarity_Priority>	Static framing or match-cutting (e.g., close-ups)	$m_{4}$
<Hybrid_Visual_Coherent>	Intricate action balancing motion and framing	$m_{3},m_{4}$
<Default_Priority>	Balanced, general-purpose retrieval	-

Appendix

Table S4. Predefined heuristic templates (

\mathbf{w}_{heu}

Figure S11. Representative shot sequences for each section of a complete video result. The sequences demonstrate the global structural alignment and local segment cohesion achieved by our framework.

Video Editing

Video editing uses AI to alter, compose, relight, or reorganize moving footage while controlling how changes persist across frames.

See all 179 papers

Auto-Director: Leveraging Multi-Agent Collaboration for Autonomous Video Editing
Oct 20, 2026Yuxia Chen, Wenbo Zhu, Yangguang Ji +3Video Editing Agentic Video Editing
University of Electronic Science and Technology of China, China · University of California, Berkeley · Nanyang Technological University +3
VideoAgent: All-in-One Framework for Video Understanding and Editing
Jun 22, 2026Hengji Zhou, Lingxuan Huang, Jian Wang +4Video Editing Benchmarks Agentic Video Editing
1Harbin Institute of Technology, Shenzhen · 2South China University of Technology · 3The University of Hong Kong +1
GLANCE: A Global-Local Coordination Multi-Agent Framework for Music-Grounded Non-Linear Video Editing
Apr 6, 2026Zihao Lin, Haibo Wang, Zhiyang Xu +7Video Editing Multi-Agent Systems
UC Davis Davis, California, USA · Fudan University Shanghai, China · Stanford University Palo Alto, California, USA +2

Oct 20, 2026J/K move · Enter open · S save

Auto-Director: Leveraging Multi-Agent Collaboration for Autonomous Video Editing

Yuxia Chen, Wenbo Zhu, Yangguang Ji, Zhenxiang Jiang+2

University of Electronic Science and Technology of China, China · University of California, Berkeley · Nanyang Technological University+3

Video Editing Agentic Video Editing Multi-Agent Systems Video Editing Benchmarks

Open paper AskPDF arXiv

Autonomous video editing from open-world resources remains a challenging problem in multimedia content creation. Retrieval-based methods are constrained by fixed source material and lack creative flexibility, while generative models struggle with long-horizon coherence and often diverge from user intent. Both paradigms fail to replicate the structured, multi-stage reasoning that underlies professional post-production. To address these limitations, we present Auto-Director, a tool-augmented multi-agent framework that mirrors professional editorial workflows: a Planning Agent decomposes abstract topics into fact-grounded scripts via retrieval-augmented planning; a Material Agent indexes open-world footage through shot segmentation and visual tagging; an Editor Agent assembles semantically aligned clips with MLLM-driven spatial adaptation; and a Critic Agent governs an iterative Reflect-and-Revise loop that checks scene-level alignment and pacing. To rigorously evaluate this new paradigm, we introduce VEdit-Bench, a benchmark of 100 expert-curated topics with human-annotated Golden Scripts and a five-dimensional metric suite covering Narrative Logic, Factual Accuracy, Semantic Alignment, Compositional Quality, and Stylistic Cohesion. Extensive experiments demonstrate that Auto-Director significantly outperforms baseline methods in both automated metrics and human evaluation, effectively bridging the gap between amateurs and professional-grade video production.

Abstract

Video Editing

Explore similar work

Auto-Director: Leveraging Multi-Agent Collaboration for Autonomous Video Editing

VideoAgent: All-in-One Framework for Video Understanding and Editing

GLANCE: A Global-Local Coordination Multi-Agent Framework for Music-Grounded Non-Linear Video Editing