Auto-Director: Leveraging Multi-Agent Collaboration for Autonomous Video Editing
Authors: Yuxia Chen, Wenbo Zhu, Yangguang Ji, Zhenxiang Jiang, Teng Zhong, Yu Zhang
Organizations: University of Electronic Science and Technology of China, China · University of California, Berkeley · Nanyang Technological University · National University of Singapore · Nanjing Normal University · Beijing Freedo Technology
Autonomous video editing from open-world resources remains a challenging problem in multimedia content creation. Retrieval-based methods are constrained by fixed source material and lack creative flexibility, while generative models struggle with long-horizon coherence and often diverge from user intent. Both paradigms fail to replicate the structured, multi-stage reasoning that underlies professional post-production. To address these limitations, we present Auto-Director, a tool-augmented multi-agent framework that mirrors professional editorial workflows: a Planning Agent decomposes abstract topics into fact-grounded scripts via retrieval-augmented planning; a Material Agent indexes open-world footage through shot segmentation and visual tagging; an Editor Agent assembles semantically aligned clips with MLLM-driven spatial adaptation; and a Critic Agent governs an iterative Reflect-and-Revise loop that checks scene-level alignment and pacing. To rigorously evaluate this new paradigm, we introduce VEdit-Bench, a benchmark of 100 expert-curated topics with human-annotated Golden Scripts and a five-dimensional metric suite covering Narrative Logic, Factual Accuracy, Semantic Alignment, Compositional Quality, and Stylistic Cohesion. Extensive experiments demonstrate that Auto-Director significantly outperforms baseline methods in both automated metrics and human evaluation, effectively bridging the gap between amateurs and professional-grade video production.
Figures & tables
Figure 1: Overview of Auto-Director: a structured multi-agent pipeline with iterative reflect-and-revise feedback, addressing the limited expressivity of retrieval-based and the weak controllability of generation-based paradigms.
Method
NL
FA
SA
CQ
SC
Overall
Seedance 2.0
2.93
2.44
3.21
3.62
3.34
3.11
Kling 3.0
2.71
2.23
2.98
3.41
3.12
2.89
Veo 3.1
2.52
2.08
2.74
3.18
2.91
2.69
Sora 2
2.34
1.93
2.51
2.97
2.68
2.49
GPT-5.4
3.64
3.82
3.23
3.11
3.34
3.43
Gemini 3.1 Pro
3.51
3.74
3.14
3.02
3.22
3.33
Table 1: Quantitative evaluation on VEdit-Bench across five dimensions: Narrative Logic (NL), Factual Accuracy (FA), Semantic Alignment (SA), Compositional Quality (CQ), Stylistic Cohesion (SC). Best scores among automated methods in bold ; human editors are a reference.
Figure 2: Qualitative comparison across three cases (factual accuracy, semantic alignment, compositional quality).
Variant
NL
FA
SA
CQ
SC
Auto-Director
4.21
4.34
4.23
4.02
4.13
w/o Fact Retrieval
4.05
3.21
4.11
3.96
4.02
w/o Visual Tagging
4.03
4.19
3.54
3.82
3.71
w/o Critic Agent
3.91
3.82
3.85
3.71
3.64
Table 2: Ablation study on VEdit-Bench across the five evaluation dimensions.
Dim.
r
ρ
MAE
NL
0.73
0.70
0.31
FA
0.78
0.75
0.28
SA
0.75
0.72
0.30
CQ
0.68
0.65
0.38
SC
0.71
0.68
0.35
All
0.73
0.70
0.32
Table 3: Correlation between MLLM and human evaluation scores.
Video editing has become essential in digital media creation, yet existing automated systems are restricted to short segment processing and domain-specific tasks. They face two critical limitations: i) inability to handle diverse video comprehension and editing operations, and ii) lack of long-video understanding for coherent narrative creation. We propose VideoAgent, an all-in-one agentic framework addressing these challenges through two key innovations. First, we develop automated video shot creation with shot planning agents for coherent narratives and cross-modal retrieval for aligned visual content. Second, we design a multi-agent orchestration framework integrating over thirty specialized editing agents. Intent parsing filters relevant tools while textual-gradient graph optimization assembles complex editing pipelines. Extensive experiments on our newly-proposed VideoEdit benchmark and public datasets demonstrate VideoAgent's superiority over existing multimodal LLMs and agentic systems. VideoAgent achieves 87-95% orchestration success rates while reducing API costs by 60%. Human evaluation across six video categories shows VideoAgent produces professional-quality content approaching human-level performance, with ratings only 4% below human-created videos. We release our code at https://github.com/HKUDS/VideoAgent.
Hengji Zhou, Lingxuan Huang, Jian Wang +4
1Harbin Institute of Technology, Shenzhen · 2South China University of Technology · 3The University of Hong Kong +1
Editing the video content with audio alignment forms a digital human-made art in current social media. However, the time-consuming and repetitive nature of manual video editing has long been a challenge for filmmakers and professional content creators alike. In this paper, we introduce CutClaw, an autonomous multi-agent framework designed to edit hours-long raw footage into meaningful short videos that leverages the capabilities of multiple Multimodal Language Models~(MLLMs) as an agent system. It produces videos with synchronized music, followed by instructions, and a visually appealing appearance. In detail, our approach begins by employing a hierarchical multimodal decomposition that captures both fine-grained details and global structures across visual and audio footage. Then, to ensure narrative consistency, a Playwriter Agent orchestrates the whole storytelling flow and structures the long-term narrative, anchoring visual scenes to musical shifts. Finally, to construct a short edited video, Editor and Reviewer Agents collaboratively optimize the final cut via selecting fine-grained visual content based on rigorous aesthetic and semantic criteria. We conduct detailed experiments to demonstrate that CutClaw significantly outperforms state-of-the-art baselines in generating high-quality, rhythm-aligned videos. The code is available at: https://github.com/GVCLab/CutClaw.
Shifang Zhao, Yihan Hu, Ying Shan +2
Beijing Jiaotong University · GVC Lab, Great Bay University · ARC Lab, Tencent
Video mashup creation represents a complex video editing paradigm that recomposes existing footage to craft engaging audio-visual experiences, demanding intricate orchestration across semantic, visual, and auditory dimensions and multiple levels. However, existing automated editing frameworks often overlook the cross-level multimodal orchestration to achieve professional-grade fluidity, resulting in disjointed sequences with abrupt visual transitions and musical misalignment. To address this, we formulate video mashup creation as a Multimodal Coherency Satisfaction Problem (MMCSP) and propose the DIRECT framework. Simulating a professional production pipeline, our hierarchical multi-agent framework decomposes the challenge into three cascade levels: the Screenwriter for source-aware global structural anchoring, the Director for instantiating adaptive editing intent and guidance, and the Editor for intent-guided shot sequence editing with fine-grained optimization. We further introduce Mashup-Bench, a comprehensive benchmark with tailored metrics for visual continuity and auditory alignment. Extensive experiments demonstrate that DIRECT significantly outperforms state-of-the-art baselines in both objective metrics and human subjective evaluation. Project page and code: https://github.com/AK-DREAM/DIRECT
Ke Li, Maoliang Li, Jialiang Chen +4
School of Electronics Engineering and Computer Science, Peking University · School of Computer Science, Peking University · Huazhong University of Science and Technology