A Benchmark and Multi-Agent System for Instruction-driven Cinematic Video Compilation
Authors: Peixuan Zhang, Chang Zhou, Ziyuan Zhang, Hualuo Liu, Chunjie Zhang, Jingqi Liu, Xiaohui Zhou, Xi Chen, Shuchen Weng, Si Li, Boxin Shi
Organizations: School of Artificial Intelligence, Beijing University of Posts and Telecommunications · AI Technology Center, Online Video Business Unit, Tencent PCG · Tsinghua University · Beijing Academy of Artificial Intelligence · School of Software and Microelectronics, Peking University · State Key Lab of Multimedia Info. Processing, School of Computer Science, Peking University · Nat’l Eng. Research Ctr. of Visual Technology, School of Computer Science, Peking University
The surging demand for adapting long-form cinematic content into short videos has motivated the need for versatile automatic video compilation systems. However, existing compilation methods are limited to predefined tasks, and the community lacks a comprehensive benchmark to evaluate the cinematic compilation. To address this, we introduce CineBench, the first benchmark for instruction-driven cinematic video compilation, featuring diverse user instructions and high-quality ground-truth compilations annotated by professional editors. To overcome contextual collapse and temporal fragmentation, we present CineAgents, a multi-agent system that reformulates cinematic video compilation into ``design-and-compose'' paradigm. CineAgents performs script reverse-engineering to construct a hierarchical narrative memory to provide multi-level context and employs an iterative narrative planning process that refines a creative blueprint into a final compiled script. Extensive experiments demonstrate that CineAgents significantly outperforms existing methods, generating compilations with superior narrative coherence and logical coherence.
Figures & tables
Figure 1: Comparison of video compilation paradigms. (a) Existing video compilation methods typically rely on a “retrieve-and-rank” paradigm. When applied to complex cinematic videos, this isolated approach inevitably leads to contextual collapse and temporal fragmentation, leading to narratively incoherent results. (b) In contrast, our proposed CineAgents introduces a novel “design-and-compose” paradigm. Through multi-agent collaboration, the system first reverse-engineers the source video into a hierarchical narrative memory, and then performs iterative narrative planning to produce a logically coherent and compelling final compilation.
Figure 2: Overview and statistic of CineBench. (a) Examples of instructions and compiled video clips. (b) Distribution of video release years, types and instruction lengths. (c) Word cloud of instruction keyword frequencies.
Figure 3: Overview of the proposed CineAgents. Given a user instruction and a set of source videos, our system generates a final compiled video through: (a) Script reverse-engineering: The script agent integrates the multimodal information from the source videos to construct a hierarchical narrative memory, which provides a multi-level context to overcome contextual collapse ( Sec. 4.2 ). (b) Cinematic sequence production: The director agent and orchestrator agent collaborate through iterative narrative planning to generate a compiled script to overcome temporal fragmentation. Subsequently, the editor agent applies external tools to assemble and enhance the final video ( Sec. 4.3 ). This entire collaborative pipeline is orchestrated within cinematic compilation environment, where a manager agent supervises the workflow and coordinates the specialized agents to achieve the final narrative goal ( Sec. 4.4 ).
Figure 4: Qualitative comparison with state-of-the-art methods. Segments in blue are correct in both content and temporal position, aligning with the human reference (GT); green segments are semantically correct but temporally misplaced; orange segments are semantically irrelevant to the instruction; and dashed regions represent areas where the model fails to retrieve the corresponding content.
Figure 5: User study results on Live-Action and Animated videos.
Figure 6: Visual example of CineAgents compilation performance on animated content. This highlights the model’s generalization capability beyond live-action video.
School of Information and Communication Engineering, Communication University of China, Beijing, China · State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University, Beijing, China