We present Mind-of-Director, a multi-modal agent-driven framework for film previz that models the collaborative decision-making process of a film production team. Given a creative idea, Mind-of-Director orchestrates multiple specialized agents to produce previz sequences within the game engine. The framework consists of four cooperative modules: Script Development, where agents draft and refine the screenplay iteratively; Virtual Scene Design, which transforms text into semantically aligned 3D environments; Character Behaviour Control, which determines character blocking and motion; and Camera Planning, which optimizes framing, movement, and composition for cinematic camera effects. A real-time visual editing system built in the game engine further enables interactive inspection and synchronized timeline adjustment across scenes, behaviours, and cameras. Extensive experiments and human evaluations show that Mind-of-Director generates high-quality, semantically grounded previz sequences in approximately 25 minutes per idea, demonstrating the effectiveness of agent collaboration for both automated prototyping and human-in-the-loop filmmaking.
Figures & tables
Fig. 2 : Overview of the Mind-of-Director Framework. Given a high-level idea, our multi-modal agent-driven framework simulates a structured collaborative decision-making workflow through four interconnected modules: (1) Script Development refines the screenplay via a Discuss-Revise-Judge process; (2) Virtual Scene Design builds consistent 3D environments using 2D-guided and rule-based generation under spatial constraints; (3) Character Behaviour Control optimizes character blocking and motion through agent feedback; (4) Camera Planning selects and validates cinematic shots via a Debate-Judge-Validation loop for physical plausibility. All modules are integrated in Unity for real-time visualization and iterative refinement.
Fig. 3 : Qualitative Comparison. We present a representative sample from Act Ai to demonstrate our framework’s performance and cross-stage consistency. The image shows results across four stages: (a) Script Development : Comparison of screenplay generated by Solo vs. Agent Collaboration; (b) Virtual Scene Design : Comparison of scene layouts from StageDesigner and our approach with improved spatial grounding; (c) Character Behaviour Control : Character positioning from FilmAgent vs. our agent-driven method; (d) Camera Planning : Camera shot selection, comparing FilmAgent and our approach.
Evaluator
Method
CA
PA
OA
Expert
Solo
29.2%
15.8%
19.2%
Agent
70.8%
84.2%
80.8%
User
Solo
42.8%
23.0%
24.8%
Agent
57.2%
77.0%
75.2%
TABLE I : Results on Script Development . Experts and users conduct a comparative evaluation under both Solo and Agent settings. Solo corresponds to a single screenwriter baseline, where the screenplay is generated in one forward LLM pass without multi-agent reasoning or iterative revision.
Fig. 4 : Unity-Based Interface. Our system provides synchronized timeline tracks for characters and cameras, enabling real-time inspection, editing, and visualization across all stages.
Character
State
Action
Clara
standing
ID 20: Thoughtful Head Shake
Elias
sitting
ID 29: Sitting Idle
Maya
standing
ID 17: Talking
Reasons: Clara reacts with restrained discomfort; Elias remains withdrawn; Maya speaks with subtle tension.
TABLE V: An annotation example for character motions.
Type
Specs
Subjects
two_static
Relation: distant Framing: OTS_pair Shot size: MS Angle: Eye
Elias, Maya
Reason: Maintains spatial logic (Elias at S1, Maya at T4), captures face-to-face interaction, and emphasizes the attention shift.
TABLE VI: An annotation example for camera template.
Fig. 7 : Overview of Human Evaluation. The questionnaire consists of four parts: Part I (Script Development) and Part II (Virtual Scene Design) are jointly rated by both experts and general users, while Part III (Behaviour Control) and Part IV (Camera Planning) require domain expertise and are therefore evaluated exclusively by film experts.
Fig. 8 : The previs sequences of the original screenplay The Last Supper , Scene I .
Fig. 9 : The previs sequences of the original screenplay The Last Supper , Scene II .
Fig. 10 : The previs sequences of the original screenplay The Last Supper , Scene III .
Fig. 11 : The previs sequences of the movie The Social Network directed by David Fincher in 2010 .
Fig. 12 : The previs sequences of the movie Good Will Hunting directed by Gus Van Sant in 1997 .
School of Information and Communication Engineering, Communication University of China, Beijing, China · State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University, Beijing, China