Video director agents orchestrate multistage production to realize users' creative visions. Emerging research on their self-evolution advances screenplay planning effectively. However, the mutual reinforcement between directing and user interaction remains largely overlooked. Improved interaction clarifies user intent and informs workflow refinement, while more reliable execution shifts collaboration from troubleshooting to creative decisions. To this end, we introduce LIT (Learning to Interact and DirecT), a framework comprises a director agent, a simulated user, and a meta engineer. Simulated user assigns creative tasks to probe the director's weaknesses, while the meta engineer uses the task traces to revise the director's interaction and production strategy. Although this approach can address discovered interaction and production issues, it still relies on passive attribution by the meta agent. This limits its ability to adapt to broader user interaction patterns and production techniques. To solve this problem, we equip LIT with a co-evolution paradigm. The meta agent proactively learns external knowledge, thereby improving its ability to discover problems and propose solutions. The meta agent, simulated user, and director together form an evolutionary closed loop, collaboratively enhances the system's directing and interaction capabilities. Across the Test and OOD Test splits, LIT improves production and Interaction scores by 19.1% and 32.0%, respectively, relative to the initial pipeline. Controlled ablations reveal the correlation between improvements in production and interaction, where the co-evolution mechanism further encourage directing skill and interaction mode exploration.
Figures & tables
Figure 1: Overview of LIT. (a) User interaction provides inspiration for improving the director’s workflow. Feedback about a change in a character’s face leads the director to create an actor card before generation. (b) The meta engineer and the agent system co-evolve through simulated user interactions. The meta engineer revises the director and simulated user. It also updates its own evaluation criteria and revision strategy. (c) The radar chart compares LIT with existing methods on visual and audio consistency, motion, plot, interaction, and intent fulfillment. (d) After evolution, the director asks for clarification and feedback on drafts. The example also shows more consistent characters and scenes.
Figure 2: LIT learns from user interaction and video production across epochs. Left: The director asks the simulated user to clarify intent and uses the answers to guide production. Center: The meta engineer analyzes videos and production traces to identify problems. It uses memory and external references to propose skill and evaluation updates. It also schedules validation and designs tasks to expose weaknesses and test new methods. Right: Accepted revisions guide the meta engineer, simulated user, and director in the next epoch.
Table 1: Comparison on Test Split and OOD Test across six dimensions. The best and second-best results are denoted by bold and underline , respectively. Ground Truth is excluded from ranking.
Figure 4: Qualitative comparison on the same plot. Baselines show duplicated characters, inconsistent props and layouts, or changes in character count and positioning. LIT shows better visual continuity in this example. Red boxes mark errors, and green boxes highlight consistent details.
Figure 5
Method
Test ↑
OOD Test ↑
Cost ↓
Consist.
Dynamic
Interact.
Intent
Consist.
Dynamic
Interact.
Intent
VLM
Image
Video
Ground Truth
0.8867
0.9610
–
–
0.8555
0.9727
–
–
–
–
–
Before Evo. (Ours)
0.6107
0.8350
0.7367
0.7352
0.6042
0.6075
0.5406
0.5609
–
–
–
LIT (Ours), GPT 5.6 luna
0.7923
0.8506
0.8281
0.7836
0.7096
0.6758
0.7531
0.6594
6.06M
385
1.01h
LIT (Ours), Gemini3.8 Flash
0.8177
0.8359
0.8734
0.8156
0.7370
0.6797
0.8031
0.6922
10.88M
448
1.18h
LIT (Ours), Qwen3.8 27B
0.7826
0.8271
0.8219
0.7789
0.7188
0.6699
0.7656
0.6609
7.73M
361
0.93h
Table 3: Ablation study on Test, OOD Test, and Cost across different methods (8 Samples per Evolution Epoch). The best and second-best results are denoted by bold and underline , respectively. Ground Truth is excluded from ranking.
Figure 6: Visual continuity before and after evolution. The examples show more consistent character appearance in the carriage scene (top) and character positioning in the office scene (bottom) after evolution. Red boxes mark inconsistencies before evolution.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Workflow and agent control in video production. Top: A framework executes six production stages in a predefined sequence. Bottom: An agent accesses the same stages as independent capabilities, selecting its next action according to context and feedback. The shared connector indicates access, not execution order.
Method
Long-horizon Video
Self Evolution
Defect Attribution
Proactive Exploration
Prompt Update
Workflow Update
User Interaction
Meta Co-evolution
ViMax ( Huang et al., 2026b )
✓
–
–
–
–
–
–
–
CHIEF ( Savytski et al., 2026 )
✓
–
–
–
✓
–
✓
–
GenMAC ( Huang et al., 2026a )
–
–
✓
–
✓
–
–
–
Co-Director ( Song et al., 2026 )
✓
–
✓
–
✓
–
–
–
AesopAgent ( Wang et al., 2024 )
✓
✓
–
–
✓
✓
–
–
VideoWeaver ( Wei et al., 2026 )
✓
✓
✓
–
✓
✓
–
–
Appendix
Table 4: Generation, interaction, and evolution mechanisms in video-production methods. ✓ : explicitly described, –: not established in the cited method. Storyboard denotes planning without end-to-end video synthesis.
Table 10
Capability
Configuration and interface
Image generation
Nano-Banana-2-Lite, default 1K, 16:9. The request includes the prompt and optional reference images. The adapter downloads the result and retains its request record.
Music
BF16 inference, seed 7, default duration 10 s. The token budget is the requested duration multiplied by 25. The server accepts 25–9,000 output tokens and returns PCM16 WAV.
Woosh-VFlow-8s, first 8 s of the reference video, Synchformer at 24 fps, seed 32001, CFG 4.5, integration tolerances 10−3 , 48 kHz WAV.
Audio separation
MelBand-RoFormer-Kim-Vocals. Inputs are converted to 44.1 kHz stereo. The output contains separate vocal and background PCM16 WAV files.
Transcription
Qwen-Audio-3.0-ASR-Flash-Filetrans, channel 0, optional language hints, asynchronous polling every 2 s with a 900 s deadline. Utterance timestamps are exported in seconds.
Appendix
Table 7: Configurations and interfaces of supporting media tools.
Number of distinct affected input–output boundaries whose improvement is claimed
Separate tests with α=αh/s
Appendix
Table 10: Design parameters for local development and validation.
u
v
90%
95%
99%
0.50
0.00
10
12
17
0.30
0.05
29
37
56
0.15
0.05
104
134
208
Appendix
Table 11: Required independent local test cases for 80% power at a single boundary. Columns vary the confidence level. u and v are planning probabilities, not measured results.
Target boundary
Positive/reference construction
Negative or contrasting construction
Plot to shot plan
Extract a contiguous screenplay–storyboard sequence. Annotate the event each shot makes visible.
Remove the shot needed to establish a specified event, or change its depicted action so that it fails the fixed event requirement.
Reference selection
Build a shot fixture with the required subject and scene references. Annotate which generation input must use each.
Omit or replace the required reference in the generation arguments. Retain the rest of the fixture.
Character or scene continuity
Extract adjacent shots with consistent identity and spatial relations. Annotate the relation being tested.
Regenerate a short shot with an omitted scene reference or an altered character/layout reference. Label the observed violation.
Audio synchronization
Extract a clip with aligned speech and image. Record the aligned interval.
Separate the voice from background audio, shift the voice, and remix it with the unchanged background.
Sound continuity
Extract a continuous scene with a coherent soundscape across its cuts.
Join shots with independently generated background tracks to create a localized soundscape discontinuity.
User response
Derive the permitted answer and response behavior from a private script, persona, and fixed conversation prefix.
Change the question or disclosure condition to require a different response. Retain the underlying fact to isolate the behavioral distinction.
Appendix
Table 12: Examples of positive and contrasting samples for local development.
Field (JSON path)
Content
Public input & interaction protocol
benchmark_case.reference_materials
Reference assets and descriptions, when available.
benchmark_case.interaction
Clarification budget and interaction mode.
Private task & persona
oracle.script
Story, characters, shot plan, and audiovisual design.
user_profile.description
Identity, knowledge, style, preferences, and patience.
Appendix
Table 13: Public input, private user context, and evaluation records in a dataset case. The director receives the opening request, references, and interaction settings. The simulated user’s private context contains the task, persona, and disclosure rules. Separate fields record evaluation requirements and construction history. Private fields are not passed directly to the director.
Intent maturity
Tasks
Reference assets
Tasks
Idea seed
17
None
22
Story outline
147
One
168
Supplied script
67
Two
39
Specified shots
9
Three or more
11
Total
240
Total
240
Appendix
Table 14: Distributions of intent maturity and reference availability across 240 tasks.
Figure 8: Illustrative responses from users with different levels of production expertise.
Method
Avg. ↑
Test ↑
OOD Test ↑
Cost ↓
Consist.
Dynamic
Interact.
Intent
Consist.
Dynamic
Interact.
Intent
VLM
Image
Video
Ground Truth
0.8867
0.9610
–
–
0.8555
0.9727
–
–
–
–
–
Before Evo. (Ours)
0.6107
0.8350
0.7367
0.7352
0.6042
0.6075
0.5406
0.5609
–
–
–
LIT (Ours), w/ Schedule
0.7690
0.7826
0.8271
0.8219
0.7789
0.7188
0.6699
0.7656
0.6609
7.73M
361
0.93h
LIT (Ours), w/o Schedule
0.7793
0.8193
0.8320
0.7625
0.7109
0.6738
0.7688
0.6391
7.65M
377
0.98h
Appendix
Table 15: Ablation study on Test, OOD Test, and Cost across different methods (8 Samples per Evolution Epoch). The best and second-best results are denoted by bold and underline , respectively. Ground Truth is excluded from ranking.
VBench-Long
ScriptAgent CriticAgent
Method
SC ↑
BC ↑
MS ↑
DD
AQ ↑
IQ ↑
CCA ↑
KBL ↑
VDF ↑
EAM ↑
NPT ↑
VideoClaw 34
0.9082
0.9292
0.9904
0.4348
0.5579
0.7053
3.958
3.896
4.104
4.042
4.271
ViMax ( Huang et al., 2026b )
0.9318
0.9484
0.9925
0.2810
0.5895
0.5924
4.417
4.208
4.542
3.625
3.938
GenMAC ( Huang et al., 2026a )
0.9440
0.9598
0.9931
0.3510
0.5769
0.6066
3.250
4.062
3.104
3.083
3.500
VideoWeaver ( Wei et al., 2026 )
0.9452
0.9536
0.9924
0.4720
0.5921
0.6475
4.396
4.083
3.625
4.292
4.021
LIT (Ours)
0.9588
0.9518
0.9907
0.5074
0.5940
0.6653
4.146
4.479
4.396
3.875
4.083
Appendix
Table 16: Quantitative comparison of five systems on VBench-Long (objective) and ScriptAgent CriticAgent (subjective) benchmarks. VBench-Long scores range from 0 to 1. CriticAgent scores range from 0 to 5. The best and second-best results are denoted by bold and underline , respectively. DD measures motion intensity and is not ranked.
Figure 9: Character and scene consistency before and after evolution across six tasks.
Figure 10: Production workflow before evolution. Shared text descriptions of characters and scenes guide timed shot prompts. Generated clips are retained in full and concatenated, then mixed with background music. The shown generation calls use no reference images or videos.
Figure 11: Production workflow after evolution. Shared character, scene, and prop references guide image preparation for each shot. Generated end frames provide visual references for later shots. The selection row distinguishes edited clips (green) from planned edits (orange), which feed into the final edit timeline.
Figure 12: Representative failure cases. Top: character appearance changes in a scene with many subjects. Bottom: a stick’s state is inconsistent with the action sequence. Red boxes highlight the errors.
Cinematic multi-shot video generation requires both narrative planning and iterative refinement. Existing methods either lack narrative reasoning or rely on feed-forward workflows without policy updates. In this paper, we propose CineEvolve, an agentic self-evolution framework for cinematic multi-shot video generation that unifies a Director and a Critic within a single vision-language policy. Emulating real-world filmmaking that couples iterative direction with critical feedback, we train CineEvolve via alternating reinforcement learning, where the Director surfaces flaws to sharpen the Critic, and the updated Critic guides the Director's targeted revisions. To stabilize co-evolution, we introduce Evolutionary GRPO (E-GRPO), leveraging historical anchoring and revision safeguarding to mitigate advantage inflation. Additionally, we contribute CineTraj, a 40K cinematic trajectory dataset spanning creation, critique, and revision. Extensive experiments demonstrate that CineEvolve outperforms existing approaches. Our code will be available at https://github.com/JingZhe-8/CineEvolve.
Jingzhe Zhang, Peixuan Zhang, Genying Li +4
School of Artificial Intelligence, Beijing University of Posts and Telecommunications · Tencent · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University +3
Autonomous video editing from open-world resources remains a challenging problem in multimedia content creation. Retrieval-based methods are constrained by fixed source material and lack creative flexibility, while generative models struggle with long-horizon coherence and often diverge from user intent. Both paradigms fail to replicate the structured, multi-stage reasoning that underlies professional post-production. To address these limitations, we present Auto-Director, a tool-augmented multi-agent framework that mirrors professional editorial workflows: a Planning Agent decomposes abstract topics into fact-grounded scripts via retrieval-augmented planning; a Material Agent indexes open-world footage through shot segmentation and visual tagging; an Editor Agent assembles semantically aligned clips with MLLM-driven spatial adaptation; and a Critic Agent governs an iterative Reflect-and-Revise loop that checks scene-level alignment and pacing. To rigorously evaluate this new paradigm, we introduce VEdit-Bench, a benchmark of 100 expert-curated topics with human-annotated Golden Scripts and a five-dimensional metric suite covering Narrative Logic, Factual Accuracy, Semantic Alignment, Compositional Quality, and Stylistic Cohesion. Extensive experiments demonstrate that Auto-Director significantly outperforms baseline methods in both automated metrics and human evaluation, effectively bridging the gap between amateurs and professional-grade video production.
Yuxia Chen, Wenbo Zhu, Yangguang Ji +3
University of Electronic Science and Technology of China, China · University of California, Berkeley · Nanyang Technological University +3
While diffusion models generate high-fidelity video clips, transforming them into coherent storytelling engines remains challenging. Current agentic pipelines automate this via chained modules but suffer from semantic drift and cascading failures due to independent, handcrafted prompting. We present Co-Director, a hierarchical multi-agent framework formalizing video storytelling as a global optimization problem. To ensure semantic coherence, we introduce hierarchical parameterization: a multi-armed bandit globally identifies promising creative directions, while a local multimodal self-refinement loop mitigates identity drift and ensures sequence-level consistency. This balances the exploration of novel narrative strategies with the exploitation of effective creative configurations. For evaluation, we introduce GenAD-Bench, a 400-scenario dataset of fictional products for personalized advertising. Experiments demonstrate that Co-Director significantly outperforms state-of-the-art baselines, offering a principled approach that seamlessly generalizes to broader cinematic narratives. Project Page: https://co-director-agent.github.io/