Learning to Interact and Direct: Co-Evolving Video Director Agents
Organizations: College of Artificial Intelligence, Zhejiang University.
Abstract
Video director agents orchestrate multistage production to realize users' creative visions. Emerging research on their self-evolution advances screenplay planning effectively. However, the mutual reinforcement between directing and user interaction remains largely overlooked. Improved interaction clarifies user intent and informs workflow refinement, while more reliable execution shifts collaboration from troubleshooting to creative decisions. To this end, we introduce LIT (Learning to Interact and DirecT), a framework comprises a director agent, a simulated user, and a meta engineer. Simulated user assigns creative tasks to probe the director's weaknesses, while the meta engineer uses the task traces to revise the director's interaction and production strategy. Although this approach can address discovered interaction and production issues, it still relies on passive attribution by the meta agent. This limits its ability to adapt to broader user interaction patterns and production techniques. To solve this problem, we equip LIT with a co-evolution paradigm. The meta agent proactively learns external knowledge, thereby improving its ability to discover problems and propose solutions. The meta agent, simulated user, and director together form an evolutionary closed loop, collaboratively enhances the system's directing and interaction capabilities. Across the Test and OOD Test splits, LIT improves production and Interaction scores by 19.1% and 32.0%, respectively, relative to the initial pipeline. Controlled ablations reveal the correlation between improvements in production and interaction, where the co-evolution mechanism further encourage directing skill and interaction mode exploration.
Figures & tables
| Method | Test Split Scores | OOD Test Scores | ||||||||||
| Visual | Audio | Motion | Plot | Inter. | Intent | Visual | Audio | Motion | Plot | Inter. | Intent | |
| Ground Truth | 0.8828 | 0.8906 | 0.9219 | 1.0000 | – | – | 0.8203 | 0.8906 | 0.9453 | 1.0000 | – | – |
| GPT-6-Astra 1 1 1 https://openai.com/index/gpt-6-astra/ . | 0.7995 | 0.8151 | 0.8398 | 0.8730 | 0.8625 | 0.7891 | 0.7500 | 0.7109 | 0.6328 | 0.7695 | 0.6813 | 0.6188 |
| MovieAgent ( Wu et al., 2025 ) | 0.4557 | 0.6654 | 0.7695 | 0.8223 | – | – | 0.5990 | 0.5651 | 0.5469 | 0.6680 | – | – |
| VideoClaw 2 2 2 VideoClaw repository: https://github.com/HITsz-TMG/VideoClaw . | 0.5156 | 0.7565 | 0.7617 | 0.8965 | 0.8398 | 0.8227 | 0.6589 | 0.7005 | 0.6094 | 0.8086 | 0.7094 | 0.6375 |
| ViMax ( Huang et al., 2026b ) | 0.4909 | 0.6901 | 0.8027 | 0.8672 | 0.7695 | 0.7797 | 0.5859 | 0.5469 | 0.5703 | 0.7383 | 0.5828 | 0.5766 |
| Method | Test | OOD Test | Cost | ||||||||
| Consist. | Dynamic | Interact. | Intent | Consist. | Dynamic | Interact. | Intent | VLM | Image | Video | |
| Ground Truth | 0.8867 | 0.9610 | – | – | 0.8555 | 0.9727 | – | – | – | – | – |
| Before Evo. (Ours) | 0.6107 | 0.8350 | 0.7367 | 0.7352 | 0.6042 | 0.6075 | 0.5406 | 0.5609 | – | – | – |
| LIT (Ours), GPT 5.6 luna | 0.7923 | 0.8506 | 0.8281 | 0.7836 | 0.7096 | 0.6758 | 0.7531 | 0.6594 | 6.06M | 385 | 1.01h |
| LIT (Ours), Gemini3.8 Flash | 0.8177 | 0.8359 | 0.8734 | 0.8156 | 0.7370 | 0.6797 | 0.8031 | 0.6922 | 10.88M | 448 | 1.18h |
| LIT (Ours), Qwen3.8 27B | 0.7826 | 0.8271 | 0.8219 | 0.7789 | 0.7188 | 0.6699 | 0.7656 | 0.6609 | 7.73M | 361 | 0.93h |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Long-horizon Video | Self Evolution | Defect Attribution | Proactive Exploration | Prompt Update | Workflow Update | User Interaction | Meta Co-evolution |
| ViMax ( Huang et al., 2026b ) | – | – | – | – | – | – | – | |
| CHIEF ( Savytski et al., 2026 ) | – | – | – | – | – | |||
| GenMAC ( Huang et al., 2026a ) | – | – | – | – | – | – | ||
| Co-Director ( Song et al., 2026 ) | – | – | – | – | – | |||
| AesopAgent ( Wang et al., 2024 ) | – | – | – | – | ||||
| VideoWeaver ( Wei et al., 2026 ) | – | – | – |
| Capability | Configuration and interface |
| Image generation | Nano-Banana-2-Lite, default 1K, 16:9. The request includes the prompt and optional reference images. The adapter downloads the result and retains its request record. |
| Music | BF16 inference, seed 7, default duration 10 s. The token budget is the requested duration multiplied by 25. The server accepts 25–9,000 output tokens and returns PCM16 WAV. |
| Text-to-sound | Woosh-DFlow, 5 s, seed 31001, Euler, 4 steps, CFG 4.5, renoising schedule , 48 kHz WAV. |
| Video-to-sound | Woosh-VFlow-8s, first 8 s of the reference video, Synchformer at 24 fps, seed 32001, CFG 4.5, integration tolerances , 48 kHz WAV. |
| Audio separation | MelBand-RoFormer-Kim-Vocals. Inputs are converted to 44.1 kHz stereo. The output contains separate vocal and background PCM16 WAV files. |
| Transcription | Qwen-Audio-3.0-ASR-Flash-Filetrans, channel 0, optional language hints, asynchronous polling every 2 s with a 900 s deadline. Utterance timestamps are exported in seconds. |
| Role | Examples |
| Production operations | Image/video generation, frame extraction, transcription, audio separation, clip assembly. |
| Directing procedures | Shot planning, H3 prompt construction, reference-asset selection, spatial-anchor instructions. |
| Runtime operations | Agent delegation, session continuation, task-result submission. |
| Optional visual analysis | DINOv3 features, SAM 3 segmentation, RelateAnything relations, Orient Anything V2 orientation, Depth Anything 3 geometry. |
| Optional 3D workbench | Panorama conditioning, SHARP Gaussian reconstruction, depth alignment, splat-to-voxel conversion. |
| Reference type | Records |
| Documents and prompt articles | 355 |
| Skills | 21 |
| Workflows | 85 |
| Code references | 12 |
| Contiguous storyboard scenes | 263 |
| Manual-feedback records | 17 |
| Parameter | Categories | Consequence |
| Difficulty | 1: direct deterministic check 2: perceptual decision within one stage 3: context-dependent or cross-stage judgment | development and validation cases |
| Importance | 1: local optional quality 2: recurring quality requirement 3: task-critical failure | |
| Scope | Number of distinct affected input–output boundaries whose improvement is claimed | Separate tests with |
| 90% | 95% | 99% | ||
| 0.50 | 0.00 | 10 | 12 | 17 |
| 0.30 | 0.05 | 29 | 37 | 56 |
| 0.15 | 0.05 | 104 | 134 | 208 |
| Target boundary | Positive/reference construction | Negative or contrasting construction |
| Plot to shot plan | Extract a contiguous screenplay–storyboard sequence. Annotate the event each shot makes visible. | Remove the shot needed to establish a specified event, or change its depicted action so that it fails the fixed event requirement. |
| Reference selection | Build a shot fixture with the required subject and scene references. Annotate which generation input must use each. | Omit or replace the required reference in the generation arguments. Retain the rest of the fixture. |
| Character or scene continuity | Extract adjacent shots with consistent identity and spatial relations. Annotate the relation being tested. | Regenerate a short shot with an omitted scene reference or an altered character/layout reference. Label the observed violation. |
| Audio synchronization | Extract a clip with aligned speech and image. Record the aligned interval. | Separate the voice from background audio, shift the voice, and remix it with the unchanged background. |
| Sound continuity | Extract a continuous scene with a coherent soundscape across its cuts. | Join shots with independently generated background tracks to create a localized soundscape discontinuity. |
| User response | Derive the permitted answer and response behavior from a private script, persona, and fixed conversation prefix. | Change the question or disclosure condition to require a different response. Retain the underlying fact to isolate the behavioral distinction. |
| Field (JSON path) | Content |
| Public input & interaction protocol | |
| benchmark_case.reference_materials | Reference assets and descriptions, when available. |
| benchmark_case.interaction | Clarification budget and interaction mode. |
| Private task & persona | |
| oracle.script | Story, characters, shot plan, and audiovisual design. |
| user_profile.description | Identity, knowledge, style, preferences, and patience. |
| Intent maturity | Tasks | Reference assets | Tasks |
| Idea seed | 17 | None | 22 |
| Story outline | 147 | One | 168 |
| Supplied script | 67 | Two | 39 |
| Specified shots | 9 | Three or more | 11 |
| Total | 240 | Total | 240 |
| Method | Avg. | Test | OOD Test | Cost | ||||||||
| Consist. | Dynamic | Interact. | Intent | Consist. | Dynamic | Interact. | Intent | VLM | Image | Video | ||
| Ground Truth | 0.8867 | 0.9610 | – | – | 0.8555 | 0.9727 | – | – | – | – | – | |
| Before Evo. (Ours) | 0.6107 | 0.8350 | 0.7367 | 0.7352 | 0.6042 | 0.6075 | 0.5406 | 0.5609 | – | – | – | |
| LIT (Ours), w/ Schedule | 0.7690 | 0.7826 | 0.8271 | 0.8219 | 0.7789 | 0.7188 | 0.6699 | 0.7656 | 0.6609 | 7.73M | 361 | 0.93h |
| LIT (Ours), w/o Schedule | 0.7793 | 0.8193 | 0.8320 | 0.7625 | 0.7109 | 0.6738 | 0.7688 | 0.6391 | 7.65M | 377 | 0.98h | |
| VBench-Long | ScriptAgent CriticAgent | ||||||||||
| Method | SC | BC | MS | DD | AQ | IQ | CCA | KBL | VDF | EAM | NPT |
| VideoClaw 34 | 0.9082 | 0.9292 | 0.9904 | 0.4348 | 0.5579 | 0.7053 | 3.958 | 3.896 | 4.104 | 4.042 | 4.271 |
| ViMax ( Huang et al., 2026b ) | 0.9318 | 0.9484 | 0.9925 | 0.2810 | 0.5895 | 0.5924 | 4.417 | 4.208 | 4.542 | 3.625 | 3.938 |
| GenMAC ( Huang et al., 2026a ) | 0.9440 | 0.9598 | 0.9931 | 0.3510 | 0.5769 | 0.6066 | 3.250 | 4.062 | 3.104 | 3.083 | 3.500 |
| VideoWeaver ( Wei et al., 2026 ) | 0.9452 | 0.9536 | 0.9924 | 0.4720 | 0.5921 | 0.6475 | 4.396 | 4.083 | 3.625 | 4.292 | 4.021 |
| LIT (Ours) | 0.9588 | 0.9518 | 0.9907 | 0.5074 | 0.5940 | 0.6653 | 4.146 | 4.479 | 4.396 | 3.875 | 4.083 |
Agentic Video Generation
Agentic video generation uses AI agents to plan, create, evaluate, and refine videos through coordinated tools and feedback.
See all 49 papers