cs.CVOct 29, 2026

Video Generation Meets Agent Harness: Agentic Video Generation, Agentic World Modeling and Recursive Self-Improvement

Authors: Liaoyuan Fan, Yuhao Zheng, Dongxing Mao, Hangyu Ran, Siyuan Hu, Weijia Mao, Rui Zhao, Weijia Wu, 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Video generation is shifting from end-to-end synthesis toward agentic generation, where planning, tool use, and feedback enable coherent, controllable multi-shot videos. In parallel, video world models provide action-conditioned observations and predicted futures, serving as interactive environments for agents. We study their intersection through the agent harness: an executable runtime that constructs context, exposes tools, maintains state, routes feedback, and manages budgets around video models. This report examines three directions. First, agentic video generation, where harnesses coordinate scripting, shot planning, asset management, production, and editing, alongside the evaluation challenges of these systems. Second, agentic world modeling, where harnesses mediate observation, control, execution, and state. We discuss hybrid designs in which game engines or agent-written programs maintain world state while video models render observations, and how agents and world models can mutually evaluate and improve one another. Third, recursive self-improvement (RSI) in video model development, distinguishing output revisions, updates to harness state, generator parameters, and evaluators, and changes to the improvement procedure itself. We identify how these improvements feed back into subsequent development. Across these directions, we analyze essential agent capabilities, including audiovisual perception, long-horizon planning, memory, code-based interaction, feedback-guided control, skill reuse, and multi-agent coordination. We conclude with challenges in autonomous video production, lifelong memory, programmable hybrid control, autonomous model development, provenance, cost accounting, and evaluation linking video quality and task performance to sustained improvements in models and harnesses.

Figures & tables

Figure 2: Agent involvement across three directions of video generation. Representative works branch from a shared video generation foundation into agentic world modeling, agentic video generation, and development toward recursive self-improvement (RSI). These directions connect video models to sustained world interaction, coordinated production, and the improvement of models and the agent harness. Vertical position reflects increasing agent involvement, from No Agent through Agent Assistance and Agent Guided Improvement to Self Improvement.
Survey / Study Year Scope Thematic Coverage Video Generation World Models Model Development RSI Agentic Video Gen. Agent–World Interaction Agent Design
Table 1: Comparison with related surveys and studies. Studies are grouped by primary research focus and compared in terms of scope, thematic coverage, and agent design. Our survey connects agentic video generation, agent–world interaction, and video model development toward Recursive Self-Improvement (RSI) through a unified agent-harness framework. Green checks ( ), gray checks ( ), and red crosses ( ) indicate systematic coverage, partial coverage, and topics outside a study’s scope, respectively.
More in this topic

Agentic Video Generation

Agentic video generation uses AI agents to plan, create, evaluate, and refine videos through coordinated tools and feedback.

See all 42 papers

Explore similar work

CardsList
Sep 21, 2026

VideoGen-Agent: Reinforcing Video Generation Agents

Recent advances in video generative models have enabled high-fidelity, temporally coherent video generation. However, these models often struggle to satisfy prompts requiring specialized knowledge, specific identities, physical consistency, or ordered events. In this paper, we present VideoGen-Agent, a multimodal agent trained through multitask agentic reinforcement learning to use external tools for video generation. The agent coordinates augmentation, generation, and verification tools through multi-turn interactions, using the prompt and intermediate observations to guide its decisions. We train a shared policy on a category-balanced dataset spanning six tasks. Supervised fine-tuning on teacher-generated trajectories establishes tool-use behavior, which is then refined through reinforcement learning. A category-aware hybrid reward evaluates tool-call validity, task-appropriate tool use, and generated video quality. We further introduce VABench, a held-out benchmark of 600 prompts covering procedural knowledge, single- and multi-entity identity preservation, physical consistency, scene composition, and multi-shot temporal structure. On VABench, VideoGen-Agent improves over its base text-to-video generator by 19.1 points, from 56.5 to 75.6. Upgrading the generation tools further raises the score to 86.1 without additional agent training. Human raters prefer the upgraded configuration over the strongest standalone baseline in 84.3% of comparisons. These results support learning tool use across video-generation tasks and show that the trained agent can benefit from subsequent advances in generation tools. Project page: https://andyca111.github.io/VideoGen_Agent/

Binxu Li, Haoyi Duan, Yuhui Zhang +9

Princeton University · Stanford University · UC Davis +4

RL for Video GenerationAgentic Video GenerationVideo GenerationMulti-Shot Video Generation
AskPDF
Jun 23, 2026

Autonomous Video Generation with Counterfactual Controllability for Self-Evolving World Models

Large-scale video generation models are increasingly described as world models because they can learn rich spatiotemporal regularities from visual data. However, we argue that an ideal world model should benefit in a self-evolving generative character. Traditional visually plausible predictions alone are not enough to establish whether an imagined future is physically actionable for a particular embodied agent, failing to provide informative feedback from environments for self-evolving improvement. To realize self-evolving world models, this article proposes the concept of autonomous video generation, which is evaluated through counterfactual controllability, i.e., the ability to i) generate intervention-conditioned futures, ii) bind these future frames to embodiment constraints, iii) verify them under distribution shifts, and iv) distil surviving branches into compact variables for decision-making. We formalize a four-stage closed-loop optimization of Generation, Binding, Verification and Distillation, together with four corresponding evaluation metrics: novelty, consistency, out-of-distribution (OOD) and efficiency. We further discuss two examples, i.e., drones and manipulators, as early embodied testbeds where wind, sensing limits, actuation delay, contact dynamics and recovery constraints can be systematically perturbed and verified. The central claim is that the framework of autonomous video generation for self-evolving world models should not be judged by video fidelity alone, but by whether the generated frames improve valid action under counterfactual interventions and various embodiment constraints.

Xin Wang, Wenxuan Liu, Tongtong Feng +1

Department of Computer Science and Technology, Tsinghua University, Haidian District, Beijing, 100084, China · Beijing National Research Center for Information Science and Technology, Haidian District, Beijing, 100084, China

Video GenerationControllable Video GenerationEmbodied World ModelsVideo World Models
AskPDF
Mar 3, 2026

ShareVerse: Multi-Agent Consistent Video Generation for Shared World Modeling

This paper presents ShareVerse, a video generation framework enabling multi-agent shared world modeling, addressing the gap in existing works that lack support for unified shared world construction with multi-agent interaction. ShareVerse leverages the generation capability of large video models and integrates three key innovations: 1) A dataset for large-scale multi-agent interactive world modeling is built on the CARLA simulation platform, featuring diverse scenes, weather conditions, and interactive trajectories with paired multi-view videos (front/ rear/ left/ right views per agent) and camera data. 2) We propose a spatial concatenation strategy for four-view videos of independent agents to model a broader environment and to ensure internal multi-view geometric consistency. 3) We integrate cross-agent attention blocks into the pretrained video model, which enable interactive transmission of spatial-temporal information across agents, guaranteeing shared world consistency in overlapping regions and reasonable generation in non-overlapping regions. ShareVerse, which supports 49-frame large-scale video generation, accurately perceives the position of dynamic agents and achieves consistent shared world modeling.

Jiayi Zhu, Jianing Zhang, Yiying Yang +2

Shanghai Jiao Tong University, China · Fudan University, China · StepFun, China

Video World ModelsMulti-View Video Generation
AskPDF