Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generation

Weijie Wang; Haomin Zhang; Ye Li; Akide Liu; Zeyu Zhang; Haoyu Zhao; Yefei He; Donny Y. Chen

cs.CVOct 14, 2026

Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generation

Authors: Weijie Wang, Haomin Zhang, Ye Li, Akide Liu, Zeyu Zhang, Haoyu Zhao, Yefei He, Donny Y. Chen,

Organizations: Zhejiang University · Monash University · BAIR, UC Berkeley · Adelaide University

Abstract

Long-form agentic video generation must turn a screenplay into shots that remain spatially and temporally consistent across nonconsecutive requests. Passing prompts or recent frames between shots does not provide a shared spatial frame for revisiting a scene or a time-indexed record for replaying an earlier event. We present Any Time, Any Space (ATAS), which compiles a screenplay into shot requests with explicit location, camera, and event-time references. Before generation, it retrieves relevant evidence from a scene memory of registered static geometry and an event memory of dynamic observations indexed by scene, event, and time. After generation, ATAS reconstructs and registers the clip. It updates memory only when review and state checks accept the observation, while film delivery remains independent of memory admission. Across four video generators, ATAS improves all reported cross-shot consistency measures and achieves the highest aggregate benchmark score for each backbone. It also outperforms the compared agents and 3D-guided methods on target-image spatial revisit, leads three event-recapture criteria, and receives the highest human ratings on all six criteria. The video results and code are available on our project page: https://lhmd.top/atas/.

Figures & tables

Fig. 1: World-state-centric agentic video generation. ATAS treats a multi-shot film as a sequence whose spatial and temporal information can be reused. Shared scene geometry anchors consistent return views, while time-indexed event evidence supports replay from a new camera. The generated shots form the visible film, and the resulting scene and event information guides subsequent requests.

Fig. 2: Workflow of ATAS. A high-level prompt is parsed into multi-shot requests and bounded shot requests. Each request contains a description, scene, spatial anchor, camera, time interval, and optional event. The shot prompt and the context-dependent prompt condition the video generator. Scene memory supplies static point-cloud references for the requested scene, while event memory supplies dynamic references for the requested event. Shot validation either requests another candidate or sends the accepted observation to the state update. The generated shot remains available as film independently of whether the observation is added to persistent state.

Fig. 3: Qualitative world-state rollout. Film frames and registered world views are shown for two story sequences. The examples illustrate how accepted observations extend the scene and event state across intervening shots.

Model	Method	VBench	ST-Bench	ViStory- Self	ViStory- Cross	Avg.
Open-Source Models
MiniMax-H3	I2V Extension	0.828 $\pm$ 0.013	0.572 $\pm$ 0.020	0.601 $\pm$ 0.028	0.418 $\pm$ 0.027	0.605
	VGoT	0.834 $\pm$ 0.014	0.612 $\pm$ 0.017	0.638 $\pm$ 0.023	0.468 $\pm$ 0.030	0.638
	Mora	0.836 $\pm$ 0.012	0.598 $\pm$ 0.019	0.628 $\pm$ 0.026	0.452 $\pm$ 0.024	0.628
	MovieAgent	0.852 $\pm$ 0.012	0.608 $\pm$ 0.016	0.642 $\pm$ 0.025	0.476 $\pm$ 0.023	0.644
	ReCA	0.844 $\pm$ 0.011	0.628 $\pm$ 0.018	0.658 $\pm$ 0.021	0.508 $\pm$ 0.022	0.659

Tab. 1: Agent-level quantitative evaluation on standard benchmarks. Models are grouped by source type and rendering backbone. Methods within a block share the generator, prompt package, duration budget, and evaluation protocol. Bold and underlined entries mark the best and second-best values within each backbone block.

Fig. 4: Spatial revisit and historical event recapture. The top row uses a target image for spatial comparison. The bottom row uses source-grounded event evidence under a requested camera.

Table 6

Fig. 5: Human evaluation of agent methods. Grouped bars show mean ratings and standard-error whiskers for six methods across six criteria.

Configuration	ViStory average $\uparrow$	C-Top10 $\uparrow$	LPIPS C $\downarrow$	Event Recapture Score $\uparrow$
ATAS w/ shot rendering	0.478	0.528	0.341	0.441
ATAS w/ scene and event resolution	0.612	0.568	0.334	0.508
ATAS w/ recursive compilation	0.638	0.684	0.288	0.711
ATAS w/ observation validation	0.651	0.702	0.286	0.726
ATAS w/ state update check	0.670	0.719	0.284	0.739

Tab. 4: Agent-component ablation. All rows use Wan 3.0 and the same requests, cameras, candidate budget, and retry policy. Components are added cumulatively from shot rendering through memory resolution, recursive compilation, observation validation, and state-update checking.

Fig. 6: Efficiency comparison across memory settings. Panels (a), (b), and (c) show runtime, candidate lifecycle, and retry and delivery measures across six memory settings.

Config.	C-Top10 $\uparrow$	LPIPS ${}_{\mathrm{C}}\downarrow$	Recap. $\uparrow$
ATAS w/o memory	0.536	0.341	0.452
ATAS w/ frame history	0.598	0.319	0.548
ATAS w/ static scene	0.617	0.289	0.583
ATAS w/ updated scene	0.658	0.286	0.585
ATAS w/ scene + event	0.703	0.285	0.728
ATAS w/ full state	0.719	0.284	0.739

Tab. 5: Memory analysis. ATAS with Wan 3.0 is fixed. Rows vary stored evidence and state checks.

Appendix figures & tables40 assets

Supplementary material from the paper’s appendix.

Appendix

Dimension	Wan 2.7 ATAS	Wan 3.0 ATAS	MiniMax-H3 ATAS	HappyHorse ATAS
Aesthetic quality	0.6387	0.6731	0.6945	0.6792
Subject consistency	0.9481	0.9643	0.9576	0.9418
Background consistency	0.9554	0.9717	0.9680	0.9522
Temporal flickering	0.9932	0.9863	0.9847	0.9884
Motion smoothness	0.9888	0.9766	0.9714	0.9795
Dynamic degree	0.5488	0.6882	0.5842	0.6314

Appendix

Tab. A: VBench dimension breakdown. Each row is one reported VBench dimension, and each method/backbone column uses the official VBench evaluator and prompt suite. Higher is better unless the official dimension defines the opposite direction.

Metric	Wan 2.7 ATAS	Wan 3.0 ATAS	MiniMax-H3 ATAS	HappyHorse ATAS	ReCA reference
Aesthetic	0.6387	0.6731	0.6945	0.6792	0.6517
PF-G	0.2174	0.2415	0.2351	0.2289	0.2216
PF-S	0.2154	0.2362	0.2308	0.2243	0.2192
C-All	0.5977	0.6844	0.6529	0.6615	0.5751
C-Top10	0.6253	0.7187	0.6908	0.6684	0.6015

Appendix

Tab. B: ST-Bench metric breakdown. The table reports official scores for aesthetic quality, global and single-shot prompt following, all-pair cross-shot consistency, and top-ten-pair cross-shot consistency.

Metric	Wan 2.7 ATAS	Wan 3.0 ATAS	MiniMax-H3 ATAS	HappyHorse ATAS	ReCA reference
Identity consistency	0.5494	0.6617	0.6421	0.5974	0.5608
Appearance consistency	0.6836	0.8139	0.7476	0.7772	0.7034
Temporal self-consistency	0.7648	0.7908	0.7392	0.7425	0.7186
Aggregate	0.6659	0.7555	0.7096	0.7057	0.6609

Appendix

Tab. C: ViStory-Self metric breakdown. Rows report the supplied within-story consistency dimensions and aggregate.

Metric	Wan 2.7 ATAS	Wan 3.0 ATAS	MiniMax-H3 ATAS	HappyHorse ATAS	ReCA reference
Cross-shot identity	0.5073	0.5992	0.5736	0.5428	0.4934
Cross-shot appearance	0.4871	0.5806	0.5392	0.5618	0.5013
Cross-shot scene consistency	0.4621	0.5729	0.5285	0.5138	0.4484
Aggregate	0.4855	0.5842	0.5471	0.5395	0.4810

Appendix

Tab. D: ViStory-Cross metric breakdown. Rows report the supplied cross-shot consistency dimensions and aggregate.

Fig. A: Recursive screenplay compilation. The compiler turns a high-level request into bounded shot requests and uses validated state updates to condition subsequent requests.

Fig. B: Scene and event memory. The figure follows one accepted shot-local observation as static evidence is written to scene memory and dynamic evidence is indexed in event memory, then shows how both references condition a later camera request.

Checkpoint	Scene size ( $\times 10^{6}$ )	Event records	Storage (GB)	Query and update time (s)	Processed shots
Initial state	33.54	1.0	0.95	93.0	1.2
After accepted update	40.92	2.1	1.38	97.8	2.8
End of rollout	49.60	4.8	2.44	103.5	6.9

Appendix

Tab. E: Memory statistics. This analysis table reports memory growth and query cost at rollout checkpoints. It is not a sample manifest.

Fig. C: Memory capacity and update sensitivity. The figure shows whether quality gains require additional state size or frequent updates.

Configuration	Capacity ( $\times 10^{6}$ )	Update interval	C-Top10	LPIPS C	Event score
Full update	49.60	1	0.719	0.284	0.739
Sparse update	49.60	3	0.685	0.286	0.732
Small capacity	12.00	1	0.638	0.301	0.701
Short event history	49.60	1	0.712	0.284	0.608

Appendix

Tab. F: Memory capacity and update sensitivity. The table supplies the values used by Fig. C .

Fig. D: Memory freshness and staleness. The figure tests whether quality degrades as future requests operate farther from the most recent accepted observation.

Shots since update	C-Top10	LPIPS C	Event Recapture Score
0	0.736	0.276	0.762
1 to 2	0.722	0.283	0.744
3 to 4	0.691	0.292	0.712
5+	0.651	0.306	0.678

Appendix

Tab. G: Memory freshness data. The table supplies the values used by Fig. D .

Fig. E: State update check trade-off. The figure shows how stricter admission changes state coverage and downstream quality.

State update setting	State update rate	C-Top10	LPIPS C	Event score
Permissive	90.0	0.696	0.293	0.708
Default	67.4	0.719	0.284	0.739
Strict	36.0	0.668	0.288	0.682

Appendix

Tab. H: State update check sensitivity. The table supplies the values used by Fig. E . It reports no independent benchmark.

Fig. F: Human evaluation interface. Each page shows one anonymized long video with its story title and synopsis. Raters watch the whole video and score it on the six criteria from 1 (very poor) to 5 (excellent). The method identity is hidden, and all six criteria must be rated before moving to the next video.

Fig. G: Distribution of human judgments. The heatmaps report mean ratings and the fraction of ratings at least four for each criterion and method. The side panel summarizes inter-rater agreement.

Fig. H: Automatic-to-human agreement. Spearman correlations compare the event-recapture and spatial-revisit diagnostics with human ratings on matched clips, alongside the human–human agreement ceiling.

Fig. I: Qualitative baseline audit. Matched frames from the record-shop and bakery stories compare methods across shot continuation, scene layout, object state, and character consistency.

Fig. J: Representative rollout timeline. The timeline follows the generated film and memory condition through scene changes, a spatial revisit, and an event recapture request.

Fig. K: Representative multi-story results. The grid shows timestamped frames from six stories, illustrating the range of scenes, actions, and camera requests used in the qualitative evaluation.

Fig. G.1: Saturday Laundry in the Flat: overview. He leaves the living room for the laundry room, adds detergent and closes the washer, sits and then lies down on the bed, opens the curtains, and later gets up from the sofa to walk back to the laundry room.

Fig. G.2: Saturday Laundry, visit 1: living room (0:02). The first view of the living room, with the sofa, the gray throw, the coffee table, and the TV unit.

Fig. G.3: Saturday Laundry, visit 2: laundry room (0:22). He loads the clothes from the basket into the washer.

Fig. G.4: Saturday Laundry, visit 3: bedroom (1:07). He sits in the armchair after opening the curtains.

Fig. G.5: Saturday Laundry, visit 4: living room, return (1:25). Back on the sofa, with the mug still on the coffee table where he set it down.

Fig. G.6: Saturday Laundry, visit 5: laundry room, return (1:45). He opens the washer he started more than a minute earlier.

Fig. G.7: Saturday Laundry in the Flat: returns. Top: the mug set down on the coffee table at the start is still there more than a minute later, next to the same sofa and throw. Bottom: the laundry room keeps its shelf, towel, basket, and drying rack, and the washer he started is still shut with the laundry inside.

Fig. G.8: Winter Fire in the Cabin: overview. She builds the fire from the log basket and watches it catch, stands by the frosted kitchen window with the heart she drew, rests in the bedroom, and returns to tend the fire, which has grown while she was away.

Fig. G.9: Winter Fire, visit 1: living room (0:12). She lights the fire in the stone hearth.

Fig. G.10: Winter Fire, visit 2: kitchen (0:35). She draws in the frost on the kitchen window.

Fig. G.11: Winter Fire, visit 3: bedroom (1:10). She lies on the bed under the lamp.

Fig. G.12: Winter Fire, visit 4: kitchen, return (1:30). The kitchen as she left it, with the heart still in the frost on the window.

Fig. G.13: Winter Fire, visit 5: living room, return (2:05). She sits by the fire she lit before leaving the room.

Fig. G.14: Winter Fire in the Cabin: returns. Top: the heart she drew in the frost is still in the lower right pane of the same kitchen window. Bottom: the living room keeps its stone fireplace, log basket, lamp, and sofa with the red plaid throw, while the hearth that was cold at the start now holds the fire she lit.

Fig. G.15: Running a Bath on a Winter Evening: overview. She leaves the laptop on the bed, turns on the tap, lights a candle as the tub fills with pink water, browses the bookshelf in the living room, opens the wardrobe back in the bedroom, and tests the water in the filled tub.

Fig. G.16: Running a Bath, visit 1: bedroom (0:07). She closes the laptop on the bed.

Fig. G.17: Running a Bath, visit 2: bathroom (0:30). The tub begins to fill with pink water.

Fig. G.18: Running a Bath, visit 3: living room (1:07). She rests on the sofa while the bath runs.

Fig. G.19: Running a Bath, visit 4: bedroom, return (1:30). She sits next to the laptop she left on the bed.

Fig. G.20: Running a Bath, visit 5: bathroom, return (2:05). She kneels by the tub, which has filled while she was away.

Fig. G.21: Running a Bath on a Winter Evening: returns. Top: from the same view, the bedroom matches the first visit, with the bed, nightstand lamp, wardrobe, and pink curtains, and the laptop she closed is still lying on the bed. Bottom: the tiles, glass shelf, mirror, and towel rail match her first entrance, while the empty tub has filled with pink water and the candle she lit is still burning.

Agentic Video Generation

Agentic video generation uses AI agents to plan, create, evaluate, and refine videos through coordinated tools and feedback.

See all 40 papers

MVAgent: Multi-Agent Video Generation via Consistent Condition Construction and Shot-Level Policy Optimization
Sep 24, 2026Xiangyu Kong, Wenjie Zhou, Fengping Tian +5Agentic Video Generation Multi-Shot Video Generation
University of Exeter, United Kingdom · Alibaba Group, China
GroundShot: Visually Consistent Multi-Shot Long Video Generation via Entity-Grounded Shot Scheduling
Jun 18, 2026Yixuan Lai, Tianjia Shao, Kun Zhou +3Long Video Generation Memory-Augmented Video Generation
State Key Lab of CAD&CG Zhejiang University Hangzhou, China · Fudan University Shanghai, China · Baidu Beijing, China
Soap2Soap: Long Cinematic Video Remaking via Multi-Agent Collaboration
May 17, 2026Yiren Song, Huilin Zhong, Kevin Qinghong Lin +2Temporal Consistency in Video Generation Multi-Shot Video Generation
Show Lab, National University of Singapore · University of Oxford · Lovart AI

Sep 24, 2026J/K move · Enter open · S save

MVAgent: Multi-Agent Video Generation via Consistent Condition Construction and Shot-Level Policy Optimization

Xiangyu Kong, Wenjie Zhou, Fengping Tian, Lihua Fang+4

University of Exeter, United Kingdom · Alibaba Group, China

Agentic Video Generation Multi-Shot Video Generation Temporal Consistency in Video Generation

Open paper AskPDF arXiv

Multi-shot agentic video generation requires consistent character appearance, stable spatial layout across camera angles, and continuous character state between shots. When every shot is a separate request to a frozen generator, repeated text does not determine appearance, layout or state. We therefore recast the problem as condition construction and present MVAgent, a multi-agent pipeline whose agents collaborate through typed conditioning inputs. Because an environment image shows one viewpoint, a Spatial Grounding agent samples views from generated camera-traversal clips and anchors each shot to the view matching its framing. As generated shots drift from the plan, an Observer records how each shot ends in a continuity memory, from which a Transition agent builds character action and spatial references for the next shot. An Orchestrator composes these inputs into each request. Since a request reveals its effect only after rendering, we train it by agentic reinforcement learning with Trunk-GDPO, which compares rendered candidates at every shot rather than once per video and continues the best as the trunk. With generator and judges frozen, MVAgent attains the highest cross-shot consistency and narrative-planning quality among the compared methods on ViMax-Bench and is preferred over the strongest agentic baseline in human evaluation.

Abstract

Agentic Video Generation

Explore similar work

MVAgent: Multi-Agent Video Generation via Consistent Condition Construction and Shot-Level Policy Optimization

GroundShot: Visually Consistent Multi-Shot Long Video Generation via Entity-Grounded Shot Scheduling

Soap2Soap: Long Cinematic Video Remaking via Multi-Agent Collaboration