UniState: Unifying Memory and Intervention for Controllable World Simulation

Jaeseok Jeong; Kyungmook Choi; Sohyun Chung; Youngsik Yun; Youngjung Uh

cs.CVOct 18, 2026

UniState: Unifying Memory and Intervention for Controllable World Simulation

Authors: Jaeseok Jeong, Kyungmook Choi, Sohyun Chung, Youngsik Yun, Youngjung Uh

Organizations: Department of Artificial Intelligence, Yonsei University · Familiar

Abstract

World models offer increasingly rich environments for exploration, yet intervening in a world while observing the consequences from freely chosen viewpoints remains challenging. We tackle these challenges by introducing UniState, a framework for controllable world simulation built on a unified world-state representation. This representation unifies spatial memory and user interventions. Users specify object motion directly within spatial memory, while independently choosing the camera trajectory from which the resulting world state is observed. This unification makes the relationship between the scene and the requested changes explicit, removing ambiguity between object motion and viewpoint change from the control specification. Crucially, specifying an intervention does not require prescribing every resulting movement: the video model can synthesize additional motion within controlled objects and plausible interactions with their surroundings. We develop training and sampling strategies that help generated videos follow the specified object motion and camera trajectory while preserving this capacity to generate responses beyond the prescribed transformations. Experiments demonstrate more accurate joint object and camera control than the existing methods, alongside qualitative evidence that intervention-induced scene changes persist across rollouts.

Figures & tables

Figure 1: UniState is a framework for controllable world simulation, enableing coherent and interactive world simulation.

Method	Spatial memory	User intervention	Unified repr.
VerseCrafter	RGB point cloud	3D Gaussians	✘
WorldDirector	Video context tokens	2D box trajectories	✘
MotionCanvas	Image latent	2D boxes / point tracks	✘
SymphoMotion	Point-cloud render	3D point trajectories	✘
Real2SAM2Real	Video latent	3D mesh	✘
Ours	UniState		✔

Table 1: Comparison of spatial memory and object-intervention representations. UniState is the only method that unifies spatial memory and user intervention within a unified representation and world coordinate system. Detailed capability and intervention-representation comparisons are provided in Table 5 and Table 6 .

Figure 2: Overview of UniState. (a) Source observations are lifted into a unified state. Users specify interventions and a camera trajectory to construct the inference conditions. (b) During training, entity latents from ground-truth videos are combined with initial-state renders by masked compositing. (c) Only the conditioning modules marked with flames are trained. $\mathcal{L}_{\mathrm{FM}}$ denotes the flow-matching loss between predicted and target latent velocities.

Figure 3: State persistence and control across rollouts. (a) A rock displaced by the controlled bulldozer remains in its updated position after the camera leaves and revisits the region. (b) Three successive rollouts demonstrate object control with a fixed camera, camera control with a fixed object pose, and joint object and camera control, while carrying the world state forward. The left column visualizes the unified state and prescribed controls; the right columns show generated frames. Colored boxes identify controlled entities, and coordinate axes indicate their prescribed poses.

Figure 4: Ablation results for Scene-Anchored Joint Guidance. The red vertical line denotes the selected setting $\alpha=0.25$ .

Training condition	WCPE (%) $\downarrow$	Cam-T $\downarrow$	Cam-R ( ∘ ) $\downarrow$
Direct source-transport †	2.857	0.0115	0.405
GT condition w/o TAA	2.574	0.0114	0.412
GT condition + TAA	2.323	0.0106	0.376

Table 2: Ablation study on Transport-Aligned Augmentation (TAA). † denotes using transported initial-frame latents as the entity condition during training.

Method	WCPE (%) $\downarrow$	Cam-T $\downarrow$	Cam-R ( ∘ ) $\downarrow$
VerseCrafter	3.039	0.0540	0.385
WorldDirector	2.487	0.0844	0.571
SymphoMotion	2.822	0.0627	0.526
MotionCanvas	2.425	0.0378	0.428
Real2SAM2Real	3.061	0.104	0.608
Ours	2.323	0.0106	0.376

Table 3: Comparison with competing methods on joint object and camera control. Bold denotes the best score, and underline denotes the second-best score for each metric.

Figure 5: Object translation and rotation control under occlusion. The ball translates from right to left and rotates behind the alphabet blocks. Green boxes and XYZ axes indicate its target position and orientation. UniState preserves object motion, occlusion, and background stability.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Training configuration	WCPE (%) $\downarrow$	Cam-T $\downarrow$	Cam-R ( ∘ ) $\downarrow$
Interleaved training	2.630	0.0117	0.413
Staged curriculum (Ours)	2.323	0.0106	0.376

Appendix

Table 4: Staged versus interleaved training under matched data exposure and computation.

Figure 6: Qualitative comparison of object translation and rotation control . The user translates the yellow robot forward and left while rotating it in yaw, and independently translates the camera to the right. Ours follows the robot’s approach and turn under the prescribed viewpoint change. VerseCrafter shows insufficient forward motion; MotionCanvas and SymphoMotion deviate from the target heading; and WorldDirector and Real2SAM2Real produce excessive enlargement accompanied by unintended scene motion.

Figure 7: Qualitative comparison of object rotation control . The user specifies vertical oscillation and pitch rotation for the ball while pitching the camera upward. VerseCrafter, WorldDirector, SymphoMotion, and Real2SAM2Real fail to reflect the prescribed ball rotation. Ours reproduces the ball’s changing orientation and vertical motion under the moving viewpoint.

Figure 8: Qualitative comparison of occlusion handling . The user moves the tractor forward behind the hay bales and then partially back while dollying the camera forward. WorldDirector, MotionCanvas, and SymphoMotion cause the hay bales to disappear. Real2SAM2Real absorbs the tractor’s intended motion into unintended viewpoint change, while VerseCrafter fails to reproduce its partial return. Ours follows both the tractor trajectory and camera dolly while preserving the bales and their occlusion of the tractor.

Method	Appearance-bearing scene memory	Explicit 3D object control	Occlusion-aware appearance transport
VerseCrafter	✔	✔	▲
WorldDirector	✘	▲	✘
MotionCanvas	✘	▲	✘
SymphoMotion	▲	✔	▲
Real2SAM2Real	✘	✔	✘
Ours	✔	✔	✔

Appendix

Table 5: Capability comparison of object-control methods. Appearance-bearing scene memory denotes spatially grounded appearance evidence that can be reused across viewpoints. Explicit 3D object control indicates that object position or pose can be directly specified in 3D. Occlusion-aware appearance transport requires transporting appearance evidence with explicit depth and visibility reasoning. ✔ , ▲ , and ✘ denote full, partial or indirect, and unsupported capabilities, respectively.

Method	User intervention
Method	Geometry	Appearance
VerseCrafter	3D Gaussian trajectories	–
WorldDirector	2D box trajectories from 3D plans	Box-aligned RGB crops
MotionCanvas	2D boxes / point tracks	–
SymphoMotion	3D point trajectories	–
Real2SAM2Real	3D mesh cache $\rightarrow$ normal maps	–

Appendix

Table 6: Detailed comparison of representations used for user intervention. Geometry denotes the representation used to instantiate the intervention, while appearance indicates whether the representation explicitly carries appearance information. A dash indicates no explicit appearance information.

Method	WCPE (%) $\downarrow$	Cam-T $\downarrow$	Cam-R ( ∘ ) $\downarrow$
MosaicMem (reproduced)	2.522	0.01249	0.5132
Ours	2.3233	0.0106	0.3761

Appendix

Table 7: Comparison with MosaicMem under the same evaluation conditions as UniState. MosaicMem supports camera control but does not natively support object control. We apply time-varying 6-DoF transformations to selected object patches for comparison. Under these interventions, we observe diverse failure modes, including camera-control failures and objects remaining stationary or disappearing instead of following the specified motion, as illustrated in Fig. 9 .

Figure 9: Qualitative failure of MosaicMem in an object-only intervention scenario with a fixed camera. The tank fails to follow the specified motion and disappears midway through the generated sequence.

Figure 10: Misalignment in source-transport conditions. Rows show the source-transport condition (top) and ground-truth video (bottom) for the same sequence. Source transport produces floating artifacts due to inaccurate geometric alignment. TAA instead augments ground-truth entity latents to introduce transport-like appearance distortions while preserving the target entity support, avoiding the misaligned floaters present in the source-transport condition.

Controllable Video Generation

Controllable video generation steers how videos look, move, and change over time using user-provided instructions and structured controls.

See all 309 papers

StateCraft: Augmenting Video World Models with Abstract-State Reasoning
Oct 14, 2026Lorenzo Venturoli, Nedko Savov, Xi Wang +2Video World Models Long-Horizon Video Generation
INSAIT, Sofia University "St. Kliment Ohridski" · ETH Zurich · University of Ulm +3
WorldCraft: From Camera Navigation to Object Manipulation in Interactive Video World Models
May 24, 2026Bohai Gu, Taiyi Wu, Yueyang Yuan +9Camera-Controlled Video Generation Object-Centric World Models
The Hong Kong University of Science and Technology · AI Technology Center, Tencent Video, Tencent · Wuhan University +1
State World Models: Execute the Rules, Generate the Pixels
Oct 19, 2026Chan Yan Ki, Yanran Zhang, Wenzhao Zheng +5World Models Action-Conditioned World Models
Department of Automation, Tsinghua University, China · GigaAI

Oct 14, 2026J/K move · Enter open · S save

StateCraft: Augmenting Video World Models with Abstract-State Reasoning

Lorenzo Venturoli, Nedko Savov, Xi Wang, Marc Pollefeys+1

INSAIT, Sofia University "St. Kliment Ohridski" · ETH Zurich · University of Ulm+3

Video World Models Long-Horizon Video Generation Memory-Augmented Video Generation Action-Conditioned Video Generation Video Diffusion Models

Open paper AskPDF arXiv

Video world models aim to simulate complex environments with realistic appearance and dynamics. Maintaining consistency over long interactions requires tracking past actions and understanding how those actions affect the environment. Video world models can only generate partial observations of the entire world at a time. Remembering past observations alone may not be sufficient in every scenario, as they may no longer reflect the world after a sequence of actions. We introduce a paradigm that decouples a persistent, updatable abstract state from the visual state. The abstract state keeps track of the entities' changes as the world evolves, and is used to condition the visual state, which represents the current view. We propose StateCraft, a novel architecture that pairs a language model with a video diffusion model to evolve the dynamic world. We evaluate our approach on generative interfaces, which pose a substantial challenge because a single action may change parts of the state that are not currently visible. To support training and evaluation, we develop a data collection pipeline that produces informative, densely annotated trajectories at scale without human or agent supervision, together with a benchmark measuring generation quality, memory, and controllability. On this benchmark, our approach improves surpasses the state-of-the-art baselines. We demonstrate that StateCraft maintains high self-consistency and strong generation quality over long trajectories. Because the abstract state is explicit, it can be edited directly, and generation follows edits to states never observed in the data. Finally, a proof-of-concept 3D experiment, in which actions change the illumination of an unobserved room, shows that the paradigm can also be applied to spatial environments. Project page: https://insait-institute.github.io/StateCraft/.

Abstract

Controllable Video Generation

Explore similar work

StateCraft: Augmenting Video World Models with Abstract-State Reasoning

WorldCraft: From Camera Navigation to Object Manipulation in Interactive Video World Models

State World Models: Execute the Rules, Generate the Pixels