Marionette: Predicting World States, Rendering Geometry, Painting Appearance

Zian Meng; Zhen Li; Chuanhao Li; Qiang Li; Kaipeng Zhang

cs.CVAug 14, 2026

Marionette: Predicting World States, Rendering Geometry, Painting Appearance

Authors: Zian Meng, Zhen Li, Chuanhao Li, Qiang Li, Kaipeng Zhang

Organizations: Alaya Lab · Shanghai Innovation Institute · Huazhong University of Science and Technology

Abstract

Interactive game world models typically autoregress visual observations directly in pixel or latent space, forcing structured properties such as pose, geometry, and occlusion to be implicitly maintained by the same generative sequence. Over long horizons, errors in these latent world properties accumulate, making consistency and controllability fragile. We explicitly model the evolving world state, delegate exact geometric computation to a fixed, zero-parameter renderer, and leave the neural model to synthesize appearance. We instantiate this idea as Marionette, a world model for interactive games with articulated characters. First, a two-stage autoregressive dynamics model predicts an explicit and interpretable 276-dimensional 3D world state comprising multi-entity articulated skeletons, metric root trajectories, and rotations. Second, a zero-parameter graphics bridge converts the predicted state into pose-control videos, computing world-space geometry and occlusion in closed form. Third, a control-conditioned video-diffusion observation model synthesizes photorealistic RGB observations from the resulting structured controls. Our experiments establish two properties of Marionette. First, the predicted world state is directly controllable. Forcing a mismatched action stream changes root-aligned joint error by 31% across 48 held-out segments. Second, long-horizon behaviour is determined in the state, and can be repaired there. Left free, the two generated characters drift to 21.2 m apart (recorded sessions stay near 5 m) and a third of frames show ground penetration. Two rules imposed on the explicit state, a terrain collider and a separation cap, cut penetration by 66% and keep the pair engaged, with no change to the observation model. Routing appearance through the predicted state costs no fidelity we can detect, at an FVD of 831 against 799 for recorded pose.

Figures & tables

Figure 1: Marionette overview. Bottom: the pipeline. ActionGPT makes discrete per-entity action decisions as a streaming token sequence; PoseGPT turns them into a continuous articulated world state; a zero-parameter graphics bridge renders the state to a pose-control video; a control-conditioned video-diffusion observation model adds appearance. Top: each stage illustrated with recorded data from one moment of a hunter–monster fight. The pose-control render and the game RGB show the same recorded frame through the same camera. Control enters the neural side at two points and nowhere else: editing the newest action token as it streams, and editing the root translation and rotation of the state. Everything the bridge computes from the state is exact by construction; appearance is not, and is the observation model’s to supply.

Panel of Figure 2 — Figure 2: The deterministic bridge on a $12$ -second generated rollout. Top: pose-control frames rasterized from the predicted state (terrain depth encoding with both entities’ skeletons). Bottom: the RGB frames the observation model renders from them. The geometry shared by the two rows is computed by the bridge, not generated by a neural model.

Metric	Marionette	Pixel-AR baseline
FVD 16 $\downarrow$	$831$ $[963,1209]$	$975$ $[1101,1346]$
FVD, final chunk of long rollout $\downarrow$	$1013$	$1198$

Table 1: Observation layer, against the end-to-end pixel-autoregressive baseline. Twelve

16

-second rollouts per model against real footage; FVD 16 is over

16

-frame clips, eight per rollout (

128

real /

96

generated), every pair started from the recorded first frame of its own held-out window, with the bridge camera initialized from the recorded view; our rollouts use the shipped configuration (terrain collider on). The final-chunk row uses the last

81

frames. Bracketed ranges are clip-level bootstrap percentile intervals ( 12 ) ; resampling duplicates clips and inflates a Fréchet distance, which is why they sit above the point estimates. The bias is common to both columns, and only the overlap is read. State-layer behaviour is in Table 2 and Figure 3 , where each number is read against the recorded state; controllability is in Table 4 .

Driving state	CFR $\downarrow$	MMP (m) $\downarrow$	Skate (m/s) $\downarrow$	Sep (m)
Recorded (reference)	0.082	0.023	1.13	4.8
Generated, no rule	$0.337$	$0.444$	$1.14$	$21.2$
+ collider	$0.114$	$0.157$	$1.14$	$21.2$
+ collider + separation cap	$0.114$	$0.155$	$1.22$	$5.1$

Table 2: The effect of each rule. Twelve held-out windows, identical checkpoint, bridge commit and seeds; only the rule applied to the state differs. Recorded state is the same measurement on the recorded trajectories of those windows, and is a reference level, not a competing method. CFR , collision-frame ratio: frames with any of

11

key joints below terrain at a

0.15

m margin. MMP , mean maximum penetration depth. Skate , contact-gated foot slide. Sep , monster–hunter distance at the final frame; the separation cap bounds it at

6

m by construction.

Figure 3: Each rule affects only its target quantity. The terrain collider reduces penetration without changing separation, and the separation cap does the opposite. Dashed line: the same measurement on recorded state over the same windows.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Indices	Component	Dimension
$[0{:}3]$	Monster root displacement (local)	$3$
$[3{:}162]$	Monster root-relative joint positions	$159$
$[162{:}165]$	Hunter root displacement (local)	$3$
$[165{:}258]$	Hunter root-relative joint positions	$93$
$[258{:}264]$	Weapon points (hunter-relative)	$6$
$[264{:}270]$	Monster root rotation (6D)	$6$

Appendix

Table 3: The

276

-dimensional world state

s

. Monster

M

and hunter

N

are each stored as a per-frame root displacement (local), root-relative joint positions, and a 6D root rotation; a small weapon sub-state is stored relative to the hunter. All positions are metric (metres); the world frame is

Y

-up, right-handed.

Figure 4: Terrain, from scan to conditioning to render. All five panels are one frame of one real generated rollout. (a) The scanned height field around the monster, one cell per metre, coloured by world height. The square is the footprint of the region the dynamics model is given and the arrow is the body heading that orients it. (b) The egocentric patch actually fed to both stages: $11\times 11$ cells of $0.4$ m, heights taken relative to the root, rotated into the body frame and biased $1$ m forward, so that the same slope yields the same feature at any position and heading. (c) The second conditioning feature: signed ground clearance of the $11$ key joints, which tells the model which parts of the body are in contact (one raised claw is clipped at the axis limit). (d) The three channels the bridge rasterizes, shown separately and contrast-windowed: height, in which terrain occupies the upper range and bones a disjoint lower one, so bones read as dark against light ground; per-joint categorical identity, on which terrain is zero; and inverse depth. Black regions are ground the scan never covered, where the render falls through to background. (e) The composite pose-control frame the observation model receives. Every step from (a) to (e) is a fixed geometric operation with no learned parameters.

Figure 5: Long-horizon behaviour on both evaluation layers (§ 4.4 ). (A) State layer: root distance to the recorded reference for

20

held-out free rollouts (faint) and their means (bold). A plausible stochastic world legitimately diverges from any single reference, so this panel is a diagnostic of divergence in absolute units and not a score. (B) Observation layer: FVD per

81

-frame chunk rises slowly and smoothly for both models, and does not separate them.

	Free	Force-GT	Force-Shuf
RA-MPJPE (m)	$0.281\pm 0.141$	$\mathbf{0.272\pm 0.135}$	$0.357\pm 0.120$
per class ( GT $\to$ Shuf )	stationary $0.24\!\to\!0.55$ attack $0.26\!\to\!0.46$ locomotion $0.27\!\to\!0.35$

Appendix

Table 4: State-layer controllability probe at scale. Hunter root-aligned MPJPE (m) at horizon

120

, seed context

64

frames, deterministic decoding;

48

held-out segments (mean

\pm

std). Bottom: per-frame error bucketed by the class of the driving token.

Panel of Figure 6 — Figure 6: Token switching executes at the commanded frame (state layer, pose-control rendering, fixed camera). All three rollouts share the seed and the sampling seed, so the frames are pixel-identical until the switch. At $5$ s the hunter’s token is held (top) or replaced by two different action ids (middle, bottom), and the generated pose diverges accordingly within a few frames.

Figure 7: Forced-token compliance by action class (

48

held-out segments, deterministic decoding). Error inflation from a wrong token is largest for the stationary and attack classes and smallest for locomotion, whose pose varies least across ids. Locomotion ids differ least in pose shape and show the smallest inflation.

Panel of Figure 8 — Figure 8: Qualitative comparison under one timestamped action schedule (stand still until $5$ s, then one large melee attack), all systems starting from the same recorded first frame. Our model receives the schedule as injected action tokens. The baseline and the commercial systems receive it as text. Commercial systems render superb appearance but follow the schedule loosely or not at all (Seedance never attacks; Grok attacks seconds late and swaps both characters’ appearance), and the pixel-autoregressive baseline exchanges the monster’s identity. Precise entity-level control at a commanded time is the axis the decoupled state interface provides.

Interactive World Models

Interactive world models predict and generate how environments change in response to actions, supporting simulation, planning, and control.

See all 117 papers

Jul 29, 2026

StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation

Recent game world models can generate visually realistic and interactive environments conditioned on player actions. However, games are not defined by pixels alone; they are governed by explicit mechanics, namely state-dependent rules that control health reduction, skill activation, and game termination. These mechanics depend on precise internal states, such as health points, skill meters, and timers, which are tightly coupled with visual observations and determine how gameplay evolves. Without modeling these state dynamics, existing game world models may generate visually plausible rollouts but violate the underlying game rules. In this paper, we propose StatePlay, a novel state-aware game world model that jointly predicts visual content and game states to promote mechanics-consistent generation. StatePlay adopts a mixture-of-transformers (MoT)-style architecture that preserves specialized visual and state representations while enabling cross-modal interaction, allowing predicted states to guide frame generation. Each branch is further optimized with a distinct objective suited to its modality. Experiments show that StatePlay achieves an average normalized L1 distance below 0.06 for state prediction. Furthermore, compared with models without explicit state modeling, our method improves mechanics fidelity in generated game rollouts by 18.6%. Overall, our work highlights the importance of state-aware game world modeling and advances beyond pixel-level realism toward complete and mechanically faithful game generation.

Zijun Lin, Zeqing Wang, Cheston Tan +2

Tencent · Nanyang Technological University · Centre for Frontier AI Research, A*STAR +1

World Models Action-Conditioned World Models Video World Models

Ask PDF

Jun 15, 2026

DreamX-World 1.0: A General-Purpose Interactive World Model

DreamX-World 1.0 is a general-purpose interactive text/image-to-video world model for controllable long-horizon generation. It supports camera navigation, revisits to previously observed regions, and promptable events across photorealistic, game-style, and stylized domains. Our data engine combines camera-accurate Unreal Engine rendering, action-rich gameplay recordings, and real-world videos with recovered camera geometry. For camera control, we introduce E-PRoPE, a lightweight variant of projective positional encoding that retains PRoPE's projective camera geometry while applying camera-aware attention to spatially reduced tokens. We convert a bidirectional video generator into a few-step autoregressive world model using causal forcing, DMD-style distillation, and long-rollout training. Training on self-generated long-horizon contexts exposes the model to its own generated history and reduces the style and color drift that accumulates across autoregressive chunks. Memory-Conditioned Scene Persistence retrieves earlier views through camera-geometry-based retrieval, while residual recycling makes the conditioning path less sensitive to imperfect memory latents. Event Instruction Tuning adds composable event control, and reinforcement learning alignment recovers camera control and visual quality after distillation. With mixed-precision DiT execution, residual reuse, 75%-pruned VAE decoding, and asynchronous pipeline parallelism, DreamX-World 1.0 reaches up to 16,FPS on eight RTX,5090 GPUs. On our 5-second basic evaluation, DreamX-World 1.0 achieves a camera-control score of 73.75 and an overall score of 84.76, outperforming HY-WorldPlay 1.5 and LingBot-World in overall score, which achieve 80.79 and 80.45, respectively.

DreamX Team, Yancheng Bai, Rui Chen +20

DreamX Team

Camera-Controlled Video Generation Real-Time Interactive Video Generation Long-Horizon Video Generation Memory-Augmented Video Generation Autoregressive Video Generation Interactive World Models

Ask PDF

Oct 14, 2026

StateCraft: Augmenting Video World Models with Abstract-State Reasoning

Video world models aim to simulate complex environments with realistic appearance and dynamics. Maintaining consistency over long interactions requires tracking past actions and understanding how those actions affect the environment. Video world models can only generate partial observations of the entire world at a time. Remembering past observations alone may not be sufficient in every scenario, as they may no longer reflect the world after a sequence of actions. We introduce a paradigm that decouples a persistent, updatable abstract state from the visual state. The abstract state keeps track of the entities' changes as the world evolves, and is used to condition the visual state, which represents the current view. We propose StateCraft, a novel architecture that pairs a language model with a video diffusion model to evolve the dynamic world. We evaluate our approach on generative interfaces, which pose a substantial challenge because a single action may change parts of the state that are not currently visible. To support training and evaluation, we develop a data collection pipeline that produces informative, densely annotated trajectories at scale without human or agent supervision, together with a benchmark measuring generation quality, memory, and controllability. On this benchmark, our approach improves surpasses the state-of-the-art baselines. We demonstrate that StateCraft maintains high self-consistency and strong generation quality over long trajectories. Because the abstract state is explicit, it can be edited directly, and generation follows edits to states never observed in the data. Finally, a proof-of-concept 3D experiment, in which actions change the illumination of an unobserved room, shows that the paradigm can also be applied to spatial environments. Project page: https://insait-institute.github.io/StateCraft/.

Lorenzo Venturoli, Nedko Savov, Xi Wang +2

INSAIT, Sofia University "St. Kliment Ohridski" · ETH Zurich · University of Ulm +3

Video World Models Long-Horizon Video Generation Memory-Augmented Video Generation Action-Conditioned Video Generation Video Diffusion Models

Ask PDF

Abstract

Interactive World Models

Explore similar work

StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation

DreamX-World 1.0: A General-Purpose Interactive World Model

StateCraft: Augmenting Video World Models with Abstract-State Reasoning