Synchronised action annotations are needed to train controllable world models and these datasets remain elusive. Existing approaches make use of instrumented platforms with calibrated sensors, costly manual annotation, or latent-action models which lack grounding. We instead turn ordinary unlabelled video into action-supervised training data by recovering (without training) a data-derived egomotion basis. We track pixel displacements across frames and exploit the recurring coherent structure induced by egomotion to obtain grounded control signals directly. Using a method as simple as principal components analysis perform this, we find that the leading components provide signed, scalable, and composable throttle--yaw controls, although the method can recover only motion axes represented in the data. To prevent a high-capacity video DiT from exploiting pixel-level supervision, an online latent critic distils a frozen decoder--tracker--PCA (Principal Components Analysis) teacher without backpropagating through the decoder or tracker. Finally we critique the use of video generation metrics to evaluate WMs and introduce an example of an alternative, reference-free evaluation method. We measure controllability, plausibility, conjuring (creating objects out of thin air) and geometric integrity, revealing failures that conventional video metrics miss. We show that most baselines follow familiar action directions but struggle to reverse or remain stationary. Our model handles both while retaining compositional control and generation quality. Despite backwards actions being less than 1% of our training data, we find that the model learns to reverse, scale its response linearly, and compose throttle with steering, all simply by learning through a grounded action space.
Figures & tables
Figure 1: Tracked displacement fields from two real-world FrodoBots-2K sequences. Columns show successive 0.5 s chunks, and arrows show net grid-point displacement. Coherent egomotion patterns persist despite local motion and occlusion. Source video frames are from the FrodoBots-2K dataset, licensed under CC BY-SA 4.0; tracking overlays are ours.
Figure 2: Training-time cosine agreement for recorded (GT) and sign-reversed (FLIP) commands. Positive FLIP agreement indicates genuine command response; panel titles give source-direction duration in minutes. The full eight-direction result appears in Fig. 10 .
Figure 3: Settled throttle and steering responses of the ablation variants, measured by the CoTracker–PCA readout. Most points use sign-flipped commands on held-out windows; positive throttle instead uses sampled forward commands on held-out no-op contexts. Dashed lines mark the approximate training support.
Figure 4: Realised throttle g0 under eight commands. Each wedge is a collation of one direction; opacity shows density, solid outlines medians, and dashed arcs interquartile ranges. Blue denotes forward and red backward motion. Panels use separate 95 th-percentile rims, so compare sign and shape rather than absolute radius.
Figure 5: Wedge plots for the ablation suite. The No Critic and the No AdaLN model exhibit markedly weaker and less coherent realised responses than the other models, showing that control performance is strongly dependent on these axes.
Command
Default
WorldPlay
Matrix-Game
WorldCam
Astra
Yume
minWM
↑
7
13
7
23
67
3
67
↓
7
3
0
93
87
40
60
←
3
0
0
0
7
0
23
→
3
0
0
0
3
3
30
↖
7
0
0
3
3
0
27
↗
0
0
3
0
3
0
27
Table 1: Control failure by command (%) for our flagship and the external baselines; lower is better.
Command
Default
pca4
pca2
Batch64
Batch16
No Act Tok
No AdaLN
No Critic
↑
7
3
10
17
7
7
47
87
↓
7
27
33
10
27
37
97
100
←
3
0
0
0
0
3
77
97
→
3
0
0
3
0
0
13
80
↖
7
3
7
7
7
7
40
77
↗
0
0
0
0
0
0
7
77
Table 2: Control failure by command (%) for the ablation grid, with the same definitions and populations as Table 1 .
Model
Style %
Geom. %
Reloc. %
Conj. %
HF %
WorldPlay
16
80
78
0
13
Matrix-Game
62
71
96
0
9
WorldCam
62
68
88
0.4
45
Astra
13
52
68
0.4
11
Yume
5
9
59
0.4
13
minWM
4
1
4
20
5
Table 3: Generation-quality failure rates (%); lower is better.
Model
Style %
Geom. %
Reloc. %
Conj. %
HF %
Default
3
17
20
1
16
pca4
3
38
28
0
22
pca2
3
40
25
0
28
Batch64
3
26
17
0
14
Batch16
5
32
42
1
44
No Act Tok
3
29
31
1
29
Table 4: Generation-quality failure rates (%) for the ablations. Each variant replaces Default as the sole family representative when scored.
Default
WorldPlay
Matrix-Game
WorldCam
Astra
Yume
minWM
REAL
No-op fail (%)
0
0
0
22
75
44
3
6
Table 5: No-op control failure (%) over 32 contexts, using the 0.1 action-magnitude threshold defined in the text. REAL reports false positives from noisy action labelling on real no-op continuations, not a lower bound on model failure rates.
Default
pca4
pca2
Batch64
Batch16
No Act Tok
No AdaLN
No Critic
No-op fail (%)
0
0
0
0
0
0
78
3
Table 6: Stationary control failure (%) for the ablation grid, with the same definition as Table 5 .
Figure 6: Per-rollout failure profiles for (a) Default and external baselines and (b) our ablations. Neutral cells pass all criteria; coloTurs show the first failure in priority order: control > geometry > relocation > HF > style > conjuration. Headings report the number passing all criteria.
Appendix figures & tables37 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Representative rollouts across environments and cities. The commanded action is shown in the leftmost panel of each row.
Figure 8: Principal motion components recovered from the fixed CoTracker–PCA pipeline. Each panel shows the displacement field associated with one PCA coordinate. The exposed control space uses g0 for longitudinal motion and g1 for yaw; g6 captures roll-like motion. The remaining components describe secondary camera and scene dynamics and are used only for critic supervision.
Model
Output resolution
Ours
832×480
minWM
832×480
WorldCam
832×480
HY-World 1.5 WorldPlay
832×480
Astra
832×480
Matrix-Game 2.0
640×352
Appendix
Table 7: Native output resolutions used by the action-forcing benchmark.
Command
Keyboard input
Mouse yaw
↑
forward =1
0
↓
backward =1
0
→
none
+0.1
←
none
−0.1
↗
forward =1
+0.1
↖
forward =1
−0.1
Appendix
Table 8: Matrix-Game 2.0 controls used for the eight-direction benchmark.
Command
Person clause
Camera clause
↑
moves forward (W)
remains still
↓
moves backward (S)
remains still
←
stands still
turns left
→
stands still
turns right
↖
moves forward and left (W+A)
turns left
↗
moves forward and right (W+D)
turns right
Appendix
Table 9: Yume-1.5 caption clauses used for the eight-direction benchmark.
Figure 9: Conditioning-pathway ablation under counterfactual FLIP evaluation, grouped by source-ride motion direction. Dual injection through AdaLN and action tokens converges fastest and reaches the strongest response. AdaLN-only conditioning learns control more slowly, while tokens-only conditioning remains anti-aligned on throttle and unstable on steering. Panels and smoothing follow Fig. 2 .
Figure 10: Training-time command following for the default model across all eight source-motion groups. Blue uses the recorded command (GT), while red applies the sign-flipped counterfactual (FLIP) to the same context. Positive FLIP agreement indicates genuine command response. Panel titles denote the source-context motion, and curves use a 9 -point ( ≈450 -step) centred rolling mean.
Figure 11: GT and FLIP agreement for the no-AdaLN, tokens-only variant. High GT agreement but negative or unstructured FLIP response shows that the model continues source motion without acquiring reliable command control. Panel titles denote source-context motion.
Figure 12: GT and FLIP agreement for the Batch64 model. FLIP becomes positive across all eight directions, demonstrating genuine command response. Panel titles denote source-context motion.
Figure 13: Batch-size ablation using global batch sizes 16 , 32 , and 64 under FLIP evaluation. All batch sizes recover the same directional structure by step 5,000 , but larger batches acquire the hardest throttle responses earlier and retain stronger reverse motion.
Figure 14: Critic-supervision ablation using the leading 2 , 4 , or 8 PCA components. Two supervised components suffice to recover directional control, while richer supervision improves convergence and realized response strength.
Figure 15: Settled throttle response g0 by commanded direction for the ablation variants. Each translucent wedge represents one of 256 rollouts; blue denotes realised forward motion and red realised backward motion. Solid outlines show medians, and panel rims show the local 95 th percentile. The tokens-only model renders every command as forward motion.
Figure 16: Settled steering response g1 by commanded direction for the ablation variants. Blue denotes realised rightward yaw and red realised leftward yaw; opacity, outlines, and panel normalization otherwise follow Fig. 15 . Every functional variant shows the expected four-lobe structure.
Figure 17: Cross-model action-forcing response curves from 32 low-egomotion held-out contexts. Realized motion is measured using the common CoTracker–PCA readout, and shaded bands show ±1 SEM. Baselines are shown only at strengths supported by their released interfaces. The dotted diagonal denotes unit gain as a visual reference.
Figure 18: Steering response g1 by commanded direction for all models. Blue denotes realised rightward yaw and red realised leftward yaw; other wedge conventions follow the main-paper throttle figure. Every model recovers the expected steering sign, although response magnitude and variance differ.
Figure 19: Uncommanded roll response g6 under each planar command. Radius indicates roll magnitude; blue denotes clockwise and red counter-clockwise motion. Direction-dependent responses reveal coupling between commanded planar motion and out-of-plane camera motion.
Model or variant
Feature-valid
Active
WorldPlay
240
238
Matrix-Game
240
240
WorldCam
240
216
Astra
240
215
Yume
240
237
minWM
240
187
Appendix
Table 10: Quality-evaluation populations. The complete directional fleet contains 256 rollouts per model. Two wet-lens source contexts, corresponding to 16 rollouts per model, are excluded uniformly because reference-based feature matching is undefined, leaving 240 feature-valid rollouts. Active further excludes near-static continuations using the ORB-based criterion. Style, relocation, and deployed HF rates use the Active denominator.
Variant
Median B
B>150 (%)
n/N
Default
−78
16
39/239
Batch64
−51
14
33/237
pca4
−10
22
52/237
No Action Tokens
+48
29
66/229
pca2
+61
28
66/232
Batch16
+125
44
104/236
Appendix
Table 11: High-frequency degradation under HF-Cal-v2 . B measures sharpness loss relative to the same-scene median of the six external baselines and the scored variant. Rates use model-specific active populations; n/N counts rollouts with B>150 . This is a peer-relative exceedance rate, not absolute degradation prevalence. No additional sharpness-quantile exclusion is applied.
Figure 20: Distribution of the high-frequency-degradation score B under HF-Cal-v2 (reference-relative sharpness loss; higher is more degraded). No AdaLN and Batch16 degrade most, while Default , Batch64 , and pca4 lie at or below zero and preserve detail better than the same-scene reference continuations.
Figure 21: Cross-model distribution of the high-frequency-degradation score B under HF-Cal-v2 for the six external baselines, Default and Batch64, using their active evaluation populations ( 1,809 rollouts in total). Higher values indicate greater reference-relative loss of sharpness. Green denotes our models, while blue denotes external baselines. Black ticks mark medians, the grey dashed line marks B=0 , and the red dotted line marks the descriptive threshold B>150 .
Figure 22: Geometry-instrument examples for the external baselines, showing representative human-positive and human-negative cases. Yume r27 relocates without visible geometric corruption, illustrating why relocation needs a separate instrument.
Figure 23: Geometry-instrument examples for our ablation variants, showing representative human-positive and human-negative cases.
Detector
AUC
VLM geometric-corruption probe (Qwen3-VL-8B)
0.78
same probe, InternVL3-8B
0.75
same probe, Cosmos-Reason1-7B
0.74
PAL4VST artifact fraction
0.77
VLM melt probe, Cosmos-Reason1-7B
0.71
VLM melt probe, Qwen2.5-VL-7B
0.68
Appendix
Table 12: Geometric-corruption detectors scored against 36 human mangle labels on the 100 -rollout subset. The deployed VLM probe (Qwen3-VL-8B) agrees best with human judgement; the same probe is evaluated across three open VLMs. The depth-curvature signal, part of an earlier ensemble, performs near chance ( 0.52 ) and is not used.
Signal
AUC
DINOv2 embedding drift (deployed)
0.72
CLIP embedding distance
0.70
Cosmos-Reason1-7B style probe
0.67
Qwen3-VL-8B style probe
0.66
InternVL3-8B style probe
0.55
VGG Gram-matrix distance
0.55
Appendix
Table 13: Candidate style-shift measures evaluated by AUC on the human-labelled set containing 20 positives among 100 examples.
Figure 24: Style-shift-instrument examples for the external baselines, showing representative flagged and clean rollouts.
Figure 25: Style-shift-instrument examples for our ablation variants, showing representative flagged and clean rollouts.
Detector
AUC
ORB + RANSAC inliers (deployed)
0.70
DINOv2 embedding drift
0.72
AKAZE + RANSAC inliers
0.63
SIFT + RANSAC inliers
0.62
ORB matches, no RANSAC
0.54
Appendix
Table 14: Scene-relocation detectors evaluated on a 100 -video human-labelled subset containing 19 relocation positives. Higher AUC is better. ORB+RANSAC is deployed because it provides an absolute inlier count and also supports static-rollout detection. The deployed relocation decision uses this inlier count against the frozen peer panel and is therefore panel-relative rather than a standalone absolute rate.
Figure 26: Scene-relocation-instrument examples for the external baselines, showing representative relocated and retained-scene rollouts.
Figure 27: Scene-relocation-instrument examples for our ablation variants. Post-collision content and severe lens occlusion illustrate ambiguous flagged cases that suppress geometric correspondences without necessarily changing the underlying scene.
Backbone
Prec.
Rec.
F1
fps
RT-DETR R50 (deployed)
1.00
0.80
0.89
94.8
Faster R-CNN R50-FPNv2
1.00
0.80
0.89
56.3
RetinaNet R50-FPNv2
1.00
0.73
0.85
69.6
Appendix
Table 15: Detection backbones under an identical temporal pipeline, scored on the enriched evaluation set: 15 confirmed conjurations drawn from several models together with 22 rollouts confirmed clean, so that both precision and recall are measurable. Throughput is end-to-end detection plus temporal analysis on a single RTX 5090.
Figure 28: Conjuration-instrument examples. Top: flagged rollouts centred on object birth, showing empty ground beforehand and a persistent object afterwards. Bottom: hard negatives in which new objects arrive legitimately; annotations identify the prior-appearance test that rejected each candidate.
Detector
AUC
Laplacian drift, original analysis
0.85
Laplacian drift, original-panel rescore
0.78
Laplacian drift, family-balanced (deployed)
0.68
Dark-channel haze drift, sibling-relative
0.82
Dark-channel haze drift, raw
0.77
Laplacian-variance drift, raw
0.68
Appendix
Table 16: High-frequency-degradation validation on 112 rollouts with 13 haze positives. The first three rows separate the original Laplacian analysis from rescoring under the original and family-balanced references. Other candidate results are from the original comparison, not a new matched comparison against v2 .
The high-frequency-degradation instrument on our ablation variants. High- B rollouts (top) visibly wash out over the generated span—texture and contrast are lost by the end—while low- B rollouts (bottom) preserve detail. The same source scenes appear in both classes, so the effect is model-dependent, not scene-dependent.
Model
Movement
Animation
REAL (reference)
0.0
1.1
Ours (Default)
2.3
1.4
Ours (pca4)
2.1
0.0
Ours (pca2)
2.2
0.0
Ours (Batch64)
2.0
0.6
Ours (Batch16)
2.0
0.0
Appendix
Table 17: Stationarity and animation over 32 null-command contexts. Movement is the mean distance between generated and real-continuation readouts in raw PCA-projection units, before tanh; lower is better and REAL =0.0 by construction. Animation is the mean count of alive cells in an 8×8 grid after removing the top three PCA motion modes, not a count of objects. A faithful no-op should hold the viewpoint without freezing the world. Animation is most interpretable for low-Movement models and is partly confounded by native bitrate.
Figure 29: Residual egomotion under the null command. Each translucent wedge represents one of 32 contexts. Its angle and radius describe the two-component raw PCA departure from the real continuation; radius uses the same raw PCA-projection units as Movement, before tanh squashing. Blue denotes forward drift and red backward drift. Solid outlines show medians and dashed arcs show interquartile ranges.
Model
Style %
Geom. %
Reloc. %
Conj. %
HF %
Default
0
0
3
0
3
pca4
0
0
3
0
0
pca2
0
0
3
0
9
Batch64
0
0
3
0
0
Batch16
0
0
3
0
12
No Act. Tok.
0
0
3
0
3
Appendix
Table 18: Stationary-rollout failure rates (%); lower is better. Style, Geom., Reloc., Conj., and HF denote style shift, geometric corruption, scene relocation, conjuration, and high-frequency degradation. HF and relocation are peer-referenced against the family-balanced v2 reference (one seat per family; a variant of ours occupies our seat itself when scored); the remaining quality axes are panel-independent.
Variant
Nodes
GPUs
Wall-clock
GPU-h
Default (PCA-8)
8
32
26.9 h
862
Batch64
16
64
28.4 h
1819
Batch16
4
16
27.4 h
439
PCA-4
8
32
27.5 h
879
PCA-2
8
32
26.9 h
860
No-Act-Tokens
8
32
21.0 h
671
Appendix
Table 19: Measured training cost per reported model.
Controllable video world models target interactive world exploration, where models must faithfully execute explicit action commands while preserving visual quality and temporal coherence. However, most existing approaches rely on continuous camera trajectories as action conditions, which often lead to unreliable action following, especially under complex motion sequences. In this work, we identify action representation entanglement as a key bottleneck in controllable video generation, and show that continuous camera representations lead to high feature similarity across distinct motion patterns, degrading action controllability. Based on this insight, we propose DisCo, a controllable video world model that conditions generation on a compact set of discrete action primitives to improve action separability. We further introduce DisCoBench, a comprehensive benchmark for evaluating the ability of models in short-term, long-horizon, and highly dynamic exploration scenarios. Extensive experiments demonstrate that DisCo achieves significantly more reliable action following while preserving visual quality.
Video world models have achieved strong visual realism, but this does not ensure that their dynamics are truly governed by actions. In this work, we argue that action faithfulness should be understood through the compositional structure of actions, which in many embodied settings follows a group structure (e.g., SE(2) for navigation). Based on this insight, we formalize action-conditioned world modeling as realizing a group action on the state space, providing a principled criterion for evaluating dynamics beyond visual quality. To operationalize this framework, we propose a unified approach that enforces identity, inverse, and composition consistency via latent-space regularization with synthesized supervision, avoiding additional data collection. We further introduce two metrics: Group-Action Consistency (GAC) and Group-Action Robustness (GAR), to evaluate structural correctness and rollout stability. Extensive experimental results show that our method consistently improves both GAC and GAR in state-of-the-art video world models without degrading perceptual quality.
Zijie Wang, Wei Zhang, Weiming Zhang +4
Sun Yat-sen University · Shenzhen Loop Area Institute · Baidu Inc. +2
World action models (WAMs) jointly predict actions (intent) and visual future (foresight). Standard training adds noise to recorded actions and video simultaneously, but such training paradigms introduce a mismatch: perturbed actions imply counterfactual future visual, while the noised video remains tied to the GT recording. In low-noise regime, the scene geometry and even the dynamic behavior remain clearly visible from the noisy future frames despite the added noise. We present CtrlWAM, which executes perturbed actions in a simulator and pairs them with their noised visual consequences for joint WAM learning. To accommodate the different denoising requirements of video and actions, we introduce warped video--action noise schedules that aim to keep visual layout responsive as action predictions evolve. We further extend the action interface from ego-only control to a variable number of agent streams, allowing a unified model to represent predicted or commanded futures for multiple agents. Driving experiments show more accurate action forecasts, closer agreement between generated video and actions, and better following of supplied commands; robotics experiments show stronger motion fidelity and controllability. Matched controls support the benefit of off-path renders for command following and manipulation fidelity. Together, these findings contribute to a more controllable world action model. Project page: https://ctrl-wam.github.io/