Action Forcing: Training World Models on Unsupervised Video by Recovering Underlying Egomotion Bases

Ashish Sundar; Tiankuo Hou; Zhong Fan; Chunbo Luo; Xiaoyang Wang

Figure 7: Representative rollouts across environments and cities. The commanded action is shown in the leftmost panel of each row.

Figure 8: Principal motion components recovered from the fixed CoTracker–PCA pipeline. Each panel shows the displacement field associated with one PCA coordinate. The exposed control space uses

g_{0}

for longitudinal motion and

g_{1}

for yaw;

g_{6}

captures roll-like motion. The remaining components describe secondary camera and scene dynamics and are used only for critic supervision.

Model	Output resolution
Ours	$832\times 480$
minWM	$832\times 480$
WorldCam	$832\times 480$
HY-World 1.5 WorldPlay	$832\times 480$
Astra	$832\times 480$
Matrix-Game 2.0	$640\times 352$

Appendix

Table 7: Native output resolutions used by the action-forcing benchmark.

Command	Keyboard input	Mouse yaw
$\uparrow$	forward =1	$0$
$\downarrow$	backward =1	$0$
$\rightarrow$	none	$+0.1$
$\leftarrow$	none	$-0.1$
$\nearrow$	forward =1	$+0.1$
$\nwarrow$	forward =1	$-0.1$

Appendix

Table 8: Matrix-Game 2.0 controls used for the eight-direction benchmark.

Command	Person clause	Camera clause
$\uparrow$	moves forward (W)	remains still
$\downarrow$	moves backward (S)	remains still
$\leftarrow$	stands still	turns left
$\rightarrow$	stands still	turns right
$\nwarrow$	moves forward and left (W+A)	turns left
$\nearrow$	moves forward and right (W+D)	turns right

Appendix

Table 9: Yume-1.5 caption clauses used for the eight-direction benchmark.

Figure 9: Conditioning-pathway ablation under counterfactual FLIP evaluation, grouped by source-ride motion direction. Dual injection through AdaLN and action tokens converges fastest and reaches the strongest response. AdaLN-only conditioning learns control more slowly, while tokens-only conditioning remains anti-aligned on throttle and unstable on steering. Panels and smoothing follow Fig. 2 .

Figure 10: Training-time command following for the default model across all eight source-motion groups. Blue uses the recorded command (GT), while red applies the sign-flipped counterfactual (FLIP) to the same context. Positive FLIP agreement indicates genuine command response. Panel titles denote the source-context motion, and curves use a

9

-point (

\approx 450

-step) centred rolling mean.

Figure 11: GT and FLIP agreement for the no-AdaLN, tokens-only variant. High GT agreement but negative or unstructured FLIP response shows that the model continues source motion without acquiring reliable command control. Panel titles denote source-context motion.

Figure 12: GT and FLIP agreement for the Batch64 model. FLIP becomes positive across all eight directions, demonstrating genuine command response. Panel titles denote source-context motion.

Figure 13: Batch-size ablation using global batch sizes

16

,

32

, and

64

under FLIP evaluation. All batch sizes recover the same directional structure by step

5{,}000

, but larger batches acquire the hardest throttle responses earlier and retain stronger reverse motion.

Figure 14: Critic-supervision ablation using the leading

2

,

4

, or

8

PCA components. Two supervised components suffice to recover directional control, while richer supervision improves convergence and realized response strength.

Figure 15: Settled throttle response

g_{0}

by commanded direction for the ablation variants. Each translucent wedge represents one of

256

rollouts; blue denotes realised forward motion and red realised backward motion. Solid outlines show medians, and panel rims show the local

95

th percentile. The tokens-only model renders every command as forward motion.

Figure 16: Settled steering response

g_{1}

by commanded direction for the ablation variants. Blue denotes realised rightward yaw and red realised leftward yaw; opacity, outlines, and panel normalization otherwise follow Fig. 15 . Every functional variant shows the expected four-lobe structure.

Figure 17: Cross-model action-forcing response curves from

32

low-egomotion held-out contexts. Realized motion is measured using the common CoTracker–PCA readout, and shaded bands show

\pm 1

SEM. Baselines are shown only at strengths supported by their released interfaces. The dotted diagonal denotes unit gain as a visual reference.

Figure 18: Steering response

g_{1}

by commanded direction for all models. Blue denotes realised rightward yaw and red realised leftward yaw; other wedge conventions follow the main-paper throttle figure. Every model recovers the expected steering sign, although response magnitude and variance differ.

Figure 19: Uncommanded roll response

g_{6}

under each planar command. Radius indicates roll magnitude; blue denotes clockwise and red counter-clockwise motion. Direction-dependent responses reveal coupling between commanded planar motion and out-of-plane camera motion.

Model or variant	Feature-valid	Active
WorldPlay	240	238
Matrix-Game	240	240
WorldCam	240	216
Astra	240	215
Yume	240	237
minWM	240	187

Appendix

Table 10: Quality-evaluation populations. The complete directional fleet contains

256

rollouts per model. Two wet-lens source contexts, corresponding to

16

rollouts per model, are excluded uniformly because reference-based feature matching is undefined, leaving

240

feature-valid rollouts. Active further excludes near-static continuations using the ORB-based criterion. Style, relocation, and deployed HF rates use the Active denominator.

Variant	Median $\mathcal{B}$	$\mathcal{B}>150$ (%)	$n/N$
Default	$-78$	$16$	$39/239$
Batch64	$-51$	$14$	$33/237$
pca4	$-10$	$22$	$52/237$
No Action Tokens	$+48$	$29$	$66/229$
pca2	$+61$	$28$	$66/232$
Batch16	$+125$	$44$	$104/236$

Appendix

Table 11: High-frequency degradation under HF-Cal-v2 .

\mathcal{B}

measures sharpness loss relative to the same-scene median of the six external baselines and the scored variant. Rates use model-specific active populations;

n/N

counts rollouts with

\mathcal{B}>150

. This is a peer-relative exceedance rate, not absolute degradation prevalence. No additional sharpness-quantile exclusion is applied.

Figure 20: Distribution of the high-frequency-degradation score

\mathcal{B}

under HF-Cal-v2 (reference-relative sharpness loss; higher is more degraded). No AdaLN and Batch16 degrade most, while Default , Batch64 , and pca4 lie at or below zero and preserve detail better than the same-scene reference continuations.

Figure 21: Cross-model distribution of the high-frequency-degradation score

\mathcal{B}

under HF-Cal-v2 for the six external baselines, Default and Batch64, using their active evaluation populations (

1{,}809

rollouts in total). Higher values indicate greater reference-relative loss of sharpness. Green denotes our models, while blue denotes external baselines. Black ticks mark medians, the grey dashed line marks

\mathcal{B}=0

, and the red dotted line marks the descriptive threshold

\mathcal{B}>150

.

Figure 22: Geometry-instrument examples for the external baselines, showing representative human-positive and human-negative cases. Yume r27 relocates without visible geometric corruption, illustrating why relocation needs a separate instrument.

Figure 23: Geometry-instrument examples for our ablation variants, showing representative human-positive and human-negative cases.

Detector	AUC
VLM geometric-corruption probe (Qwen3-VL-8B)	0.78
same probe, InternVL3-8B	0.75
same probe, Cosmos-Reason1-7B	0.74
PAL4VST artifact fraction	0.77
VLM melt probe, Cosmos-Reason1-7B	0.71
VLM melt probe, Qwen2.5-VL-7B	0.68

Appendix

Table 12: Geometric-corruption detectors scored against

36

human mangle labels on the

100

-rollout subset. The deployed VLM probe (Qwen3-VL-8B) agrees best with human judgement; the same probe is evaluated across three open VLMs. The depth-curvature signal, part of an earlier ensemble, performs near chance (

0.52

) and is not used.

Signal	AUC
DINOv2 embedding drift (deployed)	$\mathbf{0.72}$
CLIP embedding distance	$0.70$
Cosmos-Reason1-7B style probe	$0.67$
Qwen3-VL-8B style probe	$0.66$
InternVL3-8B style probe	$0.55$
VGG Gram-matrix distance	$0.55$

Appendix

Table 13: Candidate style-shift measures evaluated by AUC on the human-labelled set containing

20

positives among

100

examples.

Figure 24: Style-shift-instrument examples for the external baselines, showing representative flagged and clean rollouts.

Figure 25: Style-shift-instrument examples for our ablation variants, showing representative flagged and clean rollouts.

Detector	AUC
ORB + RANSAC inliers (deployed)	0.70
DINOv2 embedding drift	0.72
AKAZE + RANSAC inliers	0.63
SIFT + RANSAC inliers	0.62
ORB matches, no RANSAC	0.54

Appendix

Table 14: Scene-relocation detectors evaluated on a

100

-video human-labelled subset containing

19

relocation positives. Higher AUC is better. ORB+RANSAC is deployed because it provides an absolute inlier count and also supports static-rollout detection. The deployed relocation decision uses this inlier count against the frozen peer panel and is therefore panel-relative rather than a standalone absolute rate.

Figure 26: Scene-relocation-instrument examples for the external baselines, showing representative relocated and retained-scene rollouts.

Figure 27: Scene-relocation-instrument examples for our ablation variants. Post-collision content and severe lens occlusion illustrate ambiguous flagged cases that suppress geometric correspondences without necessarily changing the underlying scene.

Backbone	Prec.	Rec.	$F_{1}$	fps
RT-DETR R50 (deployed)	1.00	0.80	0.89	94.8
Faster R-CNN R50-FPNv2	1.00	0.80	0.89	56.3
RetinaNet R50-FPNv2	1.00	0.73	0.85	69.6

Appendix

Table 15: Detection backbones under an identical temporal pipeline, scored on the enriched evaluation set:

15

confirmed conjurations drawn from several models together with

22

rollouts confirmed clean, so that both precision and recall are measurable. Throughput is end-to-end detection plus temporal analysis on a single RTX 5090.

Figure 28: Conjuration-instrument examples. Top: flagged rollouts centred on object birth, showing empty ground beforehand and a persistent object afterwards. Bottom: hard negatives in which new objects arrive legitimately; annotations identify the prior-appearance test that rejected each candidate.

Detector	AUC
Laplacian drift, original analysis	$0.85$
Laplacian drift, original-panel rescore	$0.78$
Laplacian drift, family-balanced (deployed)	$0.68$
Dark-channel haze drift, sibling-relative	$0.82$
Dark-channel haze drift, raw	$0.77$
Laplacian-variance drift, raw	$0.68$

Appendix

Table 16: High-frequency-degradation validation on

112

rollouts with

13

haze positives. The first three rows separate the original Laplacian analysis from rescoring under the original and family-balanced references. Other candidate results are from the original comparison, not a new matched comparison against v2 .

The high-frequency-degradation instrument on our ablation variants. High-

B

rollouts (top) visibly wash out over the generated span—texture and contrast are lost by the end—while low-

B

rollouts (bottom) preserve detail. The same source scenes appear in both classes, so the effect is model-dependent, not scene-dependent.

Model	Movement	Animation
REAL (reference)	$0.0$	$1.1$
Ours (Default)	$2.3$	$1.4$
Ours (pca4)	$2.1$	$0.0$
Ours (pca2)	$2.2$	$0.0$
Ours (Batch64)	$2.0$	$0.6$
Ours (Batch16)	$2.0$	$0.0$

Appendix

Table 17: Stationarity and animation over

32

null-command contexts. Movement is the mean distance between generated and real-continuation readouts in raw PCA-projection units, before tanh; lower is better and REAL

=0.0

by construction. Animation is the mean count of alive cells in an

8\times 8

grid after removing the top three PCA motion modes, not a count of objects. A faithful no-op should hold the viewpoint without freezing the world. Animation is most interpretable for low-Movement models and is partly confounded by native bitrate.

Figure 29: Residual egomotion under the null command. Each translucent wedge represents one of

32

contexts. Its angle and radius describe the two-component raw PCA departure from the real continuation; radius uses the same raw PCA-projection units as Movement, before tanh squashing. Blue denotes forward drift and red backward drift. Solid outlines show medians and dashed arcs show interquartile ranges.

Model	Reloc. %	HF %
Default	3	3
pca4	3	0
pca2	3	9
Batch64	3	0
Batch16	3	12
No Act. Tok.	3	3

Appendix

Table 18: Stationary-rollout failure rates (%); lower is better. Style, Geom., Reloc., Conj., and HF denote style shift, geometric corruption, scene relocation, conjuration, and high-frequency degradation. HF and relocation are peer-referenced against the family-balanced v2 reference (one seat per family; a variant of ours occupies our seat itself when scored); the remaining quality axes are panel-independent.

Variant	Nodes	GPUs	Wall-clock	GPU-h
Default (PCA-8)	8	32	26.9 h	862
Batch64	16	64	28.4 h	1819
Batch16	4	16	27.4 h	439
PCA-4	8	32	27.5 h	879
PCA-2	8	32	26.9 h	860
No-Act-Tokens	8	32	21.0 h	671

Appendix

Table 19: Measured training cost per reported model.

Action Forcing: Training World Models on Unsupervised Video by Recovering Underlying Egomotion Bases

Abstract

Figures & tables

Appendix

Controllable Video Generation

DisCo: World Models with Discrete Camera Motion Control

World Models as Group Actions

CtrlWAM: Controllable World Action Models with Aligned Intent and Foresight

Abstract

Controllable Video Generation

Explore similar work

DisCo: World Models with Discrete Camera Motion Control

World Models as Group Actions

CtrlWAM: Controllable World Action Models with Aligned Intent and Foresight