A robot policy is trained with one of two action parameterizations: absolute joint targets, or deltas relative to the current state. The choice is a live engineering decision in robot learning, and a world model conditioned on actions inherits it silently. We show the inheritance is catastrophic. A latent dynamics model trained on one parameterization and handed the identical commanded trajectory written in the other collapses: retrieval degrades by 2.6-13.4x across three robot datasets and two morphologies, goal-conditioned action selection falls from 53% to 15%, and on PushT the two beliefs about the same future are near-orthogonal (cos = 0.067, worst case -0.377), so the predictor does not degrade gracefully, it answers a different question. This is not a distribution-shift artifact in the usual sense: the two encodings are mutually reconstructible at R^2 = 0.996 given the joint input, so no information is lost, and we give the test that separates a valid re-parameterization from a lossy summary or a sensor swap. The test rejected three of the four axes we proposed. The defect lives in the action channel, which the invariance literature for visual models does not examine: work there concerns crops, jitter and camera pose, while the parameterization of the commands goes unaudited. The repair is averaging over the two encodings, and where it goes matters. Averaging the objective restores task performance by itself; averaging the outputs, safe for probabilities by concavity, is not available for direction-valued prediction, where the normalized mean can score below every member of the orbit. What objective-averaging leaves behind is the tail: worst-case agreement stays at 0.78, a disagreement penalty closes it to 0.995, and over a latent rollout it is the difference between a worst case that erodes and one that holds. On PushT, averaging alone does not repair the axis.
Figures & tables
Figure 1: The same failure on frames. Each row is one test instance: the observation at t , the true frame at t+Δ , and the frames retrieved when the predictor is handed the identical action window in absolute and in relative encoding. Green marks a correct retrieval, orange an incorrect one. Rows are restricted to misses from a different episode, since a same-episode miss is a near-duplicate frame and would show the reader nothing. Of 384 test instances, 125 are absolute-correct and relative-wrong, 95 of them from a different episode; two are shown.
Retrieval rank
cos
Dataset
Morphology
Canon.
Held out
Collapse
mean
worst
aloha_mobile_cabinet
14-D bimanual, 50 fps
6.19
61.5
9.9×
0.370
−0.178
aloha_static_coffee
14-D bimanual, new task
3.97
53.3
13.4×
0.371
−0.096
pusht
2-D planar pusher, 10 fps
66.2
169.8
2.6×
0.067
−0.377
Table 1: Orbit dispersion under the action-encoding axis. Δ=4 s, 3 seeds, episode-disjoint splits, pool size 384. Collapse is held-out rank over canonical rank; cos is agreement between the two encodings’ predictions, mean and worst case. ALOHA datasets ( Zhao et al., 2023 ) ; PushT ( Chi et al., 2023 ) ; all via Cadene et al. (2024) .
Figure 2: Left: retrieval rank on each encoding. Right: cross-encoding agreement, mean and worst case over the evaluation set. The teacher is competent on the encoding it was trained on and collapses on the equivalent one. Pooled training ( λ=0 ) is shown because it, not mean-matching, is the baseline that competes: it recovers accuracy but leaves worst-case agreement at 0.779 , which only the consistency term lifts. The teacher’s worst case is −0.178 : on its worst input the two notations point in opposite directions.
Retrieval rank
Agreement
Action selection
Canon.
Held out
mean
worst
Canon.
Held out
Teacher
6.19
61.5
0.370
—
0.531
0.154
Orbit-mean student
11.55
15.1
0.942
0.804
—
—
λ=0
5.46
5.98
0.937
0.779
0.443
0.401
λ=1
5.51
5.58
0.981
0.939
0.408
0.406
λ=4
5.25
5.31
0.993
0.973
0.426
0.402
Table 2: Repair on aloha_mobile_cabinet , 3 seeds, all rows from the same runs. λ is the disagreement weight of Eq. ( 2 ); λ=0 is the same two-view pooled training with the penalty off, which is the baseline any practitioner would reach for once the axis is known; the orbit-mean row fits a student to the averaged teacher prediction instead. cos is mean and worst-case pairwise agreement across the orbit. Action selection is top-1 over 32 candidates, chance 0.031 .
Teacher
λ=0
λ=16
Dataset
mean
worst
mean
worst
mean
worst
ALOHA cabinet
0.805
0.242
0.988
0.896
1.000
0.997
ALOHA coffee
0.898
0.614
0.991
0.955
1.000
0.998
PushT
0.178
−0.497
0.677
−0.348
0.872
0.139
Table 3: Latent rollout on held-out episodes, 3 seeds, three datasets and two morphologies: cross-encoding cosine at the final hop (three hops of 4 s, two for PushT, whose episodes are short). Mean and worst case over the evaluation set. Per-hop trajectories are in Table 4 . Chains need horizon×Δ of runway and so start earlier in an episode than the pairs of Table 2 , so read the comparison across models, not against that table.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Mean cos
Worst-case cos
Dataset
Model
hop 1
hop 2
hop 3
hop 1
hop 2
hop 3
ALOHA
Teacher
0.836
0.803
0.805
0.515
0.350
0.242
cabinet
λ=0
0.992
0.990
0.988
0.972
0.951
0.896
λ=16
1.000
1.000
1.000
1.000
0.999
0.997
ALOHA
Teacher
0.916
0.889
0.898
0.707
0.676
0.614
coffee
λ=0
0.995
0.993
0.991
0.987
0.973
0.955
Appendix
Table 4: Per-hop cross-encoding cosine for Table 3 : 3 seeds, three datasets and two morphologies; cross-encoding cosine between the absolute and relative rollouts at each hop. PushT episodes are short and admit two hops rather than three. Read the trend across hops, not the absolute values: chains need horizon×Δ of runway and so start earlier in an episode than the pairs of Table 2 , where the encodings are more similar at the input ( cos0.46 against 0.27 ).
Axis
Varied
From
To
Data
n=150→1200
collapse 2.5×→9.9×
Optimization
800→8000 steps
cos0.370
cos0.242
Capacity
hidden 1024/2→2048/3
cos0.370
cos0.125
Appendix
Table 5: The incentive strengthens along all three axes the threshold implicates. Cross-encoding cosine; lower is more broken.
d
∣Sd∣
ntrain
Acc (unconstr.)
Acc (invariant)
Orbit disp.
8
4×104
1,000
0.858
0.979
0.141
8
4×104
10,000
0.942
0.993
0.059
8
4×104
100,000
0.968
0.995
0.024
16
2.1×1013
1,000
0.756
0.974
0.229
16
2.1×1013
10,000
0.828
0.991
0.163
16
2.1×1013
100,000
0.918
0.994
0.062
Appendix
Table 6: True symmetry breaking at finite n . Target Y=1{median(x)>0∧std(x)>0.8∧range(x)>2} is exactly Sd -invariant. The unconstrained model’s orbit dispersion under random coordinate permutations (TV to the orbit mean) is large at finite n and grows with ∣Sd∣ , confirming that compression selects non-invariant predictors even when the target is invariant.
Figure 3: More data improves the trained encoding and barely improves the equivalent one, so the collapse ratio grows from 2.5× to 9.9× . Panel b is a quotient of the two curves in panel a, and should be read as the growing cost of a fixed disagreement, not as a growing disagreement: cross-encoding cosine is flat across the same sweep ( 0.357,0.337,0.374,0.370 ).
Figure 4: The cost of orbit-constancy as a function of λ , relative to λ=0 . Negative is better. Left: the benefit shrinks and finally reverses as n grows. Right: the same, earlier, with more optimization.
Axis
Family
Teacher dispersion
Reduction
Certificate
Angle wrap (Kepler)
gauge
8.6×10−15
—
bijective
Time origin (Lorenz)
gauge
4.2×10−12
—
bijective
Circular shift
gauge
2.6×10−2
100%
quotient
Node permutation (Ax=b)
exchangeability
3.3×102
83.4%
quotient
Appendix
Table 7: Admissibility certificates on four further numerical axes, 3 seeds each. All are admissible; two exhibit no measurable departure because the featurization already removes the nuisance.
World models are increasingly used in embodied intelligence and generative simulation, yet their scope remains ambiguous across communities. This tutorial presents a design-space view of world models as action-conditioned predictive models that estimate the future evolution of task-relevant observations or states. We categorize existing methods into observation-space and state-space world models, comparing their trade-offs in visual fidelity, spatial structure, physical interpretability, and control usability. We further introduce world action models, which connect predicted futures with executable robot actions, and summarize four representative paradigms: imagine-then-execute, video-feature-conditioned action prediction, joint video-action modeling, and auxiliary video prediction for policy learning. The goal of this tutorial is to clarify the conceptual scope of world (action) models and provide a structured taxonomy for embodied prediction and control.
Xiaoxiong Zhang, Xiong Zeng, Wei Zhang
School of Automation and Intelligent Manufacturing, Southern University of Science and Technology, Shenzhen, China · LimX Dynamics, Shenzhen, China
The diversity of robot embodiments and action spaces makes it challenging to build robot world models that generalize across different embodiments. We introduce the Latent Action-Conditioned Robot World Model (LAC-WM), which operates within a learned unified latent action space shared across diverse embodiments. This unified action space improves the world model's performance when adapted to previously unseen robot embodiments. We compare LAC-WM with an Explicit Action-Conditioned World Model (EAC-WM), which conditions on explicit motion labels. Our results show that explicit action conditioning leads to disjoint action representations across embodiments, limiting downstream performance when adapting to new robots. We evaluate both models on dexterous manipulation tasks and a modified LIBERO benchmark. LAC-WM improves downstream performance over EAC-WM by up to 46.7% on dexterous manipulation and 11.7% on LIBERO. Crucially, the unified latent action space allows LAC-WM's downstream performance to scale positively with the number of embodiments used during pretraining. In contrast, the disjoint action space in EAC-WM leads to decreased performance as the number of pretraining embodiments increases. These results highlight the importance of a unified action space for efficient cross-embodiment learning, addressing a key challenge in robotics. Project website: https://lacwm.github.io/
Huang Huang, Sriram Yenamandra, Arjun Majumdar +9
Department of Computer Science, Stanford University, US · Work partially done while at Meta FAIR Robotics.
Action-conditioned world models increasingly serve as learned simulators for robotic policy improvement, yet their reliability rests on an unverified assumption: generated futures faithfully reflect the consequences of arbitrary valid actions. Systematic evaluation against action-specific ground truth remains limited beyond expert demonstrations. To address this gap, we introduce WorldEcho, which evaluates action following over a broader action distribution through visual integrity and SE(3) end-effector trajectory alignment. Our diagnosis shows that expert-trained models perform better on demonstrated actions but struggle with off-expert trajectories, producing motion inconsistent with the commanded actions or visually invalid rollouts. To mitigate these failures, we propose WorldSync, which broadens action coverage, grounds intermediate video representations in action-induced robot dynamics through an Action-Forcing Expert, and aligns predicted and ground-truth changes under action interventions. Experiments show that WorldSync reduces integrity-gated error on RoboTwin and, when used for iterative policy improvement, yields higher success rates in simulation and on real robots.
Sixiang Chen, Jiaming Liu, Jixian Wu +11
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Beijing Innovation Center of Humanoid Robotics · New York University +3