Robot World Models Are Not Invariant to How the Actions Are Written

Ahmed Karim; Leon Chlon

cs.ROSep 19, 2026

Robot World Models Are Not Invariant to How the Actions Are Written

Authors: Ahmed Karim, Leon Chlon

Organizations: Hassana Labs · University College London · University of Oxford

Abstract

A robot policy is trained with one of two action parameterizations: absolute joint targets, or deltas relative to the current state. The choice is a live engineering decision in robot learning, and a world model conditioned on actions inherits it silently. We show the inheritance is catastrophic. A latent dynamics model trained on one parameterization and handed the identical commanded trajectory written in the other collapses: retrieval degrades by 2.6-13.4x across three robot datasets and two morphologies, goal-conditioned action selection falls from 53% to 15%, and on PushT the two beliefs about the same future are near-orthogonal (cos = 0.067, worst case -0.377), so the predictor does not degrade gracefully, it answers a different question. This is not a distribution-shift artifact in the usual sense: the two encodings are mutually reconstructible at R^2 = 0.996 given the joint input, so no information is lost, and we give the test that separates a valid re-parameterization from a lossy summary or a sensor swap. The test rejected three of the four axes we proposed. The defect lives in the action channel, which the invariance literature for visual models does not examine: work there concerns crops, jitter and camera pose, while the parameterization of the commands goes unaudited. The repair is averaging over the two encodings, and where it goes matters. Averaging the objective restores task performance by itself; averaging the outputs, safe for probabilities by concavity, is not available for direction-valued prediction, where the normalized mean can score below every member of the orbit. What objective-averaging leaves behind is the tail: worst-case agreement stays at 0.78, a disagreement penalty closes it to 0.995, and over a latent rollout it is the difference between a worst case that erodes and one that holds. On PushT, averaging alone does not repair the axis.

Figures & tables

Figure 1: The same failure on frames. Each row is one test instance: the observation at $t$ , the true frame at $t+\Delta$ , and the frames retrieved when the predictor is handed the identical action window in absolute and in relative encoding. Green marks a correct retrieval, orange an incorrect one. Rows are restricted to misses from a different episode, since a same-episode miss is a near-duplicate frame and would show the reader nothing. Of $384$ test instances, $125$ are absolute-correct and relative-wrong, $95$ of them from a different episode; two are shown.

Dataset	Morphology	Canon.	Held out	Collapse	mean	worst
		Retrieval rank			$\cos$
aloha_mobile_cabinet	14-D bimanual, 50 fps	6.19	61.5	$9.9\times$	0.370	$-0.178$
aloha_static_coffee	14-D bimanual, new task	3.97	53.3	$13.4\times$	0.371	$-0.096$
pusht	2-D planar pusher, 10 fps	66.2	169.8	$2.6\times$	0.067	$\mathbf{-0.377}$

Table 1: Orbit dispersion under the action-encoding axis.

\Delta=4

s, 3 seeds, episode-disjoint splits, pool size 384. Collapse is held-out rank over canonical rank;

\cos

is agreement between the two encodings’ predictions, mean and worst case. ALOHA datasets ( Zhao et al., 2023 ) ; PushT ( Chi et al., 2023 ) ; all via Cadene et al. (2024) .

Figure 2: Left: retrieval rank on each encoding. Right: cross-encoding agreement, mean and worst case over the evaluation set. The teacher is competent on the encoding it was trained on and collapses on the equivalent one. Pooled training (

\lambda=0

) is shown because it, not mean-matching, is the baseline that competes: it recovers accuracy but leaves worst-case agreement at

0.779

, which only the consistency term lifts. The teacher’s worst case is

-0.178

: on its worst input the two notations point in opposite directions.

	Retrieval rank		Agreement		Action selection
	Canon.	Held out	mean	worst	Canon.	Held out
Teacher	6.19	61.5	0.370	—	$0.531$	$0.154$
Orbit-mean student	11.55	15.1	0.942	0.804	—	—
$\lambda=0$	5.46	5.98	0.937	0.779	$\mathbf{0.443}$	$0.401$
$\lambda=1$	5.51	5.58	0.981	0.939	$0.408$	$0.406$
$\lambda=4$	5.25	5.31	0.993	0.973	$0.426$	$0.402$

Table 2: Repair on aloha_mobile_cabinet , 3 seeds, all rows from the same runs.

\lambda

is the disagreement weight of Eq. ( 2 );

\lambda=0

is the same two-view pooled training with the penalty off, which is the baseline any practitioner would reach for once the axis is known; the orbit-mean row fits a student to the averaged teacher prediction instead.

\cos

is mean and worst-case pairwise agreement across the orbit. Action selection is top-1 over 32 candidates, chance

0.031

	Teacher		$\lambda=0$		$\lambda=16$
Dataset	mean	worst	mean	worst	mean	worst
ALOHA cabinet	0.805	0.242	0.988	0.896	1.000	0.997
ALOHA coffee	0.898	0.614	0.991	0.955	1.000	0.998
PushT	0.178	$-0.497$	0.677	$-0.348$	0.872	0.139

Table 3: Latent rollout on held-out episodes, 3 seeds, three datasets and two morphologies: cross-encoding cosine at the final hop (three hops of

4

s, two for PushT, whose episodes are short). Mean and worst case over the evaluation set. Per-hop trajectories are in Table 4 . Chains need

\text{horizon}\times\Delta

of runway and so start earlier in an episode than the pairs of Table 2 , so read the comparison across models, not against that table.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

		Mean $\cos$			Worst-case $\cos$
Dataset	Model	hop 1	hop 2	hop 3	hop 1	hop 2	hop 3
ALOHA	Teacher	0.836	0.803	0.805	0.515	0.350	0.242
cabinet	$\lambda=0$	0.992	0.990	0.988	0.972	0.951	0.896
	$\lambda=16$	1.000	1.000	1.000	1.000	0.999	0.997
ALOHA	Teacher	0.916	0.889	0.898	0.707	0.676	0.614
coffee	$\lambda=0$	0.995	0.993	0.991	0.987	0.973	0.955

Appendix

Table 4: Per-hop cross-encoding cosine for Table 3 : 3 seeds, three datasets and two morphologies; cross-encoding cosine between the absolute and relative rollouts at each hop. PushT episodes are short and admit two hops rather than three. Read the trend across hops, not the absolute values: chains need

\text{horizon}\times\Delta

of runway and so start earlier in an episode than the pairs of Table 2 , where the encodings are more similar at the input (

\cos 0.46

against

0.27

Axis	Varied	From	To
Data	$n=150\to 1200$	collapse $2.5\times\to 9.9\times$
Optimization	$800\to 8000$ steps	$\cos 0.370$	$\cos 0.242$
Capacity	hidden $1024/2\to 2048/3$	$\cos 0.370$	$\cos 0.125$

Appendix

Table 5: The incentive strengthens along all three axes the threshold implicates. Cross-encoding cosine; lower is more broken.

$d$	$\|S_{d}\|$	$n_{\mathrm{train}}$	Acc (unconstr.)	Acc (invariant)	Orbit disp.
8	$4\times 10^{4}$	1,000	0.858	0.979	0.141
8	$4\times 10^{4}$	10,000	0.942	0.993	0.059
8	$4\times 10^{4}$	100,000	0.968	0.995	0.024
16	$2.1\times 10^{13}$	1,000	0.756	0.974	0.229
16	$2.1\times 10^{13}$	10,000	0.828	0.991	0.163
16	$2.1\times 10^{13}$	100,000	0.918	0.994	0.062

Appendix

Table 6: True symmetry breaking at finite

n

. Target

Y=\mathbf{1}\{\mathrm{median}(\mathbf{x})>0\;\wedge\;\mathrm{std}(\mathbf{x})>0.8\;\wedge\;\mathrm{range}(\mathbf{x})>2\}

is exactly

S_{d}

-invariant. The unconstrained model’s orbit dispersion under random coordinate permutations (TV to the orbit mean) is large at finite

n

and grows with

|S_{d}|

, confirming that compression selects non-invariant predictors even when the target is invariant.

Figure 3: More data improves the trained encoding and barely improves the equivalent one, so the collapse ratio grows from

2.5\times

9.9\times

. Panel b is a quotient of the two curves in panel a, and should be read as the growing cost of a fixed disagreement, not as a growing disagreement: cross-encoding cosine is flat across the same sweep (

0.357,0.337,0.374,0.370

Figure 4: The cost of orbit-constancy as a function of

\lambda

, relative to

\lambda=0

. Negative is better. Left: the benefit shrinks and finally reverses as

n

grows. Right: the same, earlier, with more optimization.

Axis	Family	Teacher dispersion	Reduction	Certificate
Angle wrap (Kepler)	gauge	$8.6\times 10^{-15}$	—	bijective
Time origin (Lorenz)	gauge	$4.2\times 10^{-12}$	—	bijective
Circular shift	gauge	$2.6\times 10^{-2}$	$100\%$	quotient
Node permutation (Ax=b)	exchangeability	$3.3\times 10^{2}$	$83.4\%$	quotient

Appendix

Table 7: Admissibility certificates on four further numerical axes, 3 seeds each. All are admissible; two exhibit no measurable departure because the featurization already removes the nuisance.

Task	rank $\downarrow$ (no-act)	rank $\downarrow$ (act)	rank $\downarrow$ (shuf)	R@10 $\uparrow$ (no-act)	R@10 $\uparrow$ (act)	Verdict
ALOHA	186.2	97.7	635.8	0.068	0.124	make
LIBERO-10	80.9	49.6	55.5	0.136	0.242	make
PushT	179.9	349.9	547.7	0.079	0.093	break

Appendix

Table 8: Action conditioning: action-shuffle degrades performance, confirming causal use.

Metric	persist	no action	action	shuf
mean rank $\downarrow$	167.5	172.6	31.3	173.9
Recall@10 $\uparrow$	0.203	0.185	0.471	0.124

Appendix

Table 9: iPhyRE:

5.5\times

retrieval improvement with action conditioning.

Encoder	Acc (base)	Acc (stud.)	TV-mean (base)	TV-mean (stud.)	Verdict
RDF ( $\sigma{=}0.01$ )	0.894	0.893	0.017	0.016	make
RDF ( $\sigma{=}0.03$ )	0.894	0.893	0.012	0.011	make
Flatten ( $\sigma{=}0.01$ )	0.856	0.456	0.449	0.433	break

Appendix

Table 10: Same symmetry, two representations. The defect is caused by the representation.

World Models for Robotics

World models for robotics predict how scenes and robot states change under actions to support planning, control, and learning.

See all 384 papers

From World Models to World Action Models: A Concise Tutorial for Robotics
Jul 1, 2026Xiaoxiong Zhang, Xiong Zeng, Wei ZhangWorld Model Learning World Models for Robotics
School of Automation and Intelligent Manufacturing, Southern University of Science and Technology, Shenzhen, China · LimX Dynamics, Shenzhen, China
Cross-Embodiment Robot Foundation World Models with Latent Actions
Oct 7, 2026Huang Huang, Sriram Yenamandra, Arjun Majumdar +9World Models for Robotics Action-Conditioned World Models
Department of Computer Science, Stanford University, US · Work partially done while at Meta FAIR Robotics.
Do Robotic World Models Really Follow Actions? Diagnosing and Aligning Action-Conditioned Generation for Policy Learning
Aug 25, 2026Sixiang Chen, Jiaming Liu, Jixian Wu +11Action-Conditioned World Models World Model Evaluation
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Beijing Innovation Center of Humanoid Robotics · New York University +3

Jul 1, 2026J/K move · Enter open · S save

From World Models to World Action Models: A Concise Tutorial for Robotics

Xiaoxiong Zhang, Xiong Zeng, Wei Zhang

School of Automation and Intelligent Manufacturing, Southern University of Science and Technology, Shenzhen, China · LimX Dynamics, Shenzhen, China

World Model Learning World Models for Robotics Action-Conditioned World Models World Action Models Embodied World Models

Open paper AskPDF arXiv

World models are increasingly used in embodied intelligence and generative simulation, yet their scope remains ambiguous across communities. This tutorial presents a design-space view of world models as action-conditioned predictive models that estimate the future evolution of task-relevant observations or states. We categorize existing methods into observation-space and state-space world models, comparing their trade-offs in visual fidelity, spatial structure, physical interpretability, and control usability. We further introduce world action models, which connect predicted futures with executable robot actions, and summarize four representative paradigms: imagine-then-execute, video-feature-conditioned action prediction, joint video-action modeling, and auxiliary video prediction for policy learning. The goal of this tutorial is to clarify the conceptual scope of world (action) models and provide a structured taxonomy for embodied prediction and control.

Abstract

World Models for Robotics

Explore similar work

From World Models to World Action Models: A Concise Tutorial for Robotics

Cross-Embodiment Robot Foundation World Models with Latent Actions

Do Robotic World Models Really Follow Actions? Diagnosing and Aligning Action-Conditioned Generation for Policy Learning