Identifying Habit, Physics, and Nuisance in Robot World Models

Jinting Hang; Zhenhui Cai

cs.ROSep 6, 2026

Identifying Habit, Physics, and Nuisance in Robot World Models

Authors: Jinting Hang, Zhenhui Cai

Organizations: Harvest Praxis

Abstract

Teleoperated demonstrations are often multimodal even when the underlying dynamics are nearly deterministic given the executed action. We argue that this multimodality typically mixes three factors--operator habit in action selection, shared physics, and observation nuisance--and that entangled next-observation predictors absorb all three. We formalize the split with a structural causal model a=g(h,z,u), z'=f(z,a), o=r(z,c), and test it with complementary interventions: replacing or shuffling actions at fixed state sharply increases next-state error, whereas appearance and camera changes should not; habit-aware reverse scoring improves ranking of feasible pasts without rewriting the dynamics. The associated adaptation rule is to freeze a shared physics readout and update only a thin interface. On StackCube, DROID, and RH20T this rule improves low-shot transfer relative to training from scratch, retains cleaner dynamics under corrupted adaptation data, and extends from proprioception to pixel observations with multi-view and multi-step checks. We do not equate latent actions with operator habit, and we do not target large-scale video generation benchmarks.

Figures & tables

Panel of Figure 1 — Fig. 1: From entangled next-observation predictors (A) to an SCM that separates habit, physics, and nuisance (B), and a freeze-and-adapt interface (C).

Panel of Figure 2 — Fig. 2: Graphical summary of ( 1 )–( 3 ). Solid arrows are allowed mechanisms; the dashed edge marks the exclusion $h\nrightarrow z$ .

Intervention	Proxy	Physics readout	Habit / note
Wrong action	shuffle at fixed $z$	Kernel ${\sim}10\times$ ; DROID ${\sim}13\times$ ; RH20T ${\sim}10^{3}\times$	action changed
Swap user	other operator’s $a$	RH20T ${\sim}1362\times$	within-task habit
Habit fingerprint	$a{-}\hat{\pi}$ ; scramble	—	ID $0.52{\to}0.21$
Corrupted adapt.	$a{\times}2$ demos	freeze ${\sim}150\times$ cleaner	finetune absorbs habit
Appearance $do(c)$	color / blur	pixel ratio ${\sim}0.99$	next-RGB sensitive
Second view	held-out camera	cross ${\sim}1.42\times$	harder nuisance

TABLE I: Intervention design. Physics-facing readouts emphasize next-state or next-joint error; habit-facing readouts emphasize policy residuals and ranking.

Fig. 3: Compact status summary of the main diagnosis and transfer checks.

Setting	Metric	Outcome
Kernel $z_{p}$ mix / true	${\sim}10.4\times$	strong
DROID proprio wrong / true	${\sim}13\times$	strong
RH20T swap / shuffle	${\sim}1362\times$ / ${\sim}1352\times$	strong
RH20T LOUO $f_{\mathrm{all}}/f_{\mathrm{self}}$	$0.38\times$	strong
Fingerprint ID / scramble	$0.52{\to}0.21$ ( $-31$ pp)	strong
Corrupted adapt., freeze vs. finetune	${\sim}150\times$ cleaner	strong

TABLE II: Selected quantitative results. “Partial” denotes a weak but non-null effect; “weak” denotes an inconclusive cross-task test.

Panel of Figure 6 — Fig. 4: StackCube Kernel: (a) wrong-action / idle attribution; (b) freeze+ $g$ vs. finetune vs. scratch.

Panel of Figure 7 — Fig. 5: DROID proprioception: (a) predictive spread; (b) freeze+ $g$ on in-domain dynamics.

Condition	MSE	vs. true	Role
true $a$	$2.09{\times}10^{-7}$	$1\times$	reference
swap-user $a$	$2.84{\times}10^{-4}$	${\sim}1362\times$	within-task habit
shuffle $a$	$2.82{\times}10^{-4}$	${\sim}1352\times$	unstructured action
$a{=}0$	$1.41{\times}10^{-4}$	—	idle

TABLE III: RH20T interventions on the action argument of a shared dynamics model.

Panel of Figure 9 — Fig. 6: RH20T: (a) freeze under $a{\times}2$ demos; (b) freeze+ $g$ with mid-lossy $u$ .

Fig. 7: Effect of dynamics training choices on low-shot freeze+

g

transfer.

Quantity	Value
Appearance $do(c)$ ratio	$0.988{\pm}0.054$
Second-camera ratio	$1.42{\pm}0.06$
Wrong-action ratio (structural)	$9.5{\pm}3.7\times$
Multi-step wrong-action ( $H{=}1/5/10$ )	$1.9$ / $7.3$ / $13.7$
Downstream freeze MSE at $N{=}32$	$0.194{\pm}0.018$
Reverse mass lift vs. physics-only	$+0.040{\pm}0.002$

TABLE IV: DROID vision and nuisance summary (mean

\pm

std over three seeds).

Panel of Figure 12 — Fig. 8: (a) Wrong-action ratios grow with $H$ while appearance $do(c)$ stays near one. (b) Habit raises GT posterior mass; action-ID remains weak.

Fig. 9: Pixel low-shot next-joint error after freezing Enc and probe (mean

\pm

std over seeds).

$N$	freeze+ $\mathrm{delta}$	finetune	scratch
16	0.29	0.38	1.01
32	0.22	0.35	0.65
64	0.11	0.19	0.18

TABLE V: Pixel low-shot next-joint MSE (seed 0). Across three seeds, freeze beats scratch on a majority of budgets; at

N{=}32

freeze error is

0.194{\pm}0.018

Probe	Result
Seed action std	$0.79$
Video shuffle / true MSE	$5.0\times$
Language-induced $\overline{\Delta a^{2}}$	$0.033$
State shuffle / true MSE	$1.00\times$

TABLE VI: External action-model probes (boundary experiments, not the proposed method).

Method	$N$	MSE $\pm$ SE	$\Delta$ vs. base
Base	—	$0.098\pm 0.018$	—
LoRA	16	$0.107\pm 0.021$	$+0.010$
Kernel bridge	16	$0.096\pm 0.018$	$-0.001$

TABLE VII: Matched open-loop action MSE on the external model.

World Models for Robotics

World models for robotics predict how scenes and robot states change under actions to support planning, control, and learning.

See all 384 papers

From World Models to World Action Models: A Concise Tutorial for Robotics
Jul 1, 2026Xiaoxiong Zhang, Xiong Zeng, Wei ZhangWorld Model Learning World Models for Robotics
School of Automation and Intelligent Manufacturing, Southern University of Science and Technology, Shenzhen, China · LimX Dynamics, Shenzhen, China
Learning Transferable Dynamics Priors from Action to World Modeling
Jun 28, 2026Ze Huang, Jiahui Zhang, Hairuo Liu +3Action-Conditioned Video Generation Robot Policy Learning
School of Data Science, Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China +1
From World Models to World Action Models: Rethinking Next-State Prediction
Sep 28, 2026Tingyu Yuan, Ziming Ji, Biaoliang Guan +9Robot Dynamics Learning Action-Conditioned World Models
CASIA · UCAS · BUPT +4

Jul 1, 2026J/K move · Enter open · S save

From World Models to World Action Models: A Concise Tutorial for Robotics

Xiaoxiong Zhang, Xiong Zeng, Wei Zhang

School of Automation and Intelligent Manufacturing, Southern University of Science and Technology, Shenzhen, China · LimX Dynamics, Shenzhen, China

World Model Learning World Models for Robotics Action-Conditioned World Models World Action Models Embodied World Models

Open paper AskPDF arXiv

World models are increasingly used in embodied intelligence and generative simulation, yet their scope remains ambiguous across communities. This tutorial presents a design-space view of world models as action-conditioned predictive models that estimate the future evolution of task-relevant observations or states. We categorize existing methods into observation-space and state-space world models, comparing their trade-offs in visual fidelity, spatial structure, physical interpretability, and control usability. We further introduce world action models, which connect predicted futures with executable robot actions, and summarize four representative paradigms: imagine-then-execute, video-feature-conditioned action prediction, joint video-action modeling, and auxiliary video prediction for policy learning. The goal of this tutorial is to clarify the conceptual scope of world (action) models and provide a structured taxonomy for embodied prediction and control.

Abstract

World Models for Robotics

Explore similar work

From World Models to World Action Models: A Concise Tutorial for Robotics

Learning Transferable Dynamics Priors from Action to World Modeling

From World Models to World Action Models: Rethinking Next-State Prediction