OneWorld: Learning Consistent Physics Across Actions in World Models

Ke He; Yichen Ding; Bin Yang

cs.CVSep 25, 2026

OneWorld: Learning Consistent Physics Across Actions in World Models

Authors: Ke He, Yichen Ding, Bin Yang

Organizations: Wuhan University Wuhan, China

Abstract

Action-conditioned video world models aim to predict scene evolution under different actions, a capability that is essential for reliable planning, decision-making, and interaction in dynamic environments. However, futures generated independently from the same initial scene may each appear plausible while implying incompatible physical properties, such as friction or mass. This inconsistency can lead to contradictory predictions across interventions, making it difficult for the model to maintain a coherent understanding of the underlying world and limiting its reliability for planning and decision-making. To address these issues, we propose OneWorld, a shared-mechanism counterfactual generation framework that jointly models multiple action-conditioned futures under a common latent physical mechanism. A physical mechanism interpreter first infers a distribution over latent mechanisms from each action-outcome branch. These distributions are then aggregated into shared-world evidence, which captures whether the branches admit a common physical explanation while accounting for uncertainty in less informative branches. This evidence constrains flow training and guides sampling, encouraging consistency in the underlying physical mechanism while preserving the distinct outcomes induced by different actions. We further introduce a multi-intervention evaluation protocol in controlled environments, following the interaction settings of ACWM-Phys, to assess whether generated futures can be jointly explained by the same physical parameters, alongside standard measures of single-rollout prediction quality. Experiments in these environments show that OneWorld improves cross-intervention physical consistency while maintaining competitive single-rollout prediction quality.

Figures & tables

Figure 1: Motivating probes of action-conditioned prediction. ACWM-DiT predictions under different actions from the same initial observation within each scene, illustrating the need to evaluate whether alternative futures remain compatible with a shared physical world.

Figure 2: Overview of OneWorld . Given the same initial scene and multiple action sequences, OneWorld predicts a future for each intervention branch. A physical mechanism interpreter infers a mechanism distribution from each action–outcome pair. The shared-world evidence score assesses their compatibility and provides the signal for shared-world flow training and shared-world flow guidance.

Category	Environment	Method	Prediction	Physical fit and consistency
Category	Environment	Method	M-MSE $\downarrow$	$E_{\mathrm{ind}}\downarrow$	$E_{\mathrm{shared}}\downarrow$	$\Delta_{\mathrm{world}}\downarrow$
Rigid body	Push Cube	ACWM-DiT	$0.03477\pm 0.00034$	$0.07535\pm 0.00112$	$0.09788\pm 0.00173$	$0.02253\pm 0.00109$
		Vid2World	$0.03428\pm 0.00029$	$0.07488\pm 0.00096$	$0.09346\pm 0.00161$	$0.01858\pm 0.00094$
		CoCo	$0.03416\pm 0.00031$	$0.07452\pm 0.00088$	$0.08891\pm 0.00139$	$0.01439\pm 0.00082$
		Twin Rollouts	$0.03435\pm 0.00027$	$0.07497\pm 0.00104$	$0.09078\pm 0.00148$	$0.01581\pm 0.00087$
		Shared Noise	$0.03464\pm 0.00036$	$0.08073\pm 0.00121$	$0.11122\pm 0.00196$	$0.03049\pm 0.00128$

Table 1: Multi-intervention comparison: individual versus shared physical fits (

E_{\mathrm{ind}}

E_{\mathrm{shared}}

) and their gap

\Delta_{\mathrm{world}}

. Values are mean

\pm

standard deviation over three independent runs; lower is better for all metrics.

Figure 3: Different actions from the same initial world on RoboDesk. Branches share the initial state and hidden physics but differ in action. Consistency concerns a common physical explanation after accounting for contact and release conditions, rather than identical displacements.

Method	Components			Metrics
	Mismatch Neg.	Shared Train	Flow Guide	M-MSE $\downarrow$	$E_{\mathrm{shared}}\downarrow$	$\Delta_{\mathrm{world}}\downarrow$
ACWM-DiT	–	–	–	0.03477	0.09788	0.02253
No Mismatch Neg.	✗	✓	✓	0.03442	0.08495	0.01084
No Shared Train	✓	✗	✓	0.03431	0.08260	0.00831
No Flow Guide	✓	✓	✗	0.03418	0.08024	0.00625
OneWorld	✓	✓	✓	0.03396	0.07741	0.00357

Table 2: Ablation of OneWorld. “Mismatch Neg.” denotes mismatched-world supervision for the mechanism interpreter. “Shared Train” denotes shared-world flow training with

\mathcal{L}_{\mathrm{shared}}

, and “Flow Guide” denotes shared-world flow guidance during sampling.

Figure 4: Sensitivity of OneWorld to the number of intervention branches

K

, shared-loss weight

\lambda_{\mathrm{shared}}

, maximum guidance strength

\lambda_{\max}

, and the number of importance samples

M

on Push Cube.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Resolution	Context	Prediction	Evaluation clips
BAIR	$64\times 64$	2 frames	12 frames	2,048
RoboNet	$64\times 64$	2 frames	10 frames	2,048

Appendix

Table 3: Protocol for external action-conditioned video prediction.

Setting	BAIR	RoboNet
Backbone	Action-conditioned DiT	Action-conditioned DiT
Video representation	Frozen Wan2.1 causal VAE	Frozen Wan2.1 causal VAE
Optimizer	AdamW	AdamW
Learning rate	$1\times 10^{-5}$	$1\times 10^{-5}$
Batch size	8	8
Training updates	40,000	50,000

Appendix

Table 4: Training configuration for the external video prediction benchmarks.

Dataset	Method	Venue	FVD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
BAIR	MaskViT	ICLR’23	70.5	–	–	–
	iVideoGPT	NeurIPS’24	60.8	24.5	90.2	5.0
	DPM	ICLR’25 ∗	48.5	25.9	92.0	4.5
	DyDiff	ICLR’25	45.0	26.2	92.4	4.2
	SAMPO	NeurIPS’25	55.5	26.7	94.7	3.7
	Our DiT backbone	Ours	41.2	27.9	95.8	2.8

Appendix

Table 5: Comparison with existing action-conditioned video prediction methods on BAIR and RoboNet at

64\times 64

resolution. Lower is better for FVD and LPIPS; higher is better for PSNR and SSIM. SSIM and LPIPS are multiplied by

100

. ∗ DPM denotes the diffusion baseline reported with DyDiff.

Task	Initial states	Groups	$K$	Hidden physical factors	Configs.
Open Drawer	192/48/48	384/96/96	3	rail friction, joint damping	$3\times 3$
Sliding Door	192/48/48	384/96/96	3	track friction, guide damping	$3\times 3$

Appendix

Table 6: RoboDesk setup used for the qualitative comparison.

Component	Setting
Latent mechanism $W$	32-dimensional diagonal Gaussian
Posterior update	Additive natural-parameter evidence, $\Delta\lambda=\operatorname{softplus}(\rho)$
Clean-future projector	2-layer MLP, width 512
Action encoder	2-layer MLP, width 256
Flow-time embedding	128-dimensional sinusoidal embedding
Interpreter fusion network	3-layer MLP, width 1024, SiLU

Appendix

Table 7: Implementation details for the mechanism model and OneWorld training.

Compute path	Latency (s/group)	Peak mem. (GiB/GPU)	Relative	Throughput (groups/s)
ACWM-DiT	$13.247\pm 0.012$	2.403	$1.000\times$	0.603
OneWorld w/o guidance	$13.245\pm 0.014$	2.403	$1.000\times$	0.602
OneWorld full	$\mathbf{21.684\pm 0.041}$	7.864	$\mathbf{1.637\times}$	0.369

Appendix

Table 8: Inference cost on 8 NVIDIA A100-SXM4-80GB GPUs. Each group contains three branches with 37 frames at

240\times 240

resolution.

Environment	Initial states	Groups	$K$	Video	Physical parameters	Configs.
Push Cube	128/32/32	256/64/64	3	37 frames, $240^{2}$ , 10 Hz	drag, elasticity	$3\times 3$
Push Rope	128/32/32	256/64/64	3	37 frames, $240^{2}$ , 10 Hz	bending stiffness, bending damping	$3\times 3$
Pour Water	160/40/40	320/80/80	3	37 frames, $240^{2}$ , 10 Hz	viscosity, particle cohesion	$4\times 4$

Appendix

Table 9: Controlled environments used for shared-world evaluation.

Environment	Observable state	Validation error
Push Cube	2D center and orientation	position RMSE $0.011$ , angle error $2.8^{\circ}$
Push Rope	16 ordered rope keypoints	mean point RMSE $0.017$
Pour Water	$16\times 16$ occupancy and centroid	occupancy IoU $0.928$ , centroid RMSE $0.014$

Appendix

Table 10: Validation accuracy of the frozen state extractors used by the simulator-based evaluator.

Environment	Method	$3\times 3$	$5\times 5$	$9\times 9$
Push Cube	ACWM-DiT	0.02253	0.02187	0.02161
	OneWorld	0.00357	0.00341	0.00336
Push Rope	ACWM-DiT	0.01389	0.01343	0.01326
	OneWorld	0.00209	0.00197	0.00191

Appendix

Table 11: Sensitivity of the shared-world gap to evaluator grid density. Lower is better. The model outputs are fixed across all columns.

Method	Push Cube $\uparrow$	Push Rope $\uparrow$	Pour Water $\uparrow$	Avg. $\uparrow$
ACWM-DiT	59.4	62.5	50.0	57.3
Vid2World	62.5	65.6	52.5	60.2
CoCo	65.6	71.9	57.5	65.0
Twin Rollouts	65.6	68.8	55.0	63.1
Shared Noise	53.1	59.4	45.0	52.5
Posterior Alignment	71.9	75.0	60.0	69.0

Appendix

Table 12: Recovery accuracy (%) of ground-truth physical configurations on held-out same-world groups. Higher is better.

Environment	Physical factor	Training values	Interpolation values
Push Cube	drag	$\{0,\ 0.5,\ 1\}$	$\{0.25,\ 0.75\}$
	elasticity	$\{0,\ 0.5,\ 1\}$	$\{0.25,\ 0.75\}$
Push Rope	bending stiffness	$\{0,\ 0.5,\ 1\}$	$\{0.25,\ 0.75\}$
	bending damping	$\{0,\ 0.5,\ 1\}$	$\{0.25,\ 0.75\}$
Pour Water	viscosity	$\{0,\ \frac{1}{3},\ \frac{2}{3},\ 1\}$	$\{\frac{1}{6},\ \frac{1}{2},\ \frac{5}{6}\}$
	particle cohesion	$\{0,\ \frac{1}{3},\ \frac{2}{3},\ 1\}$	$\{\frac{1}{6},\ \frac{1}{2},\ \frac{5}{6}\}$

Appendix

Table 13: Physics interpolation split. Values are shown in range-normalized parameter coordinates. Interpolation values are excluded from mechanism learning and world-model training and are used only for held-out evaluation.

Method	Push Cube		Push Rope		Pour Water		Avg. NPE $\downarrow$
Method	$\Delta_{\mathrm{world}}\downarrow$	NPE $\downarrow$	$\Delta_{\mathrm{world}}\downarrow$	NPE $\downarrow$	$\Delta_{\mathrm{world}}\downarrow$	NPE $\downarrow$	Avg. NPE $\downarrow$
ACWM-DiT	0.02431	0.218	0.01542	0.194	0.03876	0.267	0.226
Vid2World	0.02046	0.193	0.01327	0.176	0.03418	0.239	0.203
CoCo	0.01683	0.164	0.01109	0.151	0.02874	0.208	0.174
Twin Rollouts	0.01812	0.178	0.01221	0.162	0.03136	0.221	0.187
Shared Noise	0.03268	0.241	0.01821	0.216	0.04357	0.291	0.249

Appendix

Table 14: Generalization to unseen interpolated physics. We report the shared-world gap and normalized physical parameter error (NPE) on held-out joint interpolation configurations. Lower is better for both metrics.

Action-Conditioned World Models

Action-conditioned world models predict how environments change under actions to support planning, control, and simulation.

See all 343 papers

Oct 21, 2026

PhysiCalWorld: In-Context Physics Calibration for World Models

Existing world models typically perceive a scene through current visual frame or a narrow history window. However, these observations may not reveal the physics properties that determine transition dynamics, such as an object's mass distribution, joint damping, or surface friction. Without calibrating these physics factors for world models, predictions tend to regress toward an empirical prior and induce severe drift in novel test scenes. To this end, we propose PhysiCalWorld, which calibrates action-conditioned world models to the instance-specific dynamics via in-context prompting, without fine-tuning or analytical physics modeling. Our method encodes an exploratory interaction clip collected within the test scene into a compact physics descriptor token, which grounds subsequent predictions in the scene's actual physical behavior. PhysiCalWorld is trained end-to-end with the latent prediction loss alone, without physics labels. At inference time, the descriptor token is extracted once and can be reused across rollouts with negligible compute overhead. We evaluate PhysiCalWorld across four environments covering diverse physics variations and show that it substantially improves rollout prediction accuracy and visual planning success rate. The learned descriptors also form a physically structured latent space and generalize beyond the physics ranges seen during training.

Yunhao Luo, Wentao He, Yinpei Dai +5

University of Michigan

World Models Action-Conditioned World Models World Model-Based Planning ICL for Robotics

Ask PDF

Aug 10, 2026

Twin Rollouts: Noise-Coupled Counterfactual Branching in Interactive Video World Models

Interactive video world models generate rollouts autoregressively under an action stream, yet they are trained and evaluated almost exclusively on factual prediction. We study counterfactual generation inside the rollout: given a trajectory the model has itself generated, what would have happened had the actions differed from step t* onward? We formalize noise-coupled twin rollouts --- a factual and a counterfactual branch sharing the generated prefix and the future exogenous noise sequence, diverging only in the action stream at an intervention point. Because the factual branch is self-generated, its exogenous noise is known exactly: the abduction step of Pearl's counterfactual procedure is exact by construction, sidestepping the approximate-inversion problem faced by editing-based pipelines. Noise coupling further turns the minimal-change principle into a per-sample verifiable property: we define a spatiotemporal locality metric that penalizes divergence outside the causal descendants of the intervention, computable against simulator ground truth without a learned judge. Forking the simulator state at t* yields ground-truth counterfactual re-renders, which we use as verifiable rewards for post-training. This note establishes the formal framework, metric definitions, and positioning; experiments are forthcoming.

Yu Ma, Hongli Shi, Xinran Xu

Action-Conditioned Video Generation Counterfactual Evaluation Causal World Models Causal Counterfactual Generation Video Generation Video World Models

Ask PDF

May 26, 2026

What-If World: A Causal Benchmark for General World Models in Embodied Scenarios

Video generation models are increasingly used as world simulators for tasks like driving and robotic manipulation. What matters in these settings is not whether a single video looks right, but whether the model's output changes when its input changes. We test this by giving a model two prompts describing the same scene with one physical detail varied, and checking whether the two videos diverge the way physics predicts. The wording difference between the prompts is small by design, since only one variable is changed, but the correct physical difference is not. A model that misses this can still produce two videos that each look plausible individually, and existing benchmarks score videos one at a time and cannot detect this failure. We introduce What-If World, 319 such prompt pairs built on real frames from nuScenes and DROID, organized by a taxonomy of six physical variables shared across driving and manipulation. Each pair is scored with APEO, a four-part rubric checking whether each video follows its prompt (Adherence), is physically consistent (Physics), preserves the shared scene (Environment), and ends in the correct difference (Outcome). Across nine state-of-the-art models, no system exceeds 52% on the paired score, and open-source models cluster near 28%. Every model tested fails on a large fraction of causal interventions, indicating substantial room before these models can reliably support action-conditioned simulation or model-based planning. Where models do score well, performance appears to track the visual prominence of the intervention rather than the tractability of its underlying physics. Some visually subtle interventions score as low as 14.2%, while visually pronounced ones reach 40.4%.

Kunlin Cai, Rui Song, Jinghuai Zhang +7

1UCLA · University of Tennessee · 3Amazon

Physical Consistency in Video Generation Causal World Models Causal Counterfactual Generation Video Generation Video Reasoning Causal Reasoning in Videos+2

Ask PDF

Abstract

Action-Conditioned World Models

Explore similar work

PhysiCalWorld: In-Context Physics Calibration for World Models

Twin Rollouts: Noise-Coupled Counterfactual Branching in Interactive Video World Models

What-If World: A Causal Benchmark for General World Models in Embodied Scenarios