Action-conditioned video prediction models (often referred to as world models) have shown strong potential for robotics applications, but existing approaches are often slow and struggle to capture physically consistent interactions over long horizons, limiting their usefulness for scalable robot policy training and evaluation. We present Interactive World Simulator, a framework for building interactive world models from a moderate-sized robot interaction dataset. Our approach leverages consistency models for both image decoding and latent-space dynamics prediction, enabling fast and stable simulation of physical interactions. In our experiments, the learned world models produce interaction-consistent pixel-level predictions and support stable long-horizon interactions for more than 10 minutes at 15 FPS on a single RTX 4090 GPU. Our framework enables scalable demonstration collection solely within the world models to train state-of-the-art imitation policies. Through extensive real-world evaluation across diverse tasks involving rigid objects, deformable objects, object piles, and their interactions, we find that policies trained on world-model-generated data perform comparably to those trained on the same amount of real-world data. Additionally, we evaluate policies both within the world models and in the real world across diverse tasks, and observe a strong correlation between simulated and real-world performance. Together, these results establish the Interactive World Simulator as a stable and physically consistent surrogate for scalable robotic data generation and faithful, reproducible policy evaluation.
Figures & tables
Fig. 2: Method Overview. Model training proceeds in two stages. In Stage 1, we train an autoencoder that maps RGB observations into compact 2D latent representations using a CNN encoder Eϕ and a consistency-model decoder Dθ , enabling efficient and high-fidelity reconstruction. In Stage 2, we freeze the autoencoder and train an action-conditioned consistency model Fψ in latent space to model future visual dynamics using next-frame supervision. We choose Fψ as a consistency model because it efficiently represents multimodal distributions over possible future outcomes. The dynamics model learns to denoise noisy latent states conditioned on robot actions, past latents, and noise levels, and is instantiated as a stack of 3D convolutional blocks with FiLM modulation and spatiotemporal attention. During inference, Fψ autoregressively predicts future latents, which are decoded by Dθ to generate long-horizon video predictions.
Fig. 3: Qualitative Comparison of Long-Horizon Action-Conditioned Video Prediction. We compare the proposed Interactive World Simulator with state-of-the-art action-conditioned video prediction models across multiple manipulation tasks. (a) Intermediate predicted frames for the Pile Sweeping task illustrate long-horizon rollout behavior, where baseline methods exhibit inaccurate dynamics, robot pose drift, accumulated artifacts, or loss of fine-grained details over time. (b) Terminal predicted frames for additional tasks, including Box Packing , Rope Routing , T Pushing , Rope Collecting , and Mug Grasping , highlight differences in visual fidelity and interaction consistency. In contrast to baseline methods, our model preserves coherent robot–object interactions and maintains stable predictions over extended horizons.
Metric
Ours
DINO-WM [ 3 ]
UVA [ 8 ]
Dreamer4 [ 16 ]
Cosmos [ 6 ]
MSE ↓
0.005 ± 0.005
0.028 ± 0.032
0.023 ± 0.015
0.012 ± 0.009
0.019 ± 0.010
LPIPS ↓
0.051 ± 0.019
0.270 ± 0.093
0.272 ± 0.077
0.163 ± 0.052
0.224 ± 0.060
FID ↓
63.50 ± 13.78
200.77 ± 79.02
142.55 ± 49.43
239.97 ± 45.75
200.74 ± 31.53
PSNR ↑
25.82 ± 2.72
17.79 ± 2.84
17.87 ± 2.21
20.81 ± 2.21
18.91 ± 1.73
SSIM ↑
0.831 ± 0.039
0.652 ± 0.059
0.650 ± 0.059
0.693 ± 0.045
0.647 ± 0.040
UIQI ↑
0.960 ± 0.019
0.875 ± 0.029
0.884 ± 0.025
0.919 ± 0.024
0.883 ± 0.029
TABLE I: Quantitative Comparison of Video Prediction Performance. We quantitatively evaluate state-of-the-art action-conditioned video prediction models across a diverse set of manipulation tasks involving rigid objects, deformable objects, articulated objects, and object piles. The table reports results aggregated across all tasks; per-task results are provided in the supplementary material. Performance is measured using standard video prediction metrics, including MSE, LPIPS, FID, PSNR, SSIM, UIQI, and FVD, computed between predicted and ground-truth video rollouts. All results are reported for action-conditioned predictions over extended horizons (192 steps). The proposed Interactive World Simulator consistently outperforms prior world models across metrics, indicating improved visual fidelity, interaction consistency, and long-horizon stability, while maintaining efficient inference speed.
Fig. 4: Teleoperation Interface for Real-Time Interaction with the World Simulator. We build a kinematic teleoperation device that allows users to control robot actions through the Interactive World Simulator. The left column shows a user issuing commands through the teleoperation device (e.g., moving forward, opening the gripper, and lifting the end-effector). The right columns show the corresponding frames generated by the world simulator in real time. The simulator follows the teleoperation commands and produces consistent robot–object interactions.
Fig. 5: Imitation Policy Training with Mixtures of Real-World Data and Our World Simulator Data. “WS %” represents the percentage of data from our world simulator, and “Real %” represents the percentage of data from the real world. From left to right, the mixtures range from 100% world simulator data to 100% real-world data. We use a total of 100 training episodes for each policy and data mixture. Performance among the policies trained on different data mixtures across different manipulation tasks is observed to be consistently high with comparable task scores, suggesting that data generated by our world simulator is comparable in quality to real-world demonstrations.
Fig. 6: Effect of Data Scaling on Policy Performance . We compare the performance of ACT and DP trained on datasets of varying sizes, ranging from 5 to 100 expert demonstration episodes. For each dataset size, policies are trained using either 100% MuJoCo data or 100% data generated from our world simulator. Policy performance improves consistently as the number of demonstrations increases across both data sources, suggesting that the scaling behavior of policy training is similar when using world simulator data.
Fig. 7: Correlation Between Policy Performance in Our World Simulator and the Real World. Real-world task performance of imitation policies is plotted against their corresponding performance in our world simulator across multiple manipulation tasks. Each point corresponds to a trained policy evaluated under identical settings in both environments. Strong positive correlations are observed, indicating that evaluation within our world simulator closely reflects relative policy performance in the real world.
Action-conditioned video models offer a promising path to building general-purpose robot simulators that can improve directly from data. Yet, despite training on large-scale robot datasets, current state-of-the-art video models still struggle to predict physically consistent robot-object interactions that are crucial in robotic manipulation. To close this gap, we present PlayWorld, a simple, scalable, and fully autonomous pipeline for training high-fidelity video world simulators from interaction experience. In contrast to prior approaches that rely on success-biased human demonstrations, PlayWorld is the first system capable of learning entirely from unsupervised robot self-play, enabling naturally scalable data collection while capturing complex, long-tailed physical interactions essential for modeling realistic object dynamics. Experiments across diverse manipulation tasks show that PlayWorld generates high-quality, physically consistent predictions for contact-rich interactions that are not captured by world models trained on human-collected data. We further demonstrate the versatility of PlayWorld in enabling fine-grained failure prediction and policy evaluation, with up to 40% improvements over human-collected data. Finally, we demonstrate how PlayWorld enables reinforcement learning in the world model, improving policy performance by 65% in success rates when deployed in the real world.
Video world models are emerging as a scalable alternative for evaluating generalist robot policies, bypassing the physical constraints and engineering burdens of real-world deployment. However, evaluating policies with video world models remains challenging, as world-model errors can make generated rollouts unreliable and slow inference limits large-scale throughput. We introduce RoboWorld, an automated evaluation pipeline that pairs a fast autoregressive video world model with a task-progress-aware vision-language model scoring. To enable reliable long-horizon autoregressive world-model rollouts, we propose Step Forcing, which combines anchored and one-step self-forwarded contexts to reduce train-test mismatch while preserving action-observation dynamics. Together, these components enable RoboWorld to align strongly with real-world robot evaluation across tasks and environments, achieving Pearson's r = 0.989 and Spearman's ρ = 0.970.
Interactive world simulators can provide scalable environments for robot planning, policy training, and evaluation by predicting action consequences while reducing reliance on repeated physical rollouts. To serve these applications, they must generate future image sequences that respond faithfully to robot actions and preserve the dynamics of robot-object interactions over long horizons. However, existing world models typically predict the entire next latent state and often fail to capture subtle changes induced by robot actions. Such omissions can produce physically implausible outcomes, including object interpenetration and excessive deformation. To address this limitation, we propose DeltaWorld, a physically consistent interactive world simulator for robotic manipulation. Our method introduces the Delta Latent Transition Model (Delta-LTM), which predicts action-induced latent feature changes and adds them to the current latent state to obtain the next state, rather than predicting the next latent state directly. To mitigate object interpenetration and excessive deformation in predicted future frames, Interaction-aware Latent Alignment is introduced to construct counterfactual interaction regions and supervise interaction-related latent changes. DeltaWorld is evaluated on the IWS manipulation benchmark and a self-collected cross-robot dataset covering multiple robot embodiments and manipulation tasks. On the cross-robot dataset, DeltaWorld reduces FVD by 46.6% and LPIPS by 31.1% relative to the IWS baseline. These results highlight the potential of DeltaWorld for long-horizon action-conditioned video prediction in robotic manipulation.
Boyuan Hou, Xiaoge Cao, Chaofan Zhang +2
Imprintx Robotics · Electric Power Research Institute of State Grid Jibei Electric Power Co., Ltd. · North China Electric Power Research Institute Co., Ltd. +1