Coupled Local and Global World Models for Efficient First Order RL

Joseph Amigo; Rooholla Khorrambakht; Nicolas Mansard; Ludovic Righetti

cs.ROFeb 5, 2026

Coupled Local and Global World Models for Efficient First Order RL

Authors: Joseph Amigo, Rooholla Khorrambakht, Nicolas Mansard, Ludovic Righetti

Organizations: Machines in Motion Laboratory, New York University, USA · LAAS-CNRS, Universit´e de Toulouse, CNRS, Toulouse, France · Artificial and Natural Intelligence Toulouse Institute, Toulouse, France

Abstract

World models offer a promising avenue for more faithfully capturing complex dynamics, including contacts and non-rigidity, as well as complex sensory information, such as visual perception, in situations where standard simulators struggle. However, these models are computationally complex to evaluate, posing a challenge for popular RL approaches that have been successfully used with simulators to solve complex locomotion tasks but yet struggle with manipulation. This paper introduces a method that bypasses simulators entirely, training RL policies inside world models learned from robots' interactions with real environments. At its core, our approach enables policy training with large-scale diffusion models via a novel decoupled first-order gradient (FoG) method: a full-scale world model generates accurate forward trajectories, while a lightweight latent-space surrogate approximates its local dynamics for efficient gradient computation. This coupling of a local and global world model ensures high-fidelity unrolling alongside computationally tractable differentiation. We demonstrate the efficacy of our method on the Push-T manipulation task, where it significantly outperforms PPO in sample efficiency. We further evaluate our approach through an ego-centric object manipulation task with a quadruped. Together, these results demonstrate that learning inside data-driven world models is a promising pathway for solving hard-to-model RL tasks in image space without reliance on hand-crafted physics simulators.

Figures & tables

Figure 1: Real-world robotic tasks solved zero-shot by policies trained entirely from scratch within our learned world models: Push-T with a tabletop manipulator (left), Ego-Centric Grasp and Lift with a G1 humanoid (center), and Ego-Centric Push Cube with a Go2 quadruped (right).

Figure 2: Overview of the proposed approach. Global world/reward models are learned from play/demonstration data (Steps 1 and 2). Then, local world/reward models are pre-trained (Step 3). Policy is optimized with the DMO algorithm (Step 4): forward simulation uses the diffusion-based global world model in pixel space, trained on real robot data, to produce high-fidelity rollouts. The backward pass uses gradients computed from a local world model operating in a low-dimensional latent space. The local world/reward model is further fine-tuned during policy optimization.

Figure 3: Real-world zero-shot deployment of our DMO policies trained in the world model.

Figure 4: Unrolling of real and model-predicted trajectories on the G1 humanoid manipulation task. We compare the real camera observation ( top ), the local DreamerV3 RSSM trajectory ( middle ), and the global DreamerV4 diffusion trajectory ( bottom ) at time steps $0,5,10,\dots,55$ ( $5.5$ s at $10$ Hz), with both models initialized from the first real frame. The global diffusion model maintains high visual fidelity and accurate contact dynamics over the long horizon.

Figure 5: Aggregated performance metrics across all evaluation tasks. Left: Normalized reward relative to normalized sample number; offline RL (IQL) uses no environment samples and is shown as a horizontal line at its best evaluation. Right: Normalized reward relative to normalized wall-clock time. Every online experiment was run with 4 seeds.

Method	Success Rate ( $\uparrow$ )	Steps to Success ( $\downarrow$ )	Straightness Index ( $\uparrow$ )	Curvature (rad/m) ( $\downarrow$ )
DMO (Ours)	90%	177.2 $\pm$ 52.2	0.520 $\pm$ 0.266	0.692 $\pm$ 0.206
BC (ACT) †	60%	371.0 $\pm$ 94.0	0.330 $\pm$ 0.147	0.761 $\pm$ 0.151
PPO	40%	173.5 $\pm$ 20.3	0.233 $\pm$ 0.182	1.222 $\pm$ 0.306
No Diffusion	0%	–	–	–

Table 1: Trajectory Quality Analysis. We report the mean

\pm

std over

N=10

randomly sampled evaluation episodes on the real robot for the Push Cube task. Success Rate : fraction of episodes reaching the goal; Steps : lower is better; Straightness : displacement/path length,

\uparrow

is straighter; Curvature : yaw change per meter,

\downarrow

is smoother. Metrics computed over successful episodes only. † For the BC baseline, success is defined as guiding the cube to the immediate pre-goal vicinity, as the cube never fully enters the goal.

Table 7

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: (a) Unrolling of real and model-predicted trajectories comparing the real trajectory (top), the DIAMOND diffusion trajectory (middle), and the DreamerV3 trajectory (bottom), every 5 steps ( $12$ s). (b) Unrolling of real and model-predicted trajectories comparing the real trajectory (top), the DreamerV3 trajectory (middle), and the DreamerV4 diffusion trajectory (bottom), every 5 steps ( $12$ s), with both models initialized with zero context, and the cube initially occluded. Note that the local model violates object permanence by spawning the cube throughout the rollout.

Figure 7: Visual comparison of the world model before and after patching on the Unitree G1 Grab Box task. Rows labeled original display the unpatched world model where the RL policy discovered physical exploits to artificially maximize reward. In the first rollout ( top two rows ), the policy exploits the model by moving the hand into a lifting pose, causing the box to spontaneously teleport into its grasp. In the second rollout ( bottom two rows ), the hand remains distant but pulls the box toward itself using an “invisible force.” Rows labeled patched show the corrected world model rollouts after being fine-tuned with real-world data collected by deploying these exploiting policies, effectively eliminating the physical hallucinations.

Panel of Figure 8 — Figure 8: (a) Efficiency comparison on the Push-T task: sample efficiency of DMO (4M samples) versus PPO (48M samples), SAC, and the No Diffusion and No RSSM Finetuning ablations (left), and corresponding time efficiency comparison (right). (b) Efficiency comparison on the Push Cube task: sample efficiency of DMO (4M samples) versus PPO (25M samples), SAC, and the No Diffusion and No RSSM Finetuning ablations (left), and corresponding time efficiency comparison (right). (c) Efficiency comparison on the G1 Grab Box task: sample efficiency of DMO (4M samples) versus PPO (25M samples), SAC, and the No Diffusion and No RSSM Finetuning ablations (left), and corresponding time efficiency comparison (right). The dashed line is the best evaluation of offline RL (IQL), which uses no environment samples. Every experiment was run with 4 seeds.

Figure 9: Three real-robot Push-T trajectories executed by the policy learned with our approach.

Figure 10: Three real-robot G1 Grab Box trajectories executed by the policy learned with our approach.

Figure 11: Comparison of search strategies under partial observability. Top: When the cube is out of view, the BC policy (ACT) executes a naive backward retreat, which fails to locate the object if it is directly behind. Bottom: The DMO policy exhibits an emergent active search behavior, rotating to visually scan the environment and successfully localize the target.

Figure 12: Real-world execution of the DMO-learned policy for the Go2 Push Cube task when using a different low-level policy than seen during training. Top: Standard successful push trajectory. Middle: Emergent active search behavior; when the object is initially out of view (or lost), the agent retreats and scans the environment to localize the cube before pushing. Bottom: Successful approach and push from a different initial configuration.

Figure 13: Task completion comparison. Top: The Behavior Cloning (ACT) policy successfully approaches the cube but stops pushing before the object enters the goal, reflecting the sub-optimal demonstration distribution. Bottom: Our RL Policy (DMO) generalizes beyond the demonstrations, learning to push the cube fully into the net to maximize the task reward.

RL Algorithms (All Tasks)
Parameter	PPO	DMO-SAPO
Image Size (Input)	$64\times 64$	$64\times 64$
Horizon ( $H$ )	64	16
Gamma ( $\gamma$ )	0.99	0.99
Lambda ( $\lambda$ )	0.95	0.95
Mini-batch Size	4096	1024

Appendix

Table 4: Hyperparameters. Settings for PPO and DMO-SAPO across the three tasks (in the RL block, values that differ across tasks are given as Push-T / Push Cube / G1 Grab Box), along with the specific World Model architectures used in our framework. DIAMOND is used for Push-T, while DreamerV4 is used for both Push Cube and G1 Grab Box.

Model-Based RL

Model-based reinforcement learning uses environment models to plan actions and learn policies with fewer real-world interactions.

See all 243 papers

WorldSample: Closed-loop Real-robot RL with World Modelling
Jul 2, 2026Yuquan Xue, Le Xu, Zeyi Liu +5Robot Skill Learning World Models for Robotics
1PINE Lab, School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Department of Electronic Engineering, Tsinghua University, Beijing, China · School of Automation, Central South University, Changsha, China +1
Interactive World Simulator for Robot Policy Training and Evaluation
Mar 9, 2026Yixuan Wang, Rhythm Syed, Fangyu Wu +7Interactive World Models World Model Learning
Columbia University · Toyota Research Institute · Amazon +1
DriftWorld: Fast World Modeling through Drifting
Jul 16, 2026Susie Lu, Haonan Chen, Weirui Ye +1World Models for Robotics World Model-Based Planning
Massachusetts Institute of Technology · Harvard University

Jul 2, 2026J/K move · Enter open · S save

WorldSample: Closed-loop Real-robot RL with World Modelling

Yuquan Xue, Le Xu, Zeyi Liu, Zhenyu Wu+4

1PINE Lab, School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Department of Electronic Engineering, Tsinghua University, Beijing, China · School of Automation, Central South University, Changsha, China+1

Robot Skill Learning World Models for Robotics Synthetic Data Generation for Robotics Robotic RL Robotic Manipulation Sample-Efficient RL

Open paper AskPDF arXiv

Reinforcement learning (RL) can overcome the demonstration-coverage limitation of imitation learning (IL) by allowing robots to improve through trial-and-error interaction beyond the states observed in demonstrations. However, deploying RL on real robots remains constrained by high interaction costs, since each physical rollout is costly and reflects only one realized action-outcome path. To address this challenge, we propose WorldSample, a physically grounded data augmentation framework for real-robot RL that closes a real-synthetic loop between physical rollouts, world-model generation, and policy improvement. Grounded on real rollouts, WorldSample generates high-fidelity synthetic transitions through a post-trained world model, which greatly lowers the visual hallucination. Specifically, rather than simply using these transitions as real-world experience, WorldSample introduces Policy-Paced Learning (PPL) to regulate the training process through sample selection and scheduling, balancing useful augmentation against value overestimation and mitigating the hallucination-induced noise. Experiments on robot manipulation tasks involving contact-rich and precise tasks show that WorldSample improves policy success rate by 28% while reducing training steps by 59% compared with baselines. Furthermore, WorldSample improves world model visual fidelity by 19.4dB in PSNR and 0.47 in SSIM over demonstration-only post-training, validating the effectiveness of the real-synthetic loop for both policy and world model performance.

Abstract

Model-Based RL

Explore similar work

WorldSample: Closed-loop Real-robot RL with World Modelling

Interactive World Simulator for Robot Policy Training and Evaluation

DriftWorld: Fast World Modeling through Drifting