Persistent Robot World Models: Stabilizing Multi-Step Rollouts via Reinforcement Learning

Jai Bardhan; Patrik Drozdik; Josef Sivic; Vladimir Petrik

cs.ROMar 26, 2026

Persistent Robot World Models: Stabilizing Multi-Step Rollouts via Reinforcement Learning

Authors: Jai Bardhan, Patrik Drozdik, Josef Sivic, Vladimir Petrik

Organizations: Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague

Abstract

Action-conditioned robot world models generate future video frames of the manipulated scene given a robot action sequence, offering a promising alternative for simulating tasks that are difficult to model with traditional physics engines. However, these models are optimized for short-term prediction and break down when deployed autoregressively: each predicted clip feeds back as context for the next, causing errors to compound and visual quality to rapidly degrade. We address this through the following contributions. First, we introduce a reinforcement learning (RL) post-training scheme that trains the world model on its own autoregressive rollouts rather than on ground-truth histories. We achieve this by adapting a recent contrastive RL objective for diffusion models to our setting and show that its convergence guarantees carry over exactly. Second, we design a training protocol that generates and compares multiple candidate variable-length futures from the same rollout state, reinforcing higher-fidelity predictions over lower-fidelity ones. Third, we develop efficient, multi-view visual fidelity rewards that combine complementary perceptual metrics across camera views and are aggregated at the clip level for dense, low-variance training signal. Fourth, we show that our approach establishes a new state-of-the-art for rollout fidelity on the DROID dataset, outperforming the strongest baseline on all metrics (e.g., LPIPS reduced by 14% on external cameras, SSIM improved by 9.1% on the wrist camera), winning 98% of paired comparisons, and achieving an 80% preference rate in a blind human study.

Figures & tables

Figure 1: Autoregressive video rollout quality. We use an action-conditioned robot world model to generate multi-view image predictions from a single observed state. The ground-truth ( blue ) is compared against the baseline ( red ) and our post-trained model PersistWorld ( green ). While the baseline accumulates error and destroys the object (cyan bowl) within seconds, our method maintains structural integrity and spatial consistency, establishing a new state-of-the-art in rollout fidelity.

Figure 2: Overview of our method: (Top) Autoregressive inference: A robot policy generates actions fed to the world model, which produces multi-view frames that are appended to the history buffer and condition the next generation step. (Bottom) RL post-training: (S1) A shared variable-length prefix is rolled out autoregressively from a ground-truth initial condition. (S2) $K$ independent candidate continuations are branched from the frozen prefix state. (S3) Candidates are scored against ground-truth using multi-view perceptual rewards. (S4) Reward weights $r$ scale implicit positive/negative $\mathbf{x}_{0}$ predictions used in contrastive model updates via loss $L$ .

Evaluated Cameras	Model	Pixel/Structure		Perceptual
Evaluated Cameras	Model	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
External Camera	WPE ∗	0.77	20.33	0.131
	IRASim ∗	0.77	21.36	0.117
	Ctrl-World ∗	0.83	23.56	0.091
	Ctrl-World †	0.84	23.02	0.081
	Ours	0.86	24.42	0.070

Table 1: Visual quality metrics for 14-step autoregressive rollouts (

\approx

11 s) on the DROID validation split. Values represent averages over the full rollout duration. Results marked with ∗ are from [ 12 ] ; † indicates our reproduction.

Panel of Figure 3 — Figure 3: Qualitative comparison of autoregressive rollout stability. We compare long-horizon (11 s) generations from the baseline [ 12 ] against our PersistWorld for the wrist camera. Left: Object-centric fidelity. The baseline model suffers from rapid decoherence; as errors compound in the history buffer, manipulated objects like the cup lose their structural identity and dissolve into amorphous textures. In contrast, our method maintains the spatial consistency and structural integrity of the object throughout the rollout. Right: Robot-centric consistency. The baseline exhibits significant robot decoherence , where the generated robot arm loses their geometric structure. Our approach maintains structural persistence. Please see additional video results on the associated project page.

Panel of Figure 4 — Figure 4: $\Delta_{\text{metric}}$ of paired videos from the validation dataset. On $1-1$ paired comparison, our PersistWorld world model is better than the baseline on $\sim 98\%$ of the sample ( $p<10^{-6}$ ).

Panel of Figure 5 — Figure 5: Temporal evolution of wrist camera metrics. While both models exhibit natural degradation over longer horizons (x-axis), our post-trained model, PersistWorld (green), consistently maintains higher fidelity and slower error accumulation compared to the baseline (orange). Specifically, our method preserves a higher PSNR and SSIM while suppressing LPIPS drift, effectively extending the stable prediction horizon for complex, fine-grained interactions. See Fig. 11 in Appendix. 0.E.2 for external camera results.

Evaluated Cameras	Model	Object-Only			Robot-Only
Evaluated Cameras	Model	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
External Camera	Ctrl-World †	0.88	22.25	0.025	0.82	17.62	0.039
External Camera	Ours	0.89	23.60	0.021	0.86	19.25	0.033
Wrist Camera	Ctrl-World †	0.73	18.52	0.088	0.83	25.50	0.027
Wrist Camera	Ours	0.76	19.87	0.078	0.86	27.24	0.023

Table 2: Masked visual metrics for 14-step autoregressive rollouts (

\approx

11 s) . We isolate object-only and robot-only pixels to evaluate task-relevant spatial and control consistency. Our model demonstrates superior fidelity in these dynamic regions compared to baselines, confirming that performance gains are driven by accurate interaction modeling rather than background reconstruction. All metrics are averaged over the full rollout horizon.

Figure 6: Human preference results.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Learning Rate	External			Wrist
Learning Rate	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
$3\times 10^{-4}$	$\underline{0.865}$	$\underline{25.02}$	$\underline{0.1230}$	$\underline{0.706}$	$\underline{19.95}$	$\underline{0.3603}$
$1\times 10^{-4}$ (Ours)	$\mathbf{0.872}$	$\mathbf{25.52}$	$\mathbf{0.1162}$	$\mathbf{0.721}$	$\mathbf{20.40}$	$\mathbf{0.3387}$

Appendix

Table 3: Ablation of the post-training learning rate.

Reward	External			Wrist
Reward	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
LPIPS only	$0.868$	$25.17$	$\mathbf{0.1156}$	$0.708$	$19.90$	$\mathbf{0.3242}$
SSIM only	$\mathbf{0.872}$	$25.40$	$0.1178$	$\mathbf{0.724}$	$20.24$	$0.3516$
PSNR only	$\underline{0.871}$	$\mathbf{25.57}$	$0.1189$	$0.716$	$\mathbf{20.45}$	$0.3544$
Combined (Ours)	$\mathbf{0.872}$	$\underline{25.52}$	$\underline{0.1162}$	$\underline{0.721}$	$\underline{20.40}$	$\underline{0.3387}$

Appendix

Table 4: Ablation of the reward signal used during DiffusionNFT post-training.

Strategy	External			Wrist
Strategy	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
Fixed
Size 3	$\underline{0.872}$	$\mathbf{25.53}$	$0.1170$	$\mathbf{0.723}$	$\mathbf{20.51}$	$\mathbf{0.3381}$
Size 6	$0.871$	$25.41$	$0.1174$	$0.719$	$20.33$	$0.3423$
Curriculum [ 41 ]	$0.871$	$25.50$	$0.1169$	$0.721$	$20.43$	$0.3412$
Random

Appendix

Table 5: Ablation of the context-window curriculum. Curriculum is the growing schedule from [ 41 ] ; Random samples uniformly from the specified range each step; Fixed keeps a constant window size throughout training.

Config	External			Wrist
Config	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
Group Size 4	$\underline{0.875}$	$25.62$	$0.1142$	$0.724$	$20.56$	$0.3379$
Group Size 8	$0.873$	$25.61$	$0.1176$	$0.721$	$20.45$	$0.3471$
Group Size 16
No best of $N$	$0.874$	$\underline{25.69}$	$\underline{0.1127}$	$\underline{0.725}$	$\underline{20.64}$	$\underline{0.3255}$
Best of 5 (Ours)	$0.872$	$25.52$	$0.1162$	$0.721$	$20.40$	$0.3387$

Appendix

Table 6: Ablation of the GRPO group size and the number of candidates retained via best-of-

N

filtering (evaluated at group size 16).

Horizon $H$	External			Wrist
Horizon $H$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
$H=3$	$\mathbf{0.872}$	$\underline{25.41}$	$\underline{0.1169}$	$\underline{0.719}$	$\underline{20.31}$	$\underline{0.3424}$
$H=1$ (Ours)	$\mathbf{0.872}$	$\mathbf{25.52}$	$\mathbf{0.1162}$	$\mathbf{0.721}$	$\mathbf{20.40}$	$\mathbf{0.3387}$

Appendix

Table 7: Ablation of the future prediction horizon

H

(number of frames generated per autoregressive step) used during post-training.

$w_{\mathrm{wrist}}$	External			Wrist
$w_{\mathrm{wrist}}$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
$w_{\mathrm{wrist}}=2$	$\mathbf{0.872}$	$\underline{25.45}$	$\underline{0.1169}$	$\mathbf{0.722}$	$\mathbf{20.47}$	$\mathbf{0.3357}$
$w_{\mathrm{wrist}}=1$ (Ours)	$\mathbf{0.872}$	$\mathbf{25.52}$	$\mathbf{0.1162}$	$\underline{0.721}$	$\underline{20.40}$	$\underline{0.3387}$

Appendix

Table 8: Ablation of the wrist-camera loss weight

w_{\mathrm{wrist}}

relative to the external cameras (weight 1).

KL Reg.	External			Wrist
KL Reg.	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
Without	$\mathbf{0.873}$	$\underline{25.50}$	$\underline{0.1163}$	$\underline{0.720}$	$\underline{20.38}$	$\underline{0.3392}$
With (Ours)	$\underline{0.872}$	$\mathbf{25.52}$	$\mathbf{0.1162}$	$\mathbf{0.721}$	$\mathbf{20.40}$	$\mathbf{0.3387}$

Appendix

Table 9: Effect of KL regularisation applied to the post-training objective.

Reference Policy EMA	External			Wrist
Reference Policy EMA	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
No EMA schedule	$\mathbf{0.872}$	$\underline{25.51}$	$\mathbf{0.1162}$	$\mathbf{0.722}$	$\mathbf{20.46}$	$\underline{0.3390}$
EMA rising to $0.5$ (Ours)	$\mathbf{0.872}$	$\mathbf{25.52}$	$\mathbf{0.1162}$	$\underline{0.721}$	$\underline{20.40}$	$\mathbf{0.3387}$

Appendix

Table 10: Effect of applying a warming EMA schedule to the frozen reference (old) policy, which gradually interpolates it toward the learning policy during training.

Policy EMA	External			Wrist
Policy EMA	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
EMA $=0.9$	$0.876$	$25.81$	$0.1105$	$0.726$	$20.68$	$0.3215$
No EMA (Group Size 16, BoN 2)	$0.876$	$25.79$	$0.1107$	$0.727$	$20.69$	$0.3212$

Appendix

Table 11: Effect of applying an exponential moving average (EMA) to the learning (fine-tuned) policy weights during post-training.

Panel of Figure 7 — Figure 7: Initial Conditions for the WM-to-Real Correlation. We run rollouts in the world models given these starting conditions for the three tasks. For the policy rollout, the middle view (external camera) and the wrist view are provided as inputs.

Skill	Task	Progression
Put	Put Banana in Box	Reach $\rightarrow$ Grasp $\rightarrow$ Lift $\rightarrow$ Move Close $\rightarrow$ IsInside
Put	Put Green Block in Bowl
Rotate	Rotate Marker	Reach $\rightarrow$ Grasp $\rightarrow$ Rotate $45^{\circ}$

Appendix

Table 12: Partial Progress of the Different Tasks .

Figure 8: WM to Real comparison: While both the world models tend to make the task easier for the policies (shown by the higher values task performance on the WM), our post-trained model maintains better correlation and MMRV amongst the policies compared to the baseline [ 12 ] .

Figure 9: 2AFC Website Onboarding: We provide a small description of the task and potential hints to look at when judging the quality of the generated rollouts compared to the reference.

Figure 10: 2AFC Website Comparison UI: On the comparison webpage, the users are provided with the reference video on the top and the two generated videos below it. The users can use the scrubber to move through the frames or watch the entire video. After which, the user needs to choose between Option A or Option B.

Panel of Figure 11 — Figure 11: Temporal evolution of external camera metrics. While both models exhibit natural degradation over longer horizons (x-axis), our post-trained model, PersistWorld (green), consistently maintains higher fidelity and slower error accumulation compared to the baseline (orange). Specifically, our method preserves a higher PSNR and SSIM while suppressing LPIPS drift, effectively extending the stable prediction horizon for complex, fine-grained interactions.

Reward	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	training sec/iter $\downarrow$
HPSv3 (theirs)	0.802	22.58	0.193	129.2
Ours (top- $k{=}2$ )	0.826	24.09	0.181	41.3

Appendix

Table 13: Our reward vs HPSv3 [ 30 ] : To compare to WorldCompass [ 41 ] , we compare their reward structure with HPSv3 [ 30 ] to our light-weight rewards.

Panel of Figure 12 — Figure 12: Qualitative comparison of autoregressive rollout stability on Bridge [ 9 ] and RT-1 [ 7 ] . We compare long-horizon (11s) generations from the baseline against our PersistWorld . In both the cases, we see that the baseline model struggles to maintain object consistency. In left, we see that the saucepan has completely disappeared in the baseline model, while PersistWorld is able to maintain it. In right, we see that the baseline model has lost the bottle, and instead hallucinates a different object within the robot’s gripper. In contrast, PersistWorld is able to maintain the bottle and correctly shows the gripper grasping the bottle.

Dataset	Method	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
Bridge	IRASim [ 49 ] (long-traj) †	–	23.26	–
	SVD backbone	0.864	25.20	0.1061
	+ PersistWorld ( Ours )	0.876	26.36	0.0957
RT-1	IRASim [ 49 ] (long-traj) †	–	24.62	–
	SVD backbone	0.833	23.81	0.1683
	+ PersistWorld ( Ours )	0.865	26.63	0.1305

Appendix

Table 14: Generalization Beyond DROID: We compare the performance of the standard action conditioned pretraining on SVD backbone to PersistWorld on a long rollout with

10

consecutive interactions. We also report IRASim [ 49 ] on their long trajectory setup (

\dagger

indicates the number is taken from original paper).

Figure 13: Ours vs SF style baseline: Our reward exceeds the SF-style baseline. Our procedure achieves the same performance

\sim 10\times

faster and achieves a much higher reward at the same number of steps.

Figure 14: Policy Improvement Setup: We place 4 blocks on a table and command the robot to pick up the top-right/top-left/bottom-right/bottom-left block in the $2\times 2$ grid. We also place some distractors on the table such as the bowl and the tape.

World Models for Robotics

World models for robotics predict how scenes and robot states change under actions to support planning, control, and learning.

See all 380 papers

FutureWorlds: Learning Robotic World Models from Alternative Futures
Oct 1, 2026Hao Wu, Shengju Qian, Weiyan Wang +5Action-Conditioned Video Generation World Model Learning
HKUST (GZ) · CUHK · Tencent +3
RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation
Jul 1, 2026Byeongguk Jeon, Seonghyeon Ye, JaeHyeok Doo +4World Model Learning Robotic Policy Evaluation
1KAIST · 2Config
DriftWorld: Fast World Modeling through Drifting
Jul 16, 2026Susie Lu, Haonan Chen, Weirui Ye +1World Models for Robotics World Model-Based Planning
Massachusetts Institute of Technology · Harvard University

Oct 1, 2026J/K move · Enter open · S save

FutureWorlds: Learning Robotic World Models from Alternative Futures

Hao Wu, Shengju Qian, Weiyan Wang, Fan Xu+4

HKUST (GZ) · CUHK · Tencent+3

Action-Conditioned Video Generation World Model Learning World Models for Robotics RL for Generative Models Video World Models

Open paper AskPDF arXiv

Robotic world models predict action-conditioned future scenes, providing a foundation for understanding action outcomes. However, turning alternative predictions into useful learning signals remains challenging: similar candidates limit informative quality comparisons, while diverging trajectories require persistent maintenance of their individual histories. We introduce FutureWorlds, a framework that unifies candidate construction, history maintenance, and learning from relative quality. Built on a multimodal discrete autoregressive model, FutureWorlds uses diverse beam search during reinforcement learning to construct candidate futures that balance confidence and diversity. Candidate-specific bounded memory preserves scene states and ensures that generation and policy scoring use matching histories. We further propose MemSPO (Memory-Conditioned Search-Guided Policy Optimization), which converts video trajectory rewards into group-relative advantages to optimize the world model. On RT-1, BridgeV2, and RoboCasa, FutureWorlds reduces LPIPS for 32-frame predictions by 14.78%, 20.84%, and 9.12%, respectively, relative to the strongest baseline on each dataset. Under fixed evaluation configurations, only 200 MemSPO updates further improve generation quality and support continued prediction beyond the training horizon. Memory ablations, decoding sensitivity analysis, and optical-flow evaluation show that these gains extend beyond visual quality to more accurate motion prediction and more consistent object states. Project page and code: https://github.com/Alexander-wu/FutureWorlds.

Abstract

World Models for Robotics

Explore similar work

FutureWorlds: Learning Robotic World Models from Alternative Futures

RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation

DriftWorld: Fast World Modeling through Drifting