Organizations: National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · Mila - Quebec AI Institute · Université de Montréal
Vision-Language-Action (VLA) models show strong generalization for robotic control, but finetuning them with reinforcement learning (RL) is constrained by the high cost and safety risks of real-world interaction. Training VLA models in interactive world models avoids these issues but introduces several challenges, including pixel-level world modeling, multi-view consistency, and compounding errors under sparse rewards. Building on recent advances across large multimodal models and model-based RL, we propose VLA-MBPO, a practical framework to tackle these problems in VLA finetuning. Our approach has three key design choices: (i) adapting unified multimodal models (UMMs) for data-efficient world modeling; (ii) an interleaved view decoding mechanism to enforce multi-view consistency; and (iii) chunk-level branched rollout to mitigate error compounding. Theoretical analysis and experiments across simulation and real-world tasks demonstrate that VLA-MBPO significantly improves policy performance and sample efficiency, underscoring its robustness and scalability for real-world robotic deployment.
Figures & tables
Figure 1: The framework illustration of VLA-MBPO, comprising (A) a UMM-based world model with interleaved view decoding for multi-view observation and reward prediction, and (B) a stable and scalable policy update algorithm with chunked-level branch rollout. (C) illustration of both simulated and real-world task designs in our work.
Figure 2: Frame-skipping scheme in UMM-based World Model.
Model
Dynamics Model
Reward Model
Head View
Wrist View
Inf. Time ↓
ACC ↑
F1 ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
Ctrl-World
0.150
21.95
0.882
0.435
13.87
0.680
21
–
–
Qwen3-VL-8B
–
–
–
–
–
–
–
97.0
0.841
UMM-World
0.094
23.29
0.906
0.254
18.76
0.751
10
98.4
0.861
- w/o IVD
0.116
21.71
0.895
0.454
13.38
0.559
8
98.5
0.799
Table 1: Performance comparison of various world models across multiple metrics, including dynamics and reward model evaluations.
Model
LIBERO
Spatial
Object
Goal
Long
Avg
One-Trajectory SFT
π0.5 (SFT)
78.2
88.6
85.8
54.6
76.8
BC (WM)
80.6
89.8
85.0
48.6
76.0
πRL
86.0
92.4
90.8
61.2
82.6
IDQL
79.0
92.4
86.4
52.2
77.5
Table 2: Performance comparison on LIBERO benchmark.
Figure 3: The learning dynamics of value models. The blue lines represent the ground-truth return, while the orange lines represent the return estimated by the value model.
Figure 4: Qualitative visualization results of UMM-World.
Figure 5: Performance comparison on real-world tasks.
Rollouts Scheme
LIBERO Long
Branched Rollout
Full Horizon
1
2
4
VLA-MBPO
63.9
66.8
62.9
52.8
Table 3: Results of different rollouts scheme for VLA-MBPO.
Figure 6: Results of different sample size for VLA-MBPO.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: An illustration of real-world robots.
Figure 8: An illustration of real world assets.
Figure 9: An illustration of real world tasks. We test VLA-MBPO on bimanual and whole-body robots.
Figure 10: Causal Mask in UMM World Model during training . VAE and ViT denote VAE features and ViT features, respectively. t is the noise timestep and t=0 means no noise.
Parameters
Value
lr
2e−5
Cross entropy weight
0.01
MSE weight
1.0
Max latent size
64
Timestep shift
4
Cfg interval
[0.4, 1.0]
Appendix
Table 4: Hyperparameters for UMM-World.
Parameters
Libero
Real-World
Spatial
Object
Goal
Long
Sample size
512
512
512
1280
512
Batch size
512
512
512
512
512
Rollout chunk
2
2
2
2
2
Actor lr
5e−6
5e−6
5e−6
5e−6
5e−6
Critic lr
1e−4
1e−4
1e−4
1e−4
1e−4
Appendix
Table 5: Hyperparameters for RL.
Figure 11: Qualitative comparison of UMM-World prediction against Ground Truth across both simulated (LIBERO) and real-world tasks.
Figure 12: An illustration showing spatial consistency of UMM-World generation.
Figure 13: An illustration of failure case caused of partial observability.
Figure 14: An illustration of failure cases caused by large prediction horizon.
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China. · Huazhong University of Science and Technology, Wuhan, China. · Westlake University, Hangzhou, China. +1