Organizations: Institute of Automation, Chinese Academy of Sciences. · School of Artificial Intelligence, University of Chinese Academy of Sciences. · Zhongguancun Academy. · Xi’an Jiaotong-Liverpool University. · Tsinghua University. · Anyverse Dynamics.
Action-conditioned world models can reduce costly real-robot interaction in reinforcement learning for Vision-Language-Action models (VLA-RL), but hallucinations can mislead policy optimization. Extending them to richer VLA interfaces and long-horizon tasks raises challenges in spatiotemporal and cross-modal consistency, exploration-reliability balance in RL, and compounding rollout errors. We present WoVR+, extending WoVR's hallucination-aware WM-RL framework by redesigning the learned environment and imagined interaction protocol: A VLA-interface-aligned world model provides multi-view prediction and proprioceptive updates while mitigating hallucinations. Chunk-wise noise preserves flow-based policy exploration while limiting prediction-degrading action jitter. For long-horizon tasks, stage-local imagined training combines stage-entry resets, stage-specific rewards, local horizons, and controllable sampling to shorten generated prefixes and balance training across stages. WoVR+ improves success over SFT by 21.09% and 16.7% on RoboTwin and 6 real-robot tasks. Long-horizon phone-packaging success rises from 30% to 75%. Videos are available at https://youmo445.github.io/WoVR_Plus.github.io/.
Figures & tables
Fig. 1: Overview of motivation, extensions, and evaluation of WoVR+. (A) Shared core problem. World-model hallucinations can produce spurious imagined successes that fail to transfer to real execution, motivating hallucination-aware WM-RL. (B) Extended scope. WoVR+ extends WoVR’s reliability-oriented framework to complete VLA interfaces with multi-view observations and proprioceptive feedback, as well as long-horizon, multi-stage manipulation. (C) Simulator results. WoVR+ improves multi-view video-generation fidelity and enables effective policy optimization through WM-RL. (D) Real-world results. The resulting policies improve performance across six real-robot manipulation tasks and support end-to-end optimization of a 4-stage long-horizon task.
Fig. 2: WM-RL framework in WoVR, redrawn based on [ 25 ] . WoVR treats the world model as a generative simulator and controls hallucination across three levels.
Fig. 3: Overview of WoVR+. WoVR+ extends the reliability-oriented framework with a VLA-aligned multi-view world model with proprioceptive feedback, chunk-wise Flow-SDE exploration and stage-balanced reset-based local rollouts for long-horizon policy optimization.
Fig. 4: Architecture of the VLA-aligned world model. Action maps and camera ray maps are explicitly encoded as geometric conditions and injected into the backbone through a parallel DiT control branch, enabling multi-view video prediction.
Fig. 5: Construction of Action Map in Fig. 4 . Projected EEF axes encode orientation, circle color indicates gripper openness, and depth-dependent circle size supplies perspective information. Semi-transparent compositing retains overlapping bimanual control cues.
Fig. 6: Construction of latent-aligned geometric condition in Fig. 4 . The VAE-encoded action map is concatenated with temporally grouped ray origins and directions, projected into the VACE hidden space, and augmented with view embeddings. Hz,Wz denote the assembled video-latent size; the patch projection produces Hp,Wp .
Fig. 7: Action-space exploration for flow-based action-chunk policies. (A) I.I.D. SDE applies independent noise to each action token, whereas chunk-wise SDE shares one noise vector across the horizon. (B) I.I.D. noise causes high-frequency action fluctuations, while chunk-wise noise preserves per-token exploration with temporally coherent deviations from ODE.
Fig. 8: Stage-balanced imagined interaction. Stage resets directly initialize GRPO rollout groups from data-supported stage-entry states, improving late-stage coverage while avoiding long model-generated prefixes.
Method
NFE
Head View
Wrist Views
FPS ↑
LPIPS ↓
FID ↓
FVD ↓
SSIM ↑
PSNR ↑
LPIPS ↓
FID ↓
FVD ↓
SSIM ↑
PSNR ↑
Ctrl-World
5
0.318
73.22
229.83
0.812
16.39
0.350
117.45
305.39
0.863
22.38
4.16
50
0.305
47.16
147.46
0.802
15.86
0.346
73.67
186.66
0.849
22.14
0.53
EVAC
27
0.337
50.93
196.72
0.768
13.76
0.328
68.49
185.15
0.833
20.89
1.71
WoVR (1 view)
5
0.182
20.74
75.09
0.858
19.12
–
–
–
–
–
22.56
WoVR (3 views)
5
0.231
27.01
109.33
0.825
16.96
0.269
47.55
124.41
0.825
23.16
14.01
TABLE I: Video prediction quality. Wrist metrics are averaged over the two wrist cameras. Best and second-best results are shown in bold and underlined, respectively. “–” denotes an unavailable view.
Fig. 9: Qualitative comparison of multi-view prediction.
Fig. 10: Long-horizon prediction fidelity over a fixed cohort of 20 held-out trajectories.
Method
NFE
Head View
Wrist Views
LPIPS ↓
FID ↓
FVD ↓
SSIM ↑
PSNR ↑
LPIPS ↓
FID ↓
FVD ↓
SSIM ↑
PSNR ↑
Ctrl-World
5
0.264
80.42
198.17
0.847
15.99
0.311
97.43
491.56
0.913
19.57
50
0.278
51.86
191.81
0.823
14.89
0.338
58.02
229.34
0.890
18.71
EVAC
27
0.304
56.74
147.66
0.809
14.12
0.291
56.59
278.99
0.887
20.00
WoVR (1 view)
5
0.228
34.58
131.12
0.854
16.36
–
–
–
–
–
WoVR (3 views)
5
0.236
36.91
144.73
0.847
16.09
0.322
49.34
297.26
0.856
19.21
TABLE II: Prediction fidelity under policy-induced action distribution shift.
RayMap Encoding
View Emb.
NFE
Head View
Wrist Views
LPIPS ↓
FID ↓
FVD ↓
SSIM ↑
PSNR ↑
LPIPS ↓
FID ↓
FVD ↓
SSIM ↑
PSNR ↑
No RayMap
No
5
0.2335
32.67
101.07
0.8275
16.91
0.2842
61.14
130.27
0.8685
26.57
50
0.2360
27.46
81.22
0.8243
16.80
0.2875
52.06
114.71
0.8589
26.24
Direct Latent
No
5
0.1829
27.73
79.99
0.8593
20.10
0.2439
53.30
105.46
0.8780
27.81
50
0.1862
24.36
64.72
0.8553
19.82
0.2447
44.30
94.14
0.8698
27.71
Image + VAE
No
5
0.2148
33.29
103.37
0.8433
18.43
0.2634
57.55
120.34
0.8735
27.02
TABLE III: Ablation of camera raymap encoding and view embeddings.
Training Configuration
NFE
Head View
Wrist Views
LPIPS ↓
FID ↓
FVD ↓
SSIM ↑
PSNR ↑
LPIPS ↓
FID ↓
FVD ↓
SSIM ↑
PSNR ↑
Raw
5
0.1801
26.79
76.29
0.8613
20.29
0.2333
53.04
101.11
0.8875
28.29
50
0.1850
22.88
63.50
0.8549
19.96
0.2330
44.49
87.95
0.8783
28.13
+ Sparse History
5
0.1623
25.52
73.77
0.8743
21.33
0.2263
48.57
95.57
0.8898
29.05
50
0.1660
21.03
55.35
0.8689
21.04
0.2254
39.71
77.52
0.8817
28.82
+ Geometry-Aware Blur
5
0.1721
25.82
74.94
0.8681
20.92
0.2288
49.21
92.84
0.8885
28.69
TABLE IV: Ablation of world-model training strategies. Each individual variant adds the indicated strategy to the raw configuration, while All combines all the strategies.
Fig. 11: Real-robot and world-model success rates across six task–collection groups. Colors denote tasks; circles and squares distinguish different checkpoints.
Exploration
Updates
Adjust Bottle
Click Bell
Fixed
Random
Fixed
Random
Initial SFT
0
79.00
70.70
71.00
72.66
Chunk-wise
10
76.00
76.95
87.00
81.25
20
80.00
80.86
89.00
86.72
30
85.00
86.33
87.00
86.33
40
86.00
88.28
88.00
89.45
TABLE V: Success rates (%) using I.I.D. and chunk-wise Flow-SDE noise. Fixed evaluation uses 100 shared initial states.
Fig. 12: Chunk-wise and I.I.D. Flow-SDE exploration. (a) Across the evaluated noise scales, chunk-wise exploration achieves a better trade-off among EEF ensemble spread, coherent exploration magnitude, and jitter suppression. (b) DCT spectra of standardized action residuals show that chunk-wise exploration concentrates perturbations in low-frequency components and exhibits substantially less high-frequency power than I.I.D. exploration.
Fig. 13: Executed EEF swept workspace.
RL Environment
Adjust Bottle
Click Bell
Fixed
Random
Fixed
Random
Single-view VLA
None (SFT)
68.00
58.98
59.00
40.63
Direct RL (matched)
69.00
60.16
60.00
37.89
Direct RL (converged)
78.00
76.95
89.00
83.59
WoVR
69.00
63.67
77.00
68.36
TABLE VI: VLA success rates on RoboTwin (%). Fixed uses 100 fixed initial states and Random uses 256 random states.
Fig. 14: Real-robot tasks for multi-task WM-RL. Three single-arm and three bimanual tasks covering contact execution, object reorientation, precise insertion, multi-object placement, and bimanual deformable-object manipulation
Fig. 15: Real-world multi-task WM-RL. A single policy is jointly optimized across three single-arm and three bimanual tasks. Bars report success over 20 matched trials per task.
Fig. 16: Stage-reset WM-RL for long-horizon phone packaging. 4 sequential stages spanning approximately 700 control steps. All policies are evaluated from the task initial state without intermediate resets.
Fig. 17: Conditional stage success and end-to-end success on the long-horizon Phone Packaging task. For each stage, the percentage denotes the conditional probability of completing that stage among trials that successfully reach it; the fraction below reports the corresponding number of successful completions over stage arrivals. All policies are evaluated from the task initial state without intermediate resets.
Fig. 18: Stage-completion estimates under full-start and stage-reset imagination. Full-start cumulative completion rates are measured directly, whereas stage-reset cumulative estimates are computed as products of local completion rates.
Fig. 19: Effect of generated-prefix length on ACWM replay. Actions from the same recorded trajectory are replayed from either the task initial state or the beginning of Stage 4. Full-start replay exhibits object disappearance and positional drift, while the stage-entry reset avoids the long generated prefix.
Fig. 20: Informative-group generation under stage-reset and full-start training. (a) Uniform stage-reset sampling allocates equal rollout groups across stages, while informative-group counts vary. (b) Full-start training with stage-wise rewards produces more informative groups per update. (c) Shorter stage-local rollouts reduce update time, yielding 1.58× the hourly informative-group throughput of full-start training.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 21: Long-horizon proprioception prediction. Endpoint NRMSE over 75 task-internal and 75 cross-task trajectories. Predictions are recursively propagated in eight-step blocks for 200 control steps.
Fig. 22: Dual-arm ARX-5 manipulation platform.
Task
Objective
Success Criterion
Button Pressing
(Single)
Press the target button.
The target button is successfully actuated.
Bottle Righting
(Single)
Grasp and reorient a fallen bottle upright.
The bottle remains stably upright.
Test-Tube Transfer
(Single)
Transfer a test tube between racks.
The tube is inserted into the target rack.
Appendix
TABLE VII: Definitions and success criteria of the six real-robot manipulation tasks.
Stage
Operation
Completion Criterion
Stage 1
Base positioning
Drag the box base to the target position.
The base reaches the required pose for phone placement.
Stage 2
Phone placement
Grasp and place the phone into the base.
The phone is completely inside the base.
Stage 3
Lid handover
Reorient the lid and transfer it to the right gripper.
The lid is correctly oriented and held by the right gripper.
Appendix
TABLE VIII: Stage definitions and completion criteria for the long-horizon Phone Packaging task.
Hyperparameter
Configuration
Architecture and conditioning
Video backbone
Wan2.2-TI2V-5B
Temporal window
13 frames: 5 context + 8 predicted frames
VACE branch
15 blocks initialized from main-DiT blocks {0,2,…,28}
Trainable modules
Main DiT and VACE branch; VAE and UMT5 are frozen
Optimization and distributed training
Appendix
TABLE IX: Architecture and training hyperparameters of the three-view world model.
Reinforcement learning (RL) promises to unlock capabilities beyond imitation learning for Vision--Language--Action (VLA) models, but its requirement for massive real-world interaction prevents direct deployment on physical robots. Recent work attempts to use learned world models as simulators for policy optimization, yet closed-loop imagined rollouts inevitably suffer from hallucination and long-horizon error accumulation. Such errors not only degrade visual fidelity, but also mislead policy optimization by providing unreliable learning signals. We propose WoVR, a reliable world-model-based RL framework for post-training VLA policies. Instead of assuming a faithful world model, WoVR explicitly regulates how RL interacts with imperfect imagined dynamics. It improves rollout stability through a controllable action-conditioned video world model, reshapes imagined interaction to reduce effective error depth via Keyframe-Initialized Rollouts, and maintains policy--simulator alignment through World Model-Policy co-evolution. Extensive experiments demonstrate that WoVR enables stable long-horizon imagined rollouts and effective policy optimization, achieving superior LIBERO performance and consistent real-world gains across multiple robotic platforms. These results show that world models can serve as practical simulators for RL when hallucination is explicitly controlled. Additional visualization results are available at https://wovr-corl.github.io.
Zhennan Jiang, Shangqing Zhou, Yutong Jiang +11
University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Zhongguancun Academy +2
Generalist robot policies have demonstrated strong generalization across robotic manipulation tasks, yet their success rates remain limited in com- plex long-horizon scenarios. Recent methods improve Visual-Language-Action (VLA) policies through online reinforcement learning on real robots, but such training relies on costly physical interactions, suffers from low sample efficiency, and may introduce hardware and safety risks. World models offer a promising alternative by enabling policy optimization with imagined rollouts. However, long-horizon rollouts generated by world models often suffer from prediction hal- lucinations, producing biased state transitions that can mislead policy learning. To address this issue, we propose Hallucination-aware World Model-based Pol- icy Optimization (HaWMPO), a closed-loop reinforcement learning pipeline for VLA policy post-training with world models. Specifically, HaWMPO introduces an action-conditioned hallucination-aware model to estimate the reliability of gen- erated image sequences, and incorporates hallucination scores into group relative policy optimization through a Reward-Soft mechanism, suppressing unreliable ac- tion chunks during training. On the LIBERO benchmark, HaWMPO achieves the best average success rate, with gains of 15.0% over the base model and 2.8% over the strongest baseline; real-world experiments on a G1 robot further validate its effectiveness, raising the average success rate on two manipulation tasks from 67.5% to 80.0%.
Zengjue Chen, Peidong Liu, Jiawei Li +1
Joy Future Academy, JD · School of Artificial Intelligence, Jilin University
Vision-Language-Action (VLA) models show strong generalization for robotic control, but finetuning them with reinforcement learning (RL) is constrained by the high cost and safety risks of real-world interaction. Training VLA models in interactive world models avoids these issues but introduces several challenges, including pixel-level world modeling, multi-view consistency, and compounding errors under sparse rewards. Building on recent advances across large multimodal models and model-based RL, we propose VLA-MBPO, a practical framework to tackle these problems in VLA finetuning. Our approach has three key design choices: (i) adapting unified multimodal models (UMMs) for data-efficient world modeling; (ii) an interleaved view decoding mechanism to enforce multi-view consistency; and (iii) chunk-level branched rollout to mitigate error compounding. Theoretical analysis and experiments across simulation and real-world tasks demonstrate that VLA-MBPO significantly improves policy performance and sample efficiency, underscoring its robustness and scalability for real-world robotic deployment.
Zhilong Zhang, Haoxiang Ren, Yihao Sun +6
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · Mila - Quebec AI Institute +1