WoVR+: World Model as Reliable Simulator for Versatile VLA Policy Optimization with RL

Shangqing Zhou; Zhennan Jiang; Ning Ma; Xin Fu; Yupeng Zheng; Shuai Tian; Huangrui Li; Jiahao Liu

cs.ROOct 13, 2026

WoVR+: World Model as Reliable Simulator for Versatile VLA Policy Optimization with RL

Authors: Shangqing Zhou, Zhennan Jiang, Ning Ma, Xin Fu, Yupeng Zheng, Shuai Tian, Huangrui Li, Jiahao Liu,

Organizations: Institute of Automation, Chinese Academy of Sciences. · School of Artificial Intelligence, University of Chinese Academy of Sciences. · Zhongguancun Academy. · Xi’an Jiaotong-Liverpool University. ·

Abstract

Action-conditioned world models can reduce costly real-robot interaction in reinforcement learning for Vision-Language-Action models (VLA-RL), but hallucinations can mislead policy optimization. Extending them to richer VLA interfaces and long-horizon tasks raises challenges in spatiotemporal and cross-modal consistency, exploration-reliability balance in RL, and compounding rollout errors. We present WoVR+, extending WoVR's hallucination-aware WM-RL framework by redesigning the learned environment and imagined interaction protocol: A VLA-interface-aligned world model provides multi-view prediction and proprioceptive updates while mitigating hallucinations. Chunk-wise noise preserves flow-based policy exploration while limiting prediction-degrading action jitter. For long-horizon tasks, stage-local imagined training combines stage-entry resets, stage-specific rewards, local horizons, and controllable sampling to shorten generated prefixes and balance training across stages. WoVR+ improves success over SFT by 21.09% and 16.7% on RoboTwin and 6 real-robot tasks. Long-horizon phone-packaging success rises from 30% to 75%. Videos are available at https://youmo445.github.io/WoVR_Plus.github.io/.

Figures & tables

Fig. 1: Overview of motivation, extensions, and evaluation of WoVR+. (A) Shared core problem. World-model hallucinations can produce spurious imagined successes that fail to transfer to real execution, motivating hallucination-aware WM-RL. (B) Extended scope. WoVR+ extends WoVR’s reliability-oriented framework to complete VLA interfaces with multi-view observations and proprioceptive feedback, as well as long-horizon, multi-stage manipulation. (C) Simulator results. WoVR+ improves multi-view video-generation fidelity and enables effective policy optimization through WM-RL. (D) Real-world results. The resulting policies improve performance across six real-robot manipulation tasks and support end-to-end optimization of a 4-stage long-horizon task.

Fig. 2: WM-RL framework in WoVR, redrawn based on [ 25 ] . WoVR treats the world model as a generative simulator and controls hallucination across three levels.

Fig. 3: Overview of WoVR+. WoVR+ extends the reliability-oriented framework with a VLA-aligned multi-view world model with proprioceptive feedback, chunk-wise Flow-SDE exploration and stage-balanced reset-based local rollouts for long-horizon policy optimization.

Fig. 4: Architecture of the VLA-aligned world model. Action maps and camera ray maps are explicitly encoded as geometric conditions and injected into the backbone through a parallel DiT control branch, enabling multi-view video prediction.

Fig. 5: Construction of Action Map in Fig. 4 . Projected EEF axes encode orientation, circle color indicates gripper openness, and depth-dependent circle size supplies perspective information. Semi-transparent compositing retains overlapping bimanual control cues.

Fig. 6: Construction of latent-aligned geometric condition in Fig. 4 . The VAE-encoded action map is concatenated with temporally grouped ray origins and directions, projected into the VACE hidden space, and augmented with view embeddings. $H_{z},W_{z}$ denote the assembled video-latent size; the patch projection produces $H_{p},W_{p}$ .

Fig. 7: Action-space exploration for flow-based action-chunk policies. (A) I.I.D. SDE applies independent noise to each action token, whereas chunk-wise SDE shares one noise vector across the horizon. (B) I.I.D. noise causes high-frequency action fluctuations, while chunk-wise noise preserves per-token exploration with temporally coherent deviations from ODE.

Fig. 8: Stage-balanced imagined interaction. Stage resets directly initialize GRPO rollout groups from data-supported stage-entry states, improving late-stage coverage while avoiding long model-generated prefixes.

Method	NFE	Head View					Wrist Views					FPS $\uparrow$
Method	NFE	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	FPS $\uparrow$
Ctrl-World	5	0.318	73.22	229.83	0.812	16.39	0.350	117.45	305.39	0.863	22.38	4.16
	50	0.305	47.16	147.46	0.802	15.86	0.346	73.67	186.66	0.849	22.14	0.53
EVAC	27	0.337	50.93	196.72	0.768	13.76	0.328	68.49	185.15	0.833	20.89	1.71
WoVR (1 view)	5	0.182	20.74	75.09	0.858	19.12	–	–	–	–	–	22.56
WoVR (3 views)	5	0.231	27.01	109.33	0.825	16.96	0.269	47.55	124.41	0.825	23.16	14.01

TABLE I: Video prediction quality. Wrist metrics are averaged over the two wrist cameras. Best and second-best results are shown in bold and underlined, respectively. “–” denotes an unavailable view.

Fig. 9: Qualitative comparison of multi-view prediction.

Panel of Figure 11 — Fig. 10: Long-horizon prediction fidelity over a fixed cohort of 20 held-out trajectories.

Method	NFE	Head View					Wrist Views
Method	NFE	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$
Ctrl-World	5	0.264	80.42	198.17	0.847	15.99	0.311	97.43	491.56	0.913	19.57
	50	0.278	51.86	191.81	0.823	14.89	0.338	58.02	229.34	0.890	18.71
EVAC	27	0.304	56.74	147.66	0.809	14.12	0.291	56.59	278.99	0.887	20.00
WoVR (1 view)	5	0.228	34.58	131.12	0.854	16.36	–	–	–	–	–
WoVR (3 views)	5	0.236	36.91	144.73	0.847	16.09	0.322	49.34	297.26	0.856	19.21

TABLE II: Prediction fidelity under policy-induced action distribution shift.

RayMap Encoding	View Emb.	NFE	Head View					Wrist Views
RayMap Encoding	View Emb.	NFE	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$
No RayMap	No	5	0.2335	32.67	101.07	0.8275	16.91	0.2842	61.14	130.27	0.8685	26.57
No RayMap	No	50	0.2360	27.46	81.22	0.8243	16.80	0.2875	52.06	114.71	0.8589	26.24
Direct Latent	No	5	0.1829	27.73	79.99	0.8593	20.10	0.2439	53.30	105.46	0.8780	27.81
Direct Latent	No	50	0.1862	24.36	64.72	0.8553	19.82	0.2447	44.30	94.14	0.8698	27.71
Image + VAE	No	5	0.2148	33.29	103.37	0.8433	18.43	0.2634	57.55	120.34	0.8735	27.02

TABLE III: Ablation of camera raymap encoding and view embeddings.

Training Configuration	NFE	Head View					Wrist Views
Training Configuration	NFE	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$
Raw	5	0.1801	26.79	76.29	0.8613	20.29	0.2333	53.04	101.11	0.8875	28.29
Raw	50	0.1850	22.88	63.50	0.8549	19.96	0.2330	44.49	87.95	0.8783	28.13
+ Sparse History	5	0.1623	25.52	73.77	0.8743	21.33	0.2263	48.57	95.57	0.8898	29.05
+ Sparse History	50	0.1660	21.03	55.35	0.8689	21.04	0.2254	39.71	77.52	0.8817	28.82
+ Geometry-Aware Blur	5	0.1721	25.82	74.94	0.8681	20.92	0.2288	49.21	92.84	0.8885	28.69

TABLE IV: Ablation of world-model training strategies. Each individual variant adds the indicated strategy to the raw configuration, while All combines all the strategies.

Fig. 11: Real-robot and world-model success rates across six task–collection groups. Colors denote tasks; circles and squares distinguish different checkpoints.

Exploration	Updates	Adjust Bottle		Click Bell
Exploration	Updates	Fixed	Random	Fixed	Random
Initial SFT	0	79.00	70.70	71.00	72.66
Chunk-wise	10	76.00	76.95	87.00	81.25
	20	80.00	80.86	89.00	86.72
	30	85.00	86.33	87.00	86.33
	40	86.00	88.28	88.00	89.45

TABLE V: Success rates (%) using I.I.D. and chunk-wise Flow-SDE noise. Fixed evaluation uses 100 shared initial states.

Fig. 12: Chunk-wise and I.I.D. Flow-SDE exploration. (a) Across the evaluated noise scales, chunk-wise exploration achieves a better trade-off among EEF ensemble spread, coherent exploration magnitude, and jitter suppression. (b) DCT spectra of standardized action residuals show that chunk-wise exploration concentrates perturbations in low-frequency components and exhibits substantially less high-frequency power than I.I.D. exploration.

Single-view VLA
RL Environment	Adjust Bottle		Click Bell
RL Environment	Fixed	Random	Fixed	Random
None (SFT)	68.00	58.98	59.00	40.63
Direct RL (matched)	69.00	60.16	60.00	37.89
Direct RL (converged)	78.00	76.95	89.00	83.59
WoVR	69.00	63.67	77.00	68.36

TABLE VI: VLA success rates on RoboTwin (%). Fixed uses 100 fixed initial states and Random uses 256 random states.

Fig. 14: Real-robot tasks for multi-task WM-RL. Three single-arm and three bimanual tasks covering contact execution, object reorientation, precise insertion, multi-object placement, and bimanual deformable-object manipulation

Fig. 15: Real-world multi-task WM-RL. A single policy is jointly optimized across three single-arm and three bimanual tasks. Bars report success over 20 matched trials per task.

Fig. 16: Stage-reset WM-RL for long-horizon phone packaging. 4 sequential stages spanning approximately 700 control steps. All policies are evaluated from the task initial state without intermediate resets.

Fig. 17: Conditional stage success and end-to-end success on the long-horizon Phone Packaging task. For each stage, the percentage denotes the conditional probability of completing that stage among trials that successfully reach it; the fraction below reports the corresponding number of successful completions over stage arrivals. All policies are evaluated from the task initial state without intermediate resets.

Fig. 18: Stage-completion estimates under full-start and stage-reset imagination. Full-start cumulative completion rates are measured directly, whereas stage-reset cumulative estimates are computed as products of local completion rates.

Fig. 19: Effect of generated-prefix length on ACWM replay. Actions from the same recorded trajectory are replayed from either the task initial state or the beginning of Stage 4. Full-start replay exhibits object disappearance and positional drift, while the stage-entry reset avoids the long generated prefix.

Fig. 20: Informative-group generation under stage-reset and full-start training. (a) Uniform stage-reset sampling allocates equal rollout groups across stages, while informative-group counts vary. (b) Full-start training with stage-wise rewards produces more informative groups per update. (c) Shorter stage-local rollouts reduce update time, yielding

1.58\times

the hourly informative-group throughput of full-start training.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 21: Long-horizon proprioception prediction. Endpoint NRMSE over 75 task-internal and 75 cross-task trajectories. Predictions are recursively propagated in eight-step blocks for 200 control steps.

Fig. 22: Dual-arm ARX-5 manipulation platform.

Task	Objective	Success Criterion
Button Pressing
(Single)	Press the target button.	The target button is successfully actuated.
Bottle Righting
(Single)	Grasp and reorient a fallen bottle upright.	The bottle remains stably upright.
Test-Tube Transfer
(Single)	Transfer a test tube between racks.	The tube is inserted into the target rack.

Appendix

TABLE VII: Definitions and success criteria of the six real-robot manipulation tasks.

Stage	Operation	Completion Criterion
Stage 1
Base positioning	Drag the box base to the target position.	The base reaches the required pose for phone placement.
Stage 2
Phone placement	Grasp and place the phone into the base.	The phone is completely inside the base.
Stage 3
Lid handover	Reorient the lid and transfer it to the right gripper.	The lid is correctly oriented and held by the right gripper.

Appendix

TABLE VIII: Stage definitions and completion criteria for the long-horizon Phone Packaging task.

Hyperparameter	Configuration
Architecture and conditioning
Video backbone	Wan2.2-TI2V-5B
Temporal window	13 frames: 5 context + 8 predicted frames
VACE branch	15 blocks initialized from main-DiT blocks $\{0,2,\ldots,28\}$
Trainable modules	Main DiT and VACE branch; VAE and UMT5 are frozen
Optimization and distributed training

Appendix

TABLE IX: Architecture and training hyperparameters of the three-view world model.

Configuration	Setting
Shared policy and exploration
Policy observation	3 RGB views, 14-D proprioception, and instruction
Policy action	14-D absolute joint-and-gripper action; horizon $H=16$
Policy sampling	5 flow steps; chunk-wise Flow-SDE noise at 1 sampled step
Noise scale	$0.2$
Trainable module	Action expert only; no value head

Appendix

TABLE X: Main configurations for the real-robot WM-RL experiments.

Robotic RL

Robotic reinforcement learning uses trial-and-error learning to help robots control, navigate, and manipulate objects.

See all 300 papers

WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL
Feb 15, 2026Zhennan Jiang, Shangqing Zhou, Yutong Jiang +11World Models for Robotics World Models
University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Zhongguancun Academy +2
HaWMPO: Hallucination-Aware World Model-based Policy Optimization for Generalist Robot Policy
Sep 9, 2026Zengjue Chen, Peidong Liu, Jiawei Li +1Robot Policy Learning World Models
Joy Future Academy, JD · School of Artificial Intelligence, Jilin University
Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models
Mar 21, 2026Zhilong Zhang, Haoxiang Ren, Yihao Sun +6RL for VLMs Vision-Language-Action Models
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · Mila - Quebec AI Institute +1

Feb 15, 2026J/K move · Enter open · S save

WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL

Zhennan Jiang, Shangqing Zhou, Yutong Jiang, Zefang Huang+10

University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Zhongguancun Academy+2

World Models for Robotics World Models Action-Conditioned World Models Vision-Language-Action Models RL for VLMs Model-Based RL+1

Open paper AskPDF arXiv

Reinforcement learning (RL) promises to unlock capabilities beyond imitation learning for Vision--Language--Action (VLA) models, but its requirement for massive real-world interaction prevents direct deployment on physical robots. Recent work attempts to use learned world models as simulators for policy optimization, yet closed-loop imagined rollouts inevitably suffer from hallucination and long-horizon error accumulation. Such errors not only degrade visual fidelity, but also mislead policy optimization by providing unreliable learning signals. We propose WoVR, a reliable world-model-based RL framework for post-training VLA policies. Instead of assuming a faithful world model, WoVR explicitly regulates how RL interacts with imperfect imagined dynamics. It improves rollout stability through a controllable action-conditioned video world model, reshapes imagined interaction to reduce effective error depth via Keyframe-Initialized Rollouts, and maintains policy--simulator alignment through World Model-Policy co-evolution. Extensive experiments demonstrate that WoVR enables stable long-horizon imagined rollouts and effective policy optimization, achieving superior LIBERO performance and consistent real-world gains across multiple robotic platforms. These results show that world models can serve as practical simulators for RL when hallucination is explicitly controlled. Additional visualization results are available at https://wovr-corl.github.io.

Abstract

Robotic RL

Explore similar work

WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL

HaWMPO: Hallucination-Aware World Model-based Policy Optimization for Generalist Robot Policy

Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models