DRILL: Training World Models to Improve Policies, Not Predict Pixels

Zhaolu Kang; Tailong Luo; Lei Wei; Guangyuan Dong; Sihan Liu; Yingjie He; Siheng Wang; Shuaibo Li

cs.CVOct 22, 2026

DRILL: Training World Models to Improve Policies, Not Predict Pixels

Authors: Zhaolu Kang, Tailong Luo, Lei Wei, Guangyuan Dong, Sihan Liu, Yingjie He, Siheng Wang, Shuaibo Li,

Organizations: Hunyuan, Tencent · Peking University

Abstract

World models are increasingly used to reduce environment interaction when post-training vision-language-action (VLA) policies with reinforcement learning. Yet they are typically trained to predict pixels one step ahead, even though their role here is not merely to render visually faithful videos but to produce learning signals from imagined rollouts that improve the policy. We show that this mismatch is substantial: lower pixel MSE can produce worse policy gradients, and a world model trained for policy utility can yield better downstream policies despite 17% higher pixel MSE on the next frame. We propose the Downstream-Return Imagination Learning Loop (DRILL), a bilevel framework that updates the world model to maximize the policy's return after an inner GRPO update on imagined rollouts. DRILL has two instantiations. DRILL-VWI is a closed-form surrogate, requiring no additional simulator rollouts, that reweights prediction errors by policy visitation, the group-relative GRPO advantage, and model confidence. DRILL-IMG is a full meta-gradient method that differentiates through the inner update using truncated Hessian-vector products. We show that DRILL-VWI recovers the local one-step component of the DRILL-IMG meta-gradient under score-compatibility and small-residual assumptions. Across five manipulation simulators and two VLA backbones, DRILL-IMG raises mean success rate by 13.1 points over WMPO and 7.9 points over RLVR-World, and complements RLVR pretraining rather than competing with it. More broadly, we believe video world models for VLA reinforcement learning can be productively trained by the policy gradients they produce, not only by pixel-level prediction.

Figures & tables

Figure 1: Overview of DRILL. The policy first learns inside the current world model through imagined GRPO. The resulting policy-improvement signal is then fed back to the world model: DRILL-VWI uses the same imagined rollouts to form a signed, visitation-weighted local surrogate, whereas DRILL-IMG differentiates through the inner update and grounds the outer gradient with a small simulator batch.

Method	LIBERO	CALVIN	RoboCasa	MT50	Simpler	MSE $\downarrow$	ESNR $\uparrow$	$\mathrm{PUR}$ $\downarrow$
Scratch-GRPO	$18.2\!\pm\!2.7$	$22.5\!\pm\!3.1$	$\phantom{0}8.1\!\pm\!1.6$	$21.4\!\pm\!2.4$	$15.0\!\pm\!2.2$	N/A	N/A	N/A
Sim-PPO	$28.5\!\pm\!2.1$	$32.1\!\pm\!2.5$	$12.4\!\pm\!1.9$	$31.6\!\pm\!2.0$	$22.8\!\pm\!1.8$	N/A	N/A	N/A
WMPO	$64.2\!\pm\!1.6$	$71.8\!\pm\!1.3$	$38.5\!\pm\!2.1$	$67.4\!\pm\!1.5$	$55.1\!\pm\!2.0$	$8.4\!\pm\!0.3$	$1.00$	$1.42\!\pm\!0.08$
VLAW	$65.1\!\pm\!2.4$	$74.0\!\pm\!1.8$	$39.6\!\pm\!2.2$	$68.2\!\pm\!2.1$	$57.9\!\pm\!1.7$	$8.7\!\pm\!0.4$	$1.09$	$1.37\!\pm\!0.09$
RLVR-World †	$70.8\!\pm\!1.4$	$75.3\!\pm\!1.6$	$42.7\!\pm\!1.8$	$72.9\!\pm\!1.5$	$61.6\!\pm\!1.9$	$\mathbf{7.2\!\pm\!0.3}$	$1.39$	$0.97\!\pm\!0.06$
DRILL-VWI †	$76.1\!\pm\!1.2$	$80.7\!\pm\!1.1$	$51.8\!\pm\!1.7$	$75.6\!\pm\!1.4$	$\mathbf{64.0\!\pm\!1.9}$	$9.3\!\pm\!0.3$	$1.68$	$0.73\!\pm\!0.05$

Table 1: Main results on

\pi_{0}

-3B. Values are mean

\pm

std over five seeds. Bold marks column best; † denotes

p\!<\!0.05

versus WMPO. MSE is in

10^{-3}

and ESNR is relative to WMPO; RLVR-WM denotes the RLVR-World WM.

Panel of Figure 2 — Figure 2: MSE–success decoupling on LIBERO-Long ( $\pi_{0}$ -3B). Faint markers are seeds; bold markers are mean $\pm$ std.

Panel of Figure 3 — Figure 3: Three compact diagnostics on LIBERO-Long. (a) DRILL variants pass baselines early and continue improving. (b) Larger rollout horizon widens the DRILL-IMG–DRILL-VWI gap until imagined trajectories become noisy. (c) Lower $\mathrm{PUR}$ tracks higher policy utility during training.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
Observations, actions, language
$o_{t}\in\mathcal{O}$	Observation (image) at time $t$ .
$o_{\leq t}$	Observation history $(o_{0},\dots,o_{t})$ .
$a_{t}\in\mathcal{A}$	Action at time $t$ .
$a_{t:t+H_{a}}$	$H_{a}$ -step action chunk produced by $\pi_{\theta}$ .
$\ell$	Natural-language instruction.

Appendix

Table 2: Notation used throughout the paper.

Panel of Figure 4 — Figure 4: Exact-vs-truncated meta-gradient cosine similarity on low-dimensional environments where the exact meta-gradient is analytically tractable. Error bars show $\pm 1$ std over 10 seeds $\times$ 1000 states. $K\!=\!2$ already exceeds the $0.7$ threshold on both environments (on MT10 just barely at $0.71\!\pm\!0.05$ , the lower end of which falls slightly below threshold); $K\!\geq\!3$ improves cosine by $<\!0.07$ at linearly more wall-clock cost, hence the choice of $K\!=\!2$ at scale.

Env	VLA	WM	$\eta_{\theta}$	$\eta_{\phi}$	batch	$H$	$H_{a}$	$K$	$\alpha$	seeds
LIBERO-Long	$\pi_{0}$ -3B	1.2B Ctrl-World	$1\!\times\!10^{-5}$	$5\!\times\!10^{-6}$	512	50	16	2	1	5
CALVIN ABC $\to$ D	$\pi_{0}$ -3B	1.2B Ctrl-World	$1\!\times\!10^{-5}$	$5\!\times\!10^{-6}$	512	50	16	2	1	5
RoboCasa	$\pi_{0}$ -3B	1.2B Ctrl-World	$8\!\times\!10^{-6}$	$4\!\times\!10^{-6}$	384	50	16	2	1	5
Meta-World MT50	$\pi_{0}$ -3B	1.2B Ctrl-World	$1\!\times\!10^{-5}$	$5\!\times\!10^{-6}$	512	50	16	2	1	5
SimplerEnv	$\pi_{0}$ -3B	1.2B Ctrl-World	$1\!\times\!10^{-5}$	$5\!\times\!10^{-6}$	512	50	16	2	1	5
LIBERO-Long	OpenVLA-7B	1.2B Ctrl-World	$5\!\times\!10^{-6}$	$2\!\times\!10^{-6}$	256	50	16	2	1	5

Appendix

Table 3: Training hyperparameters. Values shared across all rows: group size

G\!=\!8

for GRPO; outer-loop optimizer AdamW with

\beta_{1}\!=\!0.9

\beta_{2}\!=\!0.95

, weight decay

10^{-4}

; inner-loop optimizer plain SGD with momentum

0.9

; gradient clipping at

\lVert\hat{g}_{\phi}\rVert\leq 10

for outer and

\lVert\nabla_{\theta}\hat{J}\rVert\leq 1

for inner; mixed-precision bfloat16 forward, float32 HVP accumulation; 5 seeds for headline rows, 3 seeds for ablations. Effective batch is the per-GPU batch times 8 H20 GPUs (data-parallel across the 24-GPU pool, with a 16-way fragment used for the WM forward and an 8-way fragment used for evaluation in parallel).

Phase	GPUs	H20-hours	% total	shared with baselines
WM pretrain	24	7,200	42%	yes (WMPO/RLVR-World)
Inner loop (GRPO + imagined)	16	4,800	28%	yes (WMPO/RLVR-World)
Outer loop (sim rollouts + HVP)	16	3,840	23%	no (DRILL-IMG-only)
Evaluation (reported tasks)	8	1,200	7%	yes (all methods)
Total	–	17,040	100%	–

Appendix

Table 4: Compute budget breakdown for the reported DRILL-IMG runs (five

\pi_{0}

-3B benchmarks plus three OpenVLA-7B replication benchmarks, each with five seeds), measured on a 24-GPU NVIDIA H20 cluster over

\sim

4 weeks of wall-clock. The marginal cost of running DRILL-IMG on top of an existing WMPO pipeline is the “Outer loop” row alone (

\sim

3.8k H20-h), since the WM-pretrain and inner-loop phases are shared with the WMPO/RLVR-World baselines.

Method	Outer steps	Mean SR
WMPO	10k	$64.2\!\pm\!1.6$
WMPO, 1.4 $\times$ compute	14k	$65.1\!\pm\!1.9$
WMPO, 2.0 $\times$ compute	20k	$64.4\!\pm\!2.2$
DRILL-IMG	10k	$\mathbf{81.3\!\pm\!1.3}$

Appendix

Table 5: Compute-matched control on LIBERO-Long. Extending WMPO outer updates yields marginal, non-monotone changes, confirming that DRILL’s gain is not merely compute-driven.

Coaching signal
Variant	Mean SR	MSE $\downarrow$	$\mathrm{PUR}$ $\downarrow$
WMPO	$58.2\!\pm\!1.7$	$8.4\!\pm\!0.3$	$1.42\!\pm\!0.08$
Visitation only	$64.2\!\pm\!1.5$	$8.6\!\pm\!0.3$	$1.18\!\pm\!0.07$
Advantage only	$66.8\!\pm\!1.4$	$8.9\!\pm\!0.4$	$1.06\!\pm\!0.06$
Stability only	$60.1\!\pm\!1.6$	$8.5\!\pm\!0.3$	$1.31\!\pm\!0.08$
Advantage exponent

Appendix

Table 6: Objective ablations on LIBERO, CALVIN, and RoboCasa. Values are mean

\pm

std over three seeds; MSE is in

10^{-3}

$H$	WMPO	VWI	IMG	IMG $-$ WMPO	IMG $-$ VWI
20	$67.4\!\pm\!1.5$	$72.6\!\pm\!1.3$	$74.4\!\pm\!1.5$	$7.0$	$1.8$
50	$64.2\!\pm\!1.6$	$76.1\!\pm\!1.2$	$\mathbf{81.3\!\pm\!1.3}$	$17.1$	$5.2$
100	$58.0\!\pm\!2.2$	$75.9\!\pm\!1.5$	$83.4\!\pm\!1.7$	$25.4$	$7.5$
200	$49.8\!\pm\!2.8$	$70.5\!\pm\!2.1$	$78.6\!\pm\!2.5$	$28.8$	$8.1$

Appendix

Table 7: Horizon sweep on LIBERO-Long. VWI and IMG denote DRILL-VWI and DRILL-IMG; five seeds, with

H{=}50

as the default.

Panel of Figure 5 — Figure 5: Imagined rollout quality on LIBERO-Long. WMPO has lower pixel MSE, while DRILL-IMG obtains higher task-relevant progress prediction (TRPS). Data are presented as a phase-space trajectory where time progresses along the curves.

Const.	Estimator	Range	Used
$L$	finite-difference Lipschitz slope of $\nabla_{\phi}J$ in 64 random directions	$0.13$ – $0.41$	$0.42$
$\sigma_{g}$	std. of the outer gradient estimator across simulator-rollout minibatches	$0.10$ – $0.21$	$0.22$
$B$	intercept of the fit $\lVert g_{K}-g_{\mathrm{exact}}\rVert\leq B\rho^{K}$ on gridworld/MT10	$0.47$ – $0.78$	$0.82$
$\rho$	spectral radius from power iteration on the inner-loop Jacobian	$0.64$ – $0.81$	$0.82$

Appendix

Table 8: Empirical ranges for the constants used in Theorem 1 . Ranges are over the five

\pi_{0}

-3B headline benchmarks, 5 seeds each. The final column is the conservative value used when evaluating the bound.

Method	LIBERO-L	CALVIN	RoboCasa	MSE $\downarrow$	$\mathrm{PUR}$ $\downarrow$
Zero-shot	$41.2\!\pm\!1.8$	$48.5\!\pm\!1.6$	$19.6\!\pm\!1.9$	N/A	N/A
WMPO	$59.7\!\pm\!1.5$	$66.9\!\pm\!1.4$	$34.7\!\pm\!2.0$	$9.8\!\pm\!0.4$	$1.35\!\pm\!0.08$
RLVR-World	$64.9\!\pm\!1.3$	$70.6\!\pm\!1.5$	$38.8\!\pm\!1.9$	$\mathbf{8.5\!\pm\!0.3}$	$0.94\!\pm\!0.06$
DRILL-VWI	$72.1\!\pm\!1.4$	$75.8\!\pm\!1.5$	$\mathbf{48.7\!\pm\!2.1}$	$10.4\!\pm\!0.4$	$0.77\!\pm\!0.06$
DRILL-IMG	$\mathbf{77.1\!\pm\!1.5}$	$\mathbf{78.9\!\pm\!1.6}$	$\mathbf{50.8\!\pm\!2.3}$	$11.1\!\pm\!0.5$	$\mathbf{0.43\!\pm\!0.05}$

Appendix

Table 9: Cross-backbone replication on OpenVLA-7B (mean

\pm

std, 5 seeds). Method rankings match Table 1 , except on RoboCasa where DRILL-IMG and DRILL-VWI tie statistically (

p\!=\!0.12

). MSE is scaled by

10^{-3}

Reward model	WMPO	VWI	IMG	IMG $-$ WMPO
Qwen3-VL-4B (default)	$64.2\!\pm\!1.6$	$76.1\!\pm\!1.2$	$\mathbf{81.3\!\pm\!1.3}$	$17.1$
LLaVA-CoT-7B	$63.7\!\pm\!2.0$	$77.8\!\pm\!1.6$	$82.1\!\pm\!1.5$	$18.4$
InternVL-reward-4B	$65.3\!\pm\!1.7$	$75.4\!\pm\!1.4$	$80.9\!\pm\!1.8$	$15.6$

Appendix

Table 10: Reward-model swap on LIBERO-Long (mean SR

\pm

std, 5 seeds). VWI and IMG denote DRILL-VWI and DRILL-IMG. The IMG gain over WMPO stays in the

15.6

–

18.4

pt range, but is not constant across reward models.

Method	Novel	Distr.	Light/tex.	Pose	Avg
WMPO	$37.4\!\pm\!2.1$	$25.3\!\pm\!1.8$	$30.4\!\pm\!2.0$	$27.6\!\pm\!1.9$	$30.2$
RLVR-World	$31.8\!\pm\!1.9$	$20.4\!\pm\!1.7$	$26.0\!\pm\!1.8$	$24.1\!\pm\!1.9$	$25.6$
DRILL-VWI	$26.4\!\pm\!1.8$	$17.1\!\pm\!1.5$	$23.5\!\pm\!1.9$	$\mathbf{18.0\!\pm\!1.6}$	$21.3$
DRILL-IMG	$\mathbf{22.5\!\pm\!1.7}$	$\mathbf{15.3\!\pm\!1.5}$	$\mathbf{17.6\!\pm\!1.8}$	$18.9\!\pm\!1.7$	$\mathbf{18.6}$
DRILL-IMG (OpenVLA)	$23.9\!\pm\!1.9$	$16.4\!\pm\!1.6$	$19.2\!\pm\!2.0$	$20.0\!\pm\!1.7$	$19.9$

Appendix

Table 11: Zero-shot out-of-distribution relative SR drop on LIBERO (%,

\downarrow

). DRILL-IMG exhibits the highest robustness in three of four conditions, while DRILL-VWI marginally leads under pose perturbation.

Comparison	DRILL (%)	Tie (%)	Base. (%)	$p$
WMPO quality	$33$	$26$	$41$	$0.29$
RLVR-World quality	$30$	$28$	$42$	$0.17$
WMPO task accepted	$\mathbf{62}$	$23$	$15$	$<\!0.001$

Appendix

Table 12: Forced-choice human evaluation of imagined rollouts (

N\!=\!120

trials/row, 6 raters). Raters show no significant preference in visual quality (rows 1-2), but strongly prefer DRILL-IMG when task completion is revealed (row 3), validating the MSE-success trade-off.

World Models for VLA Models

See all 42 papers

Feb 15, 2026

WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL

Reinforcement learning (RL) promises to unlock capabilities beyond imitation learning for Vision--Language--Action (VLA) models, but its requirement for massive real-world interaction prevents direct deployment on physical robots. Recent work attempts to use learned world models as simulators for policy optimization, yet closed-loop imagined rollouts inevitably suffer from hallucination and long-horizon error accumulation. Such errors not only degrade visual fidelity, but also mislead policy optimization by providing unreliable learning signals. We propose WoVR, a reliable world-model-based RL framework for post-training VLA policies. Instead of assuming a faithful world model, WoVR explicitly regulates how RL interacts with imperfect imagined dynamics. It improves rollout stability through a controllable action-conditioned video world model, reshapes imagined interaction to reduce effective error depth via Keyframe-Initialized Rollouts, and maintains policy--simulator alignment through World Model-Policy co-evolution. Extensive experiments demonstrate that WoVR enables stable long-horizon imagined rollouts and effective policy optimization, achieving superior LIBERO performance and consistent real-world gains across multiple robotic platforms. These results show that world models can serve as practical simulators for RL when hallucination is explicitly controlled. Additional visualization results are available at https://wovr-corl.github.io.

Zhennan Jiang, Shangqing Zhou, Yutong Jiang +11

University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Zhongguancun Academy +2

World Models for Robotics World Models Action-Conditioned World Models Vision-Language-Action Models RL for VLMs Model-Based RL+1

Ask PDF

Jun 4, 2026

PiL-World: A Chunk-Wise World Model for VLA Policy-in-the-Loop Evaluation

Vision-language-action (VLA) policies operate in a closed loop in real-world robot tasks: a robot observes the scene, executes an action chunk, and conditions its next decision on the resulting observation. However, most existing world models for robot action evaluation are limited to open-loop prediction along pre-collected action trajectories. This prevents them from supporting closed-loop VLA evaluation, where each action chunk must be conditioned on the observation generated by the previous execution. To address this gap, we propose PiL-World, a chunk-wise world model designed for policy-in-the-loop VLA evaluation. Given the current observation and the action trajectory rolled out by a VLA policy, PiL-World generates multi-view future observations that are consistent with the VLA rollout and match the image inputs required by the policy. By alternating between VLA inference and world-model prediction, PiL-World enables closed-loop evaluation without real robot execution at every step. To improve rollout fidelity, PiL-World conditions video generation on action-derived visual control from head-view robot motion and latent histories that encode task execution context, while jointly predicting complementary multi-view observations. Beyond successful teleoperated demonstrations, it also learns from failed execution trajectories, helping the imagined rollouts better match the distribution of real policy executions. We evaluate PiL-World on three real dual-arm manipulation tasks. PiL-World generates imagined rollouts that are highly consistent with real robot executions. More importantly, compared with the baseline, it reduces the error between VLA success rates measured in real-world rollouts and those estimated through closed-loop world-model evaluation from 63.2% to 12.0%.

Chong Ma, Taiyi Su, Jian Zhu +4

1Tongji University · 2AIRC, Midea Group

Action-Conditioned World Models Robotic Policy Evaluation Policy Evaluation Vision-Language-Action Models World Model Evaluation Embodied World Models+1

Ask PDF

Mar 11, 2026

World2Act: Latent Action Post-Training from World Model Dynamics

World Models (WMs) offer a promising mechanism for post-training Vision-Language-Action (VLA) policies by providing dynamics priors that improve generalization under task and scene variation. However, most WM-based post-training methods rely on pixel-space supervision, making policies sensitive to visual artifacts introduced by imperfect WM rollouts. We present World2Act, a latent-space post-training framework that transfers WM dynamics to the VLA policy without pixel-space supervision. World2Act operates in two stages: 1) it induces a shared video-action latent space by contrastively aligning WM-dynamics latents with action embeddings, and 2) it post-trains the VLA by guiding policy action representations toward WM-imagined dynamics rather than decoded pixels. Built on GR00T-N1.6, World2Act delivers absolute success-rate gains of up to +2.5% on simulation benchmarks (RoboCasa, LIBERO, Bridge-SIMPLER) and +6.7% on a real robot over finetuned VLA baselines. Notably, it outperforms pixel-space WM supervision by up to +6.0%, including on LIBERO where pixel supervision degrades the baseline, suggesting that latent WM dynamics offer a more stable WM-based post-training alternative to pixel-space transfer.

An Dinh Vuong, Tuan Van Vo, Abdullah Sohail +6

MBZUAI

VLA Model Post-Training Vision-Language-Action Models World Models for VLA Models VLMs for Robotics Latent Action Learning

Ask PDF

Abstract

World Models for VLA Models

Explore similar work

WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL

PiL-World: A Chunk-Wise World Model for VLA Policy-in-the-Loop Evaluation

World2Act: Latent Action Post-Training from World Model Dynamics