Deploying safety-critical agents requires anticipating the consequences of actions before they are executed. While world models offer a paradigm for this proactive foresight, current approaches relying on visual simulation incur prohibitive latencies, often exceeding several seconds per step. In this work, we challenge the assumption that visual processing is necessary for failure prevention. We show that a trained policy's latent state, combined with its planned actions, already encodes sufficient information to anticipate action outcomes, making visual simulation redundant for failure prevention. To this end, we introduce DILLO (DIstiLLed Language-ActiOn World Model), a fast steering layer that shifts the paradigm from "simulate-then-act" to "describe-then-act." DILLO is trained via cross-modal distillation, where a privileged Vision Language Model teacher annotates offline trajectories and a latent-conditioned Large Language Model student learns to predict semantic outcomes. This creates a text-only inference path, bypassing heavy visual generation entirely, achieving a 14x speedup over baselines. Experiments on MetaWorld and LIBERO demonstrate that DILLO produces high-fidelity descriptions of the next state and is able to steer the policy, improving episode success rate by up to 15 pp and 9.3 pp on average across tasks. Code is available at github.com/MaxPappa/DILLO.
Figures & tables
Figure 1: ( left ) Standard approaches rely on heavy visual world models to simulate future outcomes using expensive rendering. In contrast, ( right ) DILLO avoids this visual dependency entirely. By distilling the foresight of a simulator-privileged Teacher VLM into a fast Language-Action World Model, DILLO predicts future outcome descriptions and success verdicts directly from policy latents ( zt ) and action chunks ( at:t+k ). This shift from “simulate-then-act” to “describe-then-act” enables fast proactive steering on consumer hardware.
Figure 2: DILLO Training Pipeline. We distill the foresight of a VLM teacher. The pipeline bypasses visual processing by projecting the fixed policy’s internal latents ( zt ) and action chunks ( at:t+k ) directly into the LLM’s embedding space.
Figure 3: Proactive Policy Steering via Latent Rejection Sampling. At inference time, DILLO acts as a high-speed safety filter. The policy proposes N candidate action chunks ( at:t+ki ). Instead of re-rendering future images for each candidate, DILLO projects the policy latent ( zt ) directly into the Language-Action World Model. The model predicts a semantic description and verdict for each chunk. Negative trajectories (red × ) are rejected, and the first verified positive plan (green ✓ ) is executed.
1B Parameter Models
4B Parameter Models
Environment
Task
DILLO-1B
FS
ZS
CAP
DILLO-4B
FS
ZS
CAP
MetaWorld
Soccer
0.518
0.495
0.398
0.477
0.586
0.501
0.540
0.626
Sweep-Into
0.562
0.329
0.528
0.271
0.554
0.630
0.687
0.725
Drawer-Open
0.706
0.626
0.419
0.612
0.705
0.555
0.553
0.609
LIBERO
Goal
0.727
0.262
0.287
0.723
0.710
0.422
0.241
0.905
Object
0.622
0.388
0.204
0.820
0.587
0.546
0.225
0.877
Table 1: Fidelity Results. We measure the alignment between the predicted description and the actual simulator state. We compare DILLO against LLM and VLM baselines (Few-Shot, Zero-Shot, and CAP) across two model scales (1B and 4B parameters) on both the MetaWorld and LIBERO suites. DILLO consistently achieves high fidelity and outperforms or matches baselines, validating that the latent state is a sufficient statistic for semantic prediction.
1B Parameter Models
4B Parameter Models
Environment
Task
DILLO-1B
FS
ZS
CAP
DILLO-4B
FS
ZS
CAP
MetaWorld
Soccer
0.471
0.301
0.204
0.277
0.504
0.361
0.328
0.425
Sweep-Into
0.592
0.372
0.284
0.388
0.607
0.391
0.381
0.364
Drawer-Open
0.655
0.510
0.189
0.427
0.628
0.497
0.212
0.252
LIBERO
Goal
0.834
0.259
0.271
0.686
0.830
0.487
0.278
0.816
Object
0.803
0.380
0.208
0.668
0.723
0.519
0.243
0.684
Table 2: Reasoning Based LLM Score. We compare DILLO against VLM baselines (Few-Shot, Zero-Shot, and CAP) across two model scales (1B and 4B parameters). DILLO consistently outperforms standard baselines in semantic reasoning, even approaching or exceeding the CAP in the 1B regime.
Figure 4: Qualitative validation of DILLO’s foresight for ‘Positive’ (left) and ‘Negative’ (right) outcomes. DILLO generates the predicted description ( d ) and verdict (check/cross) using only the policy’s latent state ( zt ) and the proposed action chunk. The initial observation ( ot ) and ground-truth future ( ot+K ) are shown purely for visual comparison, confirming the high fidelity of the latent-based prediction.
Success Rate
Verdict Accuracy
Task
Base
DILLO-1B
DILLO-4B
DILLO-1B
DILLO-4B
MetaWorld
Soccer
65.0%
75.0%
80.0%
82.7%
87.4%
Sweep-Into
80.0%
90.0%
85.0%
97.7%
92.1%
Drawer-Open
75.0%
70.0%
90.0%
80.6%
94.7%
LIBERO
Goal
86.0%
87.5%
89.0%
80.0%
87.5%
Object
82.0%
86.5%
88.0%
79.8%
81.2%
Table 3: Steering Success Rates and Verdict Accuracy. We report Steering Success Rates (SR) and Safety Verdict Accuracy (%). High success rates and accuracy confirm that the model effectively steers rollouts while discriminating successful dynamics from failures.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
1B Parameter Models
4B Parameter Models
Suite
Task
DILLO-1B
FS
ZS
CAP
DILLO-4B
FS
ZS
CAP
MetaWorld
Soccer
0.523
0.451
0.289
0.475
0.655
0.495
0.519
0.617
Sweep-Into
0.693
0.494
0.202
0.140
0.649
0.592
0.567
0.594
Drawer-Open
0.531
0.578
0.245
0.570
0.735
0.685
0.245
0.621
LIBERO
Goal
0.727
0.262
0.287
0.723
0.710
0.422
0.241
0.905
Object
0.622
0.388
0.204
0.820
0.587
0.546
0.225
0.877
Appendix
Table 4: Fidelity Text-to-Text Results. We compare DILLO against predictive baselines (Few-Shot and Zero-Shot) across two model scales (1B and 4B parameters) on both the MetaWorld and LIBERO suites. Unlike the predictive baselines, the captioning model (CAP) has privileged access to past and future information. DILLO consistently outperforms the fair predictive baselines, validating that the latent state is a sufficient statistic for semantic prediction.
Figure 6: Qualitative examples of DILLO for Drawer-Open (Top), Soccer (Middle), and Sweep-Into (Bottom) tasks.
Figure 7: Qualitative examples of DILLO for LIBERO-90 (Top), LIBERO-10 (Middle), and LIBERO-Goal (Bottom) tasks. Descriptions are trimmed for readability.
Figure 9: AI judge preference percentage results for the 1B and 4B models across the three MetaWorld tasks (Sweep-Into, Soccer, and Drawer-Open) and the five LIBERO suites (Spatial, Object, Goal, LIBERO-10, and LIBERO-90).
DILLO-1B
DILLO-4B
Suite
Base
Multistage
Stage 3 only
Verdict only
Multistage
Stage 3 only
Verdict only
MLP
LIBERO-Goal
86.0
87.5
79.0
85.0
89.0
92.0
83.0
86.0
LIBERO-Object
82.0
86.5
80.0
83.0
88.0
81.0
81.0
80.5
LIBERO-Spatial
70.0
76.0
69.0
71.0
73.0
54.0
64.0
70.5
LIBERO-10
64.0
71.5
59.0
65.0
74.0
67.0
63.0
64.0
LIBERO-90
71.4
68.6
70.8
71.5
70.5
73.6
70.1
70.2
Appendix
Table 5: Steering ablation on LIBERO validation suites. Episode success rates (%). Multistage denotes the complete DILLO training procedure.
Pretrained vision-language-action (VLA) policies show promising zero-shot generalization, but often fail under deployment-time distribution shift, leading to decreased robustness and inconsistent instruction following. While prior work commonly tackles this by finetuning on in-distribution data, it assumes demonstrations collected on tasks in the target environment. In this work, we propose DREAMSTEER, a deployment-time steering framework for pretrained VLAs without any finetuning or parameter modifications. The key insight in DREAMSTEER is to leverage a latent world model and a value model to steer pretrained VLA policies. During deployment, DREAMSTEER samples candidate action chunks from a VLA policy and predefined motion primitives, imagines their outcomes using an action-conditioned latent world model, and ranks the imagined trajectories with a language-conditioned value model. Across four real-world manipulation benchmarks with unseen objects, DREAMSTEER improves task success rate from 23.75% to 66.25% and instruction-following accuracy from 38.75% to 56.25% over the base VLA policy.
Hanchen Cui, Sergio Arnaud, Arjun Majumdar +5
1Fundamental AI Research (FAIR), Meta · University of Minnesota Twin Cities
As large language model (LLM) agents increasingly invoke external tools and interact with real-world systems, unsafe actions may cause irreversible consequences on external states, user data, and downstream services. Recent runtime guardrails mitigate such risks by checking proposed actions before execution, but many remain reactive: they primarily assess the apparent safety of the current action, lacking an explicit model of how risk evolves across the trajectory. This limitation creates a critical blind spot for long-horizon risks, where individually benign-looking actions can gradually drift the agent toward hazardous states. In response, we propose DreamGuard, a proactive guardrail for LLM agents built around a risk-aware world model. The world model maintains a compact recurrent latent state over the trajectory and predicts future latent states from which DreamGuard derives immediate-hazard and prefix-risk evidence. It then fuses these multi-horizon signals into intervention decisions before execution. Experiments across four benchmarks and an online guardrail evaluation show that DreamGuard outperforms generic, reactive, and proactive guardrail baselines, achieves the best safety-utility trade-off among evaluated guardrails, and maintains an average end-to-end latency of 25 ms per call.
Wenhao Lin, Chenyu Yu, Xingwei Lin +6
Zhejiang University · Nanjing University of Posts and Telecommunications · Sun Yat-sen University
Latent world models offer a promising way to improve Vision-Language-Action policies by capturing the consequences of actions. However, models trained primarily on expert demonstrations have limited exposure to failure outcomes and may struggle to distinguish visually similar successful and failed interactions. We propose WorldGuide, a framework that learns these distinctions in latent space and uses them to guide policy training. WorldGuide combines predictive pretraining on successful and failed trajectories with contrastive learning on matched success--failure pairs. The learned predictor then provides a differentiable reward to guide joint optimization of the policy and visual encoder. The predictor is discarded after training, so deployment requires no additional world-model inference. Extensive experiments show that WorldGuide substantially improves VLA reliability and achieves state of the art performance on LIBERO 100 and SimplerEnv, reaching 96.8% and 72.0%, respectively. Code will be publicly available.
Lin Liu, Lu Zhang, Ziying Song +6
School of Information and Communication Engineering Dalian University of Technology & Beta Infinity · School of Information and Communication, Engineering Dalian University of Technology · Nanyang Technological University +1