Describe-Then-Act: Proactive Agent Steering via Distilled Language-Action World Models

Massimiliano Pappa; Luca Romani; Valentino Sacco; Alessio Palma; Stéphane Lathuilière; Fabio Galasso; Xavier Alameda-Pineda; Indro Spinelli

cs.AIMar 24, 2026

Describe-Then-Act: Proactive Agent Steering via Distilled Language-Action World Models

Authors: Massimiliano Pappa, Luca Romani, Valentino Sacco, Alessio Palma, Stéphane Lathuilière, Fabio Galasso, Xavier Alameda-Pineda, Indro Spinelli

Organizations: Sapienza University of Rome, Italy · Inria, Univ. Grenoble Alpes, CNRS, LJK

Abstract

Deploying safety-critical agents requires anticipating the consequences of actions before they are executed. While world models offer a paradigm for this proactive foresight, current approaches relying on visual simulation incur prohibitive latencies, often exceeding several seconds per step. In this work, we challenge the assumption that visual processing is necessary for failure prevention. We show that a trained policy's latent state, combined with its planned actions, already encodes sufficient information to anticipate action outcomes, making visual simulation redundant for failure prevention. To this end, we introduce DILLO (DIstiLLed Language-ActiOn World Model), a fast steering layer that shifts the paradigm from "simulate-then-act" to "describe-then-act." DILLO is trained via cross-modal distillation, where a privileged Vision Language Model teacher annotates offline trajectories and a latent-conditioned Large Language Model student learns to predict semantic outcomes. This creates a text-only inference path, bypassing heavy visual generation entirely, achieving a 14x speedup over baselines. Experiments on MetaWorld and LIBERO demonstrate that DILLO produces high-fidelity descriptions of the next state and is able to steer the policy, improving episode success rate by up to 15 pp and 9.3 pp on average across tasks. Code is available at github.com/MaxPappa/DILLO.

Figures & tables

Figure 1: ( left ) Standard approaches rely on heavy visual world models to simulate future outcomes using expensive rendering. In contrast, ( right ) DILLO avoids this visual dependency entirely. By distilling the foresight of a simulator-privileged Teacher VLM into a fast Language-Action World Model, DILLO predicts future outcome descriptions and success verdicts directly from policy latents ( $z_{t}$ ) and action chunks ( $a_{t:t+k}$ ). This shift from “simulate-then-act” to “describe-then-act” enables fast proactive steering on consumer hardware.

Figure 2: DILLO Training Pipeline. We distill the foresight of a VLM teacher. The pipeline bypasses visual processing by projecting the fixed policy’s internal latents ( $z_{t}$ ) and action chunks ( $a_{t:t+k}$ ) directly into the LLM’s embedding space.

Figure 3: Proactive Policy Steering via Latent Rejection Sampling. At inference time, DILLO acts as a high-speed safety filter. The policy proposes $N$ candidate action chunks ( $a^{i}_{t:t+k}$ ). Instead of re-rendering future images for each candidate, DILLO projects the policy latent ( $z_{t}$ ) directly into the Language-Action World Model. The model predicts a semantic description and verdict for each chunk. Negative trajectories (red $\times$ ) are rejected, and the first verified positive plan (green $\checkmark$ ) is executed.

		1B Parameter Models				4B Parameter Models
Environment	Task	DILLO-1B	FS	ZS	CAP	DILLO-4B	FS	ZS	CAP
MetaWorld	Soccer	0.518	0.495	0.398	0.477	0.586	0.501	0.540	0.626
	Sweep-Into	0.562	0.329	0.528	0.271	0.554	0.630	0.687	0.725
	Drawer-Open	0.706	0.626	0.419	0.612	0.705	0.555	0.553	0.609
LIBERO	Goal	0.727	0.262	0.287	0.723	0.710	0.422	0.241	0.905
	Object	0.622	0.388	0.204	0.820	0.587	0.546	0.225	0.877

Table 1: Fidelity Results. We measure the alignment between the predicted description and the actual simulator state. We compare DILLO against LLM and VLM baselines (Few-Shot, Zero-Shot, and CAP) across two model scales (1B and 4B parameters) on both the MetaWorld and LIBERO suites. DILLO consistently achieves high fidelity and outperforms or matches baselines, validating that the latent state is a sufficient statistic for semantic prediction.

		1B Parameter Models				4B Parameter Models
Environment	Task	DILLO-1B	FS	ZS	CAP	DILLO-4B	FS	ZS	CAP
MetaWorld	Soccer	0.471	0.301	0.204	0.277	0.504	0.361	0.328	0.425
	Sweep-Into	0.592	0.372	0.284	0.388	0.607	0.391	0.381	0.364
	Drawer-Open	0.655	0.510	0.189	0.427	0.628	0.497	0.212	0.252
LIBERO	Goal	0.834	0.259	0.271	0.686	0.830	0.487	0.278	0.816
	Object	0.803	0.380	0.208	0.668	0.723	0.519	0.243	0.684

Table 2: Reasoning Based LLM Score. We compare DILLO against VLM baselines (Few-Shot, Zero-Shot, and CAP) across two model scales (1B and 4B parameters). DILLO consistently outperforms standard baselines in semantic reasoning, even approaching or exceeding the CAP in the 1B regime.

Figure 4: Qualitative validation of DILLO’s foresight for ‘Positive’ (left) and ‘Negative’ (right) outcomes. DILLO generates the predicted description ( $d$ ) and verdict (check/cross) using only the policy’s latent state ( $z_{t}$ ) and the proposed action chunk. The initial observation ( $o_{t}$ ) and ground-truth future ( $o_{t+K}$ ) are shown purely for visual comparison, confirming the high fidelity of the latent-based prediction.

		Success Rate			Verdict Accuracy
	Task	Base	DILLO-1B	DILLO-4B	DILLO-1B	DILLO-4B
MetaWorld	Soccer	65.0%	75.0%	80.0%	82.7%	87.4%
	Sweep-Into	80.0%	90.0%	85.0%	97.7%	92.1%
	Drawer-Open	75.0%	70.0%	90.0%	80.6%	94.7%
LIBERO	Goal	86.0%	87.5%	89.0%	80.0%	87.5%
	Object	82.0%	86.5%	88.0%	79.8%	81.2%

Table 3: Steering Success Rates and Verdict Accuracy. We report Steering Success Rates (SR) and Safety Verdict Accuracy (%). High success rates and accuracy confirm that the model effectively steers rollouts while discriminating successful dynamics from failures.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

		1B Parameter Models				4B Parameter Models
Suite	Task	DILLO-1B	FS	ZS	CAP	DILLO-4B	FS	ZS	CAP
MetaWorld	Soccer	0.523	0.451	0.289	0.475	0.655	0.495	0.519	0.617
	Sweep-Into	0.693	0.494	0.202	0.140	0.649	0.592	0.567	0.594
	Drawer-Open	0.531	0.578	0.245	0.570	0.735	0.685	0.245	0.621
LIBERO	Goal	0.727	0.262	0.287	0.723	0.710	0.422	0.241	0.905
	Object	0.622	0.388	0.204	0.820	0.587	0.546	0.225	0.877

Appendix

Table 4: Fidelity Text-to-Text Results. We compare DILLO against predictive baselines (Few-Shot and Zero-Shot) across two model scales (1B and 4B parameters) on both the MetaWorld and LIBERO suites. Unlike the predictive baselines, the captioning model (CAP) has privileged access to past and future information. DILLO consistently outperforms the fair predictive baselines, validating that the latent state is a sufficient statistic for semantic prediction.

Figure 6: Qualitative examples of DILLO for Drawer-Open (Top), Soccer (Middle), and Sweep-Into (Bottom) tasks.

Figure 7: Qualitative examples of DILLO for LIBERO-90 (Top), LIBERO-10 (Middle), and LIBERO-Goal (Bottom) tasks. Descriptions are trimmed for readability.

Panel of Figure 9 — Figure 9: AI judge preference percentage results for the 1B and 4B models across the three MetaWorld tasks (Sweep-Into, Soccer, and Drawer-Open) and the five LIBERO suites (Spatial, Object, Goal, LIBERO-10, and LIBERO-90).

Suite	Base	Multistage	Stage 3 only	Verdict only	Multistage	Stage 3 only	Verdict only	MLP
		DILLO-1B			DILLO-4B
LIBERO-Goal	86.0	87.5	79.0	85.0	89.0	92.0	83.0	86.0
LIBERO-Object	82.0	86.5	80.0	83.0	88.0	81.0	81.0	80.5
LIBERO-Spatial	70.0	76.0	69.0	71.0	73.0	54.0	64.0	70.5
LIBERO-10	64.0	71.5	59.0	65.0	74.0	67.0	63.0	64.0
LIBERO-90	71.4	68.6	70.8	71.5	70.5	73.6	70.1	70.2

Appendix

Table 5: Steering ablation on LIBERO validation suites. Episode success rates (%). Multistage denotes the complete DILLO training procedure.

Efficient Inference for World Action Models

Efficient inference for world action models cuts the cost of predicting action-conditioned futures for responsive control and planning.

See all 80 papers

DREAMSTEER: Latent World Models Can Steer VLA Policies During Deployment Without Any Finetuning
Jul 3, 2026Hanchen Cui, Sergio Arnaud, Arjun Majumdar +5World Model-Based Planning Robotic Manipulation
1Fundamental AI Research (FAIR), Meta · University of Minnesota Twin Cities
DreamGuard: Efficient Runtime Guardrail for LLM Agents via Risk-Aware World Model
Aug 6, 2026Wenhao Lin, Chenyu Yu, Xingwei Lin +6LLM World Models LLM Guardrails
Zhejiang University · Nanjing University of Posts and Telecommunications · Sun Yat-sen University
WorldGuide: Learning Success-Failure Boundaries in Latent World Models for Vision-Language-Action Policies
Sep 28, 2026Lin Liu, Lu Zhang, Ziying Song +6Robotic Manipulation Vision-Language-Action Models
School of Information and Communication Engineering Dalian University of Technology & Beta Infinity · School of Information and Communication, Engineering Dalian University of Technology · Nanyang Technological University +1

Jul 3, 2026J/K move · Enter open · S save

DREAMSTEER: Latent World Models Can Steer VLA Policies During Deployment Without Any Finetuning

Hanchen Cui, Sergio Arnaud, Arjun Majumdar, Daniel Dugas+4

1Fundamental AI Research (FAIR), Meta · University of Minnesota Twin Cities

World Model-Based Planning Robotic Manipulation Vision-Language-Action Models Test-Time Adaptation of VLA Models Robustness of VLA Models

Open paper AskPDF arXiv

Pretrained vision-language-action (VLA) policies show promising zero-shot generalization, but often fail under deployment-time distribution shift, leading to decreased robustness and inconsistent instruction following. While prior work commonly tackles this by finetuning on in-distribution data, it assumes demonstrations collected on tasks in the target environment. In this work, we propose DREAMSTEER, a deployment-time steering framework for pretrained VLAs without any finetuning or parameter modifications. The key insight in DREAMSTEER is to leverage a latent world model and a value model to steer pretrained VLA policies. During deployment, DREAMSTEER samples candidate action chunks from a VLA policy and predefined motion primitives, imagines their outcomes using an action-conditioned latent world model, and ranks the imagined trajectories with a language-conditioned value model. Across four real-world manipulation benchmarks with unseen objects, DREAMSTEER improves task success rate from 23.75% to 66.25% and instruction-following accuracy from 38.75% to 56.25% over the base VLA policy.

Abstract

Efficient Inference for World Action Models

Explore similar work

DREAMSTEER: Latent World Models Can Steer VLA Policies During Deployment Without Any Finetuning

DreamGuard: Efficient Runtime Guardrail for LLM Agents via Risk-Aware World Model

WorldGuide: Learning Success-Failure Boundaries in Latent World Models for Vision-Language-Action Policies