DREAM: Deployment-Time Demonstration Generation via Real-to-Sim for Scalable Policy Adaptation

Makoto Sato; Tatsuya Matsushima; Yutaka Matsuo; Yusuke Iwasawa

cs.ROAug 29, 2026

DREAM: Deployment-Time Demonstration Generation via Real-to-Sim for Scalable Policy Adaptation

Authors: Makoto Sato, Tatsuya Matsushima, Yutaka Matsuo, Yusuke Iwasawa

Organizations: The University of Tokyo, Tokyo, Japan.

Abstract

Vision-language-action (VLA) models have made strong progress in language-conditioned robot manipulation, but improving their performance in a new workspace still often requires action-labeled data from that environment. Collecting such data by human teleoperation is costly, especially when each workspace, object arrangement, or task may require new demonstrations. We present DREAM, a framework that generates fine-tuning data for a pretrained VLA from a captured workspace and a language instruction, without requiring a task-specific human demonstration. DREAM reconstructs the workspace, automatically translates the instruction into symbolic task goals and success criteria using a large language model, and uses task-and-motion planning to generate feasible robot trajectories. The planned trajectories are augmented across randomized object configurations, verified by the generated success criteria, and rendered into image-action examples for VLA fine-tuning. Through real-robot experiments on language-conditioned manipulation tasks, we study whether DREAM can serve as a scalable data-collection system for the deployment workspace by examining whether fine-tuning on its automatically generated data improves success over direct deployment and how its data-collection cost compares with human teleoperation when adapting a VLA to a new workspace.

Figures & tables

Method	Teleop- free	Demo- free	Real-to- Sim	Language task	Long- horizon
Scaling Up [ 9 ]	✓	✓	–	✓	✓
OPTIMUS [ 12 ]	✓	✓	–	–	✓
SkillMimicGen [ 15 ]	–	–	–	–	✓
RialTo [ 6 ]	–	–	✓	–	–
RoboSplat [ 20 ]	–	–	✓	–	–
R2R2R [ 5 ]	✓	–	✓	–	–

TABLE I: Comparison of representative robot data-generation methods. ✓ denotes support, while – denotes that the capability is not provided.

Fig. 2: The DREAM pipeline. Given a short RGB video of the target workspace and a language instruction, DREAM reconstructs a metrically aligned digital twin, translates the instruction into symbolic goals and staged success criteria with a large language model, generates feasible robot trajectories with TAMP, augments them across randomized object configurations, and renders them into photorealistic image–action demonstrations. The resulting synthetic dataset is used to fine-tune a pretrained VLA policy, which is then deployed directly in the physical workspace.

Panel of Figure 3 — Fig. 3: The two tabletop manipulation tasks. For each task, the left image compares the initial state in the reconstructed digital twin (Simulation) with the corresponding real workspace (Real), and the right image shows the target state, rendered in the reconstructed simulation, together with the language instruction. In (a) BlockIntoBowl , the robot places a blue block into a red bowl. In (b) FruitPacking , the robot places a banana and a peach onto a red plate in sequence.

Fig. 4: Sim-to-real correlation of policy performance. Each point represents a policy checkpoint trained with a particular dataset size and random seed. The horizontal axis shows real-world success rate, and the vertical axis shows success rate in the reconstructed simulation. Colors distinguish the two tasks, while marker shapes distinguish the three policy architectures (Diffusion Policy, SmolVLA, and $\pi_{0.5}$ ). The dashed line represents equal simulated and real-world performance.

Panel of Figure 5 — Fig. 5: Real-world success rate as a function of cumulative data acquisition time for BlockIntoBowl (top) and FruitPacking (bottom). Each panel compares SmolVLA and $\pi_{0.5}$ trained on demonstrations obtained through human teleoperation or generated by DREAM . Human-teleoperation datasets contain 10, 50, and 100 demonstrations, whereas DREAM datasets contain 100, 500, and 1000 demonstrations.

Synthetic Data Generation for Robotics

Synthetic data generation for robotics creates simulated scenes, sensor readings, and action trajectories to train and evaluate robots.

See all 286 papers

DREAMSTEER: Latent World Models Can Steer VLA Policies During Deployment Without Any Finetuning
Jul 3, 2026Hanchen Cui, Sergio Arnaud, Arjun Majumdar +5World Model-Based Planning Robotic Manipulation
1Fundamental AI Research (FAIR), Meta · University of Minnesota Twin Cities
RDGen: Demonstration Generation for High-Quality Robot Learning via Reinforcement Learning
May 29, 2026Zijian Zhu, Menglin Zou, Zhuang Li +2RL for Robotics Robot Skill Learning
DATAFARM: Distribution-Aligned Task and Motion Planning for Fine-Tuning Vision-Language-Action Models
Sep 11, 2026Samrat Sahoo, Yixuan Huang, Tom SilverSynthetic Data Generation for Robotics Robotic Manipulation
Stanford University · Princeton University

Jul 3, 2026J/K move · Enter open · S save

DREAMSTEER: Latent World Models Can Steer VLA Policies During Deployment Without Any Finetuning

Hanchen Cui, Sergio Arnaud, Arjun Majumdar, Daniel Dugas+4

1Fundamental AI Research (FAIR), Meta · University of Minnesota Twin Cities

World Model-Based Planning Robotic Manipulation Vision-Language-Action Models Test-Time Adaptation of VLA Models Robustness of VLA Models

Open paper AskPDF arXiv

Pretrained vision-language-action (VLA) policies show promising zero-shot generalization, but often fail under deployment-time distribution shift, leading to decreased robustness and inconsistent instruction following. While prior work commonly tackles this by finetuning on in-distribution data, it assumes demonstrations collected on tasks in the target environment. In this work, we propose DREAMSTEER, a deployment-time steering framework for pretrained VLAs without any finetuning or parameter modifications. The key insight in DREAMSTEER is to leverage a latent world model and a value model to steer pretrained VLA policies. During deployment, DREAMSTEER samples candidate action chunks from a VLA policy and predefined motion primitives, imagines their outcomes using an action-conditioned latent world model, and ranks the imagined trajectories with a language-conditioned value model. Across four real-world manipulation benchmarks with unseen objects, DREAMSTEER improves task success rate from 23.75% to 66.25% and instruction-following accuracy from 38.75% to 56.25% over the base VLA policy.

Abstract

Synthetic Data Generation for Robotics

Explore similar work

DREAMSTEER: Latent World Models Can Steer VLA Policies During Deployment Without Any Finetuning

RDGen: Demonstration Generation for High-Quality Robot Learning via Reinforcement Learning

DATAFARM: Distribution-Aligned Task and Motion Planning for Fine-Tuning Vision-Language-Action Models