Human teleoperators spend substantial time demonstrating behaviors that robots can already perform autonomously, limiting the scalability of data collection for robot foundation models. Task and motion planning (TAMP) can automate many of these behaviors, but a fixed planning domain may not support every stage of a long-horizon manipulation task. We present TANDEM (Tamp with As-Needed Demonstrations for Efficient Model fine-tuning), a system that combines TAMP with selective human teleoperation to collect demonstrations for tasks beyond the planner's capabilities. Our key idea is to represent human assistance as an on-demand planning capability. Given a language instruction and visual observation, TANDEM uses pretrained vision-language models to extend the planning domain with missing predicates and human-executed magic operators. This allows the planner to interleave autonomous and human-executed stages without task-specific intervention points. After each human stage, TANDEM re-perceives the scene and checks whether the intended effects hold before resuming autonomous planning. To support fine-tuning vision-language-action (VLA) models, TANDEM also uses example pretraining trajectories to align planner-generated motions with the target model's pretraining distribution. We evaluate TANDEM on five long-horizon manipulation tasks beyond the TAMP domain's capabilities. On a representative long-horizon task, TANDEM collects 2.9x as many demonstrations as full-task teleoperation at the same human intervention time. Fine-tuning a pretrained π_{0.5}-DROID model on 20 TANDEM demonstrations per task increases average task success from 0% to 60% across the five tasks.
Figures & tables
Fig. 2 : Overview of TANDEM. Given a language instruction and an initial visual observation, TANDEM uses a VLM to identify task requirements beyond the capabilities of the existing TAMP domain and automatically extends the domain with invented predicates and human-executed magic operators. The resulting task plan interleaves autonomous TAMP phases with human-assisted phases. TAMP executes the supported stages, while the human performs unsupported stages through teleoperation. After each human phase, TANDEM re-perceives the scene and verifies the intended effects before resuming autonomous planning.
Fig. 3 : Overview of the five long-horizon manipulation tasks used to evaluate TANDEM. Each task combines routine pick-and-place behaviors that TAMP can execute autonomously with manipulation stages beyond the underlying TAMP domain. The task sequences illustrate how TANDEM interleaves autonomous TAMP execution (green) with human teleoperation (blue) to complete the full task.
Fig. 4 : Failure case analysis of TANDEM across 130 execution attempts. Most failures are caused by the underlying TAMP executions on the real robot, while predicate/operator invention accounts for 6.7% of the failed episodes.
Task
Raw TAMP
TANDEM
Cover Bread Rolls
0% (0/20)
64.5% (20/31)
Solve Constrained Puzzle
0% (0/20)
87.0% (20/23)
Sort & Cover Snacks
0% (0/20)
90.9% (20/22)
Open Obstructed Book
0% (0/20)
80.0% (20/25)
Store Bread in Closed Box
0% (0/20)
69.0% (20/29)
Total
0% (0/100)
76.9% (100/130)
TABLE I : Demonstration collection success rates across the five long-horizon tasks. We report successful demonstrations over total execution attempts for TANDEM. Raw TAMP cannot complete these tasks because each contains at least one stage beyond its pick-and-place capabilities.
Fig. 5 : Scaling with matched human effort on Cover Bread Rolls. Each point corresponds to a matched human-effort budget. (a) TANDEM collects substantially more demonstrations than full-task human teleoperation, achieving approximately 2.9× higher data throughput across the evaluated budgets. (b) The additional demonstrations translate into improved downstream VLA performance, with TANDEM consistently outperforming human teleoperation at the same human-effort budget.
Fig. 6 : Downstream policy performance of π0.5 -DROID before and after fine-tuning with TANDEM demonstrations, compared with HITL-TAMP across five long-horizon tasks. Bars show mean success rate and task progress over 20 evaluation trials per task; error bars indicate standard error. TANDEM fine-tuning substantially improves both task progress and success rate over the pretrained policy π0.5 -DROID, and outperforms the HITL-TAMP baseline in success rate on all five tasks.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Subgoal
Criterion
Cover Bread Rolls “place the 3 breads on the plate and then cover the plate with the cloth”
one_bread
One piece of bread ends up on the plate
two_bread
A second piece of bread ends up on the plate
three_bread
All three pieces of bread end up on the plate
cover_plate
The plate is covered with the cloth
Solve Constrained Puzzle “place the toy on the cloth and solve the puzzle”
Appendix
TABLE II : Task-specific evaluation subgoals and success criteria, with the language instruction given to the policy for each task.
Failure attribution
Task
Successes
Failures
Predicate/operator invention
TAMP (planning)
TAMP (execution)
Human teleoperation
Cover Bread Rolls
20
11
0% (0/11)
0% (0/11)
100% (11/11)
0% (0/11)
Solve Constrained Puzzle
20
3
0% (0/3)
0% (0/3)
100% (3/3)
0% (0/3)
Sort & Cover Snacks
20
2
0% (0/2)
0% (0/2)
100% (2/2)
0% (0/2)
Open Obstructed Book
20
5
0% (0/5)
0% (0/5)
100% (5/5)
0% (0/5)
Store Bread in Closed Box
20
9
22.2% (2/9)
0% (0/9)
77.8% (7/9)
0% (0/9)
Appendix
TABLE III : Failure analysis of TANDEM demonstration collection across the five tasks. For each task, we report the number of successful demonstrations and failed episodes, together with the fraction of failures attributed to predicate and operator invention, TAMP planning, TAMP execution, and human teleoperation.
Failure attribution
Task
Approach
Failures
Grasping
Manipulation / placement
OOD joint configurations
Semantic
Timeout (stuck)
Timeout (unfinished)
Cover Bread Rolls
π0.5 -DROID
20
0%
5.0% (1/20)
0%
95.0% (19/20)
0%
0%
HITL-TAMP
14
71.4% (10/14)
28.6% (4/14)
0%
0%
0%
0%
TANDEM
11
0%
0%
9.1% (1/11)
72.7% (8/11)
18.2% (2/11)
0%
Solve Constrained Puzzle
π0.5 -DROID
20
0%
15.0% (3/20)
0%
85.0% (17/20)
0%
0%
HITL-TAMP
20
15.0% (3/20)
85.0% (17/20)
0%
0%
0%
0%
Appendix
TABLE IV : Failure analysis of downstream policy execution across the five tasks. For each task and approach, we report the number of failed evaluation trials and their attribution across six failure modes: grasping, manipulation/placement, out-of-distribution (OOD) joint configurations, semantic errors, stuck timeouts, and unfinished timeouts. Percentages are computed over the failed trials in each row, where the most common failure mode is highlighted and the runner-up is underlined .
Collecting high-quality robot data remains a fundamental challenge for training robot foundation models. Task and motion planning (TAMP) offers a scalable way to generate demonstrations, but our experiments show that raw TAMP trajectories provide surprisingly little benefit when used to fine-tune pretrained vision-language-action (VLA) models, despite successfully solving the target tasks. We hypothesize that this failure arises from a behavioral distribution mismatch between planner-generated trajectories and the data used to pretrain the VLA. To address this mismatch, we introduce DATAFARM: Distribution-Aligned Task And motion planning for Fine-tuning A Robot foundation Model, an approach that incorporates the pretraining distribution directly into TAMP trajectory generation. DATAFARM aligns generated trajectories with the pretraining data in robot joint configurations, motion style, and temporal execution profiles. We evaluate DATAFARM on three tabletop manipulation tasks that TAMP can perform and a cloth-folding task beyond the capability of TAMP. DATAFARM achieves an average success rate of 56.7%, substantially outperforming raw TAMP (8.3%) while approaching human teleoperation (61.7%). On Deformable Object Manipulation, which is outside the fine-tuning distribution, the fine-tuned model retains 85% success, compared with 90% for the pretrained model. These results show that aligning planner-generated demonstrations with the pretraining distribution can make TAMP an effective source of data for VLA fine-tuning. Website and code: https://prpl-group.com/datafarm/
Vision-language-action (VLA) models have made strong progress in language-conditioned robot manipulation, but improving their performance in a new workspace still often requires action-labeled data from that environment. Collecting such data by human teleoperation is costly, especially when each workspace, object arrangement, or task may require new demonstrations. We present DREAM, a framework that generates fine-tuning data for a pretrained VLA from a captured workspace and a language instruction, without requiring a task-specific human demonstration. DREAM reconstructs the workspace, automatically translates the instruction into symbolic task goals and success criteria using a large language model, and uses task-and-motion planning to generate feasible robot trajectories. The planned trajectories are augmented across randomized object configurations, verified by the generated success criteria, and rendered into image-action examples for VLA fine-tuning. Through real-robot experiments on language-conditioned manipulation tasks, we study whether DREAM can serve as a scalable data-collection system for the deployment workspace by examining whether fine-tuning on its automatically generated data improves success over direct deployment and how its data-collection cost compares with human teleoperation when adapting a VLA to a new workspace.
Robotic manipulation requires sophisticated commonsense reasoning, a capability naturally possessed by large-scale Vision-Language Models (VLMs). While VLMs show promise as zero-shot planners, their lack of grounded physical understanding often leads to compounding errors and low success rates when deployed in complex real-world environments, particularly for challenging tasks like deformable object manipulation. Although Reinforcement Learning (RL) can adapt these planners to specific task dynamics, directly fine-tuning VLMs via real-world interaction is prohibitively expensive, unsafe, and sample-inefficient. To overcome this bottleneck, we introduce DreamPlan, a novel framework for the reinforcement fine-tuning of VLM planners via video world models. Instead of relying on costly physical rollouts, DreamPlan first leverages the zero-shot VLM to collect exploratory interaction data. We demonstrate that this sub-optimal data is sufficient to train an action-conditioned video generation model, which implicitly captures complex real-world physics. Subsequently, the VLM planner is fine-tuned entirely within the "imagination" of this video world model using Odds Ratio Policy Optimization (ORPO). By utilizing these virtual rollouts, physical and task-specific knowledge is efficiently injected into the VLM. Our results indicate that DreamPlan bridges the gap between semantic reasoning and physical grounding, significantly improving manipulation success rates without the need for large-scale real-world data collection. Our project page is https://psi-lab.ai/DreamPlan/.
Emily Yue-Ting Jia, Weiduo Yuan, Tianheng Shi +3
USC Physical Superintelligence Lab · Toyota Research Institute