On the Strengths and Weaknesses of Data for Open-set Embodied Assistance

Pradyumna Tambwekar; Andrew Silva; Deepak Gopinath; Jonathan DeCastro; Xiongyi Cui; Guy Rosman

cs.ROMar 5, 2026

On the Strengths and Weaknesses of Data for Open-set Embodied Assistance

Authors: Pradyumna Tambwekar, Andrew Silva, Deepak Gopinath, Jonathan DeCastro, Xiongyi Cui, Guy Rosman

Organizations: Distyl AI · Toyota Research Institute

Abstract

Embodied foundation models are increasingly performant in real-world domains such as robotics or autonomous driving. These models are often deployed in interactive or assistive settings, where it is important that these assistive models generalize to new users and new tasks. Diverse interactive data generation offers a promising avenue for providing data-efficient generalization capabilities for interactive embodied foundation models. In this paper, we investigate the generalization capabilities of a multimodal foundation model fine-tuned on diverse interactive assistance data in a synthetic domain. We explore generalization along two axes: a) assistance with unseen categories of user behavior and b) providing guidance in new configurations not encountered during training. We study a broad capability called Open-Set Corrective Assistance, in which the model needs to inspect lengthy user behavior and provide assistance through either corrective actions or language-based feedback. This task remains unsolved in prior work, which typically assumes closed corrective categories or relies on external planners, making it a challenging testbed for evaluating the limits of assistive data. To support this task, we generate synthetic assistive datasets in Overcooked and fine-tune a LLaMA-based model to evaluate generalization to novel tasks and user behaviors. Our approach provides key insights into the nature of assistive datasets required to enable open-set assistive intelligence. In particular, we show that performant models benefit from datasets that cover different aspects of assistance, including multimodal grounding, defect inference, and exposure to diverse scenarios.

Figures & tables

Figure 1: We simulate synthetic users in Overcooked to generate multimodal (image + text) gameplay trajectories, which used to distill complementary synthetic datasets. These datasets are designed to (1) ground actions to environmental outcomes and (2) support behavior understanding and correction from trajectories. By training an embodied model on these data, we evaluate whether an embodied foundation model can generalize to unseen defective behaviors and novel task configurations.

	Held-Out Defects		Task Generalization
Baseline	Coaching	Corrections	Coaching	Corrections
Behavior Critic	21.00	20.40	34.21	9.17
Behavior Critic + Summaries	55.70	19.80	71.05	15.83
Ours 1B	76.60	55.70	50.88	50.83
Ours 8B	77.80	54.60	85.96	56.67

Table 1: Generalization across defects and tasks. We compare assistive performance when generalizing to (left) unseen defective behaviors and (right) novel task configurations (recipes). Models trained entirely on diverse assistive data outperform behavior-critic baselines when adapting to new defects with only a few examples, with performance saturating at the 1B scale. In contrast, generalization to new tasks benefits substantially from model scaling, likely due to the stronger multi-modal grounding capacity offered by the larger model.

	Held-out Defects		Task Generalization
Baseline	Coaching	Corrections	Coaching	Corrections
Behavior Critic	21.00	20.40	34.21	9.17
Ours 8B	34.30	39.70	0.00	55.00
Ours 8B + reasoning	18.10	32.60	8.77	43.30

Table 2: The 8B variant of our model outperforms the behavior critic baseline in all settings except while coaching in the task generalization setting. Coaching users to perform new tasks proves to be a challenging setting for the model, however, the inclusion of reasoning traces provides an 8% boost in performance.

	Unseen Defects		Novel Tasks
Training Data	Coaching	Corrections	Coaching	Corrections
1B ( $\mathcal{D}_{train}$ )	76.60	55.70	50.88	50.83
1B ( $\mathcal{D}^{coach}$ )	38.10	–	54.39	–
1B ( $\mathcal{D}^{correct}$ )	–	29.40	–	33.34

Table 3: Effect of dataset ablations on assistive generalization. Training jointly across coaching, correction and defect delineation improves downstream assistive performance compared to individually training on

\mathcal{D}^{coach}

\mathcal{D}^{correct}

Baseline	Coaching	Corrections
1B	50.88	50.83
1B ( $\mathcal{D}_{ground}+\mathcal{D}_{train}$ )	58.77	53.33
1B ( $\mathcal{D}^{I}+\mathcal{D}_{train}$ )	69.30	46.67
1B ( $\mathcal{D}^{V}+\mathcal{D}_{train}$ )	46.49	42.50
1B ( $\mathcal{D}^{T}+\mathcal{D}_{train}$ )	78.95	52.50

Table 4: Inclusion of additional grounding datasets improves model’s ability to assist in new task configurations. This figure depicts the performance of the assistive model, on the task-generalization experiment, after co-training with varying configurations of the grounding datasets,

\mathcal{D}_{ground}

. Among the grounding datasets, the Trajectory-QA dataset,

\mathcal{D}^{T}

, appears to be the most effective, whereas the Video-QA dataset,

\mathcal{D}^{V}

, appears to reduce performance.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model-name	Training Datasets	Data Aug	Learning rate	Training Steps	Per-device batch size	Weight Decay
Ours (1B)	$D_{train}$	No	$5e-5$	15000	1	0.01
1B + grounding datasets	$\mathcal{D}_{train}+\mathcal{D}_{ground}$	Yes	$5e-5$	16500	1	0.01
1B Coaching Only	$\mathcal{D}^{coach}$	Yes	$5e-5$	5000	1	0.01
1B Correction Only	$\mathcal{D}^{correct}$	No	$5e-5$	5000	1	0.01
Ours (1B) + I-QA	$\mathcal{D}_{train}+\mathcal{D}^{I}$	Yes	$5e-5$	16500	1	0.01
Ours (1B) + T-QA	$\mathcal{D}_{train}+\mathcal{D}^{T}$	Yes	$5e-5$	16500	1	0.01

Appendix

Table 5: Experimental parameters per baseline. All used image + llm LoRA with r, alpha = 16. Image -> LLM projection layer trained from scratch. 500 warmup steps.

Figure 2: This figure provides an abstract depiction of each task synthesized in this paper to train our assistive model. Top to Bottom - Trajectory-QA, Video-QA, Image-QA, Corrections, Coaching, Defect Delineation

Assistive Robotics

Assistive robotics develops robots and wearable devices that support mobility, daily activities, rehabilitation, and care.

See all 207 papers

Apr 16, 2026

$π_{0.7}$ : a Steerable Generalist Robotic Foundation Model with Emergent Capabilities

We present a new robotic foundation model, called

π_{0.7}

, that can enable strong out-of-the-box performance in a wide range of scenarios.

π_{0.7}

can follow diverse language instructions in unseen environments, including multi-stage tasks with various kitchen appliances, provide zero-shot cross-embodiment generalization, for example enabling a robot to fold laundry without seeing the task before, and perform challenging tasks such as operating an espresso machine out of the box at a level of performance that matches much more specialized RL-finetuned models. The main idea behind

π_{0.7}

is to use diverse context conditioning during training. This conditioning information, contained in the prompt, makes it possible to steer the model precisely to perform many tasks with different strategies. It is conditioned not just on a language command that describes what it should do, but on additional multimodal information that also describes the manner or strategy in which it should do it, including metadata about task performance and subgoal images. This enables

π_{0.7}

to use very diverse data, including demonstrations, potentially suboptimal (autonomous) data including failures, and data from non-robot sources. Our experiments evaluate

π_{0.7}

across numerous tasks with multiple robot platforms, on tasks that require speed and dexterity, language following, and compositional task generalization.

Physical Intelligence, Bo Ai, Ali Amin +85

Robot Policy Learning Robot Skill Learning Language-Conditioned Robot Manipulation Robot Foundation Models Generalization in Robotic Manipulation Cross-Embodiment Robot Learning+2

Ask PDF

Sep 8, 2026

DYAD: A Multimodal Dataset of Co-Located Human Assistance

An embodied assistant working beside a person must track task state, recognize help seeking, choose how to intervene, and produce an appropriate response. Existing procedural datasets richly describe individual execution, while interactive datasets capture remote verbal instruction or undifferentiated co-working. They do not jointly link a co-located helper's verbal and physical interventions to performer requests, task state, assistance triggers, and outcomes. We introduce DYAD (DYadic Assistance Dataset), a synchronized multimodal record of human-human assistance during gearbox assembly. Across 20 sessions, one trained helper follows a guidance-first policy while assisting HoloLens 2 wearers. DYAD links 528 task-step intervals and 611 performer requests with 851 valid assistance records spanning verbal and physical help. DYAD's annotations span the assistance process; three reference tasks evaluate selected components rather than an end-to-end system: causal step understanding, pre-onset mode anticipation, and instructor response generation. On 829 eligible mode events, the strongest four-seed RGB mean is 0.548 +/- 0.007 macro-F1; causal metadata reaches 0.624 and a privileged trigger mapping 0.915, revealing information not recovered from pre-onset RGB. DYAD's contribution is not scale, but a linked interaction structure spanning help seeking, intervention choice, execution, and outcome under egocentric and workspace sensing.

Akhil Ajikumar, Mahya Qorbani, Sakib Reza +2

Georgia Institute of Technology, Atlanta, GA, USA · Northeastern University, Boston, MA, USA · Microsoft Research, Redmond, WA, USA

Human-AI Interaction Assistive Robotics Human-Robot Interaction Multimodal Understanding

Ask PDF

Jun 23, 2026

OpenThoughts-Agent: Data Recipes for Agentic Models

Agentic language models dramatically expand the applications of AI yet little is publicly known about how to curate training data for broadly capable agents. Existing open efforts such as SWE-Smith, SERA, and Nemotron-Terminal typically target a single benchmark, leaving open the question of how to train models that generalize across diverse agentic tasks. The OpenThoughts-Agent (OT-Agent) project addresses this gap with a fully open data curation pipeline for training agentic models. We conduct more than 100 controlled ablation experiments to systematically investigate each stage of the pipeline, yielding insights on the importance of task sources and diversity. We then assemble a training set of 100K examples from our pipeline and fine-tune Qwen3-32B on this dataset, which yields an average accuracy of 44.8% across seven agentic benchmarks and a 3.9 percentage point improvement over the strongest existing open data agentic model (Nemotron-Terminal-32B, 40.9%). Moreover, our training data exhibits strong scaling properties, outperforming alternative open datasets at every training set size in compute-controlled comparisons. We publicly release our training sets, data pipeline, experimental data, and models at openthoughts.ai to support future open research on agentic model training.

Negin Raoof, Richard Zhuang, Marianna Nezhurina +47

Training Data Curation LLM Agent Training

Ask PDF

Abstract

Assistive Robotics

Explore similar work

π0.7π_{0.7}π0.7​: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities

DYAD: A Multimodal Dataset of Co-Located Human Assistance

OpenThoughts-Agent: Data Recipes for Agentic Models

$π_{0.7}$ : a Steerable Generalist Robotic Foundation Model with Emergent Capabilities