GUILD: Guided Iterative Data Collection for Imitation Learning

Shivin Dass; Kevin Nguyen; Rohan Yelandur; Junhong Xu; Roberto Martín-Martín

cs.RONov 3, 2026

GUILD: Guided Iterative Data Collection for Imitation Learning

Authors: Shivin Dass, Kevin Nguyen, Rohan Yelandur, Junhong Xu, Roberto Martín-Martín

Organizations: University of Texas at Austin

Abstract

Imitation learning policies in robotics are trained on datasets collected through costly real-world teleoperation. Often, the data-collection process is unguided, relying on the teleoperator's intuition, resulting in data that contributes little to overall policy performance. Recent work has shown that curating such unguided datasets can substantially improve policy performance, but these approaches operate by filtering existing datasets after collection and therefore do not prevent the acquisition of uninformative data. In this work, we propose GUILD, a novel Guided Data Collection (GDC) method that actively reasons about which data should be collected to maximize downstream policy performance. We introduce an iterative framework that alternates between 1) task proposal and data collection and 2) data evaluation to close the loop. For task proposal, we leverage semantic priors from vision-language models to identify relevant environments, object configurations, and goals for data collection. However, since semantic relevance does not imply policy improvement, we perform data evaluation. By leveraging predictive data attribution we approximate how the collected trajectories influence policy performance on a target deployment setting. This signal is then used as feedback to guide subsequent data collection, closing the data collection loop. We evaluate our approach in simulation and real robot settings, demonstrating that guided data collection substantially improves sample efficiency and deployment performance compared to passive data collection strategies, multiple baselines and experienced roboticists. More information at https://robin-lab.cs.utexas.edu/guild2026/

Figures & tables

Figure 1: Overview of GUILD Pipeline. Given a limited-access deployment setting and a set of accessible data collection environments, GUILD uses VLM priors to propose candidate scenes, object placements and tasks for data collection. Collected trajectories are then evaluated using predictive data attribution to estimate their influence on deployment-task performance. These attribution scores are used as feedback to guide subsequent data collection and task proposals—closing the loop.

Panel of Figure 2 — Algorithm 1 GUILD Pseudocode

Figure 2: LIBERO guided data collection. We generate data using various strategies and train policies across different points of the process. GUILD either matches or surpasses the next best baseline. For each method on each task, we train 3 different data collection seeds. For every evaluation, we train 5 policies, measure their performance over 100 rollouts and average their performances. The means and standard error are computed across the 3 seeds of data collection process.

Figure 3: Real world guided data collection. We evaluate the policies trained on the data generated by different strategies. GUILD consistently achieves the highest success under the same data collection budget, highlighting its efficacy in guiding data collection towards useful tasks for the deployment setting.

Figure 4: Human Comparison. Avg. performance on 3 LIBERO tasks of datasets collected via various methods and humans.

Panel of Figure 5 — Figure 5: Real-world qualitative analysis. For each target task, we visualize the final allocation of collected trajectories across auxiliary tasks ( middle ), and tasks with the highest data allocation auxiliary tasks ( bottom ). Bar colors indicate the outer-loop iteration in which each auxiliary task was introduced.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Relationship between estimated utility and downstream policy performance. Each point corresponds to a dataset collected by one of the evaluated data-collection strategies. We plot the dataset’s mean task utility against the success rate of the policy trained on that dataset.

	No. of	No. of	No. of Data	Outer-loop	No. of tasks	Batch size	No. of Initial
	Training Envs	Target Demos	Collection Iter.	Schedule	Every Iter. ( $m$ )	( $b$ )	Task Demos
LIBERO
Book-Caddy	16	3	30	[2,4,8,16]	5	20	5
Soup-butter-tray	16	5	30	[2,4,8,16]	5	20	5
Bowl-ketchup-drawer	16	5	30	[2,4,8,16]	5	10	5
Real World

Appendix

Table 1: Task-specific hyperparameters for Guided Data Collection. Data collection iterations denote the total number of inner-loop collection iterations. Outer-loop schedule specifies the inner-loop iterations at which support expansion is performed (Appendix A.1 ).

m

denotes the number of new task parameters proposed during each outer-loop iteration, and initial task demos denotes the number of demonstrations collected when a new task is introduced. Batch size (

b

) specifies the number of demonstrations collected during each inner-loop iteration using attribution-guided task selection.

Panel of Table 2 — Table 2: Task-parameter spaces used across experimental settings. We summarize the components of the task parameter $\phi$ that may be varied by the task proposer in each setting. Scene selection and object configuration indicate whether these components are explicitly chosen during task generation, while the task specification describes how the manipulation objective is represented. The candidate-space column indicates whether proposals are restricted to a fixed task set or generated within a structured/open-ended space.

Panel of Figure 7 — Figure 7: Data Composition of data collected via GUILD. We demonstrate the composition of the final datasets from each of the LIBERO tasks. Top. A visualization of the target task for which the data is collected. Bottom. Final data composition plot, where each bar shows the number of trajectories collected for a particular task.

Figure 8: Task-wise human-guided data collection comparison on LIBERO

Tasks	book-caddy	soup-butter-tray	bowl-ketchup-drawer	Average
GUILD	$68.2\pm 1.3$	$69.6\pm 3.6$	$56.9\pm 1.8$	$64.9$
GUILD (soft-sampling)	$66.1\pm 0.5$	$63.2\pm 2.3$	$54.7\pm 4.5$	$61.3$
GUILD (no-feedback)	$65.5\pm 1.1$	$55.3\pm 3.9$	$56.9\pm 1.7$	$59.2$
GUILD (gemini)	$67.2\pm 2.0$	$52.2\pm 1.5$	$63.8\pm 0.6$	$61.1$

Appendix

Table 3: Mean performance with standard error of policies trained on the final aggregated dataset by GUILD , GUILD (soft-sampling), GUILD (no-feedback) and GUILD (gemini) for simulation tasks. The evaluation methodology is the same as our main experiments described in Appendix B .

Figure 9: Stage-wise real world results. We perform 20 rollouts of a policy on a task and measure the progress of the policy at different stages. The success rates are absolute for each stage, counting it as a fail even if the policy fails in an earlier stage.

Hyperparameter	LIBERO	Real-World
Policy architecture	Octo-Small	Octo-Small
Warmup steps $N_{w}$	Adaptive	Adaptive
Metagradient steps $N_{m}$	Adaptive	Adaptive
Max step bound $C$	500	300
Batch size $B$	512	512
Optimizer	Adam	Adam

Appendix

Table 4: Hyperparameters for computing metagradients.

Scene Image	Pick Objects	Place Objects
	akita_black_bowl_1, butter_1, butter_2, chocolate_pudding_1	wooden_cabinet_1_top_side, wooden_cabinet_1_top_region, akita_black_bowl_1, butter_1, butter_2, chocolate_pudding_1
	akita_black_bowl_1, plate_1	wooden_cabinet_1_top_side, wooden_cabinet_1_top_region, akita_black_bowl_1, plate_1
	akita_black_bowl_1, akita_black_bowl_2, akita_black_bowl_3, plate_1	wooden_cabinet_1_top_side, wooden_cabinet_1_top_region, akita_black_bowl_1, akita_black_bowl_2, akita_black_bowl_3, plate_1
	akita_black_bowl_1, wine_bottle_1	white_cabinet_1_top_side, white_cabinet_1_top_region, white_cabinet_1_bottom_region, wine_rack_1_top_region, akita_black_bowl_1, wine_bottle_1
	akita_black_bowl_1, plate_1, ketchup_1	white_cabinet_1_top_side, white_cabinet_1_top_region, white_cabinet_1_bottom_region, akita_black_bowl_1, plate_1, ketchup_1
	white_bowl_1, plate_1	microwave_1_top_side, plate_1

Appendix

Table 15

Panel of Table 6 — Table 6: Real world scene list

Scene ID	Initial Object Placement	Task
k1	White cup was placed on the plate	Put white cup in the sink
k1	Bread was placed on the plate	Put bread in the oven
k1	Black bowl was placed in the oven	Put black bowl in the sink
k1	Orange was placed on the plate	Put orange in the oven
k1	Coke can was placed in the drying rack	Put coke can on the plate
k1	White cup was placed in the oven and filled with some colored paper balls	Pour the contents of white cup into the sink

Appendix

Table 17

Robotic Data Collection

Robotic data collection develops ways to gather, label, and validate robot and human-generated experience for learning, evaluation, and sensing.

See all 187 papers

Mulligan: Performance-Guided Data Collection for Efficient On-Robot Learning
Oct 5, 2026Lars Ankile, Perry Dong, Rohan Bhowmik +4Robotic Data Collection Robotic Manipulation
Stanford University
Robotic Data Inoculation
Oct 28, 2026Qiyang Li, Zhiyuan Zhou, Charles Xu +1Robotic Data Collection Data-Efficient Robot Learning
UC Berkeley
Learning Beyond What Humans Can Demonstrate
Sep 21, 2026Yuchen Song, Aditya Mittal, Unnat JainRobotic Data Collection Robotic Manipulation
University of California, Irvine

Oct 5, 2026J/K move · Enter open · S save

Mulligan: Performance-Guided Data Collection for Efficient On-Robot Learning

Lars Ankile, Perry Dong, Rohan Bhowmik, Aneesh Muppidi+3

Stanford University

Robotic Data Collection Robotic Manipulation Data-Efficient Robot Learning Interactive Imitation Learning

Open paper AskPDF arXiv

Learning from human demonstrations is a reliable way to teach robots new tasks, but the gains from each additional demonstration shrink as the policy improves. Continued improvement can instead come from supervised deployment, where an operator places the objects and intervenes when the policy fails. We ask how to maximize improvement from a fixed budget of supervised episodes on high-precision manipulation tasks with wide ranges of object placements. We observe that failures can concentrate in a small subset of initial states, so uniform collection spends much of the operator's time on states the policy already handles. Mulligan makes the initial-state distribution a decision, starting each round's episodes at observed failures and untried states. To further improve data efficiency, we augment interactive imitation learning with a value function trained on all data, including failures that imitation discards. Across three real-world tasks evaluated on 2,550 held-out, blinded episodes and two simulated tasks, Mulligan outperforms uniform initial-state sampling at matched collection budgets, and combined with value-based action selection, HiL-IDQL+Mulligan, improves final real-task success by 10-34 percentage points. With operator interventions, the human-robot team completes 98% of collection episodes, remaining productive while the policy learns. Videos, code, and data are available at https://mulligan.page/.

Abstract

Robotic Data Collection

Explore similar work

Mulligan: Performance-Guided Data Collection for Efficient On-Robot Learning

Robotic Data Inoculation

Learning Beyond What Humans Can Demonstrate