Imitation learning policies in robotics are trained on datasets collected through costly real-world teleoperation. Often, the data-collection process is unguided, relying on the teleoperator's intuition, resulting in data that contributes little to overall policy performance. Recent work has shown that curating such unguided datasets can substantially improve policy performance, but these approaches operate by filtering existing datasets after collection and therefore do not prevent the acquisition of uninformative data. In this work, we propose GUILD, a novel Guided Data Collection (GDC) method that actively reasons about which data should be collected to maximize downstream policy performance. We introduce an iterative framework that alternates between 1) task proposal and data collection and 2) data evaluation to close the loop. For task proposal, we leverage semantic priors from vision-language models to identify relevant environments, object configurations, and goals for data collection. However, since semantic relevance does not imply policy improvement, we perform data evaluation. By leveraging predictive data attribution we approximate how the collected trajectories influence policy performance on a target deployment setting. This signal is then used as feedback to guide subsequent data collection, closing the data collection loop. We evaluate our approach in simulation and real robot settings, demonstrating that guided data collection substantially improves sample efficiency and deployment performance compared to passive data collection strategies, multiple baselines and experienced roboticists. More information at https://robin-lab.cs.utexas.edu/guild2026/
Figures & tables
Figure 1: Overview of GUILD Pipeline. Given a limited-access deployment setting and a set of accessible data collection environments, GUILD uses VLM priors to propose candidate scenes, object placements and tasks for data collection. Collected trajectories are then evaluated using predictive data attribution to estimate their influence on deployment-task performance. These attribution scores are used as feedback to guide subsequent data collection and task proposals—closing the loop.
Algorithm 1 GUILD Pseudocode
Figure 2: LIBERO guided data collection. We generate data using various strategies and train policies across different points of the process. GUILD either matches or surpasses the next best baseline. For each method on each task, we train 3 different data collection seeds. For every evaluation, we train 5 policies, measure their performance over 100 rollouts and average their performances. The means and standard error are computed across the 3 seeds of data collection process.
Figure 3: Real world guided data collection. We evaluate the policies trained on the data generated by different strategies. GUILD consistently achieves the highest success under the same data collection budget, highlighting its efficacy in guiding data collection towards useful tasks for the deployment setting.
Figure 4: Human Comparison. Avg. performance on 3 LIBERO tasks of datasets collected via various methods and humans.
Figure 5: Real-world qualitative analysis. For each target task, we visualize the final allocation of collected trajectories across auxiliary tasks ( middle ), and tasks with the highest data allocation auxiliary tasks ( bottom ). Bar colors indicate the outer-loop iteration in which each auxiliary task was introduced.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Relationship between estimated utility and downstream policy performance. Each point corresponds to a dataset collected by one of the evaluated data-collection strategies. We plot the dataset’s mean task utility against the success rate of the policy trained on that dataset.
No. of
No. of
No. of Data
Outer-loop
No. of tasks
Batch size
No. of Initial
Training Envs
Target Demos
Collection Iter.
Schedule
Every Iter. ( m )
( b )
Task Demos
LIBERO
Book-Caddy
16
3
30
[2,4,8,16]
5
20
5
Soup-butter-tray
16
5
30
[2,4,8,16]
5
20
5
Bowl-ketchup-drawer
16
5
30
[2,4,8,16]
5
10
5
Real World
Appendix
Table 1: Task-specific hyperparameters for Guided Data Collection. Data collection iterations denote the total number of inner-loop collection iterations. Outer-loop schedule specifies the inner-loop iterations at which support expansion is performed (Appendix A.1 ). m denotes the number of new task parameters proposed during each outer-loop iteration, and initial task demos denotes the number of demonstrations collected when a new task is introduced. Batch size ( b ) specifies the number of demonstrations collected during each inner-loop iteration using attribution-guided task selection.
Table 2: Task-parameter spaces used across experimental settings. We summarize the components of the task parameter ϕ that may be varied by the task proposer in each setting. Scene selection and object configuration indicate whether these components are explicitly chosen during task generation, while the task specification describes how the manipulation objective is represented. The candidate-space column indicates whether proposals are restricted to a fixed task set or generated within a structured/open-ended space.
Figure 7: Data Composition of data collected via GUILD. We demonstrate the composition of the final datasets from each of the LIBERO tasks. Top. A visualization of the target task for which the data is collected. Bottom. Final data composition plot, where each bar shows the number of trajectories collected for a particular task.
Figure 8: Task-wise human-guided data collection comparison on LIBERO
Tasks
book-caddy
soup-butter-tray
bowl-ketchup-drawer
Average
GUILD
68.2±1.3
69.6±3.6
56.9±1.8
64.9
GUILD (soft-sampling)
66.1±0.5
63.2±2.3
54.7±4.5
61.3
GUILD (no-feedback)
65.5±1.1
55.3±3.9
56.9±1.7
59.2
GUILD (gemini)
67.2±2.0
52.2±1.5
63.8±0.6
61.1
Appendix
Table 3: Mean performance with standard error of policies trained on the final aggregated dataset by GUILD , GUILD (soft-sampling), GUILD (no-feedback) and GUILD (gemini) for simulation tasks. The evaluation methodology is the same as our main experiments described in Appendix B .
Figure 9: Stage-wise real world results. We perform 20 rollouts of a policy on a task and measure the progress of the policy at different stages. The success rates are absolute for each stage, counting it as a fail even if the policy fails in an earlier stage.
Hyperparameter
LIBERO
Real-World
Policy architecture
Octo-Small
Octo-Small
Warmup steps Nw
Adaptive
Adaptive
Metagradient steps Nm
Adaptive
Adaptive
Max step bound C
500
300
Batch size B
512
512
Optimizer
Adam
Adam
Appendix
Table 4: Hyperparameters for computing metagradients.
Learning from human demonstrations is a reliable way to teach robots new tasks, but the gains from each additional demonstration shrink as the policy improves. Continued improvement can instead come from supervised deployment, where an operator places the objects and intervenes when the policy fails. We ask how to maximize improvement from a fixed budget of supervised episodes on high-precision manipulation tasks with wide ranges of object placements. We observe that failures can concentrate in a small subset of initial states, so uniform collection spends much of the operator's time on states the policy already handles. Mulligan makes the initial-state distribution a decision, starting each round's episodes at observed failures and untried states. To further improve data efficiency, we augment interactive imitation learning with a value function trained on all data, including failures that imitation discards. Across three real-world tasks evaluated on 2,550 held-out, blinded episodes and two simulated tasks, Mulligan outperforms uniform initial-state sampling at matched collection budgets, and combined with value-based action selection, HiL-IDQL+Mulligan, improves final real-task success by 10-34 percentage points. With operator interventions, the human-robot team completes 98% of collection episodes, remaining productive while the policy learns. Videos, code, and data are available at https://mulligan.page/.
Modern robot learning systems feed on high-quality demonstration data, and their performance hinges on coverage. Without good coverage, small errors compound and the policy can quickly go into out-of-distribution states from which it cannot recover. Yet there is no principled way of collecting high-coverage data, and data collection often relies on heuristic instructions to human operators. Purely online reinforcement learning (RL) explores at the cost of safety and sample efficiency, while DAgger-style interventions require the policy to fail by chance before a human can correct it. In this paper, we ask instead whether we can modify how the robot behaves during data collection to collect more useful data. We propose Data Inoculation (DI), a shared-control framework in which a human provide closed-loop teleoperation commands while a residual policy perturbs these commands. The residual policy is trained online, on the growing dataset, with the same reward-maximizing objective later used for policy extraction, so its perturbations steer data collection toward the deviations a policy trained on the data would make. These perturbations expose the robot to a wider range of states, while the human's continuous corrections provide recovery demonstrations, "inoculating" the dataset against the learner's future mistakes while the human remains in control. On three challenging real-world robotic tasks, DI consistently improves or matches the success rate of policies trained with both behavioral cloning and offline RL, showing DI as an effective data collection strategy for improving downstream policy performance.
Behavior cloning for robot manipulation relies on expert demonstrations. However, for tasks that require dynamic stability, precise contact timing, or dexterous coordination, human operators may find it hard or even impossible to collect data. We study this infeasible-demonstration regime and propose GLIDE: Guardrails for Learning from Infeasible Demonstrations Efficiently, a framework that infers task-specific failure modes and converts them into executable guardrails for data collection and policy deployment. Given a task description and the conditioning teleoperation code, GLIDE writes guardrails that use system states to filter teleoperation and policy commands, constrain failure-prone actions, and iteratively improve from trajectory feedback. Across three tasks, GLIDE discovers emergent guardrails that go beyond domain-expert hardcoded ones, improving data collection over naive VR teleoperation and domain-expert hardcoded guardrails. After refinement, GLIDE raises data-collection success from 0-10 percent to 70-90 percent across the three tasks. During policy execution, mixed-data guarded policies reach 70 percent, 60 percent, and 60 percent success on Tomato plate transfer, Marker handover and stand, and Wine serving tasks. These results show that GLIDE can support policy learning when direct demonstrations are infeasible. Project website: http://guardrail-policy.github.io/