Programming robots to perform complex tasks is often difficult and time consuming, requiring expert knowledge and skills in robot software and sometimes hardware. Imitation learning is a method for training robots to perform tasks by leveraging human expertise through demonstrations. Typically, the assumption is that those demonstrations are performed by a single, highly competent expert. However, in many real-world applications that use user demonstrations for tasks or incorporate both user data and pretrained data, such as home robotics including assistive robots, this is unlikely to be the case. This paper presents research towards a system which can leverage suboptimal demonstrations to solve ambiguous tasks; and particularly learn from its own failures. This is a negative-feedback system which achieves significant improvement over purely positive imitation learning for ambiguous tasks, achieving a 90% improvement in success rate against a system that does not utilise negative feedback, compared to a 50% improvement in success rate when utilised on a real robot, as well as demonstrating higher efficacy, memory efficiency and time efficiency than a comparable negative feedback scheme. The novel scheme presented in this paper is validated through simulated and real-robot experiments.
Figures & tables
Fig. 1: An illustration of the ambiguity problem in a task avoiding an obstacle while navigating to a goal. The two blue lines represent demonstrations, both successes (neither collide with the obstacle), but the central green line, learned from a combination of the two, is a failure, as it averages the two behaviours and collides with the obstacle.
Fig. 2: An illustration of the mask for an obstacle avoidance task, with zeroes, regions where Nthreshold or greater trajectories pass through, shown in white and ones, where fewer than Nthreshold trajectoreis pass through, shown in blue, and the obstacle being avoided shown in red.
Fig. 3: An illustration of the simple ambiguous task tested here, with the two success modes demonstrated, over (left) and under (right)
Fig. 4: The Negative Feedback graph for the simple ambiguous task, showing the improvement in success rate with increasing cycles of negative feedback, comparing three different methods for applying negative feedback - a negative weighting method, a mixture of experts method, and finally the product of experts method presented in this paper
Fig. 5: The graph showing the impact of different methods of trajectory selection on the same five-cycle run of negative feedback described in 4 . Here, ’Mask 75%’ and ’Mask 50%’ refer to the impacts of using an Nthreshold of 75% of the trajectories and 50% of the trajectories respectively.
Fig. 6: The ’slalom’ variant of the task, which involves navigating around a more complex course, shown in the left image. The right image shows the gaps which the route can traverse, labeled from A to E and from A to D. Any combination of the first row and second row is a viable route through the slalom.
Fig. 7: The Negative Feedback graph for the slalom task, showing the success rate for the three different methods increasing with the number of feedback cycles.
Fig. 8: The Negative Feedback graph for four different masking methods on the slalom task. In this instance, only central trajectory and three different masking distributions - 25%, 50%, and 75% - were considered.
Fig. 9: An illustration of the same pick-and-place with obstacle task as used in the real robot experiments, with the Franka Emika Research 3 arm visible. The highlighted lines illustrate three possible expert behaviours, and then the combination of these, which due to the ambiguity of choice between the three behaviours results in a collision.
Fig. 10: The Negative Feedback graph for the real robot, showing the success rate per 10 runs with an increasing number of negative feedback cycles for the same three methods used in the Simulated experiments.
Fig. 11: The Negative Feedback graph for the efficiency comparison, showing the time taken to complete each step of negative feedback under the Product of Experts and Negative weighting methods, each step averaged over twenty runs.
Imitation learning for robotic tasks has relied primarily on policies trained only on successful demonstrations, although failures are unavoidable during human data collection. Many existing approaches for exploiting failure data require additional data processing or iterative policy updates through autonomous rollouts, making it difficult to directly and stably utilize failure data accumulated during data collection. In this work, we propose a method that learns latent representations of success-failure discrepancies and incorporates them into the attention mechanism. During inference, an appropriate latent mode is selected from the initial observation to improve action stability. Furthermore, we introduce a post-training metric that quantifies the attention discrepancy between each failure sample and successful demonstrations to select failure data. Simulation results show that the proposed method improves task success rates when trained with failure data and that the proposed metric identifies failure samples that are beneficial for learning when combined with successful demonstrations. These results suggest that the proposed method can support more efficient use of collected demonstrations in robotic data collection pipelines.
Kana Miyamoto, Kanata Suzuki, Tetsuya Ogata
Faculty of Science and Engineering, Waseda University, Tokyo, Japan · Physical AI Laboratory, Fujitsu Limited, Kanagawa, Japan
A robot that can be taught a new task from a handful of demonstrations has to work out for itself what it still cannot do, and then ask for exactly that. Interactive imitation learning takes a step in that direction by letting a policy practice on its own and calling an expert when it goes wrong. Existing methods decide when to interrupt the learner. A further 2 decisions are left to whichever episode happened to trigger the interruption: which failure to correct, and where the demonstration should start. This paper is a first attempt at making both of them deliberately. DISEIL (Demonstration dIstillation for Sample-Efficient Imitation Learning) marks each failed episode at the step where the policy first becomes unreliable, represents that moment with a geometric descriptor, and groups the failures into recurring failure modes. A vision-language model and a language model read the selected mode and write a request for the next demonstration, and a store of task constraints checks that the request can be carried out before any expert time is spent. No model produces a robot action. Across 5 simulated tasks under state and image observations, changing only what the expert is asked for gives the highest mean held-out success rate in all 10 settings, with a tie in 1, and the margin is widest at the smallest budget we tested. The scope is narrow: a single round of practice at a time, in simulation, with experts that are mostly scripted. The longer-term aim is a learner that also tracks what its demonstration set already covers, and that asks a human teacher for the missing behavior in proportion to the effort each request costs them.
Suyog Khanal, Arun Kumar A, Santu Rana
Deakin Applied Artificial Intelligence Initiative Geelong, Australia
Imitation learning offers a promising framework for enabling robots to acquire diverse skills from human users. However, most imitation learning algorithms assume access to high-quality demonstrations an unrealistic expectation when collecting data from non-expert users, whose demonstrations often contain inadvertent errors. Naively learning from such demonstrations can result in unsafe policy behavior, while discarding entire demonstrations due to occasional mistakes wastes valuable data, especially in low-data settings. In this work, we introduce GiB (Good-in-Bad), an algorithm that automatically identifies and discards erroneous subtasks within demonstrations while preserving high-quality subtasks. The filtered data can then be used by any policy learning algorithm to train more robust policies. GiB first trains a self-supervised model to learn latent features and assigns binary weights to label each demonstration as good or bad. It then models the latent feature distribution of high-quality segments and uses the Mahalanobis distance to detect and evaluate poor-quality subtasks. We validate GiB on the Franka robot in both simulated and real-world multi-step tasks, demonstrating improved policy performance when learning from mixed-quality human demonstrations.
Noushad Sojib, Ola Ghattas, Momotaz Begum
Department of Computer Science, University of New Hampshire, USA