Organizations: Shenzhen Key Laboratory of Robotics and Computer Vision, Southern University of Science and Technology, Shenzhen, China. · Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, China. · Department of Robotics, Perception, and Learning, KTH Royal Institute of Technology, Stockholm, Sweden.
Interactive Imitation Learning (IIL) typically relies on extensive human involvement for both offline demonstration and online interaction. Prior work primarily focuses on reducing human effort in passive monitoring rather than active operation. Notably, structured model-based imitation approaches achieve comparable performance with significantly fewer demonstrations than end-to-end imitation learning policies in the low-data regime. However, these methods are typically surpassed by end-to-end policies as the data increases. Leveraging this insight, we propose Easy-IIL, a framework that utilizes off-the-shelf model-based imitation methods as an assistant expert to replace active human operation for the majority of data collection. The human expert only provides a single demonstration to initialize the assistant expert and intervenes in critical states where the task is approaching failure. Furthermore, Easy-IIL can maintain IIL performance by preserving both offline and online data quality. Extensive simulation and real-world experiments demonstrate that Easy-IIL significantly reduces human operational burden while maintaining performance comparable to mainstream IIL baselines. User studies further confirm that Easy-IIL reduces subjective workload on the human expert.
Figures & tables
Fig. 1 : (a) illustrates the standard IIL interaction mechanism, where a human expert is involved in both offline and online stages. (b) depicts Easy-IIL, which delegates most offline and online data collection and interactions to an assistant expert, substantially reducing active human operational effort.
Fig. 2 : Data collection strategy in Easy-IIL for (a) the first one offline demonstration and (b) the remaining offline demonstrations (“Offline” mode) and online interaction data (“Online” mode). Specifically, aH=πH(x) , aR=πR(x,o) , aA=πA(o) , aN=πN(o) . x and o represent state and observation, respectively.
Fig. 3 : Visualization of an online interaction trajectory of Easy-IIL on the “Hang the Cup” task. The robot arm is controlled by (a) the novice policy; (b) the assistant expert, which yields a skewed grasping pose that risks failure; (c) the human expert, who intervenes and successfully picks up the cup; and (d) the assistant expert, which moves the arm normally toward the cup holder.
Fig. 4 : Four task scenarios. Basketball in Hoop (left-top), Take Chicken in Saucepan (right-top), Hang the Cup (left-bottom), and Put Duck in Cooker (right-bottom).
Fig. 5 : Main experiment results evaluating Succ. Rate (a) and Intv. Rate (b) across four tasks for Easy-IIL, HG-DAgger, IWR, and Sirius. Easy-IIL is configured with H=8 and σ=0.3 .
Fig. 6 : Ablation studies results evaluating (a) Succ. Rate and (b) Intv. Rate across two simulated tasks for online strategy of Easy-IIL. Easy-IIL is configured with H=8 and σ=0.3 .
Succ. Rate (%)
Intv. Rate (%)
Method
Basketball
Chicken
Basketball
Chicken
Best Baseline
74.4 ± 2.1
68.3 ± 5.3
100.0 ± 0.0
100.0 ± 0.0
Easy-IIL
77.8 ± 2.6
65.0 ± 4.7
4.6 ± 0.1
2.8 ± 0.2
TABLE I : Ablation study results evaluating the offline Succ. Rate and Intv. Rate across two simulated tasks for the best baseline and Easy-IIL. Results are presented as Mean ± Standard Deviation.
Succ. Rate (%)
H=4
H=8
H=12
Basketball
90.0 ± 3.1
90.0 ± 2.2
88.9 ± 1.9
Chicken
84.4 ± 3.5
86.1 ± 3.8
83.3 ± 5.1
TABLE II : Ablation study results evaluating Succ. Rate across two simulated tasks for parameter variations in the online strategy of Easy-IIL. Top: varying H (with σ=0.3 ); Bottom: varying σ (with H=8 ). Results are presented as Mean ± Standard Deviation.
Succ. Rate (%)
Assist. Expert
DP-30
DP-50
DP-80
Basketball
91.7 ± 2.1
90.0 ± 2.2
95.6 ± 2.5
97.8 ± 1.9
Chicken
88.9 ± 3.5
86.1 ± 3.8
90.6 ± 3.4
94.4 ± 3.9
TABLE III : The additional experiment evaluating Succ. Rate for assistant expert (Assist. Expert) and diffusion policy trained via Easy-IIL with 30 (DP-30), 50 (DP-50), and 80 demonstrations (DP-80). Results are presented as Mean ± Standard Deviation.
Succ. Rate (%)
Assist. Expert
π0.5 -1
π0.5 -5
Basketball
91.7±2.1
21.7 ± 5.1
59.4 ± 2.8
Chicken
88.9±3.5
14.4 ± 3.4
47.8 ± 7.2
TABLE IV : Success rates of the assistant expert (Assist. Expert) and π0.5 fully fine-tuned on one ( π0.5 -1) and five ( π0.5 -5) offline demonstrations. Results are presented as Mean ± Standard Deviation.
Raw TLX scores
Basketball
Chicken
Cup
Duck
Best Baseline
56.5±10.9
70.3±12.0
62.5±8.8
81.7±15.6
Easy-IIL
25.3±11.9
33.4±13.3
28.8±10.7
38.3±14.9
TABLE V : User study results evaluating Raw TLX scores for the best baseline and Easy-IIL. Raw TLX scores are presented as Mean ± Standard Deviation. ( p<0.01 , Wilcoxon Signed-Rank Test).
Imitation learning (IL)---training an agent to replicate expert behavior from demonstrations---underpins applications from robotics to language model training. Standard approaches such as Behavior Cloning (BC) are known to suffer from compounding errors and performance plateaus, particularly when the learner cannot perfectly represent the expert's policy (as is typical, e.g., in distillation). Two interventions are widely understood empirically to improve performance: querying the expert interactively along the learner's own trajectories, and using value function estimation en route to generating a policy rather than directly fitting the expert's full action distribution. We investigate the nature of these improvements and their potentially surprising interplay. Our main finding is that expert interaction relaxes the representational demands on the learner: one only needs a model capable of realizing the expert's value function, bypassing the (often stricter) requirement of realizing the expert's policy itself. Concretely, we introduce OVI, an interactive on-policy IL algorithm that is statistically efficient whenever the learner can represent the expert's value function and computationally efficient given access to a linear maximization oracle. We complement this with a negative result showing that interaction is necessary. Namely, without stronger assumptions beyond expert-value realizability alone, any offline IL algorithm must scale with the complexity of the expert policy class. Our findings bear out empirically. OVI outperforms offline policy-based (BC), interactive policy-based (DAgger), and offline value-based IL methods, with the largest gains when the learner network is substantially less expressive than the expert's.
A robot that can be taught a new task from a handful of demonstrations has to work out for itself what it still cannot do, and then ask for exactly that. Interactive imitation learning takes a step in that direction by letting a policy practice on its own and calling an expert when it goes wrong. Existing methods decide when to interrupt the learner. A further 2 decisions are left to whichever episode happened to trigger the interruption: which failure to correct, and where the demonstration should start. This paper is a first attempt at making both of them deliberately. DISEIL (Demonstration dIstillation for Sample-Efficient Imitation Learning) marks each failed episode at the step where the policy first becomes unreliable, represents that moment with a geometric descriptor, and groups the failures into recurring failure modes. A vision-language model and a language model read the selected mode and write a request for the next demonstration, and a store of task constraints checks that the request can be carried out before any expert time is spent. No model produces a robot action. Across 5 simulated tasks under state and image observations, changing only what the expert is asked for gives the highest mean held-out success rate in all 10 settings, with a tie in 1, and the margin is widest at the smallest budget we tested. The scope is narrow: a single round of practice at a time, in simulation, with experts that are mostly scripted. The longer-term aim is a learner that also tracks what its demonstration set already covers, and that asks a human teacher for the missing behavior in proportion to the effort each request costs them.
Suyog Khanal, Arun Kumar A, Santu Rana
Deakin Applied Artificial Intelligence Initiative Geelong, Australia
Imitation Learning (IL) enables agents to mimic expert behavior by learning from demonstrations. However, traditional IL methods require large amounts of medium-to-high-quality demonstrations as well as actions of expert demonstrations, both of which are often unavailable. To reduce this need, we propose Latent Wasserstein Adversarial Imitation Learning (LWAIL), a novel adversarial imitation learning framework that focuses on state-only distribution matching. It benefits from the Wasserstein distance computed in a dynamics-aware latent space. This dynamics-aware latent space differs from prior work and is obtained via a pre-training stage, where we train the Intention Conditioned Value Function (ICVF) to capture a dynamics-aware structure of the state space using a small set of randomly generated state-only data. We show that this enhances the policy's understanding of state transitions, enabling the learning process to use only one or a few state-only expert episodes to achieve expert-level performance. Through experiments on multiple MuJoCo environments, we demonstrate that our method outperforms prior Wasserstein-based IL methods and prior adversarial IL methods, achieving better results across various tasks.