CubeDAgger: Interactive Imitation Learning for Dynamic Systems with Efficient yet Low-risk Interaction

Taisuke Kobayashi

cs.ROMay 8, 2025

CubeDAgger: Interactive Imitation Learning for Dynamic Systems with Efficient yet Low-risk Interaction

Authors: Taisuke Kobayashi

Organizations: T. Kobayashi is with the National Institute of Informatics (NII) and with The Graduate University for Advanced Studies (SOKENDAI), 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo, 101-8430, Japan

Abstract

Interactive imitation learning makes an agent's control policy robust by stepwise supervisions from an expert. The recent algorithms mostly employ expert-agent switching systems to reduce the expert's burden by limitedly selecting the supervision timing. However, this approach is useful only for static tasks; in dynamic tasks, timing discrepancies cause abrupt changes in actions, losing the robot's dynamic stability. This paper therefore proposes a novel method, named CubeDAgger, which improves robustness with less dynamic stability violations even for dynamic tasks. The proposed method is designed on a baseline, EnsembleDAgger, with three improvements. The first adds a regularization to explicitly activate the threshold for deciding the supervision timing. The second transforms the expert-agent switching system to an optimal consensus system of multiple action candidates. Third, autoregressive colored noise is injected to the agent's actions for time-consistent exploration. These improvements are verified by simulations, showing that the trained policies are sufficiently robust while maintaining dynamic stability during interaction. Finally, real-robot scooping experiments with a human expert demonstrate that the proposed method can learn robust policies from scratch based on just 30 minutes of interaction. https://youtu.be/kBl3SCTnVEM

Figures & tables

Fig. 1 : Scooping three balls by a net attached on a quadruped

Panel of Figure 2 — Fig. 2 : Geometries for designing $p_{i}$ and $w_{k}$

Fig. 3 : Time consistency of red noise (

\Delta t=0.05

and

T=3

)

Fig. 4 : Trajectories during data collection (dashed lines: the experts’ average scores)

Fig. 5 : Statistical evaluation with 21 random seeds for each condition (dashed lines: the experts’ worst-case scores)

Fig. 6 : Evaluation of real-world scooping task

Safe Imitation Learning

See all 8 papers

Reward-DAgger: Robot-Gated Interactive Imitation Learning with General-Purpose Progress-Based Reward Models
Oct 2, 2026Ryan Li, Yigit Korkmaz, Erdem BıyıkRobot Imitation Learning Interactive Imitation Learning
Department of Mathematics, University of Southern California · Thomas Lord Department of Computer Science, University of Southern California
Policy-Calibrated DAgger: Offline Calibrated Noise Injection for Imitation Learning
Sep 24, 2026Jenny Wang, George KantorRobot Imitation Learning Covariate Shift
Carnegie Mellon University, Pittsburgh PA, USA
TimelyDAgger: Timing-Aware Expert Querying for VLA Policy Improvement
Sep 27, 2026Zhixuan Zhao, Peiyan Li, Enhao Zhang +10Robot Policy Learning Robot Imitation Learning
Tsinghua University · SEEN·E Robotics · Imperial College London +3

Oct 2, 2026J/K move · Enter open · S save

Reward-DAgger: Robot-Gated Interactive Imitation Learning with General-Purpose Progress-Based Reward Models

Ryan Li, Yigit Korkmaz, Erdem Bıyık

Department of Mathematics, University of Southern California · Thomas Lord Department of Computer Science, University of Southern California

Robot Imitation Learning Interactive Imitation Learning

Open paper AskPDF arXiv

Recent advances in robot learning have enabled generalist control policies capable of completing a wide range of tasks. However, their performance degrades when deployed in unseen environments, making it critical to detect failures and teach recovery behaviors. Existing runtime monitoring methods often require task- and policy-specific training or hyperparameter tuning, limiting cross-task deployment and introducing additional overhead during iterative policy updates. We present Reward-DAgger, a robot-gated interactive imitation learning framework that uses dense progress signals from a general-purpose reward model to determine when human intervention is needed. Our approach is agnostic to the underlying policy architecture, requires no access to policy internals, and can be applied across tasks without retuning the gating mechanism. Our results show that Reward-DAgger achieves a better failure-detection accuracy-latency tradeoff than existing runtime monitoring baselines. Across eight simulated and real-world tasks, Reward-DAgger consistently improves the downstream policy's autonomous success rate throughout interactive learning and achieves strong return on human effort, outperforming the baselines in most settings. Importantly, the same gating configuration is used across tasks without task-specific hyperparameter tuning, demonstrating transfer across tasks, environments, and policy architectures. Code and videos are available at https://liralab.usc.edu/reward-dagger.

Abstract

Safe Imitation Learning

Explore similar work

Reward-DAgger: Robot-Gated Interactive Imitation Learning with General-Purpose Progress-Based Reward Models

Policy-Calibrated DAgger: Offline Calibrated Noise Injection for Imitation Learning

TimelyDAgger: Timing-Aware Expert Querying for VLA Policy Improvement