BlenDAgger: Blended Shared Control for Interactive Imitation Learning

Cailyn Smith; Geoffrey Sun; Henny Admoni; Zackory Erickson

cs.ROSep 29, 2026

BlenDAgger: Blended Shared Control for Interactive Imitation Learning

Authors: Cailyn Smith, Geoffrey Sun, Henny Admoni, Zackory Erickson

Organizations: The Robotics Institute, School of Computer Science Carnegie Mellon University Pittsburgh, USA · School of Computer Science Carnegie Mellon University Pittsburgh, USA

Abstract

Robot policies are frequently trained from human corrections, yet teleoperating a robot to provide corrections is burdensome, and human demonstrators are not always optimal. We propose Blended DAgger (BlenDAgger), an approach for collecting data to train imitation learning policies by using shared control to blend the policy's and demonstrator's actions during interventions. By blending human and policy actions, we aim to improve the autonomous performance of manipulation policies. We validate our approach across five manipulation tasks, two in the real world and three in simulation. Our approach achieves higher autonomous performance by 30 or more percentage points on two real-world tasks compared to a typical human-gated correction approach (HG-DAgger). We also investigate the advantages of BlenDAgger that allow for higher autonomous performance, finding that BlenDAgger results in 57% smoother transitions between policy control and human interventions, and 14% higher trajectory similarity to the training data. In a user study (n=14) on two real-world tasks, we find that BlenDAgger results in faster data collection (BF=13.32), and we do not find a difference in subjective perceptions. These results show that blended shared control leads to higher autonomous performance compared to typical methods for fine-tuning robot policies from fully teleoperated interventions.

Figures & tables

Fig. 1 : BlenDAgger shares control between the robot’s policy and the human demonstrator during corrections, resulting in higher autonomous success and faster data collection.

Fig. 2 : BlenDAgger pipeline. When the demonstrator chooses to intervene, their action is blended with the policy’s predicted action. The resulting blended action is executed on the robot and used to fine-tune the policy.

Panel of Figure 3 — Fig. 3 : Experiment Tasks.

Fig. 4 : $\alpha$ over an example 30-second segment. Highlighted sections show timesteps where the participant is intervening.

Panel of Figure 5 — Fig. 5 : Autonomous performance on real-world (a–b) and simulation (c–e) tasks. Bar colors indicate the highest subtask reached during an episode. Full task success is labeled.

	Simulation	Real
Architecture
External / wrist cameras	2 / 1	1 / 1
Image resolution	$128\times 128$
Random crop	$116\times 116$	$112\times 112$
Proprioception dim.	16	14
Action dim.	12	10

TABLE I : Policy architecture and training hyperparameters.

	Object			Robot base		RoboCasa
Task	Object	$x$ / $y$ Pos. (cm)	Yaw (rad)	$x$ / $y$ Pos. (cm)	Yaw (rad)	Layout	Style
Prepare Coffee	Mug	$\pm$ 10 / 10	$0$	$\pm$ 2.5 / 4	$\pm 0.17$	7	9
Microwave Thawing	Carrot	$\pm$ 5 / 5	$0$	$\pm$ 1.5 / 4	$\pm 0.17$	4	0
Pick and Place	Lemon	$\pm$ 10 / 10	$2\pi$	$0$	$0$	1	1
Cupboard Stowing	Can	$\pm$ 11 / 11	$2\pi$	$0$	$0$	—	—
	Cupboard	$0$ / $0$	$\pi/6$	$0$	$0$	—	—

TABLE II : Randomization for data collection and evaluation.

Panel of Figure 8 — Fig. 6 : Participants complete the task faster and with fewer intervention steps with BlenDAgger. Error bars show 95% CIs.

Panel of Figure 9 — Fig. 7 : Autonomous performance (user study data).

Panel of Figure 10 — Fig. 8 : BlenDAgger has smoother action transitions and lower trajectory distance to training data. Error bars show 95% CIs.

Metric	Experts	Participants
Action discontinuity, intervention start/end	$6.2\mathrm{e}{7}$ / $1.5\mathrm{e}{6}$	$15.34$ / $3.8\mathrm{e}{36}$
Highly OOD states visited	$0.34$	$17.07$
Trajectory distance to training data	$3.35$	$3.10$
Intervention timing, intervention start/end	$0.51$ / $0.53$	$0.99$ / $1.66$

TABLE III : BFs comparing BlenDAgger to HG-DAgger on analysis metrics. Bold indicates evidence of an effect.

	Performance			Responsive-
Method	R1	R2	R3	ness (s)
HG-DAgger	0.01 ± 0.00	0.02 ± 0.01	0.05 ± 0.01	0.15
BlenDAgger	0.08 ± 0.03	0.15 ± 0.02	0.27 ± 0.04	0.20
Without uncertainty	0.00	0.14	0.16	0.28
Without similarity	0.01	0.15	0.22	0.20
Without magnitude	0.03	0.13	0.17	0.20

TABLE IV : Ablation performance and responsiveness. Bold marks the best overall and the best BlenDAgger variant.

Explore similar work

VR-DAgger: Immersive VR for Dexterous Data Collection and Uncertainty-Guided On-Policy Correction
May 26, 2026René Zurbrügg, Tifanny Portela, Arjun Bhardwaj +3Teleoperation Behavior Cloning
The Authors are with the Robotics Systems Lab ETH Zürich.
Instrumentation for Imitation Learning: Enhancing Training Datasets for Clothes Hanger Insertion
May 22, 2026Remko Proesmans, Thomas Lips, Francis wyffelsImitation Learning Robotic Manipulation
AI and Robotics Lab (IDLab-AIRO), Ghent University—imec, Ghent, Belgium
Learning Beyond What Humans Can Demonstrate
Sep 21, 2026Yuchen Song, Aditya Mittal, Unnat JainTeleoperation Behavior Cloning
University of California, Irvine

May 26, 2026, cs.ROJ/K move · Enter open · S save

VR-DAgger: Immersive VR for Dexterous Data Collection and Uncertainty-Guided On-Policy Correction

René Zurbrügg, Tifanny Portela, Arjun Bhardwaj, Aravind Elanjimattathil Vijayan+2

The Authors are with the Robotics Systems Lab ETH Zürich.

Teleoperation Behavior Cloning Human-In-The-Loop Virtual Reality On-Policy

Open paper AskPDF arXiv

Learning from demonstrations is effective for robotic manipulation, but collecting sufficient task-specific data remains a major bottleneck. Under distribution shift, small errors compound, performance degrades, and expert time is often spent on redundant, low-value corrections instead of the few critical failure cases. We present VR-DAgger, a human-in-the-loop framework centered on an immersive VR application for dexterous teleoperation, demonstration collection, and selective policy correction. The VR client provides intuitive hand control with synchronized scene visualization, while a backend workstation runs simulation and learning, enabling autonomous rollouts without continuous operator oversight. We use Monte Carlo (MC) dropout to score uncertainty during Isaac Lab rollouts of a diffusion policy and select informative failure segments for correction. These segments are replayed in VR as clips, where the operator selectively labels and corrects the policy's behavior, concentrating supervision where uncertainty is highest without full-rollout monitoring or a separate intervention classifier. We evaluate on three dexterous manipulation tasks (Pan pick-and-place, Drawer opening, Valve turning) with a 10-DoF XHand under standard and challenging initial configurations. Active labeling consistently improves over behavioral cloning across all tasks, with gains of up to 23 percentage points. Compared to unguided human-in-the-loop inspection, VR-DAgger reduces per-sample collection time by approximately 40% by focusing review on selected segments rather than full rollouts.

Abstract

Explore similar work

VR-DAgger: Immersive VR for Dexterous Data Collection and Uncertainty-Guided On-Policy Correction

Instrumentation for Imitation Learning: Enhancing Training Datasets for Clothes Hanger Insertion

Learning Beyond What Humans Can Demonstrate