Improving Policy Learning from Delayed Interventions

Xiru Zhu; David Meger; Gregory Dudek

cs.ROOct 12, 2026

Improving Policy Learning from Delayed Interventions

Authors: Xiru Zhu, David Meger, Gregory Dudek

Organizations: McGill University, Computer Science

Abstract

In this work, we explore the idea of expert intervention delays in human-supervised robotic systems for imitation learning system. Previous works in imitation learning have primarily examined the impact of suboptimal experts, where the expert policy makes mistakes, but assumed expert interventions occur immediately after errors are made. Although delayed expert intervention can be viewed as a form of noise that might improve policy robustness, our results show that learning from delayed expert interventions is overall less effective than learning from non-delayed expert interventions; delayed expert interventions are less effective at demonstrating error avoidance behaviors. To address this, we propose Learning from Delayed Intervention (LfDI), which learns a cost function based on where and when the expert intervenes, combined with limited information about the extent of intervention delays provided by the expert. This cost function is then used to infer additional supervision for states without expert data, guiding the policy to avoid states that would likely trigger interventions. We evaluate our method, Learning from Delayed Intervention (LfDI), on simulated environments and show that it outperforms other methods that do not account for intervention delays.

Figures & tables

Fig. 1: To evaluate our proposed solution, we experimented on several environments in the highway-env [ 6 ] . The Roundabout environment is configured such that the vehicle starts on the south route outside the roundabout and must enter the roundabout and exit to the north route to complete the task. In this figure above, the expert intervened and braked, preventing the controlled vehicle(yellow) from crashing into another vehicle.

Fig. 2: Consider a vehicle attempting to stay on the road. During data sampling, the current policy may make mistakes and gradually drift from the intended path. Depending on the intervention threshold $\alpha$ , the expert may intervene at different points. When $\alpha$ is low, the acceptable total error over the window $w$ is small, and the expert intervenes promptly after errors occur. In contrast, when the expert is distracted, their $\alpha$ is higher, and interventions are delayed.

Fig. 3: Learning from Delayed Interventions (LfDI) has two phases: data sampling and training. During sampling, expert interventions occur when the cost exceeds a threshold $\sigma$ , with higher thresholds allowing longer delays and tolerating more errors. After collecting trajectories, the policy is retrained with the new data. Finally, the cost function is trained, and the policy is retrained from scratch with Equation 8 .

Fig. 4: We evaluated our method on the Intersection environment from highway-env [ 6 ] . In this environment, the agent must drive north to the intersection and make an unprotected left turn. Due to the limited vehicle logic in highway-env, other vehicles can get stuck and crash into each other. Consequently, the overall success rate for even an expert is not perfect, making it critical for the agent to learn when to stop and when to turn safely.

Fig. 5: The Highway environment from highway-env [ 6 ] consists of a long 4-lane highway with a controlled vehicle. The objective is to travel as far (in terms of x-position) as possible within a time limit of 500 steps without crashing or leaving the road.

Fig. 6: We plot the average Highway environment termination rate for our method(LFDI) with and without noisy labels, EIL, noisy expert, and HG-Dagger. LFDI_noisy is a policy trained with LFDI and noisy and imprecise supervision-level labels.

Method	Dist / Task Comp. $\uparrow$	Term. $\downarrow$	Speed
Highway
EIL	$129.7\pm 9.0$	$1.00\pm 0.00$	$\mathbf{2.77\pm 0.11}$
LfDI	$\mathbf{927.7\pm 55.8}$	$\mathbf{0.34\pm 0.06}$	$2.23\pm 0.09$
HG-DAgger	$509.5\pm 47.7$	$0.85\pm 0.04$	$2.34\pm 0.10$
human-pref	$512.7\pm 32.1$	$0.83\pm 0.05$	$2.30\pm 0.16$
noisy-expert	$381.4\pm 70.0$	$0.84\pm 0.08$	$2.18\pm 0.32$

TABLE I: Results for three highway-env environments. Highway shows distance traveled[sim. units] where the distance is in simulated units, termination rate(lower is better), and speed[sim. units/s]. Roundabout and Intersection show task completion rate instead and termination rate.

Method	Distance $\uparrow$	Term. $\downarrow$	Speed
LfDI	$928.2\pm 55.6$	$0.34\pm 0.06$	$2.23\pm 0.09$
LfDI(noisy labels)	$705.7\pm 104.5$	$0.58\pm 0.10$	$2.20\pm 0.12$
LfDI(no unsup)	$832.9\pm 36.9$	$0.50\pm 0.07$	$2.29\pm 0.08$
LfDI(no noisy)	$380.9\pm 30.2$	$0.99\pm 0.00$	$3.01\pm 0.06$
LfDI(ensemble)	$970.0\pm 24.8$	$0.28\pm 0.04$	$2.22\pm 0.06$

TABLE II: Highway Ablation Study Results (Last 10 Steps). We provide a set of results of our method with different ablation settings. Note that distance and speed are in simulator units.

Fig. 7: We plot the termination rate (crash rate) of our method(LFDI) against other methods. We can see that our method shows improvements in reducing crash rates without decreasing completion rates, as shown in Table I .

High Delay Dataset
Method	Reward $\uparrow$	Term. $\downarrow$
EIL	$2433\pm 282$	$15.10\pm 3.18$
HG-DAgger	$3344\pm 324$	$6.68\pm 1.89$
LfDI	$\mathbf{3863\pm 310}$	$\mathbf{3.99\pm 1.34}$
human-pref	$3739\pm 324$	$4.36\pm 1.54$
noisy-exp	$3587\pm 353$	$5.56\pm 1.92$

TABLE III: Humanoid Environment Results (Last 10 Steps). Humanoid results (top) show all methods with limited expert interventions. For the bottom, we performed a simple experiment showing the performance of HG-DAgger with different levels of expert intervention delays.

Explore similar work

Does Imitation Learning Preserve Temporal Robustness in Dexterous Manipulation? An Expert-Learner Comparison Across Task Execution Speeds
Sep 1, 2026Clinton Enwerem, John S. Baras, Calin BeltaContact-Rich Robotic Manipulation Robotic Policy Evaluation
Institute for Systems Research University of Maryland College Park, MD, U.S.A.
GAINS: Leveraging Inconsistent Human Intervention Signals in Reinforcement Learning
Aug 16, 2026Xinyi Zhang, Yinuo Zhao, Pei Ren +7Robotic RL RL from Human Feedback
Beijing Institute of Technology · Beijing Innovation Center of Humanoid Robotics · City University of Hong Kong +1
TimelyDAgger: Timing-Aware Expert Querying for VLA Policy Improvement
Sep 27, 2026Zhixuan Zhao, Peiyan Li, Enhao Zhang +10Robot Policy Learning Robot Imitation Learning
Tsinghua University · SEEN·E Robotics · Imperial College London +3

Sep 1, 2026J/K move · Enter open · S save

Does Imitation Learning Preserve Temporal Robustness in Dexterous Manipulation? An Expert-Learner Comparison Across Task Execution Speeds

Clinton Enwerem, John S. Baras, Calin Belta

Institute for Systems Research University of Maryland College Park, MD, U.S.A.

Contact-Rich Robotic Manipulation Robotic Policy Evaluation Robotic Manipulation Robot Imitation Learning

Open paper AskPDF arXiv

Dexterous manipulation policies learned by imitation are typically evaluated for robustness to variation in scenes, objects, or instructions, but their performance across task execution speeds is less often examined. This leaves open how much temporal robustness a learner retains relative to the expert it imitates. We compare an expert and learner under the same task conditions, initial-condition draws, and speedup factors. We instantiate the evaluation in ParcelStow, a contact-rich task in which the robot acquires, reorients, and inserts a parcel. The demonstrations span the speedup range for the manipulation phases after parcel acquisition. A scripted expert and an Action Chunking with Transformers (ACT) policy trained from the expert's demonstrations both achieve 100 percent task success at nominal speed. Their success rates diverge within the demonstrated range: at its maximum, expert success is 84 percent and ACT success is 53 percent. Two ACT policies with different parameter initializations show similar degradation, decreasing by 34 and 48 percentage points from nominal speed to the maximum demonstrated speed, compared with 16 points for the expert. Stage-level analysis shows that 35 of ACT's 47 failures at the maximum demonstrated speed are insertion misalignments. Under the relative-motion handoff, every ACT acquisition retains the parcel through reorientation and transfer in free space, but only 64 percent complete the overall task, compared with 95 percent after expert acquisition. Across all evaluated policies and speeds, none of the 414 acquisitions without force closure completes the task. Equal nominal task success therefore does not imply preservation of expert performance across execution speeds. Code, data, and evaluation scripts are available at https://github.com/coenwerem/parcelstow.

Abstract

Explore similar work

Does Imitation Learning Preserve Temporal Robustness in Dexterous Manipulation? An Expert-Learner Comparison Across Task Execution Speeds

GAINS: Leveraging Inconsistent Human Intervention Signals in Reinforcement Learning

TimelyDAgger: Timing-Aware Expert Querying for VLA Policy Improvement