TimelyDAgger: Timing-Aware Expert Querying for VLA Policy Improvement

Zhixuan Zhao; Peiyan Li; Enhao Zhang; Yueran Tao; Hao Wang; Chenghao Yue; Lei Lv; Wentao Zhao

cs.ROSep 27, 2026

TimelyDAgger: Timing-Aware Expert Querying for VLA Policy Improvement

Authors: Zhixuan Zhao, Peiyan Li, Enhao Zhang, Yueran Tao, Hao Wang, Chenghao Yue, Lei Lv, Wentao Zhao, +5 more

Organizations: Tsinghua University · SEEN·E Robotics · Imperial College London · Dalian University of Technology · Tongji University · Peking University

Abstract

DAgger improves robot policies by aggregating expert supervision from states visited during policy execution. Robot-gated DAgger automates expert queries, allowing the robot to decide when to request expert takeover. While existing gates emphasize detecting the need for assistance, takeover timing also shapes the content of these demonstrations and their value for policy learning. We propose TimelyDAgger, combining Bridge-PCA monitoring of internal vision-language-action (VLA) features with Feedback-guided Threshold Adaptation based on expert behavior to improve takeover timing. We introduce an evaluation framework linking failure detection, takeover timing, and policy improvement, including Target-Aligned Supervision Ratio (TASR) for assessing supervision quality without retraining. Experiments show that takeover timing affects policy learning, with TimelyDAgger achieving competitive failure detection and higher post-training success in most evaluated settings under matched expert-action budgets. Project website: https://seen-e.github.io/TimelyDagger/.

Figures & tables

Fig. 1: (A) Robot-gated DAgger. The robot requests expert assistance and uses the resulting demonstrations to update its policy. (B) Takeover timing affects policy improvement. Early takeover can collect redundant behavior, while late takeover may miss part of the behavior the policy needs to learn. Timely takeover concentrates supervision on the target behavior, supporting more effective policy learning.

Fig. 2: Overview of TimelyDAgger. (a) Offline construction: Bridge-PCA fits a principal subspace to ID bridge features and calibrates an initial takeover threshold. (b) Online collection: Bridge-PCA monitors policy execution and requests expert takeover, while FTA uses intervention feedback to adjust the threshold for better timing in subsequent episodes. (c) Policy update: The collected expert suffixes are combined with the original ID demonstrations to fine-tune the policy.

Fig. 3: Comparison of alarm timing across monitoring locations. A larger proportion of bridge alarms falls within the shaded preferred takeover window than is observed for input and action alarm, indicating closer alignment with the desired intervention timing.

Fig. 4: Bridge-feature geometry. Gray points denote ID reference features; blue and orange points denote features from successful and failed policy executions. The dark line illustrates the fitted PCA subspace in the visualization. The separation relative to this reference motivates the residual used by Bridge-PCA.

Fig. 5: ID-to-OOD task design. (a) Representative position, pose, and object variations across tasks. (b) Variations affecting different stages of OpenDrawer: handle position for opening, object pose for grasping, and target position for placement. Each pair contrasts the ID configuration with its corresponding OOD variation.

VLA	Task	AUPRC (%) $\uparrow$						TASR (%) $\uparrow$				Success Rate (%) $\uparrow$
VLA	Task	FID	CRS	Diff	ACC	STAC	Ours	BC	HG	Diff	Ours	BC	HG	Diff	Ours
$\pi_{0.5}$	SC–Green	96.74	94.67	80.88	72.11	63.82	99.72	49.24	24.22	22.20	87.82	49	29	34	81
	SC–Red	98.36	99.57	82.94	71.13	65.87	99.30	35.01	35.36	31.66	46.25	48	39	55	76
	OD–Handle	96.82	98.70	83.32	54.31	60.69	98.80	18.45	19.85	20.59	24.71	46	41	55	68
	OD–Place	96.66	98.30	87.79	68.77	63.94	97.73	17.46	18.89	26.41	28.50	43	37	52	64
	SP–Green	93.25	94.33	89.94	64.25	73.67	97.35	28.24	25.85	26.52	33.49	50	47	46	53

TABLE I: Simulation results across tasks and VLA backbones.

Fig. 6: Qualitative illustrations and recorded monitoring scores for OpenDrawer under (a) grasp-pose OOD and (b) target-position OOD. These examples illustrate the goal of requesting expert assistance before the task stage associated with the capability gap, so that the collected demonstrations better support learning the target behavior.

Fig. 7: Real-world task and an illustrative FTA case. Top: The task comprises cable pickup, handover, and precise connector insertion into the target board. Bottom: Expert correction following a late takeover provides feedback for FTA to lower the takeover threshold, enabling an earlier request for assistance in the next rollout.

Fig. 8: (a) Takeover Timing Study. Post-training success rate and TASR across six fixed takeover steps on OpenDrawer (Grasp-OOD) with matched retained expert-action budgets and training settings. (b) FTA Ablation. TASR and post-training success with and without FTA on StackCube–Green and OpenDrawer–Pose.

Fig. 9: Real-world ID and OOD configurations for cable insertion. The target board is fixed at its training position in ID and moved to two different positions in OOD1 and OOD2 for data collection and evaluation. The percentage at the upper right of each panel indicates the base policy’s success rate.

Fig. 10: Real-world success rates under two OOD conditions. Overall combines both conditions.

Explore similar work

Reward-DAgger: Robot-Gated Interactive Imitation Learning with General-Purpose Progress-Based Reward Models
Oct 2, 2026Ryan Li, Yigit Korkmaz, Erdem BıyıkRobot Imitation Learning Interactive Imitation Learning
Department of Mathematics, University of Southern California · Thomas Lord Department of Computer Science, University of Southern California
PathTime-VLA: Path-Time Decoupling for Factorized Post-Training of Vision-Language-Action Policies
Oct 8, 2026Qing Huang, Yifei Yang, Ziqing Zou +5Robot Policy Learning Vision-Language-Action Models
Zhejiang University
Visual Verification Enables Inference-time Steering and Autonomous Policy Improvement
Jun 16, 2026Mingtong Zhang, Dhruv ShahRobot Policy Learning Robot Policy Adaptation
Princeton University

Oct 2, 2026J/K move · Enter open · S save

Reward-DAgger: Robot-Gated Interactive Imitation Learning with General-Purpose Progress-Based Reward Models

Ryan Li, Yigit Korkmaz, Erdem Bıyık

Department of Mathematics, University of Southern California · Thomas Lord Department of Computer Science, University of Southern California

Robot Imitation Learning Interactive Imitation Learning

Open paper AskPDF arXiv

Recent advances in robot learning have enabled generalist control policies capable of completing a wide range of tasks. However, their performance degrades when deployed in unseen environments, making it critical to detect failures and teach recovery behaviors. Existing runtime monitoring methods often require task- and policy-specific training or hyperparameter tuning, limiting cross-task deployment and introducing additional overhead during iterative policy updates. We present Reward-DAgger, a robot-gated interactive imitation learning framework that uses dense progress signals from a general-purpose reward model to determine when human intervention is needed. Our approach is agnostic to the underlying policy architecture, requires no access to policy internals, and can be applied across tasks without retuning the gating mechanism. Our results show that Reward-DAgger achieves a better failure-detection accuracy-latency tradeoff than existing runtime monitoring baselines. Across eight simulated and real-world tasks, Reward-DAgger consistently improves the downstream policy's autonomous success rate throughout interactive learning and achieves strong return on human effort, outperforming the baselines in most settings. Importantly, the same gating configuration is used across tasks without task-specific hyperparameter tuning, demonstrating transfer across tasks, environments, and policy architectures. Code and videos are available at https://liralab.usc.edu/reward-dagger.

Abstract

Explore similar work

Reward-DAgger: Robot-Gated Interactive Imitation Learning with General-Purpose Progress-Based Reward Models

PathTime-VLA: Path-Time Decoupling for Factorized Post-Training of Vision-Language-Action Policies

Visual Verification Enables Inference-time Steering and Autonomous Policy Improvement