TILDE: Teacher-in-the-Loop Distillation from Embodied Foundation Models

Yuxiang Lu; Yantong Liu; Jinghua Hou; Xueli Sun; Ruihua Han; Zhe Liu; Hengshuang Zhao

cs.ROOct 18, 2026

TILDE: Teacher-in-the-Loop Distillation from Embodied Foundation Models

Authors: Yuxiang Lu, Yantong Liu, Jinghua Hou, Xueli Sun, Ruihua Han, Zhe Liu, Hengshuang Zhao

Organizations: The University of Hong Kong

Abstract

Large-scale action pretraining equips Embodied Foundation Models (EFMs) with strong manipulation capabilities, with particular strengths in generalization and closed-loop robustness, but reproducing such pretraining requires substantial robot data and computation. To transfer these capabilities from pretrained EFMs to policies without action pretraining, we introduce TILDE (Teacher-In-the-Loop Distillation from EFMs), which synergizes on-policy distillation at student-induced states with learning from teacher-executed recovery trajectories. We further propose AutoDAgger, an autonomous data aggregation pipeline in which a robotic reward model is utilized to monitor task progress and trigger teacher intervention, thereby collecting recovery experience without human monitoring or manual control. AutoDAgger accelerates recovery-trajectory collection by more than $2\times$ compared with the human-in-the-loop method. By unifying learning from demonstrations, on-policy teacher supervision, and teacher's corrective behavior, TILDE improves task performance by up to $4.1\times$ over behavior cloning across three real-world tasks and supports effective cross-architecture distillation.

Figures & tables

Figure 1: Left: Policies with ( $\pi_{0.5}$ & X-VLA) and without large-scale action pretraining (“w/o PT” & VLA-Adapter-Pro) perform similarly on standard benchmarks (LIBERO-Spatial & Object), while larger gaps emerge on long-horizon tasks (LIBERO-10), under perturbations (LIBERO-Plus), and on the real-world Pick Beverage task. These results motivate transferring capabilities from pretrained EFMs to student policies without repeating action pretraining. Right: (a) Offline distillation is restricted to the demonstration distribution, leading to distribution mismatch during student execution. (b) On-Policy Distillation extends supervision to student-induced states but does not execute teacher corrections to observe the subsequent recovery states. (c) Human-in-the-loop methods collect recovery trajectories through human monitoring and intervention, but their coverage is limited by the bandwidth of human supervision. (d) TILDE jointly learns from demonstration, student-induced rollout, and failure recovery distributions, with AutoDAgger automating progress monitoring and teacher intervention for efficient data collection.

Figure 2: AutoDAgger pipeline. A reward-model-based task monitor estimates task progress and success probability. The execution manager uses these signals to coordinate policy execution, trigger teacher intervention upon detected student failure, and select successful recovery episodes for data aggregation. Pre-intervention student rollout and successful teacher recovery are recorded without human monitoring or takeover. Bottom: A real-world Fold Dishcloth example; green/blue denote student/teacher execution, while red marks the detected failure and intervention point.

Figure 3: Comparison of real-world task performance. TILDE is compared with behavior cloning (BC), offline distillation, and on-policy distillation (OPD), with the teacher policy serving as a reference. TILDE outperforms all student baselines on all three tasks.

Figure 4: Ablation study of training data and supervision. Left: Starting from offline distillation, we add teacher supervision at student-rollout data (+OPD), supervision on teacher recovery trajectories (+Recovery Distill), or both (TILDE). Combining both sources gives the highest scores on all three tasks. Right: Recovery BC augments demonstration BC with imitation of recorded recovery actions. Recovery Distill additionally incorporates distillation on both demonstration and recovery samples, further improving performance.

Table 5

Data Source	Distill	Fold Dishcloth		Pick Pen		Pick Beverage
Data Source	Distill	#Episodes	Score	#Episodes	Score	#Episodes	Score
Human Demo	✗	200	20.8	100	37.0	100	16.7
+Extra Demo	✗	230	27.8 (+ 7.0)	130	45.4 (+ 8.4)	130	44.2 (+ 27.5)
+AutoDAgger	✗	230	48.6 (+ 27.8)	130	63.0 (+ 26.0)	130	50.0 (+ 33.3)
Human Demo	✓	200	27.8	100	45.4	100	26.7
+Extra Demo	✓	230	29.2 (+ 1.4)	130	54.6 (+ 9.2)	130	48.3 (+ 21.6)

Table 3: Analysis of the effectiveness of additional human demonstrations and AutoDAgger data. “Distill” indicates the use of teacher action targets, “#Episodes” denotes the total number of training episodes, and parenthesized values indicate absolute score improvements over Human Demo under the same distillation setting. With the same number of 30 additional episodes, AutoDAgger yields substantially larger performance gains.

Method	Distill	Fold Dishcloth			Pick Pen			Pick Beverage
Method	Distill	Score	Rec. Time	Rec. SR(%)	Score	Rec. Time	Rec. SR(%)	Score	Rec. Time	Rec. SR(%)
HG-DAgger	✗	34.7	54.4s	76.7	75.9	29.4s	76.7	60.0	32.2s	90.0
HG-DAgger	✓	65.3	54.4s	76.7	80.6	29.4s	76.7	63.3	32.2s	90.0
AutoDAgger	✗	48.6	21.5s	73.3	63.0	13.2s	66.7	50.0	12.4s	90.0
AutoDAgger	✓	63.9	(2.53 $\times$ )	73.3	80.6	(2.23 $\times$ )	66.7	69.2	(2.60 $\times$ )	90.0

Table 4: Comparison of HG-DAgger and AutoDAgger in policy performance and recovery-data collection. “Rec. Time” denotes the time required by the human operator or teacher policy to complete a recovery after takeover, while “Rec. SR” denotes the fraction of interventions that successfully recover and complete the task. Both metrics are evaluated over 30 extra trials. Parenthesized values indicate the recovery-time speedup of AutoDAgger over HG-DAgger.

Table 8

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Task	$T_{\mathrm{fail}}$ (s)		$T_{\mathrm{succ}}$ (s)
Task	Student	Teacher	Student	Teacher
Fold Dishcloth	6	14	5	5
Pick Pen	3	8	4	4
Pick Beverage	2	6	4	4

Appendix

Table 7: Task- and policy-specific temporal thresholds used during AutoDAgger data collection.

T_{\mathrm{fail}}

determines the failure-detection timeout, while

T_{\mathrm{succ}}

determines the duration of continuous success predictions required to confirm task success.

Figure 6: Visualization of three real-world tasks.

Model	$\pi_{0.5}$		X-VLA
Task	AgileX	LIBERO	AgileX	LIBERO
Prediction Horizon $H$	50	10	30	30
Action Space	Relative Joint	Delta EEF	Abs Joint	Abs EE6D
Global Batch Size	128	256	128	128
Training Steps	50k	30k	50k	60k
Optimizer	AdamW	AdamW	AdamW	AdamW

Appendix

Table 8: Hyperparameter settings for training

\pi_{0.5}

and X-VLA policies.

Model	VLA-Adapter-Pro		SmolVLA
Task	AgileX	LIBERO	LIBERO
Prediction Horizon $H$	30	8	10
Action Space	Abs Joint	Delta EEF	Delta EEF
Global Batch Size	128	64	64
Training Steps	50k	100k	100k
Optimizer	AdamW	AdamW	AdamW

Appendix

Table 9: Hyperparameter settings for training VLA-Adapter-Pro and SmolVLA policies.

Method	$\mathcal{D}_{\mathrm{demo}}$	$\mathcal{D}_{\mathrm{roll}}$	$\mathcal{D}_{\mathrm{rec}}$
BC Baseline	BC	–	–
Offline Distill	BC+KD	–	–
OPD	BC+KD	KD	–
Recovery BC	BC	–	BC
Recovery Distill	BC+KD	–	BC+KD
TILDE	BC+KD	KD	BC+KD

Appendix

Table 10: Comparison of different training configurations.

Prediction	Ground-truth outcome		Total
Prediction	True	False	Total
Positive	8 (TP)	1 (FP)	9
Negative	2 (FN)	9 (TN)	11
Total	10	10	20

Appendix

Table 11: Rollout outcome prediction results on real-world tasks.

Method	Fold Dishcloth		Pick Pen		Pick Beverage
Method	Mean	96% CI	Mean	96% CI	Mean	96% CI
Teacher	87.5	(69.4, 98.6)	87.0	(78.7, 94.4)	79.2	(65.0, 91.7)
BC Baseline	20.8	(2.8, 41.7)	37.0	(22.2, 51.9)	16.7	(8.3, 25.8)
Offline Distill	27.8	(6.9, 51.4)	45.4	(31.5, 60.2)	26.7	(14.2, 40.0)
OPD	43.1	(16.7, 69.4)	58.3	(45.4, 70.4)	40.0	(25.0, 55.0)
Recovery BC	48.6	(22.2, 75.0)	63.0	(48.1, 76.9)	50.0	(32.5, 67.5)

Appendix

Table 12: Performance statistics for the real-world tasks, corresponding to Figure 3 and Figure 4 . The 96% confidence intervals are computed using bootstrap resampling.

Data Source	Distill	Fold Dishcloth		Pick Pen		Pick Beverage
Data Source	Distill	Mean	96% CI	Mean	96% CI	Mean	96% CI
Human Demo	✗	20.8	(2.8, 41.7)	37.0	(22.2, 51.9)	16.7	(8.3, 25.8)
+Extra Demo	✗	27.8	(6.9, 51.4)	45.4	(27.8, 63.0)	44.2	(28.3, 60.0)
+AutoDAgger	✗	48.6	(22.2, 75.0)	63.0	(48.1, 76.9)	50.0	(32.5, 67.5)
Human Demo	✓	27.8	(6.9, 51.4)	45.4	(31.5, 60.2)	26.7	(14.2, 40.0)
+Extra Demo	✓	29.2	(9.7, 51.4)	54.6	(40.7, 68.5)	48.3	(33.3, 63.3)

Appendix

Table 13: Performance statistics for the real-world tasks corresponding to Table 3 .

Method	Distill	Fold Dishcloth		Pick Pen		Pick Beverage
Method	Distill	Mean	96% CI	Mean	96% CI	Mean	96% CI
HG-DAgger	✗	34.7	(11.1, 59.7)	75.9	(60.2, 89.8)	60.0	(46.7, 73.3)
HG-DAgger	✓	65.3	(33.3, 90.3)	80.6	(68.5, 90.7)	63.3	(48.3, 77.5)
AutoDAgger	✗	48.6	(22.2, 75.0)	63.0	(48.1, 76.9)	50.0	(32.5, 67.5)
AutoDAgger	✓	63.9	(38.9, 87.5)	80.6	(72.2, 88.0)	69.2	(53.3, 83.3)

Appendix

Table 14: Performance statistics for the real-world tasks corresponding to Table 4 .

Task	HG-DAgger			AutoDAgger
Task	Rec. Time	Rec. SR (%)	96% CI	Rec. Time	Rec. SR (%)	96% CI
Fold Dishcloth	$54.4\pm 10.9$ s	76.7	(60.0, 90.0)	$21.5\pm 5.8$ s	73.3	(56.7, 90.0)
Pick Pen	$29.4\pm 2.4$ s	76.7	(60.0, 90.0)	$13.2\pm 1.4$ s	66.7	(50.0, 83.3)
Pick Beverage	$32.2\pm 2.8$ s	90.0	(76.7, 100.0)	$12.4\pm 1.8$ s	90.0	(76.7, 100.0)

Appendix

Table 15: Recovery data collection statistics for the real-world tasks corresponding to Table 4 . Rec. Time is reported as mean

\pm

standard deviation from successful recovery.

Panel of Figure 7 — Figure 7: Representative AutoDAgger collection trajectories. Examples for Fold Dishcloth, Pick Pen, and Pick Beverage are shown from top to bottom. Green, red, and blue frame borders denote student execution, detected failure, and teacher recovery, respectively. The red dashed line marks the success threshold $\theta_{s}=0.5$ .

Knowledge Distillation

Knowledge distillation transfers useful behavior or representations from larger or specialized models into more compact or adaptable learners.

See all 814 papers

CAPEX: Efficiently Distilling Foundation Model Behavior into Deployable Robot Policies through Experience-Adaptive Reasoning
Sep 26, 2026Shivam Aarya, Zhang Xi-Jia, Chengyue Huang +6Robot Demonstration Generation Synthetic Data Generation for Robotics
School of Interactive Computing Georgia Institute of Technology Atlanta, GA 30332, USA
Data Pyramid for Embodied Manipulation: A Survey
Jul 27, 2026Yifan Ye, Yankai Fu, Yaoxu Lv +26Robotic Data Collection Robot Foundation Models
PKU · NTU · HKUST +8
WAM-OPD: Sharpening World Action Models via On-Policy Distillation
Sep 28, 2026Panjun Liu, Xiaohan Lei, Shiqi Zhang +8Robot Policy Learning Robot Policy Adaptation
University of Science and Technology of China

Sep 26, 2026J/K move · Enter open · S save

CAPEX: Efficiently Distilling Foundation Model Behavior into Deployable Robot Policies through Experience-Adaptive Reasoning

Shivam Aarya, Zhang Xi-Jia, Chengyue Huang, Junhyun Kim+5

School of Interactive Computing Georgia Institute of Technology Atlanta, GA 30332, USA

Robot Demonstration Generation Synthetic Data Generation for Robotics Data-Efficient Robot Learning Robot Imitation Learning Robot Foundation Models

Open paper AskPDF arXiv

Robot learning has largely relied on human-teleoperated demonstrations to acquire effective learnable behaviors. However, human-operated data collection processes can be unintuitive, difficult to scale, and inherently asynchronous. We explore an alternative: distilling physical behavior from general-purpose multimodal foundation models into deployable robot policies by using the foundation model itself as an autonomous demonstrator. While sufficiently capable models can generate successful zero-shot manipulation trajectories, repeatedly invoking them during physical execution is slow and expensive, limiting their utility as scalable data generators. As a solution, we introduce CAPEX, an experience-conditioned demonstration collection framework that uses execution experience from previous attempts to adapt how frequently the foundation model must observe, reason, and replan. We evaluate across RoboCasa tasks and on physical Franka and bimanual YAM-arm platforms, measuring task success, model calls, token usage, collection time, and cost. We further train Diffusion Policy and ACT on matched sets of human-teleoperated and foundation-model-generated demonstrations to evaluate the downstream learning value of autonomously collected data. We find that CAPEX increases the number of successful demonstrations by 4.3x while reducing the cost per successful demonstration by 80%. Policies trained on CAPEX-generated data approach the performance of those trained on matched human demonstrations; with longer training, this gap largely closes for policies trained from scratch. These results suggest that foundation models can serve as scalable sources of reusable robot experience. Project page: https://capex-paper.github.io/

Abstract

Knowledge Distillation

Explore similar work

CAPEX: Efficiently Distilling Foundation Model Behavior into Deployable Robot Policies through Experience-Adaptive Reasoning

Data Pyramid for Embodied Manipulation: A Survey

WAM-OPD: Sharpening World Action Models via On-Policy Distillation