CIDER: Continual Interactive Distillation for Embodied Reinforcement Learning

Houlin Li; Minghui Xu; Guo Xu; Xuan Du; Xiaohan Yan; Chun Wang; Yuxiang Yan; Shukai Yang

cs.ROAug 22, 2026

CIDER: Continual Interactive Distillation for Embodied Reinforcement Learning

Authors: Houlin Li, Minghui Xu, Guo Xu, Xuan Du, Xiaohan Yan, Chun Wang, Yuxiang Yan, Shukai Yang,

Organizations: AgiBot · Shanghai Jiao Tong University

Abstract

Human-in-the-loop real-world reinforcement learning enables rapid acquisition of effective robotic manipulation policies for individual tasks, often within tens of minutes. Yet it remains unclear how to extend this paradigm to continual learning, where a single policy must acquire new skills without losing previously learned behaviors. Existing real-world continual learning methods do not explicitly constrain prior behaviors, leading to severe catastrophic forgetting. We introduce Continual Interactive Distillation for Embodied Reinforcement Learning (CIDER), a continual reinforcement learning framework that freezes the accumulated historical policy as a teacher before learning each new task and interleaves task learning with distillation-based retention. We further introduce gradient routing to separate the gradients used for acquiring new tasks from those used for preserving prior behaviors. We evaluate our method with a single shared actor on six real-world household and industrial manipulation tasks. Interactive Distillation maintains high measured success on previously learned tasks across our six-task real-robot sequence while acquiring each new task in 10 to 20 minutes, whereas every baseline forgets at least one previous task. Additional ablations reveal the key design choices that govern the tradeoff between stability and plasticity in real-world continual reinforcement learning.

Figures & tables

Figure 1: Overview of CIDER. Naive sequential RL forgets previous skills, while Interactive Distillation interleaves current-task online RL with rehearsal with a frozen expert on states from previous tasks.

Figure 2: The six real-world manipulation tasks.

Figure 3: Continual-learning performance across the six-task sequence, evaluated over five independent evaluation sets with 20 rollouts each. (a) Mean success rate over all tasks seen so far after each training stage; (b) per-method success-rate matrices where row $k$ shows $S_{i,k}$ for each task $\mathcal{M}_{i}$ and the left margin reports the stage average $A_{k}$ .

Figure 4: Success rate as a function of training steps for the CIDER policy (ours) and a single-task baseline across six manipulation tasks, averaged over three independent training runs per task.

Figure 5: Adapter gradient routing ablation: actor loss, mean Q value, and distillation loss under three routing configurations.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Examples of tasks.

Panel of Figure 8 — Figure 8: Examples of tasks.

Parameter	Value
Action space	6D twist
Reward function	Human-Assisted Binary classifier
Initial offline demonstrations	15
Environment update frequency	10 Hz
Maximum episode length	200 environment steps
Reset method	Human reset

Appendix

Table 2: Common Experimental Parameters For All Tasks.

Task	Observation space	Training steps
Socket Insertion	wrist_1, mock	21000
Toothbrush Hanging	wrist_1, mock	17000
Phone-Box Closing	wrist_1, mock	28000
Industrial Connectors Plugging	wrist_1, mock	29000
Safety Hook Coupling	wrist_1, mock	25000
Chip-Tray Placement	wrist_1, wrist_2	23000

Appendix

Table 3: Varied Experimental Parameters in Different Tasks.

Parameter	Finetune with EWC	Sequential BC
BC loss	NLL	NLL
EWC $\lambda$	1.0	—
FIM batches $N_{\mathrm{FIM}}$	1,000	—
Penalty normalization	$1/\|\theta\|$	—
Training data	All transitions (demo buffer + replay buffer)	Demo buffer
Historical replay	None	None

Appendix

Table 4: Hyperparameters for Finetune with EWC and Sequential BC.

Parameter	SAC	TD3+BC
Training steps/task	256,000	256,000
Auto entropy tuning	True	—
TD3+BC $\alpha$	—	10.0
Policy noise $\sigma$	—	0.2
Noise clip	—	0.5
Training data	All transitions	All transitions

Appendix

Table 5: Hyperparameters for Sequential Offline RL baselines.

Parameter	Value
Encoder	resnet10
Latent Dimension	64
Teacher RL Algorithm	HIL-RL
Student Network	MLP(256,256)
Distillation Loss	reverse KL
Distillation Learning Rate	$3\times 10^{-4}$

Appendix

Table 6: Implementation Details for DisCoRL

Method	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5	Stage 6
CIDER	$100.0\pm 0.0$	$100.0\pm 0.0$	$100.0\pm 0.0$	$100.0\pm 0.0$	$100.0\pm 0.0$	$100.0\pm 0.0$
DisCoRL	$100.0\pm 0.0$	$100.0\pm 0.0$	$100.0\pm 0.0$	$93.0\pm 0.8$	$55.8\pm 1.5$	$51.3\pm 1.4$
Finetune + Replay	$100.0\pm 0.0$	$81.5\pm 2.5$	$62.3\pm 1.2$	$56.5\pm 1.8$	$41.8\pm 2.5$	$26.2\pm 2.9$
Finetune + EWC	$100.0\pm 0.0$	$56.5\pm 9.0$	$38.7\pm 2.4$	$25.0\pm 0.0$	$20.0\pm 0.0$	$16.7\pm 0.0$
Finetune	$100.0\pm 0.0$	$50.0\pm 0.0$	$33.3\pm 0.0$	$25.0\pm 0.0$	$20.0\pm 0.0$	$16.7\pm 0.0$
Sequential Offline RL	$98.0\pm 0.2$	$42.0\pm 0.6$	$17.3\pm 1.7$	$25.0\pm 0.0$	$17.0\pm 1.3$	$16.2\pm 1.2$

Appendix

Table 7: Success rates with STD across continual learning stages.

RL updates per distillation update	Final avg	Current-task speed
0.5	100 $\pm$ 0.0	34.86 min
1(Ours)	100 $\pm$ 0.0	19.85 min
2	97.0 $\pm$ 2.0	19.88 min

Appendix

Table 8: Distillation update-ratio ablation.

Figure 10: Intervened Steps

Sampling	Final avg
Dataset-proportional	78.0 $\pm$ 1.8
Task-balanced (Ours)	100 $\pm$ 0.0

Appendix

Table 9: Historical sampling ablation.

Sampling	Final avg
Ours $(M_{1}\sim M_{6})$	100 $\pm$ 0.0
$M_{3},M_{4},M_{5},M_{6},M_{1},M_{2}$	96.2 $\pm$ 1.4
$M_{1},M_{4},M_{2},M_{5},M_{3},M_{6}$	98.4 $\pm$ 0.7

Appendix

Table 10: Task order robustness.

Loss	Toothbrush Hanging (current)	Socket Insertion (retention)
NLL	100 $\pm$ 0.0	100 $\pm$ 0.0
Forward KL	100 $\pm$ 0.0	100 $\pm$ 0.0
Reverse KL (Ours)	100 $\pm$ 0.0	100 $\pm$ 0.0
MSE	100 $\pm$ 0.0	17.0 $\pm$ 1.2

Appendix

Table 11: Toothbrush Hanging is evaluated as the current task after stage 2, and Socket Insertion is re-evaluated as the retention check.

Continual Robot Learning

Continual robot learning develops robots that acquire new skills and adapt to changing conditions while retaining previously learned behavior.

See all 89 papers

VE2VF: Vision-Enabled to Vision-Free Distillation via Real-world Reinforcement Learning for Robust Contact-Rich Manipulation
May 28, 2026Victor Kowalski, Chengxi Li, Dongheui LeeCross-Modal Knowledge Distillation Robot Skill Learning
Autonomous Systems, Technische Universitaet Wien (TU Wien), Vienna, Austria · Institute of Robotics and Mechatronics (DLR), German Aerospace Center, Wessling, Germany
Res-HIL: Human-Guided Residual Reinforcement Learning for Sample-Efficient Dexterous Manipulation
Sep 24, 2026Mariia Iavorskaia, Christian Dietz, Sebastian Albrecht +1Contact-Rich Robotic Manipulation Human-in-the-Loop AI
Siemens AG, Research and Predevelopment, Germany. · Technical University of Munich, Germany.
ProgAgent:A Continual RL Agent with Progress-Aware Rewards
Mar 8, 2026Jinzhou Tan, Gabriel Adineera, Jinoh KimContinual RL Continual Robot Learning
University of California, San Diego, USA · Texas A&M University-Commerce, USA

May 28, 2026J/K move · Enter open · S save

VE2VF: Vision-Enabled to Vision-Free Distillation via Real-world Reinforcement Learning for Robust Contact-Rich Manipulation

Victor Kowalski, Chengxi Li, Dongheui Lee

Autonomous Systems, Technische Universitaet Wien (TU Wien), Vienna, Austria · Institute of Robotics and Mechatronics (DLR), German Aerospace Center, Wessling, Germany

Cross-Modal Knowledge Distillation Robot Skill Learning Contact-Rich Robotic Manipulation Generalization in Robotic Manipulation Robotic RL Robot Policy Generalization+2

Open paper AskPDF arXiv

When using reinforcement learning (RL) for contact-rich robotic manipulation, vision can provide task-relevant information that complements robot proprioception. However, vision-enabled policies tend to overfit to the visual conditions seen during training, limiting their robustness and transferability. We present a human-in-the-loop RL framework that employs teacher-student distillation to achieve robust performance across multiple task variants, trained entirely in the real world without requiring domain randomization or data augmentation. A vision-enabled teacher distills its knowledge into a vision-free student that relies solely on pose, twist, and wrench sensing, combining fast training with strong task generalization. On the real-world NIST assembly benchmark board, our approach achieves 95% overall success after approximately 50 minutes of training on 3 representative tasks, including robust generalization to 8 unseen task variants. Fine-tuning with distillation achieves full success on the most challenging task. We demonstrate that the resulting policies outperform baselines in both robustness and adaptability. Page: https://tuwien-asl.github.io/VE2VF/.

Abstract

Continual Robot Learning

Explore similar work

VE2VF: Vision-Enabled to Vision-Free Distillation via Real-world Reinforcement Learning for Robust Contact-Rich Manipulation

Res-HIL: Human-Guided Residual Reinforcement Learning for Sample-Efficient Dexterous Manipulation

ProgAgent:A Continual RL Agent with Progress-Aware Rewards