Online Self-Training for Co-Adaptation in Hierarchical Diffusion Policies

Clemence Grislain; Mathilde Kappel; Olivier Sigaud; Mohamed Chetouani

cs.ROMar 5, 2026

Online Self-Training for Co-Adaptation in Hierarchical Diffusion Policies

Authors: Clemence Grislain, Mathilde Kappel, Olivier Sigaud, Mohamed Chetouani

Organizations: ISIR, Sorbonne Université, CNRS, Paris, France

Abstract

Hierarchical policies decompose language-conditioned long-horizon robotic manipulation into a high-level planner and a low-level controller. However, effective coordination between HL and LL requires that both components operate on compatible subgoal distributions. We propose ORCHID, a self-training framework that enables stable online improvement of hierarchical diffusion policies by aligning planning and control through iterative refinement. By filtering policy samples via environment feedback, ORCHID identifies trajectories where the planner and controller are jointly successful and distills them back into both modules via supervised learning. This process induces a bidirectional co-adaptation: the planner grounds its subgoals in the actual reaching capabilities of the controller, while the controller specializes in the trajectory structures the planner produces. By relying on supervised distillation of filtered on-policy samples, ORCHID avoids the instability typical of online hierarchical gradient-based RL training with diffusion models. On the CALVIN benchmark, ORCHID allows a lightweight, initially weak model to outperform pure offline methods, including a Vision-Language-Action model twice its size.

Figures & tables

Figure 1: ORCHID compared with common hierarchical policy training paradigms. (a) : Independent training of HL and LL on a fixed offline dataset $D_{0}$ , prone to HL-LL coupling mismatch. (b) : ORCHID iteratively refines both components through: (1) supervised updates on $D_{t}$ ; (2) on-policy rollout collection filtered by sparse environment reward, producing $\mathcal{R}_{t}$ — successful trajectories from which we extracts subgoals to train HL reachable by LL, and successful actions for LL conditioned on HL’s plans; and (3) dataset aggregation. Right: During rollout collection, $\pi_{\text{HL}}$ generates $K$ subgoal sequences $\hat{\zeta}$ for each context in $\mathcal{C}(D_{t})$ , executed by $\pi_{\text{LL}}$ via action chunks $a_{c}$ .

	Method	No. Instructions in a Row (1000 chains)					Avg. Len. ( $\uparrow$ )
		1	2	3	4	5
Baselines	HULC*	82.7%	64.9%	50.4%	38.5%	28.3%	2.64 ( $\pm$ 0.05)
	SuSIE*	87.7%	67.4%	49.8%	41.9%	33.7%	2.80 ( $\pm$ 0.15)
	LDC*	88.7%	69.9%	54.5%	42.7%	32.2%	2.88 ( $\pm$ 0.11)
	TaKSIE*	90.4%	73.9%	61.7%	51.2%	40.8%	3.18 ( $\pm$ 0.02)
	MDT*	93.7%	84.5%	74.1%	64.4%	55.6%	3.72 ( $\pm$ 0.05)

Table 1: ORCHID improves HD policies to achieve SOTA results on CALVIN LH-MTLC. HD policies trained with different variants of our method (up to 4 iterations) against baselines on CALVIN LH-MTLC. We report the success rate for completing 1 to 5 consecutive tasks, alongside the average successful sequence length (Avg. Len.). Mean and standard error over 3 seeds. Best performance are highlighted in bold , while second best underlined . Results from previous work are marked with

*

\dagger

indicates training until convergence.

Figure 4: ORCHID improves HD policy in low data regime in the Franka-3Blocks environment. SR of HD policy after 0 (trained solely on 10% of

D_{0}

) and up to 10 iter of ORCHID. Mean SR and standard error across 3 seeds.

		$HL_{D_{0}}$	$HL_{R_{1}}$	$HL_{D_{0}}$	$HL_{R_{1}}$
		$LL_{D_{0}}$	$LL_{D_{0}}$	$LL_{R_{1}}$	$LL_{R_{1}}$
Avg. Len. ( $\uparrow$ )		2.69 ${\pm 0.01}$	2.45 ${\pm 0.02}$	2.86 $\pm 0.01$	2.89 ${\pm 0.01}$
5-tasks SR		29.0% ${\pm 0.7}$	24.4% ${\pm 0.4}$	33.4% ${\pm 0.2}$	34.7% ${\pm 0.2}$
RE-pixel ( $\downarrow$ )	All	12.30 ${\pm 0.08}$	11.42 ${\pm 0.07}$	12.76 ${\pm 0.07}$	12.56 ${\pm 0.07}$
RE-pixel ( $\downarrow$ )	Last	16.73 ${\pm 0.26}$	14.58 ${\pm 0.27}$	15.60 $\pm 0.23$	12.80 ${\pm 0.23}$
RE-R3M ( $\downarrow$ ) (1e-3)	All	4.78 ${\pm 0.07}$	4.33 ${\pm 0.06}$	4.90 $\pm 0.06$	4.76 ${\pm 0.07}$

Table 2: Cross-evaluation of HL-LL combinations trained on expert data (

D_{0}

) or on-policy data (

R_{1}

) , mean and standard errors over 3 seeds of task sequence length (Avg. Len.), 5-tasks SR and reachability error (RE) computed over all subgoals and the last subgoal, across pixel, R3M, and DINOv2 embedding spaces.

	GT
	$LL_{D_{0}}$	$LL_{R_{1}}$
SR ( $\uparrow$ )	90.0 $\pm 0.1$	87.6 $\pm 0.1$

Table 3: Isolating LL performances trained on

\mathcal{D}_{0}

and

R_{1}

with GT subgoals. Mean SR on MTLC and standard errors over 3 seeds.

HL	LL	SR (%)
GT	ORCHID (iter 3)	91.2 ( $\pm$ 0.5)
ORCHID (iter 3) w/o replan.		92.1 ( $\pm$ 0.3)

Table 4: Fixed ORCHID LL (iter 3) guided by GT subgoals, and the respective HL without replanning. Results on MTLC.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
$x\sim\mathcal{X}$	$x$ sampled from a uniform distribution over the set $\mathcal{X}$
$\mathcal{X}^{*}$	Set of all finite sequences over $\mathcal{X}$
$S$	Environment state space
$\rho_{0}\in\mathcal{P}(S)$	Initial state distribution
$\rho_{\text{reset}}\in\mathcal{P}(S)$	Environment reset state distribution
$\Omega=\mathbb{R}^{3\times H\times W}$	Observation space (pixel space)

Appendix

Table 5: List of Notations

Environment	Component	Batch size	$N_{0}$
CALVIN	HL	16	5 $e^{5}$
	LL ACT	128	5 $e^{5}$
	LL DP	64	1 $e^{6}$
Franka-3Blocks	HL	64	1 $e^{5}$
Franka-3Blocks	LL DP	64	3 $e^{5}$

Appendix

Table 6: Training hyperparameters for both environments and the different component architectures.

Figure 6: Task distributions in CALVIN LH-MTLC and MTLC (uniform).

Figure 7: Training paradigm of the offline hierarchical baselines a) independent HL-LL training, b) ’glue’ models c) cross-level representation space.

	Method	Success Rate ( $\uparrow$ )
Baselines	LDC*	74.01 %
	SuSIE*	79.73 %
	HULC	81.77 ( $\pm$ 0.55) %
	TaKSIE*	86.80 %
	MDT	89.53 ( $\pm$ 0.27) %
Ours	HD (iter 0)	89.77 ( $\pm$ 0.46) %

Appendix

Table 9: Both versions of ORCHID improve HD policies and achieve SOTA performances on CALVIN MTLC after only one iteration. Mean success rate across tasks with standard error over 3 seeds of HD policies fine-tuned with 0 to 3 iterations of ORCHID and ORCHID-ft compared with baselines. Best results in bold , second best underlined , and results from prior work marked with

*

Panel of Figure 8 — Figure 8: replayed contexts increase the diversity of the context set, enhancing exploration. PCA visualization of the environment initial states for contexts in $D_{0}$ , in the replayed set extracted from $D_{0}$ , and sampled from the environment reset state distribution.

Training dataset	$\mathcal{D}_{0}$	$\mathcal{D}_{1}$	$\mathcal{D}_{2}$	$\mathcal{D}_{3}$
Convex hull area	79.7	85.6	90.7	94.5

Appendix

Table 11: State coverage across ORCHID iterations. Convex hull area in PCA space of initial states from which the policy successfully completes a task, for each training dataset

\mathcal{D}_{t}

, from

t=0

(offline only) to

t=3

HL	SR (%)
HD (iter 0) w/o replan.	83.3 ( $\pm$ 0.1)
HD (iter 0)	89.8 ( $\pm$ 0.5)
ORCHID (iter 3) w/o replan.	92.1 ( $\pm$ 0.3)
ORCHID (iter 3)	95.2 ( $\pm$ 0.2)

Appendix

Table 12: Evaluation of HD policies with and without replanning when trained solely on

D_{0}

(HD (iter 0) or after 3 iterations of ORCHID. Results are reported as mean success rates and standard errors across 3 seeds on CALVIN MTLC.

Figure 11: Mean success rates by task category in the Franka-3Blocks environment starting from 10% of

\mathcal{D}_{0}

. Results are shown over 0-10 iterations of ORCHID, with shaded regions indicating the standard error across tasks within each category.

Figure 12: All reachability errors decrease monotonically as performance increases across ORCHID iterations. The plot shows average successful sequence length (Avg. Len.) and reachability errors (all and last subgoals in pixel, R3M and DinoV2 embedding spaces) on LH-MTLC. Scatter points represent means, and shaded areas represent standard errors across 3 seeds.

Panel of Figure 13 — Figure 13: ORCHID achieves better performance but requires more computation than ORCHID-ft. Total number of gradient updates for training HL and LL (log scale) for ORCHID and ORCHID-ft against performance and training iterations.

Figure 14: ORCHID enables the generation of more task-relevant and feasible plans. Qualitative analyses comparing plans generated by the HD policy trained solely on $D_{0}$ (iter 0) versus plans generated after three iterations of ORCHID (iter 3) for same validation contexts in CALVIN MTLC. The symbol check indicates that the plan led to success, while the symbol cross indicates failure.

Figure 15: Planning with the DDIM noise scheduler reduces inference time without degrading performance. Average successful sequence length in LH-MTLC with standard error when sampling with the DDIM noise scheduler (10, 30, 50, and 70 diffusion steps) compared to DDPM (100 diffusion steps, matching training conditions). We also report the mean inference time for a single action prediction, including both planning and control, for each noise scheduler. Mean and standard error are reported over 3 seeds.

Robot Policy Learning

Robot policy learning develops controllers that map sensory observations, task goals, and interaction history to actions for embodied robots.

See all 372 papers

From Noise to Control: Parameterized Diffusion Policies
May 29, 2026Renhao Zhang, Haotian Fu, Mingxi Jia +3Robot Policy Learning Robot Policy Adaptation
University of Massachusetts · 2Brown University · 3Harvard
Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
Sep 28, 2026Hengrui Zhang, Yuhu Cheng, C. L. Philip Chen +1Diffusion-Based Planning Hierarchical RL
School of Information and Control Engineering, China University of Mining and Technology · School of Computer Science and Engineering, South China University of Technology
Beyond Flat Policies: Hierarchical Post-Training for Embodied Agents in Robotic Manipulation
Aug 6, 2026He Kong, Zengjue Chen, Qi Wang +6Long-Horizon Robotic Manipulation Hierarchical RL
School of Artificial Intelligence, Jilin University · College of computer science and technology, Jilin University · Joy Future Academy, JD

May 29, 2026J/K move · Enter open · S save

From Noise to Control: Parameterized Diffusion Policies

Renhao Zhang, Haotian Fu, Mingxi Jia, George Konidaris+2

University of Massachusetts · 2Brown University · 3Harvard

Robot Policy Learning Robot Policy Adaptation Diffusion Policy Robot Policy Generalization Policy Learning

Open paper AskPDF arXiv

We propose Parameterized Diffusion Policy (PDP), a framework for learning diffusion policies conditioned on low-dimensional, continuous parameters embedded in a learned behavior manifold. By constructing this manifold so that distances between latent representations reflect the semantic similarity between physical trajectories, we transform diffusion from a mechanism for stochastic diversity into a precise and optimizable tool for behavior steering. Our approach enables smooth interpolation between known strategies and efficient adaptation to novel constraints without updating policy weights. We demonstrate that PDP significantly improves adaptation performance on complex multimodal benchmarks in both simulated and real-robot experiments compared to standard diffusion policies, particularly in scenarios requiring the synthesis of novel behaviors.

Abstract

Robot Policy Learning

Explore similar work

From Noise to Control: Parameterized Diffusion Policies

Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning

Beyond Flat Policies: Hierarchical Post-Training for Embodied Agents in Robotic Manipulation