Hierarchical policies decompose language-conditioned long-horizon robotic manipulation into a high-level planner and a low-level controller. However, effective coordination between HL and LL requires that both components operate on compatible subgoal distributions. We propose ORCHID, a self-training framework that enables stable online improvement of hierarchical diffusion policies by aligning planning and control through iterative refinement. By filtering policy samples via environment feedback, ORCHID identifies trajectories where the planner and controller are jointly successful and distills them back into both modules via supervised learning. This process induces a bidirectional co-adaptation: the planner grounds its subgoals in the actual reaching capabilities of the controller, while the controller specializes in the trajectory structures the planner produces. By relying on supervised distillation of filtered on-policy samples, ORCHID avoids the instability typical of online hierarchical gradient-based RL training with diffusion models. On the CALVIN benchmark, ORCHID allows a lightweight, initially weak model to outperform pure offline methods, including a Vision-Language-Action model twice its size.
Figures & tables
Figure 1: ORCHID compared with common hierarchical policy training paradigms. (a) : Independent training of HL and LL on a fixed offline dataset D0 , prone to HL-LL coupling mismatch. (b) : ORCHID iteratively refines both components through: (1) supervised updates on Dt ; (2) on-policy rollout collection filtered by sparse environment reward, producing Rt — successful trajectories from which we extracts subgoals to train HL reachable by LL, and successful actions for LL conditioned on HL’s plans; and (3) dataset aggregation. Right: During rollout collection, πHL generates K subgoal sequences ζ^ for each context in C(Dt) , executed by πLL via action chunks ac .
Method
No. Instructions in a Row (1000 chains)
Avg. Len. ( ↑ )
1
2
3
4
5
Baselines
HULC*
82.7%
64.9%
50.4%
38.5%
28.3%
2.64 ( ± 0.05)
SuSIE*
87.7%
67.4%
49.8%
41.9%
33.7%
2.80 ( ± 0.15)
LDC*
88.7%
69.9%
54.5%
42.7%
32.2%
2.88 ( ± 0.11)
TaKSIE*
90.4%
73.9%
61.7%
51.2%
40.8%
3.18 ( ± 0.02)
MDT*
93.7%
84.5%
74.1%
64.4%
55.6%
3.72 ( ± 0.05)
Table 1: ORCHID improves HD policies to achieve SOTA results on CALVIN LH-MTLC. HD policies trained with different variants of our method (up to 4 iterations) against baselines on CALVIN LH-MTLC. We report the success rate for completing 1 to 5 consecutive tasks, alongside the average successful sequence length (Avg. Len.). Mean and standard error over 3 seeds. Best performance are highlighted in bold , while second best underlined . Results from previous work are marked with ∗ . † indicates training until convergence.
Figure 3
Figure 4: ORCHID improves HD policy in low data regime in the Franka-3Blocks environment. SR of HD policy after 0 (trained solely on 10% of D0 ) and up to 10 iter of ORCHID. Mean SR and standard error across 3 seeds.
HLD0
HLR1
HLD0
HLR1
LLD0
LLD0
LLR1
LLR1
Avg. Len. ( ↑ )
2.69 ±0.01
2.45 ±0.02
2.86 ±0.01
2.89 ±0.01
5-tasks SR
29.0% ±0.7
24.4% ±0.4
33.4% ±0.2
34.7% ±0.2
RE-pixel ( ↓ )
All
12.30 ±0.08
11.42 ±0.07
12.76 ±0.07
12.56 ±0.07
Last
16.73 ±0.26
14.58 ±0.27
15.60 ±0.23
12.80 ±0.23
RE-R3M ( ↓ ) (1e-3)
All
4.78 ±0.07
4.33 ±0.06
4.90 ±0.06
4.76 ±0.07
Table 2: Cross-evaluation of HL-LL combinations trained on expert data ( D0 ) or on-policy data ( R1 ) , mean and standard errors over 3 seeds of task sequence length (Avg. Len.), 5-tasks SR and reachability error (RE) computed over all subgoals and the last subgoal, across pixel, R3M, and DINOv2 embedding spaces.
GT
LLD0
LLR1
SR ( ↑ )
90.0 ±0.1
87.6 ±0.1
Table 3: Isolating LL performances trained on D0 and R1 with GT subgoals. Mean SR on MTLC and standard errors over 3 seeds.
HL
LL
SR (%)
GT
ORCHID (iter 3)
91.2 ( ± 0.5)
ORCHID (iter 3) w/o replan.
92.1 ( ± 0.3)
Table 4: Fixed ORCHID LL (iter 3) guided by GT subgoals, and the respective HL without replanning. Results on MTLC.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Description
x∼X
x sampled from a uniform distribution over the set X
X∗
Set of all finite sequences over X
S
Environment state space
ρ0∈P(S)
Initial state distribution
ρreset∈P(S)
Environment reset state distribution
Ω=R3×H×W
Observation space (pixel space)
Appendix
Table 5: List of Notations
Environment
Component
Batch size
N0
CALVIN
HL
16
5 e5
LL ACT
128
5 e5
LL DP
64
1 e6
Franka-3Blocks
HL
64
1 e5
LL DP
64
3 e5
Appendix
Table 6: Training hyperparameters for both environments and the different component architectures.
Figure 10
Figure 6: Task distributions in CALVIN LH-MTLC and MTLC (uniform).
Figure 7: Training paradigm of the offline hierarchical baselines a) independent HL-LL training, b) ’glue’ models c) cross-level representation space.
Method
Success Rate ( ↑ )
Baselines
LDC*
74.01 %
SuSIE*
79.73 %
HULC
81.77 ( ± 0.55) %
TaKSIE*
86.80 %
MDT
89.53 ( ± 0.27) %
Ours
HD (iter 0)
89.77 ( ± 0.46) %
Appendix
Table 9: Both versions of ORCHID improve HD policies and achieve SOTA performances on CALVIN MTLC after only one iteration. Mean success rate across tasks with standard error over 3 seeds of HD policies fine-tuned with 0 to 3 iterations of ORCHID and ORCHID-ft compared with baselines. Best results in bold , second best underlined , and results from prior work marked with ∗ .
Figure 8: replayed contexts increase the diversity of the context set, enhancing exploration. PCA visualization of the environment initial states for contexts in D0 , in the replayed set extracted from D0 , and sampled from the environment reset state distribution.
Training dataset
D0
D1
D2
D3
Convex hull area
79.7
85.6
90.7
94.5
Appendix
Table 11: State coverage across ORCHID iterations. Convex hull area in PCA space of initial states from which the policy successfully completes a task, for each training dataset Dt , from t=0 (offline only) to t=3 .
HL
SR (%)
HD (iter 0) w/o replan.
83.3 ( ± 0.1)
HD (iter 0)
89.8 ( ± 0.5)
ORCHID (iter 3) w/o replan.
92.1 ( ± 0.3)
ORCHID (iter 3)
95.2 ( ± 0.2)
Appendix
Table 12: Evaluation of HD policies with and without replanning when trained solely on D0 (HD (iter 0) or after 3 iterations of ORCHID. Results are reported as mean success rates and standard errors across 3 seeds on CALVIN MTLC.
Figure 17
Figure 11: Mean success rates by task category in the Franka-3Blocks environment starting from 10% of D0 . Results are shown over 0-10 iterations of ORCHID, with shaded regions indicating the standard error across tasks within each category.
Figure 12: All reachability errors decrease monotonically as performance increases across ORCHID iterations. The plot shows average successful sequence length (Avg. Len.) and reachability errors (all and last subgoals in pixel, R3M and DinoV2 embedding spaces) on LH-MTLC. Scatter points represent means, and shaded areas represent standard errors across 3 seeds.
Figure 13: ORCHID achieves better performance but requires more computation than ORCHID-ft. Total number of gradient updates for training HL and LL (log scale) for ORCHID and ORCHID-ft against performance and training iterations.
Figure 14: ORCHID enables the generation of more task-relevant and feasible plans. Qualitative analyses comparing plans generated by the HD policy trained solely on D0 (iter 0) versus plans generated after three iterations of ORCHID (iter 3) for same validation contexts in CALVIN MTLC. The symbol check indicates that the plan led to success, while the symbol cross indicates failure.
Figure 15: Planning with the DDIM noise scheduler reduces inference time without degrading performance. Average successful sequence length in LH-MTLC with standard error when sampling with the DDIM noise scheduler (10, 30, 50, and 70 diffusion steps) compared to DDPM (100 diffusion steps, matching training conditions). We also report the mean inference time for a single action prediction, including both planning and control, for each noise scheduler. Mean and standard error are reported over 3 seeds.
We propose Parameterized Diffusion Policy (PDP), a framework for learning diffusion policies conditioned on low-dimensional, continuous parameters embedded in a learned behavior manifold. By constructing this manifold so that distances between latent representations reflect the semantic similarity between physical trajectories, we transform diffusion from a mechanism for stochastic diversity into a precise and optimizable tool for behavior steering. Our approach enables smooth interpolation between known strategies and efficient adaptation to novel constraints without updating policy weights. We demonstrate that PDP significantly improves adaptation performance on complex multimodal benchmarks in both simulated and real-robot experiments compared to standard diffusion policies, particularly in scenarios requiring the synthesis of novel behaviors.
Renhao Zhang, Haotian Fu, Mingxi Jia +3
University of Massachusetts · 2Brown University · 3Harvard
Offline goal-conditioned reinforcement learning (GCRL) learns goal-directed policies from reward-free data, but in long-horizon tasks, goal-conditioned value functions often provide unstable guidance due to sparse rewards and discounting. Hierarchical methods partially mitigate this issue via subgoal decomposition; however, high-level decision-making still relies on noise-sensitive value estimates, leading to unstable behavior in complex environments. We address this limitation by proposing Diffusion Subgoal Planning (DSP), a diffusion-based framework for high-level subgoal generation. DSP casts high-level planning as guided generative inference over goal-conditioned subgoals and learns both conditional and unconditional flows, enabling classifier-free guidance to introduce a goal-directed bias at inference time. By removing explicit value-based guidance from high-level planning, DSP generates reachable and goal-directed subgoals through a generative model while retaining hierarchical execution. Experiments on offline GCRL benchmarks demonstrate that DSP outperforms prior methods on a range of navigation and manipulation tasks, with particularly strong performance in maze environments that require multi-step subgoal planning.
Hengrui Zhang, Yuhu Cheng, C. L. Philip Chen +1
School of Information and Control Engineering, China University of Mining and Technology · School of Computer Science and Engineering, South China University of Technology
Vision-language-action (VLA) models have demonstrated remarkable capabilities in robotic manipulation by leveraging pretrained vision-language models. However, existing post-training methods predominantly optimize VLA models as flat policies, making it difficult to explicitly model task progression and perform robust long-horizon manipulation. Although hierarchical approaches introduce task decomposition, they mainly rely on supervised learning from offline demonstrations and cannot effectively improve execution through online interaction. To address this limitation, we propose Hierarchical Robotic Control (HiRoC), a hierarchical post-training framework that decouples high-level task planning from low-level action execution. The planner decomposes complex tasks into executable subgoals to provide explicit semantic guidance, while the executor continuously improves subgoal-conditioned action generation through reinforcement learning. To enable effective collaboration between the two modules, we further align the executor with planner-generated subgoals before reinforcement learning, mitigating the distribution misalignment between planning and execution. Extensive experiments across diverse robotic manipulation benchmarks demonstrate that HiRoC consistently outperforms strong baselines. Comprehensive analyses further validate the effectiveness of hierarchical post-training and the contribution of each key component.
He Kong, Zengjue Chen, Qi Wang +6
School of Artificial Intelligence, Jilin University · College of computer science and technology, Jilin University · Joy Future Academy, JD