Learning Hierarchical Skill Policies with Offline Quality-Diversity Reinforcement Learning

Tanachai Anakewat; Takayuki Osa; Tatsuya Harada

cs.AIAug 20, 2026

Learning Hierarchical Skill Policies with Offline Quality-Diversity Reinforcement Learning

Authors: Tanachai Anakewat, Takayuki Osa, Tatsuya Harada

Organizations: The University of Tokyo · RIKEN Center for Advanced Intelligence Project

Abstract

Recent studies investigate how to leverage pre-collected datasets to improve the policy performance and sample efficiency of RL. One promising approach to achieve this goal is to employ a two-stage strategy: In the first stage, diverse skills are extracted as a low-level policy from a given dataset, and a high-level policy is trained to solve a specific task in the second stage. Typically, extraction of the low-level policy is performed based on unsupervised learning such as trajectory VAE. However, a limitation of this approach is that the quality of the low-level policy highly depends on the quality of the dataset. To address this issue, we introduce QDOS (Quality-Diversity Offline Skill learning), a unified pipeline for robust offline-to-online learning. Our approach incorporates an Advantage-Weighted Quality-Diversity pretraining objective, which weights the skill extraction and diversity objectives by the estimated advantage of each trajectory segment. This approach allows the model to extract diverse and high-value skills. By providing robust and task-relevant skill representations, QDOS significantly improves the quality of the embedded skill space used by the low-level policy. We further integrate this with a dual dataset reuse strategy, where offline data is used both for skill pretraining and for populating the online replay buffer via pseudo-labeling. Experiments demonstrate that QDOS significantly outperforms strong baselines in structured manipulation tasks and unstructured locomotion tasks, confirming its ability to accelerate exploration and improve final returns in challenging sparse-reward domains.

Figures & tables

Fig. 1: Overview of QDOS Framework. (1) Offline Pretraining : The agent learns low-level skills ( $\pi_{\theta}$ ) from a mixed-quality dataset containing good moves, failed attempts, and noise. We employ an Advantage-Weighted Quality-Diversity objective to filter out sub-optimal data (failed attempts and noise) while extracting diverse, high-quality skills. (2) Online RL with Dual Dataset Reuse : The same offline data is reused to populate the high-level replay buffer via pseudo-labeling, enabling the high-level policy to utilize prior knowledge for efficient online exploration.

Fig. 2: The Mixed-Quality Dilemma: Standard VAEs encode all behaviors equally, entangling noise with useful skills.

\mathcal{L}_{\mathrm{VAE}}^{\mathrm{AW}}=\frac{1}{B}\sum_{i=1}^{B}w^{(i)}\Big[\beta D_{\mathrm{KL}}-\mathbb{E}_{z}[\log\pi_{\theta}]\Big].

Algorithm 1 QDOS: Advantage-Weighted Quality-Diversity Skill Pretraining and Online RL

Fig. 3: Learning curves across environments. QDOS is shown in red ( $\alpha=0.2$ ) and blue ( $\alpha=0.3$ ); Baseline (SUPE), ExPLORe, Trajectory Skills, HILP Online, and Diff BC JSRL are shown in black, green, purple, brown, and orange, respectively.

Environment	BC [ 11 ] [ 26 ]	IQL [ 11 ] [ 26 ]	SUPE	QDOS ( $\alpha=0.2$ )	QDOS ( $\alpha=0.3$ )
antmaze-large	$41\pm 7$	$64\pm 10$	$83.0\pm 21.0$	$\underline{86.0\pm 6.0}$	$\mathbf{96.0\pm 6.0}$
antsoccer-arena	$5\pm 1$	$50\pm 2$	$27.0\pm 31.0$	$\underline{73.0\pm 31.0}$	$\mathbf{80.0\pm 10.0}$
humanoidmaze-medium	$8\pm 2$	$27\pm 2$	$\underline{97.0\pm 6.0}$	$\underline{97.0\pm 6.0}$	$\mathbf{100.0\pm 0.0}$
kitchen-mixed	51.5	51.0	$\underline{85\pm 13.25}$	$\mathbf{100.00\pm 0.00}$	$83.25\pm 28.75$
kitchen-partial	38.0	46.3	$\mathbf{83.25\pm 14.50}$	$\mathbf{85.75\pm 24.75}$	$\mathbf{82.50\pm 30.25}$
scene-task1	$5\pm 1$	$51\pm 4$	$\mathbf{100.0\pm 0.0}$	$\mathbf{100.0\pm 0.0}$	$\mathbf{100.0\pm 0.0}$

TABLE I: Normalized evaluation returns comparing BC, IQL, SUPE, and QDOS across various domains. Results for SUPE and QDOS are averaged over 3 seeds.

Fig. 4: Training steps to first goal finding. QDOS (Red) finds the goal significantly faster than baselines, indicating more efficient exploration.

Environment	Method	Steps
antmaze-large	SUPE	$13334\pm 5774$
antmaze-large	QDOS ( $\alpha=0.2$ )	$\mathbf{11668\pm 9465}$
humanoidmaze	SUPE	$23334\pm 6292$
humanoidmaze	QDOS ( $\alpha=0.3$ )	$\mathbf{19168\pm 8036}$
scene	SUPE	$5834\pm 1443$
	QDOS ( $\alpha=0.2$ )	$\mathbf{5001\pm 0}$

TABLE II: Training steps to reach the goal for the first time (Mean

\pm

Std).

Fig. 5: Experimental results on the cube datasets. Reducing the diversity weight $\alpha_{\text{loss}}$ leads to performance closer to the SUPE baseline, indicating that explicit diversity enforcement is less critical for high-quality datasets.

Fig. 6: t-SNE visualization (top) and skill usage timeline (bottom) across kitchen , antmaze , and antsoccer . Orange points represent QDOS ( $\alpha=0.3$ for antsoccer and antmaze , $\alpha=0.2$ for kitchen ), and Blue points represent SUPE. The t-SNE plots indicate that QDOS tends to utilize a broader region of the latent space, suggesting the capture of more diverse behaviors compared to the more restricted latent usage of SUPE.

Fig. 7: QDOS skill trajectory visualization for kitchen-mixed . Red crosses mark selected timesteps ( $t=0,7,14,22$ ) with corresponding scene snapshots.

Table 11

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

Structure-Conditioned Actor-Critic Branches for Quality-Diversity Reinforcement Learning
Jun 7, 2026Lianrong Zuo, Peilan Xu, Yong Liu +1Reinforcement Learning Policy Optimization
Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies, Institute of Cyberspace Security, School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen 518055, Guangdong, China
Exploiting Local Dynamics Regularity for Reusable Skills in Offline Hierarchical RL
May 25, 2026Sarthak Dayal, Abhinav Peri, Carl Qi +4Hierarchical RL Offline RL
Department of Computer Science University of Texas at Austin · UT Austin · OpenAI
Dynamic Dual-Granularity Skill Bank for Agentic RL
Mar 30, 2026Songjun Tu, Chengdong Xu, Qichao Zhang +5LLM Agent Skill Learning Agent Skill Learning
Institute of Automation, Chinese Academy of Sciences · Pengcheng Laboratory · Sun Yat-Sen University +1

Jun 7, 2026J/K move · Enter open · S save

Structure-Conditioned Actor-Critic Branches for Quality-Diversity Reinforcement Learning

Lianrong Zuo, Peilan Xu, Yong Liu, Wenjian Luo

Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies, Institute of Cyberspace Security, School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen 518055, Guangdong, China

Reinforcement Learning Policy Optimization Actor-Critic Methods Behavioral Diversity in RL

Open paper AskPDF arXiv

Quality-diversity reinforcement learning (QD-RL) aims to construct policy repertoires that contain both high-performing and behaviorally diverse policies. Existing QD-RL methods mainly diversify policy instances after rollout evaluation or use learned value information to improve policy quality and behavior targeting, while the learning branches that generate candidate policies remain less explored. This paper proposes SV-QD-RL, a structure-value coupled framework that represents each candidate as a structure-conditioned actor-critic branch. Each branch contains an actor, a structural mask, a branch-specific critic, a replay state, and evaluation attributes including behavior, return, sparsity, and value profile. The structural mask defines the actor subspace in which the branch learns, while the branch-specific critic and replay state shape its value-learning trajectory. A branch-aware QD archive then evaluates and retains branches according to behavioral quality, structural footprint, and value-profile information. Experiments on MuJoCo continuous-control tasks show that SV-QD-RL constructs policy repertoires with strong archive quality and behaviorally useful diversity. Ablation and diagnostic analyses further indicate that structural conditioning, critic differentiation, and memory-consistent refinement make complementary contributions to behavioral specialization. Schedule-aware repertoire evaluation shows that the learned archive provides selectable policy alternatives under changing behavior-level requirements. These results suggest that coupling actor structure with branch-specific value learning is an effective mechanism for generating diverse QD-RL policy repertoires.

Abstract

Reinforcement Learning

Explore similar work

Structure-Conditioned Actor-Critic Branches for Quality-Diversity Reinforcement Learning

Exploiting Local Dynamics Regularity for Reusable Skills in Offline Hierarchical RL

Dynamic Dual-Granularity Skill Bank for Agentic RL