Learning Hierarchical Skill Policies with Offline Quality-Diversity Reinforcement Learning

Tanachai Anakewat; Takayuki Osa; Tatsuya Harada

cs.AIAug 20, 2026

Learning Hierarchical Skill Policies with Offline Quality-Diversity Reinforcement Learning

Authors: Tanachai Anakewat, Takayuki Osa, Tatsuya Harada

Organizations: The University of Tokyo · RIKEN Center for Advanced Intelligence Project

Abstract

Recent studies investigate how to leverage pre-collected datasets to improve the policy performance and sample efficiency of RL. One promising approach to achieve this goal is to employ a two-stage strategy: In the first stage, diverse skills are extracted as a low-level policy from a given dataset, and a high-level policy is trained to solve a specific task in the second stage. Typically, extraction of the low-level policy is performed based on unsupervised learning such as trajectory VAE. However, a limitation of this approach is that the quality of the low-level policy highly depends on the quality of the dataset. To address this issue, we introduce QDOS (Quality-Diversity Offline Skill learning), a unified pipeline for robust offline-to-online learning. Our approach incorporates an Advantage-Weighted Quality-Diversity pretraining objective, which weights the skill extraction and diversity objectives by the estimated advantage of each trajectory segment. This approach allows the model to extract diverse and high-value skills. By providing robust and task-relevant skill representations, QDOS significantly improves the quality of the embedded skill space used by the low-level policy. We further integrate this with a dual dataset reuse strategy, where offline data is used both for skill pretraining and for populating the online replay buffer via pseudo-labeling. Experiments demonstrate that QDOS significantly outperforms strong baselines in structured manipulation tasks and unstructured locomotion tasks, confirming its ability to accelerate exploration and improve final returns in challenging sparse-reward domains.

Figures & tables

Fig. 1: Overview of QDOS Framework. (1) Offline Pretraining : The agent learns low-level skills ( $\pi_{\theta}$ ) from a mixed-quality dataset containing good moves, failed attempts, and noise. We employ an Advantage-Weighted Quality-Diversity objective to filter out sub-optimal data (failed attempts and noise) while extracting diverse, high-quality skills. (2) Online RL with Dual Dataset Reuse : The same offline data is reused to populate the high-level replay buffer via pseudo-labeling, enabling the high-level policy to utilize prior knowledge for efficient online exploration.

Fig. 2: The Mixed-Quality Dilemma: Standard VAEs encode all behaviors equally, entangling noise with useful skills.

\mathcal{L}_{\mathrm{VAE}}^{\mathrm{AW}}=\frac{1}{B}\sum_{i=1}^{B}w^{(i)}\Big[\beta D_{\mathrm{KL}}-\mathbb{E}_{z}[\log\pi_{\theta}]\Big].

Algorithm 1 QDOS: Advantage-Weighted Quality-Diversity Skill Pretraining and Online RL

Fig. 3: Learning curves across environments. QDOS is shown in red ( $\alpha=0.2$ ) and blue ( $\alpha=0.3$ ); Baseline (SUPE), ExPLORe, Trajectory Skills, HILP Online, and Diff BC JSRL are shown in black, green, purple, brown, and orange, respectively.

Environment	BC [ 11 ] [ 26 ]	IQL [ 11 ] [ 26 ]	SUPE	QDOS ( $\alpha=0.2$ )	QDOS ( $\alpha=0.3$ )
antmaze-large	$41\pm 7$	$64\pm 10$	$83.0\pm 21.0$	$\underline{86.0\pm 6.0}$	$\mathbf{96.0\pm 6.0}$
antsoccer-arena	$5\pm 1$	$50\pm 2$	$27.0\pm 31.0$	$\underline{73.0\pm 31.0}$	$\mathbf{80.0\pm 10.0}$
humanoidmaze-medium	$8\pm 2$	$27\pm 2$	$\underline{97.0\pm 6.0}$	$\underline{97.0\pm 6.0}$	$\mathbf{100.0\pm 0.0}$
kitchen-mixed	51.5	51.0	$\underline{85\pm 13.25}$	$\mathbf{100.00\pm 0.00}$	$83.25\pm 28.75$
kitchen-partial	38.0	46.3	$\mathbf{83.25\pm 14.50}$	$\mathbf{85.75\pm 24.75}$	$\mathbf{82.50\pm 30.25}$
scene-task1	$5\pm 1$	$51\pm 4$	$\mathbf{100.0\pm 0.0}$	$\mathbf{100.0\pm 0.0}$	$\mathbf{100.0\pm 0.0}$

TABLE I: Normalized evaluation returns comparing BC, IQL, SUPE, and QDOS across various domains. Results for SUPE and QDOS are averaged over 3 seeds.

Fig. 4: Training steps to first goal finding. QDOS (Red) finds the goal significantly faster than baselines, indicating more efficient exploration.

Environment	Method	Steps
antmaze-large	SUPE	$13334\pm 5774$
antmaze-large	QDOS ( $\alpha=0.2$ )	$\mathbf{11668\pm 9465}$
humanoidmaze	SUPE	$23334\pm 6292$
humanoidmaze	QDOS ( $\alpha=0.3$ )	$\mathbf{19168\pm 8036}$
scene	SUPE	$5834\pm 1443$
	QDOS ( $\alpha=0.2$ )	$\mathbf{5001\pm 0}$

TABLE II: Training steps to reach the goal for the first time (Mean

\pm

Std).

Fig. 5: Experimental results on the cube datasets. Reducing the diversity weight $\alpha_{\text{loss}}$ leads to performance closer to the SUPE baseline, indicating that explicit diversity enforcement is less critical for high-quality datasets.

Fig. 6: t-SNE visualization (top) and skill usage timeline (bottom) across kitchen , antmaze , and antsoccer . Orange points represent QDOS ( $\alpha=0.3$ for antsoccer and antmaze , $\alpha=0.2$ for kitchen ), and Blue points represent SUPE. The t-SNE plots indicate that QDOS tends to utilize a broader region of the latent space, suggesting the capture of more diverse behaviors compared to the more restricted latent usage of SUPE.

Fig. 7: QDOS skill trajectory visualization for kitchen-mixed . Red crosses mark selected timesteps ( $t=0,7,14,22$ ) with corresponding scene snapshots.

Table 11

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

Jun 7, 2026

Structure-Conditioned Actor-Critic Branches for Quality-Diversity Reinforcement Learning

Quality-diversity reinforcement learning (QD-RL) aims to construct policy repertoires that contain both high-performing and behaviorally diverse policies. Existing QD-RL methods mainly diversify policy instances after rollout evaluation or use learned value information to improve policy quality and behavior targeting, while the learning branches that generate candidate policies remain less explored. This paper proposes SV-QD-RL, a structure-value coupled framework that represents each candidate as a structure-conditioned actor-critic branch. Each branch contains an actor, a structural mask, a branch-specific critic, a replay state, and evaluation attributes including behavior, return, sparsity, and value profile. The structural mask defines the actor subspace in which the branch learns, while the branch-specific critic and replay state shape its value-learning trajectory. A branch-aware QD archive then evaluates and retains branches according to behavioral quality, structural footprint, and value-profile information. Experiments on MuJoCo continuous-control tasks show that SV-QD-RL constructs policy repertoires with strong archive quality and behaviorally useful diversity. Ablation and diagnostic analyses further indicate that structural conditioning, critic differentiation, and memory-consistent refinement make complementary contributions to behavioral specialization. Schedule-aware repertoire evaluation shows that the learned archive provides selectable policy alternatives under changing behavior-level requirements. These results suggest that coupling actor structure with branch-specific value learning is an effective mechanism for generating diverse QD-RL policy repertoires.

Lianrong Zuo, Peilan Xu, Yong Liu +1

Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies, Institute of Cyberspace Security, School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen 518055, Guangdong, China

Reinforcement Learning Policy Optimization Actor-Critic Methods Behavioral Diversity in RL

Ask PDF

May 25, 2026

Exploiting Local Dynamics Regularity for Reusable Skills in Offline Hierarchical RL

Hierarchical Reinforcement Learning (HRL) promises to solve long-horizon Reinforcement Learning (RL) tasks more efficiently than non-hierarchical counterparts by discovering and reusing temporally-extended skills. However, obtaining skills that are actually reusable remains an open challenge. Towards this end, we focus on abstractions that exploit the intuition of local dynamics: local transitions in different global contexts require similar kinds of action sequences. By aligning these contexts with the action sequences they require, we are able to learn which skills to reuse and where to reuse them. In principle, this information should benefit many HRL algorithms, where high-level policies have to reason about the low-level skills they use. The resulting algorithm CARL (Contrastive Action-based Representations for Reusable Local Control) shows both qualitative clustering of meaningful skills in complex humanoid environments and improved downstream performance on the OGBench benchmark when integrated with HIQL.

Sarthak Dayal, Abhinav Peri, Carl Qi +4

Department of Computer Science University of Texas at Austin · UT Austin · OpenAI

Hierarchical RL Offline RL Representation Learning for RL Agent Skill Learning

Ask PDF

Mar 30, 2026

Dynamic Dual-Granularity Skill Bank for Agentic RL

Agentic RL can benefit substantially from reusable experience, yet existing skill-based methods mainly extract trajectory-level guidance and often lack principled mechanisms for maintaining an evolving skill memory. We propose D2Skill, a dynamic dual-granularity skill bank for agentic RL that organizes reusable experience into task skills for high-level guidance and step skills for fine-grained decision support and error correction. D2Skill jointly trains the policy and skill bank through paired baseline and skill-injected rollouts under the same policy, using their performance gap to derive hindsight utility signals for both skill updating and policy optimization. Built entirely from training-time experience, the skill bank is continuously expanded through reflection and maintained with utility-aware retrieval and pruning. Experiments on ALFWorld, WebShop, and Search-Augmented QA tasks show that D2Skill substantially improves performance over skill-free baselines across models of different scales. Further ablations and analyses show that both dual-granularity skill modeling and dynamic skill maintenance are critical to these gains, while the learned skills exhibit higher utility, transfer across evaluation settings, and introduce only modest training overhead.

Songjun Tu, Chengdong Xu, Qichao Zhang +5

Institute of Automation, Chinese Academy of Sciences · Pengcheng Laboratory · Sun Yat-Sen University +1

LLM Agent Skill Learning Agent Skill Learning Agentic RL Memory-Augmented Agents

Ask PDF

Abstract

Reinforcement Learning

Explore similar work

Structure-Conditioned Actor-Critic Branches for Quality-Diversity Reinforcement Learning

Exploiting Local Dynamics Regularity for Reusable Skills in Offline Hierarchical RL

Dynamic Dual-Granularity Skill Bank for Agentic RL