Learning Transferable Skills using Goal-Conditioned Bisimulation

Mohammad Amin Abbasfar; Farbod Azimmohseni; Mohammad Hossein Rohban

cs.LGSep 30, 2026

Learning Transferable Skills using Goal-Conditioned Bisimulation

Authors: Mohammad Amin Abbasfar, Farbod Azimmohseni, Mohammad Hossein Rohban

Organizations: Department of Computer Engineering Sharif University of Technology

Abstract

Unsupervised skill discovery has emerged as a promising approach for leveraging reward-free datasets to pretrain general-purpose policies. However, current skill discovery methods either require access to expert data or exhibit limited generalization, failing to transfer effectively to previously unseen layouts. A key challenge is to learn representations that capture the temporal structure of the environment while remaining robust to variations across layouts. To address this issue, we present an objective for learning action-aware temporal representations that satisfy the functional equivariance property while preserving the local temporal structure of the environment. Building upon this embedding, we further propose unsupervised skill discovery using bisimulation, which learns transferable skills by conditioning the behavior of skills exclusively on the subset of state features that directly affect their execution. This enforces invariant behavior across different layouts, enabling skills to transfer effectively to other configurations. Finally, through comprehensive empirical evaluations, we show that skills learned in a given environment can be effectively applied to solve downstream tasks in various environment layouts, demonstrating strong out-of-distribution generalization.

Figures & tables

Panel of Figure 1 — Figure 1: (Left) HILP’s skills overfit to the training layout despite spanning the state space, limiting their ability to generalize to new layouts. (Right) USB+ATR extracts more fine-grained and layout-agnostic skills, which transfer effectively to unseen layouts. 3 3 3 This figure is adapted from Park et al. (2024a) .

Panel of Figure 2 — Figure 2: (a) We first train a state-goal pair encoder $\psi$ using goal-conditioned bisimulation, which encodes pairs with similar underlying action sequences close to each other. (b) Next, we train an action-aware temporal representation $\phi$ that preserves local temporal dependencies while respecting functional equivariance enforced by $\psi$ . (c) Based on the similarity between the local structures shaped in representation $\phi$ , we train a representation $\beta$ using bisimulation to encode dynamically similar states to spatially close latent vectors. (d) Lastly, we train the skill policy using HILP directional state-spanning rewards by conditioning these skills on the $\beta$ latents. Unlike HILPs, which exploit the layout structure to greedily maximize the reward (dashed line), USB instead learns monotonic behaviors across dynamically similar states, making the resulting skills transferable to unseen layouts (solid line).

Task			Method
Environment	Dataset	Size	Flat	OPAL	HILP	USB+ATR
pointmaze	navigate	medium	$0.0\scriptstyle\pm 0.0$	$0.0\scriptstyle\pm 0.0$	$40.0\scriptstyle\pm 7.4$	$\mathbf{91.3\scriptstyle\pm 4.2}$
	navigate	large	$0.0\scriptstyle\pm 0.0$	$30.7\scriptstyle\pm 6.9$	$44.4\scriptstyle\pm 7.5$	$\mathbf{100.0\scriptstyle\pm 0.0}$
	stitch	medium	$0.0\scriptstyle\pm 0.0$	$2.1\scriptstyle\pm 2.1$	$11.1\scriptstyle\pm 4.7$	$\mathbf{75.6\scriptstyle\pm 6.5}$
	stitch	large	$0.0\scriptstyle\pm 0.0$	$20.0\scriptstyle\pm 6.0$	$28.9\scriptstyle\pm 6.8$	$\mathbf{81.9\scriptstyle\pm 5.8}$
antmaze	navigate	medium	$2.1\scriptstyle\pm 0.7$	$\mathbf{99.0\scriptstyle\pm 0.2}$	$57.7\scriptstyle\pm 6.3$	${92.3\scriptstyle\pm 1.9}$

Table 1: Evaluation of transferability to new layouts with a high-level controller. The average success rates with standard errors are calculated across 3 layouts and 5 seeds for each (15 runs in total).

Task			Method
Environment	Dataset	Size	GCIQL	QRL	CRL	HIQL	HILP	USB+ATR
pointmaze	navigate	medium	$\mathbf{89.1\scriptstyle\pm 4.2}$	$\mathbf{90.7\scriptstyle\pm 5.5}$	${0.0\scriptstyle\pm 0.0}$	${0.0\scriptstyle\pm 0.0}$	${0.0\scriptstyle\pm 0.0}$	${36.9\scriptstyle\pm 9.0}$
	navigate	large	${72.2\scriptstyle\pm 9.1}$	$\mathbf{80.0\scriptstyle\pm 8.2}$	${0.0\scriptstyle\pm 0.0}$	${0.5\scriptstyle\pm 0.5}$	${0.0\scriptstyle\pm 0.0}$	${42.0\scriptstyle\pm 9.4}$
	stitch	medium	$\mathbf{99.3\scriptstyle\pm 0.6}$	${57.0\scriptstyle\pm 9.5}$	${0.0\scriptstyle\pm 0.0}$	${0.0\scriptstyle\pm 0.0}$	${0.0\scriptstyle\pm 0.0}$	${33.7\scriptstyle\pm 9.3}$
	stitch	large	$\mathbf{99.7\scriptstyle\pm 0.2}$	${80.0\scriptstyle\pm 8.2}$	${0.0\scriptstyle\pm 0.0}$	${14.2\scriptstyle\pm 6.8}$	${0.0\scriptstyle\pm 0.0}$	${52.5\scriptstyle\pm 9.8}$
antmaze	navigate	medium	${6.3\scriptstyle\pm 2.3}$	${0.4\scriptstyle\pm 0.4}$	${28.5\scriptstyle\pm 5.8}$	${5.0\scriptstyle\pm 1.0}$	${18.2\scriptstyle\pm 5.1}$	$\mathbf{75.7\scriptstyle\pm 2.2}$

Table 2: Evaluation of transferability to new layouts using test-time planning. The average success rates with standard errors are calculated across 5 layouts and 5 seeds for each (25 runs in total).

Method	Task Generalization	Layout Generalization
HILP+UTR	$\mathbf{71.7\scriptstyle\pm 4.8}$	$44.9\scriptstyle\pm 5.4$
HILP+ATR	$57.6\scriptstyle\pm 5.3$	$25.1\scriptstyle\pm 4.3$
USB+UTR	$0.0\scriptstyle\pm 0.0$	$0.0\scriptstyle\pm 0.0$
USB+GCB	$0.0\scriptstyle\pm 0.0$	$0.0\scriptstyle\pm 0.0$
USB+Raw	$44.2\scriptstyle\pm 5.5$	$28.4\scriptstyle\pm 4.1$
USB+ATR	${66.4\scriptstyle\pm 6.4}$	$\mathbf{83.9\scriptstyle\pm 2.9}$

Table 3: Ablation study of the proposed components in the hierarchical setup. We evaluate the task and layout generalization capabilities of each variant in the antmaze-large-navigate environment. Average success rates with standard errors are reported across 3 layouts/tasks, and 5 random seeds per setting (15 runs in total).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: We visualize the universal temporal representation learned by HILP and ATR on a sample maze layout in two environments. ATR’s functional equivariance preserves the environment’s structural layout and aligns similar action sequences along parallel directions. In contrast, HILP distorts this structure by encoding global temporal distances and learns consistently changing behaviors.

Figure 4: Visualization of the $\beta$ representation in the antmaze-medium-navigate environment. This embedding maps states with similar underlying dynamics to nearby representations while disregarding their temporal proximity.

Task			Method
Environment	Dataset	Size	GCIQL	QRL	CRL	HIQL	HILP	USB+ATR
pointmaze	navigate	medium	${53.0\scriptstyle\pm 9.9}$	$\mathbf{94.3\scriptstyle\pm 4.3}$	${16.0\scriptstyle\pm 6.6}$	${87.6\scriptstyle\pm 6.2}$	${51.9\scriptstyle\pm 9.9}$	${66.1\scriptstyle\pm 8.6}$
	navigate	large	${30.1\scriptstyle\pm 8.3}$	$\mathbf{84.0\scriptstyle\pm 7.5}$	${34.9\scriptstyle\pm 9.5}$	${54.9\scriptstyle\pm 8.8}$	${36.5\scriptstyle\pm 9.3}$	${70.1\scriptstyle\pm 8.1}$
	stitch	medium	${18.9\scriptstyle\pm 5.9}$	${79.0\scriptstyle\pm 7.2}$	${7.5\scriptstyle\pm 5.2}$	$\mathbf{85.0\scriptstyle\pm 4.6}$	${22.2\scriptstyle\pm 7.9}$	${76.4\scriptstyle\pm 8.3}$
	stitch	large	${34.2\scriptstyle\pm 8.8}$	$\mathbf{87.4\scriptstyle\pm 6.6}$	${0.0\scriptstyle\pm 0.0}$	${10.2\scriptstyle\pm 5.8}$	${58.4\scriptstyle\pm 9.4}$	${76.0\scriptstyle\pm 8.7}$
antmaze	navigate	medium	${78.3\scriptstyle\pm 2.2}$	${89.1\scriptstyle\pm 1.8}$	$\mathbf{95.5\scriptstyle\pm 0.7}$	$\mathbf{95.0\scriptstyle\pm 0.7}$	${77.0\scriptstyle\pm 3.6}$	${83.8\scriptstyle\pm 3.7}$

Appendix

Table 4: Evaluation of in-distribution task generalization using test-time planning. The average success rates with standard errors are calculated across 5 tasks and 5 seeds for each (25 runs in total).

Normalizing Term	Success Rate
$\psi(s_{i},s_{i})$	$0.0\scriptstyle\pm 0.0$
$\psi(s_{i}^{\prime},s_{i}^{\prime})$ ( Hansen-Estruch et al., 2022 )	$0.4\scriptstyle\pm 0.2$
$\mathbb{E}_{s\in\mathcal{S}}[\psi(s,s)]$ (Ours)	$\mathbf{66.3\scriptstyle\pm 3.9}$

Appendix

Table 5: Ablation study of the

\psi

normalization term in the test-time planning setup. We evaluate the layout generalization of USB+ATR using different choices of the normalization term in the antmaze-large-navigate environment. Average success rates with standard errors are reported across 5 layouts and 5 random seeds per setting (25 runs in total).

Model Size	Success Rate
Small	$59.1\scriptstyle\pm 6.4$
Medium	$69.2\scriptstyle\pm 5.5$
Large (Ours)	$\mathbf{83.9\scriptstyle\pm 2.9}$

Appendix

Table 6: Ablation study of sensitivity to the dynamics model in the hierarchical setup. We evaluate the layout generalization of USB+ATR using dynamics models with different capacities in the antmaze-large-navigate environment. Average success rates with standard errors are reported across 3 layouts and 5 random seeds per setting (15 runs in total).

Panel of Figure 5 — Figure 5: Randomly generated layouts in two sizes, used for evaluating the transferability of skills across different layouts

Hyperparameter	Value
# gradient steps	$10^{6}$
Learning rate	$0.0003$
Optimizer	Adam ( Kingma and Ba, 2015 )
Batch size	$1024$
MLP hidden dimensions	$(512,512,512)$
Target smoothing coefficient $\tau$	$0.005$

Appendix

Table 7: Hyperparameters for USB+ATR

Explore similar work

Learning Generalizable Skill Policy with Data-Efficient Unsupervised RL
Jul 1, 2026Jongchan Park, Seungjun Oh, Seungho Baek +1Unsupervised Skill Discovery Goal-Conditioned Value Function
Department of Artificial Intelligence, Sungkyunkwan University, Suwon, Republic of Korea · Department of Computer Science and Engineering, Sungkyunkwan University, Suwon, Republic of Korea
Skill0.5: Joint Skill Internalization and Utilization for Out-of-Distribution Generalization in Agentic Reinforcement Learning
May 27, 2026Jiapeng Zhu, Jianxiang Yu, Yibo Zhao +5Agentic Reinforcement Learning Offline Reinforcement Learning
1East China Normal University, 2Meituan Longcat Team · 1East China Normal University · 2Meituan Longcat Team
Decompose and Recompose: Reasoning New Skills from Existing Abilities for Cross-Task Robotic Manipulation
May 2, 2026Xitie Zhang, Aming Wu, Yahong HanHuman-To-Robot Transfer Robotic Manipulation
School of Artificial Intelligence, College of Intelligence and Computing, Tianjin University, China · School of Computer Science and Information Engineering, Hefei University of Technology, China

Jul 1, 2026, cs.LGJ/K move · Enter open · S save

Learning Generalizable Skill Policy with Data-Efficient Unsupervised RL

Jongchan Park, Seungjun Oh, Seungho Baek, Yusung Kim

Department of Artificial Intelligence, Sungkyunkwan University, Suwon, Republic of Korea · Department of Computer Science and Engineering, Sungkyunkwan University, Suwon, Republic of Korea

Unsupervised Skill Discovery Goal-Conditioned Value Function Skills Agentic Framework Fragility

Open paper AskPDF arXiv

Unsupervised Reinforcement Learning (URL) aims to pre-train scalable, skill-conditioned policies without extrinsic rewards, serving as a foundation for downstream control tasks. Despite recent progress, we argue that current off-policy URL methods are limited by two critical, overlooked bottlenecks: (1) non-stationary skill semantics and (2) brittle generalization. To address these challenges, we propose GenDa (Generalizable Data-efficient Agent), a unified framework for robust unsupervised reinforcement learning. First, we introduce a skill relabeling mechanism to mitigate non-stationarity and significantly improve data efficiency for pre-training. Second, we propose a Complementary Information Bottleneck (CIB), encouraging the learned skill policy to focus on ego-centric features and become robust to distribution shifts for downstream tasks. Through various experiments, we demonstrate that GenDa significantly enhances the scalability of URL with superior generalizability and data efficiency. Our code and videos are available at https://ihatebroccoli.github.io/official-GenDa.

Abstract

Explore similar work

Learning Generalizable Skill Policy with Data-Efficient Unsupervised RL

Skill0.5: Joint Skill Internalization and Utilization for Out-of-Distribution Generalization in Agentic Reinforcement Learning

Decompose and Recompose: Reasoning New Skills from Existing Abilities for Cross-Task Robotic Manipulation