Saliency-Guided Representation with Consistency Policy Learning for Visual Unsupervised Reinforcement Learning

Jingbo Sun; Qichao Zhang; Songjun Tu; Xing Fang; Yupeng Zheng; Haoran Li; Ke Chen; Dongbin Zhao

cs.CVApr 7, 2026

Saliency-Guided Representation with Consistency Policy Learning for Visual Unsupervised Reinforcement Learning

Authors: Jingbo Sun, Qichao Zhang, Songjun Tu, Xing Fang, Yupeng Zheng, Haoran Li, Ke Chen, Dongbin Zhao

Organizations: SKL-MAIS, Institute of Automation, Chinese Academy of Sciences · Pengcheng Laboratory · School of Artificial Intelligence, University of Chinese A

Abstract

Zero-shot unsupervised reinforcement learning (URL) offers a promising direction for building generalist agents capable of generalizing to unseen tasks without additional supervision. Among existing approaches, successor representations (SR) have emerged as a prominent paradigm due to their effectiveness in structured, low-dimensional settings. However, SR methods struggle to scale to high-dimensional visual environments. Through empirical analysis, we identify two key limitations of SR in visual URL: (1) SR objectives often lead to suboptimal representations that attend to dynamics-irrelevant regions, resulting in inaccurate successor measures and degraded task generalization; and (2) these flawed representations hinder SR policies from modeling multi-modal skill-conditioned action distributions and ensuring skill controllability. To address these limitations, we propose Saliency-Guided Representation with Consistency Policy Learning (SRCP), a novel framework that improves zero-shot generalization of SR methods in visual URL. SRCP decouples representation learning from successor training by introducing a saliency-guided dynamics task to capture dynamics-relevant representations, thereby improving successor measure and task generalization. Moreover, it integrates a fast-sampling consistency policy with URL-specific classifier-free guidance and tailored training objectives to improve skill-conditioned policy modeling and controllability. Extensive experiments on 16 tasks across 4 datasets from the ExORL benchmark demonstrate that SRCP achieves state-of-the-art zero-shot generalization in visual URL and is compatible with various SR methods.

Figures & tables

Figure 1: Illustration of traditional RL and SR methods: (a) Traditional RL trains with task-specific rewards. (b) Traditional RL generalizes poorly to tasks with different rewards. (c) SR methods learn skill-conditioned skills without reward. (d) SR agents infer skills from minimal task information and generalize across tasks.

Figure 2: Generalization performance and attention analysis of prior methods and SCPL. (a) Task generalization performance of previous SR methods (HILP and FB) under low-dimensional state and high-dimensional visual inputs. (b) Comparison of saliency mask maps and attention heatmaps of prior methods and SRCP in the DMC Walker and Quadruped domains.

Figure 3: Value and performance analysis of methods in Walker Stand task. (a) Implicit reward and value estimations of methods across trajectories with varying returns; (b) Spearman correlations of reward and value with return, and performance across methods.

Figure 4: Trajectory comparison of methods with random and walking skills in the walker domain.

Figure 5: SRCP pretraining framework. SRCP first leverages unsupervised data to generate saliency maps that guide the learning of saliency-aware dynamic representations. The resulting encoder is shared between successor measure training and consistent policy learning, enabling effective successor measure and expressive policy behaviors, thereby enhancing generalization.

Domain	Task	AE [ 31 ]	CL [ 2 ]	LAP [ 45 ]	LRA-SR [ 45 ]	FDM [ 41 ]	FB [ 41 ]	HILP [ 27 ]	SRCP
Walker	Stand	$614\pm 85$	$255\pm 95$	$343\pm 128$	$246\pm 114$	$697\pm 120$	$234\pm 62$	$424\pm 58$	$\textbf{752}\pm\textbf{72}$
	Walk	$301\pm 122$	$72\pm 39$	$87\pm 41$	$88\pm 67$	$459\pm 91$	$100\pm 48$	$260\pm 151$	$\textbf{486}\pm\textbf{42}$
	Run	$127\pm 24$	$46\pm 17$	$60\pm 21$	$52\pm 25$	$195\pm 60$	$61\pm 25$	$112\pm 51$	$\textbf{204}\pm\textbf{21}$
	Flip	$224\pm 95$	$83\pm 37$	$71\pm 24$	$62\pm 32$	$251\pm 73$	$63\pm 16$	$154\pm 62$	$\textbf{369}\pm\textbf{46}$
	Average	$317$	$114$	$140$	$112$	$401$	$115$	$238$	453
Quadruped	Stand	$351\pm 136$	$105\pm 19$	$265\pm 123$	$390\pm 141$	$355\pm 80$	$270\pm 91$	$344\pm 150$	$\textbf{514}\pm\textbf{111}$

Table 1: Zero-shot generalization results across 16 tasks. Each score is averaged over 4 datasets and 4 seeds (i.e. 16 runs).

Domain	HILP	SRCP w/o SE	SRCP w/o CP	SRCP
Walker	$231$	$345$	$396$	$\mathbf{439}$
Quadruped	$305$	$352$	$406$	$\mathbf{485}$
Cheetah	$599$	$600$	$598$	$\mathbf{602}$
Jaco	$34$	$44$	$43$	$\mathbf{50}$

Table 2: Ablation study of SRCP on the RND dataset across 4 domains. Each score is the averaged return over 4 tasks per domain, with 4 random seeds per task (i.e., 16 runs).

Figure 6: Generalization performance of HILP with various representation learning methods.

Figure 7: 2D trajectories of methods in walker domain. Solid lines represent trajectories with walking skill, while dashed colored lines are trajectories with random skills.

Figure 8: Generalization performance of FB and SRCP(FB).

Quadruped	$\omega=0$	$\omega=2$	$\omega=3$	$\omega=4$	$\omega=6$
Stand	$370\pm 38$	$657\pm 45$	${703}\pm{22}$	$\textbf{724}\pm\textbf{21}$	$698\pm 28$
Walk	$184\pm 42$	$267\pm 32$	$\textbf{339}\pm\textbf{27}$	$337\pm 27$	$321\pm 14$
Run	$164\pm 20$	$289\pm 13$	$\textbf{361}\pm\textbf{24}$	$334\pm 14$	$360\pm 18$
Flip	$218\pm 45$	${452}\pm{45}$	$\textbf{535}\pm\textbf{14}$	$451\pm 18$	$442\pm 55$
Average	$234$	$416$	485	$462$	$455$

Table 3: Ablation study of parameter

\omega

in SRCP on Quadruped domain in RND dataset, with 4 random seeds per task.

Walker	$\beta=0$	$\beta=0.2$	$\beta=0.5$	$\beta=1$
Stand	$544\pm 20$	$550\pm 26$	$\textbf{671}\pm\textbf{51}$	$583\pm 18$
Walk	$354\pm 42$	$458\pm 19$	$\textbf{520}\pm\textbf{32}$	$484\pm 48$
Run	$156\pm 12$	$170\pm 35$	$\textbf{194}\pm\textbf{38}$	$169\pm 23$
Flip	$310\pm 15$	$315\pm 11$	$\textbf{369}\pm\textbf{25}$	$346\pm 5$
Average	$341$	$373$	439	$396$

Table 4: Ablation study of parameter

\beta

in SRCP on Walker domain in RND dataset, with 4 random seeds per task.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Zero-shot generalization performance on visual tasks. (a) Overall performance of each method evaluated on 4 datasets, 4 domains, and 4 tasks per domain using 4 random seeds (i.e., 256 values in total). (b) Performance in 4 domains.

Figure 10: Experimental results on the pixel-based ExORL benchmark for each task, aggregated over four datasets and four random seeds (i.e., 16 runs in total).

Dataset	Domain	Task	AE	CL	LAP	LRA-SR	FDM	FB	HILP	SRCP
RND	Walker	Stand	$546\pm 30$	$144\pm 4$	$296\pm 35$	$213\pm 49$	$557\pm 99$	$135\pm 12$	$386\pm 98$	$671\pm 51$
		Walk	$358\pm 12$	$36\pm 1$	$49\pm 5$	$147\pm 16$	$452\pm 52$	$48\pm 9$	$326\pm 31$	$520\pm 32$
		Run	$125\pm 7$	$25\pm 2$	$43\pm 2$	$82\pm 3$	$146\pm 60$	$31\pm 10$	$121\pm 20$	$194\pm 38$
		Flip	$265\pm 14$	$50\pm 3$	$35\pm 2$	$101\pm 14$	$282\pm 52$	$36\pm 15$	$92\pm 35$	$369\pm 25$
		Average	$324$	$64$	$106$	$147$	$359$	$63$	$231$	439
	Quadruped	Stand	$467\pm 31$	$114\pm 12$	$446\pm 18$	$476\pm 37$	$374\pm 85$	$257\pm 42$	$455\pm 84$	$703\pm 22$

Appendix

Table 5: Zero-shot Performance across Multiple Domains

Hyper-parameter	Setting
Pre-training frames	$5e^{5}$
Zero-shot selection frames	$1e^{4}$
RL replay buffer size	$1e6$
Frame stack	$3$
Action repeat	$1$
$z$ vector dimensions	$50$

Appendix

Table 6: Hyper-parameter settings.

Figure 11: Experimental results of training an encoder using physical states as supervision in the Walker domain. (a) Prediction loss of the encoder during training; (b) Comparison of HILP and FB across different tasks under state input, visual input, and HILP with state-supervised encoder.

Figure 12: Detailed attention heat map of SR methods.

Figure 13: Prediction loss of physical states in walker domain.

Domain	Task	HILP	SRCP w/o SE	SRCP w/o CM	SRCP
Walker	Stand	$496\pm 73$	$519\pm 12$	$658\pm 38$	$671\pm 51$
	Walk	$376\pm 52$	$423\pm 12$	$449\pm 75$	$520\pm 32$
	Run	$126\pm 8$	$145\pm 4$	$173\pm 21$	$194\pm 38$
	Flip	$232\pm 41$	$293\pm 5$	$303\pm 13$	$369\pm 25$
	Average	$308$	$345$	$396$	$439$
Quadruped	Stand	$327\pm 126$	$476\pm 9$	$595\pm 30$	$703\pm 22$

Appendix

Table 7: Full Ablation Study

Domain	Task	HILP	HILP-TACO	HILP-TACOP	HILP-SE	SRCP
Walker	Stand	$496\pm 73$	$388\pm 8$	$622\pm 24$	$658\pm 38$	$671\pm 51$
	Walk	$376\pm 52$	$111\pm 6$	$339\pm 14$	$449\pm 75$	$520\pm 32$
	Run	$126\pm 8$	$73\pm 3$	$135\pm 10$	$173\pm 21$	$194\pm 38$
	Flip	$232\pm 41$	$99\pm 8$	$199\pm 24$	$303\pm 13$	$369\pm 25$
	Average	$308$	$168$	$324$	$396$	$439$
Quadruped	Stand	$327\pm 126$	$234\pm 18$	$195\pm 16$	$595\pm 30$	$703\pm 22$

Appendix

Table 8: Zero-shot Performance of HILP Variants on the RND Dataset

Domain	Task	FB	SRCP(FB)
Walker	Stand	$304\pm 122$	$680\pm 14$
	Walk	$81\pm 14$	$478\pm 18$
	Run	$51\pm 13$	$152\pm 12$
	Flip	$66\pm 8$	$274\pm 13$
	Average	$126$	396
Quadruped	Stand	$421\pm 16$	$583\pm 21$

Appendix

Table 9: Zero-shot Performance of FB and SRCP(FB) on the APS Dataset

Walker	$\omega=0$	$\omega=2$	$\omega=3$	$\omega=4$	$\omega=6$
Stand	$176\pm 7$	$609\pm 40$	$\textbf{671}\pm\textbf{51}$	$498\pm 30$	$363\pm 16$
Walk	$42\pm 4$	$415\pm 40$	$\textbf{520}\pm\textbf{32}$	$464\pm 16$	$79\pm 13$
Run	$32\pm 6$	$136\pm 18$	$\textbf{194}\pm\textbf{38}$	$164\pm 11$	$70\pm 14$
Flip	$59\pm 7$	$\textbf{374}\pm\textbf{18}$	$369\pm 25$	$358\pm 18$	$206\pm 10$
Average	$77$	$384$	439	$371$	$180$

Appendix

Table 10: Ablation study of parameter

\omega

in SRCP on Walker domain in RND dataset, with 4 random seeds per task.

Walker	$\beta=0$	$\beta=0.2$	$\beta=0.5$	$\beta=1$
Stand	$544\pm 20$	$550\pm 26$	$\textbf{671}\pm\textbf{51}$	$583\pm 18$
Walk	$354\pm 42$	$458\pm 19$	$\textbf{520}\pm\textbf{32}$	$484\pm 48$
Run	$156\pm 12$	$170\pm 35$	$\textbf{194}\pm\textbf{38}$	$169\pm 23$
Flip	$310\pm 15$	$315\pm 11$	$\textbf{369}\pm\textbf{25}$	$346\pm 5$
Average	$341$	$373$	439	$396$

Appendix

Table 11: Ablation study of parameter

\beta

in SRCP on Walker domain in RND dataset, with 4 random seeds per task.

Metric	FB	HILP	SRCP
Training Time (h) $\downarrow$	23.1	24.2	24.4
Memory Usage (GB) $\downarrow$	9.6	8.8	9.1

Appendix

Table 12: Computational cost on an A800 GPU for 5M steps.

		Color Easy			Color Hard			Video Easy
Domain	Task	FB	HILP	SRCP	FB	HILP	SRCP	FB	HILP	SRCP
Walker	Stand	$243\pm 80$	$364\pm 89$	$639\pm 69$	$165\pm 66$	$282\pm 74$	$506\pm 58$	$127\pm 42$	$176\pm 32$	$277\pm 48$
	Walk	$62\pm 29$	$110\pm 54$	$423\pm 47$	$54\pm 27$	$73\pm 34$	$279\pm 45$	$47\pm 14$	$62\pm 13$	$162\pm 47$
	Run	$46\pm 25$	$55\pm 18$	$165\pm 31$	$39\pm 16$	$49\pm 21$	$113\pm 36$	$38\pm 8$	$40\pm 5$	$78\pm 10$
	Flip	$59\pm 22$	$98\pm 34$	$371\pm 42$	$51\pm 25$	$69\pm 36$	$156\pm 38$	$47\pm 11$	$56\pm 11$	$123\pm 44$
	Average	103	157	400	77	118	264	65	84	160

Appendix

Table 13: Performance on DMC-GB Walker.

Method	Reach Top Left	Reach Top Right	Reach Bottom Left	Reach Bottom Right	Average
FB	$14\pm 5$	$3\pm 1$	$1\pm 1$	$0\pm 0$	$5$
HILP	$471\pm 51$	$112\pm 19$	$36\pm 10$	$2\pm 1$	$155$
SRCP	$635\pm 95$	$166\pm 55$	$67\pm 18$	$10\pm 6$	220

Appendix

Table 14: Performance on Point Mass Maze navigation tasks.

Method	Stand $\uparrow$	Walk $\uparrow$	Run $\uparrow$	Jump $\uparrow$	Average $\uparrow$	Time (h) $\downarrow$
Diffusion (1 step)	$626\pm 50$	$308\pm 37$	$324\pm 18$	$397\pm 40$	414	24.7
Diffusion (40 steps)	$724\pm 30$	$357\pm 43$	$354\pm 17$	$531\pm 26$	492	64.8
SRCP	$703\pm 22$	$339\pm 27$	$361\pm 24$	$535\pm 14$	485	25.4

Appendix

Table 15: Comparison between diffusion models and SRCP.

Inference Transitions:	100	500	1k	5k	10k	20k
Stand	$828\pm 26$	$817\pm 19$	$823\pm 35$	$834\pm 34$	$830\pm 39$	$834\pm 32$
Walk	$466\pm 87$	$478\pm 35$	$508\pm 47$	$504\pm 46$	$504\pm 66$	$506\pm 31$
Run	$207\pm 42$	$216\pm 33$	$224\pm 22$	$220\pm 26$	$227\pm 16$	$238\pm 12$
Flip	$376\pm 86$	$418\pm 26$	$413\pm 24$	$434\pm 35$	$428\pm 75$	$420\pm 28$
Average	469	482	492	498	497	500

Appendix

Table 16: Effect of inference transition on SRCP performance.

Explore similar work

Learning Generalizable Skill Policy with Data-Efficient Unsupervised RL
Jul 1, 2026Jongchan Park, Seungjun Oh, Seungho Baek +1Reinforcement Learning Robot Policy Generalization
Department of Artificial Intelligence, Sungkyunkwan University, Suwon, Republic of Korea · Department of Computer Science and Engineering, Sungkyunkwan University, Suwon, Republic of Korea
Task-Relevant Representation Decoupling for Visual Reinforcement Learning Generalization
Jul 1, 2026Jinwen Wang, Youfang Lin, Xiaobo Hu +4Representation Learning Representation Learning for RL
Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, School of Computer Science & Technology, Beijing Jiaotong University, China · CSSC Intelligent Innovation Research Institute, China and Zhejiang University, China
VIGOR: Zero-Shot Visual Generalization via Latent-Space Consistency in Model-Based Reinforcement Learning
Oct 2, 2026Mingyu Park, Samyeul Noh, Hyun Myung +1Generalization in RL Model-Based RL
Robotics Program KAIST, ETRI · Electrical Engineering KAIST

Jul 1, 2026J/K move · Enter open · S save

Learning Generalizable Skill Policy with Data-Efficient Unsupervised RL

Jongchan Park, Seungjun Oh, Seungho Baek, Yusung Kim

Department of Artificial Intelligence, Sungkyunkwan University, Suwon, Republic of Korea · Department of Computer Science and Engineering, Sungkyunkwan University, Suwon, Republic of Korea

Reinforcement Learning Robot Policy Generalization Robust RL Data-Efficient Learning

Open paper AskPDF arXiv

Unsupervised Reinforcement Learning (URL) aims to pre-train scalable, skill-conditioned policies without extrinsic rewards, serving as a foundation for downstream control tasks. Despite recent progress, we argue that current off-policy URL methods are limited by two critical, overlooked bottlenecks: (1) non-stationary skill semantics and (2) brittle generalization. To address these challenges, we propose GenDa (Generalizable Data-efficient Agent), a unified framework for robust unsupervised reinforcement learning. First, we introduce a skill relabeling mechanism to mitigate non-stationarity and significantly improve data efficiency for pre-training. Second, we propose a Complementary Information Bottleneck (CIB), encouraging the learned skill policy to focus on ego-centric features and become robust to distribution shifts for downstream tasks. Through various experiments, we demonstrate that GenDa significantly enhances the scalability of URL with superior generalizability and data efficiency. Our code and videos are available at https://ihatebroccoli.github.io/official-GenDa.

Abstract

Explore similar work

Learning Generalizable Skill Policy with Data-Efficient Unsupervised RL

Task-Relevant Representation Decoupling for Visual Reinforcement Learning Generalization

VIGOR: Zero-Shot Visual Generalization via Latent-Space Consistency in Model-Based Reinforcement Learning