Saliency-Guided Representation with Consistency Policy Learning for Visual Unsupervised Reinforcement Learning

Jingbo Sun; Qichao Zhang; Songjun Tu; Xing Fang; Yupeng Zheng; Haoran Li; Ke Chen; Dongbin Zhao

cs.CVApr 7, 2026

Saliency-Guided Representation with Consistency Policy Learning for Visual Unsupervised Reinforcement Learning

Authors: Jingbo Sun, Qichao Zhang, Songjun Tu, Xing Fang, Yupeng Zheng, Haoran Li, Ke Chen, Dongbin Zhao

Organizations: SKL-MAIS, Institute of Automation, Chinese Academy of Sciences · Pengcheng Laboratory · School of Artificial Intelligence, University of Chinese A

Abstract

Zero-shot unsupervised reinforcement learning (URL) offers a promising direction for building generalist agents capable of generalizing to unseen tasks without additional supervision. Among existing approaches, successor representations (SR) have emerged as a prominent paradigm due to their effectiveness in structured, low-dimensional settings. However, SR methods struggle to scale to high-dimensional visual environments. Through empirical analysis, we identify two key limitations of SR in visual URL: (1) SR objectives often lead to suboptimal representations that attend to dynamics-irrelevant regions, resulting in inaccurate successor measures and degraded task generalization; and (2) these flawed representations hinder SR policies from modeling multi-modal skill-conditioned action distributions and ensuring skill controllability. To address these limitations, we propose Saliency-Guided Representation with Consistency Policy Learning (SRCP), a novel framework that improves zero-shot generalization of SR methods in visual URL. SRCP decouples representation learning from successor training by introducing a saliency-guided dynamics task to capture dynamics-relevant representations, thereby improving successor measure and task generalization. Moreover, it integrates a fast-sampling consistency policy with URL-specific classifier-free guidance and tailored training objectives to improve skill-conditioned policy modeling and controllability. Extensive experiments on 16 tasks across 4 datasets from the ExORL benchmark demonstrate that SRCP achieves state-of-the-art zero-shot generalization in visual URL and is compatible with various SR methods.

Figures & tables

Figure 1: Illustration of traditional RL and SR methods: (a) Traditional RL trains with task-specific rewards. (b) Traditional RL generalizes poorly to tasks with different rewards. (c) SR methods learn skill-conditioned skills without reward. (d) SR agents infer skills from minimal task information and generalize across tasks.

Figure 2: Generalization performance and attention analysis of prior methods and SCPL. (a) Task generalization performance of previous SR methods (HILP and FB) under low-dimensional state and high-dimensional visual inputs. (b) Comparison of saliency mask maps and attention heatmaps of prior methods and SRCP in the DMC Walker and Quadruped domains.

Figure 3: Value and performance analysis of methods in Walker Stand task. (a) Implicit reward and value estimations of methods across trajectories with varying returns; (b) Spearman correlations of reward and value with return, and performance across methods.

Figure 4: Trajectory comparison of methods with random and walking skills in the walker domain.

Figure 5: SRCP pretraining framework. SRCP first leverages unsupervised data to generate saliency maps that guide the learning of saliency-aware dynamic representations. The resulting encoder is shared between successor measure training and consistent policy learning, enabling effective successor measure and expressive policy behaviors, thereby enhancing generalization.

Domain	Task	AE [ 31 ]	CL [ 2 ]	LAP [ 45 ]	LRA-SR [ 45 ]	FDM [ 41 ]	FB [ 41 ]	HILP [ 27 ]	SRCP
Walker	Stand	$614\pm 85$	$255\pm 95$	$343\pm 128$	$246\pm 114$	$697\pm 120$	$234\pm 62$	$424\pm 58$	$\textbf{752}\pm\textbf{72}$
	Walk	$301\pm 122$	$72\pm 39$	$87\pm 41$	$88\pm 67$	$459\pm 91$	$100\pm 48$	$260\pm 151$	$\textbf{486}\pm\textbf{42}$
	Run	$127\pm 24$	$46\pm 17$	$60\pm 21$	$52\pm 25$	$195\pm 60$	$61\pm 25$	$112\pm 51$	$\textbf{204}\pm\textbf{21}$
	Flip	$224\pm 95$	$83\pm 37$	$71\pm 24$	$62\pm 32$	$251\pm 73$	$63\pm 16$	$154\pm 62$	$\textbf{369}\pm\textbf{46}$
	Average	$317$	$114$	$140$	$112$	$401$	$115$	$238$	453
Quadruped	Stand	$351\pm 136$	$105\pm 19$	$265\pm 123$	$390\pm 141$	$355\pm 80$	$270\pm 91$	$344\pm 150$	$\textbf{514}\pm\textbf{111}$

Table 1: Zero-shot generalization results across 16 tasks. Each score is averaged over 4 datasets and 4 seeds (i.e. 16 runs).

Domain	HILP	SRCP w/o SE	SRCP w/o CP	SRCP
Walker	$231$	$345$	$396$	$\mathbf{439}$
Quadruped	$305$	$352$	$406$	$\mathbf{485}$
Cheetah	$599$	$600$	$598$	$\mathbf{602}$
Jaco	$34$	$44$	$43$	$\mathbf{50}$

Table 2: Ablation study of SRCP on the RND dataset across 4 domains. Each score is the averaged return over 4 tasks per domain, with 4 random seeds per task (i.e., 16 runs).

Figure 6: Generalization performance of HILP with various representation learning methods.

Figure 7: 2D trajectories of methods in walker domain. Solid lines represent trajectories with walking skill, while dashed colored lines are trajectories with random skills.

Figure 8: Generalization performance of FB and SRCP(FB).

Quadruped	$\omega=0$	$\omega=2$	$\omega=3$	$\omega=4$	$\omega=6$
Stand	$370\pm 38$	$657\pm 45$	${703}\pm{22}$	$\textbf{724}\pm\textbf{21}$	$698\pm 28$
Walk	$184\pm 42$	$267\pm 32$	$\textbf{339}\pm\textbf{27}$	$337\pm 27$	$321\pm 14$
Run	$164\pm 20$	$289\pm 13$	$\textbf{361}\pm\textbf{24}$	$334\pm 14$	$360\pm 18$
Flip	$218\pm 45$	${452}\pm{45}$	$\textbf{535}\pm\textbf{14}$	$451\pm 18$	$442\pm 55$
Average	$234$	$416$	485	$462$	$455$

Table 3: Ablation study of parameter

\omega

in SRCP on Quadruped domain in RND dataset, with 4 random seeds per task.

Walker	$\beta=0$	$\beta=0.2$	$\beta=0.5$	$\beta=1$
Stand	$544\pm 20$	$550\pm 26$	$\textbf{671}\pm\textbf{51}$	$583\pm 18$
Walk	$354\pm 42$	$458\pm 19$	$\textbf{520}\pm\textbf{32}$	$484\pm 48$
Run	$156\pm 12$	$170\pm 35$	$\textbf{194}\pm\textbf{38}$	$169\pm 23$
Flip	$310\pm 15$	$315\pm 11$	$\textbf{369}\pm\textbf{25}$	$346\pm 5$
Average	$341$	$373$	439	$396$

Table 4: Ablation study of parameter

\beta

in SRCP on Walker domain in RND dataset, with 4 random seeds per task.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Zero-shot generalization performance on visual tasks. (a) Overall performance of each method evaluated on 4 datasets, 4 domains, and 4 tasks per domain using 4 random seeds (i.e., 256 values in total). (b) Performance in 4 domains.

Figure 10: Experimental results on the pixel-based ExORL benchmark for each task, aggregated over four datasets and four random seeds (i.e., 16 runs in total).

Dataset	Domain	Task	AE	CL	LAP	LRA-SR	FDM	FB	HILP	SRCP
RND	Walker	Stand	$546\pm 30$	$144\pm 4$	$296\pm 35$	$213\pm 49$	$557\pm 99$	$135\pm 12$	$386\pm 98$	$671\pm 51$
		Walk	$358\pm 12$	$36\pm 1$	$49\pm 5$	$147\pm 16$	$452\pm 52$	$48\pm 9$	$326\pm 31$	$520\pm 32$
		Run	$125\pm 7$	$25\pm 2$	$43\pm 2$	$82\pm 3$	$146\pm 60$	$31\pm 10$	$121\pm 20$	$194\pm 38$
		Flip	$265\pm 14$	$50\pm 3$	$35\pm 2$	$101\pm 14$	$282\pm 52$	$36\pm 15$	$92\pm 35$	$369\pm 25$
		Average	$324$	$64$	$106$	$147$	$359$	$63$	$231$	439
	Quadruped	Stand	$467\pm 31$	$114\pm 12$	$446\pm 18$	$476\pm 37$	$374\pm 85$	$257\pm 42$	$455\pm 84$	$703\pm 22$

Appendix

Table 5: Zero-shot Performance across Multiple Domains

Hyper-parameter	Setting
Pre-training frames	$5e^{5}$
Zero-shot selection frames	$1e^{4}$
RL replay buffer size	$1e6$
Frame stack	$3$
Action repeat	$1$
$z$ vector dimensions	$50$

Appendix

Table 6: Hyper-parameter settings.

Figure 11: Experimental results of training an encoder using physical states as supervision in the Walker domain. (a) Prediction loss of the encoder during training; (b) Comparison of HILP and FB across different tasks under state input, visual input, and HILP with state-supervised encoder.

Figure 12: Detailed attention heat map of SR methods.

Figure 13: Prediction loss of physical states in walker domain.

Domain	Task	HILP	SRCP w/o SE	SRCP w/o CM	SRCP
Walker	Stand	$496\pm 73$	$519\pm 12$	$658\pm 38$	$671\pm 51$
	Walk	$376\pm 52$	$423\pm 12$	$449\pm 75$	$520\pm 32$
	Run	$126\pm 8$	$145\pm 4$	$173\pm 21$	$194\pm 38$
	Flip	$232\pm 41$	$293\pm 5$	$303\pm 13$	$369\pm 25$
	Average	$308$	$345$	$396$	$439$
Quadruped	Stand	$327\pm 126$	$476\pm 9$	$595\pm 30$	$703\pm 22$

Appendix

Table 7: Full Ablation Study

Domain	Task	HILP	HILP-TACO	HILP-TACOP	HILP-SE	SRCP
Walker	Stand	$496\pm 73$	$388\pm 8$	$622\pm 24$	$658\pm 38$	$671\pm 51$
	Walk	$376\pm 52$	$111\pm 6$	$339\pm 14$	$449\pm 75$	$520\pm 32$
	Run	$126\pm 8$	$73\pm 3$	$135\pm 10$	$173\pm 21$	$194\pm 38$
	Flip	$232\pm 41$	$99\pm 8$	$199\pm 24$	$303\pm 13$	$369\pm 25$
	Average	$308$	$168$	$324$	$396$	$439$
Quadruped	Stand	$327\pm 126$	$234\pm 18$	$195\pm 16$	$595\pm 30$	$703\pm 22$

Appendix

Table 8: Zero-shot Performance of HILP Variants on the RND Dataset

Domain	Task	FB	SRCP(FB)
Walker	Stand	$304\pm 122$	$680\pm 14$
	Walk	$81\pm 14$	$478\pm 18$
	Run	$51\pm 13$	$152\pm 12$
	Flip	$66\pm 8$	$274\pm 13$
	Average	$126$	396
Quadruped	Stand	$421\pm 16$	$583\pm 21$

Appendix

Table 9: Zero-shot Performance of FB and SRCP(FB) on the APS Dataset

Walker	$\omega=0$	$\omega=2$	$\omega=3$	$\omega=4$	$\omega=6$
Stand	$176\pm 7$	$609\pm 40$	$\textbf{671}\pm\textbf{51}$	$498\pm 30$	$363\pm 16$
Walk	$42\pm 4$	$415\pm 40$	$\textbf{520}\pm\textbf{32}$	$464\pm 16$	$79\pm 13$
Run	$32\pm 6$	$136\pm 18$	$\textbf{194}\pm\textbf{38}$	$164\pm 11$	$70\pm 14$
Flip	$59\pm 7$	$\textbf{374}\pm\textbf{18}$	$369\pm 25$	$358\pm 18$	$206\pm 10$
Average	$77$	$384$	439	$371$	$180$

Appendix

Table 10: Ablation study of parameter

\omega

in SRCP on Walker domain in RND dataset, with 4 random seeds per task.

Walker	$\beta=0$	$\beta=0.2$	$\beta=0.5$	$\beta=1$
Stand	$544\pm 20$	$550\pm 26$	$\textbf{671}\pm\textbf{51}$	$583\pm 18$
Walk	$354\pm 42$	$458\pm 19$	$\textbf{520}\pm\textbf{32}$	$484\pm 48$
Run	$156\pm 12$	$170\pm 35$	$\textbf{194}\pm\textbf{38}$	$169\pm 23$
Flip	$310\pm 15$	$315\pm 11$	$\textbf{369}\pm\textbf{25}$	$346\pm 5$
Average	$341$	$373$	439	$396$

Appendix

Table 11: Ablation study of parameter

\beta

in SRCP on Walker domain in RND dataset, with 4 random seeds per task.

Metric	FB	HILP	SRCP
Training Time (h) $\downarrow$	23.1	24.2	24.4
Memory Usage (GB) $\downarrow$	9.6	8.8	9.1

Appendix

Table 12: Computational cost on an A800 GPU for 5M steps.

		Color Easy			Color Hard			Video Easy
Domain	Task	FB	HILP	SRCP	FB	HILP	SRCP	FB	HILP	SRCP
Walker	Stand	$243\pm 80$	$364\pm 89$	$639\pm 69$	$165\pm 66$	$282\pm 74$	$506\pm 58$	$127\pm 42$	$176\pm 32$	$277\pm 48$
	Walk	$62\pm 29$	$110\pm 54$	$423\pm 47$	$54\pm 27$	$73\pm 34$	$279\pm 45$	$47\pm 14$	$62\pm 13$	$162\pm 47$
	Run	$46\pm 25$	$55\pm 18$	$165\pm 31$	$39\pm 16$	$49\pm 21$	$113\pm 36$	$38\pm 8$	$40\pm 5$	$78\pm 10$
	Flip	$59\pm 22$	$98\pm 34$	$371\pm 42$	$51\pm 25$	$69\pm 36$	$156\pm 38$	$47\pm 11$	$56\pm 11$	$123\pm 44$
	Average	103	157	400	77	118	264	65	84	160

Appendix

Table 13: Performance on DMC-GB Walker.

Method	Reach Top Left	Reach Top Right	Reach Bottom Left	Reach Bottom Right	Average
FB	$14\pm 5$	$3\pm 1$	$1\pm 1$	$0\pm 0$	$5$
HILP	$471\pm 51$	$112\pm 19$	$36\pm 10$	$2\pm 1$	$155$
SRCP	$635\pm 95$	$166\pm 55$	$67\pm 18$	$10\pm 6$	220

Appendix

Table 14: Performance on Point Mass Maze navigation tasks.

Method	Stand $\uparrow$	Walk $\uparrow$	Run $\uparrow$	Jump $\uparrow$	Average $\uparrow$	Time (h) $\downarrow$
Diffusion (1 step)	$626\pm 50$	$308\pm 37$	$324\pm 18$	$397\pm 40$	414	24.7
Diffusion (40 steps)	$724\pm 30$	$357\pm 43$	$354\pm 17$	$531\pm 26$	492	64.8
SRCP	$703\pm 22$	$339\pm 27$	$361\pm 24$	$535\pm 14$	485	25.4

Appendix

Table 15: Comparison between diffusion models and SRCP.

Inference Transitions:	100	500	1k	5k	10k	20k
Stand	$828\pm 26$	$817\pm 19$	$823\pm 35$	$834\pm 34$	$830\pm 39$	$834\pm 32$
Walk	$466\pm 87$	$478\pm 35$	$508\pm 47$	$504\pm 46$	$504\pm 66$	$506\pm 31$
Run	$207\pm 42$	$216\pm 33$	$224\pm 22$	$220\pm 26$	$227\pm 16$	$238\pm 12$
Flip	$376\pm 86$	$418\pm 26$	$413\pm 24$	$434\pm 35$	$428\pm 75$	$420\pm 28$
Average	469	482	492	498	497	500

Appendix

Table 16: Effect of inference transition on SRCP performance.

Explore similar work

Jul 1, 2026

Learning Generalizable Skill Policy with Data-Efficient Unsupervised RL

Unsupervised Reinforcement Learning (URL) aims to pre-train scalable, skill-conditioned policies without extrinsic rewards, serving as a foundation for downstream control tasks. Despite recent progress, we argue that current off-policy URL methods are limited by two critical, overlooked bottlenecks: (1) non-stationary skill semantics and (2) brittle generalization. To address these challenges, we propose GenDa (Generalizable Data-efficient Agent), a unified framework for robust unsupervised reinforcement learning. First, we introduce a skill relabeling mechanism to mitigate non-stationarity and significantly improve data efficiency for pre-training. Second, we propose a Complementary Information Bottleneck (CIB), encouraging the learned skill policy to focus on ego-centric features and become robust to distribution shifts for downstream tasks. Through various experiments, we demonstrate that GenDa significantly enhances the scalability of URL with superior generalizability and data efficiency. Our code and videos are available at https://ihatebroccoli.github.io/official-GenDa.

Jongchan Park, Seungjun Oh, Seungho Baek +1

Department of Artificial Intelligence, Sungkyunkwan University, Suwon, Republic of Korea · Department of Computer Science and Engineering, Sungkyunkwan University, Suwon, Republic of Korea

Reinforcement Learning Robot Policy Generalization Robust RL Data-Efficient Learning

Ask PDF

Jul 1, 2026

Task-Relevant Representation Decoupling for Visual Reinforcement Learning Generalization

Visual Reinforcement Learning (VRL) has achieved considerable success in solving control tasks. However, generalizing learned policies to new environments remains a major challenge, as agents often overfit to task-irrelevant features in the training environment. To solve this problem, we introduce the concept of decoupling observations into task-relevant and task-irrelevant representations. Building on this idea, we propose a self-supervised Task-Relevant Representation Decoupling (T2RD) algorithm for VRL. This algorithm consists of three components: task-relevant representation consistency, cross-reconstruction, and cross-dynamic prediction. The first two components achieve the decoupling of content and style features, but the resulting content representations are not necessarily task-relevant. To further refine task-relevant features from content representations, we design the third component that introduces dynamic prediction. T2RD achieves State-Of-The-Art (SOTA) generalization performance and sample efficiency in the DeepMind Control Suite and Robotic Manipulation tasks.

Jinwen Wang, Youfang Lin, Xiaobo Hu +4

Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, School of Computer Science & Technology, Beijing Jiaotong University, China · CSSC Intelligent Innovation Research Institute, China and Zhejiang University, China

Representation Learning Representation Learning for RL Visual RL Generalization in RL Self-Supervised Visual Representation Learning

Ask PDF

Oct 2, 2026

VIGOR: Zero-Shot Visual Generalization via Latent-Space Consistency in Model-Based Reinforcement Learning

Model-based reinforcement learning (MBRL) achieves strong sample efficiency by planning within learned latent dynamics, yet its performance degrades substantially under unseen visual distractions such as background variations, lighting changes, or camera shifts. Unlike model-free RL, where encoder perturbations affect only single-step predictions, MBRL suffers from a two-level vulnerability: visual distractions first push encoder outputs out of distribution, and these errors then compound through recursive latent rollouts over the planning horizon. We propose visual generalization via latent-space consistency in model-based RL (VIGOR), a framework that enables zero-shot generalization to unseen visual distractions while retaining the sample efficiency of its MBRL backbone. VIGOR integrates three interdependent components: (i) asymmetric weak-to-strong augmentation, which pairs weak-only and weak-to-strong latent views within a single batch; (ii) dynamics-level consistency, which enforces augmentation-invariant transition predictions through direct latent regression; and (iii) encoder-level stabilization, which prevents encoder drift under the cross-augmentation supervision imposed by dynamics-level consistency. Evaluations on the DeepMind Control Suite (DMC) and Robosuite show that VIGOR outperforms state-of-the-art model-free and model-based baselines, surpassing the second-best baseline by 3.4% on DMC and 43.6% on Robosuite. Ablations further show that VIGOR's robustness is augmentation-agnostic: replacing the default augmentation with alternatives from distinct perturbation families preserves strong generalization, confirming that latent-space consistency, not the augmentation choice, drives robustness.

Mingyu Park, Samyeul Noh, Hyun Myung +1

Robotics Program KAIST, ETRI · Electrical Engineering KAIST

Generalization in RL Model-Based RL Robust RL Consistency Regularization

Ask PDF