cs.LGOct 14, 2026

Hyperspherical Contrastive Representations Unlock Planning and Exploration

Authors: Jonghae Park, Sungil Son, Jonghun Shin, Junseo Lee, H. Jin Kim, Benjamin Eysenbach

Organizations: Seoul National University · Princeton University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We develop a self-supervised reinforcement learning algorithm that learns to build towers of up to eight cubes from scratch, without prior data, rewards, demonstrations, or pretrained models. Our method, Contrastive Planning via Spherical Linear Interpolation (C-Slerp), uses contrastive representations as reusable building blocks and repeatedly composes them to infer a sequence of intermediate waypoints. Under our hyperspherical parameterization, the learned representations allow intermediate waypoints to be inferred in closed form through spherical linear interpolation, without training a separate high-level policy or performing graph search. Importantly, the representation geometry also tells us how to adjust the number of waypoints based on the difficulty of each goal-reaching problem. Experiments show that our method succeeds on long-horizon navigation and robotic manipulation tasks, including settings where strong baselines fail to achieve a single success. Ablations further show that using a fixed number of waypoints substantially degrades performance, highlighting the importance of adapting the plan to task difficulty.

Figures & tables

Figure 1: Our paper studies long-horizon, sparse-reward tasks, including eight-cube stacking. Given only the current and goal states, the agent aims to manipulate the eight cubes into the goal state without receiving any external reward along the way. Strong exploration methods ( Pathak et al., 2017 ; Burda et al., 2018 ) fail to discover a single successful state in one billion environment steps.
Figure 2: Geometric interpretation of C-Slerp. The number of waypoints KK is selected using the alignment between (R⊤)k+1ψ(s)(R^{\top})^{k+1}\psi(s) and ψ(g)\psi(g) . Given the selected KK , waypoint mean directions are obtained by spherical interpolation from ψ(s)\psi(s) to RK+1ψ(g)R^{K+1}\psi(g) , followed by mapping back via (R⊤)i(R^{\top})^{i} . The policy then pursues the furthest waypoint whose alignment with ϕ(s,a)\phi(s,a) meets the threshold ϵ\epsilon .
Figure 3: C-Slerp solves long-horizon tasks where baselines never achieve a single success. Learning curves show success rates on BuilderBench manipulation tasks ( top ) and JaxGCRL locomotion tasks ( bottom ). C-Slerp achieves the highest success rates on most tasks across both benchmarks and is the only method to succeed on all cube-stacking tasks with a gripper.
Figure 4
Figure 6: Evaluation of C-Slerp in single-goal environments. To assess exploration capability, we evaluate C-Slerp with a single fixed goal throughout training. The environments and their fixed goals are shown in (a), and success rates for C-Slerp and the baselines are presented in (b).
Figure 7: Adaptive planning in C-Slerp. The top row shows successive states in the 5-tower task with a gripper. The bottom row shows spherical PCA ( Liu et al., 2019 ) projections of the latent trajectory, inferred waypoints, and selected subgoal. As the agent approaches the goal, it replans by adjusting the number of waypoints and selecting a waypoint to pursue. We provide complete rollout visualizations in Figures 15 and 16 and videos on our project page .
Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Ablation on the threshold ϵ\epsilon .
Figure 9: Ablation on learning the rotation matrix RR .
Figure 10: Ablation on network depth.
Figure 11: Ablation on representation dimension dd .
Figure 12: Comparison with CRL with larger discount factors γ\gamma .
Figure 13: Evolution of KK and i∗i^{*} over time in JaxGCRL environments. We plot the mean and sampled values of K+1K+1 , together with the selected waypoint index i∗i^{*} , against environment steps. As the agent approaches the goal, the predicted number of waypoints decreases.
Figure 14: Evolution of KK and i∗i^{*} over time in BuilderBench environments. We plot the mean and sampled values of K+1K+1 , together with the selected waypoint index i∗i^{*} , against environment steps. As the agent approaches the goal, the predicted number of waypoints decreases.
Figure 14
Figure 15
Figure 15: Visualization of C-Slerp rollouts and corresponding latent representations in JaxGCRL. For each environment, the top row shows successive states, and the bottom row shows spherical PCA ( Liu et al., 2019 ) projections of the corresponding state representations and inferred waypoints.
Figure 17
Figure 16: Visualization of C-Slerp rollouts and corresponding latent representations in BuilderBench. For each environment, the top row shows successive states, and the bottom row shows spherical PCA ( Liu et al., 2019 ) projections of the corresponding state representations and inferred waypoints.
Figure 19
Figure 17: Visualization of waypoint representations in maze environments. We visualize the inferred latent waypoints z1:Kz_{1:K} and the corresponding score heatmaps. For each environment, the top row shows the (x,y)(x,y) positions corresponding to the inferred latent waypoints, where denotes a candidate waypoint ziz_{i} , highlights the selected waypoint zi∗z_{i^{*}} , and represents the goal. The bottom row shows heatmaps of fL(s,a,s′)=ϕ(s,a)⊤ψ(s′)f_{L}(s,a,s^{\prime})=\phi(s,a)^{\top}\psi(s^{\prime}) , with the current state marked by and the goal indicated by .
Figure 18: Visualization of benchmark environments. The top row shows the BuilderBench manipulation tasks, and the bottom row shows the JaxGCRL navigation tasks. The goal used for evaluation in each JaxGCRL environment is indicated by . We provide videos demonstrating C-Slerp on all benchmark tasks on our project page .
ParameterJaxGCRLBuilderBench
Actor learning rate10−410^{-4}10−410^{-4}
Representation learning rate10−410^{-4}10−410^{-4}
Actor entropy learning rate3×10−43{\times}10^{-4}3×10−43{\times}10^{-4}
Waypoint entropy learning rate3×10−33{\times}10^{-3}3×10−33{\times}10^{-3}
Batch size51251210241024
Representation dimension dd128128512512
Appendix
Table 1: Learning hyperparameters.
MethodParameterSearch space
CRLActor learning rate{3×10−4,2×10−4,10−4}\{3\times 10^{-4},2\times 10^{-4},10^{-4}\}
 Critic learning rate{3×10−4,2×10−4,10−4}\{3\times 10^{-4},2\times 10^{-4},10^{-4}\}
RISActor learning rate{3×10−5,10−4,3×10−4}\{3\times 10^{-5},10^{-4},3\times 10^{-4}\}
 Critic learning rate{10−4,3×10−4,10−3}\{10^{-4},3\times 10^{-4},10^{-3}\}
 KL coefficient{0.1,0.3}\{0.1,0.3\}
 Prior samples{3,10}\{3,10\}
Appendix
Table 2: Hyperparameter search spaces for baseline tuning.
ParameterJaxGCRLBuilderBench
Actor learning rate3×10−43\times 10^{-4}3×10−43\times 10^{-4}
Critic learning rate3×10−43\times 10^{-4}3×10−43\times 10^{-4}
Energy functionL2 normL2 norm
Appendix
Table 3: CRL-specific parameters.
ParameterJaxGCRLBuilderBench
Actor/subgoal learning rate10−410^{-4}10−410^{-4}
Critic learning rate10−410^{-4}10−310^{-3}
KL coefficient0.10.1
Bottleneck temperature0.10.1
Prior samples1010
Appendix
Table 4: RIS-specific parameters.
ParameterJaxGCRLBuilderBench
Actor/critic learning rate10−410^{-4}10−410^{-4}
Entropy learning rate3×10−43\times 10^{-4}3×10−43\times 10^{-4}
HER probability0.80.8
Initial entropy coefficient1.00.01
Target entropy coefficient−0.5-0.5−1.0-1.0
Appendix
Table 5: SAC+HER-specific parameters.
ParameterJaxGCRLBuilderBench
Learning rate3×10−43\times 10^{-4}10−410^{-4}
Expectile0.70.7
βhigh\beta_{\mathrm{high}}0.20.5
βlow\beta_{\mathrm{low}}0.3100
Target horizon KK2510
Direct-goal bootstrap fraction00
Appendix
Table 6: Online HIQL-specific parameters.
ParameterJaxGCRLBuilderBench
Actor/critic/value learning rate10−410^{-4}10−410^{-4}
Autoencoder/cluster learning rate3×10−43\times 10^{-4}3×10−43\times 10^{-4}
Target-network update period10 learner steps10 learner steps
Latent dimension1616
Action regularization0.050.05
Learned landmarks5050
Appendix
Table 7: L3P-specific parameters.
More in this topic

Robotic RL

Robotic reinforcement learning uses trial-and-error learning to help robots control, navigate, and manipulate objects.

See all 297 papers

Explore similar work

CardsList

  1. Outperforming 1000 Euclidean Layers With Hyperbolic Goal-Conditioned RL

    Oct 14, 2026Timo Klein, Thomas Lang, Andrii Shkabrii +3Reinforcement LearningContrastive RL

    Faculty of Computer Science, University of Vienna, Vienna, Austria · UniVie Doctoral School Computer Science, University of Vienna, Vienna, Austria · Department of Machine Learning and Systems Biology, Max Planck Institute of Biochemistry, Martinsried, Germany +1

  2. Learning to Navigate with Minimal Parameters: Decomposing Visual Navigation Through Closed-Form Geometric Interfaces

    Jul 13, 2026Edward Beng Wai Tan, Siew-Kei LamVisual NavigationRobot Policy Learning

    College of Computing and Data Science, Nanyang Technological University, Singapore.

  3. Reinforced Planning with Latent World Models

    Aug 19, 2026Armin Sommer, Jannik SchillingWorld Model-Based PlanningRobotic RL

    Pantheon Industries