We develop a self-supervised reinforcement learning algorithm that learns to build towers of up to eight cubes from scratch, without prior data, rewards, demonstrations, or pretrained models. Our method, Contrastive Planning via Spherical Linear Interpolation (C-Slerp), uses contrastive representations as reusable building blocks and repeatedly composes them to infer a sequence of intermediate waypoints. Under our hyperspherical parameterization, the learned representations allow intermediate waypoints to be inferred in closed form through spherical linear interpolation, without training a separate high-level policy or performing graph search. Importantly, the representation geometry also tells us how to adjust the number of waypoints based on the difficulty of each goal-reaching problem. Experiments show that our method succeeds on long-horizon navigation and robotic manipulation tasks, including settings where strong baselines fail to achieve a single success. Ablations further show that using a fixed number of waypoints substantially degrades performance, highlighting the importance of adapting the plan to task difficulty.
Figures & tables
Figure 1: Our paper studies long-horizon, sparse-reward tasks, including eight-cube stacking. Given only the current and goal states, the agent aims to manipulate the eight cubes into the goal state without receiving any external reward along the way. Strong exploration methods ( Pathak et al., 2017 ; Burda et al., 2018 ) fail to discover a single successful state in one billion environment steps.
Figure 2: Geometric interpretation of C-Slerp. The number of waypoints K is selected using the alignment between (R⊤)k+1ψ(s) and ψ(g) . Given the selected K , waypoint mean directions are obtained by spherical interpolation from ψ(s) to RK+1ψ(g) , followed by mapping back via (R⊤)i . The policy then pursues the furthest waypoint whose alignment with ϕ(s,a) meets the threshold ϵ .
Figure 3: C-Slerp solves long-horizon tasks where baselines never achieve a single success. Learning curves show success rates on BuilderBench manipulation tasks ( top ) and JaxGCRL locomotion tasks ( bottom ). C-Slerp achieves the highest success rates on most tasks across both benchmarks and is the only method to succeed on all cube-stacking tasks with a gripper.
Figure 4
Figure 6: Evaluation of C-Slerp in single-goal environments. To assess exploration capability, we evaluate C-Slerp with a single fixed goal throughout training. The environments and their fixed goals are shown in (a), and success rates for C-Slerp and the baselines are presented in (b).
Figure 7: Adaptive planning in C-Slerp. The top row shows successive states in the 5-tower task with a gripper. The bottom row shows spherical PCA ( Liu et al., 2019 ) projections of the latent trajectory, inferred waypoints, and selected subgoal. As the agent approaches the goal, it replans by adjusting the number of waypoints and selecting a waypoint to pursue. We provide complete rollout visualizations in Figures 15 and 16 and videos on our project page .
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Ablation on the threshold ϵ .
Figure 9: Ablation on learning the rotation matrix R .
Figure 10: Ablation on network depth.
Figure 11: Ablation on representation dimension d .
Figure 12: Comparison with CRL with larger discount factors γ .
Figure 13: Evolution of K and i∗ over time in JaxGCRL environments. We plot the mean and sampled values of K+1 , together with the selected waypoint index i∗ , against environment steps. As the agent approaches the goal, the predicted number of waypoints decreases.
Figure 14: Evolution of K and i∗ over time in BuilderBench environments. We plot the mean and sampled values of K+1 , together with the selected waypoint index i∗ , against environment steps. As the agent approaches the goal, the predicted number of waypoints decreases.
Figure 14Figure 15
Figure 15: Visualization of C-Slerp rollouts and corresponding latent representations in JaxGCRL. For each environment, the top row shows successive states, and the bottom row shows spherical PCA ( Liu et al., 2019 ) projections of the corresponding state representations and inferred waypoints.
Figure 17
Figure 16: Visualization of C-Slerp rollouts and corresponding latent representations in BuilderBench. For each environment, the top row shows successive states, and the bottom row shows spherical PCA ( Liu et al., 2019 ) projections of the corresponding state representations and inferred waypoints.
Figure 19
Figure 17: Visualization of waypoint representations in maze environments. We visualize the inferred latent waypoints z1:K and the corresponding score heatmaps. For each environment, the top row shows the (x,y) positions corresponding to the inferred latent waypoints, where denotes a candidate waypoint zi , highlights the selected waypoint zi∗ , and represents the goal. The bottom row shows heatmaps of fL(s,a,s′)=ϕ(s,a)⊤ψ(s′) , with the current state marked by and the goal indicated by .
Figure 18: Visualization of benchmark environments. The top row shows the BuilderBench manipulation tasks, and the bottom row shows the JaxGCRL navigation tasks. The goal used for evaluation in each JaxGCRL environment is indicated by . We provide videos demonstrating C-Slerp on all benchmark tasks on our project page .
Parameter
JaxGCRL
BuilderBench
Actor learning rate
10−4
10−4
Representation learning rate
10−4
10−4
Actor entropy learning rate
3×10−4
3×10−4
Waypoint entropy learning rate
3×10−3
3×10−3
Batch size
512
1024
Representation dimension d
128
512
Appendix
Table 1: Learning hyperparameters.
Method
Parameter
Search space
CRL
Actor learning rate
{3×10−4,2×10−4,10−4}
Critic learning rate
{3×10−4,2×10−4,10−4}
RIS
Actor learning rate
{3×10−5,10−4,3×10−4}
Critic learning rate
{10−4,3×10−4,10−3}
KL coefficient
{0.1,0.3}
Prior samples
{3,10}
Appendix
Table 2: Hyperparameter search spaces for baseline tuning.
Scaling self-supervised contrastive reinforcement learning (CRL) to 1000-layer Euclidean neural networks has enabled state-of-the-art goal-reaching performance, yet wall-clock time scales linearly with depth. We find that changing the geometry achieves stronger performance with far shallower networks. This matters in complex navigation environments, where walls can make route distance deviate substantially from Euclidean distance. In a U-shaped maze, we show that the longer the route, the more it deviates from Euclidean distance. Hyperbolic CRL addresses this mismatch by replacing Euclidean distances in CRL's contrastive objective with geodesic distances in hyperbolic space. On hard navigation tasks, inductive bias beats scale: depth 64 hyperbolic agents outperform 1024-layer-deep Euclidean networks while training more than 12× faster and using about 13× fewer parameters. We observe that hyperbolic agents behave qualitatively better in hard humanoid mazes, where they follow corridors to reach goals, whereas Euclidean agents try straight-line shortcuts over walls. Our results show that matching the representation geometry to the task can substitute for substantial network depth. Code is at https://github.com/Probabilistic-and-Interactive-ML/HypGCRL .
Timo Klein, Thomas Lang, Andrii Shkabrii +3
Faculty of Computer Science, University of Vienna, Vienna, Austria · UniVie Doctoral School Computer Science, University of Vienna, Vienna, Austria · Department of Machine Learning and Systems Biology, Max Planck Institute of Biochemistry, Martinsried, Germany +1
Visual navigation policies have grown to hundreds of millions of parameters trained on billions of frames, with geometry, mapping, and control learned implicitly. We propose a decomposed point-goal navigation system in which operations with known closed-form structure, such as projective geometry, occupancy, and coordinate transforms, are computed analytically and serve as interfaces between three small learned modules: an egress predictor that grounds the episode goal as a local subgoal in the current view, a navigation predictor that estimates a goal-conditioned posterior over where trajectories travel, and an endpoint-pinned residual diffusion generator that samples trajectory shapes from this posterior. Only 0.58M out of 23M parameters are trained, on 44k frames, in under one GPU-hour. Across 6060 point-goal episodes in 60 environments, the system attains competitive success rates with the lowest collision rate among evaluated methods. We further show that under this decomposition, the frozen image encoder can be replaced by a 0.54M MobileNetV2 at a -2.0 SR cost, bringing the full system under 1.2M parameters. It also transfers to no-goal exploration by retraining only the 123k-parameter egress head, and its failure modes under sensor corruption are transparent and analytically correctable. We deploy and evaluate the system zero-shot on a low-cost UGV, running navigation and localization on a Jetson Orin Nano in real-time.
Edward Beng Wai Tan, Siew-Kei Lam
College of Computing and Data Science, Nanyang Technological University, Singapore.
Humans solve complex problems by constructing plans and mentally simulating their outcomes with an internal model of the world. Machine learning has made substantial progress in learning world models that predict the consequences of action sequences, yet the procedures used to plan with these models remain largely hand-designed. Most planners rely on fixed search or optimization rules; approaches that learn aspects of search typically imitate a predefined optimizer or use planning to inform an amortized policy, rather improving multi-step plans. We introduce Reinforced Planning, a method that learns the plan-update itself by reinforcing update rules that produce better plans, using gradients propagated through a differentiable world model. We instantiate Reinforced Planning in RP1, which learns a critic over imagined outcomes via temporal-difference learning and a neural plan-improvement operator trained via imagined rollouts with a pretrained world model. RP1 can be trained fully offline without environment interaction; environment episodes are used only for checkpoint selection. Across visual navigation, arm reaching, and robotic manipulation on two world-model backbones, RP1 matches or exceeds existing planners, achieving near-perfect success in several settings while using 1,000× fewer world-model rollouts than the strongest alternative (CEM) and planning up to 67× faster under concurrent planners inference.