Outperforming 1000 Euclidean Layers With Hyperbolic Goal-Conditioned RL
Authors: Timo Klein, Thomas Lang, Andrii Shkabrii, Lukas Miklautz, Yllka Velaj, Sebastian Tschiatschek
Organizations: Faculty of Computer Science, University of Vienna, Vienna, Austria · UniVie Doctoral School Computer Science, University of Vienna, Vienna, Austria · Department of Machine Learning and Systems Biology, Max Planck Institute of Biochemistry, Martinsried, Germany · ds:UniVie, University of Vienna, Vienna, Austria
Scaling self-supervised contrastive reinforcement learning (CRL) to 1000-layer Euclidean neural networks has enabled state-of-the-art goal-reaching performance, yet wall-clock time scales linearly with depth. We find that changing the geometry achieves stronger performance with far shallower networks. This matters in complex navigation environments, where walls can make route distance deviate substantially from Euclidean distance. In a U-shaped maze, we show that the longer the route, the more it deviates from Euclidean distance. Hyperbolic CRL addresses this mismatch by replacing Euclidean distances in CRL's contrastive objective with geodesic distances in hyperbolic space. On hard navigation tasks, inductive bias beats scale: depth 64 hyperbolic agents outperform 1024-layer-deep Euclidean networks while training more than 12× faster and using about 13× fewer parameters. We observe that hyperbolic agents behave qualitatively better in hard humanoid mazes, where they follow corridors to reach goals, whereas Euclidean agents try straight-line shortcuts over walls. Our results show that matching the representation geometry to the task can substitute for substantial network depth. Code is at https://github.com/Probabilistic-and-Interactive-ML/HypGCRL .
Figures & tables
Figure 1: Hyperbolic agents outperform much deeper Euclidean ones. A hyperbolic agent of depth 64 reaches a higher steps at goal than the depth 1024 Euclidean agent while using 12× less wall-clock time on Humanoid U-Maze. Euclidean results are taken from Wang et al. (2025, Figure 12) , we use their step budget. Wall-clock times are reported in Appendix F .
Figure 2: Hyperbolic critics better reflect the geometric structure of Ant U5-Maze. 2(a) ) The maze becomes increasingly more tree-like at longer task horizons under route distance, when walls are taken into account . We report relative δ -hyperbolicity of Ant U5-Maze (lower is more tree-like) under increasing task horizons. 2(b) ) Hyperbolic embedding distances track route distances more closely than Euclidean embedding distances . P(ρH>ρE) estimates the probability that a hyperbolic run has a higher Spearman correlation than a Euclidean run. 2(c) ) We sample maze locations with similar straight-line distance but different route distance. The percentage shows how often the critic assigns the longer route a larger embedding distance. The hyperbolic critic guides the agent along the correct route more frequently. In 2(b) and 2(c) , we show the 10 seeds per method with means and 95% bootstrap confidence intervals.
Figure 3: Critic values and gradients in Ant U5-Maze. Both critics have depth 16. 3(a) ) The Euclidean critic assigns high Q-values across the wall near the start, whereas the hyperbolic critic values increase along the route towards the goal (brighter colors) . ⋆ = start, G = goal, and — = invalid shortcut. 3(b) ) For state-goal pairs with long routes but short straight-line distances, we measure the cosine similarity between ∇x,yQ , and two directions: along the optimal maze route and straight towards the goal. Euclidean gradients tend to point towards the wall, while hyperbolic critics do not exhibit this bias . Results are averaged over 10 seeds.
Figure 4: Hyperbolic agents outperform Euclidean ones at fixed depth. Hyperbolic improvements are particularly pronounced in hard navigation environments, e.g., Humanoid mazes. Humanoid is not suitable for curved representations at c=0.5 because route distances are straight lines. All agents use depth 16, beyond which gains flatten (Figure 8 ). We report the mean across 10 seeds with 95% bootstrapped confidence intervals.
Figure 5: Behavioral evidence of straight-line bias . Similar to Wang et al. (2025) , we observe Euclidean agents trying to vault themselves over walls instead of learning to navigate the maze. In contrast, hyperbolic agents follow the corridors, learn locomotion behavior earlier, and achieve higher steps at goal.
Figure 6: Stitching on Ant U-Maze . Markers show the mean steps at goal with 95% bootstrap CIs.
Figure 7: Effect of curvature. The best curvature setting depends on the environment, with Humanoid preferring low curvature, and Ant environments preferring higher curvature. Mean over the final 10 evaluation epochs over five seeds.
Figure 8: Effect of actor and critic depth scaling. Hyperbolic agents generally perform better as the Euclidean encoder grows deeper, although the size of the gain varies across environments. We report the mean over five seeds with 95% bootstrap confidence intervals.
Figure 9: Offline results on OGBench . The hyperbolic agent reaches a higher mean success rate on all eight tasks, with the largest relative gains on the giant mazes. Percentages give the relative improvement of the hyperbolic mean over the Euclidean mean. Bars show the mean over 8 seeds with 95% bootstrap CIs.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Geometric interpretation of the Gromov product. Left: three points w , x , z of a metric space, with their pairwise distances. Right: the associated comparison tripod, i.e. the unique metric tree with leaves w , x , z joined at a branching point p in which the path between any two leaves is of the same length as the corresponding distance on the left. The quantity (x,z)w=21(d(w,x)+d(w,z)−d(x,z)) is the length of the initial segment that shortest paths from w to x and from w to z share in the tripod before they separate at p .
Figure 11: Example Environments. We use the environments implemented in JaxGCRL ( Bortkiewicz et al., 2025a ) . Figure taken from Bortkiewicz et al. (2025a) .
Figure 12: A Euclidean encoder maps observations to feature vectors, which are then viewed as tangent vectors at the origin of the hyperboloid, mapped to the manifold via the exponential map followed by one hyperbolic linear layer (HTC, Yang et al. (2024) ). The similarity for the online goal-oriented contrastive learning is the negative geodesic distance between any two given points.
Hyperparameter
Value
num_timesteps
100M–400M (varying across tasks)
update-to-data (UTD) ratio
1:40
max_replay_size
10,000
min_replay_size
1,000
episode_length
1,000
discounting
0.99
Appendix
Table 1: Online hyperparameters. Unshaded rows are Wang et al. (2025, Table 7) and are shared by the Euclidean and the hybrid agents. Blue rows exist only for hyperbolic agents. Red rows differ from the released code’s defaults. Total number of steps per task: ant_u4_maze, ant_ball, arm_push_easy, arm_push_hard, arm_binpick_hard 100M; ant_hardest_maze 200M; humanoid 300M; ant_u5_maze, humanoid_u_maze, humanoid_big_maze 400M.
Hyperparameter
Value
train_steps
1M
batch_size
1,024
optimizer / learning rate
Adam, 3e-4
actor MLP
(512, 512, 512), GELU, no LayerNorm
critic MLP ( ϕ , ψ )
(512, 512, 512), GELU, LayerNorm
representation dimension
512
Appendix
Table 2: Offline (OGBench) hyperparameters. Unshaded rows are the OGBench reference CRL configuration Park et al. (2025) and are shared by the Euclidean and the hyperbolic agent. Blue rows exist only for the hyperbolic agent. Red rows differ from the reference.
Figure 13: Full curvature ablation. Same Figure as Figure 7 , including Ant Ball and the Euclidean agent as baseline. No single curvature setting c works best across all environments, and the best c depends on the environment. Hyperbolic agents with the curvature set per environment match Euclidean performance on Humanoid and Arm Push. Mean over the final 10 evaluation epochs over five seeds.
Figure 14: Behavior on Ant Ball.
Hyperbolic critic
Task
Euclidean CRL
Euclidean actor
Hyperbolic actor
AntMaze
Giant navigate
16.9 ± 4.8
28.0 ± 2.3
38.7 ± 4.6
Medium stitch
43.7 ± 8.7
80.2 ± 6.8
77.2 ± 7.4
Large stitch
17.6 ± 3.2
21.2 ± 6.6
22.6 ± 7.5
HumanoidMaze
Appendix
Table 3: With a Euclidean actor, the hyperbolic critic reaches a higher mean success rate than Euclidean CRL on seven of eight tasks, and adding the hyperbolic actor raises it further on seven of eight. Entries are mean OGBench success rates (%) ± standard deviation over 8 seeds. Bold denotes hyperbolic entries whose 95% bootstrap CI does not overlap that of Euclidean CRL. The last column is the full offline hyperbolic agent, whose hybrid hyperbolic actor is regularized with Hyper++ ( Klein et al., 2026 ) .
Figure 15: Runtime on Ant U4-Maze versus network depth on one H100 GPU. 15(a) ) Median SPS over retained epochs. 15(b) ) JAX compilation time. 15(c) ) Peak host memory. All vertical axes are logarithmic.
Method
Actor
Critic
Params
Median SPS
Est. compile
Peak RSS
depth
depth
(M)
( ↑ )
(min, ↓ )
(GB, ↓ )
Humanoid U-Maze
Hyperbolic
64
64
12.9
3,722
1.62
7.6
Euclidean
64
64
12.9
3,788
1.54
7.6
Euclidean
512
512
102.0
512
32.52
83.2
Euclidean
512
1024
169.9
303
92.79
299.4
Appendix
Table 4: Runtime on the Humanoid mazes on one H100 GPU. SPS are measured as median environment steps per second, excluding evaluation. We estimate the compile time as first-epoch duration minus median later-epoch duration. Peak RSS does not refer to GPU VRAM. The hyperbolic head adds 8,448 parameters irrespective of depth.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Scaling up self-supervised learning has driven breakthroughs in language and vision, yet comparable progress has remained elusive in reinforcement learning (RL). In this paper, we study building blocks for self-supervised RL that unlock substantial improvements in scalability, with network depth serving as a critical factor. Whereas most RL papers in recent years have relied on shallow architectures (around 2 - 5 layers), we demonstrate that increasing the depth up to 1024 layers can significantly boost performance. Our experiments are conducted in an unsupervised goal-conditioned setting, where no demonstrations or rewards are provided, so an agent must explore (from scratch) and learn how to maximize the likelihood of reaching commanded goals. Evaluated on simulated locomotion and manipulation tasks, our approach increases performance on the self-supervised contrastive RL algorithm by 2× - 50×, outperforming other goal-conditioned baselines. Increasing the model depth not only increases success rates but also qualitatively changes the behaviors learned. The project webpage and code can be found here: https://wang-kevin3290.github.io/scaling-crl/.
Kevin Wang, Ishaan Javali, Michał Bortkiewicz +2
Princeton University · Warsaw University of Technology · Warsaw University of Technology, Tooploox, IDEAS Research Institute
We develop a self-supervised reinforcement learning algorithm that learns to build towers of up to eight cubes from scratch, without prior data, rewards, demonstrations, or pretrained models. Our method, Contrastive Planning via Spherical Linear Interpolation (C-Slerp), uses contrastive representations as reusable building blocks and repeatedly composes them to infer a sequence of intermediate waypoints. Under our hyperspherical parameterization, the learned representations allow intermediate waypoints to be inferred in closed form through spherical linear interpolation, without training a separate high-level policy or performing graph search. Importantly, the representation geometry also tells us how to adjust the number of waypoints based on the difficulty of each goal-reaching problem. Experiments show that our method succeeds on long-horizon navigation and robotic manipulation tasks, including settings where strong baselines fail to achieve a single success. Ablations further show that using a fixed number of waypoints substantially degrades performance, highlighting the importance of adapting the plan to task difficulty.
A goal that is close in space can be far away in time. Obstacles, terrain, and the agent's own capabilities determine how long it takes to get there. Yet, critics in contrastive and survival reinforcement learning do not measure the distances in their representation space in units of time. We therefore introduce ChronoSRL, which gives the critic's embeddings an explicit temporal geometry. The distance between state-action and goal embeddings is trained to match the time that the agent takes to reach the goal (goal-reaching time), while goals that were not reached, and goals from other trajectories, are pushed at least one discount horizon away. Furthermore, reaching a goal quickly once does not mean that reaching it is reliable in general, so the policy should not follow the temporal distance directly. Instead, we build on survival reinforcement learning and predict from our temporal embeddings not only the full distribution of goal-reaching times but also the time spent near the goal. Thereby, the policy is trained to favor actions that reach the goal sooner and more reliably and that keep the agent near it. ChronoSRL learns faster and reaches higher performance than contrastive, action-chunked contrastive, and survival reinforcement learning baselines on seven standard locomotion and navigation benchmarks, even with much smaller networks. To test the limits of self-supervised reinforcement learning, we introduce velocity tracking, goal-position reaching, and box climbing tasks with a quadruped robot in a realistic sim-to-real locomotion setup, and show how the shaping terms that are typical for robotics can be naturally incorporated into our framework. ChronoSRL is the only one of the tested self-supervised reinforcement learning methods that learns to stay at the commanded velocities and goal positions, and climbs the highest boxes.
Nico Bohlinger, Jan Peters
Technical University of Darmstadt, Germany · Robotics Institute Germany (RIG); German Research Center for AI (DFKI); hessian.AI