Outperforming 1000 Euclidean Layers With Hyperbolic Goal-Conditioned RL

Timo Klein; Thomas Lang; Andrii Shkabrii; Lukas Miklautz; Yllka Velaj; Sebastian Tschiatschek

cs.LGOct 14, 2026

Outperforming 1000 Euclidean Layers With Hyperbolic Goal-Conditioned RL

Authors: Timo Klein, Thomas Lang, Andrii Shkabrii, Lukas Miklautz, Yllka Velaj, Sebastian Tschiatschek

Organizations: Faculty of Computer Science, University of Vienna, Vienna, Austria · UniVie Doctoral School Computer Science, University of Vienna, Vienna, Austria · Department of Machine Learning and Systems Biology, Max Planck Institute of Biochemistry, Martinsried, Germany · ds:UniVie, University of Vienna, Vienna, Austria

Abstract

Scaling self-supervised contrastive reinforcement learning (CRL) to 1000-layer Euclidean neural networks has enabled state-of-the-art goal-reaching performance, yet wall-clock time scales linearly with depth. We find that changing the geometry achieves stronger performance with far shallower networks. This matters in complex navigation environments, where walls can make route distance deviate substantially from Euclidean distance. In a U-shaped maze, we show that the longer the route, the more it deviates from Euclidean distance. Hyperbolic CRL addresses this mismatch by replacing Euclidean distances in CRL's contrastive objective with geodesic distances in hyperbolic space. On hard navigation tasks, inductive bias beats scale: depth 64 hyperbolic agents outperform 1024-layer-deep Euclidean networks while training more than $12 \times$ faster and using about $13 \times$ fewer parameters. We observe that hyperbolic agents behave qualitatively better in hard humanoid mazes, where they follow corridors to reach goals, whereas Euclidean agents try straight-line shortcuts over walls. Our results show that matching the representation geometry to the task can substitute for substantial network depth. Code is at https://github.com/Probabilistic-and-Interactive-ML/HypGCRL .

Figures & tables

Figure 1: Hyperbolic agents outperform much deeper Euclidean ones. A hyperbolic agent of depth 64 reaches a higher steps at goal than the depth 1024 Euclidean agent while using

12\times

less wall-clock time on Humanoid U-Maze. Euclidean results are taken from Wang et al. (2025, Figure 12) , we use their step budget. Wall-clock times are reported in Appendix F .

Panel of Figure 2 — Figure 2: Hyperbolic critics better reflect the geometric structure of Ant U5-Maze. 2(a) ) The maze becomes increasingly more tree-like at longer task horizons under route distance, when walls are taken into account . We report relative $\delta$ -hyperbolicity of Ant U5-Maze (lower is more tree-like) under increasing task horizons. 2(b) ) Hyperbolic embedding distances track route distances more closely than Euclidean embedding distances . $P(\rho_{H}>\rho_{E})$ estimates the probability that a hyperbolic run has a higher Spearman correlation than a Euclidean run. 2(c) ) We sample maze locations with similar straight-line distance but different route distance. The percentage shows how often the critic assigns the longer route a larger embedding distance. The hyperbolic critic guides the agent along the correct route more frequently. In 2(b) and 2(c) , we show the 10 seeds per method with means and 95% bootstrap confidence intervals.

Panel of Figure 3 — Figure 3: Critic values and gradients in Ant U5-Maze. Both critics have depth 16. 3(a) ) The Euclidean critic assigns high Q-values across the wall near the start, whereas the hyperbolic critic values increase along the route towards the goal (brighter colors) . $\bm{\star}$ = start, G = goal, and — = invalid shortcut. 3(b) ) For state-goal pairs with long routes but short straight-line distances, we measure the cosine similarity between $\nabla_{x,y}Q$ , and two directions: along the optimal maze route and straight towards the goal. Euclidean gradients tend to point towards the wall, while hyperbolic critics do not exhibit this bias . Results are averaged over 10 seeds.

Figure 4: Hyperbolic agents outperform Euclidean ones at fixed depth. Hyperbolic improvements are particularly pronounced in hard navigation environments, e.g., Humanoid mazes. Humanoid is not suitable for curved representations at

c=0.5

because route distances are straight lines. All agents use depth 16, beyond which gains flatten (Figure 8 ). We report the mean across 10 seeds with 95% bootstrapped confidence intervals.

Figure 5: Behavioral evidence of straight-line bias . Similar to Wang et al. (2025) , we observe Euclidean agents trying to vault themselves over walls instead of learning to navigate the maze. In contrast, hyperbolic agents follow the corridors, learn locomotion behavior earlier, and achieve higher steps at goal.

Figure 6: Stitching on Ant U-Maze . Markers show the mean steps at goal with 95% bootstrap CIs.

Figure 7: Effect of curvature. The best curvature setting depends on the environment, with Humanoid preferring low curvature, and Ant environments preferring higher curvature. Mean over the final 10 evaluation epochs over five seeds.

Panel of Figure 8 — Figure 8: Effect of actor and critic depth scaling. Hyperbolic agents generally perform better as the Euclidean encoder grows deeper, although the size of the gain varies across environments. We report the mean over five seeds with 95% bootstrap confidence intervals.

Panel of Figure 9 — Figure 9: Offline results on OGBench . The hyperbolic agent reaches a higher mean success rate on all eight tasks, with the largest relative gains on the giant mazes. Percentages give the relative improvement of the hyperbolic mean over the Euclidean mean. Bars show the mean over 8 seeds with 95% bootstrap CIs.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 10 — Figure 10: Geometric interpretation of the Gromov product. Left: three points $w$ , $x$ , $z$ of a metric space, with their pairwise distances. Right: the associated comparison tripod, i.e. the unique metric tree with leaves $w$ , $x$ , $z$ joined at a branching point $p$ in which the path between any two leaves is of the same length as the corresponding distance on the left. The quantity $(x,z)_{w}=\tfrac{1}{2}\bigl(d(w,x)+d(w,z)-d(x,z)\bigr)$ is the length of the initial segment that shortest paths from $w$ to $x$ and from $w$ to $z$ share in the tripod before they separate at $p$ .

Figure 11: Example Environments. We use the environments implemented in JaxGCRL ( Bortkiewicz et al., 2025a ) . Figure taken from Bortkiewicz et al. (2025a) .

Figure 12: A Euclidean encoder maps observations to feature vectors, which are then viewed as tangent vectors at the origin of the hyperboloid, mapped to the manifold via the exponential map followed by one hyperbolic linear layer (HTC, Yang et al. (2024) ). The similarity for the online goal-oriented contrastive learning is the negative geodesic distance between any two given points.

Hyperparameter	Value
num_timesteps	100M–400M (varying across tasks)
update-to-data (UTD) ratio	1:40
max_replay_size	10,000
min_replay_size	1,000
episode_length	1,000
discounting	0.99

Appendix

Table 1: Online hyperparameters. Unshaded rows are Wang et al. (2025, Table 7) and are shared by the Euclidean and the hybrid agents. Blue rows exist only for hyperbolic agents. Red rows differ from the released code’s defaults. Total number of steps per task: ant_u4_maze, ant_ball, arm_push_easy, arm_push_hard, arm_binpick_hard 100M; ant_hardest_maze 200M; humanoid 300M; ant_u5_maze, humanoid_u_maze, humanoid_big_maze 400M.

Hyperparameter	Value
train_steps	1M
batch_size	1,024
optimizer / learning rate	Adam, 3e-4
actor MLP	(512, 512, 512), GELU, no LayerNorm
critic MLP ( $\phi$ , $\psi$ )	(512, 512, 512), GELU, LayerNorm
representation dimension	512

Appendix

Table 2: Offline (OGBench) hyperparameters. Unshaded rows are the OGBench reference CRL configuration Park et al. (2025) and are shared by the Euclidean and the hyperbolic agent. Blue rows exist only for the hyperbolic agent. Red rows differ from the reference.

Figure 13: Full curvature ablation. Same Figure as Figure 7 , including Ant Ball and the Euclidean agent as baseline. No single curvature setting $c$ works best across all environments, and the best $c$ depends on the environment. Hyperbolic agents with the curvature set per environment match Euclidean performance on Humanoid and Arm Push. Mean over the final 10 evaluation epochs over five seeds.

AntMaze
		Hyperbolic critic
Task	Euclidean CRL	Euclidean actor	Hyperbolic actor
Giant navigate	16.9 $\pm$ 4.8	28.0 $\pm$ 2.3	38.7 $\pm$ 4.6
Medium stitch	43.7 $\pm$ 8.7	80.2 $\pm$ 6.8	77.2 $\pm$ 7.4
Large stitch	17.6 $\pm$ 3.2	21.2 $\pm$ 6.6	22.6 $\pm$ 7.5
HumanoidMaze

Appendix

Table 3: With a Euclidean actor, the hyperbolic critic reaches a higher mean success rate than Euclidean CRL on seven of eight tasks, and adding the hyperbolic actor raises it further on seven of eight. Entries are mean OGBench success rates (%)

\pm

standard deviation over 8 seeds. Bold denotes hyperbolic entries whose 95% bootstrap CI does not overlap that of Euclidean CRL. The last column is the full offline hyperbolic agent, whose hybrid hyperbolic actor is regularized with Hyper++ ( Klein et al., 2026 ) .

Panel of Figure 15 — Figure 15: Runtime on Ant U4-Maze versus network depth on one H100 GPU. 15(a) ) Median SPS over retained epochs. 15(b) ) JAX compilation time. 15(c) ) Peak host memory. All vertical axes are logarithmic.

Method	Actor	Critic	Params	Median SPS	Est. compile	Peak RSS
	depth	depth	(M)	( $\uparrow$ )	(min, $\downarrow$ )	(GB, $\downarrow$ )
Humanoid U-Maze
Hyperbolic	64	64	$12.9$	$3,722$	$1.62$	$7.6$
Euclidean	64	64	$12.9$	$3,788$	$1.54$	$7.6$
Euclidean	512	512	$102.0$	$512$	$32.52$	$83.2$
Euclidean	512	1024	$169.9$	$303$	$92.79$	$299.4$

Appendix

Table 4: Runtime on the Humanoid mazes on one H100 GPU. SPS are measured as median environment steps per second, excluding evaluation. We estimate the compile time as first-epoch duration minus median later-epoch duration. Peak RSS does not refer to GPU VRAM. The hyperbolic head adds 8,448 parameters irrespective of depth.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities
Mar 19, 2025Kevin Wang, Ishaan Javali, Michał Bortkiewicz +2Reinforcement Learning Self-Supervised RL
Princeton University · Warsaw University of Technology · Warsaw University of Technology, Tooploox, IDEAS Research Institute
Hyperspherical Contrastive Representations Unlock Planning and Exploration
Oct 14, 2026Jonghae Park, Sungil Son, Jonghun Shin +3Robotic RL Robot Motion Planning
Seoul National University · Princeton University
ChronoSRL: Temporal Geometry for Self-Supervised Reinforcement Learning
Sep 28, 2026Nico Bohlinger, Jan PetersRL for Robotics Representation Learning
Technical University of Darmstadt, Germany · Robotics Institute Germany (RIG); German Research Center for AI (DFKI); hessian.AI

Mar 19, 2025J/K move · Enter open · S save

1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities

Kevin Wang, Ishaan Javali, Michał Bortkiewicz, Tomasz Trzciński+1

Princeton University · Warsaw University of Technology · Warsaw University of Technology, Tooploox, IDEAS Research Institute

Reinforcement Learning Self-Supervised RL Robotic RL Contrastive RL Goal-Conditioned RL

Open paper AskPDF arXiv

Scaling up self-supervised learning has driven breakthroughs in language and vision, yet comparable progress has remained elusive in reinforcement learning (RL). In this paper, we study building blocks for self-supervised RL that unlock substantial improvements in scalability, with network depth serving as a critical factor. Whereas most RL papers in recent years have relied on shallow architectures (around 2 - 5 layers), we demonstrate that increasing the depth up to 1024 layers can significantly boost performance. Our experiments are conducted in an unsupervised goal-conditioned setting, where no demonstrations or rewards are provided, so an agent must explore (from scratch) and learn how to maximize the likelihood of reaching commanded goals. Evaluated on simulated locomotion and manipulation tasks, our approach increases performance on the self-supervised contrastive RL algorithm by

2\times

50\times

, outperforming other goal-conditioned baselines. Increasing the model depth not only increases success rates but also qualitatively changes the behaviors learned. The project webpage and code can be found here: https://wang-kevin3290.github.io/scaling-crl/.

Abstract

Reinforcement Learning

Explore similar work

1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities

Hyperspherical Contrastive Representations Unlock Planning and Exploration

ChronoSRL: Temporal Geometry for Self-Supervised Reinforcement Learning