1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities

Kevin Wang; Ishaan Javali; Michał Bortkiewicz; Tomasz Trzciński; Benjamin Eysenbach

cs.LGMar 19, 2025

1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities

Authors: Kevin Wang, Ishaan Javali, Michał Bortkiewicz, Tomasz Trzciński, Benjamin Eysenbach

Organizations: Princeton University · Warsaw University of Technology · Warsaw University of Technology, Tooploox, IDEAS Research Institute

Abstract

Scaling up self-supervised learning has driven breakthroughs in language and vision, yet comparable progress has remained elusive in reinforcement learning (RL). In this paper, we study building blocks for self-supervised RL that unlock substantial improvements in scalability, with network depth serving as a critical factor. Whereas most RL papers in recent years have relied on shallow architectures (around 2 - 5 layers), we demonstrate that increasing the depth up to 1024 layers can significantly boost performance. Our experiments are conducted in an unsupervised goal-conditioned setting, where no demonstrations or rewards are provided, so an agent must explore (from scratch) and learn how to maximize the likelihood of reaching commanded goals. Evaluated on simulated locomotion and manipulation tasks, our approach increases performance on the self-supervised contrastive RL algorithm by $2\times$ - $50\times$ , outperforming other goal-conditioned baselines. Increasing the model depth not only increases success rates but also qualitatively changes the behaviors learned. The project webpage and code can be found here: https://wang-kevin3290.github.io/scaling-crl/.

Figures & tables

Figure 1: Scaling network depth yields performance gains across a suite of locomotion, navigation, and manipulation tasks, ranging from doubling performance to 50× improvements on Humanoid-based tasks. Notably, rather than scaling smoothly, performance often jumps at specific critical depths (e.g., 8 layers on Ant Big Maze, 64 on Humanoid U-Maze), which correspond to the emergence of qualitatively distinct policies (see Section 4 ).

Figure 2: Architecture. Our approach integrates residual connections into both the actor and critic networks of the Contrastive RL algorithm. The depth of this residual architecture is defined as the total number of Dense layers across the residual blocks, which, with our residual block size of 4, equates to $4N$ .

Figure 3: Increasing depth results in new capabilities: Row 1 : A depth-4 agent collapses and throws itself toward the goal. Row 2 : A depth-16 agent walks upright. Row 3 : A depth-64 agent struggles and falls. Row 4 : A depth-256 agent vaults the wall acrobatically.

Figure 4: Scaling network width vs. depth . Here, we reflect findings from previous works ( Lee et al., 2024 ; Nauman et al., 2024b ) which suggest that increasing network width can enhance performance. However, in contrast to prior work, our method is able to scale depth, yielding more impactful performance gains. For instance, in the Humanoid environment, raising the width to 2048 (depth=4) fails to match the performance achieved by simply doubling the depth to 8 (width=256). The comparative advantage of scaling depth is more pronounced as the observational dimensionality increases.

Figure 7: Deeper networks unlock batch size scaling. We find that as depth increases from 4 to 64 in Humanoid, larger networks can effectively leverage batch size scaling to achieve further improvements.

Figure 8: We disentangle the effects of exploration and expressivity on depth scaling by training three networks in parallel: a “collector,” plus one deep and one shallow learner that train only from the collector’s shared replay buffer. In all three environments, when using a deep collector (i.e. good data coverage), the deep learner outperforms the shallow learner, indicating that expressivity is crucial when controlling for good exploration. With a shallow collector (poor exploration), even the deep learner cannot overcome the limitations of insufficient data coverage. As such, the benefits of depth scaling arise from a combination of improved exploration and increased expressivity working jointly.

Figure 9: Deeper Q-functions are qualitatively different. In the U4-Maze, the start and goal positions are indicated by the $\mathbin{\vtop{\halign{#\cr$\circledcirc$\cr$\bullet$\crcr}}}$ and G symbols respectively, and the visualized Q values are computed via the $L_{2}$ distance in the learned representation space, i.e., $Q(s,a,g)=\|\phi(s,a)-\psi(g)\|_{2}$ . The shallow depth 4 network (left) naively relies on Euclidean proximity, showing high Q values near the start despite a maze wall. In contrast, the depth 64 network (right) clusters high Q values at the goal, gradually tapering along the interior. — Figure 9: Deeper Q-functions are qualitatively different. In the U4-Maze, the start and goal positions are indicated by the $\mathbin{\vtop{\halign{#\cr$ \circledcirc $\cr$ \bullet $\crcr}}}$ and G symbols respectively, and the visualized Q values are computed via the $L_{2}$ distance in the learned representation space, i.e., $Q(s,a,g)=\|\phi(s,a)-\psi(g)\|_{2}$ . The shallow depth 4 network (left) naively relies on Euclidean proximity, showing high Q values near the start despite a maze wall. In contrast, the depth 64 network (right) clusters high Q values at the goal, gradually tapering along the interior.

Figure 10: We visualize state-action embeddings from shallow (depth 4) and deep (depth 64) networks along a successful trajectory in the Humanoid task. Near the goal, embeddings from the deep network expand across a curved surface, while those from the shallow network form a tight cluster. This suggests that deeper networks may devote greater representational capacity to regions of the state space that are more frequently visited and play a more critical role in successful task completion.

Figure 11: Deeper networks exhibit improved generalization. (Top left) We modify the training setup of the Ant U-Maze environment such that start-goal pairs are separated by $\leq 3$ units. This design guarantees that no evaluation pairs (Top right) were encountered during training, testing the ability for combinatorial generalization via stitching. (Bottom) Generalization ability improves as network depth grows from 4 to 16 to 64 layers.

Figure 12: Testing the limits of scale. We extend the results from Figure 1 by scaling networks even further on the challenging Humanoid maze environments. We observe continued performance improvements with network depths of 256 and 1024 layers on Humanoid U-Maze. Note that for the 1024-layer networks, we observed the actor loss exploding at the onset of training, so we maintained the actor depth at 512 while using 1024-layer networks only for the two critic encoders.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 12: Scaled CRL (Ours) outperforms baselines CRL (original), SAC, SAC+HER, TD3+HER, GCSL, and GCBC in 8 out 10 environments.

Figure 13: Depth scaling yields limited gains for SAC, SAC+HER, TD3+HER, GCSL, and GCBC.

Task	Dim	D = 4	D = 64	Imprv.
Arm Binpick Hard	17	$38$ $\mathbf{\pm}4$	$219$ $\mathbf{\pm}15$	$5.7\times$
Arm Push Easy		$308$ $\mathbf{\pm}33$	$762$ $\mathbf{\pm}30$	$2.5\times$
Arm Push Hard		$171$ $\mathbf{\pm}11$	$410$ $\mathbf{\pm}13$	$2.4\times$
Ant U4-Maze	29	$11.4$ $\mathbf{\pm}4.1$	$286$ $\mathbf{\pm}36$	$25\times$
Ant U5-Maze		$0.97$ $\mathbf{\pm}0.7$	$61$ $\mathbf{\pm}18$	$63\times$
Ant Big Maze		$61$ $\mathbf{\pm}20$	$441$ $\mathbf{\pm}25$	$7.3\times$

Appendix

Table 1: Increasing network depth (depth

D=4\rightarrow 64

) increases performance on CRL ( Figure 1 ). Scaling depth exhibits the greatest benefits on tasks with the largest observation dimension ( Dim ).

Figure 14: Our approach successfully scales depth in offline GCBC on antmaze-medium-stitch (OGBench). In contrast, scaling depth for BC ( antmaze-giant-navigate , expert SAC data) and for both online ( FetchPush ) and offline QRL ( pointmaze-giant-stitch , OGBench) yield negative results.

Figure 15: Performance of depth scaling on CRL augmented with quasimetric architectures (CMD-1).

Figure 16: (Left) Layer Norm is essential for scaling depth. (Right) Scaling with ReLU activations leads to worse performance compared to Swish activations.

Steps to reach

\geq

200 success Depth 4 16 32 With – 50 42 Without – 64 54

Steps to reach

\geq

400 success Depth 4 16 32 With – 62 48 Without – 75 64

Steps to reach

\geq

600 success Depth 4 16 32 With – 77 67 Without – – 77

Appendix

Table 2: Integrating hyperspherical normalization in our architecture enhances the sample efficiency of depth scaling.

Figure 17: L2 norms of residual activations in networks with depths of 32, 64, 128, and 256.

Figure 18: To evaluate the scalability of our method in the offline setting, we scaled model depth on OGBench ( Park et al., 2024 ) . In two out of three environments, performance drastically declined as depth scaled from 4 to 64, while a slight improvement was seen on antmaze-medium-stitch-v0. Successfully adapting our method to scale offline GCRL is an important direction for future work.

Figure 19: The scaling results of this paper are demonstrated on the JaxGCRL benchmark, showing that they replicate across a diverse range of locomotion, navigation, and manipulation tasks. These tasks are set in the online goal-conditioned setting where there are no auxiliary rewards or demonstrations. Figure taken from ( Bortkiewicz et al., 2024 ) .

Figure 20: Scaling behavior for humanoid in two different python environments: MJX=3.2.3, Brax=0.10.5 and MJX=3.2.6, Brax=0.10.1 (ours) version of JaxGCRL. Scaling depth improves the performance significantly for both versions. In the environment we used, training requires fewer environment steps to reach a marginally better performance than in other Python environment.

Environment	Depth 4	Depth 8	Depth 16	Depth 32	Depth 64
Humanoid	1.48 $\pm$ 0.00	2.13 $\pm$ 0.01	3.40 $\pm$ 0.01	5.92 $\pm$ 0.01	10.99 $\pm$ 0.01
Ant Big Maze	2.12 $\pm$ 0.00	2.77 $\pm$ 0.00	4.04 $\pm$ 0.01	6.57 $\pm$ 0.02	11.66 $\pm$ 0.03
Ant U4-Maze	1.98 $\pm$ 0.27	2.54 $\pm$ 0.01	3.81 $\pm$ 0.01	6.35 $\pm$ 0.01	11.43 $\pm$ 0.03
Ant U5-Maze	9.46 $\pm$ 1.75	10.99 $\pm$ 0.02	16.09 $\pm$ 0.01	31.49 $\pm$ 0.34	46.40 $\pm$ 0.12
Ant Hardest Maze	5.11 $\pm$ 0.00	6.39 $\pm$ 0.00	8.94 $\pm$ 0.01	13.97 $\pm$ 0.01	23.96 $\pm$ 0.06
Arm Push Easy	9.97 $\pm$ 1.03	11.02 $\pm$ 1.29	12.20 $\pm$ 1.43	14.94 $\pm$ 1.96	19.52 $\pm$ 1.97

Appendix

Table 3: Wall-clock time (in hours) for Depth 4, 8, 16, 32, and 64 across all 10 environments.

Depth	Time (h)
4	3.23 $\pm$ 0.001
8	4.19 $\pm$ 0.003
16	6.07 $\pm$ 0.003
32	9.83 $\pm$ 0.006
64	17.33 $\pm$ 0.003
128	32.67 $\pm$ 0.124

Appendix

Table 4: Total wall-clock time (in hours) for training from Depth 4 up to Depth 1024 in the Humanoid U-Maze environment.

Environment	Scaled CRL	SAC	SAC+HER	TD3	GCSL	GCBC
Humanoid	11.0 $\pm$ 0.0	0.5 $\pm$ 0.0	0.6 $\pm$ 0.0	0.8 $\pm$ 0.0	0.4 $\pm$ 0.0	0.6 $\pm$ 0.0
Ant Big Maze	11.7 $\pm$ 0.0	1.6 $\pm$ 0.0	1.6 $\pm$ 0.0	1.7 $\pm$ 0.0	1.5 $\pm$ 0.3	1.4 $\pm$ 0.1
Ant U4-Maze	11.4 $\pm$ 0.0	1.2 $\pm$ 0.0	1.3 $\pm$ 0.0	1.3 $\pm$ 0.0	0.7 $\pm$ 0.0	1.1 $\pm$ 0.1
Ant U5-Maze	46.4 $\pm$ 0.1	5.7 $\pm$ 0.0	6.1 $\pm$ 0.0	6.2 $\pm$ 0.0	2.8 $\pm$ 0.1	5.6 $\pm$ 0.5
Ant Hardest Maze	24.0 $\pm$ 0.0	4.3 $\pm$ 0.0	4.5 $\pm$ 0.0	5.0 $\pm$ 0.0	2.1 $\pm$ 0.6	4.4 $\pm$ 0.5
Arm Push Easy	19.5 $\pm$ 0.6	8.3 $\pm$ 0.0	8.5 $\pm$ 0.0	8.4 $\pm$ 0.0	6.4 $\pm$ 0.1	8.3 $\pm$ 0.3

Appendix

Table 5: Wall-clock training time comparison of our method vs. baselines across all 10 environments.

Environment	SAC	Scaled CRL (Depth 64)
Humanoid	0.46	6.37
Ant Big Maze	1.55	0.00
Ant U4-Maze	1.16	0.00
Ant U5-Maze	5.73	0.00
Ant Hardest Maze	4.33	0.45
Arm Push Easy	8.32	1.91

Appendix

Table 6: Wall-clock time (in hours) for our approach to surpass SAC’s final performance. As shown, our approach surpasses SAC performance in less wall-clock time in 7 out of 10 environments. The N/A* entries are because in those environments, scaled CRL doesn’t outperform SAC.

Hyperparameter	Value
num_timesteps	100M-400M (varying across tasks)
update-to-data (UTD) ratio	1:40
max_replay_size	10,000
min_replay_size	1,000
episode_length	1,000
discounting	0.99

Appendix

Table 7: Hyperparameters

Explore similar work

Survival Reinforcement Learning: Toward Scalable Self-Supervised RL
May 29, 2026Franki Nguimatsia-Tiofack, Fabian Schramm, Théotime Le Hellard +1Offline Reinforcement Learning Scalable Robot Learning
Inria and École Normale Supérieure PSL Research University Paris, France
Representation Learning Enables Scalable Multitask Deep Reinforcement Learning
Jun 4, 2026Johan Obando-Ceron, Lu Li, Scott Fujimoto +3Multi-Turn Reinforcement Learning Model-Based Reinforcement Learning
Mila – Québec AI Institute · Université de Montréal · McGill University +2
ChronoSRL: Temporal Geometry for Self-Supervised Reinforcement Learning
Sep 28, 2026Nico Bohlinger, Jan PetersGoal-Conditioned Reinforcement Learning Offline Reinforcement Learning
Technical University of Darmstadt, Germany · Robotics Institute Germany (RIG); German Research Center for AI (DFKI); hessian.AI

May 29, 2026, cs.LGJ/K move · Enter open · S save

Survival Reinforcement Learning: Toward Scalable Self-Supervised RL

Franki Nguimatsia-Tiofack, Fabian Schramm, Théotime Le Hellard, Justin Carpentier

Inria and École Normale Supérieure PSL Research University Paris, France

Offline Reinforcement Learning Scalable Robot Learning Contrastive Learning

Open paper AskPDF arXiv

While self-supervised Contrastive Reinforcement Learning (CRL) has shown remarkable depth-scaling capabilities, successfully using networks over 64 layers, scaled CRL still struggles with long-horizon goal-conditioned planning due to the uniformity-tolerance dilemma inherent in contrastive losses. We introduce Survival Reinforcement Learning (SRL), an online classification-based alternative that extends the survival value learning framework by maximizing the agent's dwell time at target goals. SRL bypasses the structural constraints of CRL and mitigates the "bang-bang" control solutions inherent to survival frameworks, which often induce undesirable behavior in complex dynamical systems. Evaluated across diverse robotic benchmarks, scaled SRL matches state-of-the-art CRL on manipulation tasks and outperforms it by 2x to 8x on stable, long-horizon locomotion tasks. Our results provide strong additional evidence that classification-based methods may serve as a key primitive in the broader effort to scale reinforcement learning.

Abstract

Explore similar work

Survival Reinforcement Learning: Toward Scalable Self-Supervised RL

Representation Learning Enables Scalable Multitask Deep Reinforcement Learning

ChronoSRL: Temporal Geometry for Self-Supervised Reinforcement Learning