Arrive and Survive: Scaling Safe Goal-Conditioned Policy Learning from One-Bit Failure Signals

Guopeng Li; Yiyang Duan; Yiru Jiao; Chengcheng Xu

cs.LGAug 27, 2026

Arrive and Survive: Scaling Safe Goal-Conditioned Policy Learning from One-Bit Failure Signals

Authors: Guopeng Li, Yiyang Duan, Yiru Jiao, Chengcheng Xu

Organizations: Southeast University Nanjing, China · Lab, Yinwang Intel. Tech. Co. Ltd. Shanghai, China

Abstract

Contrastive reinforcement learning (CRL) scales effectively in goal-conditioned tasks by casting policy learning into a self-supervised contrastive objective. However, in a failure-terminated Markov decision process, established CRL considers pre-failure future goals only when constructing positive samples, without accounting for the probability mass removed by failure termination. Our theoretical analysis shows that this omission induces a systematic overestimation bias in goal-reaching values. Consequently, near-failure trajectories provide disproportionately strong supervision of success despite retaining little future occupancy. Unsafe actions can thereby be reinforced through catastrophic failure bootstrapping, leading to failed policy learning and unsustainable goal-reaching behaviours. To address this problem, we introduce two minimal yet strong corrections: mass-weighted InfoNCE corrects the overweighting of short surviving futures in critic learning, and a log-survival-mass score restores the missing survival mass in policy optimization. The resulting method, Safe Contrastive Reinforcement Learning (Safe-CRL), requires only the one-bit signal provided by failure termination to scale safe goal-conditioned policy learning. Across twelve failure-prone robot navigation and locomotion tasks, Safe-CRL consistently improves survival and substantially outperforms the Scaling-CRL baseline in goal-reaching performance. Additionally, deep Safe-CRL policies exhibit complex failure-avoidance behaviours. This study completes the CRL theory under failure termination and provides a scalable safe RL framework. The code is available via https://github.com/RomainLITUD/safe-crl.

Figures & tables

Figure 1: Comparison of time at goal between Scaling-CRL and the proposed Safe-CRL method in failure-prone navigation and locomotion tasks (mean

\pm

standard deviation over five random seeds). Both methods use 64-layer actor and critic networks (8-layer for Point Goal and Car Goal). Safe-CRL achieves higher or on-par mean time at goal across all tasks, with pronounced gains in nine.

Figure 2: Examples of unsustainable goal reaching under Scaling-CRL. (a) The Point agent reaches the goal region but subsequently collides with a moving obstacle; (b) the Ant reaches the goal at high speed and overshoots into the wall; (c–d) locomotion agents reach the goal but fall immediately afterwards. These examples illustrate that successful short-term goal reaching does not necessarily lead to a stable trajectory under failure termination.

Figure 3: Failure termination removes part of the discounted future-goal occupancy. Established CRL normalizes the remaining pre-failure futures, whereas Safe-CRL corrects the trajectory-wise normalization distortion in critic learning and restores the missing survival mass in actor updates.

Figure 4: Illustrations of the experimental environments. The goal is represented by a bright green cylinder (for Point and Car Goal) or a meshed sphere (for Ant and Humanoid locomotion tasks). The walls in Maze environments are concrete, while other obstacles, hazards (pitfalls), and moving gremlins (purple spheres in Point Goal and Car Goal) are non-colliding (ghost objects). Contacting maze walls is not a failure; entering a designated task-specific failure region triggers failure termination. The goal is randomly respawned only for Point Goal and Car Goal. The other environments use a fixed commanded goal in each episode.

Figure 5: Survival time (top two rows) and goal coverage (bottom two rows) on the 12-task benchmark (mean

\pm

std).

Figure 6: Time at goal and survival time for Safe-CRL using different actor and critic depths. The Z-encoder depth is 4. Point Goal and Car Goal are omitted from the depth-scaling study because they are easier than locomotion.

Figure 7: Predicted survival mass versus survival time in 128 evaluation runs. Actor and critic depths are 64.

Figure 8: Visualization of the policy learned by Safe-CRL. Each row contains 5 snapshots captured from one evaluation episode, showing the key failure-avoidance behaviours of the agent.

			Car Goal		Humanoid Goal		Ant Big Maze		Ant Hardest Pitfall
Method	MW	$\log Z$	Time at goal	Survival time	Time at goal	Survival time	Time at goal	Survival time	Time at goal	Survival time
Scaling-CRL	✗	✗	2.2 $\pm$ 0.1	464.9 $\pm$ 11.2	13.8 $\pm$ 1.0	113.9 $\pm$ 5.3	89.7 $\pm$ 34.2	261.1 $\pm$ 33.1	98.5 $\pm$ 17.1	381.1 $\pm$ 11.6
MW only	✓	✗	2.2 $\pm$ 0.1	471.4 $\pm$ 13.3	28.3 $\pm$ 0.7	146.1 $\pm$ 10.2	91.4 $\pm$ 31.1	291.6 $\pm$ 51.6	113.2 $\pm$ 19.3	411.0 $\pm$ 27.1
$\log Z$ only	✗	✓	2.4 $\pm$ 0.1	537.1 $\pm$ 61.3	288.7 $\pm$ 21.6	397.0 $\pm$ 21.8	337.0 $\pm$ 28.9	693.5 $\pm$ 33.7	271.2 $\pm$ 28.3	722.9 $\pm$ 32.6
Safe-CRL	✓	✓	2.4 $\pm$ 0.2	555.5 $\pm$ 49.1	297.0 $\pm$ 19.4	413.4 $\pm$ 23.9	352.9 $\pm$ 11.1	725.7 $\pm$ 31.7	313.0 $\pm$ 50.7	787.9 $\pm$ 51.1

Table 1: Core component ablation of Safe-CRL. MW denotes MW-InfoNCE. Results are averaged over the final 10% of training and reported as mean

\pm

standard deviation over three random seeds. The actor and critic depths are 16 (8 for Car Goal), and the Z-encoder depth is 4.

Figure 9: Ablations on the coefficient

\beta

\log Z

. The left four panels report time at goal, and the right four panels report survival time. We set the actor and critic depths to 16 layers (8 for Car Goal) and vary

\beta

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	Agent	Goal Respawn	Failure Termination
Point Goal	Point	Yes	Enter obstacle, hazard, or gremlin region
Car Goal	Car	Yes	Enter obstacle, hazard, or gremlin region
Ant Goal	Ant	No	Fall
Humanoid Goal	Humanoid	No	Fall
Ant H-Maze	Ant	No	Fall
Ant Cross Maze	Ant	No	Fall

Appendix

Table B.1: Summary of the experimental environments and failure mechanisms.

Actuator gear	default	ours
abdomen_y	350	100
abdomen_z	350	100
abdomen_x	350	100
right_hip_x	350	100
right_hip_z	350	100
right_hip_y	350	300

Appendix

Table B.2: Humanoid actuator gears under the default setting and our experiments.

Figure C.1: Comparison with SAC-HER on four representative failure-prone tasks. Curves report time at goal and survival time. Safe-CRL uses 16 layers for the actor and critic networks (8 for Car Goal) and 4 layers for the Z-encoder.

	Time at goal		Survival time		Goal coverage (%)
Environment	Scaling-CRL	Safe-CRL	Scaling-CRL	Safe-CRL	Scaling-CRL	Safe-CRL
Point Goal	$1.94\pm 0.11$	$\mathbf{1.98\pm 0.09}$	$555.1\pm 12.3$	$\mathbf{631.5\pm 21.0}$	$81.6\pm 3.4$	$\mathbf{83.0\pm 1.1}$
Car Goal	$2.17\pm 0.04$	$\mathbf{2.39\pm 0.06}$	$464.9\pm 6.5$	$\mathbf{550.1\pm 15.9}$	$74.2\pm 1.9$	$\mathbf{79.7\pm 0.7}$
Ant Goal	$276.83\pm 12.81$	$\mathbf{684.00\pm 14.57}$	$422.1\pm 16.2$	$\mathbf{910.8\pm 8.0}$	$\mathbf{87.2\pm 0.9}$	$81.0\pm 1.6$
Humanoid Goal	$22.19\pm 2.64$	$\mathbf{266.43\pm 20.70}$	$145.7\pm 12.9$	$\mathbf{386.3\pm 19.9}$	$60.9\pm 0.7$	$\mathbf{78.5\pm 0.9}$
Ant H-Maze	$390.60\pm 52.80$	$\mathbf{412.96\pm 17.92}$	$586.6\pm 66.6$	$\mathbf{795.6\pm 19.6}$	$\mathbf{58.3\pm 4.6}$	$54.0\pm 2.5$

Appendix

Table D.1: Final performance on the main benchmark, reported as mean

\pm

standard deviation over five random seeds.

Figure D.1: Comparison of survival time with respawned and fixed goals in Point Goal and Car Goal. Percentages show the mean advantage of Safe-CRL over Scaling-CRL during the final 10% of training.

Figure D.2: Ablation on Z-encoder depth. Actor and critic depths are fixed at 16 layers.

Figure D.3: Ablations on TD-learned Z. Actor and critic depths are set to 16 layers (8 for Car Goal).

Environment	Scaling-CRL	Safe-CRL (ours)
Point Goal	2.92 h	3.07 h
Car Goal	8.73 h	8.94 h
Ant Goal	6.65 h	6.71 h
Humanoid Goal	7.28 h	7.74 h
Ant H-Maze	10.20 h	10.58 h
Ant Cross Maze	10.48 h	10.64 h

Appendix

Table D.2: Average wall-clock training time per 100M environment steps.

Panel of Figure 19 — Figure D.4: Success rate for goals at different places in the playground. Darker colors indicate lower goal-reaching success rates. The robot initially faces the +x axis direction ( $\rightarrow$ ). The diamond marks the initial position of the robot.

Explore similar work

May 29, 2026

Survival Reinforcement Learning: Toward Scalable Self-Supervised RL

While self-supervised Contrastive Reinforcement Learning (CRL) has shown remarkable depth-scaling capabilities, successfully using networks over 64 layers, scaled CRL still struggles with long-horizon goal-conditioned planning due to the uniformity-tolerance dilemma inherent in contrastive losses. We introduce Survival Reinforcement Learning (SRL), an online classification-based alternative that extends the survival value learning framework by maximizing the agent's dwell time at target goals. SRL bypasses the structural constraints of CRL and mitigates the "bang-bang" control solutions inherent to survival frameworks, which often induce undesirable behavior in complex dynamical systems. Evaluated across diverse robotic benchmarks, scaled SRL matches state-of-the-art CRL on manipulation tasks and outperforms it by 2x to 8x on stable, long-horizon locomotion tasks. Our results provide strong additional evidence that classification-based methods may serve as a key primitive in the broader effort to scale reinforcement learning.

Franki Nguimatsia-Tiofack, Fabian Schramm, Théotime Le Hellard +1

Inria and École Normale Supérieure PSL Research University Paris, France

Reinforcement Learning Goal-Conditioned RL

Ask PDF

Oct 10, 2026

Counterfactual Rollouts for Offline Contrastive Goal-Conditioned Reinforcement Learning

Contrastive reinforcement learning (CRL) learns goal-conditioned values by associating state--action pairs with the states subsequently reached, assuming that no unobserved variable jointly affects the recorded action and its outcome. When offline demonstrations come from a demonstrator with information hidden from the learner, this assumption fails and the critic absorbs spurious action--outcome associations. We formalize this setting with confounded Markov decision processes and introduce counterfactual transitions that separate what the data identify from what they do not; minimizing over all compatible transitions yields a worst-case occupancy that lower-bounds the true one. Causal Contrastive Reinforcement Learning (CauCRL) learns a pessimistic counterfactual transition from positive and negative offline demonstrations and trains the critic on counterfactual rollouts, substantially improving success rates over standard CRL on confounded goal-reaching tasks.

Tingrui Huang, Ai Bo, Junzhe Zhang

Eindhoven University of Technology, Eindhoven, The Netherlands · Department of Electrical Engineering and Computer Science, Syracuse University, Syracuse, NY, USA

Reinforcement Learning Offline RL Contrastive RL Goal-Conditioned RL Causal RL

Ask PDF

Apr 19, 2026

SVL: Goal-Conditioned Reinforcement Learning as Survival Learning

Standard approaches to goal-conditioned reinforcement learning (GCRL) that rely on temporal-difference learning can be unstable and sample-inefficient due to bootstrapping. While recent work has explored contrastive and supervised formulations to improve stability, we present a probabilistic alternative, called survival value learning (SVL), that reframes GCRL as a survival learning problem by modeling the time-to-goal from each state as a probability distribution. This structured distributional Monte Carlo perspective yields a closed-form identity that expresses the goal-conditioned value function as a discounted sum of survival probabilities, enabling value estimation via a hazard model trained via maximum likelihood on both event and right-censored trajectories. We introduce three practical value estimators, including finite-horizon truncation and two binned infinite-horizon approximations to capture long-horizon objectives. Experiments on offline GCRL benchmarks show that SVL combined with hierarchical actors matches or surpasses strong hierarchical TD and Monte Carlo baselines, excelling on complex, long-horizon tasks. Webpage and Code: https://simple-robotics.github.io/publications/survival-value-learning/

Franki Nguimatsia Tiofack, Fabian Schramm, Théotime Le Hellard +1

Inria and École Normale Supérieure, PSL Research University, Paris, France.

Reinforcement Learning Goal-Conditioned RL

Ask PDF