Accelerating Safe Reinforcement Learning with Massive Parallelism

Joonyoung Lim; Younghwan Yoo

cs.LGNov 5, 2026

Accelerating Safe Reinforcement Learning with Massive Parallelism

Authors: Joonyoung Lim, Younghwan Yoo

Organizations: Department of Information Convergence Engineering, Pusan National University · School of Computer Science and Engineering, Pusan National University

Abstract

We investigate the problem of scaling safe reinforcement learning (safe RL) to massively parallel training regimes, where a large number of environments are exe- cuted with short rollout horizons. Although this setting improves data throughput and wall-clock efficiency, it creates a mismatch with constrained Markov decision processes (CMDPs), whose safety constraints are defined over full-episode cost returns. We address this mismatch by analyzing staggered environment resets as a phase-mixture distribution over short trajectory segments. This analysis charac- terizes the bias induced by phase coverage and motivates a phase-aggregated cost estimator that reconstructs episode-level cost estimates from short rollout batches. We further incorporate finite-sample uncertainty and distribution mismatch into a constraint-tightening framework, yielding a scalable Safe RL method compatible with on-policy optimization. Using an MJX-based implementation of Safety Gym, our experiments show that the proposed method achieves performance comparable to non-massively parallel implementations while substantially reducing wall-clock training time.

Figures & tables

Init	Cost est.	Env	Point/Goal				Car/Goal
			w/o tightening		w/ tightening		w/o tightening		w/ tightening
			Reward	Cost	Reward	Cost	Reward	Cost	Reward	Cost
Standard	LR	2K	15.13 $\pm$ 9.94	55.48 $\pm$ 24.54	–	–	29.14 $\pm$ 4.76	65.87 $\pm$ 12.06	–	–
Staggered	SC	2K	14.57 $\pm$ 2.78	27.08 $\pm$ 0.84	7.56 $\pm$ 3.06	13.88 $\pm$ 1.59	25.12 $\pm$ 2.11	28.63 $\pm$ 1.46	18.52 $\pm$ 2.83	14.38 $\pm$ 1.18
		4K	17.28 $\pm$ 5.37	26.27 $\pm$ 1.23	13.42 $\pm$ 3.52	20.00 $\pm$ 3.49	27.57 $\pm$ 1.76	26.87 $\pm$ 1.18	24.38 $\pm$ 2.04	18.43 $\pm$ 0.75
		8K	21.23 $\pm$ 1.05	27.02 $\pm$ 1.50	18.83 $\pm$ 0.73	21.19 $\pm$ 2.05	32.17 $\pm$ 0.95	26.52 $\pm$ 1.38	29.57 $\pm$ 0.70	21.06 $\pm$ 0.70

Table 1: Evaluation of CPO across 2k, 4k, and 8k parallel environments with a fixed rollout length of 10. Values are mean ± standard deviation of the last 10 epochs over eight seeds. Shading indicates violation of the cost threshold (cost

>

25). Bold highlights, for each environment count, the highest reward that satisfies the cost threshold.

Panel of Figure 1 — Figure 1: Wall-clock wise and total sample wise learning curves of all algorithms on the Goal task. Columns correspond to CPO, CRPO, and TRPO-Lag. Markers are shown every 100 epochs and report trailing 10-epoch averages, and the horizontal line marks the cost limit.

Panel of Figure 2 — Figure 2: Learning curves of CRPO on the Laikago locomotion task. Markers are shown every 100 epochs and report trailing 10-epoch averages. Costs are normalized by their respective limits, with horizontal lines indicating the normalized cost limit of 1.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Agent	Horizon	Objects	Cost event	Limit
Goal	Point	$1000$	goal, $8$ hazards, $1$ vase	hazard proximity	$25$
Goal	Car	$1000$	goal, $8$ hazards, $1$ vase	hazard proximity	$25$

Appendix

Table 2: Goal-task configurations. The cost limit specifies the maximum cumulative episodic cost.

Config	CPO	CRPO	TRPO-Lag
Critic learning rate	$10^{-3}$
Discount factor	$0.99$
Reward trace parameter	$1.0$
Cost trace parameter	$1.0$
Target KL	$0.01$
CG iterations	$15$

Appendix

Table 3: Algorithm configuration. Rows with shared values span all three algorithms.

Task

Batch size

Fixed envs (2k)

Fixed rollout (10)

Rollout

w/o tightening

w/ tightening

Envs

w/o tightening

w/ tightening

Reward

Cost

Time

Reward

Cost

Time

Reward

Cost

Time

Reward

Cost

Time

Point/Goal

20k

14.57

\pm

2.78

27.08

\pm

0.84

4.10

7.56

\pm

3.06

13.88

\pm

1.59

4.09

–

40k

17.96

\pm

1.12

27.05

\pm

1.10

7.77

13.96

\pm

1.41

15.05

\pm

0.63

7.88

17.28

\pm

5.37

26.27

\pm

1.23

6.84

13.42

\pm

3.52

20.00

\pm

3.49

7.09

80k

20.46

\pm

0.48

27.18

\pm

0.91

15.91

16.43

\pm

0.81

16.36

\pm

0.39

16.20

21.23

\pm

1.05

27.02

\pm

1.50

12.87

18.83

\pm

0.73

21.19

\pm

2.05

13.03

Car/Goal

20k

25.12

\pm

2.11

28.63

\pm

1.46

12.92

18.52

\pm

2.83

14.38

\pm

1.18

13.03

–

Appendix

Table 4: Comparison of increasing rollout length and environment scale at the same batch size using CPO with the SC surrogate. Values are mean

\pm

standard deviation of the last 10 updates over eight seeds. Shading indicates cost

>

25. Bold highlights the highest feasible reward and its corresponding cost and time for each task and batch size.

Cost est.	Envs	No tightening		$\varepsilon$ only		$\beta$ only		Both ( $\varepsilon+\beta$ )
Cost est.	Envs	Reward	Cost	Reward	Cost	Reward	Cost	Reward	Cost
SC	2k	14.57 $\pm$ 2.78	27.08 $\pm$ 0.84	12.87 $\pm$ 3.83	25.77 $\pm$ 1.87	9.55 $\pm$ 4.08	17.43 $\pm$ 2.39	7.56 $\pm$ 3.06	13.88 $\pm$ 1.59
	4k	17.28 $\pm$ 5.37	26.27 $\pm$ 1.23	16.44 $\pm$ 3.21	27.10 $\pm$ 2.27	12.71 $\pm$ 1.99	18.19 $\pm$ 1.91	13.42 $\pm$ 3.52	20.00 $\pm$ 3.49
	8k	21.23 $\pm$ 1.05	27.02 $\pm$ 1.50	21.27 $\pm$ 0.69	27.02 $\pm$ 1.62	18.68 $\pm$ 1.43	22.69 $\pm$ 1.70	18.83 $\pm$ 0.73	21.19 $\pm$ 2.05
	16k	23.07 $\pm$ 0.46	25.73 $\pm$ 0.84	23.09 $\pm$ 0.48	25.72 $\pm$ 1.20	22.09 $\pm$ 0.55	21.86 $\pm$ 1.02	22.30 $\pm$ 0.68	22.66 $\pm$ 1.41
	32k	23.81 $\pm$ 0.64	25.07 $\pm$ 0.33	24.37 $\pm$ 0.54	25.88 $\pm$ 0.74	23.48 $\pm$ 0.70	23.19 $\pm$ 0.42	23.43 $\pm$ 0.70	23.06 $\pm$ 0.59

Appendix

Table 5: Ablation of finite-sample tightening (

\beta

) and phase-initial distribution correction (

\varepsilon

) across environment scales using SC and CV surrogates. Values are mean

\pm

standard deviation of the last 10 epochs over eight seeds. Shading indicates cost

>

25.

Safe RL

Safe reinforcement learning studies how agents can learn and act while respecting safety constraints and limiting harmful outcomes.

See all 228 papers

Provably Safe, Yet Scalable Reinforcement Learning
Jun 12, 2026Kai S. Yun, Zeyang Li, Navid AzizanRL Control Safety-Critical Control
MIT
Evaluation Metrics for Safe Reinforcement Learning
Sep 14, 2026Lindsay Spoor, Aske Plaat, Thomas MoerlandRL Benchmarks Constrained RL
Leiden Institute of Advanced Computer Science, Leiden University, Leiden, the Netherlands
Sampling-Based Safe Reinforcement Learning
May 19, 2026Luca Vignola, Bruce D. Lee, Manish Prajapat +4Robot Safety Safe RL
ETH Zurich

Jun 12, 2026J/K move · Enter open · S save

Provably Safe, Yet Scalable Reinforcement Learning

Kai S. Yun, Zeyang Li, Navid Azizan

MIT

RL Control Safety-Critical Control Robotic RL Safety Filtering Safe RL

Open paper AskPDF arXiv

Safe reinforcement learning (RL) aims to learn policies that optimize rewards while satisfying constraints. Predominant approaches rely on soft-constrained policy optimization, which has achieved empirical success but does not provide formal safety guarantees for the learned policy. In contrast, methods with strict guarantees typically rely on explicit certificate functions, whose construction requires the direct synthesis and verification of control-invariant sets, a process that scales poorly with state dimension and often yields overly conservative behavior. In this paper, we present the Provably Safe, yet Scalable RL (PS2-RL) framework, a novel two-phase architecture for learning provably safe policies in a scalable manner, designed to overcome the key bottlenecks of prior methods. Rather than explicitly computing invariant sets, PS2-RL leverages a learned backup policy to forward-integrate the system dynamics, generating an implicit control-invariant set online. In the first phase, the backup policy is trained with the safe-arrival value function to achieve rapid, safe recovery to the certified base set for invariant-set construction. In the second phase, an RL policy is trained end-to-end through a differentiable projection layer that strictly enforces the safety guarantees induced by the learned backup policy. By maximizing the volume of the implicit control-invariant set in the first phase, the resulting PS2 policy from the second phase is performant and scalable, while maintaining provable safety. Crucially, PS2-RL imposes no restrictions on the underlying RL algorithm and can be plugged into any existing training pipeline. We establish theoretical guarantees for the proposed framework and evaluate it on robotic control tasks with state dimensions up to 10, a regime in which prior provably safe RL methods struggle or become impractical.

Abstract

Safe RL

Explore similar work

Provably Safe, Yet Scalable Reinforcement Learning

Evaluation Metrics for Safe Reinforcement Learning

Sampling-Based Safe Reinforcement Learning