Structural Dual Adaptation for Safe Reinforcement Learning

Runsheng Yu; Zhuoyang Song; Zengde Deng; Qixin Zhang; Li Shen; Jiaxing Zhang

cs.LGOct 31, 2026

Structural Dual Adaptation for Safe Reinforcement Learning

Authors: Runsheng Yu, Zhuoyang Song, Zengde Deng, Qixin Zhang, Li Shen, Jiaxing Zhang

Organizations: China Merchants Research Institute of Advanced Technology, China Merchants Group, Hong Kong, China · The Chinese University of Hong Kong, Hong Kong, China · Generative AI Lab, College of Computing and Data Science, Nanyang Technological University, Singapore · the School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, Shenzhen, China

Abstract

Primal-dual methods enforce safety constraints in reinforcement learning through Lagrangian relaxation. For a fixed policy, the exact bounded multiplier response exhibits bang-bang switching at the constraint boundary. Practical gradient updates can oscillate through coupling with policy optimization. We propose Structural DuaL Adaptation (SDLA), which replaces this bang-bang response with a unique continuous response through current-cost centering, trajectory weighting, and quadratic regularization. We characterize activation thresholds and response sensitivity. Ternary search computes multipliers without a dual gradient stepsize. For entropy-regularized tabular policies, we bound equilibrium constraint violations alongside reward gaps. We establish convergence of delayed updates under mild assumptions. Neural PPO/SAC experiments on OmniSafe and HASARD demonstrate competitive rewards with improved constraint tracking. On HASARD, PPO-SDLA achieves higher reported mean reward than the human baseline in six of eighteen settings while meeting the respective mean-cost budgets, including four with lower mean cost. To the best of our knowledge, this is the first report of higher mean reward at lower mean cost than the human baseline across multiple HASARD settings.

Figures & tables

Method	Inner Solve	Avoids Bang-Bang
Lagrangian	$O(1)$ (closed form)	✗
Lagrangian-PID	No closed form	—
Entropy-Regularized	$O(1)$ (closed form)	✗
SDLA (ours)	$O(\log(\lambda_{\max}/\varepsilon_{\lambda}))$	✓

TABLE I: Single-constraint multiplier responses with policy and return statistics fixed. Inner solve counts scalar optimization steps to multiplier accuracy

\varepsilon_{\lambda}

. Linear objectives admit an exact closed-form solution ( 4 ); SDLA uses ternary search. Bang-bang refers to the exact response. For PID, “No closed form” denotes feedback updates without a closed-form optimal response; — denotes not applicable. Entropy-Regularized refers to policy entropy.

Level	Method	Armament		Volcanic		Remedy		Collateral		Precipice		Detonator’s
		Burden		Venture		Rush		Damage		Plunge		Dilemma
		$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$
1	PPO	9.68	109.30	51.64	172.93	50.78	52.44	78.61	41.06	243.42	475.62	29.67	14.28
	PPOCost	5.47	3.67	30.73	5.44	28.21	6.75	68.71	24.04	237.72	0.69	27.60	8.45
	PPOLag	7.51	52.41	42.40	52.00	36.37	5.25	29.09	5.61	147.24	44.96	21.49	5.62
	PPOSaute	2.33	32.87	32.68	62.00	20.50	9.18	26.08	7.09	241.37	424.35	28.69	8.78

TABLE II: Rewards and costs over five SDLA runs and available archived baseline runs. The cost threshold is 5 for Remedy Rush, Collateral Damage, and Detonator’s Dilemma, and 50 for all other environments. Green : mean cost meets the budget. Bold Green : highest mean reward among methods meeting the mean cost budget.

Level	Method	Armament		Volcanic		Remedy		Collateral		Precipice		Detonator’s
		Burden		Venture		Rush		Damage		Plunge		Dilemma
		$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$
1	Human	18.60	42.83	57.12	45.97	51.04	5.65	42.33	4.11	243.09	46.88	31.54	4.49
1	PPO-SDLA (ours)	19.46	12.80	44.49	29.49	39.07	2.71	43.96	3.01	243.30	43.09	27.24	2.97
2	Human	16.85	31.50	42.91	45.09	34.72	1.22	17.34	3.10	226.11	44.35	34.66	3.09
2	PPO-SDLA (ours)	14.22	19.01	33.71	28.01	44.02	4.67	31.09	4.44	172.40	40.98	22.72	4.19

TABLE III: Human player rewards and costs averaged over 10 episodes compared to PPO-SDLA terminal training summaries. Bold : PPO-SDLA achieves higher reported mean reward than human.

Method	Ant		HalfCheetah		Hopper		Swimmer		Walker2d
Method	$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$
PPO	5977.73 $\pm$ 885.65	958.13 $\pm$ 134.50	6921.83 $\pm$ 1721.79	919.20 $\pm$ 173.08	2337.11 $\pm$ 942.06	550.02 $\pm$ 237.70	119.77 $\pm$ 13.80	165.27 $\pm$ 20.15	6239.52 $\pm$ 879.99	902.68 $\pm$ 100.93
PPOLag	3205.96 $\pm$ 85.16	22.46 $\pm$ 4.82	2939.58 $\pm$ 307.19	18.72 $\pm$ 2.87	961.92 $\pm$ 752.87	13.96 $\pm$ 19.33	64.74 $\pm$ 17.67	28.02 $\pm$ 4.09	2982.27 $\pm$ 681.55	13.49 $\pm$ 14.55
P3O	2636.62 $\pm$ 181.09	20.69 $\pm$ 10.23	2117.84 $\pm$ 313.55	27.60 $\pm$ 8.36	1231.52 $\pm$ 465.35	16.33 $\pm$ 11.38	38.02 $\pm$ 34.18	18.40 $\pm$ 12.13	2713.57 $\pm$ 313.20	20.51 $\pm$ 14.09
FOCOPS	3184.48 $\pm$ 305.59	14.75 $\pm$ 6.36	2965.20 $\pm$ 290.43	2.37 $\pm$ 3.50	1437.75 $\pm$ 446.87	10.13 $\pm$ 8.87	53.87 $\pm$ 17.90	29.75 $\pm$ 7.33	3117.05 $\pm$ 53.60	8.78 $\pm$ 12.38
PPOSaute	2978.74 $\pm$ 93.65	16.77 $\pm$ 0.92	2901.40 $\pm$ 25.49	16.20 $\pm$ 0.60	1650.91 $\pm$ 152.65	17.87 $\pm$ 1.33	35.61 $\pm$ 4.37	3.44 $\pm$ 1.35	2410.89 $\pm$ 241.22	18.88 $\pm$ 2.38

TABLE IV: Performance on OmniSafe’s MuJoCo Velocity Benchmark. Mean

\pm

std over 5 random seeds. Green : mean cost

\leq

25. Bold Green : highest mean reward within the same algorithm family (PPO-based or SAC-based) among methods meeting this budget.

Fig. 1: HASARD Level 1: reward (top) and cost (bottom). Lines show means and shading one standard deviation over five SDLA runs and three runs per baseline. Dashed lines mark budgets of 50 for Armament Burden and 5 for the other tasks. PPOCost shows task reward with its logged penalty added back.

Fig. 2: HASARD Level 2: reward (top) and cost (bottom) for the same four tasks, with methods, budgets, and statistical conventions as in Figure 1 .

Task	Method	Mean $\|\Delta\lambda\|$	Violation (%)	Overshoot	CrossRate (%)
Ant	PPOLag	$0.0075\pm 0.0007$	$38.7\pm 3.5$	$3.2546\pm 0.2850$	$5.2\pm 0.8$
Ant	PPO-SDLA	$0.0865\pm 0.0162$	$6.7\pm 4.5$	$0.1807\pm 0.1653$	$3.4\pm 1.8$
Walker2d	PPOLag	$0.0115\pm 0.0031$	$41.7\pm 2.1$	$4.8960\pm 0.4163$	$9.0\pm 2.1$
Walker2d	PPO-SDLA	$0.1154\pm 0.0380$	$13.0\pm 6.9$	$1.4061\pm 1.2572$	$4.0\pm 1.2$

TABLE V: Training-time constraint tracking with

d=25

. All statistics use the complete training logs. Violation and CrossRate are percentages; overshoot is averaged over all logged updates, including those below the budget.

Ant
Algorithm	Reward ( $\uparrow$ )	Cost ( $\downarrow$ )	Violation (%, $\downarrow$ )
PPO-SDLA (ours)	$3200.4\pm 75.6$	$\mathbf{19.73\pm 2.46}$	$\mathbf{6.7\pm 4.5}$
PPOEntropy	$3136.4\pm 86.6$	$40.86\pm 22.39$	$27.8\pm 3.5$
PPO-QR	$3226.8\pm 124.7$	$51.35\pm 50.00$	$52.9\pm 7.2$
PPO-QR- $\frac{d}{2}$	$3169.3\pm 112.6$	$25.47\pm 0.37$	$24.8\pm 4.9$
Swimmer

TABLE VI: Adaptive feedback versus fixed early penalization on Ant and Swimmer (

d=25

; five runs per method). Entries report mean

\pm

standard deviation. PPO-QR-

d/2

uses

\beta=0.2

versus SDLA’s

\beta=0.1

, matching their constant-cost activation threshold and response slope. Violation % is the fraction of logged updates exceeding the cost limit.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
CMDP
$\mathcal{S},\mathcal{A},P$	State space, action space, and transition kernel
$r,c,\gamma,d$	Reward, cost, discount factor, and cost budget
$J^{r}(\pi),J^{c}(\pi)$	Expected discounted reward and cost under policy $\pi$
$R,C$	Complete discounted trajectory returns
$\bar{C}$	Cost-return bound $C_{\max}/(1-\gamma)$

Appendix

TABLE S1: Notation Table

\mathcal{T}=\{\xi_{i}\}_{i=1}^{M}.

Appendix

Algorithm S2 PPO-SDLA

R_{i}=\sum_{t\geq 0}\gamma^{t}r_{t}^{i},\qquad C_{i}=\sum_{t\geq 0}\gamma^{t}c_{t}^{i},\qquad i=1,\ldots,M.

Appendix

Algorithm S3 SAC-SDLA

Fig. S1: Reward, cost, and logged multiplier trajectories on Ant (top) and Swimmer (bottom). Lines show means over five distinct seeds per method and environment; shading shows one standard deviation, clipped at zero for cost and

\lambda

. Curves use all 500 logged points without temporal smoothing. Dashed cost lines mark

d=25

. Swimmer uses

\tau=1

, with

\beta=10

for PPO-QR,

\beta=0.1

for SDLA, and

\beta=0.2

for PPO-QR-

d/2

$\beta$	$\tau$	Reward ( $\uparrow$ )	Cost ( $\downarrow$ )	Mean $\|\Delta\lambda\|$
0.1	1	$3200.44\pm 75.58$	$19.73\pm 2.46$	$0.0865\pm 0.0162$
0.5	1	$3132.15\pm 115.28$	$12.44\pm 3.12$	$0.3587\pm 0.0250$
1.0	1	$3088.90\pm 97.37$	$15.31\pm 3.31$	$0.4969\pm 0.0598$
2.0	1	$3095.12\pm 129.66$	$10.49\pm 4.61$	$0.5589\pm 0.0671$
5.0	1	$3107.92\pm 50.84$	$11.65\pm 5.40$	$0.7639\pm 0.1144$
$\infty$	1	$3086.35\pm 93.44$	$12.33\pm 5.18$	$0.8037\pm 0.1017$

Appendix

TABLE S2: Regularization ablation on SafetyAntVelocity-v1 with

d=25

. The

\beta

sweep uses a 10M-step horizon and

\tau=1

;

\beta=\infty

removes quadratic regularization. Mean

|\Delta\lambda|

measures the average change between consecutive logged multipliers.

$\tau$	Reward ( $\uparrow$ )	Cost ( $\downarrow$ )	Mean $\|\Delta\lambda\|$
1	3200.44 $\pm$ 75.58	$19.73\pm 2.46$	$0.0865\pm 0.0162$
1.5	$3192.72\pm 62.06$	$20.24\pm 1.36$	$0.08649\pm 0.01625$
2	$3098.98\pm 61.66$	$20.34\pm 1.07$	$0.08890\pm 0.00939$

Appendix

TABLE S3: Temperature ablation on SafetyAntVelocity-v1 with

d=25

and the default

\beta=0.1

. Entries report mean

\pm

standard deviation. Mean

|\Delta\lambda|

reports the average absolute change between consecutive logged multipliers. Bold marks the highest mean reward.

Panel of Figure 15 — Fig. S2: PPO-SDLA on Safety-Gym velocity tasks. Rows show episode reward, episode cost, and the dual variable $\lambda$ , respectively.

Fig. S3: Full-training constraint tracking on Ant and Walker2d, using the values in main Table V . Points show means and error bars show one standard deviation. Panels report mean absolute multiplier change, violation rate, average overshoot, and constraint-boundary crossing rate. The cost budget is

d=25

; overshoot includes all logged updates.

Environment	Near-cap frequency (%)
SafetyAntVelocity-v1	0.000
SafetyHalfCheetahVelocity-v1	0.000
SafetyHopperVelocity-v1	0.000
SafetySwimmerVelocity-v1	2.400
SafetyWalker2dVelocity-v1	0.000

Appendix

TABLE S4: Fraction of logged PPO-SDLA updates with

\lambda_{t}\geq 0.95\lambda_{\max}

, using

\lambda_{\max}=10

Panel of Figure 18 — Fig. S4: SAC-SDLA on Safety-Gym velocity tasks. Rows show episode reward, episode cost, and the dual variable $\lambda$ , respectively.

Panel of Figure 19 — Fig. S5: Learning curves on HASARD Level 1 (Part 1): Armament Burden, Volcanic Venture, Remedy Rush.

Panel of Figure 20 — Fig. S6: Learning curves on HASARD Level 1 (Part 2): Collateral Damage, Precipice Plunge, Detonator’s Dilemma.

Panel of Figure 21 — Fig. S7: Learning curves on HASARD Level 2 (Part 1): Armament Burden, Volcanic Venture, Remedy Rush.

Panel of Figure 22 — Fig. S8: Learning curves on HASARD Level 2 (Part 2): Collateral Damage, Precipice Plunge, Detonator’s Dilemma.

Panel of Figure 23 — Fig. S9: Learning curves on HASARD Level 3 (Part 1): Armament Burden, Volcanic Venture, Remedy Rush.

Panel of Figure 24 — Fig. S10: Learning curves on HASARD Level 3 (Part 2): Collateral Damage, Precipice Plunge, Detonator’s Dilemma.

Primal-Dual RL

See all 8 papers

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning
Jun 12, 2026Ayoub Belouadah, Sylvain Kubler, Yves Le TraonPrimal-Dual RL Constrained RL
SnT, University of Luxembourg
Augmented Lagrangian Multiplier Network for State-wise Safety in Reinforcement Learning
May 1, 2026Jiaming Zhang, Yujie Yang, Yao Lyu +2Constrained RL Augmented Lagrangian Method
School of Vehicle and Mobility & College of AI, Tsinghua University, Beijing, 100084, China
SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints
Dec 29, 2025Dominik Wagner, Ankit Kanwar, Luke OngSafety-Critical Control Policy Optimization
NTU Singapore · Sony Corporation

Jun 12, 2026J/K move · Enter open · S save

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

Ayoub Belouadah, Sylvain Kubler, Yves Le Traon

SnT, University of Luxembourg

Primal-Dual RL Constrained RL Safe RL

Open paper AskPDF arXiv

Safe reinforcement learning (Safe RL) aims to maximize expected return while satisfying safety constraints, typically modeled as Constrained Markov Decision Processes (CMDPs). While primal-dual methods scale well to deep RL, they often suffer from delayed constraint correction, leading to oscillatory behavior and prolonged safety violations. In this paper, we propose Constraint-Sensitive Policy Optimization (CSPO), a first-order primal-dual method that incorporates local constraint sensitivity into policy updates. CSPO augments the primal objective with a constraint-sensitive correction derived from the shortest signed distance to the safety boundary, enabling smarter recovery steps back to safety, compensating for delayed Lagrange multiplier updates, reducing oscillations near the boundary, and preserving the KKT solutions of the original constrained problem. Experiments on navigation and locomotion benchmarks demonstrate that CSPO achieves faster safety recovery and high reward preservation, resulting in higher constrained returns compared to state-of-the-art primal-dual and penalty-based methods

Abstract

Primal-Dual RL

Explore similar work

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

Augmented Lagrangian Multiplier Network for State-wise Safety in Reinforcement Learning

SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints