LOCU: Lowdin-Orthogonalized Constraint Updates for Multi-Constraint Policy Optimization

Joonyoung Lim; Younghwan Yoo

cs.AIOct 19, 2026

LOCU: Lowdin-Orthogonalized Constraint Updates for Multi-Constraint Policy Optimization

Authors: Joonyoung Lim, Younghwan Yoo

Organizations: Department of Information Convergence Engineering, Pusan National University · School of Computer Science and Engineering, Pusan National University

Abstract

We investigate multi-constraint policy optimization in constrained Markov decision processes (CMDPs), where interactions among constraint gradients often yield ill-conditioned update directions, causing numerically fragile steps and unintended cancellation or redundant overlap in constraint corrections. To address this, we propose LOCU, which mitigates numerical instability arising from constraint interactions by applying symmetric Lowdin orthogonalization to the natural gradients of the reward and constraints under the Fisher-Rao metric. Combined with Fisher-geometric screening and near-collinear compression, LOCU yields a compact reduced space in which the trust region becomes a Euclidean ball and constraints are handled symmetrically through a low-dimensional active-set solve. For infeasible iterates, LOCU constructs a Pareto-descent direction which preserves the local feasibility of satisfied constraints within the same reduced formulation. Experiments show that LOCU remains stable under high constraint coupling, narrow feasible regions, and infeasible starts, and generalizes across different cost critic architectures.

Figures & tables

Figure 1: Final evaluation on the v2 Circle tasks, with high-velocity and boundary constraints.

Figure 2: Final evaluation results on BC-v3 variants. Top: the low-velocity requirement $\kappa_{\mathrm{lo}}\in\{0.0,0.4,0.6,0.8\}$ . Bottom: redundancy variants with increasing numbers of same-type boundary and velocity constraints. The suffix “–a” marks the tightest constraint among the same type.

Figure 3: Final evaluation on the locomotion tasks Cassie and Laikago with three simultaneous constraints. Costs 0, 1, and 2 denote the balance, height, and foot timing constraints. Mean–std constraints use $\alpha_{\mathrm{lo}}\in\{0.05,0.25\}$ . Cost axes are shown on a log scale.

Figure 4: Final evaluation on the Cassie goal task with six constraints. Costs 0, 1, and 2 denote the balance, height, and foot timing constraints. And costs 3, 4 and 5 denote speed, hazard contact, energy constraints. Cost axes are shown on a log scale and 0.25 is used for $\alpha_{\mathrm{lo}}$ .

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Agents used in the multi-constraint evaluation. Ball, Car, and Drone are BulletSafetyGym agents, while Cassie and Laikago are bipedal and quadrupedal robots, respectively.

Figure 6: Geometric comparison between LOCU and PCPO updates. LOCU solves a reduced QP and produces an order-invariant update under the local model. PCPO applies sequential projections onto constraint halfspaces, which can produce order-dependent update directions.

Coefficient	Candidate values
$K_{P}$	$\{0.25,\,1\}$
$K_{I}$	$\{10^{-4},\,10^{-3},\,10^{-2},\,10^{-1},\,1\}$
$K_{D}$	$\{0.01,\,0.1,\,1\}$

Appendix

Table 3: PID coefficient candidate values used in our experiments.

Figure 7: Learning-curve comparison of PID PPO-Lagrangian under P I D coefficient variations on the Bullet Safety Gym BC-v3 task. Dashed lines in the cost plots indicate the corresponding limits.

Figure 8: Performance comparison of Löwdin and Gram–Schmidt orthogonalization under observation noise on BC-v3 Redun 3.

Task	LOCU	SDAC(A)	PCRPO	CRPO	TRPOL -GS	TRPOL -V	PPOL -V	PPOL -GS
BC v2	17:08	18:20	17:30	17:23	16:08	16:27	17:11	18:29
BC v3 Redun 3	25:02	25:40	23:00	22:53	17:02	17:05	22:28	23:40

Appendix

Table 4: Wall-clock time on the main circle task.

Task	Redun	LOCU	SDAC(A)	CRPO	TRPOL	PPOL (bs=64)	PPOL (bs=128)
PointCircle	2	10:19	10:55	10:18	10:22	21:08	15:37
	3	10:31	11:08	10:19	10:25	21:13	15:40
	5	10:44	11:19	10:22	10:29	21:20	15:45

Appendix

Table 5: Wall-clock time on Circle tasks with parallel processing.

Figure 9: Learning curves for w/o Lowdin and w/o Lowdin, Screening & compression , compared with the CPO baseline.

Figure 10: Ablation on the clustering threshold $\varepsilon_{\mathrm{dup}}$ for near-collinear constraint grouping on BC-v3 Redun : 3.

Figure 11: Sensitivity analysis on the KL divergence limit compared against SDAC(A) on BC-v3.

Figure 12: Ablation on threshold, damping, and CG tolerance values on BC-v3 Redun : 3.

Figure 13: Final evaluation comparison between the original 6-constraint Cassie task and the redundant 18-constraint version.

Phase	Basis dim.	Active set	Redun.	Inactive	Zero-grad
$0\text{--}20\%$	$5.7\pm 0.9$	$9.5\pm 1.6$	$3.8\pm 1.0$	$8.5\pm 1.6$	$2.5\pm 1.2$
$20\text{--}40\%$	$3.8\pm 1.5$	$6.5\pm 3.1$	$2.7\pm 1.6$	$11.5\pm 3.1$	$3.2\pm 1.7$
$40\text{--}60\%$	$3.9\pm 1.7$	$6.6\pm 4.0$	$2.7\pm 1.8$	$11.4\pm 4.0$	$1.6\pm 1.1$
$60\text{--}80\%$	$3.1\pm 1.5$	$5.5\pm 4.2$	$2.4\pm 1.7$	$12.5\pm 4.2$	$1.2\pm 0.5$
$80\text{--}100\%$	$2.8\pm 1.4$	$4.9\pm 4.0$	$2.1\pm 1.5$	$13.1\pm 4.0$	$0.9\pm 0.4$

Appendix

Table 6: Screening statistics during training in the 18-constraint redundant Cassie task.

Figure 14: Learning curves on the Bullet Safety Gym v2 suite for BC, CC, and DC.

Figure 15: Learning curves on BC-v3 under the low velocity sweep from the v2 case to $\kappa_{\mathrm{lo}}\in\{0.4,0.6,0.8\}$ .

Figure 16: Learning curves on BC-v3 under redundancy variants redun : 1, redun : 2, and redun : 3. The "-a" denotes the tightest redundant constraint.

Figure 17: Learning curves of LOCU with different hyperparameters on BC-v3 under redundancy variants redun : 1 and redun : 3.

Figure 18: Learning curves on the locomotion tasks with Cassie and Laikago under three simultaneous constraints.

Figure 19: Learning curves on the goal-object task with Cassie under additional three simultaneous constraints.

Figure 20: Learning curves for the goal-objective task with Cassie and the redundant 18-constraint version.

Figure 21: Learning curves for the mjx implementation of circle tasks.

Figure 22: Learning curves for the orthogonalization comparison.

Constrained RL

Constrained reinforcement learning optimizes task performance while keeping safety risks, resource use, or other costs within explicit limits.

See all 223 papers

Constrained Group Relative Policy Optimization
Feb 5, 2026Roger Girgis, Rodrigue de Schaetzen, Luke Rowe +3Group Relative Policy Optimization Constrained RL
Mila – Quebec AI Institute Université de Montréal
Augmented Lagrangian Method for Last-Iterate Convergence for Constrained MDPs
May 12, 2026Michael Lu, Max Qiushi Lin, Mo Chen +1Markov Decision Processes Last-Iterate Convergence
Simon Fraser University
Beyond Slater's Condition in Online CMDPs with Stochastic and Adversarial Constraints
Sep 24, 2025Francesco Emanuele Stradi, Eleonora Fidelia Chiefari, Matteo Castiglioni +2Markov Decision Processes Constrained RL
Politecnico di Milano

Feb 5, 2026J/K move · Enter open · S save

Constrained Group Relative Policy Optimization

Roger Girgis, Rodrigue de Schaetzen, Luke Rowe, Azalée Robitaille+2

Mila – Quebec AI Institute Université de Montréal

Group Relative Policy Optimization Constrained RL Critic-Free RL Safe RL RL Fine-Tuning Policy Learning

Open paper AskPDF arXiv

Group Relative Policy Optimization (GRPO) remains the dominant critic-free approach for fine-tuning LLMs and VLMs, but its compatibility with constrained policy optimization (e.g. for safety-critical domains) has not been carefully examined. In this work, we introduce Constrained GRPO, a Lagrangian-based extension of GRPO for constrained policy optimization. We show that the standard practice of scalarizing rewards before normalization introduces a critical Lagrangian-specific failure mode: GRPO's within-group normalization makes constrained optimization highly sensitive to how multi-component learning signals are aggregated. We show that scalarizing rewards before normalization introduces shared-denominator coupling, so that changing one multiplier alters not only the emphasis on its corresponding constraint, but also the relative weighting of the reward and other constraints. We address this with a simple but crucial modification: scalarizing standardized advantages rather than rewards. This yields a better-conditioned update by addressing the coupling induced by reward scalarization, resulting in better-behaved multiplier dynamics and more stable constraint enforcement in practice. Empirically, across a controlled gridworld, a real-world autonomous driving benchmark, and a mathematical reasoning task, Constrained GRPO consistently achieves better adherence to specified constraints while maintaining or improving task performance.

Abstract

Constrained RL

Explore similar work

Constrained Group Relative Policy Optimization

Augmented Lagrangian Method for Last-Iterate Convergence for Constrained MDPs

Beyond Slater's Condition in Online CMDPs with Stochastic and Adversarial Constraints