Residual-Controlled Multiplier Learning for Stochastic Constrained Decision-Making

Kang Liu; Jianchen Hu; Ziyu Qu; Edward Hengzhou Yan; Lun Yang; Meng Zhang

cs.LGJun 5, 2026

Residual-Controlled Multiplier Learning for Stochastic Constrained Decision-Making

Authors: Kang Liu, Jianchen Hu, Ziyu Qu, Edward Hengzhou Yan, Lun Yang, Meng Zhang

Abstract

Stochastic constrained decision-making requires optimizing performance objectives while enforcing statistical requirements such as safety or fairness. However, standard primal--dual methods struggle to update multipliers robustly under stochastic mini-batch feedback, as the noise of mini-batch gradients and constraint estimates can be directly accumulated into the multiplier memory. To address this issue, we propose Residual-Controlled Multiplier Learning (RCML), which reformulates multiplier updating as projected-pressure feedback. The central idea is to decompose the projected multiplier into an effective pressure signal for primal descent and a pressure-memory residual for finite-gain multiplier tracking. To handle heterogeneous and noisy observations, we further augment this residual-integral backbone with modular stochastic stabilization components. For the convex-affine backbone, we establish finite-gain convergence, derive a stochastic residual bound under mini-batch feedback, and show that the residual feedback law admits a local KKT-residual interpretation near regular KKT points of nonconvex problems. Experiments across optimization, allocation, and fair-ranking tasks show that RCML improves feasibility control and multiplier stability while maintaining competitive objective performance. Code is released at https://anonymous.4open.science/r/RCML-3114/.

Explore similar work

Aug 2, 2026math.OC

Learning-Based Stochastic Optimal Control with Infinite-Horizon Probabilistic Constraints

In this paper, we consider stochastic optimal control problems with infinite-horizon joint chance constraints. By means of an appropriate state augmentation, we reformulate the original problem as a constrained Markov decision process, in which both the cost and the constraint function exhibit an additive structure. We then prove that this formulation enjoys strong duality, thereby enabling us to reformulate the problem as an equivalent unconstrained one in the Lagrange dual framework. We propose a dual-ascent algorithm to solve the resulting problem and show that it converges to a deterministic Markov policy defined over the augmented state space that is both optimal and feasible. To accommodate continuous state-input spaces, we propose a dedicated learning algorithm to approximate the value function in an offline training setting, thereby significantly reducing the computational complexity of the online control phase. We then test our approach on a numerical example and demonstrate its effectiveness compared to online predictive control methods in terms of performance and computational complexity.

Francesco Cordiano, Kanghui He, Bart De Schutter

Stochastic Optimal Control Markov Decision Processes Model Predictive Control Primal-Dual Methods

Ask PDF

Jun 23, 2026cs.LG

Bias-Controlled Primal-Dual Natural Actor-Critic: Optimal Rates for Constrained Multi-Objective Average-Reward RL

Many reinforcement learning (RL) problems in the infinite-horizon average-reward setting require optimizing multiple conflicting objectives while satisfying multiple safety constraints. A common approach is concave scalarization, where the agent maximizes a utility

f(J^π_{r_1}, \ldots, J^π_{r_M})

subject to a scalarized constraint

g(J^π_{c_1}, \ldots, J^π_{c_N}) \ge 0

, where

J^π_{r_m}

and

J^π_{c_n}

denote the average-reward and cost under policy

π

. However, the nonlinearity of

f

and

g

introduces bias in policy-gradient and actor-critic methods, since gradients must be evaluated using noisy estimates of

J^π,

and

\mathbb{E}[\partial f(J^π)] \neq \partial f(\mathbb{E}[J^π]),

and this bias propagates through both primal and dual updates. We propose an MLMC-based primal-dual Natural Actor-Critic algorithm for average-reward MDPs that controls bias in scalarized objectives, constraint evaluation, and actor-critic estimation without requiring mixing-time knowledge. We show that the algorithm achieves optimal global convergence and constraint-violation rates of

\tilde{O}(1/\sqrt{T})

. To our knowledge, this is the first result establishing optimal convergence for concave scalarized multi-objective RL in the average-reward setting, both with and without constraints, and the first to do so without mixing-time information even in the absence of scalarization.

Ankur Naskar, Swetha Ganesh, Vaneet Aggarwal

Multi-Objective Reinforcement Learning Soft Actor-Critic Convex Optimization

Ask PDF

May 18, 2026cs.LG

Stochastic Penalty-Barrier Methods for Constrained Machine Learning

Constrained machine learning enables fairness-aware training, physics-informed neural networks, and integration of symbolic domain knowledge into statistical models. Despite its practical importance, no general method exists for the non-convex, non-smooth, stochastic setting that arises naturally in deep learning. We propose the Stochastic Penalty-Barrier Method (SPBM), which extends classical penalty and barrier methods to this setting via exponential dual averaging, a stabilized penalty schedule, and the Moreau envelope to handle non-smoothness. Experiments across multiple settings show that SPBM matches or outperforms existing constrained optimization baselines while incurring only linear runtime overhead compared to unconstrained Adam for up to 10,000 constraints.

Adam Bosák, Andrii Kliachkin, Jana Lepšová +2

Quadratic Slack Penalty Fairness-Aware Learning Adam Zeroth-Order Optimization Stochastic Machine Learning+1

Ask PDF