Parametric Resonance in Primal-Dual Learning for Slowly Non-Stationary Constrained MDPs

Maxime Baelde

stat.MLOct 16, 2026

Parametric Resonance in Primal-Dual Learning for Slowly Non-Stationary Constrained MDPs

Authors: Maxime Baelde

Organizations: Antvoice, Paris, France

Abstract

Primal-dual methods for constrained Markov decision processes are analyzed under a two-timescale assumption, with non-stationarity entering through two channels. In the additive channel, where the constraint level oscillates, the dual oscillation saturates at a floor set by the moving optimum and is amplified linearly by the timescale ratio. In the parametric channel, where the constraint sensitivity oscillates, a coherent modulation carries no resonant forcing at any depth: the dual update's stiffness modulation cancels the primal update's damping modulation. Instability instead comes from a mismatch between the constraint estimate used by the primal update and the measurement used by the dual update, producing a parametric resonance at twice the pair's natural frequency. The tolerated mismatch grows with the square root of the forcing bandwidth and collapses with the square root of the dual-to-primal step-size ratio. These results hold on the tangent linear-quadratic reduction around an active saddle, for one constraint and for $K$ constraints under a common modulation and an isotropy condition. Driving both updates from the same estimate removes the resonant forcing but introduces a stationary offset equal to the estimator's bias. In a Lagrangian soft actor-critic agent, the multiplier locks onto the sub-harmonic at the timescale ratios its identified linear loop predicts and a coherent cost critic suppresses the lock-in, while a proportional (PID) term, predicted to leave the lock-in in place, suppresses most of it through the projection of the multiplier.

Figures & tables

implementation	policy	multiplier	$r$
openai/safety-starter-agents , PPO-Lagrangian	$3\times 10^{-4}$	$5\times 10^{-2}$	$167$
PKU-Alignment/omnisafe , PPOLag	$3\times 10^{-4}$	$3.5\times 10^{-2}$	$117$
PKU-Alignment/Safe-Policy-Optimization , single agent	$3\times 10^{-4}$	$3.5\times 10^{-2}$	$117$

Table 1: Default step sizes in the reference Lagrangian implementations

Figure 1: Measured resonance tongue: Floquet exponent per step against the forcing frequency

$\varepsilon_{\theta}$	$\varepsilon_{\lambda}$	$h=\varepsilon_{\lambda}-\varepsilon_{\theta}$	predicted $\mu$	measured $\mu$
0	0	0	$-4.502\times 10^{-4}$	$-4.502\times 10^{-4}$
0	0.15	0.15	$-7.55\times 10^{-5}$	$-7.52\times 10^{-5}$
0	0.20	0.20	$+4.94\times 10^{-5}$	$+4.97\times 10^{-5}$
0.3	0	$-0.30$	$+2.992\times 10^{-4}$	$+2.988\times 10^{-4}$
0.2	0.5	0.30	$+2.992\times 10^{-4}$	$+2.961\times 10^{-4}$
0	0.90	0.90	$+1.798\times 10^{-3}$	$+1.755\times 10^{-3}$

Table 2: Parametric channel: predicted against measured Floquet exponent per step

$r=\alpha/\eta$	$\omega_{0}$	$\zeta$	measured threshold	$4\sigma/\omega_{0}$	$4\sigma/\tilde{\omega}_{0}$
$10^{-1}$	real spectrum	$>1$	none ( $\leq 1.98$ )	n/a	n/a
$0.25$	real spectrum	$>1$	none ( $\leq 1.98$ )	n/a	n/a
$0.96$	0.008644	0.5079	none ( $\leq 1.98$ )	2.3503	2.0315
$0.98$	0.008674	0.5026	1.9673	2.3176	2.0104
$1$	0.008704	0.4975	1.9484	2.2863	1.9899
$10^{2}$	0.009992	0.045020	0.1802	0.18022	0.18008

Table 3: Existence boundary of the phenomenon: mismatch threshold maximized over the forcing frequency

Figure 2: Lagrangian SAC: sub-harmonic amplitude of the multiplier against the forcing frequency

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

$\theta$	policy parameter, tangent coordinate	$\omega$	frequency of the non-stationarity
$\lambda$	multiplier of the active constraint	$A$	amplitude of the additive forcing
$\eta$	primal step size	$\tilde{\omega}_{0}$ , $\omega_{0}$	natural frequency, continuous and exact
$\alpha$	dual step size	$\sigma$	damping rate of the pair
$r=\alpha/\eta$	timescale ratio	$\zeta=\sigma/\tilde{\omega}_{0}$	reduced damping
$\hat{g}$	constraint estimate, consumed by the primal	$\varepsilon_{\theta}$	modulation depth seen by the primal
$g$	constraint measurement, consumed by the dual	$\varepsilon_{\lambda}$	modulation depth seen by the dual

Appendix

Table 4: Table of notation

Figure 3: Bandwidth law: mismatch threshold normalized by its own single-line threshold

$B/\sigma$	0	0.5	2	8	32
threshold (RMS)	0.1264	0.1382	0.1590	0.2310	0.4262
$/$ $\varepsilon_{0}$	0.99	1.08	1.25	1.81	3.34
$\kappa$ deduced	n/a	0.352	0.279	0.286	0.318

Appendix

Table 5: Mismatch threshold against bandwidth, and the constant of ( 21 )

$\eta$	$10^{-3}$	$\alpha$	$10^{-1}$
$r$	$100$	$\tilde{\omega}_{0}$	$0.01$
$\omega_{0}$ (discrete)	$0.00999216$	$\sigma$ (per step)	$4.502\times 10^{-4}$
$\zeta=\sigma/\tilde{\omega}_{0}$	$0.045020$	$2\omega_{0}$	$0.0199843$
threshold (peak)	$0.18022$	threshold (r.m.s.)	$0.12744$

Appendix

Table 6: Reference operating point of the parametric experiments

$\eta$	$\alpha$	$r$	$\zeta$	$\sup_{\omega}\Delta\lambda$	$/\,(2Aa/c^{2})\max\{1,r\}$	$/\,(A\alpha/\sigma)$
$10^{-3}$	$10^{-3}$	1	0.500	0.294	1.47	1.468
$10^{-3}$	$10^{-2}$	10	0.157	2.118	1.06	1.049
$10^{-3}$	$10^{-1}$	100	0.045	22.322	1.12	1.005
$10^{-2}$	$10^{-1}$	10	0.143	2.320	1.16	1.049
$10^{-1}$	$10^{-2}$	0.1	1.648	0.200	1.00	n/a

Appendix

Table 7: Additive channel: worst case over forcing frequency against the timescale ratio

$\eta$	$r=\alpha/\eta$	measured threshold	predicted $4\sigma/\omega_{0}$	ratio
$10^{-5}$	$10^{4}$	0.01801	0.01800	1.00
$10^{-4}$	$10^{3}$	0.05694	0.05693	1.00
$10^{-3}$	$10^{2}$	0.1801	0.1802	1.00
$10^{-2}$	$10$	0.5678	0.5764	0.99
$10^{-1}$	$1$	out of range	2.075	n/a

Appendix

Table 8: Parametric channel: collapse of the mismatch tolerance with the timescale ratio

$\eta\alpha$	bracket for $r^{\star}$	$r$	exponent past threshold (per step)
$10^{-2}$	$(0.85,0.90]$	–	–
$10^{-4}$	$(0.96,0.98]$	$0.98$	$+3.4\times 10^{-5}$
$10^{-4}$	$(0.96,0.98]$	$1$	$+8.4\times 10^{-5}$
$10^{-4}$	$(0.96,0.98]$	$10^{2}$	$+4.6\times 10^{-3}$

Appendix

Table 9: Existence boundary refined, and the exponent just past it

$B/\sigma$	$\varepsilon=0.3$	$\varepsilon=0.7$
0 (single line)	$-3.41\times 10^{-4}$	$-2.06\times 10^{-4}$
1	$-3.91\times 10^{-4}$	$-2.68\times 10^{-4}$
4	$-4.26\times 10^{-4}$	$-3.56\times 10^{-4}$
16	$-4.43\times 10^{-4}$	$-4.06\times 10^{-4}$

Appendix

Table 10: Coherent modulation under broadband forcing: top Lyapunov exponent per step

$\eta$	$r$	$\sigma$	$B/\sigma$	single line	band $B$	predicted band $B$
$10^{-3}$	100	$4.502\times 10^{-4}$	10	0.1277	0.2570	0.2576
$10^{-4}$	1000	$4.500\times 10^{-5}$	100	0.04222	0.2393	0.2273

Appendix

Table 11: Cancellation of the primal step size at fixed physical bandwidth

$\varepsilon_{\theta}$	$\varepsilon_{\lambda}$	Floquet exponent	$\exp(\mu T)$	$\max\lambda$ (unprojected)	$\max\lambda$ (projected)	final $\Delta\lambda$	% clipped
0	0	$-4.502\times 10^{-4}$	$5.26\times 10^{-13}$	1	1	0	0.0
0.3	0.3	$-3.748\times 10^{-4}$	$5.98\times 10^{-11}$	4.663	4.168	2.780	26.7
0.9	0.9	$-2.271\times 10^{-4}$	$6.38\times 10^{-7}$	11.21	–	–	–
0	0.3	$+2.988\times 10^{-4}$	$1.41\times 10^{8}$	$4.45\times 10^{6}$	3.933	2.685	28.0
0.3	0	$+2.988\times 10^{-4}$	$1.41\times 10^{8}$	$4.66\times 10^{5}$	–	–	–
0.2	0.5	$+2.961\times 10^{-4}$	$1.19\times 10^{8}$	$1.17\times 10^{6}$	5.669	3.649	36.6

Appendix

Table 12: Full recursion over 200 periods (62 800 steps), without and with the projection

	$\omega$	0.018	0.022	0.026	0.030	0.036
	$\omega/2\omega_{0}$	0.71	0.87	1.03	1.19	1.43
locked seeds	Lagrangian	5/10	7/10	10/10	6/10	1/10
	PID	0/10	1/10	0/10	1/10	0/10
	coherent	0/5	0/5	0/5	0/5	0/5
	PID, $\varepsilon=0.3$	–	1/10	2/10	–	–
	PID, $\varepsilon=0.35$	–	1/10	0/10	–	–

Appendix

Table 13: Lagrangian SAC at

r=100

: locking, forced response, projection, return and constraint value against the forcing frequency

Additive control (Table 15 )	$40$ forcing periods
Additive worst-case sweep (Table 7 )	$4000$ points log-spaced, $\omega\in[10^{-7},\pi]$
Floquet tongue-centre maxima	$121$ points on $\omega/2\omega_{0}\in[0.4,1.6]$ , $35$ on $[1.6,5.0]$
Figure 1 grid	$589$ integer periods, $196$ – $784$ steps
Tongue-centre mismatch bisection	$50$ halvings
Existence-boundary sweep (Table 3 )	$60$ periods log-spaced, $8$ – $3000$ steps, refined by $121$ points on $[0.55,1.45]\,\pi/\omega_{0}$
Existence-boundary bisection	$18$ halvings, mismatch split $\varepsilon_{\theta}=+d/2$ , $\varepsilon_{\lambda}=-d/2$ , $d\leq 1.98$

Appendix

Table 14: Numerical parameters of the protocol

$\omega$	coupled	frozen policy	ratio
$10^{-6}$	0.200000	2000.000	10 000
$10^{-4}$	0.199994	20.000	100
$10^{-3}$	0.199450	2.000	10
$10^{-2}$	0.159698	0.200	1.3
$10^{-1}$	0.020894	0.020008	1.0
$1$	0.002087	0.002086	1.0

Appendix

Table 15: Additive channel: peak-to-peak multiplier excursion tracking the moving optimum, coupled against frozen policy

Markov Decision Processes

Markov decision processes model sequential choices under uncertainty, linking actions, state transitions, and rewards to planning and reinforcement learning.

See all 157 papers

Residual-Controlled Multiplier Learning for Stochastic Constrained Decision-Making
Jun 5, 2026Kang Liu, Jianchen Hu, Ziyu Qu +3Stochastic Optimization Primal-Dual Optimization
School of Future Technology Xi’an Jiaotong University Xi’an, China, 710049 · School of Automation Science and Engineering Xi’an Jiaotong University Xi’an, China, 710049 · School of Mathematics and Physics China University of Geosciences Wuhan, China, 430074 +2
Rate-Optimal Algorithm for Adversarial Linear CMDPs
Oct 1, 2026Kihyun Yu, Honghao Wei, Dabeen LeeRegret Minimization in RL Markov Decision Processes
KAIST · Washington State University · Seoul National University
Hierarchical Multilevel Monte Carlo for Order-Optimal Neural Actor-Critic in Average-Reward CMDPs
Jul 30, 2026Ankur Naskar, Vaneet AggarwalAverage-Reward RL Constrained RL
Indian Institute of Science, India · Purdue University, USA

Jun 5, 2026J/K move · Enter open · S save

Residual-Controlled Multiplier Learning for Stochastic Constrained Decision-Making

Kang Liu, Jianchen Hu, Ziyu Qu, Edward Hengzhou Yan+2

School of Future Technology Xi’an Jiaotong University Xi’an, China, 710049 · School of Automation Science and Engineering Xi’an Jiaotong University Xi’an, China, 710049 · School of Mathematics and Physics China University of Geosciences Wuhan, China, 430074+2

Stochastic Optimization Primal-Dual Optimization Constrained Optimization

Open paper AskPDF arXiv

Stochastic constrained decision-making requires optimizing performance objectives while enforcing statistical requirements such as safety or fairness. However, standard primal--dual methods struggle to update multipliers robustly under stochastic mini-batch feedback, as the noise of mini-batch gradients and constraint estimates can be directly accumulated into the multiplier memory. To address this issue, we propose Residual-Controlled Multiplier Learning (RCML), which reformulates multiplier updating as projected-pressure feedback. The central idea is to decompose the projected multiplier into an effective pressure signal for primal descent and a pressure-memory residual for finite-gain multiplier tracking. To handle heterogeneous and noisy observations, we further augment this residual-integral backbone with modular stochastic stabilization components. For the convex-affine backbone, we establish finite-gain convergence, derive a stochastic residual bound under mini-batch feedback, and show that the residual feedback law admits a local KKT-residual interpretation near regular KKT points of nonconvex problems. Experiments across optimization, allocation, and fair-ranking tasks show that RCML improves feasibility control and multiplier stability while maintaining competitive objective performance. Code is released at https://anonymous.4open.science/r/RCML-3114/.

Abstract

Markov Decision Processes

Explore similar work

Residual-Controlled Multiplier Learning for Stochastic Constrained Decision-Making

Rate-Optimal Algorithm for Adversarial Linear CMDPs

Hierarchical Multilevel Monte Carlo for Order-Optimal Neural Actor-Critic in Average-Reward CMDPs