Position: Adopt Constraints Over Fixed Penalties in Deep Learning

Juan Ramirez; Meraj Hashemizadeh; Simon Lacoste-Julien

cs.LGMay 27, 2025

Position: Adopt Constraints Over Fixed Penalties in Deep Learning

Authors: Juan Ramirez, Meraj Hashemizadeh, Simon Lacoste-Julien

Organizations: Mila - Quebec AI Institute and DIRO, Université de Montréal · Canada CIFAR AI Chair

Abstract

Deep learning models are increasingly subject to explicit requirements, or constraints, such as safety levels, fairness limits, and sparsity budgets. In practice, such requirements are often handled through fixed weighted-sum penalization: the constraints are added to the task loss with fixed coefficients, and the resulting scalarized objective is minimized. This position paper argues that fixed penalization is often ill-suited to deep learning problems with non-negotiable requirements, for three main reasons. First, in non-convex settings, the penalized and constrained problems are not, in general, equivalent, so a minimizer of the penalized objective need not solve the constrained problem. Second, a penalty coefficient sets an objective-constraint trade-off rather than the constraint level, so finding an acceptable coefficient often takes trial and error, each trial a full training run. Third, fixed penalization weakens hard requirements into soft penalties to be traded off against task performance. As a result, compliance is no longer built into the problem, and requirements can be quietly relaxed. We therefore argue that, when a deep learning problem specifies non-negotiable requirements, one should solve the constrained problem itself instead of a surrogate defined by fixed penalization.

Figures & tables

Figure 1: Solving a sparsity-constrained classification problem. Starting from coefficients yielding low (25%) and high (85%) model density, five bisection search steps are needed to reach the desired 50% density . Numbers mark iterations (endpoints are “0”). Replicates Fig. 5(b) of Gallego-Posada et al. [21] (Appendix B.2 ).

$\epsilon$	$x^{*}=\arcsin(\epsilon)$	$x_{t}$ at convergence
$0.1$	$1.00\times 10^{-1}$	$1.00\times 10^{-1}$
$0.2$	$2.01\times 10^{-1}$	$2.01\times 10^{-1}$
$0.3$	$3.05\times 10^{-1}$	$3.05\times 10^{-1}$
$0.4$	$4.12\times 10^{-1}$	$4.12\times 10^{-1}$
$0.5$	$5.24\times 10^{-1}$	$5.24\times 10^{-1}$
$0.6$	$6.44\times 10^{-1}$	$6.44\times 10^{-1}$

Table 1: Solutions to Example 1 recovered by the Lagrangian approach with PI multiplier updates. At every constraint level tested, the method recovers the constrained minimizer

x^{*}=\arcsin(\epsilon)

Figure 2: Solving a binary classification task under a constraint that the average predicted probability of class “0” be at least 70%. No penalty coefficient in the sweep lands on the 70% target. Each either leaves the constraint violated or over-satisfies it at a cost in accuracy, whereas the Lagrangian approach lands on the target across a broad range of dual step-sizes. Cells are blue within one percentage point of the target and shade to red at eleven or more points away. Data and model from Figure 2 of Cotter et al. [8] (details in Appendix B.3 ).

Experiment	Penalized (median, std.)	Constrained (median, std.)	Overhead
Probability constraint	5.40 s (0.27 s)	5.68 s (0.31 s)	+5.2%
Sparsity constraint	8.17 min (0.19 min)	8.30 min (0.07 min)	+1.6%

Table 2: Wall-clock comparison over five runs on an NVIDIA L40S GPU in two illustrative settings. Constrained training adds only a small per-run overhead.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: The trade-off curve between the objective

f

and constraint

g

in Example 1 . As

x

varies over

[0,\pi/2]

, the map

x\mapsto(f(x),g(x))=(\cos x,\sin x)

traces the non-dominated frontier from

(1,0)

(0,1)

. The constrained optimum is the interior point

(\sqrt{1-\epsilon^{2}},\,\epsilon)

, whereas fixed weighted-sum penalization recovers only the two endpoints.

Iteration #	Penalty coef. $c$	Model density (%)	Train acc. (%)
0	$1.00\times 10^{-3}$	$84.5$	$99.97$
0	$1.00\times 10^{0}$	$25.3$	$99.97$
1	$3.16\times 10^{-2}$	$68.1$	$99.96$
2	$1.78\times 10^{-1}$	$54.2$	$100.00$
3	$4.22\times 10^{-1}$	$39.0$	$100.00$
4	$2.74\times 10^{-1}$	$46.9$	$99.99$

Appendix

Table 3: Sparsity-constrained neural network training using the penalized approach, targeting at most 50% model density (at least 50% sparsity), within one percentage point. The penalty coefficient is selected via a log-scale bisection search. This table complements Figure 1 .

	Penalized ( $c$ )		Lagrangian (dual step-size)
Value	$P(\hat{y}=0)$ (%)	Train acc. (%)	$P(\hat{y}=0)$ (%)	Train acc. (%)	$\lambda_{T}$
$0$	$49.96$	$99.75$	$49.96$	$99.75$	$0.00$
$1.00\times 10^{-4}$	$49.96$	$99.75$	$50.30$	$99.75$	$0.20$
$2.15\times 10^{-4}$	$49.96$	$99.75$	$50.72$	$99.75$	$0.42$
$4.60\times 10^{-4}$	$49.96$	$99.75$	$51.89$	$100.00$	$0.87$
$1.00\times 10^{-3}$	$49.96$	$99.75$	$59.56$	$96.25$	$1.63$

Appendix

Table 4: Probability-constrained linear classification (Appendix B.3 ), targeting an average predicted probability of class 0,

P(\hat{y}=0)

, of at least

70\%

. Both sweeps use the same grid of values, as penalty coefficients

c

for the penalized approach and as dual step-sizes for the Lagrangian approach. The multiplier at the end of training is

\lambda_{T}

. The Lagrangian approach lands on the target for 15 of the 20 dual step-sizes, whereas only

c=2.15

avoids a collapsed penalized solution, overshooting to

75\%

Deep Learning Optimization

Deep learning optimization develops algorithms and training strategies that help neural networks converge, generalize, and use compute efficiently.

See all 208 papers

Stochastic Penalty-Barrier Method for Constrained Machine Learning
May 18, 2026Adam Bosák, Andrii Kliachkin, Gilles Bareilles +4Constrained Optimization
Artificial Intelligence Center, CTU in Prague · CMAP, École Polytechnique, Palaiseau, France
Enhancing Reinforcement Learning Fine-Tuning with an Online Refiner
Mar 18, 2026Hao Ma, Zhiqiang Pu, Yang Liu +1RL Fine-Tuning RL for Language Models
School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences
HardNet++: Nonlinear Constraint Enforcement in Neural Networks
Apr 21, 2026Andrea Goertzen, Kaveh Alim, Youngjae Min +1Nonlinear MPC Neural Network Optimization
Massachusetts Institute of Technology

May 18, 2026J/K move · Enter open · S save

Stochastic Penalty-Barrier Method for Constrained Machine Learning

Adam Bosák, Andrii Kliachkin, Gilles Bareilles, Allen Gehret+3

Artificial Intelligence Center, CTU in Prague · CMAP, École Polytechnique, Palaiseau, France

Constrained Optimization

Open paper AskPDF arXiv

Constrained Machine Learning (CML) enables fairness-aware training, physics-informed neural networks, and integration of symbolic domain knowledge into statistical models. In this work, we introduce the Stochastic Penalty-Barrier Method (SPBM) for CML problems. SPBM extends classical penalty and barrier methods by incorporating an exponential averaging of the dual variables, a stabilized penalty schedule, and the Moreau envelope to handle non-smoothness. We analyze the bias that mini-batching introduces in the barrier function and show that the feasible set of the resulting transformed problem is contained within the original one. We compare SPBM with CML baselines across multiple fairness and physics informed neural networks experiments. We find that SPBM is competitive with state-of-the-art methods. We also observe, on our fairness-based computational benchmark, that the per-epoch runtime of CML methods is largely independent of the number of constraints, and within

1.3\times

of the per-epoch runtime of regularized Adam, for a number of constraints ranging from

90

9900

Abstract

Deep Learning Optimization

Explore similar work

Stochastic Penalty-Barrier Method for Constrained Machine Learning

Enhancing Reinforcement Learning Fine-Tuning with an Online Refiner

HardNet++: Nonlinear Constraint Enforcement in Neural Networks