Precise constraint satisfaction is a prerequisite to deploying learned models in many areas, motivating methods that repair raw neural predictions with a repair procedure. Current methods unroll multiple repair steps in training and softly penalize constraint violations that remain after the unroll. This is compute- and memory-intensive, lacks robustness when the repair fails to converge, and surrenders most of the constraint satisfaction work to the repair. Our central finding is that, contrary to common practice, a single detached projection step suffices in training. We accomplish this with a Projection-Adaptive Loss (PAL), which uses the constraint residual after this single step to adaptively weigh constraint penalties on the raw prediction. In experiments, PAL is the only method that retains virtually perfect feasibility on extremely nonlinear constraints, and matches or outperforms current methods on synthetic and engineering benchmarks. Because it only requires a single detached projection step, PAL trains 2.5x faster than the canonical repair-based method (DC3) on its own ACOPF benchmark. PAL can also be trained when constraints are expensive to evaluate (e.g., via neural surrogates), a setting where current unrolled methods are memory-intractable.
Figures & tables
Figure 1: PAL vs. SOTA repair-aware training. From raw model predictions y^ , SOTA iterates projector P and evaluator Φ for N times per training iteration and then uses a soft penalty on post-repair constraint violation c~(N) . We instead use a single and detached projection step to y~(1) . PAL then uses the post-projection residual c~(1) to adaptively drive wˉc , which acts as an adaptive penalty on the raw prediction’s constraint residual c^ .
Figure 2: Two-constraint toy problem where h1 is stiff but linear and therefore trivial to project, and h2 is non-stiff but nonlinear and difficult to project (more details in Appendix H ). Top: constraint violations before projection h(y^) . Bottom: constraint violations after the first projection step h(y~(1)) , equivalent to R1,k . Left: ALM (adaptive penalty method) trains without projection and penalizes constraints only according to pre-projection magnitude h(y^) , i.e., cannot distinguish constraints by repair difficulty. Center: Under soft-loss training (SnareNet, ENFORCE, FSNet, …), the projector drives h(y~(N))≈0 in training so that constraint pressure on the neural backbone is low, resulting in high violations before and after the first projection step. Right: PAL exerts the right amount of training pressure onto h(y^) (and thus the neural backbone) to drive post-projection residuals hk(y~(1)) below threshold τc (here 10−7 ).
Bench.
ALM+ Bolt-On
ENFORCE
DC3
FSNet
SnareNet
PAL [Ours]
Feas. (%)
S1
100±0
100±0
—
100±0
100±0
100±0
S2
100±0
100±0
97±7
100±0
100±0
100±0
S3
100±0
100±0
100±0
100±0
100±0
100±0
S4
100±1
100±0
—
100±0
100±0
100±0
S5
100±0
—
—
51±31
100±0
100±0
S6
100±0
100±0
—
100±0
100±0
100±0
Table 1: Synthetic stress-test experiments. Means and standard deviations across 10 seeds. “—”: failure on 2 or more seeds. Forward and backward evaluator passes Nϕf and Nϕb per training step are only shown for extreme cases S1 ( K=1 ) and S6 ( K=90 ), with full results in Table 12 .
Table 2: Case studies in science and engineering. Mem: peak GPU memory (GB) at batchsize=1 . Time: training wall-clock. 10 seeds each, IPOPT 5 restarts × 64 queries. ∗ : Maximum constraint violation. Bold marks the best value among feasible methods. ‡ : 49 GB on 3 seeds and >96 GB (OOM) on 7 seeds in the first two training epochs. \lx@sectionsign : batchsize=8 . † : 0% feasibility. : Non-convergence. : Structurally inapplicable. Baseline failure modes are discussed in Appendix L .
Figure 3: Feasibility, objective and constraint training pressure ∥∇θLc∥/∥∇θL∥ over curvature κ . PAL retains best-in-class feasibility at high curvature despite a single repair step, because its adaptive loss raises training pressure in response to first-step residuals.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
y,y^,y~(i)
design variable, raw neural backbone prediction, i -th projection iterate
Table 4: Ablation of PAL hyperparameters on synthetic benchmarks ( n=10 seeds). Blank cells inherit the Base value. Bold = best per column. The f column lists the argument used inside the objective term of the training loss ( y^ vs y~(1) ). Dec. marks whether the multiplier rule’s decay half is active: ✓ = bidirectional (default), ✗ = monotone-only growth.
PAL ( τc=10−4 )
PAL ( τc=10−2 )
κ
Feas. (%)
Obj.
∥∇L∥∥∇Lc∥ (%)
Feas. (%)
Obj.
∥∇L∥∥∇Lc∥ (%)
κ=0
100 ± 0
0.01 ± 0.00
0.0
100 ± 0
0.01 ± 0.00
0.0
κ=1
100 ± 0
0.01 ± 0.00
23
100 ± 0
0.01 ± 0.00
0.0
κ=10
100 ± 0
0.05 ± 0.01
63
100 ± 0
0.03 ± 0.00
25
κ=102
100 ± 0
0.66 ± 0.13
72
99.9 ± 0.1
0.48 ± 0.05
62
κ=103
100 ± 0
0.96 ± 0.40
89
99.8 ± 0.2
0.42 ± 0.20
79
Appendix
Table 5: Full results for high-curvature experiment. κ from 0 to 106 , 10 seeds, all methods trained and evaluated in float64 . ∥∇θLc∥/∥∇θL∥ is the share of constraint training pressure. Bold marks the best feasibility at each κ across all methods. † : gradient ratio carried over from the source ALM training run. Plain ALM without Bolt-On is omitted because it is 0% feasible at every κ . DC3 uses a Newton completion with Yamashita-Fukushima damping, since the undamped completion diverges for every κ>0 .
PAL- LM
PAL- SQP
PAL- IP
S1
0.02±0.02
0.01±0.01
0.01±0.01
S2
0.00±0.00
0.00±0.00
0.02±0.00
S3
0.00±0.00
0.00±0.00
0.00±0.00
S4
0.01±0.00
0.01±0.00
0.05±0.02
S5
0.00±0.00
0.00±0.00
0.00±0.00
S6
0.08±0.01
0.07±0.01
0.21±0.02
Appendix
Table 6: Objective at convergence ( f⋆=0 ) for the three repair step mechanisms on the synthetic benchmarks S1 – S6 . Shown is mean ± standard deviation across 10 seeds. All methods use default hyperparameters and consistently reach 100% constraint satisfaction on S1 – S6 , except for IP on S3 ( 86±5% ).
Figure 4: Geometric intuition. The projector eliminates gradient flow for f(y~(1)) along direction v (orthogonal to local constraint manifold M ) because, for an idealized projector, ∂y~(1)/∂y^⋅δv=0 . This leaves only gradients of f(y~(1)) which are tangent to the constraint manifold (direction u ) to be backpropagated through the projection step into the neural backbone Nθ .
Figure 5: PAL loss-placement comparison on the toy problem of Appendix H . Rows from top: pre-projection residuals h(y^) , post-projection residuals h(y~(1)) , objective values f , and objective gradient magnitudes (computed on a fixed evaluation batch). Columns: f(y^) throughout (PAL default), f(y~(1)) throughout, and a midway switch from f(y^) to f(y~(1)) at t=2500 (dotted vertical line). When using f(y~(1)) throughout, the objective never escapes the initialisation neighbourhood due to vanishing gradients normal to the manifold. The right column, having trained under f(y^) first, retains its objective and feasibility quality after the switch, demonstrating that the pathology is specific to early training.
N=1
N=2
N=5
N=10
N=100
float64
S1 (sphere track)
8.48⋅10−4
2.46⋅10−7
7.55⋅10−16
7.55⋅10−16
7.55⋅10−16
S2 (active-set switch)
1.36⋅10−4
3.41⋅10−6
2.22⋅10−16
2.22⋅10−16
2.22⋅10−16
S3 (ill-cond. tube)
8.42⋅10−6
1.03⋅10−7
3.54⋅10−15
2.22⋅10−17
2.01⋅10−17
S4 (qv coupling)
1.01⋅10−2
1.26⋅10−5
8.44⋅10−16
9.10⋅10−16
8.22⋅10−16
S5 (overdetermined)
6.77⋅10−3
4.98⋅10−4
6.33⋅10−8
8.81⋅10−9
8.83⋅10−11
Appendix
Table 7: Maximum constraint violation over the test set against the number of inference repair steps N , over 10 seeds, for the trained PAL models from the paper on the synthetic benchmarks S1 – S6 .
Figure 6: Candidate chip placement for E4 (left) and full results (right). PAL and ALM+Bolt-On run at 10 seeds, FSNet and SnareNet at 5 seeds. All methods use canonical default hyperparameters without per-method tuning .
Figure 7: E1 unmanned aircraft benchmark, candidate design. Left: outer mould-line geometry from the analytical Bézier representation, with the decoded internal wingbox (skin, spars, ribs) overlaid in blue/green. Top right: surface pressure coefficient cp predicted by the FiLM-modulated BlendedNet surrogate Φcpθ ( Sung et al., 2025 ) at iso-sliced surface points. Bottom right: structural tip deflection ∣w(y)∣ from beam integration of the pressure loads through the predicted section properties, warp-amplified 5× for visualisation. The tip-deflection constraint reads wtip at y=b/2 .
Figure 8: E1 evaluator Φ , from the design vector y to the objective f and constraints c . Dashed: geometry processing. Plain: analytical. Grey: neural surrogate. Hatched: generative model. Double border: constraint or objective.
Figure 9: E2 urban planning benchmark, three views of a candidate design. Left: top-down site plan with the 750×750 m buildable plot (green dashed) and the 10 building footprints. Clearance is evaluated pairwise and reported as the number of pairs with overlapping dmin -inflated AABBs. Middle: tail-mean wind speed predicted by the WinDiNet surrogate over the full 1100×1100 m raster domain. Right: hard pedestrian-comfort zones derived from the tail-mean speed (calm <1 m/s, acceptable 1 – 5 , uncomfortable 5 – 15 , danger >15 ). The danger constraint forbids any cell entering the red band.
Figure 10: E2 candidate design rendered in 3D with PyVista ( Sullivan & Kaszynski, 2019 ) , with the pedestrian-comfort field draped at ground level. Building heights are decoded from the design vector. The colour map shows tail-mean wind speed over the comfort range 0 – 15 m/s, with green/yellow indicating calm and acceptable zones and red indicating the danger band that the surrogate constraint forbids.
Figure 11: E2 evaluator Φ , from the design vector y to the objective f and constraints c . Dashed: geometry processing. Plain: analytical. Grey: neural surrogate. Hatched: generative model. Double border: constraint or objective.
Table 8: Selected hyperparameter values per method from the Bayesian optimization sweep. † : For ALM+Bolt-On, the optimized hyperparameters of frozen ALM models were reused, i.e., only hyperparameters of the projection were tuned for ALM+Bolt-On itself.
Method
Train (s)
Total (s)
Predict (s)
/ query (ms)
S1
PAL
106
247
0.0119
0.185
ALM
58
131
0.00734
0.115
ALM+Bolt-On
58 †
10
0.0507
0.792
ENFORCE v4
57
63
0.0152
0.237
FSNet
1720
1909
0.176
2.75
SnareNet
149
310
0.0506
0.791
Appendix
Table 9: Wall-clock times on S1 – S6 . The total time reported includes initialization, training, evaluation, and artifact saving. The overall prediction time is for 64 samples (queries). We report the median over 10 seeds (counting non-diverged seeds only). “—” indicates fully diverged or structurally inapplicable cells. † For ALM+Bolt-On, training reuses ALM checkpoints and the reported total time is inference-time projection only.
Benchmark
Method
Median wall-clock
E1 BWB
PAL
∼ 8 min
ALM
∼ 70 s
E2 Urban Planning
PAL
∼ 106 min
ALM+Bolt-On
∼ 65 min
ALM
∼ 32 s
Appendix
Table 10: End-to-end wall-clock time on engineering benchmarks. On E2 , PAL suffers from sequential Jacobian assembly (loop over K ).
Default
E1 (Unmanned Aircraft)
E2 (Urban Planning)
Training epochs
—
200
500
Batch size
—
32
16
Learning rate ( Nθ )
—
10−4
10−3
PAL
γ
10−2
10−1
10−1
Dmax
1.0
5.0
5.0
Appendix
Table 11: Hyperparameters for PAL and ALM on E1 and E2 . Bold marks deviations from defaults. ALM defaults follow Basir & Senocak (2023) . For both benchmarks, we performed minor hand-tuning of γALM and α but found defaults to work relatively well.
Bench.
ALM+ Bolt-On
ENFORCE
DC3
FSNet
SnareNet
PAL [Ours]
Feas. (%)
S1
100±0
100±0
—
100±0
100±0
100±0
S2
100±0
100±0
97±7
100±0
100±0
100±0
S3
100±0
100±0
100±0
100±0
100±0
100±0
S4
100±1
100±0
—
100±0
100±0
100±0
S5
100±0
—
—
51±31
100±0
100±0
S6
100±0
100±0
—
100±0
100±0
100±0
Appendix
Table 12: Full version of Table 1 : synthetic stress-test experiments for five baselines and PAL, including evaluator passes per training step on all benchmarks. Means and standard deviations across 10 seeds. “—” indicates failure on 2 or more seeds.
Enforcing complex (e.g., nonconvex) operational constraints is a critical challenge in real-world learning and control systems. However, existing methods struggle to efficiently enforce general classes of constraints. To address this, we propose a novel data-driven amortized approach that uses a trained autoencoder as an approximate projector to provide fast corrections to infeasible predictions. Specifically, we train an autoencoder using an adversarial objective to learn a structured, convex latent representation of the feasible set. This enables rapid correction of neural network outputs by projecting their associated latent representations onto a simple convex shape before decoding into the original feasible set. We test our approach on a diverse suite of constrained optimization and reinforcement learning problems with challenging nonconvex constraints. Results show that our method effectively enforces constraints at a low computational cost, offering a practical alternative to expensive feasibility correction techniques based on traditional solvers.
Maria Chzhen, Priya L. Donti
University of Toronto Toronto, ON, Canada · Massachusetts Institute of Technology Cambridge, MA, USA
Enforcing constraint satisfaction in neural network outputs is critical for safety, reliability, and physical fidelity in many control and decision-making applications. While soft-constrained methods penalize constraint violations during training, they do not guarantee constraint adherence during inference. Other approaches guarantee constraint satisfaction via a projection layer, but often rely on the existence of a tractable projection onto the feasible set, limiting their utility in more general problem settings. Many real-world problems of interest are nonlinear and lack the special structure admitting a tractable projection, motivating the development of methods that can enforce general nonlinear constraints. To this end, we introduce HardNet++, a constraint-satisfaction method that enforces linear and nonlinear equality and inequality constraints. Our approach iteratively adjusts the network output via damped local linearizations of the constraints. Each iteration is differentiable, admitting an end-to-end training framework, where the constraint satisfaction layer is active during training. We show that under certain regularity conditions, this procedure enforces nonlinear constraint satisfaction to arbitrary tolerance. Finally, we demonstrate tight constraint adherence without loss of optimality in a learning-for-optimization context, where we apply this method to a nonlinear model predictive control problem.
We present a novel framework for embedding hard constraint satisfaction into neural network (NN) architectures, specifically feedforward neural networks and transformers, with input-dependent affine constraints of arbitrary cardinality. Traditional constraint enforcement approaches either rely on penalty-based soft constraints, which offer no guarantee of satisfaction, or on post-processing methods that enforce constraints after the NN is trained, which may lead to suboptimality. We introduce a trainable constraint-affine (CAffine) layer into NNs, yielding CAffNet, which goes beyond enforcing affine constraints via fixed orthogonal or parallel projections and enables joint optimization with network parameters. Moreover, we impose no restrictions on the constraint space dimensions and establish that our construction preserves the universal approximation properties of NNs, while providing provable guarantees on constraint adherence for all inputs. Experimental validation demonstrates robust performance across diverse domains requiring guaranteed constraint satisfaction. The code is available at https://github.com/ice-t-lab/CAffNet.git.
Yang Zhao, Jungeun Lee, Jeong hwan Jeon +1
Department of Mechanical and Industrial Engineering, Northeastern University, Boston, MA 02115 USA · Department of Electrical Engineering, Ulsan National Institute of Science and Technology, Ulsan 44919, Republic of Korea