Organizations: Faculty of Mathematics, University of Vienna, Oskar–Morgenstern–Platz 1, A–1090 Vienna, Austria · Faculty of Engineering, Free University of Bozen–Bolzano, NOI Techpark, via Bruno Buozzi 1, 39100 Bolzano, Italy
A learning-assisted framework for Nonlinear Conjugate Gradient (NCG) methods is developed in which lightweight online prediction is used to support, rather than replace, the classical optimization process. The proposed approach retains the standard two-term NCG structure and introduces two independent online extreme learning machine models. The first model predicts an NCG coefficient during restart recovery or when persistent stagnation is detected, whereas the second model is used only after a line-search failure in finite precision arithmetic to propose a bounded correction to the deterministic fallback step size. Neural predictions are accepted only when they satisfy deterministic numerical safeguards, including descent, scale, and out-of-distribution requirements; otherwise, the corresponding deterministic recovery or fallback mechanism is retained. Both learning models are updated online using quantities already generated along the optimization trajectory and therefore require no additional function or gradient evaluations for training. These learning frameworks are applied to six well-known classical NCG methods, producing corresponding low-storage learning-assisted variants with deterministic safeguards. Numerical experiments on 480 CUTEst test problems show improved robustness for all six variants at broadly comparable evaluation costs. A PDE-constrained conductivity-identification example further illustrates the potential of the proposed approach to reduce evaluation costs and computational time for problems with expensive function and gradient evaluations. These findings suggest that restricted online learning can improve the robustness and computational performance of classical NCG methods while preserving their low-storage structure and deterministic optimization safeguards, and is particularly promising when function or gradient evaluations are expensive.
Figures & tables
Figure 1: Bird’s-eye view of the learning-assisted nonlinear CG framework. The classical CG method remains the main optimization procedure. The coefficient- ELM assists the construction of the search direction only during restart recovery or persistent stagnation, whereas the independent fallback-step- ELM is considered only after failure of cvsrch . Both learning branches return to the main optimization path after OOD screening and the corresponding deterministic safeguards.
Pair
CG solved
NN solved
NN only
CG only
ΔNsolved
HSNN - HS
384
414
31
1
+30
PRNN - PR
397
407
17
7
+10
HZNN - HZ
400
413
17
4
+13
DYNN - DY
357
371
19
5
+14
FRNN - FR
355
364
18
9
+9
DLNN - DL
394
412
23
5
+18
Table 1: Successful problems for each classical/learning-assisted pair. The columns “ NN only” and “ CG only” count problems solved exclusively by the indicated member of the pair. All counts refer to 480 problems.
Pair
ΔNsolved
Δρnf2gpp(1)
Δρngpp(1)
Δρnfpp(1)
Δρmsecpp(1)
HSNN - HS
+30
+1.04
+1.25
+1.04
−18.33
PRNN - PR
+10
0.00
0.00
0.00
−23.75
HZNN - HZ
+13
−0.21
−0.21
−0.42
−20.62
DYNN - DY
+14
+0.21
+0.62
0.00
−12.50
FRNN - FR
+9
−0.21
−0.21
−0.21
−15.21
DLNN - DL
+18
+1.67
+1.88
+1.67
−13.54
Table 2: Pairwise change of each learning-assisted method relative to its classical counterpart. The values Δρmpp(1) are percentage-point (pp) differences computed with the common 12-solver reference set and all 480 problems. A positive entry means more best-cost results, including ties, for the neural method.
Figure 2: Pairwise Dolan–Moré performance profiles for nf2g on 480 problems, using the common 12-solver baseline.
Weighted evaluations nf2g
Elapsed time msec
Pair
NN less
NN more
Tie
NN less
NN more
Tie
HSNN - HS
14
13
356
35
307
41
PRNN - PR
17
13
360
32
315
43
HZNN - HZ
4
15
377
22
334
40
DYNN - DY
13
32
307
30
287
35
FRNN - FR
5
27
314
34
281
31
Table 3: Head-to-head results restricted to problems solved by both members of each pair. Each triple gives the number of problems for which the NN variant costs less, costs more, or ties its classical counterpart, respectively.
Figure 3: Comparison of HS and HSNN on a selected synthetic inverse-conductivity instance. Left: Euclidean gradient norm against the weighted cost nf2g ; the dotted line is the common 10−6 tolerance. Right: median elapsed time over five runs, shown in seconds ( 0.1191 and 0.1114 s, or 119.1 and 111.4 msec , respectively). The time measurements include neural-model overhead.
Method
Iterations
nf
ng
nf2g
msec
HS
46
125
125
375
119.1
HSNN
36
103
103
309
111.4
Table 4: Results for the selected inverse-conductivity instance.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Role
tDL
0.1
fixed DL parameter for DL / DLNN
DY / FR restart threshold
ϵmach
criterion in ( 22 )
δa
10−8
original fallback-step seed factor
Stopping tolerance
10−6
∥g∥∞ tolerance
Evaluation budget
500000
maximum nf2g
Time limit
∞
no time-based termination
Appendix
Table 5: Classical CG , stopping, and Wolfe line-search parameters prescribed for the experiments.
Parameter
Value
Role
Coefficient- ELM and recovery
Input dimension pβ
12
coefficient feature dimension
Hidden units hβ
24
coefficient- ELM hidden layer
Archive size Nβmax
250
maximum replay samples
Minimum training samples
8
first eligible fit
Retraining interval
4
new samples between fits
Appendix
Table 6: Parameters of the learning-assisted CGNN variants.
Natural Gradient Descent (NGD) has emerged as a promising optimization algorithm for training neural network-based solvers for partial differential equations (PDEs), such as Physics-Informed Neural Networks (PINNs). However, its practical use is often limited by the high computational cost of solving linear systems involving the Gramian matrix. While matrix-free NGD methods based on the conjugate gradient (CG) method avoid explicit matrix inversion, the ill-conditioning of the Gramian significantly slows the convergence of CG. In this work, we extend matrix-free NGD to broader classes of problems than previously considered and propose the use of Randomized Numerical Linear Algebra (RandNLA) techniques for efficient preconditioning of the inner CG solver. The resulting algorithms demonstrate substantial performance improvements over existing NGD-based methods on a range of PDE problems discretized using neural networks, and offer competitive results compared to other state-of-the-art optimizers.
Ivan Bioli, Carlo Marcati, Giancarlo Sangalli
Department of Mathematics, University of Pavia, 27100 Pavia, Italy · Department of Civil Engineering and Architecture, University of Pavia, 27100 Pavia, Italy · Institut Camille Jordan, Lyon 1 Université, 69622 Villeurbanne, France +1
Interior-point methods (IPMs) are among the most widely used algorithms for constrained optimization, yet their Newton-based search directions require costly second-order information and large linear-system solves. Learning to optimize offers cheaper updates learned from data, but the singular behavior of logarithmic barriers near constraint boundaries makes IPMs highly sensitive to perturbations, complicating both warm starting and learning reliable updates. We introduce pdLIP, an IPM for smooth nonlinear programs that integrates learned preconditioning with pdProj, an all-shifted primal-dual projected-search IPM. A shared coordinate-wise recurrent network predicts a positive diagonal preconditioner that scales the right-hand side of the reduced Newton system for the primal step, and the remaining slack and multiplier directions are recovered analytically. The learned iterations avoid Hessian evaluations and Newton-system solves, using only first-order and coordinate-wise operations amenable to GPU parallelization. Training is self-supervised, with a loss based on a penalty-barrier merit function and the residual of perturbed optimality conditions, requiring neither target directions nor precomputed solutions. Primal and dual shifts mitigate the barrier's sensitivity to perturbations near constraint boundaries, enabling effective warm starting. Across four classes of 200-dimensional convex and nonconvex constrained problems, pdLIP warm starts reduce pdProj refinement iterations by 63-67% compared with cold starts at the same KKT residual tolerance of 10−8, with negligible warm-start generation cost relative to the subsequent pdProj solve. Improvements persist on box-constrained QPs with 1000 variables and extend to applications including portfolio optimization, support vector machines, and a nonlinear control example.
Abhinav Madabhushi, Jialin Liu, Minxin Zhang
Department of Mathematics, University of California, Los Angeles · School of Data, Mathematical, and Statistical Sciences, University of Central Florida
Nonlinear Parametric Optimization Network (NLPOpt-Net) is an unsupervised learning architecture to solve constrained nonlinear programs (NLP). Given the structure of an NLP, it learns the parametric solution maps with guaranteed constraint satisfaction. The architecture consists of a backbone neural network (NN) followed by a multilayer (k-layered) projection. While the NN drives toward optimality through a loss function consisting of a modified Lagrangian augmented with a consistency loss, the projection ensures feasibility by projecting the NN predictions in the original constraint manifold. Instead of typical distance minimization, our projection exploits local quadratic approximations of the original NLP. Under certain conditions (such as convexity), the projection has a descent property, which improves the NN predictions further. NLPOpt-Net deploys an inversion-free, modified Chambolle-Pock algorithm to solve the constrained quadratic projections during the forward pass and uses the implicit function theorem for efficient backpropagation. The fixed structure of the projection further allows decoupling of the NN and the projection once the training is complete. NLPOpt-Net solves large-scale convex QP, QCQP, NLP, and nonconvex problems with near zero optimality gap and constraint violations reduced to machine precision. Additionally, it provides near accurate prediction of the active sets and corresponding dual variables, thereby enabling a scalable approach for multiparametric programming. Compiling the projection in C provides order of magnitude improvement in inference time compared to JAX. We provide the codes and NLPOpt-Net as a ready to use package that includes GPU support.