An overview of machine learning-enhanced iterative methods for systems of linear and nonlinear equations
Organizations: Delft Institute of Applied Mathematics, Delft University of Technology, Mekelweg 4, Delft, 2628 CD, the Netherlands · Department of Hydrodynamics and Forecasting, Deltares, Boussinesqweg 1, Delft, 2629 HV, the Netherlands
Abstract
Systems of equations arise in a wide range of scientific and engineering applications. The present work focuses on solvers for general systems of equations, including but not limited to those arising from partial differential equations. These systems can be broadly categorized into linear and nonlinear problems. For large linear systems, iterative solvers are generally preferred over direct methods due to the latter's superlinear growth of computational costs. Although convergence theory is well-developed under certain assumptions on the coefficient matrix, many classes of systems still pose open challenges. These difficulties become even more severe for systems of nonlinear equations, where nonlinear solvers typically rely on repeated linearization. For example, Newton's method may even converge quadratically near the solution; it can also converge slowly or diverge when the initial guess is not chosen appropriately. A wide range of solvers with diverse variants and hyperparameter settings exists, and the development of efficient and robust iterative methods remains an active area of research. Recently, machine learning (ML) techniques have been applied to enhance the efficiency of classical iterative methods while preserving their interpretability and reliability. We refer to these ML-enhanced iterative methods as hybrid iterative methods, in the sense that they combine classical iterative methods with ML. This paper provides a comprehensive overview of state-of-the-art approaches to constructing hybrid iterative methods for systems of both linear and nonlinear equations, while also discussing open challenges and outlining potential directions for future research.
Figures & tables
| method class | initial guess | update function | parameters |
| stationary methods | determines the initial error | relaxation/acceleration parameters, stopping tolerance | |
| Krylov subspace methods | determines and | e . g ., CG for SPD systems, GMRES for non-SPD systems | restart length, truncation or recycling parameters, and stopping tolerances |
| ref. | tested problem | ML | iter. method | main motivation |
| [ 34 ] | Poisson eq. | FNN | Jacobi | from ML: spectral bias of neural networks |
| [ 84 ] | Poisson eq. | CNN | multigrid/BiCGSTAB | from ML: improve CNN predictions |
| [ 85 ] | Poisson eq. from incompressible flow | CNN | Jacobi | from ML: generalization issue of the CNN |
| [ 86 ] | Poisson eq. | CNN | GMRES | from classical iter.: reduce iterations |
| [ 87 ] | Poisson eq. from plasma simulations | GraphSAGE | GMRES | from classical iter.: reduce iterations |
| [ 88 ] | Parametrized elasticity & Biot problems | CAE, FNN | POD-2G | from classical iter.: reduce iterations |
| ref. | tested problem | ML | loss function | iter. method | |
| [ 137 ] | shallow water equations (time-dependent) | FNN | GCR | ||
| [ 138 ] | elliptic PDEs (different resolutions) | SNO | FCG | ||
| [ 139 , 140 ] | parametric Helmholtz equations [ 139 ] , heterogeneous fluid PDEs [ 140 ] | U-Net | FGMRES | ||
| [ 134 ] | Poisson equations from incompressible flow | CNN | PSDO | ||
| [ 136 ] | Poisson equations with mixed boundary conditions | CNN | PSDO | ||
| [ 135 ] | Poisson equations from incompressible flow | DeepONet | Richardson |
| ref. | ML model | final prec. | loss function | iter. method | |
| IC | [ 147 ] | PCG | |||
| [ 52 ] | PCG | ||||
| [ 148 ] | , | , | PCG | ||
| [ 131 ] | , | , | PCG | ||
| ILU | [ 149 ] | = | , | binary cross-entropy | prec. GMRES |
| [ 150 ] | prec. LGMRES |
| ref. | tested problem | classical iter. | ML-preconditioned iter. | learned prec. type |
| [ 37 ] | Poisson and Helmholtz eq. | Jacobi, Gauss-Seidel | DeepONet-prec. Richardson | learning the action of |
| [ 178 ] | Darcy flow and linear elasticity | Gauss-Seidel | DeepONet-prec. Richardson | learning the action of |
| [ 179 ] | Helmholtz eq. | Gauss-Seidel, GMRES | DeepONet-prec. Richardson | learning the action of |
| [ 135 ] | Poisson eq. from incompressible flow | CG or PCG | DeepONet-prec. Richardson (DLSM) | learning the action of |
| [ 180 ] | diffusion and Helmholtz eq. | Jacobi | DeepONet-prec. Richardson/Krylov | enhancing class. (subspace corr.) |
| [ 181 ] | Helmholtz scattering | Stationary methods | DeepONet-prec. Richardson | enhancing class. (subspace corr.) |
| multigrid | domain decomposition | multiscale | deflation/augmentation | |
| Sec. 3.3.2 | smoothers | subdomain solvers | – | – |
| coarse-grid solvers | coarse-grid solvers | |||
| Sec. 3.3.3 | coarsening in AMG | – | – | – |
| Sec. 3.3.4 | transfer operators / coarse spaces / deflation (or augmentation) subspaces | |||
| Sec. 3.3.5 | parameters | parameters | ✗ | ✗ |
| ref. | features | classifier | solver search space (# of candidates used for different problems) | ||
| numerical (#) | structural | numerical | structural | ||
| [ 290 ] | not specified | ✗ | ADT, SVM | ✗ | Krylov methods + preconditioners (48, 242) |
| [ 294 ] | ✗ | ADT, SVM | ✗ | Krylov methods + preconditioners (80) | |
| [ 295 ] | ✗ | RF, XGBoost, GBDT | ✗ | Krylov solver + restart parameter (6) | |
| [ 175 ] | 35 | ✗ | FNNs | ✗ | ILUTP_Mem parameters (72) |
| [ 177 ] | 32 | ✗ | reinforcement learning | ✗ | preconditioned solvers (576) |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| package name | description | download link |
| PETSc | the portable, extensible toolkit for scientific computation | https://petsc.org/release/ |
| Trilinos | collection of open-source packages for scientific application development | https://trilinos.github.io/ |
| Hypre | high-performance preconditioners | https://www.llnl.gov/casc/hypre/ |
| Ginkgo | high-performance numerical linear algebra library | https://gko-project.org/ |
| SuiteSparse Matrix Collection | formerly known as the University of Florida Sparse Matrix Collection | https://sparse.tamu.edu/ |
| ref. | description | download link |
| [ 84 ] | Poisson CNN, learning initial guesses for multigrid & BiCGSTAB | https://github.com/aligirayhanozbay/poisson_CNN |
| [ 86 ] | learning initial guesses for GMRES | https://github.com/ML4FnP/GMRES-Learning |
| [ 89 ] | NOWS, neural operator warm starts | https://github.com/eshaghi-ms/NOWS |
| [ 90 ] | a pretraining-finetuning computational framework for material homogenization | https://github.com/yizheng-wang/HomoGenius |
| [ 91 ] | a pretraining and warm-start framework for PDEs | https://github.com/yizheng-wang/PFEM |
| [ 97 ] | enhancing classical iterative methods for PDEs | https://github.com/ermongroup/Neural-PDE-Solver |
| ref. | description | download link |
| [ 137 ] | machine-learned preconditioners for GCR | https://github.com/JanAckmann/MLPrecon |
| [ 138 ] | FCG-NO, neural operator-based preconditioner for FCG | https://github.com/arudikov/FCG-NO |
| [ 130 ] | GNP, Graph neural preconditioners for FGMRES | https://github.com/jiechenjiechen/GNP |
| [ 134 ] | DCDM, deep conjugate direction method | https://github.com/ayano721/2023_DCDM |
| [ 136 ] | neural-preconditioned steepest descent with orthogonalization | https://github.com/kai-lan/MLPCG |
| [ 135 ] | DLSM and HyDEA, alternating hybrid iterative method | https://github.com/HMB9666/HyDEA |
| ref. | description | download link |
| [ 208 ] | learning multigrid smoothers | https://github.com/jerryhuangru/Learning-optimal-multigrid-smoothers-via-neural-networks |
| [ 211 ] | multigrid-augmented deep learning preconditioners | https://github.com/BGUCompSci/HelmholtzAccCNN.jl |
| [ 214 ] | multigrid-augmented deep learning preconditioners using compact implicit layers | https://github.com/BGUCompSci/CompactImplicitHelmholtz |
| [ 219 ] | algebraic multigrid coarsening using reinforcement learning | https://github.com/compdyn/rl_grid_coarsen |
| [ 225 ] | AMG-ANN, learning the strong threshold parameter using CNNs | https://github.com/MatteoCaldana/AMG-ANN |
| [ 227 ] | AutoAMG( ), learning the strong threshold parameter using GNNs | https://github.com/zhf-0/autoamg |
| ref. | description | download link |
| [ 102 ] | PCGBandit, learning preconditioners for PCG using online learning | https://github.com/mkhodak/PCGBandit |
| [ 310 ] | automated solver selection for simulation of multiphysics processes in porous media | https://github.com/Yuriyzabegaev/solver_selector |
| [ 424 ] | embedding-based methods for linear solver performance prediction | https://gitlab.com/h.liu/embedding-based-predictor |
| ref. | description | download link |
| [ 334 , 341 ] | hybrid Newton’s method | https://github.com/PINN-Well-opening-and-closing-events/Yads |
| [ 346 ] | DeepPhysics: a physics aware deep learning framework for real-time simulation | https://github.com/AlbanOdot/DeepPhysics-article |
| [ 347 ] | TherINO: Thermodynamically-informed iterative neural operators | https://github.com/conlain-k/therino |
| [ 351 ] | Newton informed neural operator for computing multiple solutions of nonlinear PDEs | https://github.com/xlliu2017/Newton-Informed-Neural-Operator |
| [ 355 ] | adaptive learning acceleration for nonlinear PDE solvers | https://github.com/vlssanonymous/mlsolveracc |
| [ 111 ] | improving pseudo-time stepping convergence for CFD simulations with neural networks | https://github.com/searhein/nn-pseudo-time-stepping-data |