The Levenberg-Marquardt (LM) algorithm is the most widely used method for solving nonlinear least-squares problems, as it combines the robustness of steepest descent with the fast local convergence of the Gauss-Newton method. However, its computational cost can become prohibitive for large-scale problems because each iteration requires solving a large damped linear system, and conventional step acceptance strategies may require repeated solves as the damping parameter is adjusted. Despite this computational challenge, many large-scale least-squares problems exhibit effective low-dimensional structure, with only a small number of parameter-space directions strongly informed by the data. We propose an adaptive hybrid subspace Levenberg-Marquardt (HSLM) algorithm that constructs a low-dimensional subspace from complementary sources of gradient, memory, Krylov-subspace, and randomized curvature information and computes a spectrally damped LM step within this subspace. A distinguishing feature of the method is a deterministic adequacy monitor that quantifies how much descent information is captured by the reduced space and adaptively enriches the subspace when necessary. Step acceptance is decoupled from damping adjustment: Armijo backtracking determines the accepted step length, while the ratio of actual to predicted reduction is used solely to update the damping parameter, thereby avoiding repeated damped-system solves during step acceptance. For the HSLM algorithm, we establish global convergence to stationarity and prove local linear and superlinear convergence. Numerical experiments on neural-network training problems show that HSLM achieves convergence behavior comparable to classical and Krylov subspace LM (KSLM) while substantially reducing per-iteration computational cost, with increasing advantages observed as the parameter dimension grows.
Figures & tables
Network
Architecture
Trainable Parameters
Training data size
1
7→35→20→1
1021
10000
2
7→60→25→1
2031
20000
3
7→80→40→1
3921
40000
Table 1: Neural network configurations with increasing model complexity and training data size. Network architecture is reported as the number of neurons in the input layer, the two hidden layers, and the output layer, respectively.
Figure 1: Training results for five representative runs of network 3 for nonlinear function approximation. A1-A2. Levenberg–Marquardt algorithm. B1-B2. Krylov subspace Levenberg–Marquardt algorithm. C1-C2. Hybrid subspace Levenberg–Marquardt algorithm. The gray dashed line indicates the variance of the added noise, σ2≈0.622 . These results indicate that HSLM achieves convergence behavior comparable to the classical LM and Krylov-subspace LM algorithms.
Classical LM
Krylov Subspace LM
HSLM
Iteration
Time / Iteration (s)
Iteration
Time / Iteration (s)
Iteration
Time / Iteration (s)
Network 1
41.7
0.492
41.1
0.262
52.7
0.131
Network 2
41.6
2.367
39.4
1.738
36.9
0.615
Network 3
34.9
15.830
36.5
13.082
35.4
3.665
Table 2: Comparison in Computational Performance of Neural Network Training Algorithms
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Krylov LM
Hybrid reduced-subspace LM
Initial Krylov directions
Up to maximum
—
Initial random probes
—
0.01n
Krylov directions per expansion
—
0.02n
Random probes per expansion
—
0.01n
Adequacy threshold
—
ηmin=0.99
Lanczos tolerance
10−5
10−5
Appendix
Table 3: Numerical settings used for the Krylov and adaptive hybrid reduced-subspace LM methods.
Figure 5Figure 6Figure 7Figure 8
Classical LM
Krylov Subspace LM
HSLM
Iteration
Time / Iteration (s)
Iteration
Time / Iteration (s)
Iteration
Time / Iteration (s)
Network 1
43.9
1.044
45.3
0.941
49.8
0.464
Network 2
38.2
3.235
37.1
3.790
37.1
1.221
Network 3
34.9
15.830
36.5
13.082
35.4
3.665
Appendix
Table 4: Comparison between algorithms for neural-network training
Nonconvex optimization develops methods for problems whose objectives or constraints can have multiple local optima, saddle points, or nonsmooth structure.
The Levenberg-Marquardt (LM) algorithm is a well-known second-order method for rapid convergence and strong robustness when training small- to medium-sized neural networks (NNs). However, its computational and memory costs increase significantly as the number of parameters in an NN grows. To address this limitation, subspace methods have been proposed, such as the Krylov subspace LM (KSLM) and the hybrid subspace LM (HSLM), making second-order algorithms more efficient. In this work, we evaluate the subspace Levenberg-Marquardt algorithms for regression and classification tasks in neural networks. We compare the performance of subspace LM variants with the classical LM method, as well as other popular first-order algorithms, such as stochastic gradient descent (SGD) and Adam.
M. Duc Hoang
Department of Mathematics, University of California, Davis, Davis, CA, USA
This paper introduces a hybrid joint-selective optimization (HJSO) framework for large-scale numerical problems in which a small subset of trainable quantities is of primary interest. We partition the full parameter vector into a high-dimensional remaining block and a low-dimensional block of parameters of interest (POIs), perform joint first-order optimization over the full parameter set, and then freeze the remaining variables while applying a reduced-space Levenberg-Marquardt (LM) refinement to the POIs. The method is designed for settings in which the POIs are low-dimensional but strongly influence the quality of the computed solution, while the full parameter space remains too large for full-space second-order methods. The framework is evaluated on three representative problems: a matrix eigenvalue problem, an inverse Bratu problem solved with a physics-informed neural network, and a 100-dimensional nonlinear Black-Scholes problem solved with the DeepBSDE method. In each test, HJSO reaches prescribed POI-error thresholds faster than the corresponding joint first-order baseline and improves the final POI accuracy for the reported solver configurations. The contribution is therefore not a universal optimizer, but a practical reduced-space strategy for problems with known low-dimensional parameters of interest and expensive high-dimensional training variables.
Muhammad Luthfi Shahab, Gabriella Alfa Indahsari, Imam Mukhlash +1
Department of Mathematics, Institut Teknologi Sepuluh Nopember, Surabaya, 60111, Indonesia · Department of Mathematics, Khalifa University of Science & Technology, Abu Dhabi, PO Box 127788, United Arab Emirates
Nonlinear least-squares objectives form the foundation of scientific machine-learning tasks, yet even curvature-aware optimizers remain geometrically inconsistent at finite step sizes: Levenberg-Marquardt (LM) derives its direction from local Riemannian metrics but realizes it as a straight parameter update. Here we introduce RNC-LM, which carries the LM direction along a locally constructed curved trajectory in Riemann normal coordinates. A recursive reformulation of the geodesic equation generates arbitrary finite-order corrections while reusing the same damped Gauss-Newton matrix factorization, and curve length is controlled separately from damping. On a reaction-diffusion physics-informed neural network benchmark, RNC-LM reduces relative L2 errors below 8×10−3, whereas L-BFGS, LM and LM with geodesic acceleration remain near one. On a large-scale machine-learning potential fitting task with 985,160 configurations, fourth-order RNC-LM reaches a fixed training-error target with a 34× wall-clock speedup over LM. These results establish finite-step geometric realization as a distinct optimizer design principle.
Jianing Liu, Dong H. Zhang
State Key Laboratory of Chemical Reaction Dynamics and Department of Chemical Physics, University of Science and Technology of China, Hefei, 230026, China · State Key Laboratory of Chemical Reaction Dynamics, Dalian Institute of Chemical Physics, Chinese Academy of Sciences, Dalian, 116023, China. · University of Chinese Academy of Sciences, Beijing, 100049, China +1