Hybrid Joint-Selective Optimization: Reduced-Space Levenberg-Marquardt Refinement of Low-Dimensional Parameters of Interest

Muhammad Luthfi Shahab; Gabriella Alfa Indahsari; Imam Mukhlash; Hadi Susanto

math.NASep 29, 2026

Hybrid Joint-Selective Optimization: Reduced-Space Levenberg-Marquardt Refinement of Low-Dimensional Parameters of Interest

Authors: Muhammad Luthfi Shahab, Gabriella Alfa Indahsari, Imam Mukhlash, Hadi Susanto

Organizations: Department of Mathematics, Institut Teknologi Sepuluh Nopember, Surabaya, 60111, Indonesia · Department of Mathematics, Khalifa University of Science & Technology, Abu Dhabi, PO Box 127788, United Arab Emirates

Abstract

This paper introduces a hybrid joint-selective optimization (HJSO) framework for large-scale numerical problems in which a small subset of trainable quantities is of primary interest. We partition the full parameter vector into a high-dimensional remaining block and a low-dimensional block of parameters of interest (POIs), perform joint first-order optimization over the full parameter set, and then freeze the remaining variables while applying a reduced-space Levenberg-Marquardt (LM) refinement to the POIs. The method is designed for settings in which the POIs are low-dimensional but strongly influence the quality of the computed solution, while the full parameter space remains too large for full-space second-order methods. The framework is evaluated on three representative problems: a matrix eigenvalue problem, an inverse Bratu problem solved with a physics-informed neural network, and a 100-dimensional nonlinear Black-Scholes problem solved with the DeepBSDE method. In each test, HJSO reaches prescribed POI-error thresholds faster than the corresponding joint first-order baseline and improves the final POI accuracy for the reported solver configurations. The contribution is therefore not a universal optimizer, but a practical reduced-space strategy for problems with known low-dimensional parameters of interest and expensive high-dimensional training variables.

Figures & tables

Setting	Eigenvalue Problem	Inverse Bratu PINNs	Black–Scholes DeepBSDE
Outer cycles, $T$	200	400	100
Joint-phase solver	MATLAB fminunc	MATLAB fminunc	TensorFlow Adam
Joint-phase algorithm	Steepest descent	Steepest descent	Adam
Joint-phase iterations, $T_{\mathrm{FO}}$	50	50	100
Joint-phase step tolerance	$10^{-12}$	$10^{-12}$	–
Selective-phase solver	MATLAB fsolve	MATLAB fsolve	SciPy least_squares

Table 1 : Optimization settings used for JO and HJSO in the numerical experiments.

Setting	Eigenvalue Problem	Inverse Bratu PINNs	Black–Scholes DeepBSDE
Problem dimension	$n=200$	One-dimensional	$d=100$
Parameter(s) of interest	$\lambda$	$(\lambda_{1},\lambda_{2})$	$u_{0}$
Initial POI(s)	$\lambda^{(0)}=120$	$(\lambda_{1}^{(0)},\lambda_{2}^{(0)})=(0,0)$	$u_{0}^{(0)}=100$
Remaining trainable parameters	Eigenvector $\bm{v}$	NN weights and biases $\bm{w}$	DeepBSDE NN parameters
Neural-network architecture	–	$\mathrm{NN}(1,20,20,1)$	Hidden layers $(110,110)$
Training/collocation points	–	$99$	Batch size $64$

Table 2 : Problem-specific settings used in the numerical experiments.

Method	Loss	$\lambda$	APE $\lambda$	Time (s)	Time to 1% APE (s)
HJSO	$8.5\times 10^{-17}$	109.2516	$8.4\times 10^{-13}$	0.54	0.13
JO	$3.4\times 10^{-2}$	109.4358	$1.6\times 10^{-1}$	22	14

Table 3 : Comparison of JO and HJSO for the largest eigenvalue of the

200\times 200

Lehmer matrix.

Method	Loss	$\lambda_{1}$	$\lambda_{2}$	APE $\lambda_{1}$	APE $\lambda_{2}$	Time (s)	Time to 1% APE (s)
HJSO	$9.9\times 10^{-6}$	1.9955	1.0082	0.2236	0.8250	99	9.1
JO	$1.9\times 10^{-3}$	2.1295	0.7559	6.4772	24.4068	109	–

Table 4 : Comparison of JO and HJSO for the inverse Bratu problem.

Method	Loss	$u_{0}$	APE $u_{0}$	Time (s)	Time to 1% APE (s)
HJSO	22.6285	57.1889	0.1938	1553	78
JO	22.7697	57.0165	0.4948	1432	1198

Table 5 : Comparison of JO and HJSO for the 100-dimensional nonlinear Black–Scholes problem.

Explore similar work

( $θ_l, θ_u$ )-Parametric Multi-Task Optimization: Joint Search in Solution and Infinite Task Spaces
Mar 11, 2025Tingyang Wei, Jiao Liu, Abhishek Gupta +2Multi-Objective Optimization Multi--Task Learning
Select-then-differentiate: Solving Bilevel Optimization with Manifold Lower-level Solution Sets
May 9, 2026Saeed Masiha, Zebang Shen, Negar Kiyavash +1Bilevel Optimization Flat Minima
EPFL School of Management of Technology, Station 5, 1015 Lausanne, Switzerland · ETH Department of Computer Science, Universitätstrasse 6, 8092 Zürich, Switzerland
Direction-Magnitude Decomposition for Low-Rank Matrix Optimization: Faster Convergence and Saddle-to-saddle Dynamics
Jun 30, 2026Yudong Wei, Liang Zhang, Bingcong Li +1Low-Rank Structure Riemannian Optimization
Department of Mathematics, ETH Zurich, Zurich, Switzerland · Department of Computer Science, ETH Zurich, Zurich, Switzerland

Mar 11, 2025, cs.NEJ/K move · Enter open · S save

( $θ_l, θ_u$ )-Parametric Multi-Task Optimization: Joint Search in Solution and Infinite Task Spaces

Tingyang Wei, Jiao Liu, Abhishek Gupta, Puay Siew Tan+1

Multi-Objective Optimization Multi--Task Learning Joint Search Robot Autonomy Engineering

Open paper AskPDF arXiv

Multi-task optimization is typically characterized by a fixed and finite set of tasks. The present paper relaxes this condition by considering a non-fixed and potentially infinite set of optimization tasks defined in a parameterized, continuous and bounded task space. We refer to this unique problem setting as parametric multi-task optimization (PMTO). Assuming the bounds of the task parameters to be (

\boldsymbolθ_l

\boldsymbolθ_u

), a novel (

\boldsymbolθ_l

\boldsymbolθ_u

)-PMTO algorithm is crafted to operate in two complementary modes. In an offline optimization mode, a joint search over solution and task spaces is carried out with the creation of two approximation models: (1) for mapping points in a unified solution space to the objective spaces of all tasks, which provably accelerates convergence by acting as a conduit for inter-task knowledge transfers, and (2) for probabilistically mapping tasks to their corresponding solutions, which facilitates evolutionary exploration of under-explored regions of the task space. In the online mode, the derived models enable direct optimization of any task within the bounds without the need to search from scratch. This outcome is validated on both synthetic test problems and practical case studies, with the significant real-world applicability of PMTO shown towards fast reconfiguration of robot controllers under changing task conditions. The potential of PMTO to vastly speedup the search for solutions to minimax optimization problems is also demonstrated through an example in robust engineering design.

Abstract

Explore similar work

(θl,θuθ_l, θ_uθl​,θu​)-Parametric Multi-Task Optimization: Joint Search in Solution and Infinite Task Spaces

Select-then-differentiate: Solving Bilevel Optimization with Manifold Lower-level Solution Sets

Direction-Magnitude Decomposition for Low-Rank Matrix Optimization: Faster Convergence and Saddle-to-saddle Dynamics

( $θ_l, θ_u$ )-Parametric Multi-Task Optimization: Joint Search in Solution and Infinite Task Spaces