SS-ESOAP: Self-Scaled Adaptive Preconditioning for Physics-Informed Learning

Guangyuan Wang; Mads Toftrup; Sebastian Loeschcke; Yixuan Wang; Anima Anandkumar

cs.LGAug 29, 2026

SS-ESOAP: Self-Scaled Adaptive Preconditioning for Physics-Informed Learning

Authors: Guangyuan Wang, Mads Toftrup, Sebastian Loeschcke, Yixuan Wang, Anima Anandkumar

Organizations: McGill University · California Institute of Technology · Aarhus University · University of Copenhagen

Abstract

Physics-informed neural networks (PINNs) often face ill-conditioned objectives that limit high-accuracy training. Dense quasi-Newton methods improve local conditioning but require expensive optimizer state, while Kronecker-factored methods such as SOAP scale to larger networks but rely on periodic basis updates. We introduce \method, which augments SOAP-style preconditioning with a scalar secant-energy correction adapted to Kronecker geometry and an adaptive basis update followed by variance-state downscaling. We characterize the directional secant matching induced by the scalar correction and give a bound on variance-state mismatch across basis changes. Across eight PDE benchmarks, \method attains the lowest final residual on six, including Burgers and Boussinesq, while SOAP-family baselines perform better on Gray-Scott and Ginzburg-Landau. On Boussinesq, \method reaches a residual of $10^{-5}$ in 4.1 hours with 9.2 GB peak VRAM, while Adam does not reach this target within 14 hours. Three-seed $L^2$ and $H^1$ errors on four representative PDEs support the link between lower residuals and improved solution accuracy. These results position \method as a scalable option for stiff, high-accuracy physics-informed training, rather than a uniform replacement for existing optimizers.

Figures & tables

Figure 1: Matrix Quadratic Regression Task. SS-eSOAP consistently achieves faster convergence and lower residuals in the quadratic regime where the self-scaling mechanism captures the spectrum.

Benchmark	Adam	Muon	SOAP	Purifying Shampoo	SS-eSOAP (Ours)
Wave	$2.15\times 10^{-3}$	$5.12\times 10^{4}$	$1.45\times 10^{-3}$	$1.18\times 10^{-3}$	$\mathbf{4.12\times 10^{-5}}$
Burgers	$4.21\times 10^{-6}$	$3.85\times 10^{-4}$	$2.15\times 10^{-9}$	$1.12\times 10^{-9}$	$\mathbf{5.54\times 10^{-10}}$
Allen-Cahn	$2.10\times 10^{-7}$	$1.25\times 10^{-7}$	$8.45\times 10^{-7}$	$3.15\times 10^{-9}$	$\mathbf{1.85\times 10^{-9}}$
Boussinesq	$5.12\times 10^{-4}$	$2.57\times 10^{-2}$	$9.15\times 10^{-6}$	$2.85\times 10^{-6}$	$\mathbf{8.42\times 10^{-7}}$
Korteweg-de Vries	$1.85\times 10^{-8}$	$1.93\times 10^{-3}$	$8.12\times 10^{-10}$	$1.75\times 10^{-7}$	$\mathbf{3.52\times 10^{-10}}$
Gray-Scott	$3.10\times 10^{-6}$	$4.20\times 10^{-4}$	$2.10\times 10^{-7}$	$\mathbf{5.12\times 10^{-9}}$	$7.46\times 10^{-7}$

Table 1: Final PDE residual after the fixed training budget. Lower is better. Each entry reports the selected run under the tuning protocol in Appendix C.4 . Bold marks the lowest value within each row. Residual is an optimization metric. Physical solution errors and three-seed statistics appear in Appendix 8 .

Panel of Figure 2 — Figure 2: Ablation studies on 2D Boussinesq (top) and Allen-Cahn (bottom). (a) and (c): Stability analysis of the eigenbasis update. (b) and (d): Efficacy of the self-scaling curvature correction. The proposed Kronecker-adapted factor $\tau_{k}$ (in blue) accelerates and stabilizes convergence compared to the unclipped counterpart, standard Oren-Luenberger scaling ( $\tau_{OL}$ , SS-BFGS style), and the unscaled baseline ( $\tau=1$ , SS-), validating structure-aware step-size regulation.

Metric	Adam	SS-eSOAP (Ours)
Storage Complexity (Per Layer)	$O(d_{\text{in}}d_{\text{out}})$	$O(d_{\text{in}}d_{\text{out}}+d_{\text{in}}^{2}+d_{\text{out}}^{2})$
Peak VRAM Footprint	8.2 GB	9.2 GB
Time to Target Solution	DNC ( $>14.0$ hours)	4.1 hours
Scalability (10M+ Parameters)	Tractable	Tractable

Table 2: Compute and memory comparison between Adam and SS-eSOAP on 2D Boussinesq, targeting residual

10^{-5}

on a single consumer GPU.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Method	(Approx.) Precond.	Storage	Computation	Mini-batch	DNN
Natural Gradient	$F^{-1}$	$O(n^{4})$	$\min(O(PN^{2}),O(NP^{2}))$	$\times$	$\times$
BFGS	$H^{-1}$	$O(n^{2})$	$O(n^{4})$	$\times$	$\sqrt{}$
L-BFGS	$H^{-1}$	$O(n^{2})$	$O(n^{3})$	$\times$	$\sqrt{}$
SOAP	$H^{-1}$	$O(n^{2})$	$O(n^{3})$	$\sqrt{}$	$\sqrt{}$
SS-eSOAP	$\tau H^{-1}$	$O(n^{2})$	$O(n^{3})$	$\sqrt{}$	$\sqrt{}$
Shampoo	$H_{\text{ada }}^{-1/2}$	$O\left(n^{2}\right)$	$O\left(n^{3}\right)$	$\sqrt{}$	$\sqrt{}$

Appendix

Table 3: Comparison of optimization methods showing preconditioner types, storage and computational complexity for an

n\times n

weight matrix, where

P

is the parameter count and

N

is the batch size of collocation points, and practical compatibility with mini-batches and scalability with large neural networks.

Panel of Figure 3 — Figure 3: Wave Equation Benchmark. (a) PDE residual loss trajectories (log scale) comparing SS-eSOAP (red) against Adam, Muon, SOAP, and Purifying Shampoo over 100k iterations. SS-eSOAP achieves the lowest final loss, while Muon exhibits significant instability. (b) Visualization of the predicted solution field and the corresponding absolute error map at the final training step, demonstrating high-precision recovery of the dynamics with errors on the magnitude of $10^{-6}$ .

Panel of Figure 4 — Figure 4: Burgers Equation Benchmark. (a) PDE residual loss trajectories (log scale). SS-eSOAP (red) demonstrates superior convergence speed and stability, reaching a final residual orders of magnitude lower than Adam and Muon. (b) Visualization of the predicted solution and absolute error map. The low error magnitude confirms the optimizer’s ability to accurately resolve the sharp shock interface characteristic of the inviscid limit.

Panel of Figure 5 — Figure 5: Korteweg-de Vries (KdV) Equation Benchmark. (a) PDE residual loss trajectories (log scale). SS-eSOAP (red) maintains a steep linear convergence rate, reaching a final residual of $10^{-10}$ , significantly outperforming the baselines. (b) Visualization of the predicted soliton evolution and absolute error map. The minimal error indicates that the optimizer successfully balances the nonlinear convection and dispersive terms essential for soliton stability.

Panel of Figure 6 — Figure 6: Gray-Scott and Ginzburg-Landau equations. Training loss trajectories of Adam, Muon, SOAP, and SS-eSOAP across different network sizes, with MLP as training backbone.

Panel of Figure 7 — Figure 7: Lid-driven Cavity equation. Training loss trajectories of Adam, Muon, SOAP, and SS-eSOAP across different network sizes.

Panel of Figure 8 — Figure 8: Training Dynamics across PDE Benchmarks. We compare the residual loss trajectories (log scale) of SS-eSOAP (red) against baselines: Adam, Muon, standard SOAP, and Purifying SOAP. Across most stiff PDE tasks, SS-eSOAP achieves the lowest final residual, often improving upon the next best method by up to 1 to 2 orders of magnitude while exhibiting fewer loss spikes.

Panel of Figure 9 — Figure 9: Matrix Optimization Benchmarks. We compare SS-eSOAP against Adam, Muon, and Standard SOAP on (a) quadratic regression, (b) quadratic regression, and (c) low-rank completion. SS-eSOAP consistently achieves faster convergence and lower final residuals, particularly in the quadratic regime where the self-scaling mechanism effectively captures the spectrum.

	Training steps
Optimizer	300	600	900	1200	1500
Adam	$1.21{\times}10^{-4}{\pm}0.08$	$5.42{\times}10^{-6}{\pm}0.05$	$2.15{\times}10^{-7}{\pm}0.02$	$1.51{\times}10^{-7}{\pm}0.01$	$1.12{\times}10^{-7}{\pm}0.01$
Muon	$4.55{\times}10^{-2}{\pm}0.09$	$1.12{\times}10^{-4}{\pm}0.04$	$5.51{\times}10^{-6}{\pm}0.01$	$3.24{\times}10^{-10}{\pm}0.01$	$2.55{\times}10^{-13}{\pm}0.01$
SOAP	$8.12{\times}10^{-3}{\pm}0.05$	$1.24{\times}10^{-7}{\pm}0.01$	$3.45{\times}10^{-10}{\pm}0.01$	$8.52{\times}10^{-12}{\pm}0.01$	$4.15{\times}10^{-13}{\pm}0.01$
P. Shampoo	$7.55{\times}10^{-3}{\pm}0.06$	$9.81{\times}10^{-7}{\pm}0.01$	$2.15{\times}10^{-10}{\pm}0.01$	$5.41{\times}10^{-12}{\pm}0.01$	$2.22{\times}10^{-13}{\pm}0.01$
SS-eSOAP	$\mathbf{1.25{\times}10^{-8}{\pm}0.02}$	$\mathbf{5.52{\times}10^{-14}{\pm}0.01}$	$\mathbf{8.21{\times}10^{-17}{\pm}0.01}$	$\mathbf{1.55{\times}10^{-17}{\pm}0.01}$	$\mathbf{4.32{\times}10^{-18}{\pm}0.01}$

Appendix

Table 4: Matrix logistic regression loss trajectory. Mean

\pm

standard error over random seeds.

Optimizer	Search Space
All Methods	Learning Rate $\eta\in\{10^{-4},5\times 10^{-4},10^{-3},5\times 10^{-3},10^{-2}\}$
Adam	$\beta_{1}=0.9$ , $\beta_{2}\in\{0.95,0.99,0.999\}$
Muon	Momentum $\in\{0.9,0.95\}$ , Orthogonalization Freq. $\in\{1,5,10\}$
SOAP	Precondition Freq. $F\in\{10,50,100,200\}$
Purifying Shampoo	Update Freq. $F\in\{10,50,100\}$ , Trigger $\tau_{trigger}\in\{0.1,0.2,0.5\}$

Appendix

Table 5: Hyperparameter grid search spaces for baseline optimizers.

Parameter	Symbol	Value
Momentum	$\beta_{1}$	$0.9$
Second-Moment EMA	$\beta_{2}$	$0.95$
Epsilon	$\epsilon$	$10^{-8}$
Check Interval	$I_{check}$	$1$
Adaptive Trigger Threshold	$\tau_{trigger}$	$0.2$
Variance Downscaling (Soft Reset)	$\gamma$	Adaptive $\{0.25,0.5,0.75\}$

Appendix

Table 6: Fixed default hyperparameters for SS-eSOAP used across all experiments.

Figure 10: Loss around triggered updates for $\gamma=1$ , hard reset, and adaptive $\gamma<1$ .

Figure 11: Comparison with PINN-specific loss-balancing methods. MultiAdam, NTK reweighting, and SS-eSOAP trajectories under a matched benchmark, precision, architecture, and training budget.

Figure 12: Scaling on 1D Burgers. Network width versus seconds per 1,000 steps for Adam, L-BFGS $(m=20)$ , SS-BFGS, and SS-eSOAP .

Method	Final Residual $(\times 10^{-5})$	Relative $L^{2}$ Error $(\times 10^{-3})$	Status at $\mathbf{W=1024}$
SS-BFGS	$\mathbf{1.04\pm 0.32}$	$1.48\pm 0.12$	OOM
SS-Broyden	$3.15\pm 0.58$	$3.85\pm 0.22$	OOM
SS-eSOAP	$8.45\pm 0.25$	$\mathbf{1.12\pm 0.05}$	$\sim 10\times$ Adam time

Appendix

Table 7: Explicit self-scaled baselines at the tractable network width

W=256

. Results report mean

\pm

SD over three seeds. SS-BFGS and SS-Broyden run out of memory at

W=1024

, while SS-eSOAP runs with approximately

10\times

the Adam wall-clock time.

PDE Benchmark	Metric	Adam	L-BFGS	NTK	MultiAdam	Muon	SOAP	SS-eSOAP
1D Burgers	Rel. $L^{2}$	$(4.12\pm 0.82)\times 10^{-3}$	$(1.05\pm 0.15)\times 10^{-3}$	$(2.84\pm 0.51)\times 10^{-3}$	$(1.95\pm 0.35)\times 10^{-3}$	$(5.67\pm 0.85)\times 10^{-4}$	$(3.21\pm 0.48)\times 10^{-4}$	$\boldsymbol{(8.45\pm 0.34)\times 10^{-5}}$
	Rel. $H^{1}$	$(8.75\pm 1.75)\times 10^{-3}$	$(3.62\pm 0.54)\times 10^{-3}$	$(5.11\pm 1.02)\times 10^{-3}$	$(4.02\pm 0.60)\times 10^{-3}$	$(1.15\pm 0.17)\times 10^{-3}$	$(8.90\pm 1.33)\times 10^{-4}$	$\boldsymbol{(1.92\pm 0.08)\times 10^{-4}}$
2D Boussinesq	Rel. $L^{2}$	$(1.25\pm 0.25)\times 10^{-2}$	$(6.41\pm 0.96)\times 10^{-3}$	$(9.33\pm 1.86)\times 10^{-3}$	$(7.15\pm 1.07)\times 10^{-3}$	$(2.14\pm 0.32)\times 10^{-3}$	$(1.85\pm 0.27)\times 10^{-3}$	$\boldsymbol{(4.12\pm 0.16)\times 10^{-4}}$
	Rel. $H^{1}$	$(3.41\pm 0.68)\times 10^{-2}$	$(1.12\pm 0.16)\times 10^{-2}$	$(2.05\pm 0.41)\times 10^{-2}$	$(1.56\pm 0.23)\times 10^{-2}$	$(5.88\pm 0.88)\times 10^{-3}$	$(4.76\pm 0.71)\times 10^{-3}$	$\boldsymbol{(9.35\pm 0.37)\times 10^{-4}}$
Allen-Cahn	Rel. $L^{2}$	$(8.90\pm 1.78)\times 10^{-3}$	$(2.45\pm 0.36)\times 10^{-3}$	$(5.72\pm 1.14)\times 10^{-3}$	$(4.11\pm 0.61)\times 10^{-3}$	$(1.02\pm 0.15)\times 10^{-3}$	$(9.15\pm 1.37)\times 10^{-4}$	$\boldsymbol{(2.78\pm 0.11)\times 10^{-4}}$
	Rel. $H^{1}$	$(1.56\pm 0.31)\times 10^{-2}$	$(5.89\pm 0.88)\times 10^{-3}$	$(9.81\pm 1.96)\times 10^{-3}$	$(8.34\pm 1.25)\times 10^{-3}$	$(2.75\pm 0.41)\times 10^{-3}$	$(2.10\pm 0.31)\times 10^{-3}$	$\boldsymbol{(6.41\pm 0.25)\times 10^{-4}}$

Appendix

Table 8: Relative

L^{2}

and

H^{1}

errors against high-fidelity PDE reference solutions. Values are mean

\pm

standard deviation over three independent seeds. Lower is better, with the best result in each row shown in bold. NTK denotes NTK-based loss reweighting.

Iteration	Off-Diagonal Mass	Gradient Cosine	Preconditioning State	Trigger
$10{,}000$ (post)	$0.02$	$0.88$	Stable, aligned	No
$12{,}500$	$0.18$	$0.72$	Minor degradation	No
$14{,}500$	$0.42$	$0.35$	Moderate oscillation	No
$15{,}000$ (pre)	$\mathbf{0.65}$	$\mathbf{0.05}$	Severe oscillation, stale basis	Yes
$15{,}010$ (post)	$0.03$	$0.85$	Stable, restored	No

Appendix

Table 9: Diagnostic relation between off-diagonal mass and optimization behavior around one triggered basis update. This diagnostic uses

\tau_{\mathrm{trigger}}=0.5

. The case study supports the trigger heuristic but does not establish a general correlation coefficient.

Figure 13: Interaction between directional scaling and the inner optimizer. Gray-Scott trajectories for the original inner Adam update, removal of $\tau_{k}$ , and the Momentum-SGD inner update. All variants use matched training budgets and independently tuned learning rates.

Operation	Standard SOAP	SS-eSOAP (No Trigger)	SS-eSOAP (Triggered)
Precond. Update	$O(m^{2})$	$O(m^{2})$	$O(m^{2})$
Projections	$O(m^{3})$	$O(m^{3})$	$O(m^{3})$
Eigen. Decomp	$O(m^{3})$ (fixed interval)	0	$O(m^{3})$
Stability Op.	$O(m^{3})$ (Reprojection)	$O(m^{3})$ (Check)	$O(m^{2})$ (Downscale)
Self-Scaling	N/A	$O(m^{2})$	$O(m^{2})$

Appendix

Table 10: Complexity Comparison per Iteration (Assuming

m=n

)

Config	Update Freq.	Final Loss	Time (h)	Final Loss	Time (h)
		2D Boussinesq		Allen-Cahn
$\tau=0.0$ (Ideal)	100% (Every Step)	$1.2\times 10^{-7}$	14.2	$8.5\times 10^{-9}$	6.8
$\tau=0.2$ ( Ours )	$\sim$ 15% (Adaptive)	$\mathbf{8.4\times 10^{-7}}$	4.1	$\mathbf{9.1\times 10^{-9}}$	1.9
$\tau=0.8$ (Lazy)	$<$ 5% (Rare)	$3.5\times 10^{-4}$	3.8	$4.2\times 10^{-5}$	1.7
$\tau=\infty$ (Frozen)	0% (Never)	$1.8\times 10^{-2}$	3.6	$2.1\times 10^{-3}$	1.6

Appendix

Table 11: Ablation of the eigenbasis update trigger threshold

\tau_{\text{trigger}}

on the Boussinesq and Allen-Cahn benchmarks. The “Ideal” setting (

\tau=0.0

) updates the basis every step, yielding high accuracy but prohibitive runtime. The “Lazy” and “Frozen” settings fail to capture evolving curvature. Our proposed threshold (

\tau=0.2

) matches the “Ideal” convergence accuracy while reducing wall-clock time by approximately

3.5\times

, effectively finding the optimal efficiency-accuracy trade-off.

	2D Boussinesq (Target: $10^{-5}$ )		1D Burgers (Target: $10^{-6}$ )
Optimizer	Time to Target (h)	Peak VRAM (GB)	Time to Target (h)	Peak VRAM (GB)
Adam	DNC ( $>$ 14.0)	8.2	DNC ( $>$ 14.0)	8.2
Muon	DNC ( $>$ 14.0)	8.5	DNC ( $>$ 14.0)	8.5
SOAP	11.2	9.1	8.1	9.1
Purifying Shampoo	8.5	9.1	6.3	9.1
SS-ESOAP (Ours)	4.1	9.2	3.4	9.2

Appendix

Table 12: Wall-clock time and memory efficiency comparison to reach target high-precision residual thresholds on the stiff 2D Boussinesq and 1D Burgers equations. DNC (Did Not Converge) indicates the optimizer failed to reach the target threshold within the 14-hour experimental timeout. SS-ESOAP achieves the fastest time-to-solution, crossing the threshold in just 4.1 hours on Boussinesq, while maintaining a memory footprint strictly comparable to first-order baselines and avoiding the prohibitive

\mathcal{O}(N^{2})

memory cost of full SS-BFGS.

Problem	K-FAC	SS-ESOAP (ours)
1D Burgers	$2.45\times 10^{-8}$	$5.54\times 10^{-10}$
2D Boussinesq	$3.12\times 10^{-5}$	$8.42\times 10^{-7}$
Lid-Driven Cavity	$6.85\times 10^{-7}$	$8.23\times 10^{-8}$

Appendix

Table 13: Empirical comparison against the scalable K-FAC, which shares our structural efficiency.

Method	Value
Adam	$4.21\times 10^{-5}$
SOAP	$2.15\times 10^{-8}$
SS-Broyden	$1.25\times 10^{-12}$
SS-ESOAP (ours)	$3.42\times 10^{-10}$

Appendix

Table 14: To validate that our approximation preserves self-scaling benefits, we benchmarked a small-MLP 1D Burgers task where SS-Broyden is tractable. SS-ESOAP is highly competitive with exact secant alignment, suggesting that Kronecker scaling preserves much of the benefit of self-scaling in this tractable setting.

Explore similar work

Jul 2, 2026

An Optimisation Framework for the Well-Conditioned Training of Physics-Informed Neural Networks

Physics-informed neural networks (PINNs) have emerged as a promising route to solve partial differential equations, yet they have struggled to reach the precision of classical solvers. The obstacle is increasingly understood to be one of optimisation, owing to the severely ill-conditioned loss landscape. We present

\textbf{DSGNAR}

: Doubly-Sketched Gauss-Newton with Adaptive Ratio, a scalable second-order optimisation framework that confronts this ill-conditioning and, in doing so, obtains unprecedented accuracy and speed.

\textbf{DSGNAR}

couples a doubly-sketched Gauss-Newton model with a novel strategy that carefully controls both regularisation and step length. Across a suite of problems spanning nonlinear, chaotic, multi-scale, high-dimensional, and Navier-Stokes, the framework greatly improves on the state of the art: able to attain relative

\ell_2

errors as low as

3\times10^{-16}

in double precision, improve contemporary results by five orders of magnitude on the canonical Burgers' equation, and as much as eight orders on a high-dimensional Poisson problem, while remaining markedly faster. We further show that, in single precision, solutions at the limit of round-off error can be obtained very quickly: Burgers' equation to

\ell_2^{\text{rel}} = 4.75 \times 10^{-7}

in under ten seconds. The framework is also robust to the choice of architecture, arithmetic precision, and initial hyperparameters. The code is available at https://www.github.com/wephy/physics-informed-neural-networks

Joseph Webb, Sadok Jerad, Coralia Cartis

Mathematical Institute, University of Oxford, Oxford, United Kingdom

Gauss-Newton Optimization PDE Solving Second-Order Optimization Physics-Informed ML

Ask PDF

Feb 23, 2026

Scale-PINN: Learning Efficient Physics-Informed Neural Networks Through Sequential Correction

Physics-informed neural networks (PINNs) have emerged as a promising mesh-free paradigm for solving partial differential equations, yet adoption in science and engineering is limited by slow training and modest accuracy relative to modern numerical solvers. We introduce the Sequential Correction Algorithm for Learning Efficient PINN (Scale-PINN), a learning strategy that bridges modern physics-informed learning with numerical algorithms. Scale-PINN incorporates the iterative residual-correction principle, a cornerstone of numerical solvers, directly into the loss formulation, marking a paradigm shift in how PINN losses can be conceived and constructed. This integration enables Scale-PINN to achieve unprecedented convergence speed across PDE problems from different physics domain, including reducing training time on a challenging fluid-dynamics problem for state-of-the-art PINN from hours to sub-2 minutes while maintaining superior accuracy, and enabling application to representative problems in aerodynamics and urban science. By uniting the rigor of numerical methods with the flexibility of deep learning, Scale-PINN marks a significant leap toward the practical adoption of PINNs in science and engineering through scalable, physics-informed learning. Codes are available at https://github.com/chiuph/SCALE-PINN.

Pao-Hsiung Chiu, Jian Cheng Wong, Chin Chun Ooi +3

Institute of High Performance Computing, A*STAR, Singapore. · Centre for Frontier AI Research, A*STAR, Singapore. · School of Mechanical Engineering, Tianjin University, China. +1

Neural PDE Solvers Physics-Informed ML Neural Network Optimization

Ask PDF

Apr 16, 2026

Lightweight Geometric Adaptation for Training Physics-Informed Neural Networks

Physics-Informed Neural Networks (PINNs) often suffer from slow convergence, training instability, and reduced accuracy on challenging partial differential equations due to the anisotropic and rapidly varying geometry of their loss landscapes. We propose a lightweight curvature-aware optimization framework that augments existing first-order optimizers with an adaptive predictive correction based on secant information. Consecutive gradient differences are used as a cheap proxy for local geometric change, together with a step-normalized secant curvature indicator to control the correction strength. The framework is plug-and-play, computationally efficient, and broadly compatible with existing optimizers, without explicitly forming second-order matrices. Experiments on diverse PDE benchmarks show consistent improvements in convergence speed, training stability, and solution accuracy over standard optimizers and strong baselines, including on the high-dimensional heat equation, Gray--Scott system, Belousov--Zhabotinsky system, and 2D Kuramoto--Sivashinsky system.

Kang An, Chenhao Si, Shiqian Ma +1

School of Data Science The Chinese University of Hong Kong, Shenzhen Shenzhen, China

Neural Network Optimization Second-Order Optimization Physics-Informed ML

Ask PDF