Physics-informed neural networks (PINNs) often suffer from imbalanced optimization of competing loss components, particularly when approximating solutions with high-frequency or multiscale features. Although existing balancing strategies have proven effective in alleviating this difficulty, the mechanisms governing the evolution of individual task contributions to Adam's parameter updates remain insufficiently understood. We introduce task-wise indicators to characterize this evolution and identify three distinct training regimes: smooth variation, intermittent spikes, and persistent oscillations. Focusing on the early smooth regime, we combine discrete-time analysis with a continuous-time singular perturbation framework under the full-batch setting and the parameter regime (1−β2)/(1−β1)≪1. A first-order approximation of the locally attracting slow manifold reveals two factors governing the step-to-step redistribution of task contributions: curvature-related alignment disparities and differences in the coordinatewise distributions of preconditioned task gradients. Both factors involve Adam's shared second-moment estimate. Inspired by AdaTask, we propose Gradient-Modulated Adam (GMAdam), which modulates incoming task gradients using task-specific second-moment estimates. Numerical experiments on high-frequency Poisson and wave equations, the Allen-Cahn equation, and two-dimensional lid-driven cavity flow (Re=100,400) demonstrate consistent improvements in predictive accuracy over AdaTask at comparable computational cost.
Figures & tables
Figure 1: Comparison of the actual and approximated update contributions: (a) Rk(i) and (b) Δk(i) .
Figure 2: Empirically identified dynamical regimes of the relative update contributions Δk(i) : (a) Phase I (Smooth Evolution); (b) Phase II (Intermittent Spikes); and (c) Phase III (Persistent Oscillations).
Figure 3: Numerical assessment of the second-moment approximation and first-order expansion. (a) Relative discrepancy ∥vk−β2vk−1∥/∥vk∥ . (b) Euclidean and (c) infinity norms of (1−β2)(1−gk−12/vk−1) .
Figure 4: Relative L2 errors of the first-order predictions of xk(i)−yk(i) given by ( 3.19 ), with respect to their actual values from the discrete Adam iterations in the 1D Poisson problem. For (β1,β2)=(0.9,0.999) : (a) η=10−4 ; (b) η=5×10−5 ; (c) η=10−5 .
Figure 5: Relative L2 errors of the first-order predictions of xk(i)−yk(i) given by ( 3.19 ), with respect to their actual values from the discrete Adam iterations in the 1D Poisson problem. For η=5×10−5 : (a) (β1,β2)=(0.9,0.99995) ; (b) (β1,β2)=(0.5,0.999) ; (c) (β1,β2)=(0.5,0.99995) .
Figure 6: Comparison of ωk and ω^k for the one-dimensional Poisson problem under an exponentially decaying learning rate, over three randomly selected iteration intervals within Phase I : (a) k=213 – 223 ; (b) k=1314 – 1324 ; (c) k=1778 – 1788 . Different vertical scales are used to resolve the local variations.
Figure 7: Relative L2 errors of the task-specific second-moment estimates v(1) and v(2) with respect to the shared estimate v , for Adam trained (a) without loss reweighting, (b) with gradient-norm-based loss reweighting, and (c) with NTK-based loss reweighting. The blue and orange curves represent ∥v−v(1)∥2/∥v∥2 and ∥v−v(2)∥2/∥v∥2 , respectively. The vertical axes use a logarithmic scale.
Figure 8: Evolution of Term A, μk(1)−μk(2) , for Adam trained (a) without loss reweighting, (b) with gradient-norm-based loss reweighting, and (c) with NTK-based loss reweighting. The vertical scales differ across panels.
Figure 9: Evolution of Term B, χk(2)−χk(1) , for Adam trained (a) without loss reweighting, (b) with gradient-norm-based loss reweighting, and (c) with NTK-based loss reweighting. The vertical scales differ across panels.
Figure 10: One-dimensional Poisson equation: results obtained with GMAdam after 105 iterations. Left: exact and predicted solutions. Middle: signed pointwise error u−uθ . Right: evolution of the boundary loss Lbc and PDE-residual loss Lr (logarithmic scale).
Method
Relative L2 error
training time (s)
AdaTask
6.56×10−4
340.49
GMAdam
1.84×10−4
349.89
Grad Norm
4.47×10−3
315.15
NTK
1.02×10−1
550.96
Table 1: Relative L2 errors and training times for the 1D Poisson equation after 105 iterations.
Figure 11: Evolution of the relative L2 error during training for (a) AdaTask, (b) GMAdam, (c) Adam with Grad Norm loss weighting, and (d) Adam with NTK-based loss weighting. The vertical axes are logarithmic.
Table 2: Relative L2 errors and training times for the one-dimensional wave equation after 8×104 iterations.
Figure 13: Evolution of the relative L2 error during training for (a) AdaTask, (b) GMAdam, (c) Adam with Grad Norm, and (d) Adam with NTK-based loss weighting. The vertical axes are logarithmic.
Figure 14: One-dimensional Allen–Cahn equation: reference solution (left), prediction obtained with GMAdam (middle), and pointwise absolute error ∣u−uθ∣ (right) after 1.5×105 iterations.
Method
Relative L2 error
Training time (s)
AdaTask
3.6097×10−3
1626.82
GMAdam
2.8518×10−3
1610.42
Grad Norm
2.9500×10−3
1487.00
NTK
3.2818×10−3
2164.36
Table 3: Relative L2 errors and training times for the one-dimensional Allen–Cahn equation after 1.5×105 iterations.
Figure 15: Evolution of the relative L2 error over 1.5×105 training iterations for (a) AdaTask, (b) GMAdam, (c) Adam with Grad Norm, and (d) Adam with NTK-based loss weighting. The vertical axes are logarithmic.
Figure 16: Lid-driven cavity flow at Re=100 : reference velocity magnitude (left), prediction obtained with GMAdam (middle), and pointwise absolute error ∣Uref−Uθ∣ (right) after 40,000 iterations.
Method
eu
ev
eU
Training time (s)
AdaTask
4.3435×10−2
3.1404×10−2
3.7977×10−2
978.62
GMAdam
4.3283×10−2
3.0456×10−2
3.7562×10−2
981.44
Grad-norm
4.3502×10−2
2.8727×10−2
3.7503×10−2
970.59
NTK
4.2995×10−2
2.6276×10−2
3.6914×10−2
1816.74
Table 4: Relative L2 errors and training times for the lid-driven cavity problem at Re=100 after 40,000 iterations.
Figure 17: Evolution of the relative L2 error in velocity magnitude over 40,000 training iterations at Re=100 for (a) AdaTask, (b) GMAdam, (c) Adam with Grad Norm, and (d) Adam with NTK-based loss weighting. The vertical axes are logarithmic.
Figure 18: Lid-driven cavity flow at Re=400 : reference velocity magnitude (left), prediction obtained with GMAdam (middle), and pointwise absolute error ∣Uref−Uθ∣ (right) after 80,000 iterations.
Method
eu
ev
eU
Training time (s)
AdaTask
2.6606×10−1
2.6578×10−1
2.3981×10−1
1893.92
GMAdam
5.9259×10−2
5.2851×10−2
5.5362×10−2
1899.63
Grad Norm
4.2040×10−2
1.9143×10−2
3.3066×10−2
1862.50
NTK
4.4458×10−2
2.5805×10−2
3.6427×10−2
3480.67
Table 5: Relative L2 errors and training times for the lid-driven cavity problem at Re=400 after 80,000 iterations.
Figure 19: Evolution of the relative L2 error in velocity magnitude over 80,000 training iterations at Re=400 for (a) AdaTask, (b) GMAdam, (c) Adam with Grad Norm, and (d) Adam with NTK-based loss weighting. The vertical axes are logarithmic.
Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.
Physics-informed neural networks (PINNs) provide a flexible framework for solving forward and inverse problems governed by partial differential equations (PDEs), but standard PINN training typically relies on soft penalty formulations that combine PDE residuals, data mismatch, and initial/boundary conditions using manually chosen weights. This often leads to ill-conditioning, sensitivity to loss weights, and poor constraint satisfaction. In this work, we reformulate PINN training as an equality-constrained optimization problem and propose a novel Adaptive Momentum Feedback Linearization Optimization for Hard Constrained PINN (AdamFLIP). The key idea is to view the constraint residuals as the output of a controlled dynamical system and to compute the Lagrange multiplier as a feedback input that locally drives these residuals toward stable linear contraction dynamics. AdamFLIP then applies Adam-style first- and second-moment adaptation to the resulting feedback-linearized Lagrangian gradient, combining principled constraint handling with the scalability and robustness of adaptive neural-network optimization. We test AdamFLIP on a range of benchmark forward and inverse PDE problem, and it consistently outperforms both the standard soft-constrained PINN and state-of-the-art constrained optimizers. Specifically, on the Navier--Stokes equations benchmark, AdamFLIP \textbf{reduces relative L2 error by more than two thirds} for the predicted solution compared to the next best method. Our AdamFLIP framework provides an effective and computationally scalable hard constraint optimization method for PINN training.
Binghang Lu, Runyu Zhang, Changhong Mou +2
Purdue University · Massachusetts Institute of Technology · Utah State University +1
The Adam optimizer is widely used to train neural networks with highly nonlinear and nonconvex objective functions. Remarkably, Adam often drives the empirical training loss nearly to zero. This phenomenon has been partially explained by neural tangent kernel (NTK) analyses of gradient descent (GD) and stochastic gradient descent (SGD). However, analyzing Adam presents additional challenges. Its first-moment momentum couples stochastic gradients across iterations, while its adaptive second-moment preconditioner depends on the same stochastic gradient used in the update. In this paper, we establish linear convergence guarantees for Adam in training sufficiently wide two-layer neural networks with either full-batch or minibatch gradients. For full-batch Adam, we construct a Lyapunov function to control the effect of historical momentum. For minibatch Adam, our analysis builds on an exact virtual-iterate representation, a conditional analysis of the dependence between stochastic gradients and adaptive preconditioners, and uniform concentration bounds for the second-moment estimates. We prove geometric decay of the empirical loss for full-batch Adam and geometric decay in restricted expectation on a high-probability event for minibatch Adam. Beyond regression, we extend our convergence analysis to physics-informed neural networks (PINNs) for solving the Poisson equation, where differential operators and separate sampling of interior and boundary points introduce additional challenges. We establish analogous global convergence guarantees for both deterministic and stochastic Adam under explicit conditions on network width and Adam's parameters, characterizing the effects of minibatch size on convergence. These results help bridge the gap between Adam's empirical success and its theoretical understanding, providing sharper convergence guarantees for PINNs than existing analyses.
Yihang Gao
Department of Mathematics, Hong Kong Baptist University
Physics-Informed Neural Networks (PINNs) are trained by optimising a composite objective that combines data fitting with physics-based constraints, typically resulting in a highly imbalanced multi-task optimisation problem. Under these conditions, existing optimisation strategies are affected by conflicting task gradients, leading to slow convergence and unstable training, particularly for stiff and high-frequency partial differential equations. We analyse gradient conflicts throughout training of PINNs with standard optimiser and investigate Multi-Task Deep Learning (MTDL) optimisation methods. In our analysis across four benchmark problems we observed that PINN optimisation exhibits three distinct phases in which angle- and magnitude-based gradient conflicts alternate, with only one present at a time. Building on these observations, we propose PAM-GS, a physics-aware gradient surgery method that adaptively mitigates task interference during training according to the observed conflict types. Experiments on four representative PDE benchmarks demonstrate that PAM-GS combines competitive solution accuracy with consistently strong task-balanced performance, outperforming existing methods on most problems.
Thomas Borsani, Giuseppe Di Fatta
Faculty of Engineering Free University of Bozen - Bolzano Bozen - Bolzano, Italy