Physics-informed neural networks (PINNs) often suffer from imbalanced optimization of competing loss components, particularly when approximating solutions with high-frequency or multiscale features. Although existing balancing strategies have proven effective in alleviating this difficulty, the mechanisms governing the evolution of individual task contributions to Adam's parameter updates remain insufficiently understood. We introduce task-wise indicators to characterize this evolution and identify three distinct training regimes: smooth variation, intermittent spikes, and persistent oscillations. Focusing on the early smooth regime, we combine discrete-time analysis with a continuous-time singular perturbation framework under the full-batch setting and the parameter regime (1−β2)/(1−β1)≪1. A first-order approximation of the locally attracting slow manifold reveals two factors governing the step-to-step redistribution of task contributions: curvature-related alignment disparities and differences in the coordinatewise distributions of preconditioned task gradients. Both factors involve Adam's shared second-moment estimate. Inspired by AdaTask, we propose Gradient-Modulated Adam (GMAdam), which modulates incoming task gradients using task-specific second-moment estimates. Numerical experiments on high-frequency Poisson and wave equations, the Allen-Cahn equation, and two-dimensional lid-driven cavity flow (Re=100,400) demonstrate consistent improvements in predictive accuracy over AdaTask at comparable computational cost.
Figures & tables
Figure 1: Comparison of the actual and approximated update contributions: (a) Rk(i) and (b) Δk(i) .
Figure 2: Empirically identified dynamical regimes of the relative update contributions Δk(i) : (a) Phase I (Smooth Evolution); (b) Phase II (Intermittent Spikes); and (c) Phase III (Persistent Oscillations).
Figure 3: Numerical assessment of the second-moment approximation and first-order expansion. (a) Relative discrepancy ∥vk−β2vk−1∥/∥vk∥ . (b) Euclidean and (c) infinity norms of (1−β2)(1−gk−12/vk−1) .
Figure 4: Relative L2 errors of the first-order predictions of xk(i)−yk(i) given by ( 3.19 ), with respect to their actual values from the discrete Adam iterations in the 1D Poisson problem. For (β1,β2)=(0.9,0.999) : (a) η=10−4 ; (b) η=5×10−5 ; (c) η=10−5 .
Figure 5: Relative L2 errors of the first-order predictions of xk(i)−yk(i) given by ( 3.19 ), with respect to their actual values from the discrete Adam iterations in the 1D Poisson problem. For η=5×10−5 : (a) (β1,β2)=(0.9,0.99995) ; (b) (β1,β2)=(0.5,0.999) ; (c) (β1,β2)=(0.5,0.99995) .
Figure 6: Comparison of ωk and ω^k for the one-dimensional Poisson problem under an exponentially decaying learning rate, over three randomly selected iteration intervals within Phase I : (a) k=213 – 223 ; (b) k=1314 – 1324 ; (c) k=1778 – 1788 . Different vertical scales are used to resolve the local variations.
Figure 7: Relative L2 errors of the task-specific second-moment estimates v(1) and v(2) with respect to the shared estimate v , for Adam trained (a) without loss reweighting, (b) with gradient-norm-based loss reweighting, and (c) with NTK-based loss reweighting. The blue and orange curves represent ∥v−v(1)∥2/∥v∥2 and ∥v−v(2)∥2/∥v∥2 , respectively. The vertical axes use a logarithmic scale.
Figure 8: Evolution of Term A, μk(1)−μk(2) , for Adam trained (a) without loss reweighting, (b) with gradient-norm-based loss reweighting, and (c) with NTK-based loss reweighting. The vertical scales differ across panels.
Figure 9: Evolution of Term B, χk(2)−χk(1) , for Adam trained (a) without loss reweighting, (b) with gradient-norm-based loss reweighting, and (c) with NTK-based loss reweighting. The vertical scales differ across panels.
Figure 10: One-dimensional Poisson equation: results obtained with GMAdam after 105 iterations. Left: exact and predicted solutions. Middle: signed pointwise error u−uθ . Right: evolution of the boundary loss Lbc and PDE-residual loss Lr (logarithmic scale).
Method
Relative L2 error
training time (s)
AdaTask
6.56×10−4
340.49
GMAdam
1.84×10−4
349.89
Grad Norm
4.47×10−3
315.15
NTK
1.02×10−1
550.96
Table 1: Relative L2 errors and training times for the 1D Poisson equation after 105 iterations.
Figure 11: Evolution of the relative L2 error during training for (a) AdaTask, (b) GMAdam, (c) Adam with Grad Norm loss weighting, and (d) Adam with NTK-based loss weighting. The vertical axes are logarithmic.
Table 2: Relative L2 errors and training times for the one-dimensional wave equation after 8×104 iterations.
Figure 13: Evolution of the relative L2 error during training for (a) AdaTask, (b) GMAdam, (c) Adam with Grad Norm, and (d) Adam with NTK-based loss weighting. The vertical axes are logarithmic.
Figure 14: One-dimensional Allen–Cahn equation: reference solution (left), prediction obtained with GMAdam (middle), and pointwise absolute error ∣u−uθ∣ (right) after 1.5×105 iterations.
Method
Relative L2 error
Training time (s)
AdaTask
3.6097×10−3
1626.82
GMAdam
2.8518×10−3
1610.42
Grad Norm
2.9500×10−3
1487.00
NTK
3.2818×10−3
2164.36
Table 3: Relative L2 errors and training times for the one-dimensional Allen–Cahn equation after 1.5×105 iterations.
Figure 15: Evolution of the relative L2 error over 1.5×105 training iterations for (a) AdaTask, (b) GMAdam, (c) Adam with Grad Norm, and (d) Adam with NTK-based loss weighting. The vertical axes are logarithmic.
Figure 16: Lid-driven cavity flow at Re=100 : reference velocity magnitude (left), prediction obtained with GMAdam (middle), and pointwise absolute error ∣Uref−Uθ∣ (right) after 40,000 iterations.
Method
eu
ev
eU
Training time (s)
AdaTask
4.3435×10−2
3.1404×10−2
3.7977×10−2
978.62
GMAdam
4.3283×10−2
3.0456×10−2
3.7562×10−2
981.44
Grad-norm
4.3502×10−2
2.8727×10−2
3.7503×10−2
970.59
NTK
4.2995×10−2
2.6276×10−2
3.6914×10−2
1816.74
Table 4: Relative L2 errors and training times for the lid-driven cavity problem at Re=100 after 40,000 iterations.
Figure 17: Evolution of the relative L2 error in velocity magnitude over 40,000 training iterations at Re=100 for (a) AdaTask, (b) GMAdam, (c) Adam with Grad Norm, and (d) Adam with NTK-based loss weighting. The vertical axes are logarithmic.
Figure 18: Lid-driven cavity flow at Re=400 : reference velocity magnitude (left), prediction obtained with GMAdam (middle), and pointwise absolute error ∣Uref−Uθ∣ (right) after 80,000 iterations.
Method
eu
ev
eU
Training time (s)
AdaTask
2.6606×10−1
2.6578×10−1
2.3981×10−1
1893.92
GMAdam
5.9259×10−2
5.2851×10−2
5.5362×10−2
1899.63
Grad Norm
4.2040×10−2
1.9143×10−2
3.3066×10−2
1862.50
NTK
4.4458×10−2
2.5805×10−2
3.6427×10−2
3480.67
Table 5: Relative L2 errors and training times for the lid-driven cavity problem at Re=400 after 80,000 iterations.
Figure 19: Evolution of the relative L2 error in velocity magnitude over 80,000 training iterations at Re=400 for (a) AdaTask, (b) GMAdam, (c) Adam with Grad Norm, and (d) Adam with NTK-based loss weighting. The vertical axes are logarithmic.
Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.