Gradient Surgery for Physics-Informed Neural Networks

Thomas Borsani; Giuseppe Di Fatta

cs.LGSep 25, 2026

Gradient Surgery for Physics-Informed Neural Networks

Authors: Thomas Borsani, Giuseppe Di Fatta

Organizations: Faculty of Engineering Free University of Bozen - Bolzano Bozen - Bolzano, Italy

Abstract

Physics-Informed Neural Networks (PINNs) are trained by optimising a composite objective that combines data fitting with physics-based constraints, typically resulting in a highly imbalanced multi-task optimisation problem. Under these conditions, existing optimisation strategies are affected by conflicting task gradients, leading to slow convergence and unstable training, particularly for stiff and high-frequency partial differential equations. We analyse gradient conflicts throughout training of PINNs with standard optimiser and investigate Multi-Task Deep Learning (MTDL) optimisation methods. In our analysis across four benchmark problems we observed that PINN optimisation exhibits three distinct phases in which angle- and magnitude-based gradient conflicts alternate, with only one present at a time. Building on these observations, we propose PAM-GS, a physics-aware gradient surgery method that adaptively mitigates task interference during training according to the observed conflict types. Experiments on four representative PDE benchmarks demonstrate that PAM-GS combines competitive solution accuracy with consistently strong task-balanced performance, outperforming existing methods on most problems.

Figures & tables

Figure 1: Evolution of gradient conflict metrics ( $cos(\phi)$ , $\Psi$ ) during PINNs training with ADAM optimiser. The red line denotes the angle-based gradient conflict threshold, while the blue lines highlight the transitions between training phases, from Fitting phase to Diffusion and to Total Diffusion.

	Constraint-wise MSE			Overall MSE
GS method	IC	BC	PDE	Mean	SD	MR	$\Delta M\%$
Base Line	1.289	0.716	1.615	1.599	0.035	0	0
FAMO	4993.946	1270.649	3719.593	3698.4728	1.087	11.0	$>1000$
UW-SO	5071.515	1348.557	3797.431	3776.3081	66.17	12.0	$>1000$
IMTL	18.171	850.346	583.555	581.59	681.7	10.0	$>1000$
Aligned-MTL	1.314	0.758	1.706	1.689	0.081	8.5	4.73

Table 1: Performance comparison of gradient surgery methods on the 1D Burgers equation. All MSE values are reported in units of

10^{-4}

GS method	BC	IC	PDE	MSE	BC	IC	PDE	MSE	BC	IC	PDE	MSE	Mean	SD	MR	$\Delta$ M (%)
	Solution $u$				Solution $v$				Magnitude $h$				Overall MSE
Base Line	1.269	1.770	35.86	35.42	0.306	0.566	55.62	54.92	1.049	1.203	3.662	3.630	31.32	4.288	0	0
UW-SO	39430	854.5	44340	44130	1042	1568	33490	33080	37390	1522	67410	66890	48030	114.3	111.6	$>1000$
FAMO	33950	3409	38650	38470	15.70	35.13	30440	30050	33800	3049	59890	59440	42650	85.74	10.73	$>1000$
IMTLg	433.9	377.8	47040	46450	566.6	299.3	36590	36130	40.16	416.8	6005	5934	29500	381.3	10.67	$>1000$
Aligned-MTL	1.515	2.699	9350	9512	1.081	1.748	13710	13540	0.874	1.208	679.0	670.3	7907	112.8	8.8	$>1000$

Table 2: Performance comparison of gradient surgery methods on the 1D Schrödinger equation. All MSE values are reported in units of

10^{-5}

GS method	MSE	MSE	MSE	MSE	Mean	SD	MR	$\Delta M$ (%)
	Pressure $p$	Velocity $u$	Velocity $v$	Velocity $w$	Overall MSE
Base Line	1.946e+01	3.211	3.571	2.897	7.284	1.505	0	0
UW-SO	602200	95830	107200	98950	226000	8734	10.90	$>1000$
FAMO	513200	85500	87890	86990	193400	25.08	10.10	$>1000$
DWA	32.48	6.318	7.212	6.321	13.08	2.953	9.75	92.50
IMTLg	33.46	3.823	3.524	4.218	11.26	0.135	8.1	36.71

Table 3: Performance comparison of gradient surgery methods on the 3D Beltrami flow. All MSE values are expressed in units of

10^{-5}

	Constraint-wise MSE		Overall MSE
GS method	BC	PDE	Mean	SD	MR	$\Delta M\%$
Base Line	7.347	5.041	5.044	3.911	0	0
UW-SO	20530000	4754000	4770000	1857	11.0	$>1000$
FAMO	5686000	2407000	2410000	1978	10.0	$>1000$
CONFIG	20.41	20.23	20.23	3.77	10.0	2.62
SAM-GS	1.452	3.764	3.762	0.345	7.3	-15.51

Table 4: Performance comparison of gradient surgery methods on the 2D Kovasznay flow. All MSE values are reported in units of

10^{-7}

Figure 2: Ablation study of the parameter

\gamma

for SAM-GS and PAM-GS gradient surgery methods during training on the 2D Kovasznay flow. Curves correspond to generalised additive model fits; shaded regions indicate 95% confidence intervals.

Figure 3: Loss landscape of the 2D Kovasznay flow across optimisers. Panels show $\log_{10}$ training loss over a 2D PCA subspace with iso-contours and learning trajectories (blue) on the loss 3D surface and 2D projection.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Memory (MB)	Time (s)
DWA	884.25	$18.375\pm 0.187$
FAMO	884.26	$18.616\pm 0.023$
ADMA	884.25	$18.971\pm 0.961$
GradDrop	1605.13	$23.588\pm 0.917$
SAM-GS	1605.33	$24.220\pm 0.277$
PAM-GS	1605.33	$24.281\pm 0.085$

Appendix

Table 5: Table B:

	Pressure $p$					Velocity $u$					Velocity $v$					Velocity $w$
Method		BC	IC	Interior	MSE		BC	IC	Interior	MSE		BC	IC	Interior	MSE		BC	IC	Interior	MSE	Total MSE	MR	$\Delta M$ (%)
Base Line		2.214e+01	1.865e+01	1.565e+01	1.946e+01		4.275	2.262	2.116	3.211		4.802	2.915	2.014	3.571		4.085	2.292	1.376	2.897	7.284	NA	0.000
UW-SO		318400	2034000	124700	602200		79230	211600	46930	95830		90850	230700	52900	107200		83540	218200	45930	98950	226000	10.90	$>1000$
FAMO		341000	1207000	337600	513200		76920	181600	36550	85500		80500	187200	34910	87890		79330	185300	35110	86990	193400	10.10	$>1000$
DWA		35.96	29.28	28.96	32.48		8.753	5.761	2.753	6.318		9.972	7.138	2.806	7.212		8.827	5.244	2.982	6.321	13.08	9.75	92.50
IMTLg		34.22	18.64	41.91	33.46		5.462	1.806	2.496	3.823		4.524	1.833	3.015	3.524		6.112	2.202	2.480	4.218	11.26	8.1	36.71

Appendix

Table 6: Performance comparison of GS methods on the 3D Beltrami flow. All MSE values are reported in units of

10^{-5}

. Lower values indicate better performance.

Physics-Informed ML

Physics-informed machine learning combines data-driven models with physical laws to solve, predict, and infer scientific systems.

See all 830 papers

Gradient-Update Mismatch: Rethinking Conflict-Free Training of Physics-Informed Neural Networks
Sep 1, 2026Jing Xiao, Xinhai Chen, Qinglin Wang +5Neural Network Optimization Gradient Interference
Laboratory of Digitizing Software for Frontier Equipment, National University of Defense Technology, Changsha 410073, China · National Key Laboratory of Parallel and Distributed Computing, National University of Defense Technology, Changsha 410073, China · School of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China
Per-Loss Adapters for Gradient Conflict in Physics-Informed Neural Networks
May 11, 2026Bum Jun Kim, Gnankan Landry Regis N'guessanGradient Interference Adaptive Loss Weighting
The University of Tokyo, Japan · Axiom Research Group · Department of Applied Mathematics and Computational Science, NM-AIST, Tanzania +3
Modularity-Free Conflict-Averse Training for Generalized PINNs
Jun 18, 2026Heejo Kong, Beomchul Park, Sung-Jin Kim +1Neural PDE Solvers Gradient Interference
Department of Brain and Cognitive Engineering, Korea University, Seoul, Republic of Korea · Department of Artificial Intelligence, Korea University, Seoul, Republic of Korea

Sep 1, 2026J/K move · Enter open · S save

Gradient-Update Mismatch: Rethinking Conflict-Free Training of Physics-Informed Neural Networks

Jing Xiao, Xinhai Chen, Qinglin Wang, Menghan Jia+4

Laboratory of Digitizing Software for Frontier Equipment, National University of Defense Technology, Changsha 410073, China · National Key Laboratory of Parallel and Distributed Computing, National University of Defense Technology, Changsha 410073, China · School of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China

Neural Network Optimization Gradient Interference Gradient Projection Physics-Informed ML

Open paper AskPDF arXiv

Training Physics-Informed Neural Networks (PINNs) requires jointly optimizing physics residual and initial/boundary condition loss terms, which often induce conflicting gradients. Gradient surgery methods mitigate this issue by constructing directions from loss-specific gradients to reduce conflict before optimizer transformation. However, even when the constructed direction is conflict-free, this property may not be preserved after optimizer transformation. Let

a_t

denote the direction constructed by gradient surgery,

u_t

the optimizer proposal, and

\mathcal{C}_t

the conflict-free cone induced by the loss-specific gradients. We show that modern optimizers can transform

a_t

through mechanisms such as historical state, adaptive scaling, preconditioning, or decoupled weight decay, so

a_t \in \mathcal{C}_t

does not generally imply

u_t \in \mathcal{C}_t

. We refer to this optimizer-induced discrepancy in conflict-freeness between

a_t

and

u_t

as Gradient-Update Mismatch (GUM). Accordingly, we propose Gradient-Update Alignment (GUA), which projects

u_t

onto

\mathcal{C}_t

to obtain the aligned update

p_t

and applies

p_t

to the parameters. When the optimizer maintains internal state, GUA further adjusts this state toward targets reconstructed from the applied update. We conduct extensive experiments and find that GUM is widespread across momentum, adaptive, and curvature-based optimizers, with conflict rates reaching up to 86.3%. Across all PINN settings, GUA achieves conflict-free applied updates and consistently improves various gradient surgery methods, reducing the relative

L_2

error by up to 98.2% in individual settings. Data and code are available at https://github.com/JingXiao10/GUA.

Abstract

Physics-Informed ML

Explore similar work

Gradient-Update Mismatch: Rethinking Conflict-Free Training of Physics-Informed Neural Networks

Per-Loss Adapters for Gradient Conflict in Physics-Informed Neural Networks

Modularity-Free Conflict-Averse Training for Generalized PINNs