Research in the domain of Physics-Informed Neural Networks applied to PDE modelling has advanced significantly in recent years, providing a wide ecosystem of novel methods. However, one frequent limitation is the inability of PINNs to scale out on High-Performance Computing architectures to address more ambitious challenges, particularly non-linear physics models with turbulent dynamics in large spatial domains. Typical studies restrain demonstrations to toy models with simple physics equations, few variables, and rarely any complex non-linear dynamics evolved over long timescales. Part of the AI community has even adopted the idea that "PINNs don't scale". This work proves otherwise. We present a mathematically robust formulation of Finite-Element-Method for Neural Networks (FEM-NN) implemented into a comprehensive, modular framework: Fuse-PINN. This formulation provides the basis for a sclable parallelisation of Physics-Informed Neural Networks on GPU-based HPC systems. Furthermore, an Energy Natural Gradient Descent optimiser is implemented to allow high-precision convergence. Fuse-PINN is demonstrated on physics models relevant to Magnetic Confinement Fusion, including non-linear convective models with fine-scale dynamics. Near-perfect scaling is achieved on multi-GPU runs with increasing problem size.
Figures & tables
Figure 1: (a) Example of a Fuse-PINN grid which can be built directly with [ 11 ] . Intentionally low resolution sampling (blue grid) is used to illustrate the parallelisation method. The slightly higher sampling near FEM-NN edges highlights the 25 FEM-NN nodes and 36 elements. (b) Simulation of Hasegawa-Wakatani demonstrated using 75 NN’s parallelised on 75 GPUs (3 physics variables for 25 FEM-NN nodes).
Figure 2: Parallelisation strategy of the Fuse-PINN implementation with FEM-NNs. Each physics variable (and its corresponding equation) is allocated to a GPU group, and within that group each FEM-NN grid node is allocated to a GPU, such that each Neural Network (on a given GPU) represents the solution of one variable on all parent elements of that FEM-NN grid node.
Figure 3: Testing of the vorticity model with 2 variables. (a) The 2D maps of the fields Φ and ∇2Φ . (b) The training loss as a function of epochs for each variable.
Figure 4: Simulation of the Hasegawa-Wakatani model in a 2D slab initialised with two neighbouring blobs. A full evolution animation is available on Zenodo [ 36 ] .
Figure 5: Comparison of the Poisson solve within the Hasegawa-Wakatani model, at time-step nt=80 . Top left: Vorticity field from a [ 5 ] simulation with identical conditions, showing agreement with Fuse-PINN. Top right: Vorticity field from Fuse-PINN. Bottom right: ∇2Φ field from Fuse-PINN showing good agreement with vorticity field and smoothness of derivatives. Bottom left: FEM-NN elements and nodes for the Fuse-PINN simulation (with lower sampling for visibility).
Figure 6: Simulation of the electrostatic Reduced-MHD model with blobs evolving radially in toroidal geometry. A full evolution animation is available on Zenodo [ 36 ] .
Figure 7
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: A single triangular element showing (a) the weight of the C1 node, w1(x,y) , (b) the sum of all weights (before normalisation), illustrating the overlap regions where the solution is shared between the node NN contributions. Where the sum is highest and reaches ∼ 2, all three node NN’s are contributing, but where the sum is lower, ∼ 1.5, only two node NN’s are contributing.
Weight Functions Control Parameters
parameter
default
description
σ
\nicefrac103100
position of the decay near pseudo-median
λ
\nicefrac20100
width of the decay (in % of side lengths)
δ1
\nicefrac20100
width of transition between distances on left and right sides
δ2
\nicefrac20100
width of transition between varying widths on left and right sides
δ3
\nicefrac20100
width of transition between edge distance and median distance
Appendix
Table 4: List of control parameters for the weight functions.
Figure 10: Ring of 4 elements around a central node, showing (a) the weight of the central node, w1(x,y) , (b) the Laplacian ∇2(w1) , demonstrating smoothness of the weight function and its derivatives across element edges.
Figure 11: Examples of grids generated with GMSH (a) with a user-defined boundary contour, (b) with an arbitrary obstacle in the middle of the grid. The grids contains 13 elements with 11 nodes in total, hence 11 PINNs for each physics variable. The background color shows an arbitrary variable being solved for, shown here for context. The blue lines are the edges of the elements. The white lines show the random sampling used inside each element for the PINN training. Higher-resolution sampling is typically used for simulations, but low-resolution was used here for illustration purposes. Note that obstacles with boundary conditions can be placed anywhere on the grid, including above grid nodes.
Figure 12: Testing of Poisson equation with the same test as in the original ENGD paper [ 14 ] . (a) Training loss as a function of epochs, comparing ENGD with Adam and L-BFGS optimisers, as well as the original JAX implementation of ENGD from Müller & Zeinhofer. (b) Same training losses but as a function of compute time. (c) Effect of Gram Matrix regularisation parameter: higher regularisation leads to slightly faster convergence, but with a trade-off in final convergence level. (d) The 2D map of the final solution Φ , together with the error compared to the analytical solution sin(πx)sin(πy) . (e) Comparison of compute cost between the JAX implementation of the original ENGD paper [ 14 ] and the PyTorch implementation of Fuse-PINN, for various NN sizes, demonstrating satisfactory performance of the Fuse-PINN implementation.
Physics-informed neural networks (PINNs) have emerged as a promising mesh-free paradigm for solving partial differential equations, yet adoption in science and engineering is limited by slow training and modest accuracy relative to modern numerical solvers. We introduce the Sequential Correction Algorithm for Learning Efficient PINN (Scale-PINN), a learning strategy that bridges modern physics-informed learning with numerical algorithms. Scale-PINN incorporates the iterative residual-correction principle, a cornerstone of numerical solvers, directly into the loss formulation, marking a paradigm shift in how PINN losses can be conceived and constructed. This integration enables Scale-PINN to achieve unprecedented convergence speed across PDE problems from different physics domain, including reducing training time on a challenging fluid-dynamics problem for state-of-the-art PINN from hours to sub-2 minutes while maintaining superior accuracy, and enabling application to representative problems in aerodynamics and urban science. By uniting the rigor of numerical methods with the flexibility of deep learning, Scale-PINN marks a significant leap toward the practical adoption of PINNs in science and engineering through scalable, physics-informed learning. Codes are available at https://github.com/chiuph/SCALE-PINN.
Pao-Hsiung Chiu, Jian Cheng Wong, Chin Chun Ooi +3
Institute of High Performance Computing, A*STAR, Singapore. · Centre for Frontier AI Research, A*STAR, Singapore. · School of Mechanical Engineering, Tianjin University, China. +1
Physics-Informed Neural Networks (PINNs) frequently fail on stiff or advection-dominated PDEs, and two recent accounts offer competing remedies: switching from FP32 to FP64 to repair an L-BFGS stopping artifact, or replacing the MLP with a state-space-model (SSM) backbone plus sub-sequence alignment to counter architectural simplicity bias. We test both under matched, seed-paired controls in a pre-registered 144-run study spanning convection, reaction, and wave, plus an independent 85-run convection/wave study; success is relative ℓ2 error below 0.05. The two remedies act on disjoint regime-and-seed slices: neither substitutes for the other. On hard convection (β=50), alignment recovers 2/5 seeds in FP32 and 3/5 in FP64, where the unaligned SSM succeeds on 0/5 seeds at either precision and the vanilla MLP moves only from 0/5 to 1/5 across the precision switch---the recoveries trace to the alignment objective, not the backbone. On reaction the backbone alone already succeeds on 3/5--4/5 seeds, so each remedy covers a regime the other does not. Responses are also seed-specific: the same precision switch flips individual seeds in opposite directions and, on wave, lowers median error with no statistically significant success gain. Tightening the inner L-BFGS tolerance in an independent repeated-step runner likewise lowers median error at a large runtime cost, with success counts unchanged. Precision, stopping, backbone, and alignment must therefore be evaluated jointly and reported per seed.
Physics-informed neural networks (PINNs) have become a powerful framework for solving PDEs by embedding physical laws into differentiable objectives. Despite their advances, training PINNs remains fragile: recent conflict-averse optimization schemes alleviate gradient interference between residual and boundary losses, but we show that their effectiveness deteriorates as model capacity increases. In this paper, we identify a capacity-induced failure mode, where overparameterized networks undergo functional modularity, self-partitioning into task-exclusive modules that suppress cross-objective interaction and hinder convergence toward Pareto-stationary points. To address this issue, we propose a novel framework, Modular-Sparsity Synchronization (ModSync), which integrates structural optimization into conflict-averse training by penalizing task-exclusive connections while preserving interaction-promoting pathways. Extensive experiments across diverse PDE benchmarks demonstrate that ModSync consistently prevents capacity-driven failures, sustains robust cross-objective coupling, and achieves state-of-the-art accuracy. Codes are available at https://github.com/heejokong/ModSync.
Heejo Kong, Beomchul Park, Sung-Jin Kim +1
Department of Brain and Cognitive Engineering, Korea University, Seoul, Republic of Korea · Department of Artificial Intelligence, Korea University, Seoul, Republic of Korea