Pseudo-differential-enhanced physics-informed neural networks

Andrew Gracyk

cs.LGFeb 16, 2026

Pseudo-differential-enhanced physics-informed neural networks

Authors: Andrew Gracyk

Organizations: Department of Mathematics, Purdue University, West Lafayette, IN 47907, United States

Abstract

We present pseudo-differential enhanced physics-informed neural networks (PINNs), an extension of gradient enhancement but in Fourier space. Gradient enhancement of PINNs dictates that the PDE residual is taken to a higher differential order than prescribed by the PDE, added to the objective as an augmented term in order to improve training and overall learning fidelity. We propose the same procedure after application via Fourier transforms, since differentiating in Fourier space is multiplication with the Fourier wavenumber under suitable decay. Our methods are fast and efficient. Our methods oftentimes achieve superior PINN versus numerical error in fewer training iterations, potentially pair well with few/fixed samples in collocation, and can achieve breakthrough rather than gradual effects on the error. Moreover, our methods are suitable for fractional derivatives. We establish that our methods, due to the dynamical effects, improve spectral eigenvalue decay of the neural tangent kernel (NTK), and so our methods contribute towards the learning of high frequencies in early training, mitigating the effects of frequency bias up to the polynomial order and possibly greater with smooth activations. In particular, our primary contribution can be characterized as gradient enhancement affects spectral bias, and we specialize to Fourier space, although our theory does not prove the ambient case exactly due to a result with the Plancherel theorem on the descent paths. Our methods accommodate advanced techniques in PINNs, such as Fourier feature embeddings. A pitfall of discrete Fourier transforms via the Fast Fourier Transform (FFT) is mesh subjugation, and so we demonstrate compatibility of our methods for greater mesh flexibility and invariance on alternative Euclidean and non-Euclidean domains via Monte Carlo methods and otherwise, although possessing trade-offs of their own.

Figures & tables

Figure 1 . We plot (left) our Fourier enhanced PINN solution pointwise error on discretizations on three instances of retraining versus (right) a vanilla PINN on a log scale on two indices corresponding to $t=0.25,0.75$ on the Allen-Cahn equation. Lower is better. In particular, we demonstrate Fourier enhanced PINNS (with FFTs) can achieve breakthrough loss effects rather than gradual loss descent effects.

Figure 2 . We demonstrate on our Navier-Stokes experiment with severe Fourier enhanced loss tuned coefficient that high frequencies are learned faster. We train a Navier-Stokes PINN with (left) and without (right) Fourier enhancement with coefficient $0.5\times\lambda_{\text{physics}}$ on a vanilla MLP with Fourier feature embedding and the SOAP optimizer for $\sim 500$ epochs. High error power at high frequency corresponds to inability to learn high frequency. Thus, our methods mitigate spectral bias since lower is better. We remark that this figure also demonstrates lower frequencies are less affected (although this is not true in general; see Figure 12 ). We also remark this example is just a slice at a specific iteration, and is not characteristic of spectral bias across the training process (we refer to Figures 3 , 16 , 17 , 18 , 20 ).

Panel of Figure 3 — Figure 3 . We plot the spectral bias maximum error power according to Algorithm 3 and high error power for the KdV equation across a Fourier enhanced PINN with FFTs and a baseline PINN across 5,000 training iterations. We use learning rate $1\text{e}{-3}$ ; (top) batch size of 600 and (bottom) a very high batch size of 1,500; and use $\lambda_{\text{Fourier}}=0.02$ . We use an enhanced MLP with $\sigma=3.0$ and initial condition $u(x,0)=-0.65\cos(\pi x)$ . We use a larger batch size in both, which is notable because with high probability the collocation points will encompass the domain (especially in 1D, which is what we have here) with high coverage, i.e. the largest approximate open intervals possible that do not contain collocation points will have small Lebesgue measure. Moreover, the FFTs will not seriously cover points "that are missed" due to such an existence of maximally-taken small-measured open intervals. This conceptually is important to several of our figures, although less applicable to our 2D Navier-Stokes.

Figure 4 . We plot maximum frequency spectral bias learning with a vanilla (non-Fourier enhanced) PINN across 3 instances of training with: (1) a moderate batch size of 300; (2) a large batch size of 2,000. Since the spectral error power is mostly equal for both, the spectral bias is largely invariant of batch size assuming it is sufficient large. Therefore, we can conclude our spectral bias results are mostly unaffected by batch size (this is notable since FFTs can possibly encompass a large batch since it is grid-based). We also refer to Figure 19 for a similar effect with FFTs.

Scenario	Method [ $\downarrow$ ]
	Total log error power [ $\downarrow$ ]	Average log error power [ $\downarrow$ ]	Frequency 50 [ $\uparrow$ ]	Frequency 90 [ $\uparrow$ ]	Ratio 0.0-0.1 [ $\downarrow$ ]	Ratio 0.1-0.25 [ $\uparrow$ ]	Ratio 0.25-0.5 [ $\uparrow$ ]
Allen-Cahn FE	${1.812}\mathrm{e}{+1}$	${1.201}\mathrm{e}{0}$	${0.000}\mathrm{e}{0}$	${7.092}\mathrm{e}{-3}$	${9.999}\mathrm{e}{-1}$	${8.789}\mathrm{e}{-5}$	${1.094}\mathrm{e}{-5}$
Allen-Cahn vanilla	${1.903}\mathrm{e}{+1}$	${1.384}\mathrm{e}{0}$	${0.000}\mathrm{e}{0}$	${3.546}\mathrm{e}{-3}$	${9.999}\mathrm{e}{-1}$	${6.346}\mathrm{e}{-5}$	${8.788}\mathrm{e}{-6}$
Burger’s FE	${1.557}\mathrm{e}{+1}$	${5.843}\mathrm{e}{-1}$	${7.092}\mathrm{e}{-3}$	${1.064}\mathrm{e}{-2}$	${9.999}\mathrm{e}{-1}$	${1.051}\mathrm{e}{-4}$	${1.696}\mathrm{e}{-5}$
Burger’s vanilla	${1.596}\mathrm{e}{+1}$	${6.753}\mathrm{e}{-1}$	${0.000}\mathrm{e}{0}$	${1.064}\mathrm{e}{-2}$	${9.999}\mathrm{e}{-1}$	${8.644}\mathrm{e}{-5}$	${1.404}\mathrm{e}{-5}$
KdV FE	${1.489}\mathrm{e}{+1}$	${8.737}\mathrm{e}{-1}$	${2.837}\mathrm{e}{-2}$	${3.901}\mathrm{e}{-2}$	${9.989}\mathrm{e}{-1}$	${9.842}\mathrm{e}{-4}$	${1.423}\mathrm{e}{-4}$

Table 1 . We list various statistics identifying the learning power of frequency bias corresponding to the data as in Algorithm 3 pertaining to radial power spectral density. All of our results are done with severe early stopping and high

\mathcal{L}_{\text{Fourier enhanced}}

coefficient to overemphasize the spectral bias learning effects. The frequency % refers to a percentile (high is better since high frequencies learned is desirable). The ratio quantifies the amount of frequency that exists in that quantile range (high ratio in high frequency is desirable).

Figure 5 . We plot reserved memory usage on a Fourier gradient enhanced versus a traditional gradient enhanced PINN on Burger’s equation with a triangular domain at a single iteration (it is about constant per epoch).

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

\text{FFT}[f]=\widehat{f}(\xi_{1},\dots,\xi_{j})=\frac{1}{\prod_{k}N_{k}}\sum_{x_{1}=0}^{N_{1}-1}\dots\sum_{x_{j}=0}^{N_{j}-1}f(x_{1},\dots,x_{j})e^{-2\pi i\sum_{k}x_{k}\xi_{k}/s_{k}},\ \ \ \text{for}\ f=u,\dot{u},\ldots\;

Appendix

Algorithm 1 Grid-based enhanced loss

\mathcal{F}_{\text{MC}}[f]\leftarrow\frac{|\Omega|}{N}\Phi f\approx\int_{\mathbb{R}^{n}}f(x,t)e^{-2\pi i\langle\xi,x\rangle}dx\ \ \ \text{for}\ f=u,\dot{u},\ldots\ \text{(not equispaced)}\;

Appendix

Algorithm 2 Mesh-invariant Monte-Carlo enhanced Loss (see 5.3 , 6 for limiting discussions)

Panel of Figure 6 — Figure 6 . We plot relative $L^{2}$ error discretized on an instance of the Burger’s equation over $8,000$ training iterations on a triangular domain (so FFTs are not used but rather Monte Carlo techniques are) as in Figure 10 but here $(x,t)$ are fixed throughout training. This includes fixed $(x,t)$ for both the physics and Fourier losses (the domain is irregular). The error is evaluated along a mesh, thus along points not necessarily seen in training. We implement fixed coefficients with $\lambda_{\text{physics}}=1,\lambda_{\text{boundary}}=10$ , and take $\lambda_{\text{enhanced}}=0.2,0.5,0.15$ (from top to bottom) as the Fourier enhanced loss. We use the Adam optimizer with $\gamma=1\text{e}{-3}$ learning rate, and the Fourier feature embedding with $\sigma=1.0$ in a vanilla MLP. We use a quantile loss with $0.99$ for the enhanced loss. We choose $N=100$ (top); $N=300$ (middle); $N=1,000$ (bottom) collocation points. Our pseudo-differential weight term here takes the form $P(\xi)=1+(2\pi i\xi)$ for top-middle and $P(\xi)=1+(2\pi i\xi)+(2\pi i\xi)^{2}$ for bottom. We remark we ensured $(x,t)$ had gradients by redefining them each epoch and setting the seed.

Figure 7 . We plot training time versus iteration on the Allen-Cahn equation over $500$ training epochs for varying collocation number $N$ with FFTs. This figure pertains to second order differentiation of the gradient enhancement for both ours and baseline, thus our methods can handle gradient enhancement of high orders rather easily.

Figure 8 . We measure the error power $||\widehat{u-u_{\theta}}||_{L^{2}}^{2}$ across frequency wavenumber $\xi$ for (left) a Fourier enhanced PINN and (right) a vanilla PINN after 2,000 epochs on KdV data. The figures are similar, but there is a twofold order of mangnitude drop-off on the left for high frequencies, thus the spectral enhanced PINN learns higher frequencies early in training.

Figure 9 . We plot relative $L^{2}$ error discretized on an instance of the KdV equation over $70,000$ training iterations. We implement fixed coefficients with $\lambda_{\text{physics}}=1,\lambda_{\text{boundary}}=10$ , and take $\lambda_{\text{enhanced}}=0.015$ as the Fourier enhanced loss. We use the Adam optimizer with $\gamma=1\text{e}{-3}$ learning rate, and the Fourier feature embedding with $\sigma=3.0$ in a vanilla MLP. We use a quantile loss with $0.9$ for the enhanced loss. We choose $N=4,000$ collocation points for the physics loss, and so we emphasize a large batch size. Our pseudo-differential weight term here takes the form $P(\xi)=1+(2\pi i\xi)+(2\pi i\xi)^{2}$ . We also remark we found a low coefficient for $\mathcal{L}_{\text{enhanced}}$ to work better here.

Figure 10 . We plot images of pointwise error discretized on an instance of the Burger’s equation using advanced techniques over $10,000$ training iterations over an irregular domain with a batch size of $200$ . Thus, Monte Carlo approximations were used for the Fourier loss, demonstrating our methods can handle irregular domains. (0) represents the true (numerical) solution. (1a) and (1b) are the Fourier-enhanced PINN at 5,000 and 10,000 iterations, and (2a) and (2c) are a vanilla PINN at 5,000 and 10,000 iterations. As a remark, note that the scales are different. Our method performs better primarily except where the Burger’s solution has a rift.

Figure 11 . We plot absolute pointwise error of the Burger’s equation over $12,000$ training iterations over an irregular domain with a batch size of $500$ . (1) in the Fourier enhanced PINN solution with coefficient $\lambda_{\text{enhanced}}=0.025$ and (2) is the vanilla PINN. As we can see, errors are mostly consistent across retraining, and the Fourier enhanced PINN outperforms. We remark that (1) and (2) look similar but the scales are different.

Figure 12 . We plot 6 spectral bias learning scenarios corresponding to Algorithm 3 on a solution to the Burger’s equation over $3,000$ training iterations on an irregular domain with a batch size of 150 with Monte Carlo techniques. The green is the Fourier enhanced spectral learning, and the orange is the baseline PINN. The top is the true PDE solution. We plot error power versus frequency. Lower is better in general. The rightmost portions of the plot demonstrate learning at the highest frequencies. In 4 out of 6 cases, our proposed methods learn the highest frequencies better. The bulk area, or the uppermost portions of the lines, correspond to high but not highest frequencies. In 6 out of 6, our methods are at least as good for all high frequencies, and for 3 out of 6, our methods outperform in high frequencies. In general, we conclude our methods appeal to spectral bias.

Figure 13 . (0) We plot the numerical solution in our Korteweg-De Vries (KdV) experiment; (1a) we plot the PINN solution with a Fourier enhanced loss; (1b) we plot the absolute pointwise error; (1c) we plot the non-truncated Fourier residual with spectral weights in a log plot; (1d) we plot the truncated at Fourier mode 10 residual plot with spectral weights in a log plot. We remark this figure pairs with Figure 9 but in a new instance of training with fewer iterations ( $\sim 35,000$ epochs).

Figure 14 . We plot the first 4,000 training epochs in 1,000 epoch chunks for (1) a Fourier enhanced PINN and (2) a baseline PINN. We use a high learning rate $1\text{e}{-3}$ . As we can see, the loss is highly stochastic in (2); however, the more interesting part is there is a clear learning of high frequency in the first 1,000 epoch chunk.

Figure 15 . We plot four instances of retraining after a mere 2,000 epochs of the KdV experiment on (left two) a Fourier enhanced PINN; and (right two) a vanilla PINN; and (top) numerical solution. We use Adam optimizer with $1\text{e}{-3}$ learning rate. We set $\lambda_{\text{physics}}=1$ , $\lambda_{\text{boundary}}=10$ , $\lambda_{\text{Fourier}}=0.01$ . We use a vanilla MLP with $\sigma=5.0$ in the Fourier feature embedding. We highlight it is possible this figure demonstrates our methods affect frequency bias, as the left demonstrates higher frequencies learned earlier in training.

Panel of Figure 16 — Figure 16 . We plot the spectral bias maximum error power and high error power according to Algorithm 3 for the KdV equation across a Fourier enhanced PINN with FFTs and a baseline PINN across 3,000 training iterations for 3 instances of training. We use learning rate $1\text{e}{-3}$ , batch size of 2,500 (which is very high batch size), $\lambda_{\text{Fourier}}=0.02$ , and 10 Fourier modes. We use an enhanced MLP with $\sigma=3.0$ and initial condition $u(x,0)=-0.65\cos(\pi x)$ . Lower is better. We primarily use a very high batch size because the FFT consumes more data points, so we want to conclude that high sampling does not cause the spectral bias mitigation we see with the Fourier enhanced loss.

Panel of Figure 17 — Figure 17 . We plot the same as Figure 16 but we use FFT grid sizes of 31-61 rather than 151-301 in order to examine effects of FFT collocation amount on spectral bias. Our batch size is 1,000. We remark that this figure also complements Figure 19 . We compare a batch size of 2,500 to a batch size of 1,000 also for a vanilla PINN by comparing to 16 . Therefore, for a similar batch size comparison, we can compare the Fourier enhanced PINN of this figure to that of the vanilla PINN of 16 , and clearly the Fourier enhanced PINN outperforms.

Panel of Figure 18 — Figure 18 . We plot the spectral bias maximum error power and high error power according to Algorithm 3 for the KdV equation across a Fourier enhanced PINN with Monte carlo methods and a baseline PINN across 3,000 training iterations for 3 instances of training. We use a batch size of 1,000. The Monte Carlo collocation points are the exact same as the physics points here. We use the same scenario as Figure 16 .

Figure 19 . We plot the spectral bias maximum error power and high error power according to Algorithm 3 for the KdV equation across two FFT scenarios of a Fourier enhanced PINN and across 3 instances of training with: (1) a small FFT spatial grid between 31,61 in dimension each iteration; (2) a large FFT spatial grid between 151, 301 in dimension each iteration. Our hyperparameters are consistent with Figure 16 . As we can see, the FFT collocation sampling has an indiscernible effect on training and spectral bias.

Panel of Figure 20 — Figure 20 . We plot the spectral bias maximum error power and high error power according to Algorithm 2 on 2 instances of training Navier-Stokes data corresponding to the scenario of Figure 21 We use Fourier feature embedding of $\sigma=6.0$ ( $2\pi$ omitted), the SOAP optimizer, learning rate of $9.5\text{e}-4$ , and $\lambda_{\text{Fourier}}=1$ . Generally, we found no change in results with the Adam optimizer. Here, we use Monte Carlo techniques with the same batch as the physics loss with batch size 1,000. It is crucial to note the two instances are near identical, thus this example is highly consistent upon retraining. We restrict time domain to have Lebesgue measure $\mu([t_{0},T])=0.5$ in this example (compare with Figure 22 and section E.4 ).

Figure 21 . We provide a PINN scenario on Navier-Stokes data corresponding to Figure 20 . We plot (1) the true solution examined at $t>t_{0}$ ; (2) the PINN solution at $t>t_{0}$ after $\sim 400$ training iterations (we remark this is still considered early training, so we examine the effects of spectral bias learned more quickly); (3) the absolute pointwise error

Figure 22 . We provide a PINN scenario on Navier-Stokes data under (1) our PINN solution, which is Fourier enhanced (with FFTs), (2) a FFT numerical solver, and (3) the absolute pointwise error. For the PINN, we use a Fourier feature embedding with $\sigma=3.0$ ( $2\pi$ omitted), the SOAP optimizer with learning rate $1\text{e}{-3}$ and lowered to $5\text{e}{-4}$ in late training, fixed weights of $\lambda_{\text{physics}}=1$ , $\lambda_{\text{boundary}}=10$ , $\lambda_{\text{Fourier}}=0.025\times\lambda_{\text{physics}}$ for some of training and some of training with the grad norm procedure of [ Wang et al., 2023 ] . We use a batch size of 3,000, and $T_{\text{max}}-T_{\text{min}}=1$ . We use a neural network width of 400 and a depth of 4 with a vanilla architecture with $\sin(\cdot)$ activation, and train for approximately $10,000$ epochs. This data was used for Table 1 .

Figure 23 . We plot our Allen-Cahn experiment with 35 collocation point sampling and FFTs in eight 30,000 training iterations batch increments. We demonstrate the $L^{2}$ error effect is a drop-off rather than gradual.

Explore similar work

Aug 6, 2026cs.LG

Alternating Levenberg-Marquardt Training of Physics-Informed Neural Networks with Fourier-Enhanced Features

Physics-informed neural networks (PINNs) often fail to accurately resolve partial differential equations (PDEs) with high-frequency or multi-scale solutions, as well as strongly nonlinear problems. Two factors underlie this difficulty: spectral bias, the tendency of neural networks to underfit high-frequency features; and representation-coefficient coupling, the entanglement of representation learning and coefficient fitting within a single nonconvex optimization objective. In this work, we propose the Fourier-enhanced alternating Levenberg--Marquardt PINN (FALM-PINN), an optimization framework that decouples representation learning from coefficient fitting. The upper-level problem learns a Fourier-enhanced basis that enriches the latent space with high-frequency components, while the lower-level problem resolves the coupling by fitting the projection coefficients on this basis, solving a nonlinear least-squares problem with the Levenberg--Marquardt algorithm. The framework applies to general nonlinear and coupled PDE systems, and reduces to a single-step convex optimization problem for linear PDEs. We prove global convergence of the alternating training scheme in both cases. Numerical examples on multiple challenging high-frequency and nonlinear PDEs show that FALM-PINN achieves relative

L^2

errors up to two orders of magnitude lower than state-of-the-art baselines.

Yulun Wu, Matthieu Barreau, Miguel Aguiar +1

Division of Decision and Control Systems, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, Stockholm, 100 44, Sweden

Parametric Physics-Informed Neural Network Partial Differential Equations Newton Fourier Neural Network

Ask PDF

May 22, 2026cs.LG

Fourier Feature Pyramids for Physics-Informed Neural Networks

We present an improved neural field architecture for solving partial differential equations (PDEs). Current physics-informed neural networks (PINNs) provide a flexible framework for solving PDEs, but they struggle to achieve highly accurate solutions and require computation that scales poorly with parameter count. Our model, which we call beignet (Bandlimited Embedding with Interpolated Grid Network), replaces the random Fourier feature embedding used by existing PINN models with a trainable multi-resolution Fourier feature pyramid. To query beignet at a continuous coordinate, we use Fourier interpolation at each level of the pyramid to return features at the input coordinate, and then decode this vector with a fully-connected neural network trunk. Our model provides multiple benefits: 1) Spatial derivatives can be computed efficiently by using the chain rule to compose derivatives of the neural network computed with automatic differentiation with derivatives of the feature grid computed spectrally by the Fast Fourier transform (FFT). 2) beignet can achieve higher accuracy in a compute-efficient manner by scaling the parameter count of this Fourier feature pyramid, instead of the less-efficient strategy of scaling the neural network architecture. 3) beignet can directly control the representation bandlimit, resulting in more stable optimization for difficult PDEs. We demonstrate that beignet finds significantly more accurate solutions on PDE benchmarks using fewer parameters than state-of-the-art PINN methods. We further evaluate beignet on the self-similar inviscid Burgers blowup problem and show that it can minimize residuals to near machine precision using Adam, an accuracy regime previously attained only by using computationally expensive higher-order optimizers.

Brandon Zhao, Yixuan Wang, Jonathan T. Barron +3

Department of Computing and Mathematical Sciences, California Institute of Technology · Google DeepMind · Cahill Center for Astronomy and Astrophysics, California Institute of Technology

Parametric Physics-Informed Neural Network Partial Differential Equations Fourier Neural Network

Ask PDF

May 19, 2026cs.LG

From Simple to Complex: Curriculum-Guided Physics-Informed Neural Networks via Gaussian Mixture Models

Physics-informed neural networks (PINNs) offer a mesh-free framework for solving partial differential equations (PDEs), yet training often suffers from gradient pathologies, spectral bias, and poor convergence, especially for problems with strong nonlinearity, sharp gradients, or multiscale features. We propose the Curriculum-Guided Gaussian Mixture Physics-Informed Neural Network (CGMPINN), which integrates Gaussian mixture modeling with dynamic curriculum learning. Specifically, a GMM is periodically fitted to the PDE residual distribution to quantify spatially varying learning difficulty. A smooth curriculum schedule progressively shifts training focus from easy to harder regions, while precision-based variance modulation suppresses unreliable clusters during early optimization. This dual curriculum is governed by a shared curriculum parameter and can be combined with self-adaptive loss balancing. We further establish theoretical guarantees, including sublinear convergence of the gradient norm for the induced time-varying loss, uniform equivalence between the curriculum-weighted and standard PDE losses, and a generalization bound with an explicit weighting-induced bias characterization. Experiments on six benchmark PDEs spanning elliptic, parabolic, hyperbolic, advection-dominated, and nonlinear reaction-diffusion types show that CGMPINN consistently achieves the lowest relative

L_2

and maximum absolute errors among all compared methods, reducing relative

L_2

error by up to 97.8% over the standard PINN at comparable cost. Our code is publicly available at https://github.com/Mathematics-Yang/CGMPINN.

Jianan Yang, Yiran Wang, Shuai Li +3

Xi’an Jiaotong University, Xi’an, China · College of International Business, Zhejiang Yuexiu University, Shaoxing, China · Shaoxing Key Laboratory of Intelligent Monitoring and Prevention of Smart Society, Shaoxing, China +1

Parametric Physics-Informed Neural Network Neural Network Gaussian Mixture Models Mixtures

Ask PDF