cs.LGFeb 16, 2026
SavePseudo-differential-enhanced physics-informed neural networks
Organizations: Department of Mathematics, Purdue University, West Lafayette, IN 47907, United States
Abstract
We present pseudo-differential enhanced physics-informed neural networks (PINNs), an extension of gradient enhancement but in Fourier space. Gradient enhancement of PINNs dictates that the PDE residual is taken to a higher differential order than prescribed by the PDE, added to the objective as an augmented term in order to improve training and overall learning fidelity. We propose the same procedure after application via Fourier transforms, since differentiating in Fourier space is multiplication with the Fourier wavenumber under suitable decay. Our methods are fast and efficient. Our methods oftentimes achieve superior PINN versus numerical error in fewer training iterations, potentially pair well with few/fixed samples in collocation, and can achieve breakthrough rather than gradual effects on the error. Moreover, our methods are suitable for fractional derivatives. We establish that our methods, due to the dynamical effects, improve spectral eigenvalue decay of the neural tangent kernel (NTK), and so our methods contribute towards the learning of high frequencies in early training, mitigating the effects of frequency bias up to the polynomial order and possibly greater with smooth activations. In particular, our primary contribution can be characterized as gradient enhancement affects spectral bias, and we specialize to Fourier space, although our theory does not prove the ambient case exactly due to a result with the Plancherel theorem on the descent paths. Our methods accommodate advanced techniques in PINNs, such as Fourier feature embeddings. A pitfall of discrete Fourier transforms via the Fast Fourier Transform (FFT) is mesh subjugation, and so we demonstrate compatibility of our methods for greater mesh flexibility and invariance on alternative Euclidean and non-Euclidean domains via Monte Carlo methods and otherwise, although possessing trade-offs of their own.
Figures & tables
Figure 1 . We plot (left) our Fourier enhanced PINN solution pointwise error on discretizations on three instances of retraining versus (right) a vanilla PINN on a log scale on two indices corresponding to on the Allen-Cahn equation. Lower is better. In particular, we demonstrate Fourier enhanced PINNS (with FFTs) can achieve breakthrough loss effects rather than gradual loss descent effects.
Figure 2 . We demonstrate on our Navier-Stokes experiment with severe Fourier enhanced loss tuned coefficient that high frequencies are learned faster. We train a Navier-Stokes PINN with (left) and without (right) Fourier enhancement with coefficient on a vanilla MLP with Fourier feature embedding and the SOAP optimizer for epochs. High error power at high frequency corresponds to inability to learn high frequency. Thus, our methods mitigate spectral bias since lower is better. We remark that this figure also demonstrates lower frequencies are less affected (although this is not true in general; see Figure 12 ). We also remark this example is just a slice at a specific iteration, and is not characteristic of spectral bias across the training process (we refer to Figures 3 , 16 , 17 , 18 , 20 ).
Figure 3 . We plot the spectral bias maximum error power according to Algorithm 3 and high error power for the KdV equation across a Fourier enhanced PINN with FFTs and a baseline PINN across 5,000 training iterations. We use learning rate ; (top) batch size of 600 and (bottom) a very high batch size of 1,500; and use . We use an enhanced MLP with and initial condition . We use a larger batch size in both, which is notable because with high probability the collocation points will encompass the domain (especially in 1D, which is what we have here) with high coverage, i.e. the largest approximate open intervals possible that do not contain collocation points will have small Lebesgue measure. Moreover, the FFTs will not seriously cover points "that are missed" due to such an existence of maximally-taken small-measured open intervals. This conceptually is important to several of our figures, although less applicable to our 2D Navier-Stokes.
Figure 4 . We plot maximum frequency spectral bias learning with a vanilla (non-Fourier enhanced) PINN across 3 instances of training with: (1) a moderate batch size of 300; (2) a large batch size of 2,000. Since the spectral error power is mostly equal for both, the spectral bias is largely invariant of batch size assuming it is sufficient large. Therefore, we can conclude our spectral bias results are mostly unaffected by batch size (this is notable since FFTs can possibly encompass a large batch since it is grid-based). We also refer to Figure 19 for a similar effect with FFTs.
| Scenario | Method [ ] | ||||||
|---|---|---|---|---|---|---|---|
| Total log error power [ ] | Average log error power [ ] | Frequency 50 [ ] | Frequency 90 [ ] | Ratio 0.0-0.1 [ ] | Ratio 0.1-0.25 [ ] | Ratio 0.25-0.5 [ ] | |
| Allen-Cahn FE | |||||||
| Allen-Cahn vanilla | |||||||
| Burger’s FE | |||||||
| Burger’s vanilla | |||||||
| KdV FE | |||||||
Table 1 . We list various statistics identifying the learning power of frequency bias corresponding to the data as in Algorithm 3 pertaining to radial power spectral density. All of our results are done with severe early stopping and high coefficient to overemphasize the spectral bias learning effects. The frequency % refers to a percentile (high is better since high frequencies learned is desirable). The ratio quantifies the amount of frequency that exists in that quantile range (high ratio in high frequency is desirable).
Figure 5 . We plot reserved memory usage on a Fourier gradient enhanced versus a traditional gradient enhanced PINN on Burger’s equation with a triangular domain at a single iteration (it is about constant per epoch).
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Algorithm 1 Grid-based enhanced loss
Algorithm 2 Mesh-invariant Monte-Carlo enhanced Loss (see 5.3 , 6 for limiting discussions)
Figure 6 . We plot relative error discretized on an instance of the Burger’s equation over training iterations on a triangular domain (so FFTs are not used but rather Monte Carlo techniques are) as in Figure 10 but here are fixed throughout training. This includes fixed for both the physics and Fourier losses (the domain is irregular). The error is evaluated along a mesh, thus along points not necessarily seen in training. We implement fixed coefficients with , and take (from top to bottom) as the Fourier enhanced loss. We use the Adam optimizer with learning rate, and the Fourier feature embedding with in a vanilla MLP. We use a quantile loss with for the enhanced loss. We choose (top); (middle); (bottom) collocation points. Our pseudo-differential weight term here takes the form for top-middle and for bottom. We remark we ensured had gradients by redefining them each epoch and setting the seed.
Figure 7 . We plot training time versus iteration on the Allen-Cahn equation over training epochs for varying collocation number with FFTs. This figure pertains to second order differentiation of the gradient enhancement for both ours and baseline, thus our methods can handle gradient enhancement of high orders rather easily.
Figure 8 . We measure the error power across frequency wavenumber for (left) a Fourier enhanced PINN and (right) a vanilla PINN after 2,000 epochs on KdV data. The figures are similar, but there is a twofold order of mangnitude drop-off on the left for high frequencies, thus the spectral enhanced PINN learns higher frequencies early in training.
Figure 9 . We plot relative error discretized on an instance of the KdV equation over training iterations. We implement fixed coefficients with , and take as the Fourier enhanced loss. We use the Adam optimizer with learning rate, and the Fourier feature embedding with in a vanilla MLP. We use a quantile loss with for the enhanced loss. We choose collocation points for the physics loss, and so we emphasize a large batch size. Our pseudo-differential weight term here takes the form . We also remark we found a low coefficient for to work better here.
Figure 10 . We plot images of pointwise error discretized on an instance of the Burger’s equation using advanced techniques over training iterations over an irregular domain with a batch size of . Thus, Monte Carlo approximations were used for the Fourier loss, demonstrating our methods can handle irregular domains. (0) represents the true (numerical) solution. (1a) and (1b) are the Fourier-enhanced PINN at 5,000 and 10,000 iterations, and (2a) and (2c) are a vanilla PINN at 5,000 and 10,000 iterations. As a remark, note that the scales are different. Our method performs better primarily except where the Burger’s solution has a rift.
Figure 11 . We plot absolute pointwise error of the Burger’s equation over training iterations over an irregular domain with a batch size of . (1) in the Fourier enhanced PINN solution with coefficient and (2) is the vanilla PINN. As we can see, errors are mostly consistent across retraining, and the Fourier enhanced PINN outperforms. We remark that (1) and (2) look similar but the scales are different.
Figure 12 . We plot 6 spectral bias learning scenarios corresponding to Algorithm 3 on a solution to the Burger’s equation over training iterations on an irregular domain with a batch size of 150 with Monte Carlo techniques. The green is the Fourier enhanced spectral learning, and the orange is the baseline PINN. The top is the true PDE solution. We plot error power versus frequency. Lower is better in general. The rightmost portions of the plot demonstrate learning at the highest frequencies. In 4 out of 6 cases, our proposed methods learn the highest frequencies better. The bulk area, or the uppermost portions of the lines, correspond to high but not highest frequencies. In 6 out of 6, our methods are at least as good for all high frequencies, and for 3 out of 6, our methods outperform in high frequencies. In general, we conclude our methods appeal to spectral bias.
Figure 13 . (0) We plot the numerical solution in our Korteweg-De Vries (KdV) experiment; (1a) we plot the PINN solution with a Fourier enhanced loss; (1b) we plot the absolute pointwise error; (1c) we plot the non-truncated Fourier residual with spectral weights in a log plot; (1d) we plot the truncated at Fourier mode 10 residual plot with spectral weights in a log plot. We remark this figure pairs with Figure 9 but in a new instance of training with fewer iterations ( epochs).
Figure 14 . We plot the first 4,000 training epochs in 1,000 epoch chunks for (1) a Fourier enhanced PINN and (2) a baseline PINN. We use a high learning rate . As we can see, the loss is highly stochastic in (2); however, the more interesting part is there is a clear learning of high frequency in the first 1,000 epoch chunk.
Figure 15 . We plot four instances of retraining after a mere 2,000 epochs of the KdV experiment on (left two) a Fourier enhanced PINN; and (right two) a vanilla PINN; and (top) numerical solution. We use Adam optimizer with learning rate. We set , , . We use a vanilla MLP with in the Fourier feature embedding. We highlight it is possible this figure demonstrates our methods affect frequency bias, as the left demonstrates higher frequencies learned earlier in training.
Figure 16 . We plot the spectral bias maximum error power and high error power according to Algorithm 3 for the KdV equation across a Fourier enhanced PINN with FFTs and a baseline PINN across 3,000 training iterations for 3 instances of training. We use learning rate , batch size of 2,500 (which is very high batch size), , and 10 Fourier modes. We use an enhanced MLP with and initial condition . Lower is better. We primarily use a very high batch size because the FFT consumes more data points, so we want to conclude that high sampling does not cause the spectral bias mitigation we see with the Fourier enhanced loss.
Figure 17 . We plot the same as Figure 16 but we use FFT grid sizes of 31-61 rather than 151-301 in order to examine effects of FFT collocation amount on spectral bias. Our batch size is 1,000. We remark that this figure also complements Figure 19 . We compare a batch size of 2,500 to a batch size of 1,000 also for a vanilla PINN by comparing to 16 . Therefore, for a similar batch size comparison, we can compare the Fourier enhanced PINN of this figure to that of the vanilla PINN of 16 , and clearly the Fourier enhanced PINN outperforms.
Figure 18 . We plot the spectral bias maximum error power and high error power according to Algorithm 3 for the KdV equation across a Fourier enhanced PINN with Monte carlo methods and a baseline PINN across 3,000 training iterations for 3 instances of training. We use a batch size of 1,000. The Monte Carlo collocation points are the exact same as the physics points here. We use the same scenario as Figure 16 .
Figure 19 . We plot the spectral bias maximum error power and high error power according to Algorithm 3 for the KdV equation across two FFT scenarios of a Fourier enhanced PINN and across 3 instances of training with: (1) a small FFT spatial grid between 31,61 in dimension each iteration; (2) a large FFT spatial grid between 151, 301 in dimension each iteration. Our hyperparameters are consistent with Figure 16 . As we can see, the FFT collocation sampling has an indiscernible effect on training and spectral bias.
Figure 20 . We plot the spectral bias maximum error power and high error power according to Algorithm 2 on 2 instances of training Navier-Stokes data corresponding to the scenario of Figure 21 We use Fourier feature embedding of ( omitted), the SOAP optimizer, learning rate of , and . Generally, we found no change in results with the Adam optimizer. Here, we use Monte Carlo techniques with the same batch as the physics loss with batch size 1,000. It is crucial to note the two instances are near identical, thus this example is highly consistent upon retraining. We restrict time domain to have Lebesgue measure in this example (compare with Figure 22 and section E.4 ).
Figure 21 . We provide a PINN scenario on Navier-Stokes data corresponding to Figure 20 . We plot (1) the true solution examined at ; (2) the PINN solution at after training iterations (we remark this is still considered early training, so we examine the effects of spectral bias learned more quickly); (3) the absolute pointwise error
Figure 22 . We provide a PINN scenario on Navier-Stokes data under (1) our PINN solution, which is Fourier enhanced (with FFTs), (2) a FFT numerical solver, and (3) the absolute pointwise error. For the PINN, we use a Fourier feature embedding with ( omitted), the SOAP optimizer with learning rate and lowered to in late training, fixed weights of , , for some of training and some of training with the grad norm procedure of [ Wang et al., 2023 ] . We use a batch size of 3,000, and . We use a neural network width of 400 and a depth of 4 with a vanilla architecture with activation, and train for approximately epochs. This data was used for Table 1 .
Figure 23 . We plot our Allen-Cahn experiment with 35 collocation point sampling and FFTs in eight 30,000 training iterations batch increments. We demonstrate the error effect is a drop-off rather than gradual.