cs.LGFeb 22, 2026

Spectral bias in physics-informed and operator learning: Analysis and mitigation guidelines

Authors: Siavash Khodakarami, Vivek Oommen, Nazanin Ahmadi Daryakenari, Maxim Beekenkamp, George Em Karniadakis

Organizations: Division of Applied Mathematics, Brown University, Providence, RI 02912, USA · School of Engineering, Brown University, Providence, RI 02912, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Solving partial differential equations (PDEs) by neural networks as well as Kolmogorov-Arnold Networks (KANs), including physics-informed neural networks (PINNs), physics-informed KANs (PIKANs), and neural operators, are known to exhibit spectral bias, whereby low-frequency components of the solution are learned significantly faster than high-frequency modes. While spectral bias is often treated as an intrinsic representational limitation of neural architectures, its interaction with optimization dynamics and physics-based loss formulations remains poorly understood. In this work, we provide a systematic investigation of spectral bias in physics-informed and operator learning frameworks, with emphasis on the coupled roles of network architecture, activation functions, loss design, and optimization strategy. We quantify spectral bias through frequency-resolved error metrics, Barron-norm diagnostics, and higher-order statistical moments, enabling a unified analysis across elliptic, hyperbolic, and dispersive PDEs. Through diverse benchmark problems, including the Korteweg-de Vries, wave and steady-state diffusion-reaction equations, turbulent flow reconstruction, and earthquake dynamics, we demonstrate that spectral bias is not simply representational but fundamentally dynamical. In particular, second-order optimization methods substantially alter the spectral learning order, enabling earlier and more accurate recovery of high-frequency modes for all PDE types. For neural operators, we further show that spectral bias is dependent on the neural operator architecture and can also be effectively mitigated through spectral-aware loss formulations without increasing the inference cost.

Figures & tables

ArchitectureRel. L2L^{2} ErrorBarron Norm Rel. Errorlog⁡(eF,p=0)\log(e_{\mathcal{F},p=0})log⁡(eF,p=2)\log(e_{\mathcal{F},p=2})log⁡(eF,p=4)\log(e_{\mathcal{F},p=4})
Case 1: Discontinuous Function     
MLP–Tanh4.96×10−44.96\times 10^{-4}2.35×10−42.35\times 10^{-4}−3.72-3.721.311.316.386.38
MLP–SIREN4.68×10−4\mathbf{4.68\times 10^{-4}}6.48×10−4\mathbf{6.48\times 10^{-4}}−3.78\mathbf{-3.78}0.01\mathbf{0.01}4.78\mathbf{4.78}
cKAN3.67×10−23.67\times 10^{-2}1.62×10−21.62\times 10^{-2}0.010.015.075.0710.1710.17
Tanh-cKAN5.19×10−45.19\times 10^{-4}3.28×10−43.28\times 10^{-4}−3.68-3.68−0.04-0.044.864.86
Case 2: Smooth Multi-Frequency Function    
Table 1: Data-driven spectral benchmark errors. Comparison of neural network architectures for Case 1 (discontinuous function) and Case 2 (smooth multi-frequency function). Performance is evaluated using physical- and frequency-domain error metrics.
Figure 1: Case 1: Training-time spectral evolution for the discontinuous benchmark. Comparison of MLP–Tanh, MLP–SIREN, cKAN, and Tanh-cKAN on the piecewise-discontinuous target function. Columns correspond to different training epochs, illustrating the evolution of the predicted signal in physical space (top row of each block) and the corresponding Fourier amplitude spectrum (bottom row of each block). The ground-truth solution is shown in black, while network predictions are shown in red. The discontinuity induces broad-band high-frequency content in the frequency domain, highlighting differences in how each architecture and optimization strategy capture sharp transitions and recover high-frequency modes over training.
Figure 2: Case 2: Training-time spectral evolution for the smooth multi-frequency benchmark. Comparison of MLP–Tanh, MLP–SIREN, cKAN, and Tanh-cKAN on the multi-scale multi-frequency target function. Columns correspond to different training epochs to illustrate the evolution of the predicted signal in physical space (top row of each block) and the corresponding Fourier amplitude spectrum (bottom row of each block). The ground-truth solution is shown in black, while network predictions are shown in red. Although the target function is smooth, the presence of multiple well-separated frequency components makes this benchmark sensitive to spectral bias to reveal differences in how each architecture captures and balances low- and high-frequency modes during training.
Figure 3: KdV equation: PINN training history with different optimizers (Adam, L-BFGS, SOAP, and SS-Broyden) and activation functions (Tanh, and SIREN) for KdV equation. The sub-figure shows the zoomed-in training history for the SS-Broyden optimizer.
Figure 4: KdV equation: Analysis of the predictions. (a, b) First four moments (mean, variance, skewness, and kurtosis) of the PINN predictions with (a) Adam optimizer and Tanh activation, and (b) Soap or SS-Broyden optimizer with Tanh activations. (c) Time-averaged absolute error at each moment for PINN predictions with different optimizers and activations. (d) Barron norm at each time-step of PINN predictions with different optimizers.
Figure 5: KdV equation: Effect of optimizer and activation function. First row: Ground truth (GT) solution, gradient of the GT solution, and Laplacian of the GT solution. The other rows in first, second and third columns show the errors in PINN solution, errors in gradient magnitude of the solution, and errors in Laplacian of the solution, respectively. The rows are organized from the most to the least accurate results (top to bottom). The axis ranges ( x∈[−1,1]x\in[-1,1] , t∈[0,1]t\in[0,1] ) shown in top left subplot is applicable to all.
Figure 6: KdV equation: Spectral bias dynamics during the training. (a-d) Errors of first four statistical moments during the training with different optimizers and activation functions. (e) Error in the frequency domain during the training. The small red arrows show the approximate slope of learning the high-frequencies (shown in third and fourth moments). The dotted black lines in the four moments shows the iteration at which the Tanh and SIREN networks trained with SOAP intersect. Note how the intersection happens earlier at higher moments. The legends in (a) are applicable to all.
 Rel. L2L^{2} ErrorBarron Norm Rel. L2L^{2} ErrorLog(eF,p=0)Log(e_{\mathcal{F},p=0})Log(eF,p=2)Log(e_{\mathcal{F},p=2})Log(eF,p=4)Log(e_{\mathcal{F},p=4})Convergence Time [hr.]Time to 1% Error
PINNs    
Adam (Tanh)2.47×10−12.47\times 10^{-1}1.58×10−11.58\times 10^{-1}-1.520.775.340.056NA
Adam+LBFGS (Tanh)8.12×10−28.12\times 10^{-2}4.14×10−24.14\times 10^{-2}-2.48-0.154.350.056+0.21NA
Adam (SIREN)6.35×10−26.35\times 10^{-2}1.06×10−21.06\times 10^{-2}-2.69-0.194.370.60NA
Adam+LBFGS (SIREN)1.76×10−31.76\times 10^{-3}1.28×10−31.28\times 10^{-3}-5.48-2.941.370.60+0.0450.29
SOAP (Tanh)7.05×10−47.05\times 10^{-4}5.62×10−45.62\times 10^{-4}-6.59-3.910.390.400.033
Table 2: KdV equation prediction errors. Comparison of the performance of PINNs with different activation functions and PIKANs with different univariate functions trained with different optimizers. All models except those trained with SS-Broyden have ∼\sim 40,000 parameters. The models trained with SS-Broyden have ∼\sim 12,900 parameters.
Figure 7: Wave equation results with different optimizers and activation functions (a-b). Relative L2L^{2} error and fourth moment error of each of the wave problems trained with different optimizers and activation functions. (c) Skewness and Kurtosis of PINN prediction with SOAP optimizer and SIREN activation. (d) Skewness and Kurtosis of PINN prediction with SS-Broyden optimizer and SIREN activation.
MethodRel. L2L^{2} ErrorBarron Norm Rel. L2L^{2} Errorlog⁡(eF,p=0)\log(e_{\mathcal{F},p=0})log⁡(eF,p=2)\log(e_{\mathcal{F},p=2})log⁡(eF,p=4)\log(e_{\mathcal{F},p=4})
PINN
Adam (Tanh)1.000.99-1.652.236.80
Adam (SIREN)1.000.95-1.652.236.80
SOAP (Tanh)4.03×10−24.03\times 10^{-2}2.12×10−22.12\times 10^{-2}-4.44-0.693.92
SOAP (SIREN)4.34×10−34.34\times 10^{-3}2.54×10−32.54\times 10^{-3}-6.37-2.272.28
SS-Broyden (Tanh)3.05×10−53.05\times 10^{-5}2.37×10−42.37\times 10^{-4}-10.67-5.220.56
Table 3: Wave equation case (iv) prediction errors. Comparison of PINN and PIKAN performance for the high-frequency wave problem. Results for simpler cases (i)–(iii) are reported in the appendix. All models except those trained with SS-Broyden have ∼\sim 40,000 parameters. The models trained with SS-Broyden have ∼\sim 12,900 parameters.
MethodRel. L2L^{2} ErrorBarron Norm Rel. L2L^{2} Errorlog⁡(eF,p=0)\log(e_{\mathcal{F},p=0})log⁡(eF,p=2)\log(e_{\mathcal{F},p=2})log⁡(eF,p=4)\log(e_{\mathcal{F},p=4})#params KConvergence Time (hr)
PINN 
Adam (Tanh)1.41×10−11.41\times 10^{-1}1.21×10−41.21\times 10^{-4}1.69-2.391.9914.60.18
Adam (SIREN)3.14×10−33.14\times 10^{-3}2.66×10−32.66\times 10^{-3}-1.600.233.4414.60.11
SOAP (Tanh)6.31×10−26.31\times 10^{-2}3.17×10−53.17\times 10^{-5}0.99-3.26-0.5614.60.38
SOAP (SIREN)1.34×10−21.34\times 10^{-2}3.93×10−63.93\times 10^{-6}-0.35-3.99-1.3914.60.29
SS-Broyden (Tanh)1.27×10−61.27\times 10^{-6}7.08×10−107.08\times 10^{-10}-8.39-12.63-10.1114.60.06
Table 4: Steady state diffusion-reaction equation prediction errors. Comparison of PINN and PIKAN performance for the diffusion-reaction problem.
Figure 8: Steady-state diffusion–reaction equation: Effect of optimizer and activation function. Comparison of ground-truth field, gradient magnitude, and Laplacian (top row) with corresponding prediction errors (bottom row) for representative PINN and PIKAN configurations. The rows are organized from the most to the least accurate results (top to bottom). The error maps illustrate qualitative differences in spatial and derivative accuracy that are not fully captured by scalar norms alone. These visualizations complement the quantitative metrics reported in Table 4 .
Figure 9: Turbulent jet: Spectral bias in neural operators. Columns show (from left to right) schlieren images of an impinging jet, the corresponding energy spectrum E(k), spatial gradients, and the Laplacian. Rows compare the ground truth against predictions from different neural operator architectures (with and without BSP training). We observe that using Binned Spectral Power (BSP) loss helps mitigate spectral bias, improving agreement with the true energy spectrum across wavenumbers. All the results are based on SOAP optimizer.
ModelField ErrorEnergy- Spectrum ErrorBarron- Norm Error# params (M)Inference time (s)
DeepONet0.05290.15660.41713.80.00086
DeepONet (BSP)0.05350.16750.40833.80.00086
DeepOKAN0.05380.12890.36593.00.00991
DeepOKAN (BSP)0.05470.04160.16623.00.00991
FNO0.05170.08650.26413.40.00433
FNO (BSP)0.05560.02560.08283.40.00433
Table 5: Comparison of neural operator errors and costs for the impinging jet problem.
ModelField ErrorLog Spectral ErrorBarron Norm Error
 BaselineBSPBaselineBSPBaselineBSP
DeepONet (SIREN)0.00080.00090.00090.12570.12570.13660.13660.00190.00190.00270.0027
DeepONet (Tanh)−-0.0006−-0.11030.1103−-0.0012
DeepOKAN0.00220.00220.00440.00440.12300.12300.17500.17500.00290.00290.00770.0077
FNO0.00380.00380.00410.00410.14850.14850.08190.08190.00450.00450.00300.0030
CNO0.01330.01330.01450.01450.13230.13230.07580.00400.00400.00430.0043
Table 6: Architecture and loss comparison for earthquake structural response prediction. SOAP optimizer, causal training for DeepONet/DeepOKAN.
Figure 10: Representative earthquake response predictions illustrating key design choices. Each row shows the time-domain response (left) and energy spectrum (right) for the same test sample. (a) DeepONet (SIREN) achieves the lowest field error (NRMSE =0.0008=0.0008 ; Table 6 ), followed by DeepOKAN ( 0.00220.0022 ), FNO ( 0.00380.0038 ), and CNO ( 0.01330.0133 ). (b) BSP improves FNO spectral fidelity, reducing log spectral error from 0.1490.149 to 0.0820.082 ( 1.8×1.8\times ), visible in the high-frequency alignment. (c) SOAP enables DeepOKAN convergence (NRMSE =0.0022=0.0022 ), whereas Adam fails across all seeds. (d) SIREN activation enables DeepONet convergence without BSP (NRMSE =0.0008=0.0008 ), while Tanh fails to converge under the same conditions. Black dashed lines denote ground truth; extended comparisons are in Figures E.4 – E.6 .
Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure B.1: PINN with adaptive Tanh activation function training history for KdV equation. Relative L2L^{2} error during training with adaptive activation functions including slope recovery loss term with different weights ( wsrw_{sr} ), trained with Adam. Note that by increasing the weight for the slope recovery term, the errors starts dropping down earlier.
PINNRel. L2L^{2} ErrorBarron Norm Rel. L2L^{2} Errorlog⁡(eF,p=0)\log(e_{\mathcal{F},p=0})log⁡(eF,p=2)\log(e_{\mathcal{F},p=2})log⁡(eF,p=4)\log(e_{\mathcal{F},p=4})
Case (i)
Adam (Tanh)5.08×10−35.08\times 10^{-3}2.94×10−32.94\times 10^{-3}-5.19-1.832.27
Adam (SIREN)6.11×10−36.11\times 10^{-3}4.45×10−34.45\times 10^{-3}-5.03-1.582.54
SOAP (Tanh)1.19×10−31.19\times 10^{-3}2.86×10−42.86\times 10^{-4}-6.44-4.280.80
SOAP (SIREN, w0=30{w}_{0}=30 )1.59×10−31.59\times 10^{-3}2.19×10−42.19\times 10^{-4}-5.11-4.240.49
SOAP (SIREN, w0=5{w}_{0}=5 )8.45×10−48.45\times 10^{-4}1.81×10−41.81\times 10^{-4}-6.59-4.200.50
Appendix
Table C.1: Wave equation prediction errors. Comparison of PINN performance with different activation functions and optimizers for cases (i)-(iii). The results for case (iv) are shown in Table 3 of the main text. Note that for case (i) with no multi-scale features or high frequencies, using SIREN with w0=5w_{0}=5 is better than using SIREN with w0=30w_{0}=30 .
Figure D.2: Optimizer effects on spectral bias in neural operators. Columns show (from left to right) Schlieren fields, the isotropic energy spectrum E(k)E(k) , spatial gradients, and the Laplacian. Rows compare the ground truth against DeepONet and CNO trained with Adam and SOAP. SOAP yields improved agreement with the reference spectrum at high wavenumbers and recovers sharper derivative-based features, indicating reduced attenuation of fine scales relative to Adam.
Figure D.3: SOAP vs Adam for earthquake response prediction. Time response and energy spectrum for a representative test sample comparing Adam (left pair) and SOAP (right pair). Adam DeepOKAN fails to converge (near-flat prediction and flat spectral tail), while SOAP enables convergence for all architectures with improved spectral accuracy at high frequencies.
ModelField ErrorLog Spectral ErrorBarron Norm ErrorAccel. Error
 AdamSOAPAdamSOAPAdamSOAPAdamSOAP
DeepONet (SIREN)0.00150.00150.00080.00080.09740.09740.12570.12570.00250.00250.00190.00190.03810.03810.05060.0506
DeepONet (SIREN, BSP)0.00130.00130.00090.00090.10260.10260.13660.13660.00200.00200.00270.00270.03140.03140.06190.0619
DeepONet (Tanh)−-−-−-−-−-−-−-−-
DeepONet (Tanh, BSP)0.00130.00130.00060.00060.10900.10900.11030.11030.00210.00210.00120.00120.03730.03730.03030.0303
DeepOKAN−-0.00220.0022−-0.12300.1230−-0.00290.0029−-0.05840.0584
Appendix
Table E.2: Optimizer, activation, and loss comparison for earthquake response prediction (causal training). −- Failed to converge.
Figure E.4: Effect of BSP on spectral fidelity for earthquake response prediction. Time response (columns 1, 3) and energy spectrum (columns 2, 4) for a representative test sample comparing baseline (left pair) and BSP (right pair) training across all four architectures. Note the difference in scale of energy values (y-axis) when comparing these results to the impinging-jet problem.
Figure E.5: Activation function effects on DeepONet. Time response and energy spectrum comparing Tanh (left pair) and SIREN (right pair) activations for both baseline (top row) and Log-BSP (bottom row) training. Tanh baseline fails to capture the structural response, but BSP enables convergence to the best overall accuracy. SIREN converges reliably without BSP but shows diminishing returns when BSP is added.
Figure E.6: Causal vs non-causal training for branch-trunk architectures. Time response and energy spectrum comparing non-causal training (left pair) and causal training (right pair) for DeepONet and DeepOKAN. Non-causal DeepOKAN fails to converge entirely, while non-causal DeepONet achieves only marginal convergence with substantially degraded spectral fidelity compared to its causal counterpart.
ModelField ErrorLog Spectral ErrorBarron Norm ErrorAccel. Error
 CNCCNCCNCCNC
DeepONet (SIREN)0.00080.00080.02110.02110.12570.12570.09610.09610.00190.00190.00540.00540.05060.05060.0277
DeepONet (SIREN, BSP)0.00090.0009−-0.13660.1366−-0.00270.0027−-0.06190.0619−-
DeepONet (Tanh)−-−-−-−-−-−-−-−-
DeepONet (Tanh, BSP)0.0006−-0.1103−-0.0012−-0.03030.0303−-
DeepOKAN0.00220.0022−-0.12300.1230−-0.00290.0029−-0.05840.0584−-
Appendix
Table E.3: Causal (C) vs non-causal (NC) training for branch-trunk architectures (SOAP optimizer). −- Failed to converge (field error ≥0.023\geq 0.023 ).
ModelTraining ModeFinal Cos Sim
DeepONet (SIREN)Causal−0.482-0.482
DeepONet (Tanh)Causal−0.181-0.181
DeepOKANCausal+0.021+0.021
DeepONet (SIREN)Non-causal+0.126+0.126
DeepONet (Tanh)Non-causal+0.107+0.107
DeepOKANNon-causal+0.042+0.042
Appendix
Table E.4: Cosine similarity between L2 field-loss and BSP spectral-loss gradients. Positive values indicate aligned optimization directions; negative values indicate competing gradients.
Figure E.7: Gradient cosine similarity over training. Cosine similarity between L2 field-loss and BSP spectral-loss gradients as a function of training completion. Dashed lines indicate causal training; solid lines indicate non-causal training. DeepONet (SIREN) under causal training diverges toward strongly negative values, confirming increasing gradient conflict over the course of training.
More in this topic

Spectral Bias

Spectral bias is the tendency of neural networks to learn low-frequency patterns before high-frequency detail, shaping accuracy and generalization.

See all 37 papers

Explore similar work

CardsList

  1. When Does Frequency Decomposition Benefit Physics-Informed Neural Networks? A Preliminary Ablation Study

    Aug 24, 2026Shubham RaiNeural PDE SolversPhysics-Informed ML

  2. Operator-informed initialization for Fourier features physics-informed neural networks

    Oct 2, 2026Juan Molina, Paris Perdikaris, Mircea Petrache +2Physics-Informed MLNeural Network Initialization

    National Center for Artificial Intelligence, Santiago, Chile · Department of Mechanical Engineering and Applied Mechanics, University of Pennsylvania, Philadelphia, PA, USA · Department of Mathematics, Pontificia Universidad Católica de Chile, Santiago, Chile +3

  3. Pseudo-differential-enhanced physics-informed neural networks

    Feb 16, 2026Andrew GracykPDE SolvingSpectral Bias

    Department of Mathematics, Purdue University, West Lafayette, IN 47907, United States