cs.LGMar 18, 2026

A Family of Adaptive Activation Functions for Mitigating Failure Modes in Physics-Informed Neural Networks

Authors: Krishna Murari

Organizations: †Department of Mathematics, IIT Madras, Chennai 600036, India

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Physics-Informed Neural Networks(PINNs) are a powerful and flexible learning framework that has gained significant attention in recent years. It has demonstrated strong performance across a wide range of scientific and engineering problems. In parallel, wavelets have been extensively used as efficient computational tools due to their strong approximation capabilities. Motivated by the common failure modes observed in standard PINNs, this work introduces a novel family of adaptive wavelet-based activation functions. The proposed activation functions significantly improve training stability and expressive power by combining trainable wavelet functions with either trainable or fixed hyperbolic tangent and softplus functions. Five distinct activation functions are developed within the PINN framework and systematically evaluated across four representative classes of partial differential equations (PDEs). Comprehensive comparisons using bar plots demonstrate improved robustness and accuracy compared to traditional activation functions. Furthermore, the proposed approach is validated through direct comparisons with baseline PINNs, transformer-based architectures such as PINNsFormer, and other deep learning models, highlighting its effectiveness and generality.

Figures & tables

Figure 1: Illustration of the PINNs framework.
Figure 2: Comparison between the Mexican hat and Morlet wavelets.
Figure 3: Hermite functions of orders 1, 2, 3, and 4.
Figure 4: Comparison between the Gaussian wavelet and function
Figure 5: Comparison between the Gabor wavelets at ω0=3\omega_{0}=3 and ω0=5\omega_{0}=5 .
Figure 6: Softplus and Tanh activation function.
CategoryConfigurationValue
Model Hyper-parametersHidden layers4
Hidden size512
Loss weightsλR=λI=λB=1\lambda_{R}=\lambda_{I}=\lambda_{B}=1
Training SettingsOptimizerL-BFGS (Strong Wolfe)
Iterations1000
Table 1: Model hyperparameter and training configurations used for the PINNs model.
Figure 7: Exact solution
Figure 8: Predicted solution and absolute error using Tanh for the 1D reaction equation.
Figure 9: Predicted solution and absolute error using SoftMaxTanh for the 1D reaction equation.
Figure 10: Predicted solution and absolute error using SoftMorTanh for the 1D reaction equation.
Figure 11: Predicted solution and absolute error using SoftGaussTanh for the 1D reaction equation.
Figure 12: Predicted solution and absolute error using SoftHerTanh for the 1D reaction equation.
Figure 13: Predicted solution and absolute error using SoftGaborTanh for the 1D reaction equation.
Table 2: Loss and error comparison across different activation functions for the 1D reaction equation.
Figure 14: Exact solution
Figure 15: Predicted solution and absolute error using Tanh for the 1D wave equation.
Figure 16: Predicted solution and absolute error using SoftMaxTanh for the 1D wave equation.
Figure 17: Predicted solution and absolute error using SoftMorTanh for the 1D wave equation.
Figure 18: Predicted solution and absolute error using SoftGaussTanh for the 1D wave equation.
Figure 19: Predicted solution and absolute error using SoftHer2Tanh for the 1D wave equation.
Figure 20: Predicted solution and absolute error using SoftGaborTanh for the 1D wave equation.
Table 3: Loss and error comparison of PINNs with different activation functions for the 1D wave equation.
Figure 21: Exact solution
Figure 22: Predicted solution and absolute error using Tanh for the 1D convection.
Figure 23: Predicted solution and absolute error using SoftGaborTanhW for the 1D convection.
Table 4: Loss and error comparison of PINNs with different activation functions for the 1D convection equation.
Figure 24: Ground truth solution of 2D Navier-Stokes Equations
Figure 25: PINNs prediction and absolute error for the 2D Navier–Stokes equation using Tanh activation.
Figure 26: PINNs prediction and absolute error for the 2D Navier–Stokes equation using SoftMaxTanh activation.
Figure 27: PINNs prediction and absolute error for the 2D Navier–Stokes equation using SoftMorTanh activation.
Figure 28: PINNs prediction and absolute error for the 2D Navier–Stokes equation using SoftGaussTanh activation.
Figure 29: PINNs prediction and absolute error for the 2D Navier–Stokes equation using SoftHer2Tanh activation.
Figure 30: PINNs prediction and absolute error for the 2D Navier–Stokes equation using SoftGaborTanh activation.
Table 5: Loss and error comparison of PINNs with different activation functions for the 2D Navier–Stokes equations.
Table 6: Loss and error comparison of PINNs with different activation functions for the 2D Navier–Stokes equations.
Figure 31: Performance comparison of activation functions using Loss, rMAE and rRMSE.
Figure 32: Performance comparison of activation functions using Loss, rMAE, and rRMSE.
More in this topic

Neural Network Activation Functions

Neural network activation functions shape how models represent nonlinear patterns, learn, remain stable, and map efficiently to hardware.

See all 78 papers

Explore similar work

CardsList

  1. An adaptive wavelet-based PINN for problems with localized high-magnitude source

    Apr 30, 2026Himanshu Pandey, Ratikanta BeheraPDE SolvingWavelet Methods

    Department of Computational and Data Sciences, Indian Institute of Science, Bangalore, 560012, India

  2. PINNMorph: Evolving Online Adaptation Policies for Physics-Informed Neural Networks

    Sep 26, 2026Xu Yang, Mingyang Yu, Jun Zhang +2Neural PDE SolversPhysics-Informed ML

    East China Normal University · Nankai University · Hanyang University

  3. Improving the accuracy of physics-informed neural networks via last-layer retraining

    Mar 4, 2026Saad Qadeer, Panos StinisNeural PDE SolversPhysics-Informed ML

    Advanced Computing, Mathematics and Data Division, Pacific Northwest National Laboratory, Richland, WA