Why Rectified Power Unit Networks Fail and How to Improve It: An Effective Field Theory Perspective
Organizations: School of Computational Sciences Korea Institute for Advanced Study Seoul 02455, South Korea · Department of Mathematical Science Seoul National University Seoul 08826, South Korea
Abstract
The Rectified Power Unit (RePU) activation function, a differentiable generalization of the Rectified Linear Unit (ReLU), has shown promise in constructing neural networks due to its smoothness properties. However, deep RePU networks often suffer from critical issues such as vanishing or exploding values during training, rendering them unstable regardless of hyperparameter initialization. Leveraging the perspective of effective field theory, we identify the root causes of these failures and propose the Modified Rectified Power Unit (MRePU) activation function. MRePU addresses RePU's limitations while preserving its advantages, such as differentiability and universal approximation properties. Theoretical analysis demonstrates that MRePU satisfies the criticality conditions in the limit of a small kernel, and a finite-kernel analysis yields an initialization rule for deep MRePU networks. Experiments on polynomial regression, physics-informed neural networks (PINNs) and real-world vision tasks show that MRePU networks initialized by this rule remain trainable at depths at which RePU networks fail, while their accuracy relative to standard activations depends on the task.
Figures & tables
| Name | Formula | References |
|---|---|---|
| Perceptron | Rosenblatt (1958) | |
| Sigmoid | Cybenko (1989) | |
| Tanh | Cybenko (1989) , LeCun et al. (2002) | |
| Sin | Parascandolo et al. (2016) , Sitzmann et al. (2020) | |
| ReLU | Nair and Hinton (2010) , Glorot et al. (2011) | |
| Leaky ReLU | He et al. (2015) , Maas et al. (2013) |
| Network | Theory | Measured | Exploded / vanished | ||
|---|---|---|---|---|---|
| ReLU | – | ||||
| – | |||||
| RePU ( ) | – | – b | |||
| MRePU ( ), ( R1 ) | |||||
| MRePU ( ) |
| MRePU | ||||
|---|---|---|---|---|
| ReLU | ||||
| GELU |
| MRePU | |||
|---|---|---|---|
| ReLU | |||
| GELU |
| PDE | Depth | tanh | GELU | SiLU | ReLU | RePU | MRePU | MRePU |
|---|---|---|---|---|---|---|---|---|
| Burgers | 3 | |||||||
| 5 | div. b (3/3) | |||||||
| 7 | NaN a | |||||||
| Poisson | 3 | |||||||
| 5 | div. b (2/3) | |||||||
| 7 | NaN a |
| Name | Block | Combination | BN scale | |
|---|---|---|---|---|
| Post-BN | ( 23 ) | |||
| Post-BN- | ( 23 ) | |||
| Post-BN- -crit | ( 23 ) | |||
| Pre-BN | ( 24 ) | unweighted sum | ||
| Pre-BN-crit | ( 24 ) | unweighted sum | ||
| NF- | ( 25 ) | – (no BN) |
| ReLU | GELU | SiLU | RePU | MRePU | MRePU | |
|---|---|---|---|---|---|---|
| Architecture | ||||||
| Post-BN | stall b (3/3) | div. a (3/3) | div. a (3/3) | |||
| Post-BN- | stall b (3/3) | div. a (3/3) | div. a (3/3) | |||
| Post-BN- -crit | stall b (3/3) | (1/3) c | stall b (3/3) | |||
| Pre-BN | stall b (3/3) | stall b (3/3) | stall b (3/3) | |||
| Pre-BN-crit | stall b (3/3) | (1/3) d |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| tanh | GELU | MRePU 2 | MRePU 3 | ||||||
|---|---|---|---|---|---|---|---|---|---|
| PDE | error | ratio | error | ratio | error | ratio | error | ratio | |
| Poisson | 5 | 1.13 (1/3) | 0.89 (2/3) | 1.56 (1/3) | 1.23 (0/3) | ||||
| 7 | 0.97 (1/3) | 1.05 (1/3) | 0.99 (2/3) | 1.21 (0/3) | |||||
| Helmholtz | 5 | 1.38 (0/3) | 0.67 (3/3) | 0.99 (2/3) | 1.50 (0/3) | ||||
| 7 | 0.94 (2/3) | 1.09 (1/3) | 1.30 (1/3) | 0.97 (2/3) | |||||
| ReLU | GELU | SiLU | RePU | MRePU | MRePU | ||
| Backbone | Architecture | ||||||
| width 128 | Post-BN | div. | div. | ||||
| Post-BN- | div. | div. | |||||
| Post-BN- -crit | |||||||
| Pre-BN | |||||||
| Pre-BN-crit |
Neural Network Activation Functions
Neural network activation functions shape how models represent nonlinear patterns, learn, remain stable, and map efficiently to hardware.
See all 84 papers