cs.LGAug 4, 2024

Why Rectified Power Unit Networks Fail and How to Improve It: An Effective Field Theory Perspective

Authors: Taeyoung Kim, Myungjoo Kang

Organizations: School of Computational Sciences Korea Institute for Advanced Study Seoul 02455, South Korea · Department of Mathematical Science Seoul National University Seoul 08826, South Korea

Abstract

The Rectified Power Unit (RePU) activation function, a differentiable generalization of the Rectified Linear Unit (ReLU), has shown promise in constructing neural networks due to its smoothness properties. However, deep RePU networks often suffer from critical issues such as vanishing or exploding values during training, rendering them unstable regardless of hyperparameter initialization. Leveraging the perspective of effective field theory, we identify the root causes of these failures and propose the Modified Rectified Power Unit (MRePU) activation function. MRePU addresses RePU's limitations while preserving its advantages, such as differentiability and universal approximation properties. Theoretical analysis demonstrates that MRePU satisfies the criticality conditions in the limit of a small kernel, and a finite-kernel analysis yields an initialization rule for deep MRePU networks. Experiments on polynomial regression, physics-informed neural networks (PINNs) and real-world vision tasks show that MRePU networks initialized by this rule remain trainable at depths at which RePU networks fail, while their accuracy relative to standard activations depends on the task.

Figures & tables

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Deep Network Approximation: Beyond ReLU to Diverse Activation Functions

    Jul 13, 2023Shijun Zhang, Jianfeng Lu, Hongkai ZhaoReLU Neural NetworksNeural Network Activation Functions

  2. Optimized Weight Initialization on the Stiefel Manifold for Deep ReLU Neural Networks

    Aug 30, 2025Hyungu Lee, Taehyeong Kim, Hayoung ChoiNeural Network InitializationReLU Neural Networks