This paper explores the expressive power of deep neural networks for a diverse range of activation functions. An activation function set
A is defined to encompass the majority of commonly used activation functions, such as
ReLU,
LeakyReLU,
ReLU2,
ELU,
CELU,
SELU,
Softplus,
GELU,
SiLU,
Swish,
Mish,
Sigmoid,
Tanh,
Arctan,
Softsign,
dSiLU, and
SRS. We demonstrate that for any activation function
ϱ∈A, a
ReLU network of width
N and depth
L can be approximated to arbitrary precision by a
ϱ-activated network of width
3N and depth
2L on any bounded set. This finding enables the extension of most approximation results achieved with
ReLU networks to a wide variety of other activation functions, albeit with slightly increased constants. Significantly, we establish that the (width,
depth) scaling factors can be further reduced from
(3,2) to
(1,1) if
ϱ falls within a specific subset of
A. This subset includes activation functions such as
ELU,
CELU,
SELU,
Softplus,
GELU,
SiLU,
Swish, and
Mish.