cs.LGNov 11, 2026

Sharp Covering-Number Bounds for Feedforward ReLU Networks via Parameter Symmetries

Authors: Ruizhi Deng, Guohao Shen, Tangrui Xiong

Organizations: Department of Applied Mathematics, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong

Abstract

In neural networks, distinct parameter configurations often realize the same input--output mapping, indicating that nominal parameter counts can overstate functional complexity. In this paper, we characterize the parameter-space symmetries of ReLU and Leaky ReLU networks across fully connected, convolutional, and residual architectures. We prove that every quasianalytic parameter map preserving the network function is a composition of positive channel scaling and channel permutation. By quotienting out both symmetries, we derive uniform covering-number upper bounds in which positive scaling reduces the complexity exponent from the total parameter count SS to S−HS-H (with HH denoting the number of scalable channels) and permutation symmetry divides the bound by the factorial count GG of channel permutations. We establish matching lower bounds showing that the exponent S−HS-H is sharp in general and that the divisor GG is attained for one-hidden-layer and generically identifiable networks. We further extend these results to general continuous piecewise-linear activated networks. Finally, we apply our covering-number bounds to derive statistical risk guarantees for empirical risk minimization and capacity control beyond i.i.d.\ settings, and show how the underlying symmetries help explain weight rebalancing in function approximation and flat directions in empirical risk optimization.

Figures & tables

Explore similar work

CardsList
  1. Most ReLU Networks Admit Identifiable Parameters

    May 5, 2026Moritz Grillo, Guido MontúfarRepresentation IdentifiabilityParameter Identifiability

  2. The Symmetries of Three-Layer ReLU Networks

    May 18, 2026Johanna Marie Gegenfurtner, Moritz Grillo, Guido MontúfarParameter IdentifiabilityReLU Neural Networks