cs.LGMay 5, 2026

Most ReLU Networks Admit Identifiable Parameters

Authors: Moritz GrilloGuido Montúfar

Organizations: Max Planck Institute for Mathematics in the Sciences, Leipzig · Departments of Mathematics and Statistics & Data Science, UCLA

Abstract

We study the realization map of deep ReLU networks, focusing on when a function determines its parameters up to scaling and permutation. To analyze hidden redundancies beyond these standard symmetries, we introduce a framework based on weighted polyhedral complexes. Our main result shows that for every architecture whose input and hidden layers have width at least two, there exists an open set of identifiable parameters. This implies that the functional dimension of every such architecture is exactly the number of parameters minus the number of hidden neurons. We further show that minimal functional representations can still have non-trivial parameter redundancies. Finally, we establish a generic depth hierarchy, whereby for an open set of parameters the realized function cannot be represented generically by any shallower network.

Explore similar work

CardsList
  1. The Symmetries of Three-Layer ReLU Networks

    May 18, 2026Johanna Marie Gegenfurtner, Moritz Grillo, Guido MontúfarRectified Linear Unit NetworksSymmetry