Affine Geometry of Gaussian ReLU Networks via Conditional Kac-Rice Formulas
Authors: Recep Özkan, Christian Hirsch
Organizations: Department of Mathematics, Middle East Technical University (METU), Ankara, Turkey · Department of Mathematics, Aarhus University, Aarhus, Denmark
We study how the affine geometry of finite ReLU networks is created at random initialization and reorganized by supervised training. We call a sign-changing zero of a hidden preactivation an activation switch and a point where the scalar network output is nondifferentiable a scalar kink. For one-dimensional input, conditioning on the preceding layers makes each preactivation Gaussian and affine on the cells of a random finite partition. This yields an exact finite-width conditional Kac-Rice formula for the expected number of activation switches along an input interval. For fixed depth and proportionally growing widths, the resulting switch intensities converge to explicit deterministic limits. A visibility estimate shows that the expected number of switches that do not produce scalar kinks is negligible, yielding an explicit leading formula for the expected number of scalar kinks and hence affine regions. In higher input dimensions d >= 2, the analogous conditional surface formula yields the leading expected (d-1)-dimensional Hausdorff measure of the scalar kink set. On the Breast Cancer Wisconsin data, the initialization formula accurately predicts switch counts along held-out segments. After training, switch counts decrease along within-class segments and increase along between-class segments. Thus, training redistributes rather than merely contracts affine complexity.
Figures & tables
Figure 1. A deterministic two-layer example separating inherited kinks from fresh activation switches. Panel (a) shows three first-layer units; in panel (b) the later preactivation inherits slope changes at earlier switches (squares) but creates new switches only at its own sign-changing zeros (circles). In panel (c), ReLU gates can transmit or erase inherited kinks. The finite Gaussian genericity argument ensures that, almost surely, these cases do not become ambiguous through coincident switches or exact downstream cancellation.
Figure 2. Monte Carlo estimates of (1+ERN(I))/N , the expected number of maximal affine intervals normalized by N , for β=γ=1 , I=[−1,1] , and equal widths. Horizontal lines are the asymptotic values 1/2 , 5/12 , and approximately 0.3545 for depths L=1,2,3 .
L
widths
N
MC
theory
L
widths
N
MC
theory
1
16
16
8.94±0.05
9.00
2
16,16
32
14.16±0.11
14.33
1
64
64
33.08±0.09
33.00
2
64,64
128
53.92±0.21
54.33
1
256
256
129.15±0.17
129.00
3
64,64,64
192
68.72±0.37
69.06
2
128,64
192
86.17±0.36
86.33
2
64,128
192
74.71±0.42
75.67
Table 1. Monte Carlo estimates of 1+ERN([−1,1]) , the expected number of maximal affine intervals, for β=γ=1 . The theory column is 1+∑ℓnℓ∫Iρℓ ; it is exact for L=1 and the leading prediction for L≥2 . These rows are a selection; Table D.1 reports the full set of runs.
d
widths
prediction
estimate
ratio
2
64
76.68
76.22±1.18
0.994
3
32,32
107.83
108.30±1.84
1.004
5
24,24,24
171.86
167.43±3.35
0.974
Table 2. Finite-network boundary measure on the unit ball with βℓ=γℓ=1 . Predictions are the leading terms from the layerwise surface integral; errors are one standard error over 40 networks. These rows are a selection covering all three input dimensions and all three depths; Table E.2 reports the full set of nine configurations.
architecture
segment family
mean
sd
same sign
(64,64)
within malignant
−24.1%
7.3
50/50
within benign
−22.5%
5.7
50/50
between class
+18.6%
5.4
50/50
off manifold (control)
−4.3%
1.8
49/50
(64,64,64)
within malignant
−27.3%
7.1
50/50
within benign
−25.9%
5.1
50/50
Table 3. Relative change in total switch count from initialization to epoch 300 , over 10 stratified splits with 5 network seeds each. Entries are the mean over the 50 split–seed combinations, one standard deviation across them, and the number of combinations in which the change has the same sign as the mean. Test accuracy is 0.970±0.013 for (64,64) and 0.971±0.013 for (64,64,64) .
Figure 3. Split-level and trajectory views of the same experiment. (a) Results behind Table 3 . Open circles are the ten split means, each averaged over the five network seeds of that split; the horizontal bar is the mean over all 50 split–seed combinations and the grey band is one standard deviation across them. The within-class families lie entirely below zero and the between-class family entirely above it in every split, while the control stays close to zero. (b) For the (64,64) network, switch counts along within-class segments decrease during training, between-class counts increase, and the off-manifold control remains comparatively flat. Counts are normalized by initialization and averaged over 20 network seeds on the single exploratory split.
segment family
layer 1
layer 2
layer 3
within malignant
−7.7±4.4
−27.1±11.1
−48.5±12.4
within benign
−7.6±3.8
−26.1±6.5
−45.1±11.0
between class
+7.6±2.8
+10.1±6.8
+30.5±12.1
off manifold (control)
+0.0±0.6
−10.4±4.1
−17.7±6.8
Table 4. Layerwise relative change for the (64,64,64) network, over the same 10×5 split–seed design as Table 3 . Entries are the mean and one standard deviation across the 50 combinations.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
L
n
N
1+ERN(I) (MC)
theory
1
16
16
8.94±0.05
9
1
64
64
33.08±0.09
33
1
256
256
129.15±0.17
129
2
16
32
14.16±0.11
14.33
2
64
128
53.92±0.21
54.33
2
256
512
213.79±0.43
214.33
Appendix
Table D.1. Monte Carlo estimates of 1+ERN(I) , the expected number of maximal affine intervals, for β=γ=1 , I=[−1,1] , equal widths nℓ=n , based on 2000 , 1000 , and 600 independent networks for L=1,2,3 ; the last block compares two unequal allocations of the same total width N=192 , based on 500 networks each. The theory column shows 1+∑ℓnℓ∫Iρℓ : for L=1 this value is exact, while for L≥2 it is the leading prediction of Theorem 2.1 .
d
a
exact
estimate
ratio
2
0.4
2.5133
2.4698±0.0153
0.983
2
0.7
4.3982
4.4011±0.0144
1.001
3
0.4
2.0106
2.0103±0.0230
1.000
3
0.7
6.1575
6.1563±0.0314
1.000
5
0.4
0.6738
0.7021±0.0212
1.042
5
0.7
6.3192
6.3093±0.0562
0.998
Appendix
Table E.1. The estimator ( E.2 ) applied to the sphere of radius a inside the unit ball. Errors are one standard error over chords. The larger relative errors in the rows with a=0.4 reflect the small proportion of chords that meet a small sphere.
d
widths
N
prediction
estimate
ratio
2
64
64
76.68
76.22±1.18
0.994
2
32,32
64
64.34
61.23±1.16
0.952
2
24,24,24
72
61.81
59.87±1.39
0.969
3
64
64
128.00
128.74±1.53
1.006
3
32,32
64
107.83
108.30±1.84
1.004
3
24,24,24
72
103.80
104.42±2.44
1.006
Appendix
Table E.2. Expected boundary measure ESN,d(D) on the unit ball, with βℓ=γℓ=1 . Errors are one standard error over the 40 networks.
Rectified linear unit (ReLU) neural networks use piecewise-linear activations to study approximation, learning dynamics, expressivity, and verification.
Piecewise-affine neural networks (e.g., with ReLU or LeakyReLU activations) implement continuous piecewise-affine maps, and the number of affine regions provides a natural proxy for expressive capacity. However, the gap between theoretical region capacity and the affine regions realized after training remains insufficiently understood. We study this gap from two complementary perspectives. First, we give a rigorous, architecture-dependent theorem for affine line-segment probes: for multilayer perceptrons with piecewise-affine activations, the number of affine pieces realized along an affine line-segment probe is upper bounded by an explicit product of layer-wise width terms (and activation breakpoint factors). This yields a neuron-threshold lower bound for representing target functions with prescribed one-dimensional piece complexity, formalizing the minimal region budget required for complex signals. Second, we exactly enumerate affine regions realized within bounded 2D and higher-dimensional domains under controlled task complexity. Under fixed architectures and training protocols, increasing input--label complexity yields trained solutions with markedly fewer realized regions in the evaluation domain, even though worst-case architectural capacity is unchanged; we call this reduced region usage expressivity saturation. Moreover, in the most challenging regimes, 2D visualizations show that region-usage collapse often coincides with degraded decision boundaries. Finally, we visualize the training dynamics of affine-region partitions and decision boundaries, revealing a consistent refinement process during optimization.
Xuan Qi, Yi Wei, Fanqi Yu +1
AI for Good (AIGO), Istituto Italiano di Tecnologia · DITEN, University of Genoa · State Key Laboratory of Novel Software Technology +1
It is well established that ReLU networks define continuous piecewise-linear functions, and that their linear regions are polyhedra in the input space. These regions form a complex that fully partitions the input space. The way these regions fit together is fundamental to the behavior of the network, as nonlinearities occur only at the boundaries where these regions connect. However, relatively little is known about the geometry of these complexes beyond bounds on the total number of regions, and calculating the complex exactly is intractable for most networks. In this work, we prove new theoretical results about these complexes that hold for all fully-connected ReLU networks, specifically about their connectivity graphs in which nodes correspond to regions and edges exist between each pair of regions connected by a face. We find that the average degree of this graph is upper bounded by twice the input dimension regardless of the width and depth of the network, and that the diameter of this graph has an upper bound that does not depend on input dimension, despite the number of regions increasing exponentially with input dimension. We corroborate our findings through experiments with networks trained on both synthetic and real-world data, which provide additional insight into the geometry of ReLU networks. Code to reproduce our results can be found at https://github.com/bl-ake/ICLR-2026.
Blake B. Gaines, Jinbo Bi
Department of Computer Science University of Connecticut
Deep networks with continuous piecewise affine activations induce polyhedral partitions of the input space, making the number of realized affine regions a natural measure of expressive capacity and a key determinant of how well the model can approximate nonlinear target functions. In practice, standard training realizes far fewer region refinements in data-visited neighborhoods than the architecture could in principle support, while existing region-count theory is primarily architectural and offers little guidance on how optimization shapes the realized partition near the data. Our theory provides a sufficient condition under which bringing neuron switching surfaces sufficiently close to data points ensures their intersection with local neighborhoods, which in turn implies a strict increase in the local affine-region count, yielding a principled training-time handle for seeding data-relevant partitions early in optimization. Guided by these results, we propose a plug-and-play region-seeding regularizer that encourages early partitioning while allowing task-driven refinement to dominate later in training. Experiments show that the regularizer increases the number of realized affine regions via exact enumeration and improves overall performance on toy datasets, while also improving early-stage accuracy and achieving comparable (or slightly improved) final accuracy on ImageNet-1k for classical models.
Yi Wei, Xuan Qi, Furao Shen
DITEN University of Genoa, Genoa, Italy · State Key Laboratory of Novel Software Technology School of Artificial Intelligence Nanjing University, Jiangsu, China