Affine Geometry of Gaussian ReLU Networks via Conditional Kac-Rice Formulas

Recep Özkan; Christian Hirsch

stat.MLSep 26, 2026

Affine Geometry of Gaussian ReLU Networks via Conditional Kac-Rice Formulas

Authors: Recep Özkan, Christian Hirsch

Organizations: Department of Mathematics, Middle East Technical University (METU), Ankara, Turkey · Department of Mathematics, Aarhus University, Aarhus, Denmark

Abstract

We study how the affine geometry of finite ReLU networks is created at random initialization and reorganized by supervised training. We call a sign-changing zero of a hidden preactivation an activation switch and a point where the scalar network output is nondifferentiable a scalar kink. For one-dimensional input, conditioning on the preceding layers makes each preactivation Gaussian and affine on the cells of a random finite partition. This yields an exact finite-width conditional Kac-Rice formula for the expected number of activation switches along an input interval. For fixed depth and proportionally growing widths, the resulting switch intensities converge to explicit deterministic limits. A visibility estimate shows that the expected number of switches that do not produce scalar kinks is negligible, yielding an explicit leading formula for the expected number of scalar kinks and hence affine regions. In higher input dimensions d >= 2, the analogous conditional surface formula yields the leading expected (d-1)-dimensional Hausdorff measure of the scalar kink set. On the Breast Cancer Wisconsin data, the initialization formula accurately predicts switch counts along held-out segments. After training, switch counts decrease along within-class segments and increase along between-class segments. Thus, training redistributes rather than merely contracts affine complexity.

Figures & tables

Figure 1. A deterministic two-layer example separating inherited kinks from fresh activation switches. Panel (a) shows three first-layer units; in panel (b) the later preactivation inherits slope changes at earlier switches (squares) but creates new switches only at its own sign-changing zeros (circles). In panel (c), ReLU gates can transmit or erase inherited kinks. The finite Gaussian genericity argument ensures that, almost surely, these cases do not become ambiguous through coincident switches or exact downstream cancellation.

Figure 2. Monte Carlo estimates of

(1+\mathbb{E}R_{N}(I))/N

, the expected number of maximal affine intervals normalized by

N

, for

\beta=\gamma=1

I=[-1,1]

, and equal widths. Horizontal lines are the asymptotic values

1/2

5/12

, and approximately

0.3545

for depths

L=1,2,3

$L$	widths	$N$	MC	theory	$L$	widths	$N$	MC	theory
1	$16$	16	$8.94\pm 0.05$	9.00	2	$16,16$	32	$14.16\pm 0.11$	14.33
1	$64$	64	$33.08\pm 0.09$	33.00	2	$64,64$	128	$53.92\pm 0.21$	54.33
1	$256$	256	$129.15\pm 0.17$	129.00	3	$64,64,64$	192	$68.72\pm 0.37$	69.06
2	$128,64$	192	$86.17\pm 0.36$	86.33	2	$64,128$	192	$74.71\pm 0.42$	75.67

Table 1. Monte Carlo estimates of

1+\mathbb{E}R_{N}([-1,1])

, the expected number of maximal affine intervals, for

\beta=\gamma=1

. The theory column is

1+\sum_{\ell}n_{\ell}\int_{I}\rho_{\ell}

; it is exact for

L=1

and the leading prediction for

L\geq 2

. These rows are a selection; Table D.1 reports the full set of runs.

$d$	widths	prediction	estimate	ratio
2	$64$	76.68	$76.22\pm 1.18$	0.994
3	$32,32$	107.83	$108.30\pm 1.84$	1.004
5	$24,24,24$	171.86	$167.43\pm 3.35$	0.974

Table 2. Finite-network boundary measure on the unit ball with

\beta_{\ell}=\gamma_{\ell}=1

. Predictions are the leading terms from the layerwise surface integral; errors are one standard error over

40

networks. These rows are a selection covering all three input dimensions and all three depths; Table E.2 reports the full set of nine configurations.

architecture	segment family	mean	sd	same sign
$(64,64)$	within malignant	$-24.1\%$	$7.3$	$50/50$
	within benign	$-22.5\%$	$5.7$	$50/50$
	between class	$+18.6\%$	$5.4$	$50/50$
	off manifold (control)	$-4.3\%$	$1.8$	$49/50$
$(64,64,64)$	within malignant	$-27.3\%$	$7.1$	$50/50$
	within benign	$-25.9\%$	$5.1$	$50/50$

Table 3. Relative change in total switch count from initialization to epoch

300

, over

10

stratified splits with

5

network seeds each. Entries are the mean over the

50

split–seed combinations, one standard deviation across them, and the number of combinations in which the change has the same sign as the mean. Test accuracy is

0.970\pm 0.013

for

(64,64)

and

0.971\pm 0.013

for

(64,64,64)

Panel of Figure 3 — Figure 3. Split-level and trajectory views of the same experiment. (a) Results behind Table 3 . Open circles are the ten split means, each averaged over the five network seeds of that split; the horizontal bar is the mean over all $50$ split–seed combinations and the grey band is one standard deviation across them. The within-class families lie entirely below zero and the between-class family entirely above it in every split, while the control stays close to zero. (b) For the $(64,64)$ network, switch counts along within-class segments decrease during training, between-class counts increase, and the off-manifold control remains comparatively flat. Counts are normalized by initialization and averaged over $20$ network seeds on the single exploratory split.

segment family	layer 1	layer 2	layer 3
within malignant	$-7.7\pm 4.4$	$-27.1\pm 11.1$	$-48.5\pm 12.4$
within benign	$-7.6\pm 3.8$	$-26.1\pm 6.5$	$-45.1\pm 11.0$
between class	$+7.6\pm 2.8$	$+10.1\pm 6.8$	$+30.5\pm 12.1$
off manifold (control)	$+0.0\pm 0.6$	$-10.4\pm 4.1$	$-17.7\pm 6.8$

Table 4. Layerwise relative change for the

(64,64,64)

network, over the same

10\times 5

split–seed design as Table 3 . Entries are the mean and one standard deviation across the

50

combinations.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

$L$	$n$	$N$	$1+\mathbb{E}R_{N}(I)$ (MC)	theory
$1$	$16$	$16$	$8.94\pm 0.05$	$9$
$1$	$64$	$64$	$33.08\pm 0.09$	$33$
$1$	$256$	$256$	$129.15\pm 0.17$	$129$
$2$	$16$	$32$	$14.16\pm 0.11$	$14.33$
$2$	$64$	$128$	$53.92\pm 0.21$	$54.33$
$2$	$256$	$512$	$213.79\pm 0.43$	$214.33$

Appendix

Table D.1. Monte Carlo estimates of

1+\mathbb{E}R_{N}(I)

, the expected number of maximal affine intervals, for

\beta=\gamma=1

I=[-1,1]

, equal widths

n_{\ell}=n

, based on

2000

1000

, and

600

independent networks for

L=1,2,3

; the last block compares two unequal allocations of the same total width

N=192

, based on

500

networks each. The theory column shows

1+\sum_{\ell}n_{\ell}\int_{I}\rho_{\ell}

: for

L=1

this value is exact, while for

L\geq 2

it is the leading prediction of Theorem 2.1 .

$d$	$a$	exact	estimate	ratio
2	0.4	2.5133	$2.4698\pm 0.0153$	0.983
2	0.7	4.3982	$4.4011\pm 0.0144$	1.001
3	0.4	2.0106	$2.0103\pm 0.0230$	1.000
3	0.7	6.1575	$6.1563\pm 0.0314$	1.000
5	0.4	0.6738	$0.7021\pm 0.0212$	1.042
5	0.7	6.3192	$6.3093\pm 0.0562$	0.998

Appendix

Table E.1. The estimator ( E.2 ) applied to the sphere of radius

a

inside the unit ball. Errors are one standard error over chords. The larger relative errors in the rows with

a=0.4

reflect the small proportion of chords that meet a small sphere.

$d$	widths	$N$	prediction	estimate	ratio
2	$64$	64	76.68	$76.22\pm 1.18$	0.994
2	$32,32$	64	64.34	$61.23\pm 1.16$	0.952
2	$24,24,24$	72	61.81	$59.87\pm 1.39$	0.969
3	$64$	64	128.00	$128.74\pm 1.53$	1.006
3	$32,32$	64	107.83	$108.30\pm 1.84$	1.004
3	$24,24,24$	72	103.80	$104.42\pm 2.44$	1.006

Appendix

Table E.2. Expected boundary measure

\mathbb{E}\mathsf{S}_{N,d}(D)

on the unit ball, with

\beta_{\ell}=\gamma_{\ell}=1

. Errors are one standard error over the

40

networks.

ReLU Neural Networks

Rectified linear unit (ReLU) neural networks use piecewise-linear activations to study approximation, learning dynamics, expressivity, and verification.

See all 113 papers

Expressivity Saturation: Reduced Affine Region Usage Under Increasing Task Complexity
Jun 19, 2026Xuan Qi, Yi Wei, Fanqi Yu +1Multilayer Perceptrons Neural Network Approximation Theory
AI for Good (AIGO), Istituto Italiano di Tecnologia · DITEN, University of Genoa · State Key Laboratory of Novel Software Technology +1
Characterizing the Discrete Geometry of ReLU Networks
Jun 5, 2026Blake B. Gaines, Jinbo BiNeural Representation Geometry ReLU Neural Networks
Department of Computer Science University of Connecticut
Region Seeding via Pre-Activation Regularization: A Geometric View of Piecewise Affine Neural Networks
May 7, 2026Yi Wei, Xuan Qi, Furao ShenNeural Network Activation Functions Neural Network Approximation Theory
DITEN University of Genoa, Genoa, Italy · State Key Laboratory of Novel Software Technology School of Artificial Intelligence Nanjing University, Jiangsu, China

Jun 19, 2026J/K move · Enter open · S save

Expressivity Saturation: Reduced Affine Region Usage Under Increasing Task Complexity

Xuan Qi, Yi Wei, Fanqi Yu, Manuel Lecha

AI for Good (AIGO), Istituto Italiano di Tecnologia · DITEN, University of Genoa · State Key Laboratory of Novel Software Technology+1

Multilayer Perceptrons Neural Network Approximation Theory Neural Network Training Dynamics Neural Network Expressivity

Open paper AskPDF arXiv

Piecewise-affine neural networks (e.g., with ReLU or LeakyReLU activations) implement continuous piecewise-affine maps, and the number of affine regions provides a natural proxy for expressive capacity. However, the gap between theoretical region capacity and the affine regions realized after training remains insufficiently understood. We study this gap from two complementary perspectives. First, we give a rigorous, architecture-dependent theorem for affine line-segment probes: for multilayer perceptrons with piecewise-affine activations, the number of affine pieces realized along an affine line-segment probe is upper bounded by an explicit product of layer-wise width terms (and activation breakpoint factors). This yields a neuron-threshold lower bound for representing target functions with prescribed one-dimensional piece complexity, formalizing the minimal region budget required for complex signals. Second, we exactly enumerate affine regions realized within bounded 2D and higher-dimensional domains under controlled task complexity. Under fixed architectures and training protocols, increasing input--label complexity yields trained solutions with markedly fewer realized regions in the evaluation domain, even though worst-case architectural capacity is unchanged; we call this reduced region usage expressivity saturation. Moreover, in the most challenging regimes, 2D visualizations show that region-usage collapse often coincides with degraded decision boundaries. Finally, we visualize the training dynamics of affine-region partitions and decision boundaries, revealing a consistent refinement process during optimization.

Abstract

ReLU Neural Networks

Explore similar work

Expressivity Saturation: Reduced Affine Region Usage Under Increasing Task Complexity

Characterizing the Discrete Geometry of ReLU Networks

Region Seeding via Pre-Activation Regularization: A Geometric View of Piecewise Affine Neural Networks