Brownian Heads for Deep ReLU Representations: Activation Mass and the Cost of Same-Sample Selection

Mahdi Mohammadigohari; Nicole Mücke

stat.MLSep 18, 2026

Brownian Heads for Deep ReLU Representations: Activation Mass and the Cost of Same-Sample Selection

Authors: Mahdi Mohammadigohari, Nicole Mücke

Organizations: Faculty of Engineering Free University of Bozen–Bolzano 39100 Bolzano, Italy · Institute for Statistics and Foundations of Machine Learning Technische Universität Braunschweig Universitätsplatz 2 38106 Braunschweig, Germany

Abstract

Deep representation learning often selects hidden features and fits the final predictor on the same sample, so fixed-feature analysis performed after selection can omit selection cost. We study the conditional empirical Rademacher complexity of deep ReLU representations followed by bounded-norm predictors in additive or Lévy-Brownian RKHSs, termed Brownian heads. For a fixed representation, we derive an exact dual identity and sharp bounds in terms of activation mass, the average norm of the observed hidden vectors. Under same-sample selection, the representation supremum induces a quadratic Rademacher process. Brownian layer-cake and Gaussian-projection identities reduce it to coordinatewise or signed projected threshold traces, separating realized scale from selection complexity. For samples with pairwise-distinct inputs, explicit scalar ReLU families match the finite-trace and VC rates up to universal constants at the realized trace-and-envelope level. Induced-norm contraction also yields architecture-level bounds for rectangular, rank-deficient ReLU networks. Experiments verify the sharp bounds and rates, exhibit a selection gap at fixed activation mass, and assess the predictive feasibility of Brownian heads.

Figures & tables

Figure 1: Exact finite-sample constructions supporting the selection theorem. Panel (a) evaluates the closed-form finite-family selection excess. Panel (b) keeps every representation at the same activation mass while the same-sample family complexity remains constant and each fixed-member trace upper decays with

n

. These are deterministic evaluations, not independent stochastic experiments.

Figure 2: Same-sample learned representation families at the fixed sample size

n=500

. Panel (a) compares the Brownian union complexity with the strongest fixed member after all candidates are rescaled to identical activation mass. Panel (b) shows the context-normalized union-minus-fixed excess; the dashed curve is a descriptive normalized

\sqrt{\log K}

reference.

Figure 3: Paired frozen-transfer differences for the Lévy–Brownian head. Small points are the five seed-level paired differences; the large point is their mean with a descriptive paired percentile-bootstrap interval. All methods share the same frozen features and labeled splits. No multiplicity-corrected significance claim is made.

Method	CIFAR-10	CIFAR-100	EuroSAT
Linear ridge	$79.63\pm 0.33$	$45.41\pm 0.75$	$90.58\pm 0.42$
RBF KRR	$80.53\pm 0.37$	$46.55\pm 0.56$	$91.49\pm 0.33$
Laplace KRR	$\mathbf{80.68\pm 0.45}$	$47.10\pm 0.43$	$91.71\pm 0.30$
Brownian $q=1$	$79.67\pm 0.53$	$45.68\pm 0.54$	$90.58\pm 0.57$
Brownian $q=2$	$80.67\pm 0.23$	$\mathbf{47.31\pm 0.48}$	$\mathbf{91.90\pm 0.30}$

Table 1: Frozen ImageNet-pretrained ResNet-18 transfer with

n_{\rm lab}=1000

labels. Values are mean

\pm

standard deviation over five paired seeds; bold marks the largest observed mean and is not a significance claim. Ridge regularization is selected by training-set GCV. Test labels are never used for fitting or model selection and are accessed only for final evaluation.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Block	Contents	Connection to the main paper
A	Proofs and proof tools for canonical geometry, ReLU propagation, the fixed-head theorem, threshold chaos, same-sample bounds, matching trace dependence, and the explicit network corollary	Direct proof destination for every formal result retained in the nine-page paper
B	Gram, centered, spectral, covariance, root-mass, width, trace, representer, ridge, finite-realization, vector-valued, generalization, and rescaling results	Optional refinements and computational consequences for the terminal-head analysis
C	Exact theorem-aligned diagnostics, learned same-sample families, frozen-feature transfer, digits checks, and reproducibility details	Full evidence and protocols supporting Section 4
D	Fixed-measure BKL links, recursive Brownian chains, dyadic and affine propagation, and sample compression	Optional independent extension; not an ingredient of the main terminal-head theorems

Appendix

Table 2: Role of each appendix block.

Figure 4: Monte Carlo empirical Rademacher complexity of frozen Brownian feature heads compared with the universal lower endpoint, the Gram-sensitive refinement in Proposition B.2 , and the trace upper endpoint. Bars show means over three seeds.

Panel of Figure 5 — Figure 5: Empirical activation mass and layer-norm product proxies during training. Both vertical axes are logarithmic; ribbons show one standard deviation over three seeds.

Figure 6: Euclidean activation mass, centered pairwise Brownian dispersion, and the bias-free centered product envelope from Corollary B.3 . The feature extractor is evaluated on a probe split independent of its training data.

Method	Test accuracy (%)	$\Pi_{q}$
Source-trained neural head	$92.74\pm 2.68$	–
Frozen-feature linear probe	$92.89\pm 3.36$	–
Additive Brownian head ( $q=1$ )	$94.22\pm 2.35$	$0.662\pm 0.162$
Lévy–Brownian head ( $q=2$ )	$94.81\pm 2.82$	$0.964\pm 0.246$

Appendix

Table 3: Digits test accuracy and the fitted scalarized head-scale diagnostic

\Pi_{q}

from ( 130 ), reported as mean

\pm

standard deviation over three seeds. The diagnostic is shown only for the kernel heads and is not directly comparable across different output parameterizations. Because the experiment has

C=10

outputs, the vector-valued upper bound in Proposition B.18 contains the additional fixed factor

\sqrt{10}

;

\Pi_{q}

is therefore not itself a certified multiclass bound.

Panel of Figure 7 — Figure 7: Regularization-path and spectral diagnostics. In (a), points correspond to the predetermined regularization grid; markers show means over three seeds and ribbons show one standard deviation. In (b), faster residual-trace decay yields smaller localized complexity and smaller rank-truncation error.

quantity	fitted slope	$R^{2}$	deterministic check
finite traces vs. $\sqrt{\log N/n}$	0.9785	0.99982	slope in $[0.95,1.02]$
VC traces vs. $\sqrt{V/n}$	0.9804	0.99981	slope in $[0.95,1.02]$

Appendix

Table 4: Deterministic theorem-diagnostic summary.

Figure 8: Complete deterministic diagnostics. The panels show the optimal fixed-feature endpoints, finite-trace

\sqrt{\log N/n}

scaling, VC

\sqrt{V/n}

scaling, and the equal-mass failure under same-sample selection.

Figure 9: Rank-deficiency construction. The aligned, sample-nullspace, and dead-ReLU networks have the same positive product of operator norms, while the latter two have zero realized sample mass. Thus the norm product is a worst-case envelope and has no universal realized converse.

Figure 10: Secondary predictive diagnostic: clean held-out accuracy of the candidate selected by corrupted-training MSE within each nested family. Clean test labels are used only for final evaluation.

Figure 11: Frozen-feature learning curves. Markers show five-seed means and error bars show one standard deviation.

Dataset	Method	$n_{\rm lab}$	Accuracy (%)	Macro-F1 (%)	Fit (s)	Predict (s)
CIFAR-10	Brownian $q=1$	500	$76.92\pm 0.40$	$76.68\pm 0.39$	0.01	0.01
CIFAR-10	Brownian $q=2$	500	$77.82\pm 0.44$	$77.62\pm 0.44$	0.01	0.01
CIFAR-10	Laplace KRR	500	$77.88\pm 0.45$	$77.65\pm 0.45$	0.01	0.01
CIFAR-10	Linear ridge	500	$77.17\pm 0.57$	$76.84\pm 0.57$	0.01	0.01
CIFAR-10	RBF KRR	500	$77.64\pm 0.54$	$77.47\pm 0.54$	0.01	0.02
CIFAR-10	Brownian $q=1$	1000	$79.67\pm 0.53$	$79.49\pm 0.54$	0.05	0.03

Appendix

Table 5: Complete frozen-transfer results for both labeled-data budgets. Values are mean

\pm

standard deviation over five paired seeds.

Figure 12: Brownian fitted-head and realized-Gram diagnostics. These are finite-sample mechanism diagnostics, not deterministic-equivalent test-risk curves.

Dataset	Brownian head	$n_{\rm lab}$	Gain over linear (pp)	Interval (pp)
CIFAR-10	Brownian $q=1$	500	-0.25	[-0.48, -0.05]
CIFAR-10	Brownian $q=2$	500	0.65	[0.49, 0.82]
CIFAR-10	Brownian $q=1$	1000	0.04	[-0.20, 0.25]
CIFAR-10	Brownian $q=2$	1000	1.03	[0.92, 1.13]
CIFAR-100	Brownian $q=1$	500	0.10	[-0.26, 0.54]
CIFAR-100	Brownian $q=2$	500	1.29	[0.90, 1.69]

Appendix

Table 6: Paired gains of the Brownian heads over linear ridge. Intervals are descriptive paired percentile-bootstrap summaries over five seeds; no multiplicity-corrected significance claim is made.

Dataset	Comparator	$n_{\rm lab}$	Difference (pp)	Interval (pp)
CIFAR-10	Laplace KRR	500	-0.06	[-0.13, 0.06]
CIFAR-10	Linear ridge	500	0.65	[0.49, 0.82]
CIFAR-10	Laplace KRR	1000	-0.01	[-0.25, 0.22]
CIFAR-10	Linear ridge	1000	1.03	[0.92, 1.13]
CIFAR-100	Laplace KRR	500	0.35	[0.18, 0.61]
CIFAR-100	Linear ridge	500	1.29	[0.90, 1.69]

Appendix

Table 7: Lévy–Brownian

q=2

paired differences against the two principal non-Brownian baselines.

Explore similar work

Depth Enables Local Entropy: Quadratic Depth Dependence in Deep Variation-Norm ReLU Regression
Aug 18, 2026Tao Jiang, Minbo Gao, Shaowei CaiReLU Neural Networks Minimax Estimation
Key Laboratory of System Software (Chinese Academy of Sciences) · State Key Laboratory of Computer Science · Institute of Software, Chinese Academy of Sciences +1
Deep Neural Variation Spaces: A Unifying Perspective on Depth and Complexity
Jul 6, 2026Julia Nakhleh, Robert D. NowakReLU Neural Networks Neural Network Approximation Theory
Department of Computer Science · University of Wisconsin-Madison · Department of Electrical & Computer Engineering
Nearly Tight Rademacher Bounds for Sparsely Activated Neural Networks
Sep 8, 2026Xiaoyu Li, Zhizhou Sha, Jiaojiao Jiang +2Neural Network Generalization ReLU Neural Networks
University of New South Wales · University of Texas at Austin · University of Sydney

Aug 18, 2026, cs.AIJ/K move · Enter open · S save

Depth Enables Local Entropy: Quadratic Depth Dependence in Deep Variation-Norm ReLU Regression

Tao Jiang, Minbo Gao, Shaowei Cai

Key Laboratory of System Software (Chinese Academy of Sciences) · State Key Laboratory of Computer Science · Institute of Software, Chinese Academy of Sciences+1

ReLU Neural Networks Minimax Estimation

Open paper AskPDF arXiv

We study Gaussian regression over the explicit vector-valued Parhi--Nowak deep-RBV^2 architecture with depth L, width w, layer-sum variation budget A, and output bound B. For this O(L w^2)-parameterized architecture, the known lower and upper bounds differ by one factor of depth. We construct a local packing showing that the quadratic depth dependence is intrinsic under an explicit sample-size-dependent radius condition. The packing has log-cardinality Omega(L^2 w^2 log w); its codewords lie in an O(lambda) L^2 ball and are pairwise Omega(lambda)-separated. The main ingredients are a bias-corrected bounded-coefficient approximation theorem and balanced amplification: multiplying a depth-D ReLU network by q can be implemented using one constant channel so that every coefficient grows by only q^(1/D). Translation to vector-valued RBV^2 blocks then has layer-sum cost O(D w^2 q^(1/D)). Gaussian Fano yields a radius-explicit lower bound governed by the output, testing, and representation scales. Under A=B=R, sigma proportional to R, and the stated radius condition, this gives minimax risk at least of order L^2 w^2 log(w) R^2/n. A pseudodimension-based finite-net upper bound gives O-tilde(L^2 w^2 R^2/n) for unbounded Gaussian responses. Thus the minimax risk has quadratic polynomial dependence on depth, up to logarithmic factors, and exhibits a transition to representation-limited behavior at smaller radius.

Abstract

Explore similar work

Depth Enables Local Entropy: Quadratic Depth Dependence in Deep Variation-Norm ReLU Regression

Deep Neural Variation Spaces: A Unifying Perspective on Depth and Complexity

Nearly Tight Rademacher Bounds for Sparsely Activated Neural Networks