Statistical Inference with Ill-Conditioned Random Weight Neural Network Sieves
Organizations: University of St. Gallen, Division of Mathematics and Statistics, Rosenbergstrasse 22, 9000 St. Gallen, Switzerland. · University of Warwick, Department of Statistics, Coventry, CV4 7AL, United Kingdom. · Imperial College London, United Kingdom.
Abstract
Random weight neural networks (RWNNs) are shallow networks whose hidden-layer parameters are randomly generated and then kept fixed, while only the output coefficients are estimated. Conditional on the sampled parameters, RWNN regression is a linear sieve problem with random approximation space. We show that such random sieves cannot remain uniformly well-conditioned as their dimension grows. For parametrized feature families satisfying polynomial covering and Lipschitz conditions, the eigenvalues of the population covariance matrix are proven to have polynomial upper bounds determined by the covering geometry of the parameter space, leading to a vanishing smallest eigenvalue. We develop approximation, estimation, and inference theory that explicitly treats the ill-conditioning. Under a -Hölder source condition, we obtain both faster-than-Monte-Carlo approximation rates and new approximation bounds based on stratified sampling. We then derive improved consistency guarantees for ridge-regularized least squares driven by ridge effective dimension, spectral, and leverage-based stability conditions. For inference, we establish asymptotic linearity and Gaussian limits for linear functional directions while allowing the smallest eigenvalue of the population covariance matrix to vanish. Nondegeneracy is directional: we characterize it with a ridge-normalized Rayleigh quotient and extend it to smooth functionals via Riesz representation. Consistency of ridge-adjusted variance estimators is proved, yielding feasible studentized inference. Numerical experiments illustrate the deterioration of random-feature conditioning and the finite-sample behavior of the resulting estimation and inference procedures.
Figures & tables
| Panel A: Approximation results | |||
| Reference/result | Setting | Error criterion | Rate |
| Selected existing results | |||
| [B93] | Trainable features; Barron class | ||
| [KB18] | Trainable ReLU features; sign source representation; weighted coefficients | ||
| [LPSZ25] | Control-neighbors integration; -Hölder integrand on a -dimensional domain | Root MSE | |
| [LMX25] | Random ReLU m features; integral Sobolev class | ||
| Gaussian–Gaussian | Gaussian–Uniform | Uniform–Uniform | |||||||||
| Method | Bias | Coverage | Length | Bias | Coverage | Length | Bias | Coverage | Length | ||
| 100 | 15 | RSZ | 0.089 | 0.891 | 0.242 | 0.086 | 0.896 | 0.244 | -0.022 | 0.877 | 0.269 |
| SNW | 0.089 | 0.901 | 0.254 | 0.086 | 0.898 | 0.258 | -0.022 | 0.900 | 0.287 | ||
| 300 | 20 | RSZ | 0.004 | 0.930 | 0.146 | 0.002 | 0.929 | 0.146 | -0.040 | 0.928 | 0.151 |
| SNW | 0.004 | 0.936 | 0.149 | 0.002 | 0.939 | 0.150 | -0.040 | 0.934 | 0.155 | ||
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Gaussian–Gaussian | Gaussian–Uniform | Uniform–Uniform | |||||||||
| Method | Bias | Coverage | Length | Bias | Coverage | Length | Bias | Coverage | Length | ||
| 100 | 15 | RSZ | -1.565 | 0.582 | 0.356 | -1.474 | 0.611 | 0.360 | -1.780 | 0.575 | 0.395 |
| SNW | -1.565 | 0.570 | 0.348 | -1.474 | 0.605 | 0.355 | -1.780 | 0.559 | 0.385 | ||
| 300 | 20 | RSZ | -1.294 | 0.318 | 0.195 | -1.231 | 0.370 | 0.199 | -1.495 | 0.297 | 0.218 |
| SNW | -1.294 | 0.293 | 0.189 | -1.231 | 0.342 | 0.194 | -1.495 | 0.273 | 0.210 | ||
| Gaussian–Gaussian | Gaussian–Uniform | Uniform–Uniform | |||||||||
| Method | Bias | Coverage | Length | Bias | Coverage | Length | Bias | Coverage | Length | ||
| 100 | 15 | RSZ | -0.115 | 0.900 | 0.259 | -0.120 | 0.896 | 0.265 | -0.335 | 0.865 | 0.298 |
| SNW | -0.115 | 0.925 | 0.269 | -0.120 | 0.925 | 0.277 | -0.335 | 0.888 | 0.311 | ||
| 300 | 20 | RSZ | -0.095 | 0.917 | 0.150 | -0.118 | 0.910 | 0.152 | -0.229 | 0.871 | 0.162 |
| SNW | -0.095 | 0.922 | 0.153 | -0.118 | 0.921 | 0.155 | -0.229 | 0.880 | 0.165 | ||
| Gaussian–Gaussian | Gaussian–Uniform | Uniform–Uniform | |||||||||
| Method | Bias | Coverage | Length | Bias | Coverage | Length | Bias | Coverage | Length | ||
| 100 | 15 | RSZ | 0.139 | 0.889 | 0.237 | 0.129 | 0.878 | 0.239 | 0.086 | 0.871 | 0.260 |
| SNW | 0.139 | 0.898 | 0.251 | 0.129 | 0.902 | 0.253 | 0.086 | 0.902 | 0.285 | ||
| 300 | 20 | RSZ | 0.021 | 0.932 | 0.144 | 0.022 | 0.928 | 0.145 | -0.000 | 0.930 | 0.148 |
| SNW | 0.021 | 0.937 | 0.149 | 0.022 | 0.935 | 0.149 | -0.000 | 0.936 | 0.152 | ||
| Weight Sampling | Scaling | ||||
| Activation | |||||
| Tanh | Gaussian | Gaussian | 5.0 | 1.0 | |
| Tanh | Gaussian | Uniform | 5.0 | 2.0 | |
| Tanh | Uniform | Uniform | 10.0 | 5.0 | |
| ReLU | Gaussian | Gaussian | 5.0 | 2.0 | |
| ReLU | Gaussian | Uniform | 5.0 | 2.0 | |
Random Feature Methods
Random feature methods approximate kernels and neural networks with finite randomized embeddings to reduce computation while retaining useful accuracy.
See all 54 papers