stat.MENov 19, 2026

Statistical Inference with Ill-Conditioned Random Weight Neural Network Sieves

Authors: Giovanni Ballarin, Lyudmila Grigoryeva, Lukas Gonon

Organizations: University of St. Gallen, Division of Mathematics and Statistics, Rosenbergstrasse 22, 9000 St. Gallen, Switzerland. · University of Warwick, Department of Statistics, Coventry, CV4 7AL, United Kingdom. · Imperial College London, United Kingdom.

Abstract

Random weight neural networks (RWNNs) are shallow networks whose hidden-layer parameters are randomly generated and then kept fixed, while only the output coefficients are estimated. Conditional on the sampled parameters, RWNN regression is a linear sieve problem with random approximation space. We show that such random sieves cannot remain uniformly well-conditioned as their dimension grows. For parametrized feature families satisfying polynomial covering and Lipschitz conditions, the eigenvalues of the population covariance matrix are proven to have polynomial upper bounds determined by the covering geometry of the parameter space, leading to a vanishing smallest eigenvalue. We develop approximation, estimation, and inference theory that explicitly treats the ill-conditioning. Under a ββ-Hölder source condition, we obtain both faster-than-Monte-Carlo L2L^2 approximation rates and new L∞L^\infty approximation bounds based on stratified sampling. We then derive improved consistency guarantees for ridge-regularized least squares driven by ridge effective dimension, spectral, and leverage-based stability conditions. For inference, we establish asymptotic linearity and Gaussian limits for linear functional directions while allowing the smallest eigenvalue of the population covariance matrix to vanish. Nondegeneracy is directional: we characterize it with a ridge-normalized Rayleigh quotient and extend it to smooth functionals via Riesz representation. Consistency of ridge-adjusted variance estimators is proved, yielding feasible studentized inference. Numerical experiments illustrate the deterioration of random-feature conditioning and the finite-sample behavior of the resulting estimation and inference procedures.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. CAWI: Copula-Aligned Weight Initialization for Randomized Neural Networks

    May 12, 2026Mushir Akhtar, M. Tanveer, Mohd. ArshadNeural Network Initialization

  2. On the Optimality of Random Features under General Conditions: Misspecified Cases and Robust Learning

    Oct 13, 2026Caixing WangRandom Feature MethodsKernel Ridge Regression

  3. Riemannian Newton Methods with Statistical Inference

    Oct 31, 2026Xiaotian Chang, Qirui Hu, Yangdi Jiang +1Covariance EstimationRiemannian Optimization