Empirical Risk Minimization

Also known as ERM

Ask about this topic

Overview

Empirical risk minimization chooses a model or decision rule to minimize average loss on a training sample, providing a foundation for regression, classification, and many neural-network objectives. Statistical analyses use localized Rademacher complexity, concentration and small-ball methods, and high-dimensional tools such as Gaussian comparison and approximate message passing to bound test error and study when structure, weighting, or constraints help. Optimization work examines gradient methods, regularization, distributed primal-dual algorithms, and privacy-preserving variants, while extensions address biased or dependent data, distribution shift, heavy tails, and selective labels. A central tension is that low empirical risk or faster fitting need not ensure reliable test performance: guarantees depend on model complexity and data conditions, while computational, privacy, and robustness constraints introduce further tradeoffs.

Latest papers

CardsList

45 papers5 papers in the last four weeks, down 29% on the four weeks before. 0.0% of all new papers.

Mar 15, 2026

Interleaved Resampling and Refitting: Data and Compute-Efficient Evaluation of Black-Box Predictors

We study the problem of evaluating the excess risk of large-scale empirical risk minimization under the square loss. Leveraging the idea of wild refitting and resampling, we assume only black-box access to the training algorithm and develop an efficient procedure for estimating the excess risk. Our evaluation algorithm is both computationally and data efficient. In particular, it requires access to only a single dataset and does not rely on any additional validation data. Computationally, it only requires refitting the model on several much smaller datasets obtained through sequential resampling, in contrast to previous wild refitting methods that require full-scale retraining and might therefore be unsuitable for large-scale trained predictors. Our algorithm has an interleaved sequential resampling-and-refitting structure. We first construct pseudo-responses through a randomized residual symmetrization procedure. At each round, we thus resample two sub-datasets from the resulting covariate pseudo-response pairs. Finally, we retrain the model separately on these two small artificial datasets. We establish high probability excess risk guarantees under both fixed design and random design settings, showing that with a suitably chosen noise scale, our interleaved resampling and refitting algorithm yields an upper bound on the prediction error. Our theoretical analysis draws on tools from empirical process theory, harmonic analysis, Toeplitz operator theory, and sharp tensor concentration inequalities.

Haichen Hu, David Simchi-Levi

Laboratory for Information and Decision Systems, Massachusetts Institute of Technology, Cambridge, MA 02139, USA

Empirical Risk Minimization
AskPDF
Mar 5, 2026

On the Statistical Optimality of Optimal Decision Trees

While globally optimal empirical risk minimization (ERM) decision trees have become computationally feasible and empirically successful, rigorous theoretical guarantees for their statistical performance remain limited. In this work, we develop a comprehensive statistical theory for ERM trees under random design in both high-dimensional regression and classification. We first establish sharp oracle inequalities that bound the excess risk of the ERM estimator relative to the best possible approximation achievable by any tree with at most LL leaves, thereby characterizing the interpretability-accuracy trade-off. We derive these results using a novel uniform concentration framework based on empirically localized Rademacher complexity. Furthermore, we derive minimax optimal rates over a novel function class: the piecewise sparse heterogeneous anisotropic Besov (PSHAB) space. This space explicitly captures three key structural features encountered in practice: sparsity, anisotropic smoothness, and spatial heterogeneity. While our main results are established under sub-Gaussianity, we also provide robust guarantees that hold under heavy-tailed noise settings. Together, these findings provide a principled foundation for the optimality of ERM trees and introduce empirical process tools broadly applicable to other highly adaptive, data-driven procedures.

Zineng Xu, Subhro Ghosh, Yan Shuo Tan

Department of Statistics and Data Science, National University of Singapore · Department of Mathematics, National University of Singapore

Decision Tree LearningStatistical Learning TheoryEmpirical Risk MinimizationMinimax Estimation
AskPDF
Feb 25, 2026

A Researcher's Guide to Empirical Risk Minimization

This guide provides a reference for high-probability regret bounds in empirical risk minimization (ERM). The presentation is modular: we begin with intuition and general proof strategies, then state broadly applicable guarantees under high-level conditions and provide tools for verifying them for specific losses and function classes. We emphasize that many ERM rate derivations can be organized around a three-step recipe -- a basic inequality, a uniform local concentration bound, and a fixed-point argument -- which yields regret bounds in terms of a critical radius, defined via localized Rademacher complexity, under a mild Bernstein-type variance-risk condition. To make these bounds concrete, we upper bound the critical radius using local maximal inequalities and metric-entropy integrals, thereby recovering familiar rates for VC-subgraph, Sobolev/Hölder, and bounded-variation classes. We also study ERM with nuisance components -- including weighted ERM and Neyman-orthogonal losses -- as they arise in causal inference, missing data, and domain adaptation. Following the orthogonal statistical learning framework, we highlight that these problems often admit regret-transfer bounds linking regret under an estimated loss to population regret under the target loss. These bounds typically decompose the regret into (i) statistical error under the estimated loss and (ii) approximation error due to nuisance estimation. Under sample splitting or cross-fitting, the first term can be controlled using standard fixed-loss ERM regret bounds, while the second depends only on nuisance-estimation accuracy. As a novel contribution, we also treat the in-sample regime, in which the nuisances and the ERM are fit on the same data, deriving regret bounds and showing that fast oracle rates remain attainable under suitable smoothness and Donsker-type conditions.

Lars van der Laan

Department of Statistics, University of Washington

Empirical Risk MinimizationStatistical Learning Theory
AskPDF
Feb 22, 2026

Why ReLU? A Bit-Model Dichotomy for Deep Network Training

Theoretical analyses of Empirical Risk Minimization (ERM) are standardly framed within the Real-RAM model of computation. In this setting, training even simple neural networks is known to be ∃R\exists \mathbb{R}-complete -- a complexity class believed to be harder than NP, that characterizes the difficulty of solving systems of polynomial inequalities over the real numbers. However, this algebraic framework diverges from the reality of digital computation with finite-precision hardware. In this work, we analyze the theoretical complexity of ERM under a realistic bit-level model (ERMbit\mathsf{ERM}_{\text{bit}}), where network parameters and inputs are constrained to be rational numbers with polynomially bounded bit-lengths. Under this model, we reveal a sharp dichotomy in tractability governed by the network's activation function. We prove that for deep networks with {\em any} polynomial activations with rational coefficients and degree at least 22, the bit-complexity of training is severe: deciding ERMbit\mathsf{ERM}_{\text{bit}} is #P\#P-Hard, hence believed to be strictly harder than NP-complete problems. Furthermore, we show that determining the sign of a single partial derivative of the empirical loss function is intractable (unlikely in BPP), and deciding a specific bit in the gradient is #P\#P-Hard. This provides a complexity-theoretic perspective for the phenomenon of exploding and vanishing gradients. In contrast, we show that for piecewise-linear activations such as ReLU, the precision requirements remain manageable: ERMbit\mathsf{ERM}_{\text{bit}} is contained within NP (specifically NP-complete), and standard backpropagation runs in polynomial time. Our results demonstrate that finite-precision constraints are not merely implementation details but fundamental determinants of learnability.

Ilan Doron-Arad, Elchanan Mossel

MIT, Cambridge, MA, USA

Empirical Risk MinimizationReLU Neural NetworksPolynomial Neural Networks
AskPDF
Feb 20, 2026

On the Generalization and Robustness in Conditional Value-at-Risk

Conditional Value-at-Risk (CVaR) is a widely used risk-sensitive objective for learning under rare but high-impact losses, yet its statistical behavior under heavy-tailed data remains poorly understood. Unlike expectation-based risk, CVaR depends on an endogenous, data-dependent quantile, which couples tail averaging with threshold estimation and fundamentally alters both generalization and robustness properties. In this work, we develop a learning-theoretic analysis of CVaR-based empirical risk minimization under heavy-tailed and contaminated data. We establish sharp, high-probability generalization and excess risk bounds under minimal moment assumptions, covering fixed hypotheses, finite and infinite classes, and extending to ββ-mixing dependent data; we further show that these rates are minimax optimal. To capture the intrinsic quantile sensitivity of CVaR, we derive a uniform Bahadur-Kiefer type expansion that isolates a threshold-driven error term absent in mean-risk ERM and essential in heavy-tailed regimes. We complement these results with robustness guarantees by proposing a truncated median-of-means CVaR estimator that achieves optimal rates under adversarial contamination. Finally, we show that CVaR decisions themselves can be intrinsically unstable under heavy tails, establishing a fundamental limitation on decision robustness even when the population optimum is well separated. Together, our results provide a principled characterization of when CVaR learning generalizes and is robust, and when instability is unavoidable due to tail scarcity.

Dinesh Karthik Mulumudi, Piyushi Manupriya, Gholamali Aminian +1

Dept. of Mathematics, IISER, Pune. · Dept. of Computer Science and Automation, Indian Institute of Sciecne, Bangalore. · The Alan Turing Institute, Greater London, England, UK.

Conditional Value-at-RiskEmpirical Risk MinimizationStatistical Learning TheoryRobust EstimationHeavy-Tailed Distributions
AskPDF
Feb 19, 2026

Topological Exploration of High-Dimensional Empirical Risk Landscapes: general approach, and applications to phase retrieval

We consider the landscape of empirical risk minimization for high-dimensional Gaussian single-index models (generalized linear models). The objective is to recover an unknown signal θ⋆∈Rd\boldsymbolθ^\star \in \mathbb{R}^d (where d≫1d \gg 1) from a loss function R^(θ)\hat{R}(\boldsymbolθ) that depends on pairs of labels (xi⋅θ,xi⋅θ⋆)i=1n(\mathbf{x}_i \cdot \boldsymbolθ, \mathbf{x}_i \cdot \boldsymbolθ^\star)_{i=1}^n, with xi∼N(0,Id)\mathbf{x}_i \sim \mathcal{N}(0, I_d), in the proportional asymptotic regime n≍dn \asymp d. Using the Kac-Rice formula, we analyze different complexities of the landscape -- defined as the expected number of critical points -- corresponding to various types of critical points, including local minima. We first show that some variational formulas previously established in the literature for these complexities can be drastically simplified, reducing to explicit variational problems over a finite number of scalar parameters that we can efficiently solve numerically. Our framework also provides detailed predictions for properties of the critical points, including the spectral properties of the Hessian and the joint distribution of labels. We apply our analysis to the real phase retrieval problem for which we derive complete topological phase diagrams of the loss landscape, characterizing notably BBP-type transitions where the Hessian at local minima (as predicted by the Kac-Rice formula) becomes unstable in the direction of the signal. We test the predictive power of our analysis to characterize gradient flow dynamics, finding excellent agreement with finite-size simulations of local optimization algorithms, and capturing fine-grained details such as the empirical distribution of labels. Overall, our results open new avenues for the asymptotic study of loss landscapes and topological trivialization phenomena in high-dimensional statistical models.

Antoine Maillard, Tony Bonnaire, Giulio Biroli

INRIA Paris & DI ENS, PSL University, Paris, France. · Université Paris-Saclay, CNRS, Institut d’Astrophysique Spatiale, 91405 Orsay, France · Laboratoire de Physique de l’École normale supérieure, ENS, Université PSL, CNRS, Sorbonne Université, Université Paris Cité, F-75005 Paris, France

Loss LandscapeLoss Landscape GeometryEmpirical Risk MinimizationHigh-Dimensional Asymptotics
AskPDF
Feb 18, 2026

Conjugate Learning Theory: Uncovering the Mechanisms of Trainability and Generalization in Deep Neural Networks

In this work, we propose a notion of practical learnability grounded in finite sample settings, and develop a conjugate learning theoretical framework based on convex conjugate duality to characterize this learnability property. Building on this foundation, we demonstrate that training deep neural networks (DNNs) with mini-batch stochastic gradient descent (SGD) achieves global optima of empirical risk by jointly controlling the extreme eigenvalues of a structure matrix and the gradient energy, and we establish a corresponding convergence theorem. We further elucidate the impact of batch size and model architecture (including depth, parameter count, sparsity, skip connections, and other characteristics) on non-convex optimization. Additionally, we derive a model-agnostic lower bound for the achievable empirical risk, theoretically demonstrating that data determines the fundamental limit of trainability. On the generalization front, we derive deterministic and probabilistic bounds on generalization error based on generalized conditional entropy measures. The former explicitly delineates the range of generalization error, while the latter characterizes the distribution of generalization error relative to the deterministic bounds under independent and identically distributed (i.i.d.) sampling conditions. Furthermore, these bounds explicitly quantify the influence of three key factors: (i) information loss induced by irreversibility in the model, (ii) the maximum attainable loss value, and (iii) the generalized conditional entropy of features with respect to labels. Moreover, they offer a unified theoretical lens for understanding the roles of regularization, irreversible transformations, and network depth in shaping the generalization behavior of deep neural networks. Extensive experiments validate all theoretical predictions, confirming the framework's correctness and consistency.

Binchuan Qi

Tongji University, Siping Street, 200092 Shanghai, China · Zhejiang Yuying College of Vocational Technology, Sihao Street, Hangzhou, 310018, Zhejiang, China

Neural Network GeneralizationNeural Network OptimizationInformation-Theoretic Generalization BoundsEmpirical Risk MinimizationStochastic Gradient Descent
AskPDF
Feb 5, 2026

Fast Rates for Nonstationary Weighted Risk Minimization

Weighted empirical risk minimization is a common approach to prediction under distribution drift. This article studies its out-of-sample prediction error under nonstationarity. We provide a general decomposition of the excess risk into a learning term and an error term associated with distribution drift, and prove oracle inequalities for the learning error under mixing conditions. The learning bound holds uniformly over arbitrary weight classes and accounts for the effective sample size induced by the weight vector, the complexity of the weight and hypothesis classes, and potential data dependence. We also analyze the drift error in the square loss setting and provide explicit bounds for structural-break and smoothly drifting processes. We then combine the learning and drift analysis in (auto-) regression problems with linear models, basis approximations, and neural networks. In all cases, the optimal weights are characterized by a bias--variance trade-off between learning and drift, while minimax-optimal rates (up to logarithmic factors) are recovered when specialized to unweighted and stationary settings.

Tobias Brock, Thomas Nagler

LMU Munich Munich Center for Machine Learning (MCML)

Empirical Risk MinimizationStatistical Learning TheoryAdaptive Loss Weighting
AskPDF
Nov 17, 2025

On the Gradient Complexity of Private Optimization with Private Oracles

We study the running time, in terms of first order oracle queries, of differentially private empirical/population risk minimization of Lipschitz convex losses. We first consider the setting where the loss is non-smooth and the optimizer interacts with a private proxy oracle, which sends only private messages about a minibatch of gradients. In this setting, we show that expected running time Ω(min⁡{dα2,dlog⁡(1/α)})Ω(\min\{\frac{\sqrt{d}}{α^2}, \frac{d}{\log(1/α)}\}) is necessary to achieve αα excess risk on problems of dimension dd when d≥1/α2d \geq 1/α^2. Upper bounds via DP-SGD show these results are tight when d>Ω~(1/α4)d>\tildeΩ(1/α^4). We further show our lower bound can be strengthened to Ω(min⁡{dmˉα2,dlog⁡(1/α)})Ω(\min\{\frac{d}{\bar{m}α^2}, \frac{d}{\log(1/α)} \}) for algorithms which use minibatches of size at most mˉ<d\bar{m} < \sqrt{d}. We next consider smooth losses, where we relax the private oracle assumption and give lower bounds under only the condition that the optimizer is private. Here, we lower bound the expected number of first order oracle calls by Ω~(dα+min⁡{1α2,n})\tildeΩ\big(\frac{\sqrt{d}}α + \min\{\frac{1}{α^2}, n\}\big), where nn is the size of the dataset. Modifications to existing algorithms show this bound is nearly tight. Compared to non-private lower bounds, our results show that differentially private optimizers pay a dimension dependent runtime penalty. Finally, as a natural extension of our proof technique, we show lower bounds in the non-smooth setting for optimizers interacting with information limited oracles. Specifically, if the proxy oracle transmits at most ΓΓ-bits of information about the gradients in the minibatch, then Ω(min⁡{dα2Γ,dlog⁡(1/α)})Ω\big(\min\{\frac{d}{α^2Γ}, \frac{d}{\log(1/α)}\}\big) oracle calls are needed. This result shows fundamental limitations of gradient quantization techniques in optimization.

Michael Menart, Aleksandar Nikolov

Department of Computer Science, University of Toronto, Vector Institute · Department of Computer Science, University of Toronto

Query ComplexityOracle ComplexityDifferentially Private Stochastic Gradient DescentDifferential PrivacyGradient QuantizationNonsmooth Convex Optimization+1
AskPDF
Oct 27, 2025

Complexity Dependent Error Rates for Physics-informed Statistical Learning via the Small-ball Method

Physics-informed statistical learning (PISL) integrates empirical data with physical knowledge to enhance the statistical performance of estimators. While PISL methods are widely used in practice, a comprehensive theoretical understanding of how informed regularization affects statistical properties is still missing. Specifically, two fundamental questions have yet to be fully addressed: (A) what is the trade-off between soft penalties and hard constraints, and (B) what is the statistical gain of incorporating physical knowledge compared to purely data-driven empirical error minimisation. In this paper, we address these questions for PISL in classes of functions whose convex hull satisfies a small-ball condition, in particular convex classes, under physical knowledge expressed as linear equations by developing appropriate complexity dependent error rates based on the small-ball method. We specialise them to linear classes of finite dimension qq in which the space of solutions of the equation has dimension k<qk < q, and show that (A) the physics-informed estimator attains the error bound of the hard-constrained empirical error minimiser whenever the penalty parameter is above an explicit threshold, and that (B) the penalty improves the error rate from q/n\sqrt{q/n} to k/n\sqrt{k/n} under a second moment condition on the noise only, having the effect of a dimensionality reduction. We also quantify how these conclusions degrade when the equation is misspecified, in terms of the excess residual of the target function. This work establishes a theoretical framework for evaluating the statistical properties of physics-informed estimators in such classes, contributing to closing the gap between statistical theory and practical PISL.

Diego Marcondes

Physics-Informed MLStatistical Learning TheoryEmpirical Risk Minimization
AskPDF
Oct 16, 2025

LLM Priors for ERM over Programs

We study program-learning methods that are efficient in both samples and computation. Classical learning theory suggests that when the target admits a short program description, for example a short piece of ``Python code'', it can be learned from few examples by ERM over the program class. However, this approach relies on enumerating candidate programs, which is typically exponential in the description length; gradient-based training avoids this explicit search but, for some families of short programs, can require exponentially many samples to succeed. We propose LLM-PV, a propose-and-verify recipe that enables ERM-style selection over a discrete program class without exhaustive enumeration: a pretrained LLM induces a proposal distribution over candidate programs, each proposal is executed and scored on a held-out validation set, and the best program is selected, with no gradient updates or validation feedback used to adapt the sampling distribution. Across algorithmic tasks including parity variants, pattern matching, and primality testing, LLM-PV often recovers the exact underlying rule from a small labeled set and generalizes far beyond the training sequence lengths, while SGD-trained transformers, fine-tuning, in-context learning, and classical ML baselines can fit the training data yet fail to generalize reliably. Together, these results suggest that pretrained LLM priors can serve as effective search biases for ERM, narrowing the gap between statistical and computational efficiency.

Shivam Singhal, Priyadarsi Mishra, Eran Malach +1

Department of Computer Science & Engineering, Texas A&M University, TX, USA · Kempner Institute, Harvard University, MA, USA

LLM-Based Program SynthesisEmpirical Risk Minimization
AskPDF
Feb 1, 2025

On the Relationship Between CoCoA and ADMM for Distributed Empirical Risk Minimization

Distributed empirical risk minimization (ERM) is often studied through two influential yet seemingly separate families of methods: CoCoA-type algorithms, derived from distributed dual coordinate ascent, and ADMM-type algorithms, derived from consensus and proximal splitting. In this paper, we investigate the connection of the two types of algorithms from a unified primal-dual perspective. We show that consensus ADMM, linearized consensus ADMM, two distributed proximal ADMM variants, and ridge-regularized CoCoA can all be written in a common update form involving a global primal variable and block dual variables. This reformulation makes several previously hidden connections explicit: For ridge-regularized ERM, CoCoA coincides with a particular proximal ADMM scheme at the level of the dual update. Moreover, consensus ADMM on the primal problem is equivalent to proximal ADMM on the dual problem under an explicit parameter mapping together with a sign reversal of the saddle objective; similar correspondences also hold for the linearized variants. These results indicates that the ADMM-type algorithms, when fine tuned, performs at least as good as CoCoA, under ridge regularized ERM problems. The unified view also yields a natural primal-dual gap stopping criterion for consensus ADMM and a unified O(1/T)O(1/T) ergodic convergence analysis for the ADMM-type methods. Experiments on synthetic regression problems and real SVM datasets support the predicted relationships, clarify the role of tuning parameters, and show that suitably tuned ADMM variants can outperform CoCoA in the ridge-regularized setting.

Runxiong Wu, Andi Wang

Department of Industrial & Systems Engineering University of Wisconsin–Madison

Primal-Dual OptimizationDistributed OptimizationAlternating Direction Method of MultipliersEmpirical Risk Minimization
AskPDF
Jul 24, 2023

A Differentially Private Weighted Empirical Risk Minimization Procedure and its Application to Outcome Weighted Learning

Data used to train predictive models via empirical risk minimization (ERM) often contain sensitive personal information. While differential privacy (DP) provides mathematically provable bounds to protect such data, previous work has focused almost exclusively on unweighted ERM. We consider weighted ERM (wERM) -- an important generalization where individual contributions to the objective function vary. We propose the first DP algorithm for general wERM with formal privacy guarantees and derive both its empirical and population excess risk bounds. Crucially, this general wERM framework provides a pathway for deriving privacy-preserving learning methods for individualized treatment rules, including the popular outcome-weighted learning (OWL) approach. We evaluate DP-wERM applied to OWL in simulated and real data experiments. Our empirical results demonstrate that training OWL models via wERM provides strong DP guarantees while maintaining robust performance, proving the method is practical for sensitive, real-world data.

Spencer Giddens, Yiwang Zhou, Kevin R. Krull +3

University of Notre Dame, Notre Dame, IN 46556, USA · St. Jude Children’s Research Hospital, Memphis, TN 38138, USA · University of Michigan, Ann Arbor, MI 48109, USA

Privacy-Preserving MLDifferential PrivacyStatistical Learning TheoryEmpirical Risk Minimization
AskPDF
Sep 5, 2022

Learning from a Biased Sample

The empirical risk minimization approach to data-driven decision making requires access to training data drawn under the same conditions as those that will be faced when the decision rule is deployed. However, in a number of settings, we may be concerned that our training sample is biased in the sense that some groups (characterized by either observable or unobservable attributes) may be under- or over-represented relative to the general population; and in this setting empirical risk minimization over the training set may fail to yield rules that perform well at deployment. We propose a model of sampling bias called conditional ΓΓ-biased sampling, where observed covariates can affect the probability of sample selection arbitrarily much but the amount of unexplained variation in the probability of sample selection is bounded by a constant factor. Applying the distributionally robust optimization framework, we propose a method for learning a decision rule that minimizes the worst-case risk incurred under a family of test distributions that can generate the training distribution under ΓΓ-biased sampling. We apply a result of Rockafellar and Uryasev to show that this problem is equivalent to an augmented convex risk minimization problem. We give statistical guarantees for learning a model that is robust to sampling bias via the method of sieves, and propose a deep learning algorithm whose loss function captures our robust learning target. We empirically validate our proposed method in a case study on prediction of mental health scores from health survey data and a case study on ICU length of stay prediction.

Roshni Sahoo, Lihua Lei, Stefan Wager

Stanford University

Distribution Shift RobustnessDistributionally Robust OptimizationSelection BiasEmpirical Risk Minimization
AskPDF
Feb 17, 2022

Universality of empirical risk minimization

We study a general class of optimization problems with decision variable Θ∈Rp×k\boldsymbolΘ \in \mathbb{R}^{p \times k} and cost function which is the sum of nn terms, each dependent on Θ\boldsymbolΘ through the kk-dimensional projection Θ⊤xi\boldsymbolΘ^\top \boldsymbol{x}_i, where xi\boldsymbol{x}_i, i≤ni \leq n are i.i.d. random vectors. This setting is general enough to include examples of current interest in statistical physics, high-dimensional statistics, and statistical learning theory. We consider the proportional asymptotics n,p→∞n, p \to \infty, with n/p=Θ(1)n/p = Θ(1), and prove that, whenever there exists a minimizer satisfying a suitable generalization of a "delocalization" condition, the minimum value is universal. Namely, (for subgaussian xi\boldsymbol{x}_i) it depends on the distribution of xi\boldsymbol{x}_i only through its asymptotic mean and covariance. This delocalization condition is essentially necessary. Earlier universality results for such problems were limited to strongly convex loss functions. We derive applications of our theory to statistical learning and prove general universality results both for train and (under additional conditions) test error. In particular, we establish universality for vectors xi\boldsymbol{x}_i generated by random 1-layer neural networks (random features models) and first-order Taylor approximations of 2-layer networks (neural tangent models). Finally, we establish that the delocalization property holds for a class of statistical learning problems under a condition that is easy to verify.

Andrea Montanari, Basil Saeed

Department of Statistics and Department of Mathematics, Stanford University · Department of Electrical Engineering, Stanford University

Statistical Learning TheoryEmpirical Risk MinimizationHigh-Dimensional Asymptotics
AskPDF