Upper Bounds for Local Learning Coefficients of Three-Layer Neural Networks

Yuki Kurumadani

cs.LGMar 13, 2026

Upper Bounds for Local Learning Coefficients of Three-Layer Neural Networks

Authors: Yuki Kurumadani

Organizations: The University of Osaka, 1 Chome-3 Machikaneyamacho, Toyonaka, Osaka 560-8531, Japan

Abstract

Three-layer neural networks are known to form singular learning models, and their Bayesian asymptotic behavior is governed by the learning coefficient, or real log canonical threshold. Although this quantity has been clarified for regular models and for some special singular models, broadly applicable methods for evaluating it in neural networks remain limited. Recently, a formula for the local learning coefficient of semiregular models was proposed, yielding an upper bound on the learning coefficient. However, this formula applies only to nonsingular points in the set of realization parameters and cannot be used at singular points. In particular, for three-layer neural networks, the resulting upper bound has been shown to differ substantially from learning coefficient values already known in some cases. In this paper, we derive a formula for an upper bound on local learning coefficients at a class of singular realization parameters in three-layer neural networks. This formula can be interpreted as a counting rule under budget, demand, and supply constraints. In the non-polynomial real-analytic case, the formula applies in general settings, whereas in the polynomial case it applies under the restriction that the true distribution has no hidden units. In particular, our result covers activation functions such as the swish function and also includes polynomial activation functions under the above restriction, thereby extending previous results to a broader class of activation functions. We further show that, when the input dimension is one, the numerical value given by the right-hand side of our upper-bound formula agrees with the previously known learning coefficient, thereby providing a useful comparison with known exact results. Our result also provides a systematic perspective on how the weight parameters of three-layer neural networks affect the learning coefficient.

Figures & tables

Activation function $\sigma(x)$	$\{s\mid\sigma^{(s)}(0)\neq 0\}$
$e^{x}-1,~xe^{x},~x\tanh(\log(1+e^{x}))$	$(1,2,3,4,5,\ldots)$
$x/(1+e^{-x}),~x\Phi(x)$	$(1,2,4,6,8,\ldots)$
$\tanh(x),\sin(x),\arctan(x)$	$(1,3,5,7,9,\ldots)$

Table 2: Non-polynomial activation functions considered

Figure 1: Upper bounds of the local learning coefficients at $P_{1}$ and $P_{2}$ for $(H^{*},N)=(2,1)$

Panel of Figure 2 — Figure 2: Procedure of the coordinate transformations $\pi$ in the proof of the Main Theorem

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Procedure of the coordinate transformations $\pi$ carried out in this example

Two-Layer Neural Networks

See all 14 papers

Aug 20, 2026

Exact Algebraic Computation of Learning Coefficients for Two-Dimensional Singular Models

Classical information criteria such as the Bayesian Information Criterion (BIC) rely on regularity assumptions that break down for singular models, leading to incorrect model selection in settings such as deep learning. The Widely Applicable Bayesian Information Criterion (WBIC) relies on local learning coefficients

λ

, which in the analytic case coincides with local Real Log Canonical Thresholds (RLCT) of the Kullback-Leibler divergence of the model, to capture correct marginal likelihood asymptotics. Exact computation of the learning coefficients has been limited to special cases, and only sampling-based estimation methods are generally applicable. We present the first deterministic algorithm that computes local RLCTs exactly for any two-dimensional model whose Kullback-Leibler distance is contact equivalent to a polynomial, derive a bound on its complexity, and demonstrate its effectiveness for a broad class of models, with applications including polynomial neural networks. Beyond providing ground truth to calibrate sampling-based estimators, exact computation reveals algebraic structure in learning coefficients that sampling cannot and out-speeds it in the shallow regime.

Grégoire Sergeant-Perthuis, Elias Tsigaridas, Jules Tsukahara

CQSB, Sorbonne Université · Ouragan Team, INRIA · Ouragan team, Inria Paris

Singular Learning Theory

Ask PDF

Jul 8, 2026

A law of robustness for two-layer neural networks with arbitrary weights

Bubeck, Li and Nagaraj conjectured that, for generic data, any two-layer neural network with

m

neurons that fits

n

noisy labels must have Lipschitz constant at least of order

\sqrt{n/m}

, with no restriction on the size of the weights. Bubeck and Sellke proved a universal version of this law for Lipschitz-parameterized classes, but under a polynomial bound on the parameters; at depth three that boundedness hypothesis is genuinely necessary. The two-layer unbounded-weight case requires a different argument. We prove the conjectured law, up to one logarithmic factor, for every continuous piecewise-linear activation, in particular for ReLU networks. For data drawn uniformly from

\mathbb{S}^{d-1}

d\ge3

, or from

N(0,I_d/d)

, labels in

[-1,1]

with noise level

σ^2>0

, and any width-

m

two-layer network with arbitrary real weights, biases and affine skip connection, fitting the data

\varepsilon

below the noise floor forces

\mathrm{Lip}(f)\ge c\,\varepsilon\sqrt{n/(\bar m\log(C\bar m nd/\varepsilon))}

\bar m=(K-1)m+1

, with high probability. A realized-kink-count version holds on the same event: every realized two-layer piecewise-linear function with

k(f)\le n

distinct kink hyperplanes obeys the bound with

\bar m

replaced by

k(f)+1

, irrespective of how many redundant hidden units parameterize it. The proof replaces parameter-space covering, impossible for unbounded weights, by a function-space covering. The central deterministic ingredient is a rigidity lemma: on

B_2

, and on

\mathbb{S}^{d-1}

for

d\ge3

, the coefficient of each canonical kink is controlled by the Lipschitz constant of the realized function, because kinks on distinct hyperplanes cannot cancel at generic points. Rigidity genuinely fails at

d=2

, and an explicit two-layer ReLU interpolant with

O(1)

Lipschitz constant at width

2n

matches the law at the overparameterized endpoint.

Yitzchak Shmalo

Neural Network Memorization Multilayer Perceptrons Lipschitz Neural Networks Neural Network Generalization Neural Network Robustness Adversarial Robustness+1

Ask PDF

Jul 26, 2026

A Statistical Difference between Single-Layer Learning and Hierarchical Learning in Wide Neural Networks

Hierarchical neural networks are widely used in artificial intelligence, yet their mathematical properties remain incompletely understood. In the infinite-width limit, two different theoretical frameworks have been proposed. One reduces deep learning to kernel regression with a fixed kernel by assuming that the parameters remain close to their initialization, whereas the other allows the parameters to move away from their initialization, requiring the kernel itself to be optimized. In this paper, we study a three-layer neural network with a finite but large number of hidden units. We show that training the input-to-hidden weights yields a smaller generalization error than keeping them fixed. Furthermore, the latter setting exhibits singularities in the parameter space, whereas the former does not. These findings indicate that singularities play an essential role even in wide neural networks.

Sumio Watanabe

RIKEN Center for Advanced Intelligence Project 1-4-1 Nihonbashi, Chuo-ku, 103-0027, Tokyo, Japan

Neural Network Generalization Hierarchical Representation Learning Neural Network Training Dynamics

Ask PDF

Abstract

Two-Layer Neural Networks

Explore similar work

Exact Algebraic Computation of Learning Coefficients for Two-Dimensional Singular Models

A law of robustness for two-layer neural networks with arbitrary weights

A Statistical Difference between Single-Layer Learning and Hierarchical Learning in Wide Neural Networks