Para-Gaussian Approximations of Random Deep Neural Networks

Mihai Nica; Janosch Ortmann

Figure 1: The distribution of a single neuron in a randomly initialized ReLU network (

\varphi(x)=\sqrt{2}\max\{x,0\}

, width

n=100

, one input) at layers

\ell=2,4,8

, estimated from

2^{24}

Monte Carlo samples per layer. Top row: Comparing the Monte Carlo estimates (grey), to the the Gaussian approximation

\mathcal{N}(0,1)

(yellow) and the para-Gaussian law (blue) with density

\frac{1}{\sqrt{2\pi}}e^{-x^{2}/2}\big[1+\frac{\kappa_{4}}{4!}\mathrm{He}_{4}(x)\big]

, where

\mathrm{He}_{4}(x)=x^{4}-6x^{2}+3

and

\kappa_{4}=3\big[(1+5/n)^{\ell}-1\big]

. Bottom row: the difference between each approximation and the Monte Carlo density, estimated by a histogram with bin width

0.1

, with both approximations averaged over the same bins. The legend shows the binned total variation distance

d_{TV,\mathrm{bin}}=\frac{1}{2}\sum_{k}\left|p_{k}-q_{k}\right|

, where

p_{k}

is the fraction of Monte Carlo samples in the

k

-th bin,

q_{k}

is the mass the approximation assigns to that bin. Notice that the deviation from the Gaussian grows with depth. Theorem 1.13 and Theorem 1.18 generalize this picture to multiple inputs and outputs to obtain the joint para-Gaussian distribution that approximates the distribution of multiple neurons in a neural network.

Figure 2: Linear network. Fourth cumulants. Quantities related to the fourth cumulant of a linear network (

\varphi(x)=x

, width

n=32

) for depths up to

\ell=32

. Each plot compares three solutes: 1. the exact values from Lemma 2.3 , 2. the values obtained from the

1/n

recursion of Theorem 1.18 , and 3. from Monte Carlo simulations. We use two orthogonal unit inputs (

K=I_{2}

) and two different neurons, so that each statistic isolates one exact coefficient; here

z_{\alpha,k}

denotes the output of neuron

k

on input

\alpha

, with

\alpha\in\{1,2\}

and

k\in\{1,2\}

. The plots show (a) Same input:

n\mathbf{Cov}\big(z_{1,1}^{2},z_{1,2}^{2}\big)=C^{\ell}(1,1;1,1)

, exactly

2v_{\ell}+w_{\ell}=n\big[(1+\frac{2}{n})^{\ell}-1\big]

and

2\ell

according to the

1/n

recursion. (b) Orthogonal inputs:

n\mathbf{Cov}\big(z_{1,1}z_{2,1},\,z_{1,2}z_{2,2}\big)=C^{\ell}(1,2;1,2)

, exactly

v_{\ell}

and

\ell

according to the

1/n

recursion. (c) Orthogonal inputs:

n\mathbf{Cov}\big(z_{1,1}^{2},z_{2,2}^{2}\big)=C^{\ell}(1,1;2,2)

, exactly

w_{\ell}

and

0

according to the

1/n

recursion. Grey: exact solution for

C^{\ell}

from Lemma 2.3 ; red:

O(1/n)

approximation from the recursion of Theorem 1.18 ; blue: Monte Carlo over

2^{16}

networks, with

\pm 1

standard error computed from batch means. The pure neural network Gaussian (NNGP) approxiamtion predicts

0

for all these quantities. Note that the

1/n

recursion is accurate while

\ell/n

is small but gets worse when

\ell/n

is of order one. Additionally, the coefficient

w_{\ell}

of the third pairing

K(\alpha,\alpha^{\prime})K(\beta,\beta^{\prime})

is of order

1/n

, so this pairing contributes to the fourth cumulant

\kappa=\frac{1}{n}C^{\ell}

only at order

n^{-2}

and is therefore absent from the

1/n

recursion but present in the exact solution and in the simulations.

Figure 3: Linear network. Two neurons outputs, one input. Uncorrelated but not independent. The distribution of two different neurons of a linear network (

\varphi(x)=x

, width

n=32

) receiving the same unit norm input, i.e.

A={\alpha}

and we look at the joint distribution of the random variables

X=z_{\alpha,1},Y=z_{\alpha,2}

. These random variables are uncorrelated, but not independent. Left three panels: the ratio

f(x,y)/\big[f_{1}(x)f_{1}(y)\big]

of the joint density of the two neurons’ outputs to the product of their marginal densities at depth

\ell=4

, which the NNGP predicts to be identically

1

. The para-Gaussian law predicts this as

1+\frac{\kappa}{4}(x^{2}-1)(y^{2}-1)

, where

\kappa=\kappa(x,x,y,y)

is the cross-neuron fourth cumulant. We show both the result

\kappa=\frac{2\ell}{n}

from the

1/n

-approximated-recursion and with the exact value

\kappa=\frac{2v_{\ell}+w_{\ell}}{n}

, next to Monte Carlo over

2^{22}

neuron pairs. Right : the cumulant

n\mathbf{Cov}(x^{2},y^{2})

against depth, with the

1/n

recursion

2\ell

, the exact value

2v_{\ell}+w_{\ell}

and Monte Carlo (

\pm 1

standard error).

Figure 4: Linear network. Two neurons, two inputs. Dependence on the input correlation. Cross-neuron fourth cumulants of a linear network (

\varphi(x)=x

, width

n=32

) for two unit norm inputs with correlation

\rho=K(1,2)

, against

\rho

for depths

\ell=1,2,4,8

; here

z_{\alpha,k}

denotes the output of neuron

k

on input

\alpha

where

\alpha\in\{1,2\}

and

k\in\{1,2\}

. (a)

n\mathbf{Cov}\big(z_{1,1}^{2},z_{2,2}^{2}\big)=C^{\ell}(1,1;2,2)

, the dependence between the first neuron on input

1

and the second neuron on input

2

:

2\ell\rho^{2}

according to the

1/n

recursion, exactly

2\rho^{2}v_{\ell}+w_{\ell}

. (b)

n\mathbf{Cov}\big(z_{1,1}z_{2,1},\,z_{1,2}z_{2,2}\big)=C^{\ell}(1,2;1,2)

, the dependence between the products of each neuron’s outputs on the two inputs:

\ell(1+\rho^{2})

according to the

1/n

recursion, exactly

(1+\rho^{2})v_{\ell}+\rho^{2}w_{\ell}

. Dashed lines:

1/n

recursion of Theorem 1.18 ; solid lines: exact solution from Lemma 2.3 ; markers: Monte Carlo over

2^{22}

neuron pairs per value of

\rho

(

\pm 1

standard error). The Gaussian prediction (NNGP) predicts

0

in all cases. At

\rho=0

the

1/n

recursion predicts no dependence in (a), while the exact cumulant

w_{\ell}

is positive and clearly visible at

\ell=8

.

Figure 5: Linear network. One output neuron, two inputs. The joint density. Representation of the joint density of the random vector

\big(z^{\ell}_{1,1},z^{\ell}_{2,1}\big)

consisting of the outputs of the first neuron of layer

\ell=4

for two unit norm inputs

A=\{1,2\}

with correlation

K^{0}(1,2)=0.5

, in a linear network (

\varphi(x)=x

, width

n=32

). The left and middle panels show the ratio of the density to the Gaussian (NNGP) predicted density

\mathcal{N}(0,K^{0})

(numbers

>1

indicate a higher density, and numbers

<1

a lower density, than the pure Gaussian prediction). Left: the para-Gaussian density of Definition 1.1 , with covariance

K^{0}

and fourth cumulant tensor

\kappa^{\ell}

for a single neuron (

i=j=k=l=1

), where

C^{\ell}

is the exact solution ( 19 ) with

v_{\ell},w_{\ell}

given by ( 20 ). Middle: Monte Carlo over

2^{22}

samples, with probabilities compared bin by bin; hatched bins contain fewer than

100

samples. Right: the difference between the two ratios, Monte Carlo minus para-Gaussian, which is the difference of the two densities measured relative to the Gaussian density. The two agree closely in the bulk of the distribution. The difference is close to

0

inside the inner ellipse and of order

0.1

in the ring between the ellipses. The largest differences are in the tails, where the para-Gaussian ratio keeps growing while the true density does not. The dashed curves are where the para-Gaussian ratio equals

1

, i.e. where the para-Gaussian and Gaussian densities agree.

Figure 6: ReLU network. Fourth cumulants. Quantities related to the fourth cumulants of a ReLU network (

\varphi(x)=\sqrt{2}\max\{x,0\}

, width

n=64

) for depths up to

\ell=32

, for two different neurons;

z_{\alpha,k}

denotes the output of neuron

k

on input

\alpha

. (a) One unit norm input:

n\mathbf{Cov}\big(z_{1,1}^{2},z_{1,2}^{2}\big)=C^{\ell}(1,1;1,1)

, which equals

n\big[(1+\frac{5}{n})^{\ell}-1\big]

exactly (Lemma 2.5 ) and

5\ell

according to the

1/n

recursion. (b) , (c) Two unit norm inputs that are orthogonal at layer

0

(

K=I

): (b)

n\mathbf{Cov}\big(z_{1,1}z_{2,1},\,z_{1,2}z_{2,2}\big)=C^{\ell}(1,2;1,2)

and (c)

n\mathbf{Cov}\big(z_{1,1}^{2},z_{2,2}^{2}\big)=C^{\ell}(1,1;2,2)

. For several inputs no exact formula is available, and we show the

1/n

recursion ( 14 ), evaluated numerically. Red:

1/n

recursion; grey: exact (one input only); blue: Monte Carlo over

2^{15}

networks, with

\pm 1

standard error computed from batch means. The Gaussian prediciton (NNGP) is

0

in every panel. The

1/n

recursion is accurate while

\ell/n

is small, and underestimates the cumulants once

\ell/n

is of order one.

Figure 7: ReLU network. Two neuron outputs, one input. Uncorrelated but not independent. The distribution of two different neurons of a ReLU network (

\varphi(x)=\sqrt{2}\max\{x,0\}

, width

n=64

) receiving the same unit norm input, i.e.

A=\{\alpha\}

and we look at the joint distribution of the random variables

X=z^{\ell}_{\alpha,1}

,

Y=z^{\ell}_{\alpha,2}

. These random variables are uncorrelated, but not independent. Left three panels: the ratio

f(x,y)/\big[f_{1}(x)f_{1}(y)\big]

of the joint density of the two neurons’ outputs to the product of their marginal densities at depth

\ell=4

, which the NNGP predicts to be identically

1

. The para-Gaussian law predicts this as

1+\frac{\kappa}{4}(x^{2}-1)(y^{2}-1)

, where

\kappa=\kappa(x,x,y,y)

is the cross-neuron fourth cumulant. We show both the result

\kappa=\frac{5\ell}{n}

from the

1/n

-approximated recursion and the exact value

\kappa=\big(1+\frac{5}{n}\big)^{\ell}-1

from Lemma 2.5 , next to Monte Carlo over

2^{21}

neuron pairs. Right : the cumulant

n\mathbf{Cov}(x^{2},y^{2})

against depth, with the

1/n

recursion

5\ell

, the exact value

n\big[\big(1+\frac{5}{n}\big)^{\ell}-1\big]

and Monte Carlo (

\pm 1

standard error).

Figure 8: ReLU network. Two neurons, two inputs. Dependence on the input correlation. Cross-neuron fourth cumulants of a ReLU network (

\varphi(x)=\sqrt{2}\max\{x,0\}

, width

n=64

) for two unit norm inputs with correlation

\rho=K^{0}(1,2)

at layer

0

, against

\rho

for depths

\ell=1,2,4,8

. Here

z_{\alpha,k}

denotes the output of neuron

k

on input

\alpha

where

\alpha\in\{1,2\}

and

k\in\{1,2\}

. (a)

n\mathbf{Cov}\big(z_{1,1}^{2},z_{2,2}^{2}\big)=C^{\ell}(1,1;2,2)

, the dependence between the first neuron on input

1

and the second neuron on input

2

. (b)

n\mathbf{Cov}\big(z_{1,1}z_{2,1},\,z_{1,2}z_{2,2}\big)=C^{\ell}(1,2;1,2)

, the dependence between the products of each neuron’s outputs on the two inputs. Solid lines:

1/n

recursion of Theorem 1.18 , evaluated numerically, since for ReLU there is no exact formula with two distinct inputs. Stars represent the exact value

n\big[(1+\frac{5}{n})^{\ell}-1\big]

at

\rho=1

, where the two inputs coincide (Lemma 2.5 ). Monte Carlo over

2^{20}

neuron pairs per value of

\rho

(

\pm 1

standard error) is shown as cirlces. The Gaussian prediction (NNGP) predicts

0

in all cases. Unlike the linear case, the cumulant in (a) is positive at

\rho=0

already according to the

1/n

recursion, since ReLU layers make inputs that are orthogonal at layer

0

positively correlated. The

1/n

recursion underestimates the cumulants for large values of

\ell/n

.

Figure 9: ReLU network. One neuron, two inputs. The joint density as depth grows. A representation of the joint density of the random vector

\big(z^{\ell}_{1,1},z^{\ell}_{2,1}\big)

, the outputs of the first neuron of layer

\ell

on two unit norm inputs with correlation

K(1,2)=0.5

at layer

0

, in a ReLU network (

\varphi(x)=\sqrt{2}\max\{x,0\}

, width

n=64

), for depths

\ell=2,4,8

(columns). Each panel shows the ratio of a density to the NNGP Gaussian density of

\mathcal{N}(0,K^{\ell})

, on a common colour scale. Top row : the para-Gaussian density of Definition 1.1 , with covariance

K^{\ell}+\frac{1}{n}\sigma^{\ell}

from ( 13 ) and fourth cumulant tensor computed with

C^{\ell}

from the

1/n

recursion ( 14 ); all Gaussian expectations are evaluated numerically. Bottom row: Monte Carlo over

2^{21}

samples per depth, with probabilities compared bin by bin; hatched bins contain fewer than

100

samples. No exact formula is available for comparison in this case.

Para-Gaussian Approximations of Random Deep Neural Networks

Abstract

Figures & tables

Neural Network Approximation Theory

Depth Separation for Neural Networks under Gaussian Measure

Universality in Deep Neural Networks: An approach via the Lindeberg exchange principle

Phase Transitions in the Fluctuations of Functionals of Random Neural Networks

Abstract

Neural Network Approximation Theory

Explore similar work

Depth Separation for Neural Networks under Gaussian Measure

Universality in Deep Neural Networks: An approach via the Lindeberg exchange principle

Phase Transitions in the Fluctuations of Functionals of Random Neural Networks