Stable initialization without the CLT

Simon Kuang; Kyle Chickering; Xinfan Lin

cs.LGSep 24, 2026

Stable initialization without the CLT

Authors: Simon Kuang, Kyle Chickering, Xinfan Lin

Organizations: Department of Mechanical and Aerospace Engineering University of California, Davis

Abstract

Successful training of deep neural networks is highly dependent on the distribution of the initial weights. If the weights are too large, network training blows up; if they are too small, the model fails to learn features. Stable initialization is the optimal moderation between these two extremes. The conventional theory of random networks uses the Central Limit Theorem to control inter-neuron dependencies, which introduces distributional approximation error and coupling between layers. For networks with sine activations, we derive the uniform-phase initialization, which obviates distributional approximation and fully decouples the layers. Ours is the first work to use the sine function's periodic symmetry. Models trained with the uniform-phase initialization outperform the state of the art in neural representation tasks like image and audio fitting. We find that our untuned models are competitive with the best-tuned baselines from previous work and support $μ$ P width scaling.

Figures & tables

Figure 1: Image fitting on the Cameraman image. Columns: the ground truth followed by one column per initialization, with the evaluation SNR beneath each. Top row: $128\times 128$ training image. Middle row: $512\times 512$ test image. Bottom row: zoomed region (red square in the middle row).

Figure 2: Grayscale and color

\omega_{0}

sweep. Test MSE versus input-layer frequency

\omega_{0}

; rows are the grayscale, RGB, and CIELAB experiments, and columns are the four images in each. A star marks the minimum of every sweep, and the dashed red line marks the uniform-phase MM test MSE. Each panel has its own logarithmic vertical scale; for absolute errors, see Tables 3 , 4 , and 5 .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Full singular value spectrum of the end-to-end Jacobian $\mathbf{J}=\partial\mathbf{h}_{L}/\partial\mathbf{h}_{1}$ as a function of depth. Four initialization schemes are compared: uniform-phase MM (moment-matched to CVP26 $\sigma_{a}{=}0$ , $L{=}4$ ), CVP26 $\sigma_{a}{=}0$ , CVP26 $\sigma_{a}{=}1$ , and the original SM20 initialization. Each spectrum is averaged over five independently initialized networks, with 10 sample points on $[-\pi,\pi]$ .

Network	Train MSE	Eval MSE	SNR (dB)
SIREN (uniform-phase MM)	$\mathbf{0.000032}$	$\mathbf{0.002706}$	$\mathbf{14.89}$
SIREN (CVP26 $\sigma_{a}\!=\!0$ )	$0.000127$	$0.002967$	$14.49$
SIREN (SM20)	$0.000000$	$0.017428$	$\phantom{0}6.80$
ReLU (Kaiming)	$0.001768$	$0.005201$	$12.05$
SiLU (Xavier)	$0.007591$	$0.011650$	$\phantom{0}8.55$
GeLU (Xavier)	$0.010906$	$0.015013$	$\phantom{0}7.45$

Appendix

Table 1: Quantitative results for the Cameraman image-fitting experiment (§ 4.1 ). Train MSE is computed on the

128\times 128

grid; eval MSE and SNR are computed on the

512\times 512

grid. Best values are in bold.

Network	Train MSE	Eval MSE	SNR (dB)
SIREN (uniform-phase MM)	$\mathbf{0.004117}$	$\mathbf{0.007824}$	$\mathbf{10.28}$
ReLU (Kaiming)	$0.006713$	$0.010689$	$\phantom{0}8.92$
GeLU (Xavier)	$0.017129$	$0.021263$	$\phantom{0}5.94$
SiLU (Xavier)	$0.018928$	$0.023064$	$\phantom{0}5.58$
SIREN (CVP26 $\sigma_{a}\!=\!0$ )	$0.033747$	$0.037765$	$\phantom{0}3.44$
SIREN (SM20)	$0.062279$	$0.068359$	$\phantom{0}0.86$

Appendix

Table 2: Quantitative results for the narrow (

N=16

) Cameraman experiment (§ 4.1 ).

Figure 4: Image fitting with narrow networks ( $N=16$ ). Layout as in figure 1 .

Figure 5: Audio fitting on the Bach recording. One row per initialization, with the ground truth in gray and the prediction in blue. Left: the full waveform, drawn as a per-pixel minimum/maximum envelope of all

308{,}207

samples. Right: the first

1000

samples, on its own vertical scale because the recording opens in near-silence. The evaluation SNR is given in each right-hand panel.

Figure 6:

\mu

P learning-rate transfer, grayscale Cameraman, geometrically time-averaged train MSE (§ 4.5 ). Rows: initialization (uniform-phase MM and the three SIREN baselines at their best

\omega_{0}

); columns:

\mu

P LR,

\mu

P LR+LL, and standard scaling. One curve per width, shaded from light (

N=8

) to dark (

N=512

), with a star at each width’s best learning rate.

Image	uniform-phase MM	SM20	CVP26 $\sigma_{a}{=}0$	CVP26 $\sigma_{a}{=}1$
Brick	0.002852	0.002317 ( $\omega{=}20.6$ )	0.002408 ( $\omega{=}82.1$ )	0.002381 ( $\omega{=}32.7$ )
Camera	0.002695	0.002763 ( $\omega{=}20.6$ )	0.002912 ( $\omega{=}130$ )	0.002855 ( $\omega{=}51.8$ )
Gravel	0.009063	0.007982 ( $\omega{=}32.7$ )	0.007994 ( $\omega{=}130$ )	0.007937 ( $\omega{=}51.8$ )
Clock	0.000036	0.000055 ( $\omega{=}20.6$ )	0.000050 ( $\omega{=}82.1$ )	0.000042 ( $\omega{=}20.6$ )

Appendix

Table 3: Grayscale sweep: best test MSE per method. The optimal

\omega_{0}

from the grid search is shown in parentheses. Bold entries are the best per row. Visual comparisons: figure 7 .

Image	uniform-phase MM	SM20	CVP26 $\sigma_{a}{=}0$	CVP26 $\sigma_{a}{=}1$
Hubble	0.003773	0.003758 ( $\omega{=}32.7$ )	0.003715 ( $\omega{=}130$ )	0.003808 ( $\omega{=}51.8$ )
Skin	0.005784	0.006044 ( $\omega{=}20.6$ )	0.006083 ( $\omega{=}130$ )	0.005870 ( $\omega{=}51.8$ )
Astronaut	0.004027	0.004076 ( $\omega{=}32.7$ )	0.004276 ( $\omega{=}130$ )	0.004191 ( $\omega{=}51.8$ )
Coffee	0.002728	0.002785 ( $\omega{=}20.6$ )	0.002847 ( $\omega{=}82.1$ )	0.002874 ( $\omega{=}51.8$ )

Appendix

Table 4: Color sweep: best test MSE per method. Conventions as in Table 3 . Visual comparisons: figure 8 .

Image	uniform-phase MM	SM20	CVP26 $\sigma_{a}{=}0$	CVP26 $\sigma_{a}{=}1$
Hubble	18.148	18.505 ( $\omega{=}13.0$ )	18.675 ( $\omega{=}82.1$ )	18.692 ( $\omega{=}51.8$ )
Skin	45.172	134.00 ( $\omega{=}13.0$ )	145.37 ( $\omega{=}8208$ )	141.75 ( $\omega{=}8.21$ )
Astronaut	20.011	83.61 ( $\omega{=}32.7$ )	72.30 ( $\omega{=}82.1$ )	84.91 ( $\omega{=}20.6$ )
Coffee	15.658	81.19 ( $\omega{=}20.6$ )	82.15 ( $\omega{=}32.7$ )	88.28 ( $\omega{=}32.7$ )

Appendix

Table 5: CIELAB sweep: best test MSE per method. MSE is computed in CIELAB coordinates. Conventions as in Table 3 . Visual comparisons: figure 9 .

Figure 7: Grayscale best- $\omega_{0}$ visual comparisons: Brick, Camera, Gravel, and Clock.

Figure 8: RGB best- $\omega_{0}$ visual comparisons: Hubble, Skin, Astronaut, and Coffee. Same layout as figure 7 .

Figure 9: CIELAB best- $\omega_{0}$ visual comparisons: Hubble, Skin, Astronaut, and Coffee. Predictions are converted from CIELAB to RGB for display; the reported MSE values are computed in CIELAB coordinates. Same layout as figure 7 .

Neural Network Initialization

Neural network initialization studies how starting weights shape signal propagation, optimization, generalization, and reproducibility.

See all 49 papers

Jacobi-Anger Method for Deterministic Initialization in Implicit Neural Representation
Jun 4, 2026Mohammed Alsakabi, Kejia Hu, John M. Dolan +1Implicit Neural Representations Neural Network Initialization
Department of Electrical and Computer Engineering, College of Engineering · The Robotics Institute, School of Computer Science Carnegie Mellon University, Pittsburgh, PA, USA
Optimal Condition for Initialization Variance in Deep Neural Networks: An SGD Dynamics Perspective
Aug 18, 2025Hiroshi Horii, Sothea HasFokker–Planck Equation Neural Network Initialization
SU · KHM
Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias
May 27, 2026Mohua Das, Pierfrancesco Beneventano, Shibshankar Dey +2Neural Network Generalization Neural Network Initialization
MIT · Northwestern University

Jun 4, 2026J/K move · Enter open · S save

Jacobi-Anger Method for Deterministic Initialization in Implicit Neural Representation

Mohammed Alsakabi, Kejia Hu, John M. Dolan, Ozan K. Tonguz

Department of Electrical and Computer Engineering, College of Engineering · The Robotics Institute, School of Computer Science Carnegie Mellon University, Pittsburgh, PA, USA

Implicit Neural Representations Neural Network Initialization ML Reproducibility Sinusoidal Representation Networks

Open paper AskPDF arXiv

Existing implicit neural representation (INR) approaches suffer from stochastic initialization that does not guarantee consistent or high-quality performance across runs, with variations reaching more than 2.5 dB (~78%) in image regression. This variation is problematic for scientific computing and simulation, where result reproducibility is crucial. To address this problem, we present Jacobi-Anger Sinusoidal Representation Network (JA-SIREN), a deterministic initialization scheme for sinusoidal networks grounded in classical spectral analysis. By computing the Discrete Sine Transform (DST) of the target signal and leveraging the Jacobi-Anger expansion, we derive closed-form weights for a two-layer sinusoidal MLP that analytically match the network's initial spectral response to the target signal, requiring no random seed or additional hyperparameter tuning. On the Kodak dataset, JA-SIREN achieves a mean PSNR of 67.18 dB, a 21.30 dB improvement over the best baseline. This is achieved with zero run-to-run variance, confirming that spectrally-informed initialization is a more effective and reproducible alternative to stochastic initialization for sinusoidal INRs.

Abstract

Neural Network Initialization

Explore similar work

Jacobi-Anger Method for Deterministic Initialization in Implicit Neural Representation

Optimal Condition for Initialization Variance in Deep Neural Networks: An SGD Dynamics Perspective

Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias