Why Heavy-Tailed Weight Spectra Predict Model Quality

Joseph Wilson; Chris van der Heide; Liam Hodgkinson; Zhichao Wang; Fred Roosta; Michael W. Mahoney

stat.MLOct 9, 2026

Why Heavy-Tailed Weight Spectra Predict Model Quality

Authors: Joseph Wilson, Chris van der Heide, Liam Hodgkinson, Zhichao Wang, Fred Roosta, Michael W. Mahoney

Organizations: School of Mathematics and Physics University of Queensland · Dept. of Electrical and Electronic Engineering University of Melbourne · School of Mathematics and Statistics University of Melbourne · ICSI and Department of Statistics University of California, Berkeley · CIRES and School of Mathematics and Physics University of Queensland · ICSI, LBNL and Department of Statistics University of California, Berkeley

Abstract

A solid understanding of the predictive success of deep neural networks (DNNs) remains elusive. For example, while many data-dependent metrics exist that seek to predict model quality of DNNs, these metrics may be expensive to compute for large datasets and/or may be impossible to compute when training datasets are not released. A practical and effective predictor of model quality involves analyzing DNN weight matrices, as prior work has established that heavy-tailed spectral distributions in trained weight matrices can be strongly predictive of model quality. However, such metrics lack a rigorous statistical justification. In this work, we aim to understand why these metrics perform well, from a statistical perspective. To do so, we employ a probabilistic framework to derive the marginal likelihood for a DNN, evaluated using only the final layer weights. Across large-scale convolutional DNNs and LLMs, we find that this marginal likelihood predicts model quality. To understand the role heavy-tailed spectra play in predictive performance, we derive the limiting log-marginal likelihood for spectra that follow the (heavy-tailed) High-Temperature Marchenko-Pastur (HTMP) distribution. We show that this limiting quantity is convex in a heaviness parameter, and we derive an optimal heaviness that empirically predicts over-fitting for large-scale DNNs.

Figures & tables

Panel of Figure 1 — Figure 1: Plots of BFE- W versus test accuracy, for (left) torchvision models trained on ImageNet-1K, (middle) Pythia models, and (right) Qwen3.5 models. Test dataset is given in brackets.

Panel of Figure 2 — Figure 2: The ESD (gray) and fitted $\textbf{HTMP}_{\kappa,\gamma,\beta,\tau}$ distribution (red dotted lines), for the final-layer weights of (top) ResNet50 trained on CIFAR-100, and (bottom) Pythia- $12B$ trained on Pile.

Panel of Figure 3 — Figure 3: Test CE, BFE- W and $\mathcal{F}_{\infty}^{\lambda}$ , vs. epochs. Model quality during training is highly correlated with BFE. Further, $\mathcal{F}_{\infty}^{\lambda}$ closely approximates BFE- W .

Panel of Figure 4 — Figure 4: Test error, BFE- W and $\mathcal{F}_{\infty}^{\lambda}$ vs. fitted $\kappa/\beta$ and $\kappa$ . For CV, $h^{*}$ (purple cross) aligns with minimum error. For LLMs, $h^{*}$ is attained early, and increasing $\kappa$ lowers test error.

Figure 5: Fitted

\textbf{HTMP}_{\gamma,\kappa,\beta,\tau}

parameters at each epoch, for ResNet50 (trained on CIFAR-10 and CIFAR-100), Pythia-1B and Pythia-12B models. All weights are final-layer weights, except for ResNet50 CIFAR-10, where flattened convolutional weight at the third layer is used.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: For ResNet18, trained on permuted CIFAR-10 dataset, we have the (top left) ESD (gray) and fitted $\textbf{HTMP}_{\kappa,\gamma,\beta,\tau}$ distribution (red dotted lines), (top right) test CE, BFE- W and $\mathcal{F}_{\infty}^{\lambda}$ , vs. epochs, where the epoch at which training accuracy reaches $0.99$ is indicated with a vertical dotted line, (bottom left) test CE, BFE- W and $\mathcal{F}_{\infty}^{\lambda}$ , vs. $h=\kappa/\beta$ for interpolating epochs, and (bottom right) fitted HTMP parameters. Even for a task where there is no feature learning, our framework holds surprisingly well. Here, the weight matrix used for analysis is the third-layer convolutional weight.

Panel of Figure 7 — Figure 7: Test CE, BFE- W and $\mathcal{F}_{\infty}^{\lambda}$ , vs. epochs. Model quality during training is highly correlated with BFE. Further, $\mathcal{F}_{\infty}^{\lambda}$ closely approximates BFE- W . Extra experimental results to accompany Figure 3 .

Panel of Figure 8 — Figure 8: Test error, BFE- W and $\mathcal{F}_{\infty}^{\lambda}$ vs. fitted $\kappa/\beta$ and $\kappa$ . For CV, $h^{*}$ (purple cross) aligns with minimum error. For LLMs, $h^{*}$ is attained early, and increasing $\kappa$ lowers test error. Extra experimental results to accompany Figure 4 .

Figure 9: Fitted

\textbf{HTMP}_{\gamma,\kappa,\beta,\tau}

parameters at each epoch. Extra experimental results to accompany Figure 5 .

Panel of Figure 10 — Figure 10: The ESD (gray) and fitted $\textbf{HTMP}_{\kappa,\gamma,\beta,\tau}$ distribution (red dotted lines), for (top) the third-layer convolutional weight of ResNet50 trained on CIFAR-10, and (bottom) the EmbedOut weight of Pythia- $1B$ trained on Pile.

Panel of Figure 11 — Figure 11: Correlation of BFE- W with model quality for deeper weights, across (top) all weights of ResNet50 on CIFAR-100, and (bottom) for Dense and QueryKey matrices at every layer for all Pythia models. For the Pythia models, model quality is measured by log perplexity for the LAMBADA test dataset. We also provide the $\lambda^{*}=d_{l}/||{\mathbf{W}}_{l,t^{*}}||_{F}^{2}$ value for all ResNet50 weights. We see that a significant proportion of deeper weights show a very strong correlation with model quality.

Model	BFE-W	$\\|{\mathbf{W}}\\|_{F}^{2}$	$\\|{\mathbf{W}}\\|_{\infty}^{2}$	$\hat{\alpha}$	$\\|{\mathbf{W}}\\|_{2\alpha}^{2\alpha}$	$\alpha$	$\lambda$
DenseNet	$-0.80$	$-1.00$	$-1.00$	$-1.00$	$-1.00$	$0.00$	$0.40$
ResNet	$-0.90$	$-1.00$	$-1.00$	$-1.00$	$-1.00$	$-0.70$	$1.00$
VGG	$0.81$	$-0.90$	$-0.98$	$-0.98$	$-0.98$	$0.95$	$0.95$
Combined	$0.48$	$-0.88$	$-0.89$	$-0.87$	$-0.87$	$0.54$	$0.86$

Appendix

Table 1: Spearman

\rho

correlation of model quality metrics with top-5 ImageNet test accuracy for Legacy models. Combined pools all models in this type. The

\alpha

and

\lambda

are the fitted PL and ET-PL parameters.

Model	BFE-W	$\\|{\mathbf{W}}\\|_{F}^{2}$	$\\|{\mathbf{W}}\\|_{\infty}^{2}$	$\hat{\alpha}$	$\\|{\mathbf{W}}\\|_{2\alpha}^{2\alpha}$	$\alpha$	$\lambda$
ConvNeXt	$-0.80$	$1.00$	$1.00$	$1.00$	$1.00$	$0.40$	$-0.80$
DenseNet	$-0.80$	$-0.40$	$-0.40$	$-0.40$	$-0.40$	$-0.20$	$0.40$
Inception	—	—	—	—	—	—	—
RegNet	$-0.92$	$0.30$	$0.15$	$0.33$	$0.35$	$-0.30$	$-0.39$
ResNeXt	$0.50$	$-0.50$	$-1.00$	$-1.00$	$-0.50$	$1.00$	$1.00$
ResNet	$-1.00$	$0.50$	$-0.20$	$-0.20$	$0.10$	$0.40$	$0.50$

Appendix

Table 2: Spearman

\rho

correlation of model quality metrics with top-5 ImageNet test accuracy for CV (non-dropout) models. Combined pools all models in this type. The

\alpha

and

\lambda

are the fitted PL and ET-PL parameters.

Model	BFE-W	$\\|{\mathbf{W}}\\|_{F}^{2}$	$\\|{\mathbf{W}}\\|_{\infty}^{2}$	$\hat{\alpha}$	$\\|{\mathbf{W}}\\|_{2\alpha}^{2\alpha}$	$\alpha$	$\lambda$
EfficientNet	$0.54$	$-0.83$	$-0.84$	$-0.81$	$-0.83$	$-0.95$	$0.69$
MNASNet	$0.80$	$-0.80$	$-0.80$	$-0.80$	$-0.80$	$0.20$	$0.40$
MobileNet	$-1.00$	$-0.50$	$-1.00$	$-0.50$	$-0.50$	$-1.00$	$0.50$
VGG	$0.81$	$-0.90$	$-0.98$	$-0.93$	$-0.93$	$0.14$	$0.95$
Combined	$0.00$	$-0.42$	$-0.66$	$-0.45$	$-0.42$	$-0.08$	$0.73$

Appendix

Table 3: Spearman

\rho

correlation of model quality metrics with top-5 ImageNet test accuracy for Dropout models. Combined pools all models in this type. The

\alpha

and

\lambda

are the fitted PL and ET-PL parameters.

Model	BFE-W	$\\|{\mathbf{W}}\\|_{F}^{2}$	$\\|{\mathbf{W}}\\|_{\infty}^{2}$	$\hat{\alpha}$	$\\|{\mathbf{W}}\\|_{2\alpha}^{2\alpha}$	$\alpha$	$\lambda$
Swin	$-0.83$	$0.66$	$0.94$	$0.89$	$0.89$	$0.83$	$-1.00$
ViT	$0.20$	$0.20$	$-0.50$	$-0.30$	$-0.30$	$0.30$	$0.30$
Combined	$0.14$	$0.17$	$-0.31$	$0.07$	$0.03$	$0.77$	$-0.09$

Appendix

Table 4: Spearman

\rho

correlation of model quality metrics with top-5 ImageNet test accuracy for Vision Transformer models. Combined pools all models in this type. The

\alpha

and

\lambda

are the fitted PL and ET-PL parameters.

Explore similar work

Spectral Signatures of Data Quality: Eigenvalue Tail Index as a Diagnostic for Label Noise in Neural Networks
Mar 29, 2026Matthew LoftusSpectral Methods Noisy-Label Learning
Independent researcher
Spectral Signatures of Large Language Models
Jul 3, 2026Zhuoying Zhang, Ishan V. Prasad, Yuanzhe Hu +4LLM Evaluation Language Model Fingerprinting
Dartmouth College Hanover, NH, USA · University of California, Berkeley Berkeley, CA, USA · Lawrence Berkeley National Laboratory Berkeley, CA, USA
Dimension-Corrected Hitting Times for Heavy-Tailed Spectral Emergence in Neural Optimizer Dynamics
Sep 14, 2026Zongmin LiuHeavy-Tailed Distributions Neural Network Training Dynamics
Stanford University

Mar 29, 2026J/K move · Enter open · S save

Spectral Signatures of Data Quality: Eigenvalue Tail Index as a Diagnostic for Label Noise in Neural Networks

Matthew Loftus

Independent researcher

Spectral Methods Noisy-Label Learning

Open paper AskPDF arXiv

We investigate whether spectral properties of neural network weight matrices can predict test accuracy. Under controlled label noise variation, the tail index alpha of the eigenvalue distribution at the network's bottleneck layer predicts test accuracy with leave-one-out R^2 = 0.984 (21 noise levels, 3 seeds per level), far exceeding all baselines: the best conventional metric (Frobenius norm of the optimal layer) achieves LOO R^2 = 0.149. This relationship holds across three architectures (MLP, CNN, ResNet-18) and two datasets (MNIST, CIFAR-10). However, under hyperparameter variation at fixed data quality (180 configurations varying width, depth, learning rate, and weight decay), all spectral and conventional measures are weak predictors (R^2 < 0.25), with simple baselines (global L_2 norm, LOO R^2 = 0.219) slightly outperforming spectral measures (tail alpha, LOO R^2 = 0.167). We therefore frame the tail index as a data quality diagnostic: a powerful detector of label corruption and training set degradation, rather than a universal generalization predictor. A noise detector calibrated on synthetic noise successfully identifies real human annotation errors in CIFAR-10N (9% noise detected with 3% error). We identify the information-processing bottleneck layer as the locus of this signature and connect the observations to the BBP phase transition in spiked random matrix models. We also report a negative result: the level spacing ratio <r> is uninformative for weight matrices due to Wishart universality.

Abstract

Explore similar work

Spectral Signatures of Data Quality: Eigenvalue Tail Index as a Diagnostic for Label Noise in Neural Networks

Spectral Signatures of Large Language Models

Dimension-Corrected Hitting Times for Heavy-Tailed Spectral Emergence in Neural Optimizer Dynamics