Data-free methods for analysing and understanding the layers of neural networks offer many metrics for quantifying notions of 'strong' versus 'weak' layers, with the promise of increased interpretability. In particular, random matrix theory (RMT) offers data-free metrics that claim predictive power over the quality of pre-trained models at a layerwise precision and the ability to identify model pathologies, indicating a unique relationship with generalisation. As a result, metrics from RMT are championed for pre-training optimisation strategies and post-training compression. We establish that RMT-based metrics are unrelated to performance or training by showing that functionally preserving reparametrisations of a pre-trained model can have arbitrary metrics. We show this across a range of architectures and scales. To create functionally preserving reparametrisations, we exploit the well-established phenomenon of criticality: some layers can be re-initialised or re-randomised without affecting the functional behaviour of the model -- they are called robust -- while others cannot -- they are called critical. Re-initialising or re-randomising robust layers provides functionally preserving reparametrisations for which RMT-based metrics are arbitrary. Moreover, we show that relationships between metrics offered by RMT are spuriously related to generalisation. We conclude by showing that if many pre-trained models have data-free metrics in a 'good' range, it is, in part, dependent on model initialisation. Project page available at: https://gmw99.github.io/data-free-metrics-dont-work
Figures & tables
Figure 1: ReLU FCN Architecture.
Metric
Layer
Test Accuracy
FC1
FC2
FC3
FC4
FC5
FC6
Alpha ( α )
4.820 ± 0.025
4.205 ± 0.039
4.126 ± 0.038
4.135 ± 0.035
4.193 ± 0.034
3.793 ± 0.805
96.822 ± 0.057
Table 1: Alpha ( α ) of the layers in ReLU FCN 5x512 and test accuracy of the model on MNIST.
Figure 2: (a) Empirical distribution of α values on a 512x512 fully connected layer, sampled from 10,000 initialisations. (b) Layer re-initialisation ( ∙ ) and re-randomisation ( ∙ ) test accuracy vs α . In § D we present the descriptive statistics for sub-figure (a) along with other initialisation schemes.
Model Layer
Δ Accuracy
Δ Loss
Prediction Similarity ( ↑ )
Jensen-Shannon Distance ( ↓ )
Procrustes Distance ( ↓ )
KL Divergence ( ↓ )
FC1
-39.859 ± 0.936
1.226 ± 0.038
0.574 ± 0.009
0.459 ± 0.003
0.340 ± 0.003
1.014 ± 0.024
FC2
-6.330 ± 0.328
0.285 ± 0.007
0.917 ± 0.003
0.272 ± 0.002
0.087 ± 0.001
0.267 ± 0.004
FC3
-1.899 ± 0.131
0.297 ± 0.003
0.962 ± 0.001
0.294 ± 0.001
0.053 ± 0.001
0.289 ± 0.002
FC4
-1.384 ± 0.080
0.317 ± 0.003
0.968 ± 0.001
0.308 ± 0.001
0.051 ± 0.001
0.314 ± 0.002
FC5
-1.716 ± 0.114
0.364 ± 0.003
0.963 ± 0.001
0.329 ± 0.001
0.054 ± 0.001
0.356 ± 0.002
FC6
-6.966 ± 0.332
0.522 ± 0.004
0.910 ± 0.003
0.383 ± 0.001
0.061 ± 0.001
0.487 ± 0.003
Table 2: Functional preservation of the ReLU FCN 5x512 by layer under the re-initialisation reparametrisation. ( ↑/↓ ) denotes the preferred direction for each metric.
Model Layer
Δ Accuracy
Δ Loss
Prediction Similarity ( ↑ )
Jensen-Shannon Distance ( ↓ )
Procrustes Distance ( ↓ )
KL Divergence ( ↓ )
FC1
-86.928 ± 0.217
3.704 ± 0.052
0.102 ± 0.002
0.721 ± 0.001
0.894 ± 0.003
2.601 ± 0.020
FC2
-86.692 ± 0.273
2.533 ± 0.019
0.104 ± 0.003
0.712 ± 0.001
0.806 ± 0.005
2.303 ± 0.010
FC3
-86.562 ± 0.332
2.348 ± 0.009
0.105 ± 0.003
0.711 ± 0.000
0.647 ± 0.008
2.237 ± 0.006
FC4
-87.350 ± 0.415
2.300 ± 0.007
0.097 ± 0.004
0.711 ± 0.000
0.466 ± 0.008
2.228 ± 0.006
FC5
-87.007 ± 0.477
2.315 ± 0.009
0.101 ± 0.005
0.711 ± 0.001
0.335 ± 0.007
2.232 ± 0.006
FC6
-86.848 ± 0.680
2.356 ± 0.012
0.102 ± 0.007
0.711 ± 0.001
0.221 ± 0.007
2.263 ± 0.011
Table 3: Functional preservation of the ReLU FCN 5x512 by layer under the re-randomisation reparametrisation. ( ↑/↓ ) denotes the preferred direction for each metric.
Similarity Metric
ρ
R2
RMSE
K- τ
Prediction Similarity
0.997
1.000
0.004
0.964
Jensen-Shannon Distance
-0.869
0.977
0.030
-0.672
Procrustes Distance
-0.809
0.635
0.189
-0.604
KL Divergence
-0.864
0.976
0.149
-0.665
Table 4: Correlation between the change in accuracy due to re-initialisation (Table 2 ) and re-randomisation (Table 3 ) and the functional preservation of the ReLU FCN 5x512.
Model
Number of Parameters
Loss
Accuracy
Mean Layer Alpha Value
ResNet34
21.8M
1.071
73.302
3.380 ± 0.195
ViT
86.6M
0.848
79.136
4.287 ± 0.231
Table 5: Model test performance on ImageNet and mean layer Alpha ( α) value.
Figure 3: Layer re-randomisation Test Accuracy vs Alpha ( α ) on ImageNet. See stratified analysis in Appendix F .
Model
Number of Parameters
Loss
Accuracy
Perplexity
Mean Layer Alpha Value
GPT
127M
3.399
39.095
29.941
3.865 ± 0.136
GPT Large
774M
2.967
43.877
19.436
4.013 ± 0.091
Table 6: Model test performance on WikiText-103 and mean layer Alpha ( α) value.
Figure 4: Layer re-randomisation Test Accuracy vs α on WikiText103. See stratified analysis in Appendix F
Figure 5: Test Accuracy vs Model α at initialisation.
Figure 6: Test Accuracy vs Model α through training.
Figure 7: Trained model α vs initial model α ( Left ). Trained model α vs initial model α for interpolations λ=0.1 to λ=1 ( Top Right ) and λ=1 to λ=100001 ( Bottom Right ).
Fix Alpha Range
FC1
FC2
FC3
FC4
FC5
FC6
Test Accuracy
2≤α≤6
5.4799 ± 0.1312
5.0627 ± 0.1209
5.1361 ± 0.1836
4.9623 ± 0.1957
5.209 ± 0.194
5.4213 ± 0.1224
90.178 ± 0.0214
10≤α≤12
10.7025 ± 0.1442
11.0621 ± 0.2103
10.8441 ± 0.1443
10.8218 ± 0.1284
11.0165 ± 0.138
11.2001 ± 0.2024
90.223 ± 0.0237
18≤α≤20
19.0752 ± 0.1244
17.4638 ± 1.2448
18.8759 ± 0.1885
18.7325 ± 0.1573
18.6775 ± 0.2023
18.8356 ± 0.1289
90.219 ± 0.0253
Table 7: Resulting test performance of models trained with fixed Alpha ( α ) values. Mean and standard error of the mean derived from 10 trained models trained on MNIST.
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Params
Loss
Acc
Mean α
Pythia
1B
2.576
47.680
4.801 ± 0.202
Appendix
Table 8: Model test performance on WikiText-103 and mean layer Alpha ( α) value.
Layer Type
Accuracy
Alpha ( α )
% in Optimal α Range
ρ
R2
RSME
K- τ
Attention QKV
42.187 ± 0.896
29.157 ± 4.429
0.000
0.388
0.103
3.395
0.283
Attention Out
42.681 ± 1.743
21.945 ± 2.703
0.000
0.174
0.119
6.541
0.117
MLP 1
33.264 ± 2.996
25.337 ± 3.831
0.000
-0.106
0.045
11.710
-0.083
MLP 2
41.093 ± 2.729
28.064 ± 4.798
0.000
-0.247
0.563
7.216
-0.200
Appendix
Table 9: The impact of re-initialisation across a range of layers within the Pythia-1B model, alongside the correlation statistics of α and test accuracy.
Layer Type
Accuracy
Alpha ( α )
% in Optimal α Range
ρ
R2
RSME
K- τ
Attention QKV
41.581±1.193
33.577±6.225
0.000
-0.059
0.003
4.767
-0.067
Attention Out
42.437±1.817
23.361±2.651
0.000
-0.671
0.148
6.710
-0.450
MLP1
32.241±3.140
29.251±4.495
0.000
0.124
0.010
12.498
0.050
MLP2
40.954±2.727
22.579±1.961
0.000
-0.238
0.059
10.58
-0.150
Appendix
Table 10: The impact of re-randomisation across a range of layers within the Pythia-1B model, alongside the correlation statistics of α and test accuracy.
Figure 8: Pythia-1B layer re-initialisation ( ∙ ) and re-randomisation ( ∙ ) test accuracy vs α .
Table 11: Correlation between the change in accuracy due to layer re-randomisation and the functional preservation of the model.
Table 20
Metric
ρ
R2
RMSE
K- τ
Prediction Sim
0.991
0.968
0.036
0.933
Appendix
Table 13: Correlation between the change in accuracy due to re-initialisation and re-randomisation and the functional preservation of Pythia-1B.
Table 14: Distribution statistics of α across initialisation schemes and layer types.
Initialisation Scheme
xˉ
x~
σ
Skew
Kurtosis
Min
Max
% in Optimal α Range
PyTorch default
13.002
10.587
7.953
2.183
6.524
3.873
76.666
8.020
Xu
13.197
10.667
8.143
2.161
6.974
3.836
101.310
8.310
Xn
12.977
10.531
7.969
2.200
6.844
3.990
74.792
8.190
Ku
13.183
10.664
8.071
2.099
5.917
4.001
68.743
7.840
Kn
13.185
10.618
8.204
2.191
6.667
3.857
88.848
7.740
Appendix
(a) ViT Base 16 Out Projection (Linear Layer 768×768 ).
Model
AUC
F1-Score
Precision
True Positive Rate
False Positive Rate
MLP
0.520
0.242
0.167
0.439
0.387
ResNet34
0.449
0.000
0.000
0.000
0.020
ViT
0.686
0.056
0.029
1.000
0.836
GPT2
0.442
0.022
0.011
0.714
0.901
GPT2-Large
0.603
0.010
0.005
0.875
0.973
Pythia-1B
nan
0.000
0.000
nan
0.797
Appendix
Table 16: Effectiveness of α at predicting a good layer 2≤α≤6 where model performance is within 10% of the original test accuracy after layer reparametrisation.
Table 17: Stratified layer re-randomisation Test Accuracy vs α
Layer Type
Accuracy
Alpha ( α )
% in Optimal α Range
ρ
R2
RMSE
K- τ
wpe
8.905 ± 0.072
12.303 ± 1.625
10.000
-0.818
0.734
0.117
-0.644
attn.c_attn
43.581 ± 0.012
22.934 ± 0.838
0.000
-0.005
0.007
0.222
-0.002
attn.c_proj
43.041 ± 0.203
16.266 ± 0.556
1.944
0.002
0.000
3.856
0.003
mlp.c_fc
41.817 ± 0.355
24.427 ± 0.888
0.000
0.059
0.005
6.724
0.038
mlp.c_proj
42.413 ± 0.348
25.964 ± 0.999
0.000
-0.058
0.000
6.599
-0.039
Appendix
(a) GPT2-Large on WikiText103
Figure 27
Figure 10: Layer re-initialisation ( ∙ ) and re-randomisation ( ∙ ) test accuracy vs the respective metric. ρ is the Spearman correlation coefficient, RSME is the root mean square error of the linear regression (red line), and K- τ is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.
Figure 11: ResNet 34: Layer re-randomisation test accuracy vs the respective metric. ρ is the Spearman correlation coefficient, RSME is the root mean square error of the linear regression (red line), and K- τ is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.
Figure 12: ViT: Layer re-randomisation test accuracy vs the respective metric. ρ is the Spearman correlation coefficient, RSME is the root mean square error of the linear regression (red line), and K- τ is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.
Figure 13: GPT2: Layer re-randomisation test accuracy vs the respective metric. ρ is the Spearman correlation coefficient, RSME is the root mean square error of the linear regression (red line), and K- τ is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.
Figure 14: GPT2-Large: Layer re-randomisation test accuracy vs the respective metric. ρ is the Spearman correlation coefficient, RSME is the root mean square error of the linear regression (red line), and K- τ is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.
Figure 15: Pythia-1B: Layer re-initialisation ( ∙ ) and re-randomisation ( ∙ ) test accuracy vs the respective metric. ρ is the Spearman correlation coefficient, RSME is the root mean square error of the linear regression (red line), and K- τ is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.
Figure 16: Layer re-initialisation ( ∙ ), re-randomisation ( ∙ ) and signs of re-initialisation with magnitudes of re-randomisation ( ∙ ) test accuracy vs the respective metric.
Figure 17: Layer re-initialisation ( ∙ ) and re-randomisation ( ∙ ) test accuracy vs the respective metric. ρ is the Spearman correlation coefficient, RSME is the root mean square error of the linear regression (red line), and K- τ is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.
Figure 18: The first 10 singular values for each layer of a 5 hidden layer neural network with width 512 throughout training on MNIST. The viridis colormap is used to delineate epochs. Dark-purple represents initialisation and the last epoch, epoch 30, is in bright-yellow . The mean (line) is reported with the standard error of the mean (error bars) from 10 runs.
Trainable U
Trainable Σ
Trainable VT
Description
✓
✗
✗
Output Alignment Only
✗
✓
✗
Spectral Scaling Only
✗
✗
✓
Input Alignment Only
✓
✓
✗
Joint U and Σ ( UΣ )
✗
✓
✓
Joint Σ and VT ( ΣVT )
✓
✗
✓
Joint U and VT ( UVT )
Appendix
Table 19: Summary of factor interventional conditions.
Figure 19: Model test performance throughout training on MNIST. (left) accuracy and (right) loss. The dotted line ( … ) is the train performance and the solid line (–) is the test performance, the ∘ represents a epoch. Each curve corresponds to training only one or more components of the parametrisation: UΣVTUΣ , ΣVT , UVT and UΣVT . All other parameters are held fixed during training. Performance is evaluated on the at the end of each epoch. The mean (line) and standard error from the mean (hue) are reported from 10 runs.
Figure 20: Model test performance throughout training on MNIST against data-free metrics. (left) test accuracy and (right) test loss. The ∘ represents a epoch. The colour intensity displays training progress, lighter points correspond to earlier epochs with darker points corresponding to later epochs. Each curve corresponds to training only one or more components of the parametrisation: UΣVTUΣ , ΣVT , UVT and UΣVT . All other parameters are held fixed during training. Performance is evaluated on the at the end of each epoch. The mean (line) and standard error from the mean (error bars) are reported from 10 runs.
Layer
Exponential
Lognormal
Truncated Power Law
Stretched Exponential
Normalised LR
P-value
Normalised LR
P-value
Normalised LR
P-value
Normalised LR
P-value
FC1
6.838
0.000
-3.829
0.000
-5.026
0.000
-3.151
0.002
FC2
7.522
0.000
-5.502
0.000
-5.303
0.000
-5.858
0.000
FC3
7.633
0.000
-5.764
0.000
-5.535
0.000
-6.152
0.000
FC4
7.135
0.000
-6.014
0.000
-5.767
0.000
-6.431
0.000
FC5
8.224
0.000
-5.121
0.000
-4.826
0.000
-5.450
0.000
Appendix
Table 20: ReLU FCN 5x512: Goodness of fit test of the power law distribution against the alternatives with corresponding normalised likelihood ratio (LR) (a negative value indicates support for the alternative) and p-value. For reliable fitting a minimum of 100 samples were required to fit the tail.
Table 21: Comparison of distribution fits across layers using likelihood ratio tests. Counts show the number of layers favouring each distribution. Inconclusive indicates tests with a p-value greater than 0.05.
Figure 21: Number samples used for alpha estimation for each layer. Each dot represents a layer from a trained model
Figure 22: Trained model α vs initial model α ( Left ). Trained model α vs initial model α for interpolations λ=0 to λ=1 ( Top Right ) and λ=1 to λ=100001 ( Bottom Right )
Recursive models create computational depth through parameter reuse, offering a parameter-efficient alternative to increasing model size. However, most recursive models repeatedly apply one learned transformation or a prescribed sequence of transformations, restricting computation to a fixed layer order. We introduce the Random Recursive Model (RRM), which maintains a pool of L learned layers and performs T recursive steps by sampling one layer independently with replacement for each example and step. This enables flexible layer reuse while retaining the parameter efficiency of recurrence. We evaluate RRM on challenging reasoning tasks, where it matches or exceeds the baselines, often with 50-75 % fewer parameters. RRM can vary its depth at inference, including beyond that seen during training, without retraining or adding parameters, improving tasks that benefit from deeper iterative computation. RRM also supports Monte Carlo inference and probabilistic test-time scaling, both of which improve performance without retraining. These insights may open new directions in neural network architecture design.
Jama Hussein Mohamud, Mirco Ravanelli
Mila – Quebec AI Institute · Université de Montréal · Concordia University
We study the realization map of deep ReLU networks, focusing on when a function determines its parameters up to scaling and permutation. To analyze hidden redundancies beyond these standard symmetries, we introduce a framework based on weighted polyhedral complexes. Our main result shows that for every architecture whose input and hidden layers have width at least two, there exists an open set of identifiable parameters. This implies that the functional dimension of every such architecture is exactly the number of parameters minus the number of hidden neurons. We further show that minimal functional representations can still have non-trivial parameter redundancies. Finally, we establish a generic depth hierarchy, whereby for an open set of parameters the realized function cannot be represented generically by any shallower network.
Moritz Grillo, Guido Montúfar
Max Planck Institute for Mathematics in the Sciences, Leipzig · Departments of Mathematics and Statistics & Data Science, UCLA
Pretraining and fine-tuning are central stages in modern machine learning systems. In practice, feature learning plays an important role across both stages: deep neural networks learn a broad range of useful features during pretraining and further refine those features during fine-tuning. However, an end-to-end theoretical understanding of how choices of initialization impact the ability to reuse and refine features during fine-tuning has remained elusive. Here we develop an analytical theory of the pretraining fine-tuning pipeline in diagonal linear networks, deriving exact expressions for the generalization error as a function of initialization parameters and task statistics. We find that different initialization choices place the network into four distinct fine-tuning regimes that are distinguished by their ability to support feature learning and reuse and therefore by the task statistics for which they are beneficial. In particular, a smaller initialization scale in earlier layers enables the network to both reuse and refine its features, leading to superior generalization on fine-tuning tasks that rely on a subset of pretraining features. We demonstrate empirically that the same initialization parameters impact generalization in ResNets trained on CIFAR-100 and SVHN as well as Transformers trained on modular arithmetic tasks. Overall, our results demonstrate an alytically how data and network initialization interact to shape fine-tuning generalization, highlighting an important role for the relative scale of initialization across different layers in enabling continued feature learning during fine-tuning.
Nicolas Anguita, Francesco Locatello, Andrew M. Saxe +4
Department of Engineering, University of Cambridge · Institute of Science and Technology, Austria (ISTA) · Gatsby Computational Neuroscience Unit and Sainsbury Wellcome Centre, UCL +1