Data-free methods for analysing and understanding the layers of neural networks offer many metrics for quantifying notions of 'strong' versus 'weak' layers, with the promise of increased interpretability. In particular, random matrix theory (RMT) offers data-free metrics that claim predictive power over the quality of pre-trained models at a layerwise precision and the ability to identify model pathologies, indicating a unique relationship with generalisation. As a result, metrics from RMT are championed for pre-training optimisation strategies and post-training compression. We establish that RMT-based metrics are unrelated to performance or training by showing that functionally preserving reparametrisations of a pre-trained model can have arbitrary metrics. We show this across a range of architectures and scales. To create functionally preserving reparametrisations, we exploit the well-established phenomenon of criticality: some layers can be re-initialised or re-randomised without affecting the functional behaviour of the model -- they are called robust -- while others cannot -- they are called critical. Re-initialising or re-randomising robust layers provides functionally preserving reparametrisations for which RMT-based metrics are arbitrary. Moreover, we show that relationships between metrics offered by RMT are spuriously related to generalisation. We conclude by showing that if many pre-trained models have data-free metrics in a 'good' range, it is, in part, dependent on model initialisation. Project page available at: https://gmw99.github.io/data-free-metrics-dont-work
Figures & tables
Figure 1: ReLU FCN Architecture.
Metric
Layer
Test Accuracy
FC1
FC2
FC3
FC4
FC5
FC6
Alpha ( α )
4.820 ± 0.025
4.205 ± 0.039
4.126 ± 0.038
4.135 ± 0.035
4.193 ± 0.034
3.793 ± 0.805
96.822 ± 0.057
Table 1: Alpha ( α ) of the layers in ReLU FCN 5x512 and test accuracy of the model on MNIST.
Figure 2: (a) Empirical distribution of α values on a 512x512 fully connected layer, sampled from 10,000 initialisations. (b) Layer re-initialisation ( ∙ ) and re-randomisation ( ∙ ) test accuracy vs α . In § D we present the descriptive statistics for sub-figure (a) along with other initialisation schemes.
Model Layer
Δ Accuracy
Δ Loss
Prediction Similarity ( ↑ )
Jensen-Shannon Distance ( ↓ )
Procrustes Distance ( ↓ )
KL Divergence ( ↓ )
FC1
-39.859 ± 0.936
1.226 ± 0.038
0.574 ± 0.009
0.459 ± 0.003
0.340 ± 0.003
1.014 ± 0.024
FC2
-6.330 ± 0.328
0.285 ± 0.007
0.917 ± 0.003
0.272 ± 0.002
0.087 ± 0.001
0.267 ± 0.004
FC3
-1.899 ± 0.131
0.297 ± 0.003
0.962 ± 0.001
0.294 ± 0.001
0.053 ± 0.001
0.289 ± 0.002
FC4
-1.384 ± 0.080
0.317 ± 0.003
0.968 ± 0.001
0.308 ± 0.001
0.051 ± 0.001
0.314 ± 0.002
FC5
-1.716 ± 0.114
0.364 ± 0.003
0.963 ± 0.001
0.329 ± 0.001
0.054 ± 0.001
0.356 ± 0.002
FC6
-6.966 ± 0.332
0.522 ± 0.004
0.910 ± 0.003
0.383 ± 0.001
0.061 ± 0.001
0.487 ± 0.003
Table 2: Functional preservation of the ReLU FCN 5x512 by layer under the re-initialisation reparametrisation. ( ↑/↓ ) denotes the preferred direction for each metric.
Model Layer
Δ Accuracy
Δ Loss
Prediction Similarity ( ↑ )
Jensen-Shannon Distance ( ↓ )
Procrustes Distance ( ↓ )
KL Divergence ( ↓ )
FC1
-86.928 ± 0.217
3.704 ± 0.052
0.102 ± 0.002
0.721 ± 0.001
0.894 ± 0.003
2.601 ± 0.020
FC2
-86.692 ± 0.273
2.533 ± 0.019
0.104 ± 0.003
0.712 ± 0.001
0.806 ± 0.005
2.303 ± 0.010
FC3
-86.562 ± 0.332
2.348 ± 0.009
0.105 ± 0.003
0.711 ± 0.000
0.647 ± 0.008
2.237 ± 0.006
FC4
-87.350 ± 0.415
2.300 ± 0.007
0.097 ± 0.004
0.711 ± 0.000
0.466 ± 0.008
2.228 ± 0.006
FC5
-87.007 ± 0.477
2.315 ± 0.009
0.101 ± 0.005
0.711 ± 0.001
0.335 ± 0.007
2.232 ± 0.006
FC6
-86.848 ± 0.680
2.356 ± 0.012
0.102 ± 0.007
0.711 ± 0.001
0.221 ± 0.007
2.263 ± 0.011
Table 3: Functional preservation of the ReLU FCN 5x512 by layer under the re-randomisation reparametrisation. ( ↑/↓ ) denotes the preferred direction for each metric.
Similarity Metric
ρ
R2
RMSE
K- τ
Prediction Similarity
0.997
1.000
0.004
0.964
Jensen-Shannon Distance
-0.869
0.977
0.030
-0.672
Procrustes Distance
-0.809
0.635
0.189
-0.604
KL Divergence
-0.864
0.976
0.149
-0.665
Table 4: Correlation between the change in accuracy due to re-initialisation (Table 2 ) and re-randomisation (Table 3 ) and the functional preservation of the ReLU FCN 5x512.
Model
Number of Parameters
Loss
Accuracy
Mean Layer Alpha Value
ResNet34
21.8M
1.071
73.302
3.380 ± 0.195
ViT
86.6M
0.848
79.136
4.287 ± 0.231
Table 5: Model test performance on ImageNet and mean layer Alpha ( α) value.
Figure 3: Layer re-randomisation Test Accuracy vs Alpha ( α ) on ImageNet. See stratified analysis in Appendix F .
Model
Number of Parameters
Loss
Accuracy
Perplexity
Mean Layer Alpha Value
GPT
127M
3.399
39.095
29.941
3.865 ± 0.136
GPT Large
774M
2.967
43.877
19.436
4.013 ± 0.091
Table 6: Model test performance on WikiText-103 and mean layer Alpha ( α) value.
Figure 4: Layer re-randomisation Test Accuracy vs α on WikiText103. See stratified analysis in Appendix F
Figure 5: Test Accuracy vs Model α at initialisation.
Figure 6: Test Accuracy vs Model α through training.
Figure 7: Trained model α vs initial model α ( Left ). Trained model α vs initial model α for interpolations λ=0.1 to λ=1 ( Top Right ) and λ=1 to λ=100001 ( Bottom Right ).
Fix Alpha Range
FC1
FC2
FC3
FC4
FC5
FC6
Test Accuracy
2≤α≤6
5.4799 ± 0.1312
5.0627 ± 0.1209
5.1361 ± 0.1836
4.9623 ± 0.1957
5.209 ± 0.194
5.4213 ± 0.1224
90.178 ± 0.0214
10≤α≤12
10.7025 ± 0.1442
11.0621 ± 0.2103
10.8441 ± 0.1443
10.8218 ± 0.1284
11.0165 ± 0.138
11.2001 ± 0.2024
90.223 ± 0.0237
18≤α≤20
19.0752 ± 0.1244
17.4638 ± 1.2448
18.8759 ± 0.1885
18.7325 ± 0.1573
18.6775 ± 0.2023
18.8356 ± 0.1289
90.219 ± 0.0253
Table 7: Resulting test performance of models trained with fixed Alpha ( α ) values. Mean and standard error of the mean derived from 10 trained models trained on MNIST.
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Params
Loss
Acc
Mean α
Pythia
1B
2.576
47.680
4.801 ± 0.202
Appendix
Table 8: Model test performance on WikiText-103 and mean layer Alpha ( α) value.
Layer Type
Accuracy
Alpha ( α )
% in Optimal α Range
ρ
R2
RSME
K- τ
Attention QKV
42.187 ± 0.896
29.157 ± 4.429
0.000
0.388
0.103
3.395
0.283
Attention Out
42.681 ± 1.743
21.945 ± 2.703
0.000
0.174
0.119
6.541
0.117
MLP 1
33.264 ± 2.996
25.337 ± 3.831
0.000
-0.106
0.045
11.710
-0.083
MLP 2
41.093 ± 2.729
28.064 ± 4.798
0.000
-0.247
0.563
7.216
-0.200
Appendix
Table 9: The impact of re-initialisation across a range of layers within the Pythia-1B model, alongside the correlation statistics of α and test accuracy.
Layer Type
Accuracy
Alpha ( α )
% in Optimal α Range
ρ
R2
RSME
K- τ
Attention QKV
41.581±1.193
33.577±6.225
0.000
-0.059
0.003
4.767
-0.067
Attention Out
42.437±1.817
23.361±2.651
0.000
-0.671
0.148
6.710
-0.450
MLP1
32.241±3.140
29.251±4.495
0.000
0.124
0.010
12.498
0.050
MLP2
40.954±2.727
22.579±1.961
0.000
-0.238
0.059
10.58
-0.150
Appendix
Table 10: The impact of re-randomisation across a range of layers within the Pythia-1B model, alongside the correlation statistics of α and test accuracy.
Figure 8: Pythia-1B layer re-initialisation ( ∙ ) and re-randomisation ( ∙ ) test accuracy vs α .
Table 11: Correlation between the change in accuracy due to layer re-randomisation and the functional preservation of the model.
Table 20
Metric
ρ
R2
RMSE
K- τ
Prediction Sim
0.991
0.968
0.036
0.933
Appendix
Table 13: Correlation between the change in accuracy due to re-initialisation and re-randomisation and the functional preservation of Pythia-1B.
Table 14: Distribution statistics of α across initialisation schemes and layer types.
Initialisation Scheme
xˉ
x~
σ
Skew
Kurtosis
Min
Max
% in Optimal α Range
PyTorch default
13.002
10.587
7.953
2.183
6.524
3.873
76.666
8.020
Xu
13.197
10.667
8.143
2.161
6.974
3.836
101.310
8.310
Xn
12.977
10.531
7.969
2.200
6.844
3.990
74.792
8.190
Ku
13.183
10.664
8.071
2.099
5.917
4.001
68.743
7.840
Kn
13.185
10.618
8.204
2.191
6.667
3.857
88.848
7.740
Appendix
(a) ViT Base 16 Out Projection (Linear Layer 768×768 ).
Model
AUC
F1-Score
Precision
True Positive Rate
False Positive Rate
MLP
0.520
0.242
0.167
0.439
0.387
ResNet34
0.449
0.000
0.000
0.000
0.020
ViT
0.686
0.056
0.029
1.000
0.836
GPT2
0.442
0.022
0.011
0.714
0.901
GPT2-Large
0.603
0.010
0.005
0.875
0.973
Pythia-1B
nan
0.000
0.000
nan
0.797
Appendix
Table 16: Effectiveness of α at predicting a good layer 2≤α≤6 where model performance is within 10% of the original test accuracy after layer reparametrisation.
Table 17: Stratified layer re-randomisation Test Accuracy vs α
Layer Type
Accuracy
Alpha ( α )
% in Optimal α Range
ρ
R2
RMSE
K- τ
wpe
8.905 ± 0.072
12.303 ± 1.625
10.000
-0.818
0.734
0.117
-0.644
attn.c_attn
43.581 ± 0.012
22.934 ± 0.838
0.000
-0.005
0.007
0.222
-0.002
attn.c_proj
43.041 ± 0.203
16.266 ± 0.556
1.944
0.002
0.000
3.856
0.003
mlp.c_fc
41.817 ± 0.355
24.427 ± 0.888
0.000
0.059
0.005
6.724
0.038
mlp.c_proj
42.413 ± 0.348
25.964 ± 0.999
0.000
-0.058
0.000
6.599
-0.039
Appendix
(a) GPT2-Large on WikiText103
Figure 27
Figure 10: Layer re-initialisation ( ∙ ) and re-randomisation ( ∙ ) test accuracy vs the respective metric. ρ is the Spearman correlation coefficient, RSME is the root mean square error of the linear regression (red line), and K- τ is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.
Figure 11: ResNet 34: Layer re-randomisation test accuracy vs the respective metric. ρ is the Spearman correlation coefficient, RSME is the root mean square error of the linear regression (red line), and K- τ is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.
Figure 12: ViT: Layer re-randomisation test accuracy vs the respective metric. ρ is the Spearman correlation coefficient, RSME is the root mean square error of the linear regression (red line), and K- τ is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.
Figure 13: GPT2: Layer re-randomisation test accuracy vs the respective metric. ρ is the Spearman correlation coefficient, RSME is the root mean square error of the linear regression (red line), and K- τ is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.
Figure 14: GPT2-Large: Layer re-randomisation test accuracy vs the respective metric. ρ is the Spearman correlation coefficient, RSME is the root mean square error of the linear regression (red line), and K- τ is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.
Figure 15: Pythia-1B: Layer re-initialisation ( ∙ ) and re-randomisation ( ∙ ) test accuracy vs the respective metric. ρ is the Spearman correlation coefficient, RSME is the root mean square error of the linear regression (red line), and K- τ is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.
Figure 16: Layer re-initialisation ( ∙ ), re-randomisation ( ∙ ) and signs of re-initialisation with magnitudes of re-randomisation ( ∙ ) test accuracy vs the respective metric.
Figure 17: Layer re-initialisation ( ∙ ) and re-randomisation ( ∙ ) test accuracy vs the respective metric. ρ is the Spearman correlation coefficient, RSME is the root mean square error of the linear regression (red line), and K- τ is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.
Figure 18: The first 10 singular values for each layer of a 5 hidden layer neural network with width 512 throughout training on MNIST. The viridis colormap is used to delineate epochs. Dark-purple represents initialisation and the last epoch, epoch 30, is in bright-yellow . The mean (line) is reported with the standard error of the mean (error bars) from 10 runs.
Trainable U
Trainable Σ
Trainable VT
Description
✓
✗
✗
Output Alignment Only
✗
✓
✗
Spectral Scaling Only
✗
✗
✓
Input Alignment Only
✓
✓
✗
Joint U and Σ ( UΣ )
✗
✓
✓
Joint Σ and VT ( ΣVT )
✓
✗
✓
Joint U and VT ( UVT )
Appendix
Table 19: Summary of factor interventional conditions.
Figure 19: Model test performance throughout training on MNIST. (left) accuracy and (right) loss. The dotted line ( … ) is the train performance and the solid line (–) is the test performance, the ∘ represents a epoch. Each curve corresponds to training only one or more components of the parametrisation: UΣVTUΣ , ΣVT , UVT and UΣVT . All other parameters are held fixed during training. Performance is evaluated on the at the end of each epoch. The mean (line) and standard error from the mean (hue) are reported from 10 runs.
Figure 20: Model test performance throughout training on MNIST against data-free metrics. (left) test accuracy and (right) test loss. The ∘ represents a epoch. The colour intensity displays training progress, lighter points correspond to earlier epochs with darker points corresponding to later epochs. Each curve corresponds to training only one or more components of the parametrisation: UΣVTUΣ , ΣVT , UVT and UΣVT . All other parameters are held fixed during training. Performance is evaluated on the at the end of each epoch. The mean (line) and standard error from the mean (error bars) are reported from 10 runs.
Layer
Exponential
Lognormal
Truncated Power Law
Stretched Exponential
Normalised LR
P-value
Normalised LR
P-value
Normalised LR
P-value
Normalised LR
P-value
FC1
6.838
0.000
-3.829
0.000
-5.026
0.000
-3.151
0.002
FC2
7.522
0.000
-5.502
0.000
-5.303
0.000
-5.858
0.000
FC3
7.633
0.000
-5.764
0.000
-5.535
0.000
-6.152
0.000
FC4
7.135
0.000
-6.014
0.000
-5.767
0.000
-6.431
0.000
FC5
8.224
0.000
-5.121
0.000
-4.826
0.000
-5.450
0.000
Appendix
Table 20: ReLU FCN 5x512: Goodness of fit test of the power law distribution against the alternatives with corresponding normalised likelihood ratio (LR) (a negative value indicates support for the alternative) and p-value. For reliable fitting a minimum of 100 samples were required to fit the tail.
Table 21: Comparison of distribution fits across layers using likelihood ratio tests. Counts show the number of layers favouring each distribution. Inconclusive indicates tests with a p-value greater than 0.05.
Figure 21: Number samples used for alpha estimation for each layer. Each dot represents a layer from a trained model
Figure 22: Trained model α vs initial model α ( Left ). Trained model α vs initial model α for interpolations λ=0 to λ=1 ( Top Right ) and λ=1 to λ=100001 ( Bottom Right )
Department of Engineering, University of Cambridge · Institute of Science and Technology, Austria (ISTA) · Gatsby Computational Neuroscience Unit and Sainsbury Wellcome Centre, UCL +1