cs.LGNov 11, 2026

Data-Free Metrics Don't Work: The Missing Invariance Under Functionally Preserving Reparameterisations

Authors: Gabryel Mason-Williams, Israel Mason-Williams, Fredrik Dahlqvist

Organizations: King’s College London & Imperial College London London, United Kingdom

Abstract

Data-free methods for analysing and understanding the layers of neural networks offer many metrics for quantifying notions of 'strong' versus 'weak' layers, with the promise of increased interpretability. In particular, random matrix theory (RMT) offers data-free metrics that claim predictive power over the quality of pre-trained models at a layerwise precision and the ability to identify model pathologies, indicating a unique relationship with generalisation. As a result, metrics from RMT are championed for pre-training optimisation strategies and post-training compression. We establish that RMT-based metrics are unrelated to performance or training by showing that functionally preserving reparametrisations of a pre-trained model can have arbitrary metrics. We show this across a range of architectures and scales. To create functionally preserving reparametrisations, we exploit the well-established phenomenon of criticality: some layers can be re-initialised or re-randomised without affecting the functional behaviour of the model -- they are called robust -- while others cannot -- they are called critical. Re-initialising or re-randomising robust layers provides functionally preserving reparametrisations for which RMT-based metrics are arbitrary. Moreover, we show that relationships between metrics offered by RMT are spuriously related to generalisation. We conclude by showing that if many pre-trained models have data-free metrics in a 'good' range, it is, in part, dependent on model initialisation. Project page available at: https://gmw99.github.io/data-free-metrics-dont-work

Figures & tables

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Random Recursive Models

    Sep 30, 2026Jama Hussein Mohamud, Mirco RavanelliEfficient Neural Network InferenceLLM Reasoning

  2. Most ReLU Networks Admit Identifiable Parameters

    May 5, 2026Moritz Grillo, Guido MontúfarRepresentation IdentifiabilityParameter Identifiability

  3. A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning

    Feb 23, 2026Nicolas Anguita, Francesco Locatello, Andrew M. Saxe +4Neural Network GeneralizationRepresentation Learning