Beyond Accuracy: A Unified Random Matrix Theory Diagnostic Framework for Crash Classification Models

Ibne Farabi Shihab; Sanjeda Akter; Anuj Sharma

cs.LGFeb 23, 2026

Beyond Accuracy: A Unified Random Matrix Theory Diagnostic Framework for Crash Classification Models

Authors: Ibne Farabi Shihab, Sanjeda Akter, Anuj Sharma

Organizations: Department of Computer Science, Iowa State University, Ames, IA 50010 · Department of Civil, Construction and Environmental Engineering, Iowa State University, Ames, IA 50010

Abstract

Crash classification models in transportation safety are typically evaluated using accuracy, F1, or AUC, metrics that cannot reveal whether a model is silently overfitting. We introduce a spectral diagnostic framework grounded in Random Matrix Theory (RMT) and Heavy-Tailed Self-Regularization (HTSR) that spans the ML taxonomy: weight matrices for BERT/ALBERT/Qwen2.5, out-of-fold increment matrices for XGBoost/Random Forest, empirical Hessians for Logistic Regression, induced affinity matrices for Decision Trees, and Graph Laplacians for KNN. Evaluating nine model families on two Iowa DOT crash classification tasks (173,512 and 371,062 records respectively), we find that the power-law exponent $α$ provides a structural quality signal: well-regularized models consistently yield $α$ within $[2, 4]$ (mean $2.87 \pm 0.34$ ), while overfit variants show $α< 2$ or spectral collapse. We observe a strong rank correlation between $α$ and expert agreement (Spearman $ρ= 0.89$ , $p < 0.001$ ), suggesting spectral quality captures model behaviors aligned with expert reasoning. We propose an $α$ -based early stopping criterion and a spectral model selection protocol, and validate both against cross-validated F1 baselines. Sparse Lanczos approximations make the framework scalable to large datasets.

Figures & tables

Model	Matrix $\mathbf{C}$	Overfit Signal	Cost
BERT/ALBERT	$\mathbf{W}_{\ell}^{\top}\mathbf{W}_{\ell}$	$\alpha<2$ , traps	$O(d^{3})$
Qwen2.5-7B	$\mathbf{W}_{\ell}^{\top}\mathbf{W}_{\ell}$	$\alpha<2$ , traps	$O(d^{3})$
XGBoost	$\mathbf{W}_{7}^{\top}\mathbf{W}_{7}/N$	$\alpha<2$ , traps	$O(T^{3})$
Random Forest	OOB analogue	$\alpha<2$ , traps	$O(T^{3})$
Logistic Reg. †	$\mathbf{X}^{\top}\mathbf{D}\mathbf{X}/N$	$\alpha\to\infty$	$O(d^{3})$
Decision Tree †	$\mathbf{M}^{\top}\mathbf{M}$	Dirac at $\lambda\!=\!1$	$O(L)$

TABLE I: Taxonomy of Effective Representation Matrices.

N

: samples,

T

: trees,

L

: leaves,

d

: features,

k

: neighbors. Mappings marked with

\dagger

are novel proposals validated empirically in this work.

Panel of Figure 2 — Fig. 1: Unified spectral diagnostic framework. Crash data trains a diverse taxonomy of models. Weight matrices (Deep Learning) or Effective Representation Matrices (Classical ML/Ensembles) are analyzed via RMT to extract $\alpha$ and trap counts, informing safety-critical deployment decisions.

Model Family	Regime	Matrix	$\hat{\alpha}$	KS $p$	Traps	F1	AUC	$\kappa$ (Expert)
BERT	Good	Weight	$2.87\pm 0.14$	0.42	$0.4\pm 0.5$	$.874\pm.008$	$.936\pm.005$	$.68\pm.03$
BERT	Overfit-Epochs	Weight	$1.74\pm 0.19$	0.18	$4.8\pm 1.3$	$.862\pm.011$	$.921\pm.007$	$.54\pm.04$
ALBERT	Good	Weight	$3.12\pm 0.09$	0.61	$0.2\pm 0.4$	$.868\pm.007$	$.929\pm.004$	$.73\pm.02$
ALBERT	Overfit-Epochs	Weight	$2.08\pm 0.21$	0.14	$1.8\pm 0.8$	$.859\pm.009$	$.918\pm.006$	$.61\pm.04$
Qwen2.5-7B	Zero-shot	Weight	$2.94\pm 0.12$	0.52	$0.6\pm 0.5$	$.781\pm.010$	$.872\pm.007$	$.76\pm.03$
XGBoost	Good	OOF Incr.	$2.34\pm 0.17$	0.38	$0.2\pm 0.4$	$.851\pm.006$	$.922\pm.004$	$.62\pm.03$

TABLE II: Spectral diagnostics and classification performance for intersection misclassification detection (mean

\pm

std over 5 seeds). KS

p

: power-law goodness-of-fit

p

-value.

\dagger

: novel spectral mapping proposed in this work.

Model †	Task	Regime	$\hat{\alpha}$	KS $p$
Logistic Reg.	INT	Good	$3.21\pm 0.31$	0.29
Logistic Reg.	INT	Overfit	$1.68\pm 0.38$	0.09
Logistic Reg.	AIM	Good	$3.15\pm 0.27$	0.31
Logistic Reg.	AIM	Overfit	$1.72\pm 0.35$	0.08
Decision Tree	INT	Good	$2.62\pm 0.28$	0.22
Decision Tree	INT	Overfit	Collapse	$<$ 0.01

TABLE III: Power-law fit validation for novel spectral mappings (

\dagger

) across both tasks. KS

p>0.1

indicates the power-law hypothesis is not rejected.

Panel of Figure 5 — Fig. 2: Layer-wise $\alpha$ for BERT (top) and ALBERT (bottom). Shaded band = optimal range $[2,4]$ ; colored bands = min/max across 5 seeds. ALBERT’s shared weights produce more uniform $\alpha$ with tighter variance.

Panel of Figure 6 — Fig. 3: Schematic ESD (log-log) of the XGBoost OOF correlation matrix. Left: well-regularized ( $\alpha=2.34$ , no traps). Right: overfit ( $\alpha=1.62$ , correlation traps visible as isolated spikes). MP distribution (dashed gray) serves as the null model. These are illustrative; actual $\alpha$ values are fitted via MLE on computed eigenvalues.

Model	Criterion	Stop Epoch/Round	F1	$\hat{\alpha}$
BERT	Val. loss	$7.2\pm 1.1$	$.874\pm.008$	$2.87\pm.14$
	$\alpha<2.0$	$5.4\pm 0.9$	$.869\pm.010$	$2.14\pm.12$
	Joint	$5.4\pm 0.9$	$.871\pm.009$	$2.14\pm.12$
	None (20 ep.)	20	$.832\pm.018$	$1.38\pm.16$
ALBERT	Val. loss	$8.0\pm 1.3$	$.868\pm.007$	$3.12\pm.09$
	$\alpha<2.0$	$11.2\pm 2.1$	$.864\pm.008$	$2.06\pm.11$

TABLE IV: Early stopping comparison across three model families on INT task (mean

\pm

std, 5 seeds).

\alpha

-stop triggers at

\hat{\alpha}<2.0

Panel of Figure 8 — Fig. 4: BERT training dynamics (median seed, INT task). Top: validation loss minimum at epoch 7. Bottom: $\hat{\alpha}$ crosses below 2.0 at epoch 5, providing an earlier structural warning.

Panel of Figure 9 — Fig. 5: Spectral quality ( $\hat{\alpha}$ ) vs. expert agreement ( $\kappa$ ) across all model families ( $n=15$ ; DT/KNN-Overfit excluded due to collapsed spectra). Spearman $\rho=0.89$ ( $p<0.001$ ; 95% bootstrap CI: $[0.74,0.96]$ ). Shaded region = optimal $\alpha$ range. Qwen2.5-7B (zero-shot) occupies the upper-right quadrant.

Selection Strategy	Kendall’s $\tau$ vs. $\kappa$
Cross-validated F1	0.50
Validation loss	0.43
Spectral composite (Eq. 6)	0.79

TABLE V: Model selection ranking comparison on INT task. Kendall’s

\tau

measures rank correlation with expert agreement (

\kappa

) across 9 well-regularized models.

Model	Task	$\hat{\alpha}$	Traps	F1
BERT	INT	$2.87\pm 0.14$	$0.4\pm 0.5$	$.874\pm.008$
BERT	AIM	$2.71\pm 0.16$	$1.0\pm 0.7$	$.862\pm.009$
ALBERT	INT	$3.12\pm 0.09$	$0.2\pm 0.4$	$.868\pm.007$
ALBERT	AIM	$2.94\pm 0.11$	$0.4\pm 0.5$	$.855\pm.008$
Qwen2.5-7B	INT	$2.94\pm 0.12$	$0.6\pm 0.5$	$.781\pm.010$
Qwen2.5-7B	AIM	$2.88\pm 0.14$	$0.8\pm 0.8$	$.769\pm.011$

TABLE VI: Spectral diagnostics across tasks for well-regularized models (mean

\pm

std, 5 seeds). INT: intersection misclassification; AIM: alcohol inference mismatch.

Model	Train Time	Spectral Time	Overhead
BERT	4.2 h	12 min	4.8%
ALBERT	3.8 h	11 min	4.8%
Qwen2.5-7B	—	18 min	—
XGBoost	18 min	0.8 min	4.4%
Random Forest	22 min	0.9 min	4.1%
Logistic Reg.	3 min	0.2 min	6.7%

TABLE VII: Spectral extraction cost on AIM dataset. Training time excludes hyperparameter search. Qwen2.5-7B is used zero-shot (no training); spectral time reflects weight matrix extraction and eigendecomposition.

Explore similar work

Detecting overfitting in Neural Networks during long-horizon grokking using Random Matrix Theory
May 12, 2026Hari K. Prakash, Charles H MartinNeural Network Generalization Neural Network Training Dynamics
University of California San Diego Data Science and Engineering · Calculation Consulting San Francisco, CA, USA
Spectral Signatures of Data Quality: Eigenvalue Tail Index as a Diagnostic for Label Noise in Neural Networks
Mar 29, 2026Matthew LoftusSpectral Methods Noisy-Label Learning
Independent researcher
CHOIR: heterogeneity-aware conformal prediction for crash injury severity across driver safety strata
Sep 10, 2026Amir Rafe, Subasish DasDistribution Shift Robustness Group-Conditional Conformal Prediction
Civil Engineering, Texas State University, 601 University Drive, San Marcos, 78666 TX, USA

May 12, 2026J/K move · Enter open · S save

Detecting overfitting in Neural Networks during long-horizon grokking using Random Matrix Theory

Hari K. Prakash, Charles H Martin

University of California San Diego Data Science and Engineering · Calculation Consulting San Francisco, CA, USA

Neural Network Generalization Neural Network Training Dynamics Grokking

Open paper AskPDF arXiv

Training Neural Networks (NNs) without overfitting is difficult; detecting that overfitting is difficult as well. We present a novel Random Matrix Theory method that detects the onset of overfitting in deep learning models without access to train or test data. For each model layer, we randomize each weight matrix element-wise,

\mathbf{W} \to \mathbf{W}^{\mathrm{rand}}

, fit the randomized empirical spectral distribution with a Marchenko-Pastur distribution, and identify large outliers that violate self-averaging. We call these outliers Correlation Traps. During the onset of overfitting, which we call the "anti-grokking" phase in long-horizon grokking, Correlation Traps form and grow in number and scale as test accuracy decreases while train accuracy remains high. Traps may be benign or may harm generalization; we provide an empirical approach to distinguish between them by passing random data through the trained model and evaluating the JS divergence of output logits. Our findings show that anti-grokking is an additional grokking phase with high train accuracy and decreasing test accuracy, structurally distinct from pre-grokking through its Correlation Traps. More broadly, we find that some foundation-scale LLMs exhibit the same Correlation Traps, indicating potentially harmful overfitting.

Abstract

Explore similar work

Detecting overfitting in Neural Networks during long-horizon grokking using Random Matrix Theory

Spectral Signatures of Data Quality: Eigenvalue Tail Index as a Diagnostic for Label Noise in Neural Networks

CHOIR: heterogeneity-aware conformal prediction for crash injury severity across driver safety strata