cs.LGFeb 25, 2026

Structure and Redundancy in Large Language Models: A Spectral Study via Random Matrix Theory

Authors: Davide Ettori

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

This thesis addresses two persistent and closely related challenges in modern deep learning, reliability and efficiency, through a unified framework grounded in Spectral Geometry and Random Matrix Theory (RMT). As deep networks and large language models continue to scale, their internal behavior becomes increasingly opaque, leading to hallucinations, fragile generalization under distribution shift, and growing computational and energy demands. By analyzing the eigenvalue dynamics of hidden activations across layers and inputs, this work shows that spectral statistics provide a compact, stable, and interpretable lens on model behavior, capable of separating structured, causal representations from noise-dominated variability. Within this framework, the first contribution, EigenTrack, introduces a real-time method for detecting hallucinations and out-of-distribution behavior in large language and vision-language models. EigenTrack transforms streaming activations into spectral descriptors such as entropy, variance, and deviations from the Marchenko-Pastur baseline, and models their temporal evolution using lightweight recurrent classifiers, enabling early detection of reliability failures before they appear in model outputs while offering interpretable insight into representation dynamics. The second contribution, RMT-KD, presents a principled approach to compressing deep networks via random matrix theoretic knowledge distillation. By interpreting outlier eigenvalues in activation spectra as carriers of task-relevant information, RMT-KD progressively projects networks onto lower-dimensional subspaces through iterative self-distillation, yielding significantly more compact and energy-efficient models while preserving accuracy and dense, hardware-friendly structure.

Figures & tables

Figure 1: General architecture of EigenTrack: spectral features extracted from hidden activations are streamed into a recurrent discrepancy detector, which outputs early warnings.
Figure 2: Temporal evolution of spectral statistics: hallucinated sequences tend to stay closer to the MP-like regime, while factual sequences diverge toward a more structured spectra.
Figure 3: Hallucination (a, top) and OOD (b, bottom) detection metics across models and classifiers. Sliding window length is 30 tokens.
MethodLLaMa1BLLaMa3BLLaMa7B
EigenTrack0.8420.8610.894
LapEigvals0.7850.8190.871
INSIDE0.7530.8310.810
SelfCheckGPT0.7390.8040.809
HaloScope0.8200.8270.861
Table 1: AUROC comparison on LLaMa for hallucination detection on HotPotQA data.
Figure 4: AUROC and latency as a function of sequence window length of GRU on hallucination dataset (a, top) and AUROC as a function of total generated tokens (b, bottom) .
Figure 5: Overview of the iterative RMT-KD pipeline: spectral analysis estimates the MP bulk edge, outlier eigenvectors define the causal subspace, and self-distillation stabilizes training.
Figure 6: Evolution of the empirical eigenvalue distribution of activation covariances across layers of BERT on SST data during training.
Figure 7: Performance metrics of RMT-KD models relative to baselines: (a, top) accuracy and parameter reduction, (b, middle) inference speedup and power reduction, (c, bottom) memory footprint and energy per inference.
 BERT-baseBERT-tinyResNet-50
MethodRed.Acc.Red.Acc.Red.Acc.
RMT-KD80.9%+1.8%58.8%+1.4%47.7%+0.7%
DistilBERT42.7%+0.2%54.8%+0.4%—
Theseus48.3%+0.6%53.0%+0.1%—
PKD40.5%-1.0%50.1%-0.8%—
AT——42.2%+0.4%
Table 2: Compression ratio (Red.) and accuracy change (Acc.) comparison with other methods.
Figure 8: Effect of the variance initialization quantile on the compression-accuracy tradeoff.
More in this topic

LLM Compression

Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.

See all 283 papers

Explore similar work

CardsList
Jul 3, 2026

Spectral Signatures of Large Language Models

The rapidly growing repository of publicly available large language models (LLMs) presents significant challenges for systematic management and quantification at scale, such as model lineage tracing, licensing, and evaluation. However, task-specific benchmarks are insufficient for this setting, as LLMs differ widely in architectures, scales, and training procedures. To address this challenge, we adopt spectral shape-based metrics for managing and quantifying LLMs based on Heavy-Tailed Self-Regularization theory. Our approach uses the shape information of the weight empirical spectral density as a compact spectral signature of each model. This signature captures intrinsic properties of pretrained models and remains robust during post-training, making it suitable for model-level analysis. In addition, this metric is data-free, computationally-efficient, and scale-invariant, enabling large-scale analysis in practice. Moreover, we curate a large and diverse model corpus consisting of major open-source LLM families, and use it to systematically benchmark spectral and non-spectral metrics across models and downstream tasks. We show that our spectral signature supports the tracking of the model lineage, the unsupervised clustering of similar models, and the quantification of the model performance. Overall, the proposed spectral signature provides a meaningful proxy for broad performance trends across LLMs, enabling efficient organization, comparison, and analysis of large model collections.

Zhuoying Zhang, Ishan V. Prasad, Yuanzhe Hu +4

Dartmouth College Hanover, NH, USA · University of California, Berkeley Berkeley, CA, USA · Lawrence Berkeley National Laboratory Berkeley, CA, USA

LLM EvaluationLanguage Model FingerprintingSpectral Methods
AskPDF
Jun 17, 2026

Thermodynamic Signatures of Reasoning: Free-Energy and Spectral-Form-Factor Diagnostics for Hallucination Detection in Large Language Models

Hallucination detection in large language models (LLMs) is deployment-critical, and recent work shows that the spectrum of attention-derived graph Laplacians carries strong signal about reasoning quality. Prior spectral diagnostics, however, summarize the Laplacian spectrum by a handful of eigenvalues or hand-picked scalars, leaving most of its structure unused. We propose Free-Energy Signatures (Fes), a spectral descriptor that treats each layer's attention Laplacian as a Hamiltonian and extracts its thermodynamic potentials partition function, free energy, spectral entropy, heat capacity together with the random-matrix-theory (RMT) spectral form factor. We prove three results: (i)~Lipschitz stability of Fes under attention perturbation; (ii)~an expressiveness result showing that Fes enriches finite spectral summaries and approximates moment-derived spectral functionals under explicit regularity and grid-resolution assumptions; and (iii)~a finite-sample PAC bound on the AUROC of a training-free detector built from Fes. Empirically, across six open-weight LLMs and six benchmarks, a lightweight probe on Fes descriptors achieves the strongest aggregate AUROC among attention-spectral baselines, improving over LapEig by +6.5+6.5 AUROC points and over GoR-4 by +2.4+2.4 points on average, while requiring no update to the underlying LLM. In the fully unsupervised setting, an RMT-deviation score achieves mean AUROC 0.710.71, providing a label-free but weaker detector. A complementary RMT analysis shows that correct generations exhibit more Wigner-Dyson like spectral statistics, whereas hallucinations exhibit more Poisson-like statistics. The anonymized code and config are provided in the supplementary material.

Salim Khazem

Talan Research & Innovation Center

Hallucination DetectionLLM Hallucination Detection
AskPDF
May 28, 2026

Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initialization

Efficient Distillation (EDistill) compresses large language models (LLMs) by structured pruning parameters and tuning lightweight modules with high training efficiency. Although these EDistilled LLMs achieve state-of-the-art (SOTA) performance on general ability benchmarks relative to similarly sized LLMs, we identify a severe degradation in their multi-step reasoning ability, which we term reasoning collapse. We systematically analyze the geometric origins of reasoning collapse and show that the SOTA EDistill method based on width-reducing projection matrices suffers from eRank collapse, in which the effective rank (eRank) of hidden representations drops. We theoretically explain how singular values of randomly initialized projection matrices become unevenly distributed, leading to eRank collapse and thus token indistinguishability. To address this issue, we propose RED (Reasoning-preserved Efficient Distillation) for LLMs, which introduces activation-aware initialization to initialize projection matrices as channel-selection matrices, thus theoretically mitigating eRank collapse. Experiments on Llama and Qwen series demonstrate that RED substantially recovers reasoning while maintaining high training efficiency and SOTA general ability.

Junlin He, Yihong Tang, Tong Nie +5

The Hong Kong Polytechnic University, Hong Kong SAR, China · McGill University, Montreal, QC, Canada

LLM PruningLLM CompressionLanguage Model DistillationLLM Reasoning
AskPDF