stat.MLMay 7, 2026

Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimization

Authors: Andy Zeyi LiuElliot PaquetteJohn Sous

Organizations: Department of Applied Physics, Yale University, New Haven, CT 06511, United States · Department of Mathematics and Statistics, McGill University · Department of Applied Physics, Yale University, New Haven, Connecticut 06511, USA · Energy Sciences Institute, Yale University, West Haven, Connecticut 06516, USA

Abstract

Training loss and throughput can hide distinct internal representation in language-model training. To examine these hidden mechanics, we use spectral measurements as practical and operational diagnostics. Using a controlled family of decoder-only models adapted from the modded NanoGPT codebase, we introduce an empirical protocol based on activation covariance and per-sample gradient SVD spectra. This dual-view reveals three empirical findings and one mechanistic explanation. First, batch size acts as a latent determinant of representation geometry: runs that reach equal loss settle into systematically distinct activation spectra. Second, the activation covariance tail measured early in training reliably forecasts downstream token efficiency. Third, movement of the activation spectrum head (leading modes), together with gradient spectra, characterizes underlying learning-dynamics changes, separating learning-side architectural improvements from primarily execution-side gains. These predictive and diagnostic signals persist across the 12-, 36-, and 48-layer model tiers. Finally, a mechanistic model proves the main observations and explains how activation covariance spectra correlate with task-aligned feature learning.

Explore similar work

CardsList
  1. A Defense of the Quadratic Model

    Jul 23, 2026Alexandru Meterez, Pranav Ajit Nair, Depen Morwani +3Large ModelsCondition Number