cs.LGNov 13, 2026

Deep Latents, Long Memory: Latent persistence explains temporal correlations in transformer activations

Authors: Francesco Cagnetta, Santiago Acevedo

Organizations: Simplex · SISSA

Abstract

The hidden layers of Large Language Models (LLMs) are believed to encode representations of the data whose abstraction increases with depth. However, in the absence of an operational notion of abstraction, this statement is hard to test quantitatively. Here, we propose an alternative view, building on the principle that the correlations of a sequence reflect the structure of the process that generated it. We define the autocorrelation function of a layer as the average dot product between its activations at two context positions separated by a lag. Latent variables encoded persistently along the context create slowly decaying tails in the autocorrelation, and we find these tails to grow across layers in both synthetic and natural data. In synthetic data with controlled hierarchical latent structure, this growth stems from the encoding of abstract latent variables, which govern long stretches of the input. Indeed, the persistence of latent codes predicts the qualitative structure of autocorrelations and accounts for most of the empirical measures. In real text, we take sparse-autoencoder features as proxies for latent variables, and find that their persistence correlates strongly with the magnitude of autocorrelations across datasets and LLMs. In particular, the persistence time increases with depth, as do autocorrelations, implying that deeper latent features persist for longer context spans. We complement this analysis by explicitly ablating the persistent sparse-autoencoder features and observing a consequent decrease in the autocorrelations.

Figures & tables

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Persistent Sparse Autoencoders: Learning Feature-Specific Timescales in Language Model Representations

    Jul 19, 2026Haoyan Luo, Mateo Espinosa Zarlenga, Mateja JamnikSparse AutoencodersMechanistic Interpretability

  2. LLM Layers Immediately Correct Each Other

    Sep 7, 2026Arjun Patrawala, Jiahai Feng, Erik Jones +1Transformer InterpretabilityRepresentation Geometry in Language Models

  3. Layer Collapse in Diffusion Language Models

    May 7, 2026Alexander Conzelmann, Albert Catalan-Tatjer, Shiwei LiuLLM CompressionDiffusion Language Models