cs.LGNov 18, 2026

Separating Interpretability and Reconstruction in Sparse Autoencoders

Authors: Anh Tien Nguyen, Dimitris Samaras, Mahdi S. Hosseini

Organizations: Stony Brook University · Concordia University · Mila–Quebec AI Institute

Abstract

Sparse Autoencoders (SAEs) are effective tools for mechanistic interpretability by decomposing neural representations into sparse and interpretable features. However, a single sparse representation in traditional SAEs is responsible for both reconstruction fidelity and interpretability, which are not guaranteed to align. To separate these objectives, we propose DualSAE, a modular framework that augments standard SAEs with a plug-in Residual Branch. A standard SAE serves as the Sparse Branch, learning a sparse code that contains the dominant semantics of the input. The added Residual Branch models the non-semantic information omitted by the Sparse Branch. We ensure that the contributions of the two branches towards the overall reconstruction have minimal overlap through a decorrelation loss. We also adopt a two-stage training strategy that first stabilizes sparse representation learning before coupling with the Residual Branch. Across diverse encoder-SAE configurations on two vision datasets and two language corpora, DualSAE achieves mean relative improvements of approximately 29% in reconstruction fidelity and 34% in interpretability (measured by monosemanticity score). Moreover, with more interpretable representations, DualSAE yields consistent performance gains in the downstream applications of topic modeling, hypothesis generation, and LLM steering. Qualitative analysis further suggests that the Sparse Branch indeed preserves dominant semantic content, whereas the Residual Branch captures non-semantic information to improve reconstruction fidelity.

Figures & tables

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. From Input to Output: A Flexible Agent for Dual-End Interpretation of Sparse Autoencoder Features

    Sep 28, 2026Dewen Liu, Zixuan Li, Jonathan Pan +4Sparse AutoencodersMechanistic Interpretability

  2. Sparse Autoencoders for Mechanistic Interpretability in NLP: A Critical Survey of Recoverability, Interpretability, and Utility

    Oct 14, 2026Salima Lamsiyah, Adelaide Danilov, Oleksandr Marchenko Breneur +1Sparse AutoencodersLLM Interpretability

  3. Improving Robustness In Sparse Autoencoders via Masked Regularization

    Apr 7, 2026Vivek Narayanaswamy, Kowshik Thopalli, Bhavya Kailkhura +1Sparse AutoencodersMechanistic Interpretability