Sparse Autoencoders

Also known as SAE

Overview

Sparse autoencoders learn a dictionary of features that can reconstruct a model’s dense activations using only a few active features at a time, helping researchers investigate superposition in language, vision, and multimodal models. Common approaches include L1- and Top-k-based training, BatchTopK, and structured variants that model feature groups, hierarchies, or relationships across layers and modalities. Researchers use the resulting features to explain model behavior and to steer, audit, or remove targeted behaviors, while testing them with reconstruction and interpretability benchmarks as well as causal interventions. A central open tension is that sparse, human-readable features are not necessarily stable across training runs or faithful causes of behavior, and better reconstruction does not by itself guarantee reliable explanations or control.

Latest papers

CardsList

246 papers29 papers in the last four weeks, down 24% on the four weeks before. 0.2% of all new papers.

  1. Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders

    Aug 22, 2025David Chanin, Adrià Garriga-AlonsoActivation SparsityLLM Interpretability

  2. Sparse Autoencoders Can Capture Language-Specific Concepts Across Diverse Languages

    Jul 15, 2025Lyzander Marciano Andrylie, Inaya Rahmanisa, Mahardika Krisna Ihsani +3Multilingual Language ModelsRepresentation Learning

  3. Position: Use Sparse Autoencoders to Discover Unknowns

    Jun 30, 2025Kenny Peng, Rajiv Movva, Jon Kleinberg +2Explainable Artificial IntelligenceSparse Autoencoders

  4. SAEs Can Improve Unlearning: Dynamic Sparse Autoencoder Guardrails for Precision Unlearning in LLMs

    Apr 11, 2025Aashiq Muhamed, Jacopo Bonato, Mona Diab +1Machine UnlearningLLM Unlearning