Sparse Autoencoders

Also known as SAE

Ask about this topic

Overview

Sparse autoencoders learn a dictionary of features that can reconstruct a model’s dense activations using only a few active features at a time, helping researchers investigate superposition in language, vision, and multimodal models. Common approaches include L1- and Top-k-based training, BatchTopK, and structured variants that model feature groups, hierarchies, or relationships across layers and modalities. Researchers use the resulting features to explain model behavior and to steer, audit, or remove targeted behaviors, while testing them with reconstruction and interpretability benchmarks as well as causal interventions. A central open tension is that sparse, human-readable features are not necessarily stable across training runs or faithful causes of behavior, and better reconstruction does not by itself guarantee reliable explanations or control.

Latest papers

CardsList

220 papers42 papers in the last four weeks, up 133% on the four weeks before. 0.2% of all new papers.

  1. Sparse Autoencoders as Implicit Diversification Regularisers for Cross-Sectional Cryptocurrency Returns

    Oct 29, 2026Weijia Han, Lisha QuFinancePortfolio Optimization

    University of Washington, Seattle, Washington, USA · University of Washington, Bothell, Washington, USA

  2. From Error Prediction to Causal Influence: Auditing SAE Features in Multiple-Choice QA

    Oct 27, 2026Aria Nourbakhsh, Adelaide Danilov, Oleksandr Marchenko Breneur +1Sparse AutoencodersCausal Interpretability

    Department of Computer Science (DCS), University of Luxembourg

  3. Structure, Not Sequence: Tracking and Validating Folding Mechanisms in ESMFold's Trunk with Sparse Autoencoders

    Oct 26, 2026Bohan Liu, Xiaohan Fan, Wenqian Ye +1Protein Structure PredictionNeural Network Interpretability

    University of Virginia Charlottesville VA 22903, USA

  4. MI-MIDI 2: From Observation to Control with Sparse Autoencoders in Text-to-MIDI Generation

    Oct 23, 2026Jakub Poćwiardowski, Mateusz ModrzejewskiMechanistic InterpretabilitySparse Autoencoders

    Institute of Computer Science, Warsaw University of Technology, Warsaw, Poland

  5. TaxoSAE: Rethinking Sparse Autoencoders through Taxonomic Logic

    Oct 22, 2026Xinyu Guo, Minglai Yang, Clayton T. Morrison +2Sparse AutoencodersHierarchical Representation Learning

    University of Arizona · Scale AI

  6. Rethinking SAE Evaluation: An Atomic Interpretable Unit Framework Reveals Hidden Polysemanticity and Redundancy

    Oct 21, 2026Riteng Zhang, Xiaoqian WangSparse AutoencodersExplainability Evaluation

    Purdue University

  7. Building Auditable AI Text Detectors with Sparse Autoencoders

    Oct 21, 2026Liam Dugan, Callum McDougall, Matthew Denton +3AI-Generated Text DetectionExplainable Text Classification

    University of Pennsylvania · Google DeepMind

  8. VLMProbe: An Open-source Library for VLM Probing and Interpretation

    Oct 21, 2026Gexin Huang, Haisheng Fu, Yanting Yang +4VLM InterpretabilitySparse Autoencoders

    The University of British Columbia · Vector Institute

  9. The Router Already Knows

    Oct 20, 2026Andrew Bistras, Marty JohnsonSparse AutoencodersMixture-of-Experts Language Models

    University of California, Los Angeles · Astrid Cybernetics

  10. Shaped SAEs: Expanding the Toolkit for Mechanistic Interpretability

    Oct 19, 2026Oveys Delafrooz Noroozi, Bhagyashree Puranik, Upamanyu MadhowLLM InterpretabilitySparse Autoencoders

    University of California, Santa Barbara

  11. Strength Controls Change How Interventions Compare Across Sparse Autoencoder Codes

    Oct 19, 2026Jinchen Zhang, Zheyuan Liu, Yihan Zhu +1Transformer InterpretabilitySparse Autoencoders

    University of Notre Dame

  12. A Symmetry-Matching Approach to Blind-Spot Reduction in Sparse Autoencoders

    Oct 17, 2026Csaba BaloghSparse AutoencodersRepresentation Geometry in Language Models

    National Directorate General for Hospitals, Budapest, Hungary · Data Science in Health Programme, Semmelweis University, Budapest, Hungary

  13. Do LMs Know What Variables Do? Interpreting Code Representations via Variable Roles

    Oct 16, 2026Célian Vasson, Daniil Ognev, Kentaro Inui +1LLM InterpretabilityCode Language Models

    Sorbonne Université · MBZUAI · RIKEN AIP +1

  14. Beyond SAEs: Consistent Concept Discovery with Sparse Coding

    Oct 16, 2026Jinho Choi, Hyesu Lim, Jaegul Choo +1Sparse AutoencodersSparse Coding

    KAIST · Helmholtz Munich

  15. Sparse Autoencoders Can Induce Local Geometry for Mitigating Catastrophic Forgetting

    Oct 15, 2026Kamil Faber, Katarzyna Filus, Łukasz PawelaCatastrophic ForgettingRegularization-Based Continual Learning

    Faculty of Computer Science, AGH University of Krakow, Krakow, Poland · Institute of Theoretical and Applied Informatics, Polish Academy of Sciences, Gliwice, Poland

  16. Behavioral Compositionality of Sparse Autoencoder Representations

    Oct 15, 2026Haotian Ma, Ziye Mei, Frederic SalaSparse AutoencodersCompositional Representation Learning

    Department of Computer Sciences, University of Wisconsin–Madison

  17. Representation Redistribution in Sparse Autoencoders

    Oct 15, 2026Ziye Mei, Haotian MaSparse AutoencodersPrompt Sensitivity

    Department of Computer Sciences, University of Wisconsin–Madison

  18. Multi-Dimensional Featurizers Yield Fine-Grained Steering and Data Shaping in Robot Foundation Models

    Oct 14, 2026Aiden Swann, Mozes Jacobs, Mathilde Papillon +12Robot Foundation ModelsSparse Autoencoders

    Goodfire · Stanford University · Harvard University +1

  19. Diminishing Returns of Data in Sparse Autoencoders: Functional Behavior does not Guarantee Feature Reproducibility

    Oct 14, 2026Sambit Ghosh, Madelyn Mathai, R. Venkatesh Babu +1Sparse AutoencodersData-Efficient Learning

    IBM · Indian Institute of Science · University of Virginia

  20. Sparse Autoencoders for Mechanistic Interpretability in NLP: A Critical Survey of Recoverability, Interpretability, and Utility

    Oct 14, 2026Salima Lamsiyah, Adelaide Danilov, Oleksandr Marchenko Breneur +1Sparse AutoencodersLLM Interpretability

    Department of Computer Science (DCS), University of Luxembourg

  21. Sparse Transformer: a Controlled Testbed for Mechanistic Interpretability

    Oct 14, 2026Anton Korznikov, Alexey Dontsov, Elena TutubalinaMechanistic InterpretabilityNeuron-Level Interpretability

  22. Sparse Autoencoders Are a Shared Coordinate System for Symbolic Matching

    Oct 14, 2026Xiaocong Yang, Xuanyu Meng, Yuji Zhang +2Neuro-Symbolic AIRepresentation Alignment

    University of Illinois Urbana-Champaign

  23. Readout Scaling Shapes Sparse Feature Recovery

    Oct 13, 2026Brian Xiao, Sarah WilsonSparse AutoencodersSparse Recovery

    University of Waterloo · Columbia University

  24. Evaluating the Causal Faithfulness of SHAP as a Candidate Attribution Method for SAE Features

    Oct 13, 2026Samuel Xie, Noura Al-MoubayedFaithfulness of Language Model ExplanationsSHAP Feature Attribution

    Independent · Durham University

  25. What Do Sparse Autoencoder Feature Discovery Scores Identify? From Activations to Interventions

    Oct 11, 2026Yifeng Guo, Yanran Li, Xiang ChenSparse AutoencodersMechanistic Interpretability

    St. Jude Children’s Research Hospital · Columbia University

  26. Cross-Modal Latent Influence: Patch-Level Training Sample Attribution for Vision-Language Models

    Oct 9, 2026Laura Yao, Shixing Yu, Jie Peng +3VLM InterpretabilityTraining Data Attribution

    UNITES Lab, University of North Carolina at Chapel Hill · Electrical and Computer Engineering, Cornell Tech · Foci Labs +2

  27. Sparse Dictionary Learning for Measure-Valued Data via Linearized Optimal Transport: Efficient Algorithms and Sample Complexity

    Oct 9, 2026Brendan Mallery, Shuchin Aeron, James MurphyRepresentation LearningSparse Coding

    Independent Researcher · Department of Electrical and Computer Engineering, Tufts University · The NSF AI Institute for Artificial Intelligence and Fundamental Interactions (IAIFI) +1

  28. Disentangling Linguistic and Paralinguistic Information with Routed Sparse Autoencoders

    Oct 7, 2026Beimnet Bekele Guta, Xiaoyu Yang, Guangzhi Sun +1Disentangled Representation LearningSelf-Supervised Speech Representation Learning

    Department of Engineering, University of Cambridge, Cambridge, UK

  29. Inference and learning in sparse autoencoders as natural gradient flow

    Oct 5, 2026Hadi Vafaii, Tejas Rao, David Chanin +6Gradient DescentSparse Autoencoders

    Astera Institute · UC Berkeley · UCL +2

  30. Backdooring Sparse Autoencoders

    Oct 5, 2026Enrico Ahlers, Daniel Passon, Tobias Kiecker +2LLM Backdoor AttacksAdversarial Attacks

    Humboldt-Universität zu Berlin Berlin, Germany