Sparse Autoencoders

Also known as SAE

Follow this topic Ask about this topic

Momentum

16 papers in the last four weeks, up 100% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 149

All topics

Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effects
Jul 27, 2026Phu Gia Hoang, Anwoy Chatterjee, Tanmoy Chakraborty +2Sparse Autoencoders Mechanistic Interpretability
UKP Lab, Technical University of Darmstadt · Department of Electrical Engineering Indian Institute of Technology Delhi, India
Are Single-Token Sparse Autoencoder Features Causally Necessary? Layer-Depth and SAE-Family Effects
Jul 22, 2026Seonglae Cho, Zekun Wu, Kleyton Da Costa +3LLM Interpretability Sparse Autoencoders
Holistic AI · University College London
Measuring Monosemanticity in Sparse Autoencoders via Latent Activation Coherence
Jul 20, 2026Katarzyna Filus, Sebastian PokucińskiRepresentation Disentanglement Sparse Autoencoders
Institute of Theoretical and Applied Informatics, Polish Academy of Sciences, Bałtycka 5, 44-100 Gliwice, Poland · Department of Applied Informatics, Silesian University of Technology, Akademicka 16, 44-100 Gliwice, Poland
Decoder-Preserving Sparse Autoencoders: Which Readouts Survive Sparse Compression?
Jul 19, 2026Aniket DeshpandeModel Compression Activation Sparsity
Department of Physics, University of Illinois Urbana-Champaign
Persistent Sparse Autoencoders: Learning Feature-Specific Timescales in Language Model Representations
Jul 19, 2026Haoyan Luo, Mateo Espinosa Zarlenga, Mateja JamnikSparse Autoencoders Mechanistic Interpretability
University of Cambridge · University of Oxford
From Geometric Recovery to Causal Validation: A Reproducible Audit of Sparse Autoencoder Features, from Superposition Geometry to Causal Inertness
Jul 13, 2026Mohamed Abdessalem BalFeature Superposition Neural Representation Geometry
Independent Researcher, Algiers, Algeria
When Are Sparse Feature Interventions Actually Localized? Matched Evaluation for SAE-Based Safety Control
Jul 11, 2026Daming LuoSparse Autoencoders LLM Safety Evaluation
University of Technology Sydney
When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities
Jul 9, 2026Weiduo Liao, Yunqiao Yang, Ying WeiVLM Interpretability Sparse Autoencoders
Zhejiang University · Nanyang Technological University
Cross-seed explainability using Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoders
Jul 9, 2026Bendegúz Váradi, Zoltán KmettyTransformer Interpretability Sparse Autoencoders
Centre for Social Sciences, CSS-RECENS Research Group, Budapest 1097, Hungary. · Department of Sociology, Faculty of Social Sciences, Eötvös Loránd University, Budapest, Hungary.
A Coin Flip Per Token: Bernoulli Sparse Steering of Large Language Models
Jul 6, 2026Nima Eshraghi, Lovedeep Gondara, Yuqing Huang +5Language Model Steering Sparse Autoencoders
1The Vanguard Group, Inc. · University of Toronto
Expander Sparse Autoencoders: Parameter-Efficient Dictionaries for Mechanistic Interpretability
Jul 2, 2026Rodrigo Mendoza-SmithSparse Recovery Activation Sparsity
Independent Researcher.
Look But Don't Touch with Sparse Autoencoders for Unlearning in Diffusion Models
Jun 30, 2026Enrico Cassano, Riccardo Renzulli, Rayyan Ahmed +2Sparse Autoencoders Diffusion Model Unlearning
University of Turin, Computer Science Department, Italy · LTCI, Télécom Paris, Institut Polytechnique de Paris, France
Resolving superposition in AI for interpretability and cross-modal alignment in patient-neuronal images
Jun 30, 2026Jisung Park, Seohyeon Kang, Daeun Yoo +8Feature Superposition Neural Representation Geometry
1KAIST · 2Konyang University · 4UCL Queen Square Institute of Neurology & The Francis Crick Institute +1
C $^{2}$ R: Cross-sample Consistency Regularization Mitigates Feature Splitting and Absorption in Sparse Autoencoders
Jun 29, 2026Haoran Jin, Xiting Wang, Shijie Ren +2LLM Interpretability Representation Disentanglement
University of Science and Technology of China · State Key Laboratory of Cognitive Intelligence · Gaoling School of Artificial Intelligence, Renmin University of China
Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders
Jun 29, 2026Chungpa Lee, Jihoon Kwon, Kyle Min +1Cross-Modal Alignment Sparse Autoencoders
Yonsei University, Korea · Seoul National University, Korea · Oracle, USA
Turn-Averaged SAEs for Feature Discovery and Long-Context Attribution
Jun 26, 2026Kevin Der, Harish Kamath, Ben ThompsonLLM Interpretability Sparse Autoencoders
Anthropic Fellows Program · Anthropic
VASAE: Naming SAE Dictionary Directions with Vocabulary-Aligned Anchoring
Jun 26, 2026Kairui Zhang, Ziwen Yu, Zahraa S. Abdallah +1LLM Interpretability Sparse Autoencoders
Intelligent Systems Laboratory, University of Bristol, Bristol, UK · University of Bristol, Bristol, UK · University of Amsterdam, Amsterdam, Netherlands
PairSAE: Mechanistic Interpretability from Pair Representations in Protein Co-Folding
Jun 25, 2026Giosue Migliorini, Aristofanis Rontogiannis, Grigori Guitchounts +3Sparse Autoencoders Mechanistic Interpretability
University of California, Irvine · Flagship Pioneering
Beyond the Hard Budget: Sparsity Regularizers for More Interpretable Top-k Sparse Autoencoders
Jun 25, 2026Nathanaël Jacquier, Maria Vakalopoulou, Mahdi S. HosseiniActivation Sparsity Sparse Autoencoders
Université Paris-Saclay, CentraleSupélec, France · Department of Computer Science and Software Engineering (CSSE), Concordia University, Montreal, QC, Canada · Université Paris-Saclay, CentraleSupélec, Gustave Roussy, INSERM, IHU PRISM, Cancer Data Science Unit, France +2
Discovering Millions of Interpretable Features with Sparse Autoencoders
Jun 25, 2026XinYang He, Wei Wang, Bing Zhao +5LLM Interpretability Sparse Autoencoders
AI DATA, Alibaba Group Holding Limited · Beijing Institute of Technology
Steering Vision-Language Models with Joint Sparse Autoencoders
Jun 24, 2026Huizhen Shu, Xuying Li, Hongxu Lin +2VLM Interpretability Sparse Autoencoders
yunshanai · HKUST(Guangzhou) · Zidongtaichu +1
Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations
Jun 23, 2026Jonas Klotz, Cassio F. Dantas, Pallavi Jain +2VLM Interpretability Sparse Autoencoders
The Berlin Institute for the Foundations of Learning and Data (BIFOLD) · Technische Universität Berlin, Germany · INRAE +2
Do Sparse Autoencoders Learn Meaningful Concept Hierarchies?
Jun 22, 2026Nils Grandien, David Steinmann, Felix Friedrich +1Hierarchical Representation Learning Sparse Autoencoders
Computer Science Department, TU Darmstadt · Hessian Center for AI (hessian.AI) · Black Forest Labs +1
Extraction and Analysis of Multimodal Concepts in Vision Language Models through Sparse Autoencoders
Jun 19, 2026Sergio Lanza, Jae Hee Lee, Stefan WermterVision-Language Models VLM Interpretability
Knowledge Technology, Department of Informatics, University of Hamburg, Hamburg, Germany
Aligning Sentence Embeddings to Human Concepts via Sparse Autoencoders
Jun 19, 2026Wonseok Shin, Songkuk KimText Embeddings Sentence Embeddings
Yonsei University
From Sparse Features to Trustworthy Proxies: Certifying SAE-Based Interpretability
Jun 16, 2026Dibyanayan Bandyopadhyay, Asif EkbalLLM Interpretability Sparse Autoencoders
Department of Computer Science and Engineering, Indian Institute of Technology Patna, Bihar, India.
SAE Interventions are Unreliable: Post-Intervention Recovery of Suppressed Behavior
Jun 16, 2026Mingyue Cui, Linghui Shen, Xingyi YangSparse Autoencoders Mechanistic Interpretability
The Hong Kong Polytechnic University
SAE++: Cascaded Sparse Autoencoders Learn Multi-Level Visual Concepts in Multimodal LLMs
Jun 15, 2026Yusong Zhao, Hengyi Wang, Tanuja Ganu +2Hierarchical Representation Learning Multimodal Large Language Models
Rutgers University. · Microsoft.
Analyzing Visual Aircraft Representations with Sparse Autoencoders
Jun 13, 2026Deepshik SharmaVisual Representation Learning Sparse Autoencoders
Department of Computer Science & Engineering · Jain University, Bangalore, India
Size Doesn't Matter: Cosine-Scored Sparse Autoencoders
Jun 13, 2026Silen Naihin, Lev StamblerRepresentation Learning Sparse Autoencoders
Experiential Labs · Tear Labs

Jul 27, 2026, cs.LGJ/K move · Enter open · S save

Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effects

Phu Gia Hoang, Anwoy Chatterjee, Tanmoy Chakraborty, Iryna Gurevych+1

UKP Lab, Technical University of Darmstadt · Department of Electrical Engineering Indian Institute of Technology Delhi, India

Sparse Autoencoders Mechanistic Interpretability Causal Interventions in Language Models

Open paper AskPDF arXiv

The wide-scale use of sparse autoencoders (SAEs) as interpretability tools is limited by inconsistent links between SAE features and model behavior. Features with clear activation descriptions may have weak or unexpected causal effects; steering can vary across prompts or oppose the intended direction; and activation-based feature selection can miss features that produce the desired output change. Prior work has studied feature geometry inside the model, where features are computed. We instead study the geometry of changes in model logits caused by feature interventions. We introduce Feature-Effect Geometry Analysis (FEGA), an unsupervised framework that removes the same active SAE feature across contexts and analyzes the resulting cloud of logit changes. Across SAE variants, consistent one-dimensional effects are rare: few features behave like reusable directions. To interpret this variation, we distinguish value-like features, tied to static information such as factual attributes, from pointer-like features, associated with context-dependent operations. Value-like features more often exhibit structured, low-dimensional effects, although these effects typically span several directions. Pointer-like features, by contrast, predominantly exhibit diffuse effects. Our results show that a feature can be interpretable and causally relevant without providing a stable direction for steering.

Momentum

Related topics

Latest papers 149