Improving Sparse Autoencoders

Follow this topic Ask about this topic

Momentum

6 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 47

All topics

Active Budget Can Kill Sensitivity: Diagnosing and Repairing TopK Sparse Autoencoder Reliability
Sep 29, 2026Zhenting Huang, Bo Jiang, Junnan Liu +2Improving Sparse Autoencoders Top-K
Beihang University · Monash University · Zhongguancun Laboratory
Identifying ODEs from Unstructured Data with Causal Representation Learning
Sep 29, 2026Alessandro Trenta, Riccardo Massidda, Davide Bacciu +1Ordinary Differential Equations Causal Representation Learning
Department of Computer Science University of Pisa · Saarland Informatics Campus Saarland University & University of Amsterdam
Beyond Token Scale: Chunk-Level Sparse Autoencoders for Reliable Semantic Feature Discovery
Sep 28, 2026Xu Wang, Yifan Yang, TingHao YU +1Improving Sparse Autoencoders Deep Semantic Representations
The University of Hong Kong · Hunyuan Team, Tencent
Verifying the Linear Representation Hypothesis: How Interpretable Are Vision SAEs?
Sep 28, 2026Teodor Chiaburu, Franz Motzkus, Frank Haußer +1Improving Sparse Autoencoders Interpretability
Fraunhofer SIT · ATHENE National Research Center for Applied Cybersecurity · AUMOVIO +3
Decodability is Not Causality: Dissociating Probe Readouts from Behavioral Drivers via SAE Decomposition
Sep 16, 2026Devesh Tiwari, Camille Davis, Shivank Sinha +3Model Activations Causal Reasoning
WW-P High School South · Phillips Academy Andover · Dublin High School +3
What Does an LLM Learn from Reinforcement Learning? A Mechanistic Interpretability Perspective with Fixed-SAE Track
Sep 14, 2026Lingheng Du, Yiming Tang, Xufeng Duan +1Large Language Model Reinforcement Learning Improving Sparse Autoencoders
Peking University · National University of Singapore · The Chinese University of Hong Kong
MURANO: Design, Run, and Reproduce Mechanistic Interpretability Experiments as Composable Pipelines
Aug 31, 2026Alireza Bayat Makou, Emirhan Böge, Phu Gia Hoang +5Interpretability Mechanistic Interpretability
Ubiquitous Knowledge Processing (UKP) Lab, Technical University of Darmstadt, Germany · Cluster of Excellence “Reasonable Artificial Intelligence” (RAI), hessian.AI, Germany · National Research Center for Applied Cybersecurity ATHENE, Germany +1
Where You Measure Decides What You Measure: Position Selection in Ablation-Based SAE Evaluation
Aug 13, 2026Valentin NoëlImproving Sparse Autoencoders Language Model Computes
Devoteam
PEAK: Precise and Persistent Concept Erasure via k-Sparse Autoencoders
Aug 11, 2026Man Jiang, Ouxiang Li, Weibao Xue +4Concept Erasure Text-To-Image Diffusion Models
Hefei University of Technology · University of Science and Technology of China · University of Macau
Post-Hoc Sparse Coding of Latent Communication Between Vision-Language Model Agents
Aug 10, 2026Di Wu, Xiaohui ZhuImproving Sparse Autoencoders
Xi’an Jiaotong-Liverpool University
Minimizing Targeted Activations: Input-Only Suppression of Evaluation-Awareness Latents in Large Language Models
Jul 28, 2026Deepanshu Mody, Samarth Agarwal, Utkarsh Mittal +1Model Activations Large Language Model Safety
Center for Data Science, New York University
Decoder-Preserving Sparse Autoencoders: Which Readouts Survive Sparse Compression?
Jul 19, 2026Aniket DeshpandeImproving Sparse Autoencoders Reconstruction Error
Department of Physics, University of Illinois Urbana-Champaign
Persistent Sparse Autoencoders: Learning Feature-Specific Timescales in Language Model Representations
Jul 19, 2026Haoyan Luo, Mateo Espinosa Zarlenga, Mateja JamnikImproving Sparse Autoencoders Future Latent Representations
University of Cambridge · University of Oxford
When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities
Jul 9, 2026Weiduo Liao, Yunqiao Yang, Ying WeiImproving Sparse Autoencoders Modalities
Zhejiang University · Nanyang Technological University
Cross-seed explainability using Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoders
Jul 9, 2026Bendegúz Váradi, Zoltán KmettyImproving Sparse Autoencoders Bert-Based Models
Centre for Social Sciences, CSS-RECENS Research Group, Budapest 1097, Hungary. · Department of Sociology, Faculty of Social Sciences, Eötvös Loránd University, Budapest, Hungary.
Expander Sparse Autoencoders: Parameter-Efficient Dictionaries for Mechanistic Interpretability
Jul 2, 2026Rodrigo Mendoza-SmithImproving Sparse Autoencoders Intermediate Decoder Layers
Independent Researcher.
Turn-Averaged SAEs for Feature Discovery and Long-Context Attribution
Jun 26, 2026Kevin Der, Harish Kamath, Ben ThompsonSparse Autoencoder Features Improving Sparse Autoencoders
Anthropic Fellows Program · Anthropic
VASAE: Naming SAE Dictionary Directions with Vocabulary-Aligned Anchoring
Jun 26, 2026Kairui Zhang, Ziwen Yu, Zahraa S. Abdallah +1Sparse Autoencoder Features Improving Sparse Autoencoders
Intelligent Systems Laboratory, University of Bristol, Bristol, UK · University of Bristol, Bristol, UK · University of Amsterdam, Amsterdam, Netherlands
Beyond the Hard Budget: Sparsity Regularizers for More Interpretable Top-k Sparse Autoencoders
Jun 25, 2026Nathanaël Jacquier, Maria Vakalopoulou, Mahdi S. HosseiniImproving Sparse Autoencoders Sparsity
Université Paris-Saclay, CentraleSupélec, France · Department of Computer Science and Software Engineering (CSSE), Concordia University, Montreal, QC, Canada · Université Paris-Saclay, CentraleSupélec, Gustave Roussy, INSERM, IHU PRISM, Cancer Data Science Unit, France +2
Steering Vision-Language Models with Joint Sparse Autoencoders
Jun 24, 2026Huizhen Shu, Xuying Li, Hongxu Lin +2Improving Sparse Autoencoders Cross-Modal Attention
yunshanai · HKUST(Guangzhou) · Zidongtaichu +1
Aligning Sentence Embeddings to Human Concepts via Sparse Autoencoders
Jun 19, 2026Wonseok Shin, Songkuk KimRetrieval Layer Agentic Retrieval-Augmented Generation Systems
Yonsei University
Effects of sparsity and superposition on loss in simple autoencoders
Jun 16, 2026Mriganka Basu Roy Chowdhury, Eric McLaughlin WeinerImproving Sparse Autoencoders Autoencoder Architectures
Department of Statistics, UC Berkeley · Department of Materials Science, UC Berkeley
Cascaded Sparse Autoencoders Learn Multi-Level Visual Concepts in Multimodal LLMs
Jun 15, 2026Yusong Zhao, Hengyi Wang, Tanuja Ganu +2Improving Sparse Autoencoders Multimodal Large Language Models
Rutgers University · Microsoft
Rational Sparse Autoencoder
Jun 12, 2026Naiyu Yin, Yue YuImproving Sparse Autoencoders Autoencoder Architectures
Department of Mathematics · Lehigh University · Bethlehem, PA 18015
Decompose Sparsely Where You Should, Absorb Densely Where You Should No
Jun 12, 2026Ruixuan Deng, Zehao Jin, Zekun Wang +1Improving Sparse Autoencoders Sparsity
School of Psychology Georgia Institute of Technology · College of Computing Georgia Institute of Technology · School of Interactive Computing Georgia Institute of Technology
Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders
Jun 10, 2026Gleb Gerasimov, Timofei Rusalev, Nikita Balagansky +3Improving Sparse Autoencoders Feature Learning
T-Tech
Ablating Archetypes: The Stability of Archetypal SAEs is an Artifact of Initialization and Metric Design
Jun 1, 2026Michał Brzozowski, Neo Christopher ChungImproving Sparse Autoencoders Unsupervised Dictionary Learning
Samsung AI Center, Warsaw, Poland · University of Warsaw, Poland
Toward Identifiable Sparse Autoencoders
May 29, 2026Walter Nelson, Theofanis Karaletsos, Francesco LocatelloImproving Sparse Autoencoders Sparse Autoencoder Features
Institute of Science and Technology Austria · 2Pyramidal Inc. · and Achira Inc., USA.
BioRefusalAudit: Auditing Biosecurity Refusal Depth Using General and Domain-Fine-Tuned Sparse Autoencoders
May 28, 2026Caleb DeLeeuwRefusals Large Language Model Safety
Independent researcher
Sign-Aware Gated Sparse Autoencoders: Modeling Anticorrelated Features with Bi-Jump-ReLU Activations
May 27, 2026Bartosz Wieciech, Zmnako Awrahman, Marcin Czelej +2Improving Sparse Autoencoders Model Activations
Amazon Web Services
ReSAE: Residualized Sparse Autoencoders for Multi-Layer Transformer Interventions
May 27, 2026Prathyush Poduval, Calvin Yeung, Neel Desai +1Improving Sparse Autoencoders Transformer Residual Streams
Department of Computer Science University of California, Irvine Irvine, CA 92697 · Georgia Institute of Technology Atlanta, GA 30332
Universal Boosts, Specific Suppressors: Sparse Autoencoder Steering of Medical Vision-Language Models
May 24, 2026Farhad Nooralahzadeh, Benjamin Gundersen, Nicolas Deperrois +7Medical Vision-Language Models Improving Sparse Autoencoders
University of Zurich and University Hospital of Zurich, Switzerland · Zurich University of Applied Sciences, Switzerland · Kobe University, Japan +3
Polymorphism Is Rotation: Operational Mechanistic Interpretability from a Two-Layer Transformer to Pythia-70m
May 23, 2026Jordan F. McCannTransformer Architectures Improving Sparse Autoencoders
Independent Researcher
Event-Grounded Sparse Autoencoders for Vision-Language-Action Policies
May 17, 2026Xinchen Jin, Aditya Chatterjee, Pranav Kumar +1Interpretability Improving Sparse Autoencoders
Department of Computer Science, Purdue University West Lafayette, IN 47907
Mechanistic Interpretability of EEG Foundation Models via Sparse Autoencoders
May 13, 2026William Lehn-Schiøler, Magnus Ruud Kjær, Rahul Thapa +10Electroencephalography Foundation Models Improving Sparse Autoencoders
BrainCapture, Kongens Lyngby, Denmark · DTU Health Tech, Technical University of Denmark, Kongens Lyngby, Denmark · DTU Compute, Technical University of Denmark, Kongens Lyngby, Denmark +5
Domain Restriction via Multi SAE Layer Transitions
May 12, 2026Elias Shaheen, Avi MendelsonDomain-Specific Language Improving Sparse Autoencoders
Technion – Israel Institute of Technology, Haifa, Israel.
Compositional Literary Primitives in Instruction-Tuned LLMs: Cross-Architectural SAE Features for Self, Style, and Affect
May 11, 2026Joao Paulo Cavalcante Presa, Savio Salvarino Teles de OliveiraEmotion Self
Federal University of Goias (UFG)
The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling Laws
May 11, 2026Eslam Zaher, Maciej Trzaskowski, Quan Nguyen +1Improving Sparse Autoencoders Scaling Laws
ARC Training Centre for Information Resilience (CIRES) · School of Mathematics and Physics, University of Queensland · Institute for Molecular Bioscience, University of Queensland +2
Tree SAE: Learning Hierarchical Feature Structures in Sparse Autoencoders
May 8, 2026Tue M. Cao, Hoang X. Nhat, Raed Alharbi +2Improving Sparse Autoencoders Hierarchical
Hanoi University of Science and Technology, Hanoi, Vietnam · University of Florida, Florida, USA · Computer Science Department, Saudi Electronic University
Pairwise matrices for sparse autoencoders: single-feature inspection mislabels causal axes
May 4, 2026Michael A. Riegler, Birk Sebastian Frostelid Torpmann-HagenImproving Sparse Autoencoders Model Activations
SimulaMet, Norway · Simula, Norway
GeoSAE: Geometric Prior-Guided Layer-Wise Sparse Autoencoder Annotation of Brain MRI Foundation Models
May 3, 2026Favour Nerrise, Lucy Yin, Mohammad H. Abbasi +2Improving Sparse Autoencoders Biomarker
Stanford University · Stanford University, CA
MoRFI: Monotonic Sparse Autoencoder Feature Identification
Apr 29, 2026Dimitris Dimakopoulos, Shay B. Cohen, Ioannis KonstasLarge Language Model Hallucination Model Fine-Tuning
University of Edinburgh, UK · Heriot-Watt University, UK
Improving Sparse Autoencoder with Dynamic Attention
Apr 16, 2026Dongsheng Wang, Jinsen Zhang, Dawei Su +1Improving Sparse Autoencoders Dynamic Sparse Attention
College of Computer Science and Software Engineering, Shenzhen University, China
Step-Level Sparse Autoencoder for Reasoning Process Interpretation
Mar 3, 2026Xuan Yang, Jiayu Liu, Yuhang Lai +3LLM Reasoning Strategies Improving Sparse Autoencoders
Department of Data Science, City University of Hong Kong · Hong Kong Institute of AI for Science, City University of Hong Kong · Li Auto Inc +1
SynthSAEBench: Evaluating Sparse Autoencoders on Scalable Realistic Synthetic Data
Feb 16, 2026David Chanin, Adrià Garriga-AlonsoSparse Autoencoder Features Improving Sparse Autoencoders
University College London · 2MATS · 3Decode Research
Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders
Aug 22, 2025David Chanin, Adrià Garriga-AlonsoImproving Sparse Autoencoders Model Activations
University College London · MATS Research
Position: Use Sparse Autoencoders to Discover Unknowns
Jun 30, 2025Kenny Peng, Rajiv Movva, Jon Kleinberg +2Improving Sparse Autoencoders Model Discovery
1Cornell University · 2UC Berkeley

Sep 29, 2026, cs.AIJ/K move · Enter open · S save

Active Budget Can Kill Sensitivity: Diagnosing and Repairing TopK Sparse Autoencoder Reliability

Zhenting Huang, Bo Jiang, Junnan Liu, Zhixing Tan+1

Beihang University · Monash University · Zhongguancun Laboratory

Improving Sparse Autoencoders Top-K Sparse Supervision Interpretability Token Budget Allocation Diagnosis

Open paper AskPDF arXiv

Sparse autoencoders (SAEs) are increasingly scaled to wider dictionaries to recover fine-grained structure from large language model activations. However, a feature is useful for interpretation only if it remains a stable unit of analysis when the same meaning is expressed in different surface forms. We study this reliability question for TopK SAEs via feature sensitivity. Experiments demonstrate that scaling selectively reduces the sensitivity of rare features, while common features remain comparatively stable. A controlled width

\times k

factorial experiment identifies the active budget k as the root cause: the degradation arises from the selection boundary rather than dictionary width alone. We attribute this failure to the geometry of TopK selection. The active margin, the distance to the cutoff, predicts feature loss without thresholds. Guided by this margin diagnosis, we introduce pairwise rank stabilization. Our method targets ordering failures at the cutoff and improves rare-feature sensitivity by

8.83

percentage points, while keeping reconstruction and alive-feature coverage near the baseline. Overall, our results suggest that wide TopK SAEs should be evaluated not only by reconstruction, sparsity, and feature count, but also by feature reliability under semantic variation and boundary geometry for stable interpretability.

Momentum

Related topics

Latest papers 47