Sparse Coding

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 31

All topics

Inference and learning in sparse autoencoders as natural gradient flow
Oct 5, 2026Hadi Vafaii, Tejas Rao, David Chanin +6Gradient Descent Sparse Autoencoders
Astera Institute · UC Berkeley · UCL +2
NeuronEye: Query-Guided Visual Concept Activation for Vision-Language Reasoning
Sep 29, 2026Ruiyu Yan, Bowen Chen, Shaowen Wan +1VLM Interpretability VLM Reasoning
New York University · New Jersey Institute of Technology
Iterative Atom Refinement: A Monotonicity Principle for Dictionary Learning
Sep 20, 2026Alexander Christie, Miguel Moscoso, Alexei Novikov +2Sparse Recovery Dictionary Learning
Department of Mathematics, Stanford University, Stanford, CA 94305, USA · Department of Mathematics, Universidad Carlos III de Madrid, Leganés, Madrid 28911, Spain · Department of Mathematics, Pennsylvania State University, University Park, PA 16802, USA +1
Training-Adaptive Convolutional Sparse Coding via Information Bottleneck for Robust Visual Representation
Sep 16, 2026Meng'en Qin, Yinchen Liu, Mingxuan Cui +1Representation Learning Test-Time Adaptation
Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen, China · School of Mathematical Sciences, University of Electronic Science and Technology of China, Chengdu, China · School of Mathematics, Shandong University, Jinan, China
A Dominant Diffuse Phase in the Sparse Autoencoder Phase Diagram
Sep 9, 2026Alexis D. PlascenciaFeature Superposition Representation Disentanglement
Post-Hoc Sparse Coding of Latent Communication Between Vision-Language Model Agents
Aug 10, 2026Di Wu, Xiaohui ZhuSparse Autoencoders Multimodal Token Compression
Xi’an Jiaotong-Liverpool University
Toward a mechanistic understanding of inference in visual cortex and diffusion models
Jul 17, 2026Zeyu Yun, Alexander Belsten, Dasheng Bi +3Mechanistic Interpretability Visual Neuroscience
Dept. of Electrical Engineering and Computer Science, UC Berkeley · Redwood Center for Theoretical Neuroscience, UC Berkeley · Herbert Wertheim School of Optometry and Vision Science, UC Berkeley +3
Accelerating Hierarchical Sparse Predictive Coding with Hybrid Amortized Inference
Jun 26, 2026Kazuhisa FujitaSparse Recovery Efficient Inference
Department of Clinical Engineering, Komatsu University, 10-10 Doihara-Machi, Komatsu, Ishikawa, Japan 923-0921
Beyond the Hard Budget: Sparsity Regularizers for More Interpretable Top-k Sparse Autoencoders
Jun 25, 2026Nathanaël Jacquier, Maria Vakalopoulou, Mahdi S. HosseiniActivation Sparsity Sparse Autoencoders
Université Paris-Saclay, CentraleSupélec, France · Department of Computer Science and Software Engineering (CSSE), Concordia University, Montreal, QC, Canada · Université Paris-Saclay, CentraleSupélec, Gustave Roussy, INSERM, IHU PRISM, Cancer Data Science Unit, France +2
Olfactory-Inspired Sparse Combinatorial Coding for Low-Resource Named Entity Recognition
Jun 20, 2026Bhushan DeshpandeNamed Entity Recognition Representation Learning
Independent Researcher, India
Non-negative Elastic Net Decoding for Information Retrieval
Jun 16, 2026Koki Okajima, Yasutoshi Ida, Tsukasa Yoshida +1Representation Learning Information Retrieval
NTT, Inc.
Size Doesn't Matter: Cosine-Scored Sparse Autoencoders
Jun 13, 2026Silen Naihin, Lev StamblerRepresentation Learning Sparse Autoencoders
Experiential Labs · Tear Labs
Rational Sparse Autoencoder
Jun 12, 2026Naiyu Yin, Yue YuActivation Sparsity LLM Interpretability
Department of Mathematics · Lehigh University · Bethlehem, PA 18015
Decompose Sparsely Where You Should, Absorb Densely Where You Should No
Jun 12, 2026Ruixuan Deng, Zehao Jin, Zekun Wang +1Transformer Interpretability Representation Learning
School of Psychology Georgia Institute of Technology · College of Computing Georgia Institute of Technology · School of Interactive Computing Georgia Institute of Technology
Convolutional Sparse Coding via the Locally Competitive Algorithm on Loihi 2
Jun 7, 2026Geoffrey Kasenbacher, Daniel Ruepp, Gerrit A. EckeRepresentation Learning Neuromorphic Computing
Mercedes-Benz AG · Böblingen, Germany · Institut für Robotik und Kognitive Systeme, Universität zu Lübeck
Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretability
Jun 4, 2026Seyed Arshan Dalili, Mehrdad MahdaviRepresentation Disentanglement Sparse Autoencoders
The Pennsylvania State University
How Optimality Structures Sparse Dictionaries: A Theory for Understanding SAE Representations
Jun 1, 2026William DorrellRepresentation Learning Sparse Autoencoders
Kempner Institute · Harvard University
Toward Identifiable Sparse Autoencoders
May 29, 2026Walter Nelson, Theofanis Karaletsos, Francesco LocatelloParameter Identifiability Sparse Autoencoders
Institute of Science and Technology Austria · 2Pyramidal Inc. · and Achira Inc., USA.
No More K-means: Single-Stage Sparse Coding for Efficient Multi-Vector Retrieval
May 28, 2026Lixuan Guo, Yifei Wang, Tiansheng Wen +3Learned Sparse Retrieval Multi-Vector Retrieval
Stony Brook University · Xidian University · Amazon AGI SF Lab +4
Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
May 28, 2026Adly Templeton, Tom Conerly, Jonathan Marcus +23LLM Interpretability Sparse Autoencoders
Anthropic
Learning Compositional Latent Structure with Vector Networks
May 27, 2026Niclas Pokel, Benjamin F. GreweRepresentation Learning Recurrent Neural Networks
Institute of Neuroinformatics, UZH / ETH Zurich ETH AI Center Zurich, Switzerland
Implicit Binarization via Complex Phase Dynamics in Combinatorial Optimization
May 23, 2026Khen Cohen, Mark Glass, Meir Feder +1Combinatorial Optimization Quadratic Unconstrained Binary Optimization
School of Physics and Astronomy, Tel Aviv University, Tel Aviv, Israel · School of Electrical and Computer Engineering, Tel Aviv University, Tel Aviv, Israel
The Rate-Distortion-Polysemanticity Tradeoff in SAEs
May 14, 2026Tommaso Mencattini, Francesco Montagna, Francesco LocatelloRate-Distortion Theory Representation Disentanglement
EPFL · Institute of Science and Technology Austria
Sparse Code Uplifting for Efficient 3D Language Gaussian Splatting
May 13, 2026Lovre Antonio Budimir, Yushi Guan, Steve Ryhner +23D Gaussian Splatting Language-Embedded 3DGS
Faculty of Electrical Engineering and Computing, University of Zagreb · Department of Computer Science, University of Toronto · Vector Institute
Joint sparse coding and temporal dynamics support context reconfiguration
May 11, 2026Qianqian Shi, Yue Che, Faqiang Liu +6Representation Learning Continual Learning
Center for Brain-Inspired Computing Research (CBICR), Department of Precision Instrument, Tsinghua University, Beijing, China · Guangdong Institute of Intelligence Science and Technology, Hengqin, China · Optical Memory National Engineering Research Center, Tsinghua University, Beijing, China +3
Feature Starvation as Geometric Instability in Sparse Autoencoders
May 6, 2026Faris Chaudhry, Keisuke Yano, Anthea MonodRepresentation Learning Sparse Autoencoders
Imperial College London South Kensington Campus, London SW7 2AZ, UK · Institute for Statistical Mathematics 10-3 Midori-cho, Tachikawa Tokyo 190-8562, Japan
Do Sparse Autoencoders Capture Concept Manifolds?
Apr 30, 2026Usha Bhalla, Thomas Fel, Can Rager +9Neural Representation Geometry Sparse Autoencoders
aHarvard University · bNortheastern University · cTechnion IIT +1
Towards Open-Ended Visual Scientific Discovery with Sparse Autoencoders
Nov 21, 2025Jacob Beattie, Samuel Stevens, Neil Rosser +2Vision Foundation Models Sparse Autoencoders
The Ohio State University
Multi-Dictionary Learning for Low Rank Sparse Coding
Sep 12, 2025Boya Ma, Abram Magner, Maxwell McNeil +1Representation Learning Sample Complexity
University at Albany – SUNY, Computer Science, Albany, NY, USA
Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders
Aug 22, 2025David Chanin, Adrià Garriga-AlonsoActivation Sparsity LLM Interpretability
University College London · MATS Research
Kronecker Factorization Improves Efficiency and Interpretability of Sparse Autoencoders
May 28, 2025Vadim Kurochkin, Yaroslav Aksenov, Daniil Laptev +2Representation Learning Activation Sparsity
T-Tech

Oct 5, 2026, cs.LGJ/K move · Enter open · S save

Inference and learning in sparse autoencoders as natural gradient flow

Hadi Vafaii, Tejas Rao, David Chanin, Thomas Fel+5

Astera Institute · UC Berkeley · UCL+2

Gradient Descent Sparse Autoencoders Mechanistic Interpretability Dictionary Learning Sparse Coding

Open paper AskPDF arXiv

Sparse autoencoders are widely used to uncover interpretable features in neural networks, yet reliable recovery remains difficult when features overlap or activate infrequently. These challenges involve both inferring which features explain an input and learning the dictionary that represents them. Here, we unify inference and dictionary learning as natural-gradient flows on a shared variational free energy. We instantiate this framework as BeFOND, an encoder-free sparse coding model with closed-form inference and learning dynamics. We show how recurrent explaining away reduces interference between overlapping features, while Fisher preconditioning can compensate for the slow learning of rare features. On synthetic data, BeFOND improves dictionary recovery and rare-feature detection, with a growing advantage over amortized baselines as superposition increases. On language-model activations, it improves single-feature concept detection and selective intervention, outperforming pretrained reference SAEs with substantially less training data. Its feature quality continues to improve with dictionary width, whereas the evaluated baselines largely plateau. Together, these results show how improving inference and learning within a unified probabilistic framework can make better use of data and dictionary capacity to interpret and intervene on neural representations.

Momentum

Related topics

Latest papers 31