Dictionary Learning

Follow this topic Ask about this topic

Dictionary learning builds a set of reusable features, or atoms, and finds coefficients that combine them to explain data, often encouraging sparse representations. Classical methods alternate between estimating coefficients and updating atoms; newer approaches include sparse autoencoders, structured or multi-dictionary models, and optimal-transport-based methods. These representations support tasks such as signal and image reconstruction, analysis of language-model activations, and learning useful coordinates for dynamical systems. A central challenge is that good reconstruction does not guarantee interpretable or recoverable features: identifiability, stability, and generalization depend on the learned dictionary and the inference procedure.

Latest papers

20 papers1 paper in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Inference and learning in sparse autoencoders as natural gradient flow
Oct 5, 2026Hadi Vafaii, Tejas Rao, David Chanin +6Gradient Descent Sparse Autoencoders
Astera Institute · UC Berkeley · UCL +2
Iterative Atom Refinement: A Monotonicity Principle for Dictionary Learning
Sep 20, 2026Alexander Christie, Miguel Moscoso, Alexei Novikov +2Sparse Recovery Dictionary Learning
Department of Mathematics, Stanford University, Stanford, CA 94305, USA · Department of Mathematics, Universidad Carlos III de Madrid, Leganés, Madrid 28911, Spain · Department of Mathematics, Pennsylvania State University, University Park, PA 16802, USA +1
A Dominant Diffuse Phase in the Sparse Autoencoder Phase Diagram
Sep 9, 2026Alexis D. PlascenciaFeature Superposition Representation Disentanglement
Physical-Support Confidence Sets for Highly Coherent Dictionaries
Aug 20, 2026Guan-Ju PengQuery Identifiability Sparse Recovery
Institute of Data Science and Information Computing, National Chung Hsing University, Taichung, Taiwan
Honest Physical-Support Inference after Latent Dictionary Learning: Collision Singularities and Minimax Resolution
Jul 18, 2026Guan-Ju PengSparse Recovery Dictionary Learning
Institute of Data Science and Information Computing, National Chung Hsing University.
Residual-Guided Dictionary Learning for Spectrally Accurate Koopman Approximation
Jun 27, 2026George Coote, Matthew J. ColbrookSpectral Regularization Koopman Operator Learning
Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Wilberforce Road, CB3 0WA, United Kingdom
On the Sparsity-Storage-Accuracy Tradeoff in Parsimoniously Activated Dictionary Learning
Jun 21, 2026Zihui Zhao, Yuanbo Tang, Yang LiStructured Sparsity Dictionary Learning
Tsinghua University, Institute of Data and Information Shenzhen Key Laboratory of Ubiquitous Data Enabling
Size Doesn't Matter: Cosine-Scored Sparse Autoencoders
Jun 13, 2026Silen Naihin, Lev StamblerRepresentation Learning Sparse Autoencoders
Experiential Labs · Tear Labs
How Optimality Structures Sparse Dictionaries: A Theory for Understanding SAE Representations
Jun 1, 2026William DorrellRepresentation Learning Sparse Autoencoders
Kempner Institute · Harvard University
Ablating Archetypes: The Stability of Archetypal SAEs is an Artifact of Initialization and Metric Design
Jun 1, 2026Michał Brzozowski, Neo Christopher ChungSparse Autoencoders Mechanistic Interpretability
Samsung AI Center, Warsaw, Poland · University of Warsaw, Poland
Toward Identifiable Sparse Autoencoders
May 29, 2026Walter Nelson, Theofanis Karaletsos, Francesco LocatelloParameter Identifiability Sparse Autoencoders
Institute of Science and Technology Austria · 2Pyramidal Inc. · and Achira Inc., USA.
Learning fMRI activations dictionaries across individual geometries via optimal transport
May 20, 2026Sonia Mazelet, Rémi Flamary, Bertrand ThirionLearning to Optimize Geometric Representation Learning
CMAP, Ecole Polytechnique · Palaiseau, France · Mind, Inria-Saclay
Dictionary learning for Kernel EDMD
Apr 28, 2026Erik Lien Bolager, Boumediene Hamzi, Houman Owhadi +2Koopman Operator Learning Nonlinear System Identification
School of Computation, Information and Technology, Munich Data Science Institute, Munich Center for Machine Learning, Technical University of Munich, Germany · Department of Computing and Mathematical Sciences, Caltech, USA · The Alan Turing Institute, UK +1
Diverse Dictionary Learning
Apr 19, 2026Yujia Zheng, Zijian Li, Shunxing Fan +2Latent Variable Models Representation Identifiability
CMU · MBZUAI · NYU
Data-Driven Variational Basis Learning Beyond Neural Networks: A Non-Neural Framework for Adaptive Basis Discovery
Apr 17, 2026Andrew KirulutaRepresentation Learning Dictionary Learning
UC Berkeley, CA
Stop Probing, Start Coding: Why Linear Probes and Sparse Autoencoders Fail at Compositional Generalisation
Mar 30, 2026Vitória Barin-Pacela, Shruti Joshi, Isabela Camacho +2Sparse Autoencoders Sparse Coding
Mila - Québec AI Institute & Université de Montréal · Santa Clara University · Cold Spring Harbor Laboratory
Unbalanced Optimal Transport Dictionary Learning for Unsupervised Hyperspectral Image Clustering
Mar 10, 2026Joshua Lentz, Nicholas Karris, Alex Cloninger +1Image Clustering Dictionary Learning
Tufts University, Department of Mathematics, Medford, MA 02155, USA · University of California San Diego, Department of Mathematics, La Jolla, CA 92093, USA · University of California San Diego, Halıcıo˘glu Data Science Institute, La Jolla, CA 92093, USA
Missing No More: Dictionary-Guided Cross-Modal Image Fusion under Missing Infrared
Mar 9, 2026Yafei Zhang, Meng Ma, Huafeng Li +1Infrared-Visible Image Fusion Multi-Source Image Fusion
Faculty of Information Engineering and Automation, Kunming University of Science and Technology · Department of Biomedical Engineering, Hefei University of Technology
Multi-Dictionary Learning for Low Rank Sparse Coding
Sep 12, 2025Boya Ma, Abram Magner, Maxwell McNeil +1Representation Learning Sample Complexity
University at Albany – SUNY, Computer Science, Albany, NY, USA
Boosting Adversarial Robustness and Generalization with Dictionary Structure
Feb 2, 2025Zhichao Hou, Weizhi Gao, Hamid Krim +2Neural Network Generalization Adversarial Robustness
Department of Computer Science, North Carolina State University, Raleigh, USA.

Oct 5, 2026J/K move · Enter open · S save

Inference and learning in sparse autoencoders as natural gradient flow

Hadi Vafaii, Tejas Rao, David Chanin, Thomas Fel+5

Astera Institute · UC Berkeley · UCL+2

Gradient Descent Sparse Autoencoders Mechanistic Interpretability Dictionary Learning Sparse Coding

Open paper AskPDF arXiv

Sparse autoencoders are widely used to uncover interpretable features in neural networks, yet reliable recovery remains difficult when features overlap or activate infrequently. These challenges involve both inferring which features explain an input and learning the dictionary that represents them. Here, we unify inference and dictionary learning as natural-gradient flows on a shared variational free energy. We instantiate this framework as BeFOND, an encoder-free sparse coding model with closed-form inference and learning dynamics. We show how recurrent explaining away reduces interference between overlapping features, while Fisher preconditioning can compensate for the slow learning of rare features. On synthetic data, BeFOND improves dictionary recovery and rare-feature detection, with a growing advantage over amortized baselines as superposition increases. On language-model activations, it improves single-feature concept detection and selective intervention, outperforming pretrained reference SAEs with substantially less training data. Its feature quality continues to improve with dictionary width, whereas the evaluated baselines largely plateau. Together, these results show how improving inference and learning within a unified probabilistic framework can make better use of data and dictionary capacity to interpret and intervene on neural representations.