Sparse Coding

Ask about this topic

Overview

Sparse coding studies how to represent signals or model activations using a small number of active elements from a dictionary, supporting compression, denoising, and interpretation. Methods include dictionary learning with LASSO or nonnegative penalties, convolutional and structured codes, and fast inference through algorithms such as FISTA and LISTA; sparse autoencoders extend these ideas to neural-network representations. In language and vision models, researchers use learned features to probe and intervene on internal computations, while also applying sparse codes to retrieval, scientific imaging, and compositional learning. A central tension is that stronger sparsity can improve compactness or interpretability but harm reconstruction, and performance often depends on learning a good dictionary rather than merely improving inference.

Latest papers

CardsList

41 papers6 papers in the last four weeks, up 100% on the four weeks before. 0.0% of all new papers.

  1. Beyond SAEs: Consistent Concept Discovery with Sparse Coding

    Oct 16, 2026Jinho Choi, Hyesu Lim, Jaegul Choo +1Sparse AutoencodersSparse Coding

    KAIST · Helmholtz Munich

  2. How Many Ranks Does a Token Need? A Noise-Floor Squelch for Rank-One LoRA Experts

    Oct 13, 2026Mei Okonkwo, Owen Castellanos, Ingrid Solberg +1Low-Rank AdaptationAdaptive Rank Allocation

    University of Edinburgh · Delft University of Technology

  3. Sparse Dictionary Learning for Measure-Valued Data via Linearized Optimal Transport: Efficient Algorithms and Sample Complexity

    Oct 9, 2026Brendan Mallery, Shuchin Aeron, James MurphyRepresentation LearningSparse Coding

    Independent Researcher · Department of Electrical and Computer Engineering, Tufts University · The NSF AI Institute for Artificial Intelligence and Fundamental Interactions (IAIFI) +1

  4. Inference and learning in sparse autoencoders as natural gradient flow

    Oct 5, 2026Hadi Vafaii, Tejas Rao, David Chanin +6Gradient DescentSparse Autoencoders

    Astera Institute · UC Berkeley · UCL +2

  5. NeuronEye: Query-Guided Visual Concept Activation for Vision-Language Reasoning

    Sep 29, 2026Ruiyu Yan, Bowen Chen, Shaowen Wan +1VLM InterpretabilityVLM Reasoning

    New York University · New Jersey Institute of Technology

  6. Invariant Atoms: Sparse Coordinates of Local Semantic Geometry in Language Model Representations

    Sep 29, 2026Muhammad Ahtesham, Xin ZhongRepresentation Geometry in Language ModelsInvariant Representation Learning

    Department of Computer Science University of Nebraska Omaha, Omaha, NE, USA

  7. Iterative Atom Refinement: A Monotonicity Principle for Dictionary Learning

    Sep 20, 2026Alexander Christie, Miguel Moscoso, Alexei Novikov +2Sparse RecoveryDictionary Learning

    Department of Mathematics, Stanford University, Stanford, CA 94305, USA · Department of Mathematics, Universidad Carlos III de Madrid, Leganés, Madrid 28911, Spain · Department of Mathematics, Pennsylvania State University, University Park, PA 16802, USA +1

  8. Training-Adaptive Convolutional Sparse Coding via Information Bottleneck for Robust Visual Representation

    Sep 16, 2026Meng'en Qin, Yinchen Liu, Mingxuan Cui +1Representation LearningTest-Time Adaptation

    Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen, China · School of Mathematical Sciences, University of Electronic Science and Technology of China, Chengdu, China · School of Mathematics, Shandong University, Jinan, China

  9. A Dominant Diffuse Phase in the Sparse Autoencoder Phase Diagram

    Sep 9, 2026Alexis D. PlascenciaFeature SuperpositionRepresentation Disentanglement

  10. HiPACE: Hierarchical Phase-Boundary Analysis and Controlled Evaluation of Feature Absorption in Sparse Autoencoders

    Aug 26, 2026Jinyuan Zhang, Peng He, Yin Yuan +2Sparse AutoencodersSparse Coding

    Hubei University, Wuhan, China

  11. Closed-Form Nonlocal Shrinkage for Multiplicative Image Denoising and SAR Despeckling

    Aug 15, 2026Xuran Hu, Mingzhe Zhu, Djordje Stanković +4Image DenoisingSparse Coding

    School of Electronic Engineering, Xidian University, Xi’an, China · Kunshan Innovation Institute of Xidian University, Kunshan, China · Division of Geoinformatics, KTH Royal Institute of Technology, Stockholm, Sweden +2

  12. PE-CSNet: An equivariant network architecture with learnable patch-based sparse representation

    Aug 11, 2026Kai Li, Haitao Long, Bo Zhang +2Deep UnfoldingCompressed Sensing

    Department of Applied Mathematics, The Hong Kong Polytechnic University, Hung Hom, Hong Kong. · SKLMS and Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China and School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · SKLMS and Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China

  13. Post-Hoc Sparse Coding of Latent Communication Between Vision-Language Model Agents

    Aug 10, 2026Di Wu, Xiaohui ZhuSparse AutoencodersMultimodal Token Compression

    Xi’an Jiaotong-Liverpool University

  14. Toward a mechanistic understanding of inference in visual cortex and diffusion models

    Jul 17, 2026Zeyu Yun, Alexander Belsten, Dasheng Bi +3Mechanistic InterpretabilityVisual Neuroscience

    Dept. of Electrical Engineering and Computer Science, UC Berkeley · Redwood Center for Theoretical Neuroscience, UC Berkeley · Herbert Wertheim School of Optometry and Vision Science, UC Berkeley +3

  15. Accelerating Hierarchical Sparse Predictive Coding with Hybrid Amortized Inference

    Jun 26, 2026Kazuhisa FujitaSparse RecoveryEfficient Inference

    Department of Clinical Engineering, Komatsu University, 10-10 Doihara-Machi, Komatsu, Ishikawa, Japan 923-0921

  16. Beyond the Hard Budget: Sparsity Regularizers for More Interpretable Top-k Sparse Autoencoders

    Jun 25, 2026Nathanaël Jacquier, Maria Vakalopoulou, Mahdi S. HosseiniActivation SparsitySparse Autoencoders

    Université Paris-Saclay, CentraleSupélec, France · Department of Computer Science and Software Engineering (CSSE), Concordia University, Montreal, QC, Canada · Université Paris-Saclay, CentraleSupélec, Gustave Roussy, INSERM, IHU PRISM, Cancer Data Science Unit, France +2

  17. Olfactory-Inspired Sparse Combinatorial Coding for Low-Resource Named Entity Recognition

    Jun 20, 2026Bhushan DeshpandeNamed Entity RecognitionRepresentation Learning

    Independent Researcher, India

  18. Non-negative Elastic Net Decoding for Information Retrieval

    Jun 16, 2026Koki Okajima, Yasutoshi Ida, Tsukasa Yoshida +1Representation LearningInformation Retrieval

    NTT, Inc.

  19. Size Doesn't Matter: Cosine-Scored Sparse Autoencoders

    Jun 13, 2026Silen Naihin, Lev StamblerRepresentation LearningSparse Autoencoders

    Experiential Labs · Tear Labs

  20. Rational Sparse Autoencoder

    Jun 12, 2026Naiyu Yin, Yue YuActivation SparsityLLM Interpretability

    Department of Mathematics · Lehigh University · Bethlehem, PA 18015

  21. Decompose Sparsely Where You Should, Absorb Densely Where You Should No

    Jun 12, 2026Ruixuan Deng, Zehao Jin, Zekun Wang +1Transformer InterpretabilityRepresentation Learning

    School of Psychology Georgia Institute of Technology · College of Computing Georgia Institute of Technology · School of Interactive Computing Georgia Institute of Technology

  22. Convolutional Sparse Coding via the Locally Competitive Algorithm on Loihi 2

    Jun 7, 2026Geoffrey Kasenbacher, Daniel Ruepp, Gerrit A. EckeRepresentation LearningNeuromorphic Computing

    Mercedes-Benz AG · Böblingen, Germany · Institut für Robotik und Kognitive Systeme, Universität zu Lübeck

  23. Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretability

    Jun 4, 2026Seyed Arshan Dalili, Mehrdad MahdaviRepresentation DisentanglementSparse Autoencoders

    The Pennsylvania State University

  24. How Optimality Structures Sparse Dictionaries: A Theory for Understanding SAE Representations

    Jun 1, 2026William DorrellRepresentation LearningSparse Autoencoders

    Kempner Institute · Harvard University

  25. Toward Identifiable Sparse Autoencoders

    May 29, 2026Walter Nelson, Theofanis Karaletsos, Francesco LocatelloParameter IdentifiabilitySparse Autoencoders

    Institute of Science and Technology Austria · 2Pyramidal Inc. · and Achira Inc., USA.

  26. No More K-means: Single-Stage Sparse Coding for Efficient Multi-Vector Retrieval

    May 28, 2026Lixuan Guo, Yifei Wang, Tiansheng Wen +3Learned Sparse RetrievalMulti-Vector Retrieval

    Stony Brook University · Xidian University · Amazon AGI SF Lab +4

  27. Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

    May 28, 2026Adly Templeton, Tom Conerly, Jonathan Marcus +23LLM InterpretabilitySparse Autoencoders

    Anthropic

  28. Learning Compositional Latent Structure with Vector Networks

    May 27, 2026Niclas Pokel, Benjamin F. GreweRepresentation LearningRecurrent Neural Networks

    Institute of Neuroinformatics, UZH / ETH Zurich ETH AI Center Zurich, Switzerland

  29. Implicit Binarization via Complex Phase Dynamics in Combinatorial Optimization

    May 23, 2026Khen Cohen, Mark Glass, Meir Feder +1Combinatorial OptimizationQuadratic Unconstrained Binary Optimization

    School of Physics and Astronomy, Tel Aviv University, Tel Aviv, Israel · School of Electrical and Computer Engineering, Tel Aviv University, Tel Aviv, Israel

  30. The Rate-Distortion-Polysemanticity Tradeoff in SAEs

    May 14, 2026Tommaso Mencattini, Francesco Montagna, Francesco LocatelloRate-Distortion TheoryRepresentation Disentanglement

    EPFL · Institute of Science and Technology Austria