Decompose Sparsely Where You Should, Absorb Densely Where You Should No

Ruixuan Deng; Zehao Jin; Zekun Wang; Zihan Dong

cs.LGJun 12, 2026

Decompose Sparsely Where You Should, Absorb Densely Where You Should No

Authors: Ruixuan Deng, Zehao Jin, Zekun Wang, Zihan Dong

Abstract

Sparse autoencoders (SAEs) are typically trained to reconstruct the \textbf{entire} residual stream through a sparse dictionary, implicitly assuming that all activation content is amenable to sparse, monosemantic decomposition. We question this assumption and hypothesize that activations contain a low-rank, dense component that is computationally important to the model yet inherently unsuitable for sparse representation, which serves as a major source of the persistent dense latents widely observed in trained SAEs. To test this, we add a small rank- $r$ linear bottleneck in parallel with standard SAEs (BatchTopK and Matryoshka), allowing dense structure to be absorbed before sparse reconstruction. On Gemma-2-2B layer 12, a rank-24 bottleneck reduces dense latent count by up to 84% while improving sparse probing and targeted probe perturbation on both architectures at matched sparsity. The absorbed component is (i) \textbf{structurally identifiable} as the top principal components and outlier dimensions; (ii) \textbf{causally necessary}, with removing it raising next-token cross-entropy by 7.5 $\times$ , far exceeding the 2.8 $\times$ from removing the geometrically near-identical top-24 PCA directions; and (iii) \textbf{redundantly encoded by sparse dictionaries}, with ablating 787 maximally aligned sparse features raising cross-entropy by only 2.9 $\times$ and ablating 2,048 topic-aligned features leaving MMLU topic classification virtually unchanged, whereas removing the scaffold drops it from 98.7% to chance. Together, our findings identify a compact, semantically informative and causally important component of residual stream activations (which we term a \textbf{computational scaffold}) that standard sparse dictionaries represent inefficiently, suggesting that the scope of sparsity-based interpretability methods warrants careful re-examination.

Explore similar work

Jul 2, 2026cs.LG

Expander Sparse Autoencoders: Parameter-Efficient Dictionaries for Mechanistic Interpretability

Sparse autoencoders (SAEs) decompose internal activations of neural networks into sparse linear combinations of learned features by fitting an overcomplete dictionary

\mathbf{W}\in\mathbb{R}^{m\times n}

with

m<n

, and inferring a sparse code

\mathbf{x}\in\mathbb{R}^n

from

\mathbf{h}\approx\mathbf{W}\mathbf{x}

. This inference problem closely resembles the canonical setup of compressed sensing, but dense decoders requires

O(mn)

learned values, which becomes costly at large feature counts. We introduce Expander SAEs: TopK SAEs whose decoder and tied encoder are supported on a left-

d

-regular expander mask with

d\ll m

, learning only

dn

decoder values while keeping the sparse-coding problem

(m,n,k)

fixed. The same structure reduces storage and turns the matching-pursuit correlation step

\mathbf{W}^\top \mathbf{r}

in OMP into an

O(dn)

gather-and-reduce operation. Our experiments show that across Pythia-70M/160M, Qwen2.5-3B, and Llama-3.2-1B residual-stream activations, varying

d

traces a consistent storage--fidelity frontier, and that at the most compressed modern-LM setting, Qwen2.5-3B with

d=7

uses

293\times

fewer learned decoder values than the full dense decoder while retaining

84

% of dense CE-loss recovered. Control experiments show that the improved storage--fidelity tradeoff is driven by sparse, diverse decoder support structure rather than by fewer learned decoder values, and that when sparse and dense decoders are compared at matched parameter count, part of the remaining gap comes from encoder amortisation. On the theoretical side, we show that expansion and column flatness are sufficient for identifiability of noiseless

k

-sparse codes, and we derive complementary sufficient conditions under which OMP recovers the support exactly.

Rodrigo Mendoza-Smith

Improving Sparse Autoencoders Sparse Autoencoders Large Language Model Decoding Mechanistic Interpretability

Ask PDF

May 27, 2026cs.LG

ReSAE: Residualized Sparse Autoencoders for Multi-Layer Transformer Interventions

Sparse autoencoders are usually trained one layer at a time, even though transformer residual stream activations are strongly coupled across depth. This creates a practical problem for multi-layer interventions: different layerwise dictionaries can spend capacity representing the same carried-forward information, and replacing several layers at once can produce interactions that are not predicted by single-layer behavior. We introduce Residualized Sparse Autoencoders (ReSAEs), which fit an affine map between selected layers and train each later-layer SAE on the unexplained residual rather than on the full activation. Reconstructions are mapped back into the original activation space through the fitted affine chain, so ReSAEs can be evaluated with the same intervention protocols as ordinary SAEs. On Pythia-1.4B and Gemma-2-9B, residualization reduces decoder redundancy and improves sparse probing and targeted perturbation in most tested settings. Despite reconstructing less of the raw activation variance, ReSAEs recover more transformer cross entropy under multi-layer replacement. This gain is clearest under teacher-forcing and at sufficient sparsity online, indicating that ReSAEs preserve the components of the activation most relevant to the model's downstream computation. These results suggest that removing linearly predictable cross-layer structure is a useful default for multi-layer SAE interventions.

Prathyush Poduval, Calvin Yeung, Neel Desai +1

Improving Sparse Autoencoders Sparse Autoencoders Transformer Residual Streams Multi-Layer Transformers Error Correction

Ask PDF

Jun 12, 2026cs.LG

Rational Sparse Autoencoder

Sparse autoencoders (SAEs) are standard tools for mechanistic interpretability, but current SAE families are constrained by fixed encoder nonlinearities such as ReLU, JumpReLU, and TopK. This hard-codes a particular sparsity mechanism into the model and can distort the reconstruction-versus-sparsity trade-off. We introduce the Rational Sparse Autoencoder (RSAE), which replaces the fixed encoder activation with a trainable rational function. Rational activations are flexible enough to uniformly approximate the activation primitives used by existing SAE families on compact domains (for TopK, the thresholded gate obtained after a separating top-k threshold is supplied), while also providing a richer function class for adapting to the observed pre-activation geometry. We realise this idea through a two-stage pipeline: an initialisation procedure that copies the pre-trained baseline SAE weights, plugs in rational coefficients obtained by the relaxed Remez exchange on synthetic data, and calibrates the scale parameters along with the rational coefficients; followed by a fine-tuning step under the standard sparsity-regularised reconstruction objective. Empirically, on residual-stream activations of three open-weight language models and across all three baseline activation families, the RSAE strictly improves on it after the fine-tuning step, both on reconstruction-side metrics and on downstream-behaviour metrics, without sacrificing feature-level interpretability under sparse probing. These gains are consistent across host language models, across baseline activation families, and across the full range of baseline sparsity we tested, while the upgrade itself adds only a handful of scalar parameters per autoencoder and runs in minutes on a single consumer GPU.

Naiyu Yin, Yue Yu

Improving Sparse Autoencoders Autoencoder Mechanistic Interpretability Sparse Supervision

Ask PDF