A central idea in mechanistic interpretability is that neural networks represent more features than they have dimensions, arranging them in superposition to form an over-complete basis. This framing has been influential, motivating dictionary learning approaches such as sparse autoencoders. However, superposition has mostly been studied in idealized settings where features are sparse and uncorrelated. In these settings, superposition is typically understood as introducing interference that must be minimized geometrically and filtered out by non-linearities such as ReLUs, yielding local structures like regular polytopes. We show that this account is incomplete for realistic data by introducing Bag-of-Words Superposition (BOWS), a controlled setting to encode binary bag-of-words representations of internet text in superposition. Using BOWS, we find that when features are correlated, interference can be constructive rather than just noise to be filtered out. This is achieved by arranging features according to their co-activation patterns, making interference between active features constructive, while still using ReLUs to avoid false positives. We show that this kind of arrangement is more prevalent in models trained with weight decay and naturally gives rise to semantic clusters and cyclical structures which have been observed in real language models yet were not explained by the standard picture of superposition. Code for this paper can be found at https://github.com/LucasPrietoAl/correlations-feature-geometry.
Figures & tables
Figure 1: BOWS, our new framework to study superposition in realistic data (left) extends our current understanding of superposition (middle) by showing that interference can be constructive, allowing words like ‘December’ to contribute to the reconstruction of correlated words like ‘Christmas’ giving rise to a circular arrangement for the months of the year (right).
Figure 2: Autoencoding synthetic correlated features shows two ways of handling interference. Weight inner products ( W⊤W ) at convergence for AEs encoding d=12 features with cyclic covariance, varying latent size m . Top (Linear AE): captures the top- m principal components projecting all 12 features on the circular structure induced by the data covariance. Bottom (ReLU AE): Matches linear AE for small m , but forms antipodal pairs for m∈{6,...,10} using the ReLU to filter interference.
Figure 3: Linear superposition appears in ReLU AEs which have small latent sizes (a) or are trained with weight decay (d), giving rise to semantic clusters. UMAP projections of word embeddings from AEs of different latent dimensions ( m ) and weight decay values ( wd ). Points are colored by semantic category (e).
Figure 4: Circular representation of months arises from data covariance via PCA. (a) Empirical correlation matrix of month words in the WikiText-103 BOWS dataset, showing cyclic correlations. (b) PCA applied directly to the 12 month dimensions of the BOWS data vectors, projected onto the top 2 PCs, reveals a circle. (c) PCA applied to the 12 learned encoder features ( W columns) for months from a ReLU AE trained on WikiText-BOWS ( V=10k , m=1000 ), projected onto their top 2 PCs, also recovers the circular structure. Seasonal words such as ‘Christmas’ and ‘summer’ align with the months with which they co-occur, allowing ‘December’ to contribute to the reconstruction of ‘Christmas’ while interference on ‘Christmas’ cancels out if all months are present (d) .
Figure 5: Constructive interference and interference filtering coexist in realistic data. (Left) Terms related to ‘Beatles’ achieve high validation R2 despite poor one-hot reconstruction, indicating that they benefit from contextual interference. (Middle) For 81% of validation samples containing ‘Beatles’, interference improves reconstruction relative to the one-hot case. (Right) In supportive contexts, correlated words contribute positive pre-activation to ‘Beatles’; when these contexts occur without the target word, the ReLU and negative bias suppress false positives.
Figure 6: Different feature structures disappear at different latent sizes. As the latent size of the AE increases, the structure in the weights appears and disappears at different times for different groups of features. We look at 2D PCA plots of the months and roman numeral embeddings at two key latent sizes. At Snapshot 1 (green line), both groups of features appear in order with some structure, while at Snapshot 2 (red line), the representations of the months have already become orthogonal while the roman numerals are still represented in an ordered structure.
Condition
MAP
Key-freq
Loss ↓
Acc. (%) ↑
Loss ↓
Acc. (%) ↑
Baseline
0.0536
97.94
0.0001
100.00
VC+
0.2879
93.16
0.1649
93.99
VC−
6.3943
22.43
11.7464
3.11
Table 1: Value-coding (VC) ablations on two datasets. VC+ keeps the VC subspace and zeros its orthogonal complement; VC− ablates VC coordinates.
Figure 7: In the embeddings of a model performing modular addition, the circular structure is isolated by projecting onto directions corresponding to sine and cosine values ( left ). In the case of the map directions dataset, two linear probes reconstruct the city coordinates, reconstructing their positions on the map for both train and validation samples. ( right ).
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Tow-layer transformer trained on Wikitext-103 for the multi-task token-recovery task exhibit semantic clusters and ordered circular representations for the months of the year. This replicates the main results in Figure 4 and Figure 3 on a different dataset. Colors correspond to different semantic categories ( Figure 3 d).
Figure 9: UMAP embeddings of features from AEs trained with context size of 20 records with wd=0 (top) and wd=4 (bottom) across different latent sizes. The plot shows that semantic structure remains for a larger fraction of context sizes when using weight decay. Colors correspond to different semantic categories ( Figure 3 d).
Figure 10: Words like ‘December’ and ‘Day’ contribute to the reconstruction of ‘Christmas’. ( left ) ‘Christmas’ is better reconstructed with than without interference. ( middle ) For most samples (81%) where ‘Christmas’ appears in the validation set, interference is beneficial, allowing for reconstructions closer to 1. ( right ) When ‘Christmas’ appears in the right context it has a positive reconstruction thanks to the interference from related terms. However, if the related context is present but ‘Christmas’ is not, the model uses the ReLU and a negative bias to avoid false positives at the cost of having smaller reconstructions for ‘Christmas’ when it appears in an unusual context.
Figure 11: Atmospheric terms like ‘barometric’ and ‘inHg’ contribute to the reconstruction of ‘mbar’. ( left ) ‘mbar’ is better reconstructed with than without interference. ( middle ) For every sample where ‘mbar’ appears in the validation set, interference is beneficial, allowing for reconstructions closer to 1. ( right ) When ‘mbar’ appears in the right context it has a positive reconstruction thanks to the interference from related terms. However, if the related context is present but ‘mbar’ is not, the model uses the ReLU and a negative bias to minimize false positives at the cost of not being able to reconstruct ‘mbar’ when it appears in an unusual context.
Figure 12: Autoencoders trained on OpenWebText exhibit semantic clusters and ordered circular representations for the days of the week. This replicates the main results in Figure 4 and Figure 3 on a different dataset. Colors correspond to different semantic categories ( Figure 3 d).
Figure 13: Reconstructions at different latent-vector sizes. Top: “Months” dataset; middle: “Roman numerals”; bottom: corresponding latent size m .
Figure 14: 3-D PCA of the embeddings for the words and the days in a WikiText BOWS setup.
Figure 15: Extension of Figure 1 to include all values of m between 2 and 12, as well as a comparison with the weight patterns for AEs trained on data drawn from data with different correlation strucutres: figure-of-eight and spherical.
Figure 16: Zooming into the cluster for science features in the UMAP plot with a latent size of 200, we observe sub-clusters within it. Medical features are in the top left while astronomy features are in the lower left and chemistry features are in the lower center.
Figure 17: UMAP embeddings at different latent-vector sizes including only features with R2<0.3 (top) and R2>0.3 (bottom). Semantic clusters at different latent sizes are still observed in both, although this effect is combined with an increase in the number of features above the threshold in the lower one.
Figure 18: We show the latent representations of the top 4000 most frequent words using UMAP (left) t-SNE (middle) and PaCMAP (right) to highlight that these semantic clustering results are not dependent on the choice of dimensionality reduction technique.
Figure 19: Larger context windows strengthen shared structure and make constructive interference more effective. (Left) Increasing the amount of text encoded by each sample ( C ) increases the fraction of ReLU-AE reconstruction variance that can be recovered by a linear probe from the latent space. (Middle) Increasing C decreases the number of principal components needed to explain 95% of the variance while increasing the average number of active features per sample. (Right) Word frequencies follow a power-law distribution ( Zipf, 1949 ) , helping explain why some feature groups, such as months, receive more capacity and retain structured geometry longer than rarer groups such as Roman numerals.
Superposition refers to neural networks representing more features than they have dimensions. It offers a possible explanation for polysemantic neurons and motivates methods for recovering interpretable features from neural activations. Theoretical models typically start with a given set of input features and assumptions about how their values vary across inputs, then study how a network encodes those values in a lower-dimensional hidden representation. Empirical work, by contrast, seeks to identify the features encoded in trained networks and determine their role in computation. In this survey, we review the geometry, learning, and computation of superposed representations, explaining how feature statistics and decoder choice affect the conclusions. To connect these theoretical accounts with evidence from trained networks, we compare practical methods for recovering and analyzing features and examine what their evaluations establish. Since accurate activation reconstruction alone does not establish feature identity or causal use, we discuss the methods' documented failures and applications in light of the evidence available for these different claims. Finally, we assess previously stated open problems and identify remaining theoretical and empirical questions about superposition in trained networks. We hope our work can pave the way for a deeper understanding of superposition and more reliable methods for interpreting neural networks.
Dai Shi, Xiaoyu Li, Andi Han +1
University of Cambridge · University of New South Wales · University of Sydney
Comparing internal representations is a central goal in neuroscience and machine learning, but standard linear alignment metrics (Representational Similarity Analysis, Centered Kernel Alignment, and linear regression) are frequently applied to neural activity coordinates rather than on the underlying features. We show this matters when neural systems operate in superposition, encoding more features than they have neurons via linear compression. Closed-form derivations prove that these metrics depend on the Gram matrices of each system's projection, not on the latent features themselves: alignment thus combines what a system represents with how it is encoded. For those interested in what features two systems share, this is a problem: Two networks can have identical feature content yet appear more dissimilar than networks exhibiting partial feature overlap. This apparent misalignment need not reflect lost information as compressed sensing guarantees sparse features remain recoverable from the compressed activity. We confirm this by training supervised TopK sparse autoencoders that realize solvable compressed sensing by construction, finding alignment on recovered latents restored even when raw-activation alignment remains deflated. We extend the result to unsupervised SAEs trained without ground-truth latents, and to pretrained vision and language model SAEs, where SAE-latent alignment exceeds raw-activation alignment, consistent with superposition in real systems.
Sunny Liu, Habon Issa, André Longon +4
Cold Spring Harbor Laboratory · UC San Diego · Anthropic +2
Neural networks can leverage feature superposition to encode more concepts than dimensions, but cross-feature interference constrains the linear accessibility of simultaneously active features. By framing linear accessibility as a compressed sensing problem, we derive high-probability bounds for fixed supports under subgaussian noise, proving the sufficient dimension scales linearly (d=Oε(klogm)) rather than prior worst-case quadratic limits. We characterize the asymmetry between active and inactive interference and the trade-off between interference and observation-noise budgets. We then validate these bounds across system parameters through Gaussian-tail approximations. We also introduce IHT-SAE, which uses learned iterative refinement to improve feature recovery beyond the limits of linear availability. These results quantify the geometric constraints of the linear representation hypothesis, providing a framework for evaluating sparse autoencoders, compositional generalization, and neural interpretability.
Enrico Vompa
Applied Artificial Intelligence Group Tallinn University of Technology, Estonia