cs.LGMar 10, 2026
SaveFrom Data Statistics to Feature Geometry: How Correlations Shape Superposition
Abstract
A central idea in mechanistic interpretability is that neural networks represent more features than they have dimensions, arranging them in superposition to form an over-complete basis. This framing has been influential, motivating dictionary learning approaches such as sparse autoencoders. However, superposition has mostly been studied in idealized settings where features are sparse and uncorrelated. In these settings, superposition is typically understood as introducing interference that must be minimized geometrically and filtered out by non-linearities such as ReLUs, yielding local structures like regular polytopes. We show that this account is incomplete for realistic data by introducing Bag-of-Words Superposition (BOWS), a controlled setting to encode binary bag-of-words representations of internet text in superposition. Using BOWS, we find that when features are correlated, interference can be constructive rather than just noise to be filtered out. This is achieved by arranging features according to their co-activation patterns, making interference between active features constructive, while still using ReLUs to avoid false positives. We show that this kind of arrangement is more prevalent in models trained with weight decay and naturally gives rise to semantic clusters and cyclical structures which have been observed in real language models yet were not explained by the standard picture of superposition. Code for this paper can be found at https://github.com/LucasPrietoAl/correlations-feature-geometry.
Figures & tables
Figure 1: BOWS, our new framework to study superposition in realistic data (left) extends our current understanding of superposition (middle) by showing that interference can be constructive, allowing words like ‘December’ to contribute to the reconstruction of correlated words like ‘Christmas’ giving rise to a circular arrangement for the months of the year (right).
Figure 2: Autoencoding synthetic correlated features shows two ways of handling interference. Weight inner products ( ) at convergence for AEs encoding features with cyclic covariance, varying latent size . Top (Linear AE): captures the top- principal components projecting all 12 features on the circular structure induced by the data covariance. Bottom (ReLU AE): Matches linear AE for small , but forms antipodal pairs for using the to filter interference.
Figure 3: Linear superposition appears in ReLU AEs which have small latent sizes (a) or are trained with weight decay (d), giving rise to semantic clusters. UMAP projections of word embeddings from AEs of different latent dimensions ( ) and weight decay values ( ). Points are colored by semantic category (e).
Figure 4: Circular representation of months arises from data covariance via PCA. (a) Empirical correlation matrix of month words in the WikiText-103 BOWS dataset, showing cyclic correlations. (b) PCA applied directly to the 12 month dimensions of the BOWS data vectors, projected onto the top 2 PCs, reveals a circle. (c) PCA applied to the 12 learned encoder features ( columns) for months from a ReLU AE trained on WikiText-BOWS ( , ), projected onto their top 2 PCs, also recovers the circular structure. Seasonal words such as ‘Christmas’ and ‘summer’ align with the months with which they co-occur, allowing ‘December’ to contribute to the reconstruction of ‘Christmas’ while interference on ‘Christmas’ cancels out if all months are present (d) .
Figure 5: Constructive interference and interference filtering coexist in realistic data. (Left) Terms related to ‘Beatles’ achieve high validation despite poor one-hot reconstruction, indicating that they benefit from contextual interference. (Middle) For 81% of validation samples containing ‘Beatles’, interference improves reconstruction relative to the one-hot case. (Right) In supportive contexts, correlated words contribute positive pre-activation to ‘Beatles’; when these contexts occur without the target word, the ReLU and negative bias suppress false positives.
Figure 6: Different feature structures disappear at different latent sizes. As the latent size of the AE increases, the structure in the weights appears and disappears at different times for different groups of features. We look at 2D PCA plots of the months and roman numeral embeddings at two key latent sizes. At Snapshot 1 (green line), both groups of features appear in order with some structure, while at Snapshot 2 (red line), the representations of the months have already become orthogonal while the roman numerals are still represented in an ordered structure.
| Condition | MAP | Key-freq | ||
|---|---|---|---|---|
| Loss | Acc. (%) | Loss | Acc. (%) | |
| Baseline | 0.0536 | 97.94 | 0.0001 | 100.00 |
| 0.2879 | 93.16 | 0.1649 | 93.99 | |
| 6.3943 | 22.43 | 11.7464 | 3.11 | |
Table 1: Value-coding (VC) ablations on two datasets. keeps the VC subspace and zeros its orthogonal complement; ablates VC coordinates.
Figure 7: In the embeddings of a model performing modular addition, the circular structure is isolated by projecting onto directions corresponding to sine and cosine values ( left ). In the case of the map directions dataset, two linear probes reconstruct the city coordinates, reconstructing their positions on the map for both train and validation samples. ( right ).
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Tow-layer transformer trained on Wikitext-103 for the multi-task token-recovery task exhibit semantic clusters and ordered circular representations for the months of the year. This replicates the main results in Figure 4 and Figure 3 on a different dataset. Colors correspond to different semantic categories ( Figure 3 d).
Figure 9: UMAP embeddings of features from AEs trained with context size of 20 records with wd=0 (top) and wd=4 (bottom) across different latent sizes. The plot shows that semantic structure remains for a larger fraction of context sizes when using weight decay. Colors correspond to different semantic categories ( Figure 3 d).
Figure 10: Words like ‘December’ and ‘Day’ contribute to the reconstruction of ‘Christmas’. ( left ) ‘Christmas’ is better reconstructed with than without interference. ( middle ) For most samples (81%) where ‘Christmas’ appears in the validation set, interference is beneficial, allowing for reconstructions closer to 1. ( right ) When ‘Christmas’ appears in the right context it has a positive reconstruction thanks to the interference from related terms. However, if the related context is present but ‘Christmas’ is not, the model uses the ReLU and a negative bias to avoid false positives at the cost of having smaller reconstructions for ‘Christmas’ when it appears in an unusual context.
Figure 11: Atmospheric terms like ‘barometric’ and ‘inHg’ contribute to the reconstruction of ‘mbar’. ( left ) ‘mbar’ is better reconstructed with than without interference. ( middle ) For every sample where ‘mbar’ appears in the validation set, interference is beneficial, allowing for reconstructions closer to 1. ( right ) When ‘mbar’ appears in the right context it has a positive reconstruction thanks to the interference from related terms. However, if the related context is present but ‘mbar’ is not, the model uses the ReLU and a negative bias to minimize false positives at the cost of not being able to reconstruct ‘mbar’ when it appears in an unusual context.
Figure 12: Autoencoders trained on OpenWebText exhibit semantic clusters and ordered circular representations for the days of the week. This replicates the main results in Figure 4 and Figure 3 on a different dataset. Colors correspond to different semantic categories ( Figure 3 d).
Figure 13: Reconstructions at different latent-vector sizes. Top: “Months” dataset; middle: “Roman numerals”; bottom: corresponding latent size .
Figure 14: 3-D PCA of the embeddings for the words and the days in a WikiText BOWS setup.
Figure 15: Extension of Figure 1 to include all values of m between 2 and 12, as well as a comparison with the weight patterns for AEs trained on data drawn from data with different correlation strucutres: figure-of-eight and spherical.
Figure 16: Zooming into the cluster for science features in the UMAP plot with a latent size of 200, we observe sub-clusters within it. Medical features are in the top left while astronomy features are in the lower left and chemistry features are in the lower center.
Figure 17: UMAP embeddings at different latent-vector sizes including only features with (top) and (bottom). Semantic clusters at different latent sizes are still observed in both, although this effect is combined with an increase in the number of features above the threshold in the lower one.
Figure 18: We show the latent representations of the top 4000 most frequent words using UMAP (left) t-SNE (middle) and PaCMAP (right) to highlight that these semantic clustering results are not dependent on the choice of dimensionality reduction technique.
Figure 19: Larger context windows strengthen shared structure and make constructive interference more effective. (Left) Increasing the amount of text encoded by each sample ( ) increases the fraction of ReLU-AE reconstruction variance that can be recovered by a linear probe from the latent space. (Middle) Increasing decreases the number of principal components needed to explain 95% of the variance while increasing the average number of active features per sample. (Right) Word frequencies follow a power-law distribution ( Zipf, 1949 ) , helping explain why some feature groups, such as months, receive more capacity and retain structured geometry longer than rarer groups such as Roman numerals.