Understanding the Emergence of Seemingly Useless Features in Next-Token Predictors
Authors: Mark Rofin, Jalal Naghiyev, Michael Hahn
Organizations: School of Computer and Communication Sciences, EPFL · Technical University of Munich · Zuse School ELIZA · Saarland University Campus, Saarland University
Trained Transformers have been shown to compute abstract features that appear redundant for predicting the immediate next token. We identify which components of the gradient signal from the next-token prediction objective give rise to this phenomenon, and we propose a method to estimate the influence of those components on the emergence of specific features. After validating our approach on toy tasks, we use it to interpret the origins of the world model in OthelloGPT and syntactic features in a small language model. Finally, we apply our framework to a pretrained LLM, showing that features with extremely high or low influence on future tokens tend to be related to formal reasoning domains such as code. Overall, our work takes a step toward understanding hidden features of Transformers through the lens of their development during training.
Figures & tables
Figure 1: Left: An illustration of the information-flow decomposition for i=2 and k=1 into direct , pre-cached , and shared components. Direct and pre-cached paths must pass through the residual stream at position i=2 and layer k=1 ; any other path is considered shared . The turquoise rectangle indicates rθ,i=2k=1 . Right: by decomposing the loss gradients, in Section 3.4 we partition the improvements in feature linearity at each training step.
Figure 2: Performance of the linear probes applied to the residual stream after the first Transformer block. Left: Majority, the feature “majority-so-far”. Right: Conditioned Majority, the feature “previous-token”. In both cases, the feature is NTP-useless until the 10th token. In both cases, ablating pre-caching and circuit sharing substantially hurts the probing performance.
Figure 3: Left: the gap in probing performance between NTP-useful and NTP-useless board squares for OthelloGPT. Right: 95% confidence intervals for the integrated influence of each component on the representation of NTP-useful and NTP-useless features.
Figure 4: Comparison of the three distinct types of features in a small language model: POS tags, dependency tags, and the positional feature. Left: probing scores for myopic and non-myopic models. Center: direct and pre-cached influence components for each feature type.
Figure 5: Development of the direct and pre-cached influence components of the positional feature in the non-myopic model during training.
Figure 6: Distribution of Q(w) for the SAE features of Gemma 2 2B. The text boxes show the compressed descriptions of 5 features from the tails of the distribution, as well as 5 random features from the mode of the distribution.
Figure 7: Correlations between cosA(WLA,w,r) and Q(w) .
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Layers
2
Heads
4
Hidden dim
128
Feedforward dim
512
Learning Rate
0.001
Batch size
256
Appendix
Table 1: Hyperparameters for the experiments with the tasks of Majority and Conditioned Majority.
Hyperparameter
Value
Layers
6
Heads
4
Hidden dim
256
Feedforward dim
1024
Learning Rate
0.001
Batch size
1024
Appendix
Table 2: Hyperparameters for the Othello experiment.
Hyperparameter
Value
Layers
4
Heads
4
Hidden dim
128
Feedforward dim
512
Learning Rate
0.001
Batch size
1024
Appendix
Table 3: Hyperparameters for the small LM experiment (TinyStories).
non-myopic
myopic
tied
10-untied
tied
10-untied
Majority
1.00±0.00
1.00±0.00
1.00±0.00
0.96±0.05
Conditioned Majority
0.99±0.02
1.00±0.00
0.76±0.01
0.76±0.09
Appendix
Table 4: Accuracy of a trained model for the first token in the output phase. All trained models successfully solve the tasks, except for the myopic models trained on Conditioned Majority.
Figure 8: Normalized integrated influence components for the tasks of Majority (left) and Conditioned Majority ( center: tied model, right: 10-untied model). In models solving Majority, the features in the input phase seem to be predominantly shared, but that flips at the 10th token – exactly the point where the feature is needed to predict the immediate next token, which is the output for the task. The images for Conditioned Majority may help explain the curious increase in the performance and probing accuracy of the 10-untied model compared to the tied model. We see that for the 10-untied model it is the pre-caching influence that causes the previous-token feature to be learned in the input phase, while for the tied model there is no clear pattern. This likely indicates that, instead of the robust circuit we expected, the tied model learned a less robust, possibly shortcut solution, due to the influence of shared gradients from the output phase.
Figure 9: The training loss and accuracy curves of the myopic and non-myopic OthelloGPT models. The training of a myopic model is much less stable and it does not converge to the same performance as a non-myopic model.
Figure 10: Performance of the probes trained to extract the board state from myopic and non-myopic Transformers for Othello. Myopic model shows a much lower degree of the board state representation. The scores presented in this plot are lower than the ones reported in the prior literature ( Nanda et al., 2023 ) due to the differences in evaluation methodology (we train the probes for regression and evaluate them using Pearson correlation, also unlike the prior work we include empty cells into evaluation). In addition, our model is smaller than the one used in the prior work (6 layers against 8 and a smaller hidden dimensionality), dictated by the limits of computational resources.
Figure 11: Integrated influence components at different positions for the 4 central cells in Othello. Each plot is an averaging of two positions.
Figure 12: Integrated influence components at different positions for the 12 cells surrounding the center in Othello. Each plot is an averaging of two positions.
Figure 13: The training loss curves of the myopic and non-myopic small language models. It can be seen that a myopic LM has a consistently much higher loss from very early in training.
Figure 14: Direct and pre-cached influence for different feature types (as in Figure 4 ) with 95% CI estimates for the average value at each layer.
Feature category
Layer
t -statistic
pgreater
pless
POS tags
1
358640
5.8×10−8
1.0
2
387970
3.1×10−16
1.0
3
374375
5.5×10−12
1.0
4
452676
3.3×10−46
1.0
Dependency tags
1
269035
9.4×10−18
1.0
2
297604
2.5×10−34
1.0
Appendix
Table 5: One-sided Wilcoxon test statistics comparing direct and pre-cached influence components, with p-values reported for each feature type and layer. In all layers, the direct influence for POS and dependency tags is significantly larger than the pre-cached influence, which is reversed in the middle layers for the positional feature.
Figure 15: Distribution of Q(w) (pre-caching degree) for the SAE features of Gemma 2 2B.
Figure 16: Distribution of Q(w) (pre-caching degree) for the SAE features of Gemma 2 2B. The features classified as related to code, math, syntax, or text structure, are plotted orange; others are plotted blue. Images to the left and to the right show the tails of the distribution.
Figure 17: Distribution of Q(w) (pre-caching degree) for the SAE features of Gemma 2 2B. The features classified as related to code or math are plotted orange; others are plotted blue. Images to the left and to the right show the tails of the distribution.
Figure 18: Distribution of Q(w) (pre-caching degree) for the SAE features of Gemma 2 2B. The features classified as related to syntax or text structure, are plotted orange; others are plotted blue. Images to the left and to the right show the tails of the distribution.
Figure 19: Number of unconditional generations classified as code (left) and average number of punctuation symbols (right) obtained by steering SAE features in Gemma 2 2B. Steering features with high pre-caching degree leads to both more code and more punctuation symbols being generated.
Group
σ^ (95% CI)
#code above median
1.479±0.030
#code below median
1.428±0.020
#punct above median
1.519±0.025
#punct below median
1.351±0.022
Appendix
Table 6: Estimated scale parameter σ of the log-normal distribution fitted to Q(w) for different feature groups. For each group, we report the estimated σ^ and its 95% confidence interval. Features are split at the median value of #code and #punct .
Feature
Q(w)
Description
15050
0.002
HTML or CSS code referencing scripts and stylesheets
3016
0.006
programming constructs related to annotations and metadata in code
3025
0.006
French pronouns and their conjugations in various contexts
4840
0.006
aspects related to graphical user interface elements
2127
0.007
identifiers and numerical values in a structured format
9124
0.010
elements related to data structures and operations in programming contexts
Appendix
Table 7: 10 features with the lowest Q(w) among the SAE features of Gemma 2 2B under study.
Feature
Q(w)
Description
4592
13710.8
programming-related terms and structures, particularly those associated with class and method definitions
12285
6946.1
references to web development, particularly relating to dependencies and library management
6579
3308.5
code structure and control flow statements
15090
2675.1
programming constructs and syntax elements
10042
1853.5
text related to software usage rights, permissions, and licensing terms
13045
1641.9
elements and objects that are part of a programming interface or user interface
Appendix
Table 8: 10 features with the highest Q(w) among the SAE features of Gemma 2 2B under study.
Feature
Q(w)
Description
9964
0.125
terms related to standards and standardization processes
14622
0.663
terms related to poverty and marginalized communities
5029
0.928
details related to activities undertaken or actions witnessed in various contexts
136
0.852
terms associated with legal prohibitions and restrictions
4010
0.435
concepts related to logistics, regulations, and technical specifications
13176
0.227
references to events or gatherings
Appendix
Table 9: 10 random features with Q(w) between 0.1 and 1.1.
Figure 20: Pearson correlations between cosA(WLA,w,r) and Q(w) among the SAE features in Gemma 2.
Recent methods in language model interpretability employ techniques such as sparse autoencoders to decompose residual stream contributions into linear, semantically meaningful features. Such methods are commonly interpreted as identifying features that persist in the residual stream and that subsequent layers build upon. We challenge this view by identifying the Transformer Layer Correction Mechanism (TLCM), wherein adjacent transformer layers systematically counteract portions of each other's contributions. TLCM appears in 5 out of 7 major open-source model families and activates across nearly all tokens in diverse texts. We show that TLCM emerges during pretraining, operates most strongly on contextually dependent tokens, and adaptively calibrates its correction strength based on the preceding layer's output. Using the layer Jacobian, we further show that TLCM selectively corrects specific subspaces while reinforcing others, which we interpret through a ``propose-and-reject'' framework in which layers propose candidate features and subsequent layers selectively remove inappropriate ones. This dynamic suggests that the residual stream at any layer contains transient proposals alongside persistent features, helping explain why SAE feature descriptions often have low specificity, why effective model steering requires extreme feature amplification, and why transcoders hold a theoretical advantage over SAEs.
We propose a five-stage methodology for causal feature analysis in transformer language models (probe design, feature extraction, causal validation, robustness testing, and deployment integration) and demonstrate it end-to-end on GPT-2 small performing the Indirect Object Identification (IOI) task. Activation patching recovers the canonical IOI circuit (layer-9 head 9 alone gives recovery +1.02). A sparse autoencoder recovers per-name selective features with effect sizes of 30 to 50 activation units. Causal validation finds these features specifically but only partially causal: ablating fifteen of them leaves the model accurate on 98% of prompts. Two NLA-inspired evaluations strengthen this picture: the fifteen selective features explain only 31% of activation variance versus the SAE's 99.7%, and selectivity ratio anticorrelates with causal force (r = -0.56). Robustness testing under three distribution shifts finds that the circuit transfers cleanly but feature ablation effects degrade substantially, exposing a gap between detection robustness and causal robustness. A cost-based deployment evaluation (assumed 50/FN,0.42/FP, 2% error rate) finds an optimal monitor configuration yielding 8.96per1000queriesagainsta1000 baseline, a 99.1% saving. Optimal composition strategy varies with cost ratio and base rate. The conjunction of stages produces findings no single stage would.
We investigate the ability of decoder-only transformer models to perform abstract symbolic reasoning; specifically solving propositional logic reasoning problems given in-context. Previous work demonstrated that models fail to generalize to problems involving variable names that were not observed during training, and it was shown that one reason behind this is the difficulty of copying (or generating) unseen tokens. We show both theoretically and empirically that a particular representational collapse also has a crucial role: the unembeddings (last-layer weights) of unseen tokens collapse to nearly the same vector during training. The collapse makes distinguishing multiple unseen variables difficult for the model (especially when the embedding and unembedding parameters are shared), and provides a mechanistic explanation for the effectiveness of existing heuristic interventions like "active forgetting", which periodically reset the token (un)embeddings. Based on these observations, we devise a combination of techniques, involving a small architecture change facilitating copying, data diversity, and freezing or resetting (un)embeddings, that achieves generalization to unseen tokens. We support our claims with extensive controlled experiments on propositional logic reasoning problems. Beyond synthetic experiments, we also observe evidence of (un)embedding collapse in the open-weight models in the Gemma 3 family, which includes 99 unused tokens reserved for downstream use. Empirically we find that the correlated embeddings of these tokens are a poor initialization for finetuning applications.