Understanding the Emergence of Seemingly Useless Features in Next-Token Predictors

Mark Rofin; Jalal Naghiyev; Michael Hahn

cs.LGMar 14, 2026

Understanding the Emergence of Seemingly Useless Features in Next-Token Predictors

Authors: Mark Rofin, Jalal Naghiyev, Michael Hahn

Organizations: School of Computer and Communication Sciences, EPFL · Technical University of Munich · Zuse School ELIZA · Saarland University Campus, Saarland University

Abstract

Trained Transformers have been shown to compute abstract features that appear redundant for predicting the immediate next token. We identify which components of the gradient signal from the next-token prediction objective give rise to this phenomenon, and we propose a method to estimate the influence of those components on the emergence of specific features. After validating our approach on toy tasks, we use it to interpret the origins of the world model in OthelloGPT and syntactic features in a small language model. Finally, we apply our framework to a pretrained LLM, showing that features with extremely high or low influence on future tokens tend to be related to formal reasoning domains such as code. Overall, our work takes a step toward understanding hidden features of Transformers through the lens of their development during training.

Figures & tables

Figure 1: Left: An illustration of the information-flow decomposition for

i=2

and

k=1

into direct , pre-cached , and shared components. Direct and pre-cached paths must pass through the residual stream at position

i=2

and layer

k=1

; any other path is considered shared . The turquoise rectangle indicates

r_{\theta,i=2}^{k=1}

. Right: by decomposing the loss gradients, in Section 3.4 we partition the improvements in feature linearity at each training step.

Figure 2: Performance of the linear probes applied to the residual stream after the first Transformer block. Left: Majority, the feature “majority-so-far”. Right: Conditioned Majority, the feature “previous-token”. In both cases, the feature is NTP-useless until the 10th token. In both cases, ablating pre-caching and circuit sharing substantially hurts the probing performance.

Panel of Figure 3 — Figure 3: Left: the gap in probing performance between NTP-useful and NTP-useless board squares for OthelloGPT. Right: 95% confidence intervals for the integrated influence of each component on the representation of NTP-useful and NTP-useless features.

Panel of Figure 4 — Figure 4: Comparison of the three distinct types of features in a small language model: POS tags, dependency tags, and the positional feature. Left: probing scores for myopic and non-myopic models. Center: direct and pre-cached influence components for each feature type.

Figure 5: Development of the direct and pre-cached influence components of the positional feature in the non-myopic model during training.

Figure 6: Distribution of $Q(w)$ for the SAE features of Gemma 2 2B. The text boxes show the compressed descriptions of 5 features from the tails of the distribution, as well as 5 random features from the mode of the distribution.

Figure 7: Correlations between

\cos A(W_{LA},w,r)

and

Q(w)

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Layers	2
Heads	4
Hidden dim	128
Feedforward dim	512
Learning Rate	0.001
Batch size	256

Appendix

Table 1: Hyperparameters for the experiments with the tasks of Majority and Conditioned Majority.

Hyperparameter	Value
Layers	6
Heads	4
Hidden dim	256
Feedforward dim	1024
Learning Rate	0.001
Batch size	1024

Appendix

Table 2: Hyperparameters for the Othello experiment.

Hyperparameter	Value
Layers	4
Heads	4
Hidden dim	128
Feedforward dim	512
Learning Rate	0.001
Batch size	1024

Appendix

Table 3: Hyperparameters for the small LM experiment (TinyStories).

	non-myopic		myopic
	tied	10-untied	tied	10-untied
Majority	$1.00\pm 0.00$	$1.00\pm 0.00$	$1.00\pm 0.00$	$0.96\pm 0.05$
Conditioned Majority	$0.99\pm 0.02$	$1.00\pm 0.00$	$0.76\pm 0.01$	$0.76\pm 0.09$

Appendix

Table 4: Accuracy of a trained model for the first token in the output phase. All trained models successfully solve the tasks, except for the myopic models trained on Conditioned Majority.

Figure 8: Normalized integrated influence components for the tasks of Majority (left) and Conditioned Majority ( center: tied model, right: 10-untied model). In models solving Majority, the features in the input phase seem to be predominantly shared, but that flips at the 10th token – exactly the point where the feature is needed to predict the immediate next token, which is the output for the task. The images for Conditioned Majority may help explain the curious increase in the performance and probing accuracy of the 10-untied model compared to the tied model. We see that for the 10-untied model it is the pre-caching influence that causes the previous-token feature to be learned in the input phase, while for the tied model there is no clear pattern. This likely indicates that, instead of the robust circuit we expected, the tied model learned a less robust, possibly shortcut solution, due to the influence of shared gradients from the output phase.

Figure 9: The training loss and accuracy curves of the myopic and non-myopic OthelloGPT models. The training of a myopic model is much less stable and it does not converge to the same performance as a non-myopic model.

Figure 10: Performance of the probes trained to extract the board state from myopic and non-myopic Transformers for Othello. Myopic model shows a much lower degree of the board state representation. The scores presented in this plot are lower than the ones reported in the prior literature ( Nanda et al., 2023 ) due to the differences in evaluation methodology (we train the probes for regression and evaluate them using Pearson correlation, also unlike the prior work we include empty cells into evaluation). In addition, our model is smaller than the one used in the prior work (6 layers against 8 and a smaller hidden dimensionality), dictated by the limits of computational resources.

Figure 11: Integrated influence components at different positions for the 4 central cells in Othello. Each plot is an averaging of two positions.

Figure 12: Integrated influence components at different positions for the 12 cells surrounding the center in Othello. Each plot is an averaging of two positions.

Figure 13: The training loss curves of the myopic and non-myopic small language models. It can be seen that a myopic LM has a consistently much higher loss from very early in training.

Panel of Figure 14 — Figure 14: Direct and pre-cached influence for different feature types (as in Figure 4 ) with 95% CI estimates for the average value at each layer.

Feature category	Layer	$t$ -statistic	$p_{\text{greater}}$	$p_{\text{less}}$
POS tags	1	358640	$5.8\times 10^{-8}$	$1.0$
	2	387970	$3.1\times 10^{-16}$	$1.0$
	3	374375	$5.5\times 10^{-12}$	$1.0$
	4	452676	$3.3\times 10^{-46}$	$1.0$
Dependency tags	1	269035	$9.4\times 10^{-18}$	$1.0$
	2	297604	$2.5\times 10^{-34}$	$1.0$

Appendix

Table 5: One-sided Wilcoxon test statistics comparing direct and pre-cached influence components, with p-values reported for each feature type and layer. In all layers, the direct influence for POS and dependency tags is significantly larger than the pre-cached influence, which is reversed in the middle layers for the positional feature.

Figure 15: Distribution of

Q(w)

(pre-caching degree) for the SAE features of Gemma 2 2B.

Figure 16: Distribution of

Q(w)

(pre-caching degree) for the SAE features of Gemma 2 2B. The features classified as related to code, math, syntax, or text structure, are plotted orange; others are plotted blue. Images to the left and to the right show the tails of the distribution.

Figure 17: Distribution of

Q(w)

(pre-caching degree) for the SAE features of Gemma 2 2B. The features classified as related to code or math are plotted orange; others are plotted blue. Images to the left and to the right show the tails of the distribution.

Figure 18: Distribution of

Q(w)

(pre-caching degree) for the SAE features of Gemma 2 2B. The features classified as related to syntax or text structure, are plotted orange; others are plotted blue. Images to the left and to the right show the tails of the distribution.

Figure 19: Number of unconditional generations classified as code (left) and average number of punctuation symbols (right) obtained by steering SAE features in Gemma 2 2B. Steering features with high pre-caching degree leads to both more code and more punctuation symbols being generated.

Group	$\hat{\sigma}$ (95% CI)
#code above median	$1.479\pm 0.030$
#code below median	$1.428\pm 0.020$
#punct above median	$1.519\pm 0.025$
#punct below median	$1.351\pm 0.022$

Appendix

Table 6: Estimated scale parameter

\sigma

of the log-normal distribution fitted to

Q(w)

for different feature groups. For each group, we report the estimated

\hat{\sigma}

and its 95% confidence interval. Features are split at the median value of #code and #punct .

Feature	$Q(w)$	Description
15050	0.002	HTML or CSS code referencing scripts and stylesheets
3016	0.006	programming constructs related to annotations and metadata in code
3025	0.006	French pronouns and their conjugations in various contexts
4840	0.006	aspects related to graphical user interface elements
2127	0.007	identifiers and numerical values in a structured format
9124	0.010	elements related to data structures and operations in programming contexts

Appendix

Table 7: 10 features with the lowest

Q(w)

among the SAE features of Gemma 2 2B under study.

Feature	$Q(w)$	Description
4592	13710.8	programming-related terms and structures, particularly those associated with class and method definitions
12285	6946.1	references to web development, particularly relating to dependencies and library management
6579	3308.5	code structure and control flow statements
15090	2675.1	programming constructs and syntax elements
10042	1853.5	text related to software usage rights, permissions, and licensing terms
13045	1641.9	elements and objects that are part of a programming interface or user interface

Appendix

Table 8: 10 features with the highest

Q(w)

among the SAE features of Gemma 2 2B under study.

Feature	$Q(w)$	Description
9964	0.125	terms related to standards and standardization processes
14622	0.663	terms related to poverty and marginalized communities
5029	0.928	details related to activities undertaken or actions witnessed in various contexts
136	0.852	terms associated with legal prohibitions and restrictions
4010	0.435	concepts related to logistics, regulations, and technical specifications
13176	0.227	references to events or gatherings

Appendix

Table 9: 10 random features with

Q(w)

between 0.1 and 1.1.

Figure 20: Pearson correlations between

\cos A(W_{LA},w,r)

and

Q(w)

among the SAE features in Gemma 2.

Transformer Interpretability

Transformer interpretability studies how attention, feed-forward layers, and hidden representations support model predictions and behavior.

See all 338 papers

LLM Layers Immediately Correct Each Other
Sep 7, 2026Arjun Patrawala, Jiahai Feng, Erik Jones +1Transformer Interpretability Representation Geometry in Language Models
University of California, Berkeley
From Correlation to Cause: A Five-Stage Methodology for Feature Analysis in Transformer Language Models
May 21, 2026Caleb MunigetyTransformer Interpretability Causal Reasoning in Language Models
Independent Researcher
To See the Unseen: on the Generalization Ability of Transformers in Symbolic Reasoning
Apr 23, 2026Nevena Lazić, Liam Fowl, András György +1Logical Reasoning Decoder-Only Language Models

Sep 7, 2026J/K move · Enter open · S save

LLM Layers Immediately Correct Each Other

Arjun Patrawala, Jiahai Feng, Erik Jones, Jacob Steinhardt

University of California, Berkeley

Transformer Interpretability Representation Geometry in Language Models

Open paper AskPDF arXiv

Recent methods in language model interpretability employ techniques such as sparse autoencoders to decompose residual stream contributions into linear, semantically meaningful features. Such methods are commonly interpreted as identifying features that persist in the residual stream and that subsequent layers build upon. We challenge this view by identifying the Transformer Layer Correction Mechanism (TLCM), wherein adjacent transformer layers systematically counteract portions of each other's contributions. TLCM appears in 5 out of 7 major open-source model families and activates across nearly all tokens in diverse texts. We show that TLCM emerges during pretraining, operates most strongly on contextually dependent tokens, and adaptively calibrates its correction strength based on the preceding layer's output. Using the layer Jacobian, we further show that TLCM selectively corrects specific subspaces while reinforcing others, which we interpret through a ``propose-and-reject'' framework in which layers propose candidate features and subsequent layers selectively remove inappropriate ones. This dynamic suggests that the residual stream at any layer contains transient proposals alongside persistent features, helping explain why SAE feature descriptions often have low specificity, why effective model steering requires extreme feature amplification, and why transcoders hold a theoretical advantage over SAEs.

Abstract

Transformer Interpretability

Explore similar work

LLM Layers Immediately Correct Each Other

From Correlation to Cause: A Five-Stage Methodology for Feature Analysis in Transformer Language Models

To See the Unseen: on the Generalization Ability of Transformers in Symbolic Reasoning