cs.AIAug 21, 2026
SaveDeep Learning Models Also Recall Features
Organizations: Ecole Polytechnique Fédérale de Lausanne (EPFL) · Idiap Research Institute
Abstract
Recent work in mechanistic interpretability has studied how large language models recall facts stored in their weights. This paper argues that factual recall points to something broader: a general kind of operation in deep learning models, which I call feature recall. The core observation is that a linear projection can be read as retrieving stored information scaled by input activations. I define feature recall, show it applies across architectures, and contrast it with the established paradigm of feature combination. I also consider how cases of feature recall might be mechanistically identified. The account gives philosophers a new conceptual tool for understanding deep learning, and points to empirical directions for mechanistic interpretability research.
Figures & tables
Figure 1: An idealized case of feature-activated factual recall. Panel (a) shows the sequential processing of “Michael Jordan plays,” word-token by word-token as they are processed along the residual stream. To complete this sentence (by predicting the “basketball” token), an attention head first routes the Michael feature from the first residual stream to the second. MLP layers then perform two distinct operations. First, they combine the Michael and Jordan features into a unified Michael Jordan feature. Second, they retrieve associated facts stored via the network’s weights—such as plays basketball or Chicago Bulls player —based on this combined feature. Another attention head then routes this retrieved information to the last residual stream, providing the context needed for accurate next-token prediction. Panel (b) details how both feature combination and factual recall could occur within a single MLP’s linear transformations in the residual stream of the “Jordan” token. The upward projection (1) recognizes the pattern of co-occurring Michael and Jordan features, leading to a strong activation of the Michael Jordan feature. The activation function (2) suppresses any spurious activations. Through the downward projection (3) this feature activates the retrieval of certain stored weights, which when added into the output cause the activation of features such as plays basketball or Chicago Bulls player . The weights connecting the Michael Jordan feature to output features directly encode factual associations, ensuring that whenever this feature activates, relevant factual information is automatically retrieved and added to the output representation. 5 5 5 Some operations such as layer normalization are left out from this idealized case. Furthermore, the figure assumes that each feature corresponds to a single neuron, a property called neuron alignment. I adopt this simplification to make the figure legible and the core mechanism accessible to readers less familiar with the field. Readers familiar with superposition (the finding that models represent far more features than they have neurons, thereby distributing features across neuron directions; Elhage et al., 2022 ) may worry that this undermines the mechanism. Thankfully, it doesn’t. A feature direction that does not align with a neuron direction still has a well-defined projection through the weight matrix. The matrix maps the input direction to an output vector that can have strong components along multiple output feature directions. The weights then still encode stored associations; the only difference is that these associations hold between directions in activation space rather than between neurons.
Figure 2: Feature combination in a CNN: an idealized case of a circle-recognition circuit. The example traces how a subset of weights of a CNN detects a circle in a 9 9 pixel image region. The process works in two stages. First, the network applies two different 3 3 pattern detectors to non-overlapping regions of the input image. Each kernel scans four 3 3 regions with stride 3, generating two separate 2 2 activation maps. The red box illustrates how this works for the top left 3 3 region and the first kernel: when the top-left kernel perfectly matches its corresponding 3 3 input region, matrix multiplication yields a strong response of 3. The top-right 3 3 region shows only partial overlap with the first kernel at the center pixel, producing a weaker activation of 1. Applying both kernels, creates two 2 2 feature maps. These are then concatenated into a single 2 2 2 tensor (though the stacking isn’t shown in the figure). Second, a higher-level 2 2 2 kernel is applied, arranged exactly like our current concatenation of the two feature maps. It will therefore get its strongest activation precisely when the input 9 9 contains a circle. So the network detects the circle hierarchically: from 9 9 pixels, to two 2 2 edge maps, to one circle feature. 8 8 8 Some key operations of CNNs such as pooling, nonlinear activation functions, and normalization are left out from this idealized case. As with Figure 5 , features are shown as neuron-aligned for simplicity.
Figure 3: A linear projection, or matrix multiplication, can be seen both in terms of (a) feature combination or (b) feature recall. In case (a), each output element is obtained by multiplying the input vector with a column of the weight matrix. This can be seen as feature combination: each column defines a pattern of how input features should be combined to obtain one specific output feature. In case (b), each input feature multiplies out one row of the weight matrix and the resulting vectors are summed (note that this amounts to the exact same operation because matrix multiplication can be decomposed as either a dot product with columns or a weighted sum of rows). Each row can then be seen as a stored set of potential feature activations that gets scaled by the input features—the operation is thus seen in terms of feature recall.
More in this topic
Neural Network Interpretability
Neural network interpretability studies how models represent information and make predictions, and how to explain, test, or control their internal computations.
See all 330 papers