cs.CLSep 26, 2026

Gradients for Interventions and Activations for Detection: Targeted Feature Learning in Language Models

Authors: Jonathan Drechsel, Steffen Herbold

Organizations: Faculty of Computer Science and Mathematics University of Passau Passau, Germany

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Model-internal features can be studied through both their ability to identify a specified concept and their causal effect when manipulated, e.g., through steering or weight editing. A prominent approach to feature learning is Sparse Autoencoders (SAEs), which learn broad feature dictionaries whose relation to particular concepts is typically identified post hoc. However, many interpretability questions are instead hypothesis-driven and concern a concept specified in advance. We study this setting as targeted feature learning, where a single feature is constructed for such a predefined concept. We present a controlled comparison across three model signals (activation values, activation gradients, and parameter gradients) and two estimators (contrastive mean and a learned one-dimensional encoder-decoder), yielding six targeted methods, with CAA and GRADIEND as existing instances and four new methods covering the remaining combinations. We compare these methods against pretrained SAEs across 15 tasks and three language models, evaluating both detection and causal intervention. Across models, the strongest detection performance is achieved by contrastive activation value methods, whereas the strongest intervention performance is achieved by gradient-based methods. Overall, our results show that targeted feature quality depends jointly on the model signal and estimator, with detection and intervention capturing complementary properties.

Figures & tables

Figure 1: Detection and intervention capture complementary properties of targeted features. Across 15 tasks and three language models, activation value methods show stronger detection and most gradient-based methods stronger intervention overall. Per-model results are shown in Figure 3 .
Figure 2: Study overview. From a predefined feature specification, we cross three model signals and two estimators, yielding six targeted feature learning methods evaluated on detection and intervention. SAEs provide an untargeted activation value reference closest to ACTIEND .
SignalContrastive meanInterpretable ENcoder Decoder (IEND)
Activation valueContrastive Activation Addition (CAA) ( Rimsky et al., 2024 )ACTivation IEND (ACTIEND) † (new)
Activation gradientContrastive Activation Gradient Addition (CAGA) (new)Activation Gradient IEND (AGIEND) (new)
Parameter gradientContrastive Gradient Addition (CGA) (new)GRADient IEND (GRADIEND) * ( Drechsel and Herbold, 2026a )
Table 1: Targeted feature learning methods across three model-derived signals and two estimators .
FamilyEval.Tasks (classes)
Semanticpwpw / 1s1sGender (male/female); Emotion (positive/negative); Race (Asian/Black/White); Religion (Christian/Muslim/Jewish)
Factualpwpw / 1s1sRavelCont (Asia/Europe/Africa); RavelCntry (United States/China/Russia); RavelLang (English/Portuguese/Spanish)
Languagepwpw / 1s1sLangID (English/French/German)
Grammaticalpwpw / 1s1sPronNum (singular/plural); PronPers (first/second/third)
Algorithmic1s1sIOI (IO/subject); KeyValue (value/other value); Induction (match/distractor); Repetition (yes/no); FuncComp (result/distractor)
Table 2: Summary of the 15 feature learning tasks. pwpw denotes pairwise and 1s1s one-sided tasks.
Figure 3: Task-averaged detection and intervention for each method. Lines connect the two estimators using the same signal and construction. The model-averaged aggregate is shown in Figure 1 .
Figure 4: Overview of task-level performance and Bayesian method comparisons.
Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

TaskFeature classesReg.Train/val/testSource and Hugging Face ID
Gendermale (M), female (F)P73,860/2,110/8,440aieng-lab/genter-ajibawa-name-filled ( Drechsel and Herbold, 2026a ) subset n1
Emotionpositive, negativeP600/200/200aieng-lab/en-sentiment-nrc ( Drechsel and Herbold, 2026b ) vocabulary-held-out split
RaceAsian, Black, WhiteP33,602/9,603/4,799aieng-lab/gradiend_race_data ( Drechsel and Herbold, 2026a )
ReligionChristian, Muslim, JewishP20,048/5,730/2,863aieng-lab/gradiend_religion_data ( Drechsel and Herbold, 2026a )
RavelContAsia, Europe, AfricaP1,344/168/168Source: hij/ravel ( Huang et al., 2024 ) ; Processed aieng-lab/gradiend-ravel-continent (new)
RavelCntryUnited States, China, RussiaP408/51/51Source: hij/ravel ( Huang et al., 2024 ) ; Processed aieng-lab/gradiend-ravel-country (new)
Appendix
Table 3: Complete task inventory. Sizes are effective labeled train/validation/test rows after task-level class mapping and before counterfactual expansion. P = both poles occur naturally and both pairwise and one-sided regimes are evaluated. O = only the first listed class is factual and the second is its constructed counterpart. We do not redistribute LangID due to licensing constraints and instead provide a script for reconstructing it from the original sources.
TaskMasked contextTarget / class
GenderCarl looked around the room, noticing tired expressions and disheartened faces. [MASK] realized then that something needed to change.he / M
EmotionI am so unbelievably [MASK] for tomorrow.excited / positive
RaceLutron alleged that products were made in Mexico and [MASK] and then imported into the United States.China / Asian
ReligionThe inscription should, from a [MASK] point of view, pinpoint where Jumne was located.Christian / Christian
RavelContToronto is a city in the continent of North America. Pilibhit is a city in the continent of [MASK] .Asia / Asia
RavelCntrySt. Petersburg is in Russia. Johnstown is in [MASK] .United States / United States
Appendix
Table 4: Representative labeled examples. Long natural contexts are shortened with an ellipsis. The masked span and local wording are retained. The right column gives the factual target token followed by its feature class. For one-sided tasks the counterfactual class is still listed in Table 3 , but does not add a second base row.
TaskSizeSource and Hugging Face ID
Gender , Race , Religion , IOI , KeyValue , Induction , Repetition , FuncComp4,593,588aieng-lab/biasneutral-ajibawa ( Drechsel and Herbold, 2026a )
Emotion1,000aieng-lab/en-sentiment-nrc-neutral ( Drechsel and Herbold, 2026b )
PronNum , PronPers10,000aieng-lab/en-pronoun-neutral ( Drechsel and Herbold, 2026b )
RavelCont5,000Source: hij/ravel ; Processed: aieng-lab/gradiend-ravel-continent-neutral (new)
RavelCntry5,000Source: hij/ravel ; Processed: aieng-lab/gradiend-ravel-country-neutral (new)
RavelLang5,000Source: hij/ravel ; Processed: aieng-lab/gradiend-ravel-language-neutral (new)
Appendix
Table 5: Task-dependent neutral dataset overview.
ModelHF checkpointSAELens releaseSAE ID / hook
Pythia-70MEleutherAI/pythia-70m-dedupedpythia-70m-deduped-res-smblocks.{L}.hook_resid_post
GPT-2gpt2gpt2-small-resid-post-v5-32kblocks.{L}.hook_resid_post
Llama-3.1-8Bmeta-llama/Llama-3.1-8Bllama_scope_lxr_32xl{L}r_32x
Appendix
Table 6: Language-model checkpoints and pretrained SAE resources used in the study.
Figure 5: Computational cost and peak GPU memory on Gender . Left: wall-clock time separated into fitting or training, detection, and intervention evaluation. Right: peak allocated and reserved CUDA memory across these stages. Reserved memory includes allocated memory and is therefore shown as an overlaid rather than additive quantity. SAE pretraining is excluded because we use pretrained dictionaries.
SettingValue
IEND optimization
GRADIEND learning rate10−410^{-4} ( GPT-2 ), 10−610^{-6} ( Pythia-70M ), 2×10−62{\times}10^{-6} ( Llama-3.1-8B )
ACTIEND encoder learning rate5×10−55{\times}10^{-5} ( Pythia-70M , GPT-2 ), 10−510^{-5} ( Llama-3.1-8B )
ACTIEND decoder learning rate1.15×10−21.15{\times}10^{-2}
AGIEND learning rate3×10−33{\times}10^{-3} ( Pythia-70M , GPT-2 , Llama-3.1-8B )
Training steps500 ( GRADIEND , AGIEND ); 100 ( ACTIEND )
Appendix
Table 7: Training, selection, and evaluation hyperparameters.
Figure 6: Bayesian pairwise comparison of methods.
 SelectedDetectionIntervention
MethodSingle layerWinMean Δ\DeltaWinMean Δ\Delta
SAE96.9%49.0%+0.035+0.03527.3%−0.013-0.013
CAA87.9%82.5%+0.082+0.08245.0%−0.001-0.001
CGA95.4%85.0%+0.153+0.15334.8%−0.041-0.041
CAGA94.2%86.3%+0.140+0.14038.3%−0.015-0.015
Overall93.0%80.4%+0.115+0.11538.1%−0.018-0.018
Appendix
Table 8: Effect of representation scope. Single-layer representations are selected by validation Detection. Δ\Delta denotes single layer minus all-layer performance on the held-out test split. Win rates report the fraction of class/contrast comparisons with Δ>0\Delta>0 , i.e., where the selected single layer outperforms the all-layer representation on the held-out test split.
Figure 7: Difference between the validation-selected best single-layer and all-layer representations for detection and intervention. Positive values indicate higher performance for the selected single layer.
Figure 8: All-layer robustness analysis. Detection-intervention comparison corresponding to Figure 3 , with SAE and contrastive mean methods restricted to their all-layer representations rather than using validation-selected scopes.
Figure 9: Layerwise detection and intervention for Pythia-70M .
Figure 10: Layerwise detection and intervention for GPT-2 .
Figure 11: Detection and Intervention for the four SAE variants obtained by varying latent-selection position (filled-target vs. pre-target) and the number of selected latents ( k=1k=1 vs. k⋆k^{\star} ).
Figure 12: Performance by task family averaged across models.
Figure 13: Pairwise versus one-sided feature construction. Model-balanced mean Detection and Intervention on tasks supporting both constructions.
Figure 14: Pythia-70M detection and intervention scores per task.
Figure 15: GPT-2 detection and intervention scores per task.
Figure 16: Llama-3.1-8B detection and intervention scores per task.
Table 9: Detection raw results for Pythia-70M .
Table 10: Intervention raw results for Pythia-70M .
Table 11: Detection raw results for GPT-2 .
Table 12: Intervention raw results for GPT-2 .
Table 13: Detection raw results for Llama-3.1-8B .
Table 14: Intervention raw results for Llama-3.1-8B .
More in this topic

LLM Interpretability

Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.

See all 781 papers

Explore similar work

CardsList
Feb 4, 2026

Towards Isolated Interventions via Almost Orthogonal Features in Language Models

A central premise in mechanistic interpretability is that meaningful concepts in language models are represented by linear features in activation space. For such features to support reliable interventions, manipulating one feature should not substantially alter the effects of others. In practice, however, feature entanglement leads to interference such that localized interventions can have unintended downstream effects. Motivated by the Independent Causal Mechanisms principle, we propose to constrain internal features to be almost orthogonal. We argue that this promotes modular representations amenable to causal intervention. We formalize this problem by characterizing the gap between an idealized isolated intervention and its realized effect on model outputs in terms of feature interference. We upper-bound the propagation of feature interference in terms of the self-coherence of the feature dictionary, and relate this discrepancy to an explicit orthogonality regularization on the dictionary itself. Empirically, we show that this regularization enables more isolated interventions on mathematical reasoning concepts while preserving model performance. Our code is available under https://github.com/mrtzmllr/sae-icm.

Moritz Miller, Florent Draye, Bernhard Schölkopf

Max Planck Institute for Intelligent Systems, Tübingen, Germany · ETH Zurich, Switzerland · Max Planck ETH Center for Learning Systems +1

Disentangled Representation LearningRepresentation LearningOrthogonal Representation LearningMechanistic InterpretabilityCausal Interventions in Language Models
AskPDF
Oct 11, 2026

What Do Sparse Autoencoder Feature Discovery Scores Identify? From Activations to Interventions

Sparse autoencoder (SAE) features are typically discovered from how their activations co-vary with behaviour, yet they are often used through interventions such as steering or ablation. We take intervention attribution, the first-order effect of a specified intervention on the model output, as our estimand. For a broad class of response functions, we show that observational feature-discovery evidence generally does not identify this attribution, and derive an exact gradient-based decomposition of the gap between linear-probe predictions and intervention attribution. Across four model-SAE pairs, median probe-to-attribution magnitude ratios range from tens to hundreds, while attributions closely track small-step steering. For single-latent ablations with nonzero effects, gradient predictions match the realized sign in 91-97% of cases, compared with 17-35% for probe-implied changes. Attribution-selected sets also produce larger absolute mean ablation effects on held-out prompts than sets selected by correlation, rank tests or probe coefficients. Together, these results establish intervention attribution as a principled, gradient-estimable target for selecting SAE features for intervention.

Yifeng Guo, Yanran Li, Xiang Chen

St. Jude Children’s Research Hospital · Columbia University

Sparse AutoencodersMechanistic InterpretabilityCausal Interventions in Language ModelsGradient-Based Attribution
AskPDF
Oct 13, 2026

Evaluating the Causal Faithfulness of SHAP as a Candidate Attribution Method for SAE Features

Sparse autoencoders decompose language model representations into interpretable features, but feature interpretation in this literature remains predominantly correlational. Attribution methods offer more direct estimates of feature influence, yet their faithfulness to actual causal effects has not been systematically evaluated in the SAE setting. We apply SHAP-based attribution to SAE latent features and evaluate it against direct causal intervention, using sentiment classification on SST-2 with GPT-2 Small and a 32,768-feature layer-7 SAE as a testbed. We compare SHAP to Integrated Gradients, Gradient×\timesInput, and probe-weight baselines across linear and MLP probe arms, scoring each against feature ablation and activation steering measured through the language model's own logit difference. Most faithfulness associations we measure do not survive multiple-comparison correction, and attribution methods that agree strongly with one another (ρ=0.42ρ= 0.42--0.800.80) agree poorly with measured causal effects, indicating that inter-method agreement is not evidence of causal faithfulness. Two effects hold under control. First, local per-example attribution retains a directional advantage over global averaged attribution, while its apparent importance advantage is explained by feature activity rather than superior causal ranking. Second, under known-ground-truth minimal pairs, attribution recovers inserted sentiment signal well above chance (AUROC 0.7920.792) but fails to localize deletions (AUROC 0.5200.520), suggesting that these methods detect the presence of a feature's influence more reliably than its absence. We argue that importance faithfulness and directional faithfulness are separable properties requiring separate evaluation, and that attribution agreement should not substitute for causal validation.

Samuel Xie, Noura Al-Moubayed

Independent · Durham University

Faithfulness of Language Model ExplanationsSHAP Feature AttributionSparse Autoencoders
AskPDF