From Attribution to Action: A Human-Centered Application of Activation Steering

Tobias Labarta; Maximilian Dreyer; Katharina Weitz; Wojciech Samek; Sebastian Lapuschkin

cs.AIApr 13, 2026

From Attribution to Action: A Human-Centered Application of Activation Steering

Authors: Tobias Labarta, Maximilian Dreyer, Katharina Weitz, Wojciech Samek, Sebastian Lapuschkin

Organizations: Fraunhofer Heinrich-Hertz-Institut · Technische Universität Berlin · BIFOLD – Berlin Institute for the Foundations of Learning and Data

Abstract

Explainable AI (XAI) methods reveal which features influence model predictions, yet provide limited means for practitioners to act on these explanations. Activation steering of components identified via XAI offers a path toward actionable explanations, although its practical utility remains understudied. We introduce an interactive workflow combining SAE-based attribution with activation steering for instance-level analysis of concept usage in vision models, implemented as a web-based tool. Based on this workflow, we conduct semi-structured expert interviews (N=8) with debugging tasks on CLIP to investigate how practitioners reason about, trust, and apply activation steering. We find that steering enables a shift from inspection to intervention-based hypothesis testing (8/8 participants), with most grounding trust in observed model responses rather than explanation plausibility alone (6/8). Participants adopted systematic debugging strategies dominated by component suppression (7/8) and highlighted risks including ripple effects and limited generalization of instance-level corrections. Overall, activation steering renders interpretability more actionable while raising important considerations for safe and effective use.

Figures & tables

Figure 1: The four-step workflow from attribution to action: practitioners review component attributions, form causal hypotheses about components’ roles, and test them via activation steering. The resulting prediction updates provide immediate feedback for hypothesis evaluation. Example shows suppressing the “woman” component to assess its influence on job role classification for “nurse”.

Figure 2: SemanticLens workflow implementation. Users select inspection samples (1), review components ranked by attribution (2), examine component details and visualizations (3), apply steering modifications (4), and observe prediction changes (5). Yellow text and bars indicate post-modification state. Example shown: a melanoma case corrected after suppressing component #30496, which responds to text artifacts.

Figure 3: A process diagram of the interview structure: Pre-questionnaire, two debugging tasks each with attribution-only phase (Phase 1) and steering phase (Phase 2), and post-questionnaire.

Figure 4: Two debugging tasks: Task 1 (typographic attack on CLIP ViT-B-32, where overlaid text causes misclassification [ 52 ] ) and Task 2 (gender bias in CLIP ViT-L-14, where gender swap shifts predictions toward lower-status professions). Red badges indicate components contributing to misclassification; green badges indicate components supporting correct class. Component #3177 (“woman”) appears among top-attributed features in both tasks.

RQ	Code	Count
RQ1	Correlational reasoning (pre-steering)	8
	Causal hypothesis formation	6
	Causal test completed	8
	Causal limitation acknowledged	5
RQ2	Plausibility-based trust	2
	Evidence-based trust (post-steering)	6

Table 1: The 22 codes used to analyze interview data, organized by research question. Count indicates the number of participants for whom each code was identified.

ID	Recog.	Comp.	Fix	Att.	Recog.	Comp.	Fix	Att.
	T1: Typographic				T2: Gender Bias
P1	✓	✓	✓	1	✓	✓	✓	1
P2	✓	✘	✓	2	✓	✓	✓	1
P3	✓	✓	✓	2	✓	✓	✓	1
P4	✓	✓	✓	3	✓	✓	✓	1
P5	✓	✓	✓	1	✓	✓	✓	3

Table 2: Task outcomes across participants. Recog. = recognized the failure mode; Comp. = identified the responsible component; Fix = successfully corrected the prediction via steering; Att. = number of steering attempts to achieve fix. T1 = typographic attack task, T2 = gender bias task.

Explainable Artificial Intelligence

Explainable artificial intelligence develops ways to make AI decisions understandable, testable, and useful to people.

See all 326 papers

Language Models Can Explain Visual Features via Steering
Mar 23, 2026Javier Ferrando, Enrique Lopez-Cuena, Pablo Agustin Martin-Torres +3VLM Interpretability
Barcelona Supercomputing Center
Steered LLM Activations are Non-Surjective
Apr 10, 2026Aayush Mishra, Daniel Khashabi, Anqi LiuActivation Steering LLM Interpretability
Johns Hopkins University
XALIGN: A Human-centric Visual Analytics Approach to Understanding AI Explanations
Oct 22, 2026Bahavathy Kathirgamanathan, Gennady Andrienko, Natalia Andrienko +1Explainable Artificial Intelligence Explainability Evaluation
University of Cologne, Cologne, Germany · Fraunhofer IAIS, Sankt Augustin, Germany · City St George’s Institute of London, London, UK

Mar 23, 2026J/K move · Enter open · S save

Language Models Can Explain Visual Features via Steering

Javier Ferrando, Enrique Lopez-Cuena, Pablo Agustin Martin-Torres, Daniel Hinjos+2

Barcelona Supercomputing Center

VLM Interpretability

Open paper AskPDF arXiv

Sparse Autoencoders uncover thousands of features in vision models, yet explaining these features without requiring human intervention remains an open challenge. While previous work has proposed generating correlation-based explanations based on top activating input examples, we present a fundamentally different alternative based on causal interventions. We leverage the structure of Vision-Language Models and steer individual SAE features in the vision encoder after providing an empty image. Then, we prompt the language model to explain what it ``sees'', effectively eliciting the visual concept represented by each feature. Results show that Steering offers an scalable alternative that complements traditional approaches based on input examples, serving as a new axis for automated interpretability in vision models. Moreover, the quality of explanations improves consistently with the scale of the language model, highlighting our method as a promising direction for future research. Finally, we propose Steering-informed Top-k, a hybrid approach that combines the strengths of causal interventions and input-based approaches to achieve state-of-the-art explanation quality without additional computational cost.

Abstract

Explainable Artificial Intelligence

Explore similar work

Language Models Can Explain Visual Features via Steering

Steered LLM Activations are Non-Surjective

XALIGN: A Human-centric Visual Analytics Approach to Understanding AI Explanations