Influence Score and Transformers interpretability: Measure of the Effective Impact of Attention Heads at inference time

Lisa Bouger; Yannick Teglia; Philippe Loubet Moundi

cs.CLSep 4, 2026

Influence Score and Transformers interpretability: Measure of the Effective Impact of Attention Heads at inference time

Authors: Lisa Bouger, Yannick Teglia, Philippe Loubet Moundi

Organizations: Thales CDI, France · Inria Paris, France · Sorbonne Université, France

Abstract

We propose an influence score to quantify the contribution of attention heads to classification decisions in Transformer-based models designed for prompt injection detection. The score combines directional influence on the logits with structural contribution within the residual stream, enabling a multi-scale analysis at the head, layer, and network levels. Applied to a DeBERTa model specialized for prompt injection detection, our framework reveals distinct decision behaviours between correct and erroneous predictions. Our method provides an effective compromise between fine-grained circuit analysis and global output-based methods, and offers a systematic way to study decision mechanisms in Transformer classifiers.

Figures & tables

Figure 1: Overview of the influence score. The impact of each head is measured by its directional influence on the logit (

\Delta_{k,h}

) weighted by its contribution to the residual stream (

C_{k,h}\alpha_{k}

Figure 2: Average attention-head contributions for layer 0 (left) and layer 11 (right), separated by category ( TN , TP , FN , FP ). In the early layers, the distributions are similar across categories and dominated by a shared head ( $H_{0}$ in layer 0). In deeper layers, contributions become highly concentrated and specialize according to the class and prediction correctness: $H_{6}$ dominates for TN , while $H_{2}$ dominates for TP . Errors exhibit profiles similar to those of the opposite class, suggesting the activation of inappropriate circuits.

Figure 3: Distribution of $S_{\mathrm{final}}$ for TP / TN / FP / FN (all layers $\mathcal{L}={0,\dots,L-1}$ ).

Panel of Figure 4 — Figure 4: Evolution of $\mathbb{E}[S_{\mathrm{final}}]$ (top) and the Injection / Benign signals (bottom) as a function of the number of accumulated layers $|\mathcal{L}|$ .

Cat.	Top $S_{k,h}>0$ (pro- Inj )	Score	Top $S_{k,h}<0$ (pro- Ben )	Score
TP	$L_{11}H_{6}$ , $L_{10}H_{1}$ , $L_{11}H_{2}$	0.03, 0.03, 0.01	$L_{11}H_{2}$ , $L_{10}H_{1}$ , $L_{11}H_{6}$	-0.01, -0.01, -0.01
TN	$L_{11}H_{6}$ , $L_{0}H_{0}$ , $L_{11}H_{2}$	0.06, 0.01, 0.01	$L_{11}H_{6}$ , $L_{11}H_{2}$ , $L_{11}H_{3}$	-0.04, -0.01, -0.01
FP	$L_{11}H_{6}$ , $L_{10}H_{1}$ , $L_{0}H_{0}$	0.13, 0.01, 0.01	$L_{11}H_{2}$ , $L_{0}H_{7}$ , $L_{11}H_{9}$	-0.02, -0.01, -0.01
FN	$L_{11}H_{6}$ , $L_{11}H_{5}$ , $L_{0}H_{0}$	0.07, 0.01, 0.01	$L_{11}H_{6}$ , $L_{11}H_{2}$ , $L_{11}H_{3}$	-0.06, -0.01, -0.04

Table 1: Dominant heads by category, defined as those with the highest mean value of

S_{k,h}(x)

(positive) or of

|S_{k,h}(x)|

under the constraint

S_{k,h}(x)<0

(negative). The reported scores correspond to the mean values of

S_{k,h}(x)

over the examples in each category.

Group	Criterion	F1	Acc.	$\boldsymbol{\Delta\text{F1}}$	$\boldsymbol{\Delta\text{TP}}$	$\boldsymbol{\Delta\text{TN}}$
TP	Score	0.898	0.944	0.014	257	$-50$
	Logit only	0.904	0.947	0.009	161	$-32$
	Contrib only	0.905	0.948	0.007	117	$-8$
TN	Score	0.903	0.946	0.009	138	9
	Logit only	0.897	0.944	0.015	273	$-47$
	Contrib only	0.904	0.947	0.009	130	$8$

Table 2: Ablation results for the dominant heads of the TP and TN groups. The

\Delta\text{F1}

\Delta\text{TP}

, and

\Delta\text{TN}

values correspond to the difference between the base model and the ablated model (positive values indicate a decrease in performance).

Panel of Figure 5 — Figure 5: Evolution of $S_{\mathrm{final}}$ distributions for the TN clusters (C0, C1, C2) as a function of the number of accumulated layers.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Proportion (%)
Instruction / assistant datasets (Alpaca 12 12 12 https://huggingface.co/datasets/tatsu-lab/alpaca Taori and others (2023) , OpenOrca 13 13 13 https://huggingface.co/datasets/Open-Orca/OpenOrca , UltraChat 14 14 14 https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k Ding and others (2023) , Open-Instruct 15 15 15 https://huggingface.co/datasets/VMware/open-instruct Wang and others (2023) )	62.8
Prompt collections (Awesome ChatGPT Prompts 16 16 16 https://github.com/f/awesome-chatgpt-prompts )	2.6
Jailbreak Classification Dataset (jackhhao/jailbreak-classification 17 17 17 https://huggingface.co/datasets/jackhhao/jailbreak-classification )	3.2
Microsoft TaskTracker Injection Datasets (TaskTracker Prompt Injection Dataset 18 18 18 https://github.com/microsoft/TaskTracker/tree/main/data/prompt_injection )	30.2
JailbreakBench (JailbreakBench Dataset 19 19 19 https://huggingface.co/datasets/JailbreakBench/JBB-Behaviors Chao and others (2024) )	1.2
Total	100

Appendix

Table 3: Contribution of the different sources to the final dataset

Cluster	#ex.	Duplication (%)	Median char. length	Median words	% $>$ 1000 car.
C0	2 112	0.52	2 103	366	74.6
C1	11 377	0.26	313	53	8.9
C2	6 942	4.78	115	19	14.3

Appendix

Table 4: Descriptive characteristics of the three TN clusters. TN .

Transformer Interpretability

Transformer interpretability studies how attention, feed-forward layers, and hidden representations support model predictions and behavior.

See all 339 papers

Prompt Injection Detection is Regime-Dependent: A Deployment-Aware Evaluation with Interpretable Structural Signals
May 26, 2026Akindoyin Akinrele, Shreyank N GowdaLanguage Model Safety Evaluation Prompt Injection Attacks on LLMs
School of Computer Science, University of Nottingham, Jubilee Campus, Wollaton Road, Nottingham, NG8 1BB, United Kingdom
Causal Localization of Evidence Effects in Transformers
Oct 27, 2026Jipeng Li, Pramod Khargonekar, Yanning ShenTransformer Interpretability Attention Head Analysis
Importance Scoring of Transformer Attention Heads in Learning Tabular Data
Aug 27, 2026Ahmad Jad Allah, Kazi F. Akhter, Md. Kamrozzaman Bhuiyan +1Attention Head Analysis Transformer Attention
Department of Computer Science Tennessee State University Nashville, TN, USA · Enosis Solutions Dhaka, Bangladesh · Department of Computer Science North Carolina Agricultural and Technical University Greensboro, NC, USA

May 26, 2026J/K move · Enter open · S save

Prompt Injection Detection is Regime-Dependent: A Deployment-Aware Evaluation with Interpretable Structural Signals

Akindoyin Akinrele, Shreyank N Gowda

School of Computer Science, University of Nottingham, Jubilee Campus, Wollaton Road, Nottingham, NG8 1BB, United Kingdom

Language Model Safety Evaluation Prompt Injection Attacks on LLMs Prompt Injection Detection

Open paper AskPDF arXiv

Prompt injection poses a critical threat to the safe deployment of large language models, yet existing detection approaches are typically evaluated under limited settings that do not reflect real-world operating constraints. In this work, we present a deployment-aware evaluation of prompt injection detection using a multi-model and multi-regime experimental framework. We compare lexical, semantic, structural, and transformer-based detectors across multiple out-of-distribution settings, repeated data splits, and both ranking and thresholded deployment metrics. We introduce interpretable structural signals that capture hierarchy overrides, system prompt spoofing, role redefinition, and evasion patterns, and assess their contribution both within sparse models and in combination with strong encoder baselines. Our results show that detection performance is highly regime-dependent and sensitive to threshold selection, with no single model dominating across all settings. Transformer-based models achieve the strongest overall performance, while structural signals provide modest but consistent gains in certain regimes and improve low false positive rate behaviour in harder scenarios. These findings highlight the gap between ranking performance and deployment effectiveness and underscore the importance of evaluating prompt injection defences under realistic operational constraints. Code will be released.

Abstract

Transformer Interpretability

Explore similar work

Prompt Injection Detection is Regime-Dependent: A Deployment-Aware Evaluation with Interpretable Structural Signals

Causal Localization of Evidence Effects in Transformers

Importance Scoring of Transformer Attention Heads in Learning Tabular Data