SYNAPSE: Framework for Neuron Analysis and Perturbation in Sequence Encoding

Jesús Sánchez Ochoa; Enrique Tomás Martínez Beltrán; Alberto Huertas Celdrán

cs.LGMar 9, 2026

SYNAPSE: Framework for Neuron Analysis and Perturbation in Sequence Encoding

Authors: Jesús Sánchez Ochoa, Enrique Tomás Martínez Beltrán, Alberto Huertas Celdrán

Organizations: Department of Information and Communications Engineering, University of Murcia, Murcia, 30100, Spain

Abstract

In recent years, Artificial Intelligence has become a powerful partner for complex tasks such as data analysis, prediction, and problem-solving, yet its lack of transparency raises concerns about its reliability. In sensitive domains such as healthcare or cybersecurity, ensuring transparency, trustworthiness, and robustness is essential, since the consequences of wrong decisions or successful attacks can be severe. Prior neuron-level interpretability approaches are primarily descriptive, task-dependent, or require retraining, which limits their use as systematic, reusable tools for evaluating internal robustness across architectures and domains. To overcome these limitations, this work proposes SYNAPSE, a systematic, training-free framework for understanding and stress-testing the internal behavior of Transformer models across domains. It extracts per-layer [CLS] representations, trains a lightweight linear probe to obtain global and per-class neuron rankings, and applies forward-hook interventions during inference. This design enables controlled experiments on internal representations without altering the original model, thereby allowing weaknesses, stability patterns, and label-specific sensitivities to be measured and compared directly across tasks and architectures. Across all experiments, SYNAPSE reveals a consistent, domain-independent organization of internal representations, in which task-relevant information is encoded in broad, overlapping neuron subsets. This redundancy provides a strong degree of functional stability, while class-wise asymmetries expose heterogeneous specialization patterns and enable label-aware analysis. In contrast, small structured manipulations in weight or logit space are sufficient to redirect predictions, highlighting complementary vulnerability profiles and illustrating how SYNAPSE can guide the development of more robust Transformer models.

Figures & tables

Reference	Year	Model Type	Domain / Purpose	Explanation Type	Scope
Li et al. [ 17 ]	2017	LSTM-based NLP classifier	NLP (sentiment, linguistic tasks)	Feature ablation	Local Global
Radford et al. [ 21 ]	2017	mLSTM	NLP (unsupervised text generation)	Emergent feature analysis	Global
Bau et al. [ 2 ]	2019	NMT	NLP (machine translation)	Neuron importance ranking and manipulation	Global
Dalvi et al. [ 6 ]	2018	Pre-trained RNN	NLP (various language tasks)	Neuron interpretation	Global
Durrani et al. [ 12 ]	2020	Transformer (BERT)	NLP (language understanding)	Neuron importance and clustering	Global
Dalvi et al. [ 7 ]	2023	Transformers	NLP (interpretability)	Toolkit supporting multiple neuron analysis methods	Local Global

Table 1: Comparison of Neuron Analysis Methods.

Reference	Year	Threat Model	Attack Type	Phase	Access	Model Type
Szegedy et al. [ 25 ]	2014	Evasion	Small perturbation adversarial examples (L-BFGS method)	Inference	White-box	DNN (CNN)
Goodfellow et al. [ 14 ]	2015	Evasion	FGSM (Fast Gradient Sign Method)	Inference	White-box	DNN (CNN)
Pravin et al. [ 19 ]	2021	Evasion	Neuron-fragility exploitation (target-specific “fragile” neurons)	Inference	White-box	DNN (ResNet)
Che et al. [ 4 ]	2024	Capability elicitation	Hidden feature/weight manipulation (latent and weight-space attacks)	Inference (and fine-tune)	White-box	LLM (GPT family)

Table 2: Adversarial and poisoning attacks in machine learning.

Figure 1: Internal structure of SYNAPSE

Model	Layers ( $L$ )	Hidden Size ( $d_{model}$ )	Total Neurons (MLP)	Max. Seq. Length	Param. Count
BERT (base)	12	768	36,864	512	110M
BigBird	12	768	36,864	4,096	128M
DistilBERT	6	768	18,432	512	66M
Longformer	12	768	36,864	4,096	149M
GoEmotions (BERT)	12	768	36,864	512	110M

Table 3: Architectural specifications of the evaluated Transformer models.

Model	Weighted F1-Score
BERT	0.6834
DistilBERT	0.6081
BigBird	0.8306
Longformer	0.8516
Baseline	0.8516

Table 4: Baseline weighted F1-Score per model.

Figure 2: Global Silencing Attack results on BigBird

Model	Baseline	5%	10%	20%	30%	50%	75%	95%	$\Delta$ F1@95% (%)
BERT	0.683	0.686	0.686	0.686	0.686	0.575	0.482	0.147	-78.4
BigBird	0.785	0.806	0.815	0.832	0.735	0.581	0.246	0.079	-89.9
DistilBERT	0.608	0.608	0.608	0.608	0.608	0.608	0.499	0.311	-48.8
Longformer	0.852	0.844	0.871	0.852	0.768	0.535	0.097	0.079	-90.7

Table 5: Global neuron silencing: weighted F1-Score for different Transformer models under increasing fractions of globally important neurons being silenced.

\Delta

F1-Score denotes the relative variation with respect to the baseline.

Panel of Figure 3 — Figure 3: Results after silencing per-class ranking for BigBird across classes.

Model	Class 0		Class 1		Class 2		Class 3		Class 4
	F1	F1 WO *	F1	F1 WO	F1	F1 WO	F1	F1 WO	F1	F1 WO
BERT	0.167	0.166	0.800	0.528	0.000	0.242	0.923	0.455	0.533	0.328
BigBird	0.200	0.220	0.250	0.071	0.000	0.034	0.222	0.326	0.000	0.034
DistilBERT	0.444	0.212	0.909	0.348	0.730	0.489	0.000	0.481	0.182	0.195
Longformer	0.364	0.253	0.000	0.332	0.588	0.425	0.400	0.384	0.462	0.355

Table 6: Final performance under class-targeted neuron silencing (Classes 0–4).

Model	F1-Score (5%)	F1-Score (10%)	F1-Score (20%)	F1-Score (30%)	F1-Score (50%)	$\Delta$ F1-Score (%)
BERT	0.5121	0.4478	0.3952	0.3520	0.3015	-23.1%
DistilBERT	0.5013	0.4205	0.3667	0.3304	0.2801	-21.2%
BigBird	0.7310	0.6558	0.5887	0.5402	0.4814	-11.9%
Longformer	0.7702	0.7021	0.6504	0.5989	0.5407	-11.0%

Table 7: Impact of FGSM attack on model performance. The table reports the weighted F1-Score under different perturbation strengths, and the relative degradation compared to the baseline F1-Score.

Figure 4: Random Noise model performance degradation

Model	$\epsilon=0.10$	$\epsilon=0.50$	$\epsilon=1.00$	Relative $\Delta$ F1-Score (%)
BERT	0.6081	0.6081	0.5214	$-23.7$ %
DistilBERT	0.6081	0.6081	0.5214	$-14.3$ %
BigBird	0.8306	0.8306	0.8306	$0.0$ %
Longformer	0.8712	0.3903	0.0344	$-95.9$ %

Table 8: Model robustness under random noise injection at different noise levels (

\epsilon

Figure 5: Logit Bias Attack results on BigBird

Model	Accuracy	F1-Score	% Pred. $\rightarrow$ Target	% Flips
		(weighted)		(non-target)
BERT	0.7000	0.6867	40.00%	28.57%
BigBird	0.7200	0.7052	36.00%	28.13%
DistilBERT	0.6800	0.5896	18.00%	6.82%
Longformer	0.6400	0.6336	52.00%	42.86%

Table 9: Logit Bias attack (bias = 8.0, target class = 3). Comparison across models based on accuracy, weighted F1-Score, percentage of predictions assigned to the target class, and percentage of flips from non-target classes.

Model	Weighted F1-Score
BERT	0.7421
DistilBERT	0.5812
BigBird	0.7624
Longformer	0.7443

Table 10: Performance under Gaussian Noise Injection (

\sigma=0.9

, 60% of neurons perturbed).

Model	Accuracy	F1-Score	Flips $\rightarrow$ Target	Pred $\rightarrow$ Target (%)
		(weighted)
BERT	0.76	0.6834	2.27	14
BigBird	0.38	0.3126	71.43	76
DistilBERT	0.56	0.4151	18.18	28
Longformer	0.82	0.8073	0.00	6

Table 11: Balanced weight-push attack results.

Model	Accuracy	F1-Score	Flips $\rightarrow$ Target (%)	Pred $\rightarrow$ Target
		(weighted)
BERT	0.76	0.6834	2.27	14
BigBird	0.76	0.6741	14.29	28
DistilBERT	0.68	0.5871	4.55	16
Longformer	0.84	0.8315	0.00	6

Table 12: Class-suppression weight attack results.

Model	Accuracy	F1-Score	Flips $\rightarrow$ Target (%)	Pred $\rightarrow$ Target
		(weighted)
BERT	0.76	0.6834	2.27	14
BigBird	0.82	0.8064	7.14	22
DistilBERT †	–	–	–	–
Longformer	0.86	0.8516	7.14	22

Table 13: Bias-only weight attack results.

Attack	Config	F1-Score (weighted)	$\Delta$ F1-Score (%)
Baseline	–	0.860	–
Global neuron silencing	60% neurons silenced globally	0.300	-65.1
Per-class silencing	Best run (class 1), peak F1-Score = 0.882	0.882	+2.6
Random noise injection	$\epsilon=1.5$ applied to top neurons	0.460	-46.5
Gaussian noise	$\sigma=0.9$ (estimated)	0.630	-26.7
Logit bias	Best stable point ( $\Delta=8$ ). Higher $\Delta$ collapses predictions (F1-Score $\downarrow$ to 0.192).	0.624	-27.4

Table 14: Robustness of the GoEmotions model under neuron- and weight-level attacks (weighted F1-Score).

Neuron-Level Interpretability

See all 20 papers

Aug 25, 2026

RACE: Scalable Statistical Estimation of Functional Consistency in LLM Neurons

Discovering stable neuron behavior across entire domains remains a challenge in mechanistic interpretability. Existing methods often rely on instance-level point estimates or computationally expensive procedures, which either obscure population-level variability or limit scalable domain-wide analysis. We present RACE (Residual Alignment for Consistency Estimation), a forward-pass statistical framework that evaluates the domain-wide functional consistency of Transformer neurons. Compared with gradient-based point estimates, RACE produces neuron rankings that yield more domain-specific effects under perturbation. Token-distribution shifts support the connection between the selected neurons and the target domain, while scoring requires roughly one-hundredth of the computational overhead of the gradient-based methods. Code is available at https://github.com/Nexround/RACE.

Runyu Wang, Bo Liu, Xiaxin Zhang +6

School of Transportation and Civil Engineering, Nantong University · Chongqing University of Post and Telecommunications · China Southern Power Grid Company Limited +1

Transformer Interpretability LLM Interpretability Neuron-Level Interpretability Mechanistic Interpretability

Ask PDF

Oct 14, 2026

Sparse Transformer: a Controlled Testbed for Mechanistic Interpretability

Mechanistic interpretability aims to decompose neural network computations into human-understandable features, yet progress is limited by the absence of ground-truth units to evaluate against. We address this with a Sparse Transformer (ST): a 5-layer language model trained on TinyStories whose wide, TopK-sparse MLP forces neuron specialization. We show that ST neurons are highly interpretable: they outperform a loss-matched dense transformer by 43% on AutoInterp, and circuits of a median of 12 neurons suffice to reproduce individual model predictions. This testbed allows us to directly compare feature learning methods: Transcoder, which approximates the MLP input-output mapping, recovers ST's neuron vectors with 52.6% higher cosine similarity than Sparse Autoencoder, which only reconstructs outputs. Furthermore, the ST enables a controlled study of polysemanticity: distilling its sparse MLP into a dense counterpart reveals how compression leads to feature superposition. These results show how a known reference basis sharpens evaluation and understanding, a role we hope the ST can serve for the field.

Anton Korznikov, Alexey Dontsov, Elena Tutubalina

Mechanistic Interpretability Neuron-Level Interpretability Transformer Sparse Autoencoders Feature Superposition

Ask PDF

Feb 18, 2026

Explainable AI: Context-Aware Layer-Wise Integrated Gradients for Explaining Transformer Models

Transformer models achieve state-of-the-art performance across domains and tasks, yet their deeply layered representations make their predictions difficult to interpret. Existing explainability methods rely on final-layer attributions, capture either local token-level attributions or global attention patterns without unification, and lack context-awareness of inter-token dependencies and structural components. They also fail to capture how relevance evolves across layers and how structural components shape decision-making. To address these limitations, we proposed the Context-Aware Layer-wise Integrated Gradients (CA-LIG) Framework, a unified hierarchical attribution framework that computes layer-wise Integrated Gradients within each Transformer block and fuses these token-level attributions with class-specific attention gradients. This integration yields signed, context-sensitive attribution maps that capture supportive and opposing evidence while tracing the hierarchical flow of relevance through the Transformer layers. We evaluate the CA-LIG Framework across diverse tasks, domains, and transformer model families, including sentiment analysis and long and multi-class document classification with BERT, hate speech detection in a low-resource language setting with XLM-R and AfroLM, and image classification with Masked Autoencoder vision Transformer model. Across all tasks and architectures, CA-LIG provides more faithful attributions, shows stronger sensitivity to contextual dependencies, and produces clearer, more semantically coherent visualizations than established explainability methods. These results indicate that CA-LIG provides a more comprehensive, context-aware, and reliable explanation of Transformer decision-making, advancing both the practical interpretability and conceptual understanding of deep neural models.

Melkamu Abay Mersha, Jugal Kalita

College of Engineering and Applied Science, University of Colorado Colorado Springs, Colorado Springs, 80918, CO, USA

Transformer Interpretability Gradient-Based Attribution Integrated Gradients Explainability Evaluation

Ask PDF

Abstract

Neuron-Level Interpretability

Explore similar work

RACE: Scalable Statistical Estimation of Functional Consistency in LLM Neurons

Sparse Transformer: a Controlled Testbed for Mechanistic Interpretability

Explainable AI: Context-Aware Layer-Wise Integrated Gradients for Explaining Transformer Models