A Mechanistic Study of AI-Text Detection Neurons in Frozen BERT: Sparse Probing and Activation Patching on RAID

Paweł Blicharz; Miłosz Grunwald

cs.CLSep 8, 2026

A Mechanistic Study of AI-Text Detection Neurons in Frozen BERT: Sparse Probing and Activation Patching on RAID

Authors: Paweł Blicharz, Miłosz Grunwald

Organizations: Gradient PG Gda´nsk University of Technology

Abstract

AI-generated text detectors achieve high accuracy on standard benchmarks, yet the internal representations that drive these predictions remain poorly understood. We study which neurons in a frozen BERT-base-uncased encoder support AI-text detection, using the RAID benchmark across six generators spanning pure-base and instruction-tuned models. We apply the L1-to-L2 sparse-probing protocol of Gurnee et al. (2023) to all 9,216 CLS hidden-state dimensions (12 layers x 768), which we call neurons. The procedure recovers a stable set of under 1% of neurons per generator, consistent across folds and seeds; a probe restricted to that set retains most of the full-feature detection accuracy. Bidirectional activation patching confirms this set's causal relevance: in both directions it flips predictions an order of magnitude more often than size-matched random sets. Mean-ablating the same neurons leaves accuracy largely intact; the signal is therefore redundantly distributed. Cross-generator analysis reveals a bipartite structure: instruction-tuned generators concentrate 30-36% of stable neurons in BERT's final layer while both base generators fall below 14%, consistent with a layer-12 footprint of post-training alignment. Leave-one-family-out evaluation shows the selected neurons retain 86-94% of the full-feature ceiling on unseen generator families, so a detector can operate on a small fixed subspace without re-identifying neurons per generator.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of the experimental pipeline. Frozen BERT-base-uncased CLS activations are concatenated across all 12 layers, yielding $\mathbf{z}\in\mathbb{R}^{9{,}216}$ . An L1-regularised probe selects a candidate set per (fold, seed) cell; neurons stable in ${\geq}80\%$ of all 15 cells become the stable set $\mathcal{S}^{*}$ . An independent L2 probe trained on all 9,216 neurons is held frozen as the decision function for both experiments. Left branch (§ 5.2 ): mean ablation. Right branch (§ 5.1 ): bidirectional activation patching, forward (AI $\to$ human) and reverse (human $\to$ AI). Numerical results are reported in the cited sections.

Generator	Val acc	AUC	$n_{\text{sel}}$	Jaccard	$n_{\text{stable}}$ / $n_{\text{obs}}$
GPT-4	$0.988\pm 0.003$	$0.999\pm 0.000$	$74.1\pm 2.9$	$0.763\pm 0.037$	60 / 106
GPT-2	$0.959\pm 0.003$	$0.992\pm 0.001$	$84.7\pm 4.1$	$0.670\pm 0.042$	60 / 146
MPT	$0.972\pm 0.003$	$0.994\pm 0.001$	$61.3\pm 3.1$	$0.740\pm 0.036$	52 / 94
Mistral	$0.971\pm 0.004$	$0.995\pm 0.001$	$57.8\pm 2.9$	$0.724\pm 0.046$	45 / 99
LLaMA	$0.986\pm 0.003$	$0.998\pm 0.001$	$63.6\pm 1.7$	$0.756\pm 0.040$	48 / 91
Cohere	$0.905\pm 0.008$	$0.968\pm 0.003$	$72.3\pm 3.1$	$0.749\pm 0.039$	62 / 113

Table 1: Sparse-probe results across six generators: 15-cell means

\pm

std (5 folds

\times

3 seeds). Val acc: L2 evaluation probe accuracy (all 9,216 neurons). AUC: area under the ROC curve.

n_{\text{sel}}

: mean candidate set size per cell. Jaccard: mean pairwise fold Jaccard over all 105 fold-pairs (C(15,2)), computed from per-fold selection sets.

n_{\text{stable}}

: neurons in

\geq

80% of cells;

n_{\text{obs}}

: total distinct neurons selected at least once. The six generators span one model family each and all post-training regimes present in RAID (two pure-base, one SFT-only, three SFT+RLHF); the remaining five RAID generators are same-family, same-regime duplicates (§ 3.1 ).

Generator	Selected (%)	Random- $k$ (%)	Ratio
GPT-4	$1.23_{\pm 0.33}$	$0.08_{\pm 0.07}$	$16\times$
GPT-2	$3.13_{\pm 0.54}$	$0.32_{\pm 0.09}$	$10\times$
MPT	$1.52_{\pm 0.21}$	$0.14_{\pm 0.08}$	$11\times$
Mistral	$1.73_{\pm 0.27}$	$0.12_{\pm 0.08}$	$14\times$
LLaMA	$1.07_{\pm 0.37}$	$0.11_{\pm 0.07}$	$10\times$
Cohere	$8.15_{\pm 1.27}$	$0.79_{\pm 0.16}$	$10\times$

Table 2: Forward-direction (AI

\to

human) selected and random-

k

flip rates (%) at

k=\text{full}

(15-cell means

\pm

std). Ratio = selected / random (rounded to nearest integer; precise values 9.7–15.7

\times

). Full

k

-sweep in Appendix J . Reverse-direction (human

\to

AI) results are reported in the text.

Figure 2: Forward flip rate (%) vs. number of patched neurons

k

on a log-scale x-axis, averaged over 15 cells (5 folds

\times

3 seeds). Instruction-tuned generators are shown in blue, base generators in red. Numbers to the right of each line are selected/random ratios at

k=50

(8–14

\times

); headline ratios at

k=\text{full}

(the per-fold L1-selected set, 53–92 neurons) are 10–16

\times

across generators (Table 13 ).

Generator	Base acc	Sel. drop	Rand. drop
GPT-4	0.988	$+0.03\pm 0.12$ pp	$+0.01$ pp
GPT-2	0.959	$+0.11\pm 0.31$ pp	$+0.00$ pp
MPT	0.972	$+0.05\pm 0.11$ pp	$+0.00$ pp
Mistral	0.971	$+0.10\pm 0.21$ pp	$+0.00$ pp
LLaMA	0.986	$+0.11\pm 0.14$ pp	$+0.02$ pp
Cohere	0.905	$\mathbf{+1.06\pm 0.52}$ pp	$+0.06$ pp

Table 3: Mean-ablation accuracy drop at

k=\text{full}

, 15-cell means

\pm

std (positive = drop). Sel. drop: ablating the per-fold L1-selected set (

k=\text{full}

, 53–92 neurons); Rand. drop: mean over 20 random

k

-subsets. Cohere is the only generator substantially above the random baseline.

Figure 3: Proportion of stable neurons in each layer group per generator. Layer 12 percentage annotated in white. Instruction-tuned generators (IT) concentrate 30–36% of stable neurons in layer 12; base generators retain

\leq

14%. Dashed line separates the two groups. The generators shown are one per RAID model family, selected to place both pure-base and both instruction-tuned post-training regimes on this axis; see § 3.1 for the selection and exclusion criteria.

Figure 4: Leave-one-family-out generalisation. For each held-out RAID family, the full-feature L2 probe (all 9,216 neurons) and the sparse L2 probe (only the 91–125 neurons selected from the four training families) are evaluated on the held-out family. Values above each bar are accuracies; the percentage inside each sparse bar is its retention relative to that family’s own full-feature ceiling. The sparse probe never sees the held-out family during selection or fitting. Numerical values, including AUC, in Table 4 .

Family	$n_{\text{stable}}$	Full L2	Sel L2	Sel/Full	Full AUC	Sel AUC
cohere	119	88.6%	76.3%	86%	0.946	0.863
gpt	91	98.2%	88.8%	90%	0.993	0.953
llama	125	99.7%	93.7%	94%	0.999	0.986
mistral	106	96.6%	85.6%	89%	0.982	0.925
mpt	110	99.1%	85.7%	87%	0.995	0.926

Table 4: Leave-one-family-out (LOGO) generalisation.

n_{\text{stable}}

: stable neurons identified from the four training families. Full L2: L2-probe accuracy using all 9,216 BERT neurons. Sel L2: L2-probe accuracy restricted to

n_{\text{stable}}

neurons. Sel/Full: ratio expressing how much of the full-feature ceiling the selected neurons retain. Full AUC: area under the ROC curve (AUC) for the full-feature L2 probe. Sel AUC: AUC-ROC for the Sel L2 probe, ranging from 0.863 (cohere) to 0.986 (llama). All metrics are means over 3 seeds

\times

5 folds = 15 cells. Test set sizes: 5,000 (llama, 1 generator) to 20,000 (gpt, 4 generators).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

GPT-4
$C$	$N=500$	$N=1000$	$N=2000$	$N=3500$	$N=5000$	$N=7500$
0.001	$1.000^{\dagger}$ (0)	$1.000^{\dagger}$ (0)	$1.000^{\dagger}$ (0)	0.867 (0)	0.689 (8)	0.808 (18)
0.002	$1.000^{\dagger}$ (0)	$1.000^{\dagger}$ (0)	0.680 (3)	0.664 (16)	0.692 (28)	0.799 (44)
0.005	$1.000^{\dagger}$ (0)	0.433 (9)	0.510 (29)	0.617 (51)	0.684 (65)	0.751 (80)
0.01	0.314 (9)	0.356 (29)	0.488 (54)	0.553 (78)	0.598 (97)	0.681 (123)
0.02	0.254 (29)	0.355 (53)	0.425 (82)	0.460 (117)	0.509 (143)	0.625 (179)

Appendix

Table 5: Mean Jaccard stability (and mean

\bar{n}

selected) across 15 random draws of the L1 selector. Rows are

C

values; columns are

N

values. Larger Jaccard is better. Bold : chosen operating point (

C=0.005

N=7{,}500

). Dagger (

\dagger

): degenerate cell (

\bar{n}<1

, Jaccard trivially 1.0 or unstable).

N=10{,}000

omitted (only

K=1

draw possible; pool exhausted).

Gen.	Seed	Fold	Acc	AUC	$n_{\text{sel}}$
GPT-4	42	0	0.9887	0.9991	74
	42	1	0.9853	0.9992	73
	42	2	0.9887	0.9991	80
	42	3	0.9853	0.9989	74
	42	4	0.9927	0.9989	70
	123	0	0.9860	0.9987	70

Appendix

Table 6: Per-cell L2-probe metrics for all six generators. Each block of 15 rows is one generator; columns are seed and fold index. Acc: accuracy; AUC: AUC-ROC;

n_{\text{sel}}

: neurons selected by L1 in this cell.

Gen.	$n_{\text{AUC}}$	$n_{\text{L1}}$	Jaccard	AUC drop	L1 drop
GPT-4	491	74	0.127	+1.02pp	+0.03pp
GPT-2	86	85	0.096	+0.06pp	+0.11pp
MPT	784	61	0.054	+6.90pp	+0.05pp
Mistral	238	58	0.148	+0.52pp	+0.10pp
LLaMA	308	64	0.139	+0.55pp	+0.11pp
Cohere	38	72	0.185	+0.28pp	+1.06pp

Appendix

Table 7: L1 vs. AUC-ROC selection: mean across 15 cells.

n

: neurons selected. Jaccard: overlap between the two selection sets. Drop: accuracy drop from baseline when complement is ablated (positive = degradation). Three regimes are visible: parsimony (GPT-4, MPT, Mistral, LLaMA: L1 selects far fewer neurons with equivalent or smaller drop), convergence (GPT-2, similar

n

and similar drop), and reversal (Cohere: AUC-ROC selects fewer with smaller drop).

Generator	Smaller model	Gap vs ablate-complement
GPT-4	$0.972\pm 0.005$	$+0.41\pm 0.42$ pp
GPT-2	$0.913\pm 0.006$	$+3.63\pm 0.81$ pp
MPT	$0.944\pm 0.005$	$+4.68\pm 1.14$ pp
Mistral	$0.934\pm 0.005$	$+0.93\pm 0.42$ pp
LLaMA	$0.960\pm 0.003$	$+1.14\pm 0.43$ pp
Cohere	$0.865\pm 0.009$	$+4.43\pm 1.02$ pp

Appendix

Table 8: Restricted probe results. Smaller model : L2 probe retrained on only the

|\mathcal{S}^{*}|

stable neurons. Gap : smaller-model accuracy minus ablate-complement accuracy (positive = stable neurons self-sufficient; larger = more dependence on non-stable neurons in the full-feature probe). 15-cell means

\pm

std.

Generator	Cosine	LR acc	$\\|\Delta\mu\\|$	$\\|w\\|$
GPT-4	$0.138\pm 0.004$	$0.988\pm 0.003$	7.34	$\mathbf{45.8}\pm 1.7$
GPT-2	$0.073\pm 0.002$	$0.963\pm 0.006$	6.84	$69.9\pm 2.9$
MPT	$0.065\pm 0.000$	$0.973\pm 0.001$	$\mathbf{9.52}$	$53.8\pm 2.7$
Mistral	$0.110\pm 0.001$	$0.965\pm 0.003$	6.38	$59.2\pm 2.3$
LLaMA	$0.128\pm 0.001$	$0.987\pm 0.003$	6.57	$48.3\pm 0.9$
Cohere	$\mathbf{0.057}\pm 0.001$	$\mathbf{0.910}\pm 0.008$	4.69	$108.3\pm 2.1$

Appendix

Table 9: CAV diagnostics on per-generator stable sets. All values mean

\pm

std across 3 random seeds. Cosine: similarity between mean-difference CAV and L1 probe weight. LR acc: logistic-regression accuracy on the CAV projection.

\|\Delta\mu\|

: AI

-

human mean-difference norm (deterministic; bold = largest).

\|w\|

: L1 weight norm (bold = smallest).

Domain	GPT-4	GPT-2	MPT	Mistral	LLaMA	Cohere
Abstracts	1.44	3.08	1.39	1.79	1.47	9.89
Books	1.08	1.93	1.47	0.92	1.21	10.06
News	1.61	4.92	2.74	2.11	1.01	9.67
Reddit	1.05	3.22	1.36	1.34	0.76	7.50
Reviews	0.97	1.96	0.55	0.62	1.14	5.48
Wiki	1.25	3.65	1.61	3.54	0.83	6.26

Appendix

Table 10: Per-domain selected flip rate (%) at

k=\text{full}

, 15-cell means. Domain means may differ slightly from the overall mean due to variation in the number of valid-human pairs per domain. GPT-2 shows elevated flip rates on news, Reddit, and wiki (3.1–4.9%) relative to books and reviews (1.9–2.0%). Cohere is uniformly high across all six domains (5.5–10.1%); its probe operates near the decision boundary throughout. Mistral shows a notable wiki spike (3.54%) while reviews are weak (0.62%). CV: coefficient of variation over each generator’s six domain flip rates. See § 5.1.2 for interpretation.

Layer	GPT-4	GPT-2	MPT	Mistral	LLaMA	Cohere
1–10	36	41	44	24	30	40
11	6	11	6	5	2	3
12	18	8	2	16	16	19
% in L12	30.0	13.3	3.8	35.6	33.3	30.6

Appendix

Table 11: Stable-neuron counts by layer group across six generators. Bold indicates the peak layer group for each generator. Instruction-tuned generators (GPT-4, Mistral, LLaMA, Cohere) concentrate 30–36% of stable neurons in layer 12; both pure-base generators (GPT-2, MPT) have

\leq

14% in layer 12.

Figure 5: Pairwise Jaccard similarity heatmap (6 $\times$ 6). Generators reordered: instruction-tuned (IT) first, base last. Dashed blue box: IT $\times$ IT block. Dashed red box: base $\times$ base block. The bipartite structure (higher within-group similarity, near-zero cross-group overlap) is immediately visible.

Pair	Jaccard	$\times$ chance
Cohere – Mistral	0.244	86 $\times$
LLaMA – Mistral	0.163	64 $\times$
GPT-4 – Mistral	0.141	51 $\times$
GPT-4 – LLaMA	0.113	39 $\times$
MPT – GPT-2	0.120	40 $\times$
Cohere – GPT-4	0.109	33 $\times$

Appendix

Table 12: Pairwise Jaccard similarity between stable sets (15 generator pairs).

\times

chance is observed Jaccard divided by the analytic random-null Jaccard for two sets of the same sizes drawn from 9,216 neurons (

\approx

0.003 per pair). The mpt–gpt2 pair (bold) is the highest base

\times

base overlap; all base

\times

instruction-tuned pairs cluster at the bottom.

Forward direction (AI donor $\to$ human target) — selected $k$ -sweep
$k$	GPT-4	GPT-2	MPT	Mistral	LLaMA	Cohere
1	0.02	0.05	0.04	0.06	0.07	0.13
5	0.09	0.17	0.14	0.20	0.14	0.58
10	0.15	0.29	0.25	0.34	0.27	0.99
20	0.35	0.58	0.47	0.56	0.49	2.02
50	0.80	1.71	1.13	1.48	0.86	5.43

Appendix

Table 13: Flip rate (%) for both patching directions, 15-cell means (5 folds

\times

3 seeds).

k=\text{full}

equals each cell’s actual

|\mathcal{S}|

(53–92 neurons); std shown only for selected

k=\text{full}

. Forward direction (AI donor

\to

human target): selected flip rate is monotonically increasing in

k

for every generator. Selected/random ratios at

k=\text{full}

are 10–16

\times

(precise values 9.7–15.7

\times

; see Table 2 ). Reverse direction (human donor

\to

AI target) is likewise monotonically increasing in

k

for every generator; selected/random ratios at

k=\text{full}

are 7–12

\times

, and absolute rates are below the forward direction at every

k

. See § 5.1.2 for the partial-redundancy reading.

Generator	Feature	$\rho$	$\rho^{2}$	Neuron
GPT-4	word length	$+0.405$	0.164	L12/308
GPT-2	word length	$+0.537$	0.288	L1/661
MPT	type–token	$-0.478$	0.229	L1/188
Mistral	word length	$-0.510$	0.261	L5/531
LLaMA	word length	$+0.441$	0.195	L12/308
Cohere	word length	$-0.454$	0.207	L5/531

Appendix

Table 14: Strongest surviving neuron–feature association per generator, over the ten highest-weight stable neurons

\times

four surface features (40 Spearman tests per generator, Benjamini–Hochberg at

q=0.05

; 31–37 tests survive per generator).

\rho^{2}

is the approximate share of variance explained. “word length” is mean word length, “type–token” the type–token ratio. Neuron is given as layer/unit. All associations are moderate and none approaches full localisation of the detection signal to a surface cue.

AI-Generated Text Detection

AI-generated text detection develops methods to distinguish human-written, machine-generated, and AI-edited text.

See all 171 papers

May 20, 2026

Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction

AI text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary. On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures (86-106% of the fine-tuned discrimination ceiling: on RoBERTa-base, raw projection exceeds fine-tuning); on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested. The same axis inverts on non-native ESL writing (AUROC 0.06-0.20) -- a falsifiable prediction unique to the typicality reading. A 24-example frozen probe matches full fine-tuning (0.900 vs 0.895). A closed-form Jacobian predictor parameterises axis-manipulating interventions with R^2 = 1.000 universal, lifts ELECTRA-CE deployment TPR from 0.000 to 0.904 at FPR = 1%, and transfers to three independently-trained third-party RoBERTa detectors at 16/16 oracle-equivalence (57% NYT-FPR reduction on the OpenAI detector). Scope: encoder family; mechanism magnitude HC3-anchored; population-level shared axis with per-text mechanisms varying across architectures. Three operationally distinct probes -- text-surface caps_rate residualisation, geometric signed-epsilon ablation, closed-form text-pair predictor -- agree at cos 0.74/0.81/1.00 across three architectures, confirming observer-invariance. Under matched-TPR-0.90 evaluation, the published intervention zoo (CC, dealign-f2c) is calibration-equivalent across 27 cells (|Delta AUROC| <= 0.0081), and >= 97% of the LoRA->full-FT bias gap on ELECTRA is calibration shift, not learned representation -- the central claim's prediction confirmed.

Alexander Smirnov

University College London

AI-Generated Text Detection Classification Language Model Auditing

Ask PDF

Oct 21, 2026

Building Auditable AI Text Detectors with Sparse Autoencoders

AI-generated text detectors are increasingly deployed in consequential settings---peer review, education, admissions, journalism, and content moderation---where their outputs directly affect individuals. Yet the strongest detectors are fine-tuned neural classifiers whose decisions are opaque, while transparent alternatives (stylometric and linguistic-feature classifiers) underperform them by wide margins. We present a pipeline that converts a fine-tuned neural detector into a classifier that operates on human-readable concepts. We accomplish this by training sparse autoencoders (SAEs) directly on a detector's residual stream, assigning auto-interpretation labels to each SAE feature, and retaining only features whose generated descriptions pass a global fidelity threshold. We then train a lightweight classifier on just the interpretable features. We find that our SAEs recover latent concepts that correspond to highly specific and predictive linguistic "tells" of AI-generated text. Classifiers built on these extracted features are not only highly interpretable but exhibit stronger performance on out-of-distribution texts than either neural or stylometric alternatives.

Liam Dugan, Callum McDougall, Matthew Denton +3

University of Pennsylvania · Google DeepMind

AI-Generated Text Detection Explainable Text Classification Robustness of Machine-Generated Text Detection Sparse Autoencoders

Ask PDF

Jul 4, 2026

Rethinking AI-Generated Text Detection: A Strong Baseline and the Distribution-Shift Problem That Remains

Recent AI-generated text detection work often introduces a new benchmark together with a specialized detector tailored to it. We revisit this practice from a baseline-first perspective. Across several benchmarks, we show that a plain, fully fine-tuned RoBERTa matches or exceeds the specialized detectors those benchmarks are built around. This suggests that much of the recent architectural complexity is not what drives strong in-distribution detection. The remaining challenge is the distribution shift. The same strong baseline degrades sharply when the topic domain or generating model changes at test time, and simply adding more source data does not close the gap. We identify a key failure mode: under distribution shift, the detector can assign high-confidence machine labels to human-written text from unseen domains. We then study two lightweight domain adaptation methods to address this problem:

K

-shot adaptation with first-order MAML over LoRA adapters, and a per-sample confidence-weighted ensemble built on top of the adapted detector. Overall, our results suggest that progress in AI-generated text detection should be measured not only by in-distribution performance, but also by robustness under distribution shift.

Zhuoer Shen, Mingyi Wang, Shaofeng Zou +1

Department of Computer Science, University of California, Santa Barbara · School of ECEE, Arizona State University

Distribution Shift Robustness AI-Generated Text Detection Domain Adaptation Distribution Shift Detection Robustness of Machine-Generated Text Detection

Ask PDF

Abstract

AI-Generated Text Detection

Explore similar work

Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction

Building Auditable AI Text Detectors with Sparse Autoencoders

Rethinking AI-Generated Text Detection: A Strong Baseline and the Distribution-Shift Problem That Remains