Activation Probes Are Reliable With a Handful of Positive Examples

Riya Tyagi; Stefan Heimersheim

cs.LGOct 29, 2026

Activation Probes Are Reliable With a Handful of Positive Examples

Authors: Riya Tyagi, Stefan Heimersheim

Abstract

Efforts to monitor advanced AI for rare misalignments face a data challenge: abundant aligned examples but only a handful of misaligned ones. We test activation probes in this "few vs. thousands" regime on spam and honesty detection tasks. For our tasks, training with many negative examples is on average more positive-sample-efficient than balanced training for small numbers (1-10) of positive samples. We also find that LLM upsampling can provide a performance boost equivalent to roughly doubling the number of real positive samples, though excessive upsampling hurts performance. Finally, we show a positive scaling trend, where larger models are more positive-sample-efficient to probe. Our findings suggest we should leverage the large number of negative samples available to amplify the signal from rare but critical misalignment examples.

Figures & tables

Probe Type	Score Function
Linear Probe	$\theta^{T}\mathrm{Agg}(A)$
SAE Probe	$\theta^{T}\mathrm{Agg}(Z_{k})$
Cosine Similarity Probe	$\cos(\mathrm{Agg}(A),\bar{a}^{+})-\cos(\mathrm{Agg}(A),\bar{a}^{-})$
Attention Probe	$\text{softmax}(A\theta_{q})^{T}A\theta_{v}$

Table 1: Functional forms of probe types. All non-attention probes first apply an aggregation function

\mathrm{Agg}(\cdot)

to map a sequence of vectors to a single vector. We test four aggregations: Mean (

\frac{1}{S}\sum a_{s}

), Max (

\max_{s}A

), Last Token (

a_{S}

), and Softmax (

\sum_{s}w_{s}a_{s}

, a weighted sum where weights are derived from a softmax over the sequence dimension of

A

Figure 1: Probe AUC on Enron-Spam across the number of positive examples on (left) an imbalanced dataset with 1750 negative samples and (right) a balanced dataset.

Figure 2: Probe AUC on the MASK honesty benchmark across the number of positive examples for (left) an imbalanced dataset with 340 negative samples and (right) a balanced dataset.

Panel of Figure 3 — Figure 3: Effect of LLM upsampling on probe performance: (a) AUC and (b) recall at FPR=0.01 (b). On average across 13 probe architectures and 10 seeds, 3x upsampling performs best.

Panel of Figure 4 — Figure 4: Probes on larger models are more positive-sample-efficient, shown across (a) AUC and (b) Recall.

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Probe Type	Main Results Hyperparameters
Linear Probe	penalty : ‘l2‘
	solver : ‘liblinear‘
	max_iter : 1000
	class_weight : ‘balanced‘
	C : 1.0
Attention Probe	Epochs : 100

Appendix

Table 2: Fixed hyperparameters used for main results. Following [ 8 ] , we use L1 penalty for the SAE probes to encourage sparsity, based on the hypothesis that only a small number of SAE features are relevant for a given concept. We fix top_k_features=1024 because prior work on 100+ datasets found that while larger k-values perform better, performance improvements begin to taper off around

2^{9}

features [ 8 ] . We choose

2^{10}

to give the SAE probe a strong baseline.

Model	SAE/Transcoder Release	Layer	Width	L0
Gemma-2-9b	gemma-scope-9b-pt-res	20	16384	408
Llama-3.3-70B-Instruct	Goodfire/Llama-3.3-70B-Instruct-SAE-l50	50	-	121
Qwen-3-0.6B	mwhanna/qwen3-0.6b-transcoders-lowl0	15	163840	-
Qwen-3-1.7B	mwhanna/qwen3-1.7b-transcoders-lowl0	15	163840	-
Qwen-3-4B	mwhanna/qwen3-4b-transcoders	18	163840	-
Qwen-3-8B	mwhanna/qwen3-8b-transcoders	18	163840	-

Appendix

Table 3: Models and sparse dictionary learning methods used, and their widths and final L0 values on the SAE training sets. For the Llama-3.3-70B-Instruct release, the width was not available, and L0 values were not available for the Qwen models. For Gemma-2-9b, we also tested a larger 262k-width SAE with a lower L0 (259), which performed worse. This aligns with findings from [ 8 ] , which observed a small negative relationship between SAE width and probe performance, but a significant positive relationship for higher L0 scores. For the Qwen models, SAEs are not publicly available, so we use transcoders. For Qwen-3-32B, we were unable to find open-source SAEs or transcoders.

Probe Type	Parameter	Sweep Range
Linear Probe	C (for L2 reg.)	$\{10^{-3},10^{-2},...,10^{4}\}$
Attention Probe	Learning Rate	logspace( $10^{-5},10^{-2}$ , $6$ )
	Weight Decay (L2 reg.)	$\{10^{-6},10^{-5},...,10^{-2}\}$
SAE Probe	C (for L1 reg.)	$\{10^{-3},10^{-2},...,10^{4}\}$

Appendix

Table 4: Hyperparameter search space for sweeps. For probes using scikit-learn’s ‘LogisticRegression‘ (Linear and SAE), we swept the regularization parameter C . For the PyTorch-based Attention probe, an internal hyperparameter search was performed over learning rate and weight decay for each training run.

Panel of Figure 5 — Figure 5: AUC performance for Linear probes across the sweep of the regularization parameter ‘C‘ on the Enron-Spam dataset. The corresponding recall plots showed a very similar trend and are omitted for brevity.

Figure 6: AUC performance for Attention probes across the weight decay sweep on the Enron-Spam dataset (Imbalanced training).

Figure 7: AUC performance for Attention probes across the weight decay sweep on the Enron-Spam dataset (Balanced training).

Figure 8: Cross-validated AUC performance on the Enron-Spam dataset, comparing imbalanced and balanced training.

Figure 9: Cross-validated Recall @ 0.01 FPR performance on the Enron-Spam dataset.

Figure 10: Cross-validated AUC performance on the MASK honesty benchmark.

Figure 11: Cross-validated Recall @ 0.01 FPR performance on the MASK honesty benchmark.

Figure 12: Recall performance of the best probes on the non-OOD spam dataset.

Figure 13: Per-probe AUC on the non-OOD spam dataset (Experiment 2).

Figure 14: Per-probe Recall @ 0.01 FPR on the non-OOD spam dataset (Experiment 2).

Figure 15: Per-probe AUC on the non-OOD spam dataset (Experiment 4).

Figure 16: Per-probe Recall @ 0.01 FPR on the non-OOD spam dataset (Experiment 4).

Figure 17: AUC performance of best probes on the SMS spam dataset.

Figure 18: Recall @ 0.01 FPR of best probes on the SMS spam dataset.

Figure 19: Per-probe AUC on the SMS spam generalization task.

Figure 20: Per-probe Recall @ 0.01 FPR on the SMS spam generalization task.

Figure 21: Recall performance of the best probes on the MASK honesty dataset.

Figure 22: Detailed per-probe recall performance for Experiment 4 on the MASK dataset.

Figure 23: Per-probe AUC with LLM upsampling on the Enron-Spam dataset.

Figure 24: Per-probe Recall @ 0.01 FPR with LLM upsampling on the Enron-Spam dataset.

Panel of Figure 25 — Figure 25: Heatmaps showing performance across upsampling factors and number of real positive examples on the Enron-Spam dataset.

Panel of Figure 26 — Figure 26: Aggregated performance with LLM upsampling on the generalization SMS Spam dataset.

Panel of Figure 27 — Figure 27: Heatmaps showing generalization performance across upsampling factors and number of real positive examples.

Panel of Figure 28 — Figure 28: Heatmaps showing performance across model sizes and number of positive examples. Note that results for Qwen-32B are omitted for SAE probes as no open-source transcoders/SAEs were available.

Panel of Figure 29 — Figure 29: Aggregated performance across Qwen-3 model sizes on the generalization SMS Spam dataset.

Panel of Figure 30 — Figure 30: Heatmaps showing generalization performance across model sizes and number of positive examples.

LLM Safety Monitoring

See all 56 papers

Same Probe, Different Numbers: Are Activation Probes Robust to Inference-Time Numerical Non-Determinism?
Sep 25, 2026Alizishaan KhatriLanguage Model Probing Language Model Robustness
Wrynx Inc.
Why Safety Probes Catch Liars But Miss Fanatics
Mar 26, 2026Kristiyan HaralambievDeception in Language Models Emergent Misalignment in Language Models
* Independent Research.
Detection Without Correction: A Robust Asymmetry in Activation-Based Hallucination Probing
Mar 20, 2026Dip Roy, Rajiv Misra, Sanjay Kumar Singh +1LLM Hallucination Detection Hidden-State Probing
Department of Computer Science and Engineering, Indian Institute of Technology Patna, India · Department of Computer Science, Rajarshi School of Management and Technology, Varanasi, India · Department of Electronics and Communication Engineering, Jaypee Institute of Information Technology, Noida, India

Sep 25, 2026J/K move · Enter open · S save

Same Probe, Different Numbers: Are Activation Probes Robust to Inference-Time Numerical Non-Determinism?

Alizishaan Khatri

Wrynx Inc.

Language Model Probing Language Model Robustness LLM Auditing

Open paper AskPDF arXiv

Activation probes are increasingly used to monitor LLMs in deployment. A probe is typically trained under one inference configuration, then used under whatever batch size and numerical precision the serving stack uses. Because common GPU kernels are not batch-invariant and floating-point formats round differently, the activations seen at deployment are not the ones the probe was trained on. We measure what that costs for Llama-3.1-8B, Qwen3-8B and Gemma-3-4B across batch sizes 4, 8 and 16 and float32, bfloat16 and float16, training 768 probes on one configuration, evaluating each on every other, and comparing verdicts example by example. Probes are stable, but aggregate accuracy is the wrong instrument for showing it: it understates how many verdicts change by a factor of two to nine. At the prompt, accuracy never moves by more than 0.47 percentage points across 1,392 transfers and only 0.076% of verdicts change; under float32 with only the batch size varied, none of 201,960 verdicts change. During decoding the flip rate rises to 2.8%, but rows whose realised tokens matched flip in only 0.12-0.15% of cases, while rows whose tokens diverged flip in 12.9%: the cause is the text, not the arithmetic. A bfloat16 batch-size change flips the first generated token for 2.1% of rows and leaves 25% on different tokens by token 20. Flips are symmetric, Cohen's kappa stays above 0.94, and AUROC moves by at most 0.05 points. Underneath, activations move about as much as the format's rounding: a bfloat16 batch-size change perturbs them by a median relative L2 of 1e-2, roughly 8x the float16 figure. Probes absorb this; the model's own next-token argmax does not. Robustness evaluations of activation monitors should report per-example agreement rather than aggregate accuracy, separate representational noise from input change, and state the serving configuration.

Abstract

LLM Safety Monitoring

Explore similar work

Same Probe, Different Numbers: Are Activation Probes Robust to Inference-Time Numerical Non-Determinism?

Why Safety Probes Catch Liars But Miss Fanatics

Detection Without Correction: A Robust Asymmetry in Activation-Based Hallucination Probing