cs.LGOct 29, 2026

Activation Probes Are Reliable With a Handful of Positive Examples

Authors: Riya Tyagi, Stefan Heimersheim

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Efforts to monitor advanced AI for rare misalignments face a data challenge: abundant aligned examples but only a handful of misaligned ones. We test activation probes in this "few vs. thousands" regime on spam and honesty detection tasks. For our tasks, training with many negative examples is on average more positive-sample-efficient than balanced training for small numbers (1-10) of positive samples. We also find that LLM upsampling can provide a performance boost equivalent to roughly doubling the number of real positive samples, though excessive upsampling hurts performance. Finally, we show a positive scaling trend, where larger models are more positive-sample-efficient to probe. Our findings suggest we should leverage the large number of negative samples available to amplify the signal from rare but critical misalignment examples.

Figures & tables

Probe TypeScore Function
Linear ProbeθTAgg(A)\theta^{T}\mathrm{Agg}(A)
SAE ProbeθTAgg(Zk)\theta^{T}\mathrm{Agg}(Z_{k})
Cosine Similarity Probecos⁡(Agg(A),aˉ+)−cos⁡(Agg(A),aˉ−)\cos(\mathrm{Agg}(A),\bar{a}^{+})-\cos(\mathrm{Agg}(A),\bar{a}^{-})
Attention Probesoftmax(Aθq)TAθv\text{softmax}(A\theta_{q})^{T}A\theta_{v}
Table 1: Functional forms of probe types. All non-attention probes first apply an aggregation function Agg(⋅)\mathrm{Agg}(\cdot) to map a sequence of vectors to a single vector. We test four aggregations: Mean ( 1S∑as\frac{1}{S}\sum a_{s} ), Max ( max⁡sA\max_{s}A ), Last Token ( aSa_{S} ), and Softmax ( ∑swsas\sum_{s}w_{s}a_{s} , a weighted sum where weights are derived from a softmax over the sequence dimension of AA ).
Figure 1: Probe AUC on Enron-Spam across the number of positive examples on (left) an imbalanced dataset with 1750 negative samples and (right) a balanced dataset.
Figure 2: Probe AUC on the MASK honesty benchmark across the number of positive examples for (left) an imbalanced dataset with 340 negative samples and (right) a balanced dataset.
Figure 3: Effect of LLM upsampling on probe performance: (a) AUC and (b) recall at FPR=0.01 (b). On average across 13 probe architectures and 10 seeds, 3x upsampling performs best.
Figure 4: Probes on larger models are more positive-sample-efficient, shown across (a) AUC and (b) Recall.
Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Probe TypeMain Results Hyperparameters
Linear Probepenalty : ‘l2‘
 solver : ‘liblinear‘
 max_iter : 1000
 class_weight : ‘balanced‘
 C : 1.0
Attention ProbeEpochs : 100
Appendix
Table 2: Fixed hyperparameters used for main results. Following [ 8 ] , we use L1 penalty for the SAE probes to encourage sparsity, based on the hypothesis that only a small number of SAE features are relevant for a given concept. We fix top_k_features=1024 because prior work on 100+ datasets found that while larger k-values perform better, performance improvements begin to taper off around 292^{9} features [ 8 ] . We choose 2102^{10} to give the SAE probe a strong baseline.
ModelSAE/Transcoder ReleaseLayerWidthL0
Gemma-2-9bgemma-scope-9b-pt-res2016384408
Llama-3.3-70B-InstructGoodfire/Llama-3.3-70B-Instruct-SAE-l5050-121
Qwen-3-0.6Bmwhanna/qwen3-0.6b-transcoders-lowl015163840-
Qwen-3-1.7Bmwhanna/qwen3-1.7b-transcoders-lowl015163840-
Qwen-3-4Bmwhanna/qwen3-4b-transcoders18163840-
Qwen-3-8Bmwhanna/qwen3-8b-transcoders18163840-
Appendix
Table 3: Models and sparse dictionary learning methods used, and their widths and final L0 values on the SAE training sets. For the Llama-3.3-70B-Instruct release, the width was not available, and L0 values were not available for the Qwen models. For Gemma-2-9b, we also tested a larger 262k-width SAE with a lower L0 (259), which performed worse. This aligns with findings from [ 8 ] , which observed a small negative relationship between SAE width and probe performance, but a significant positive relationship for higher L0 scores. For the Qwen models, SAEs are not publicly available, so we use transcoders. For Qwen-3-32B, we were unable to find open-source SAEs or transcoders.
Probe TypeParameterSweep Range
Linear ProbeC (for L2 reg.){10−3,10−2,...,104}\{10^{-3},10^{-2},...,10^{4}\}
Attention ProbeLearning Ratelogspace( 10−5,10−210^{-5},10^{-2} , 66 )
 Weight Decay (L2 reg.){10−6,10−5,...,10−2}\{10^{-6},10^{-5},...,10^{-2}\}
SAE ProbeC (for L1 reg.){10−3,10−2,...,104}\{10^{-3},10^{-2},...,10^{4}\}
Appendix
Table 4: Hyperparameter search space for sweeps. For probes using scikit-learn’s ‘LogisticRegression‘ (Linear and SAE), we swept the regularization parameter C . For the PyTorch-based Attention probe, an internal hyperparameter search was performed over learning rate and weight decay for each training run.
Figure 5: AUC performance for Linear probes across the sweep of the regularization parameter ‘C‘ on the Enron-Spam dataset. The corresponding recall plots showed a very similar trend and are omitted for brevity.
Figure 6: AUC performance for Attention probes across the weight decay sweep on the Enron-Spam dataset (Imbalanced training).
Figure 7: AUC performance for Attention probes across the weight decay sweep on the Enron-Spam dataset (Balanced training).
Figure 8: Cross-validated AUC performance on the Enron-Spam dataset, comparing imbalanced and balanced training.
Figure 9: Cross-validated Recall @ 0.01 FPR performance on the Enron-Spam dataset.
Figure 10: Cross-validated AUC performance on the MASK honesty benchmark.
Figure 11: Cross-validated Recall @ 0.01 FPR performance on the MASK honesty benchmark.
Figure 12: Recall performance of the best probes on the non-OOD spam dataset.
Figure 13: Per-probe AUC on the non-OOD spam dataset (Experiment 2).
Figure 14: Per-probe Recall @ 0.01 FPR on the non-OOD spam dataset (Experiment 2).
Figure 15: Per-probe AUC on the non-OOD spam dataset (Experiment 4).
Figure 16: Per-probe Recall @ 0.01 FPR on the non-OOD spam dataset (Experiment 4).
Figure 17: AUC performance of best probes on the SMS spam dataset.
Figure 18: Recall @ 0.01 FPR of best probes on the SMS spam dataset.
Figure 19: Per-probe AUC on the SMS spam generalization task.
Figure 20: Per-probe Recall @ 0.01 FPR on the SMS spam generalization task.
Figure 21: Recall performance of the best probes on the MASK honesty dataset.
Figure 22: Detailed per-probe recall performance for Experiment 4 on the MASK dataset.
Figure 23: Per-probe AUC with LLM upsampling on the Enron-Spam dataset.
Figure 24: Per-probe Recall @ 0.01 FPR with LLM upsampling on the Enron-Spam dataset.
Figure 25: Heatmaps showing performance across upsampling factors and number of real positive examples on the Enron-Spam dataset.
Figure 26: Aggregated performance with LLM upsampling on the generalization SMS Spam dataset.
Figure 27: Heatmaps showing generalization performance across upsampling factors and number of real positive examples.
Figure 28: Heatmaps showing performance across model sizes and number of positive examples. Note that results for Qwen-32B are omitted for SAE probes as no open-source transcoders/SAEs were available.
Figure 29: Aggregated performance across Qwen-3 model sizes on the generalization SMS Spam dataset.
Figure 30: Heatmaps showing generalization performance across model sizes and number of positive examples.
More in this topic

LLM Safety Monitoring

See all 56 papers

Explore similar work

CardsList
Sep 25, 2026

Same Probe, Different Numbers: Are Activation Probes Robust to Inference-Time Numerical Non-Determinism?

Activation probes are increasingly used to monitor LLMs in deployment. A probe is typically trained under one inference configuration, then used under whatever batch size and numerical precision the serving stack uses. Because common GPU kernels are not batch-invariant and floating-point formats round differently, the activations seen at deployment are not the ones the probe was trained on. We measure what that costs for Llama-3.1-8B, Qwen3-8B and Gemma-3-4B across batch sizes 4, 8 and 16 and float32, bfloat16 and float16, training 768 probes on one configuration, evaluating each on every other, and comparing verdicts example by example. Probes are stable, but aggregate accuracy is the wrong instrument for showing it: it understates how many verdicts change by a factor of two to nine. At the prompt, accuracy never moves by more than 0.47 percentage points across 1,392 transfers and only 0.076% of verdicts change; under float32 with only the batch size varied, none of 201,960 verdicts change. During decoding the flip rate rises to 2.8%, but rows whose realised tokens matched flip in only 0.12-0.15% of cases, while rows whose tokens diverged flip in 12.9%: the cause is the text, not the arithmetic. A bfloat16 batch-size change flips the first generated token for 2.1% of rows and leaves 25% on different tokens by token 20. Flips are symmetric, Cohen's kappa stays above 0.94, and AUROC moves by at most 0.05 points. Underneath, activations move about as much as the format's rounding: a bfloat16 batch-size change perturbs them by a median relative L2 of 1e-2, roughly 8x the float16 figure. Probes absorb this; the model's own next-token argmax does not. Robustness evaluations of activation monitors should report per-example agreement rather than aggregate accuracy, separate representational noise from input change, and state the serving configuration.

Alizishaan Khatri

Wrynx Inc.

Language Model ProbingLanguage Model RobustnessLLM Auditing
AskPDF
Mar 26, 2026

Why Safety Probes Catch Liars But Miss Fanatics

Activation-based probes have emerged as a promising approach for detecting deceptively aligned AI systems by identifying internal conflict between true and stated goals. We identify a fundamental blind spot: probes fail on coherent misalignment - models that believe their harmful behavior is virtuous rather than strategically hiding it. We prove that no polynomial-time probe can detect such misalignment with non-trivial accuracy when belief structures reach sufficient complexity (PRF-like triggers). We show the emergence of this phenomenon on a simple task by training two models with identical RLHF procedures: one producing direct hostile responses ("the Liar"), another trained towards coherent misalignment using rationalizations that frame hostility as protective ("the Fanatic"). Both exhibit identical behavior, but the Liar is detected 95%+ of the time while the Fanatic evades detection almost entirely. We term this Emergent Probe Evasion: training with belief-consistent reasoning shifts models from a detectable "deceptive" regime to an undetectable "coherent" regime - not by learning to hide, but by learning to believe.

Kristiyan Haralambiev

* Independent Research.

Deception in Language ModelsEmergent Misalignment in Language ModelsHidden-State Probing
AskPDF
Mar 20, 2026

Detection Without Correction: A Robust Asymmetry in Activation-Based Hallucination Probing

Activation-based linear probing is widely proposed as a method for both detecting and correcting hallucinations in autoregressive language models. We present an empirical study across seven models spanning 117M to 7B parameters and three architecture families (GPT-2, Pythia, Qwen-2.5) that documents a robust asymmetry: linear probes can detect hallucination signals with above-chance accuracy in larger models, but activation steering along the probe-derived direction fails to correct hallucinations in 7 of 7 models tested. We further find that output-confidence baselines outperform activation probes on raw detection AUC at every model above 410M parameters, with the gap reaching 0.157 AUC for Pythia-6.9B. The probe's distinguishing value is therefore not detection accuracy but temporal positioning: probe signals are accessible at position zero (before any output tokens are produced), enabling pre-generation flagging that output-based methods structurally cannot provide. The temporal signal is statistically significant in two of seven models (Pythia-1.4B, p = 0.012; Qwen2.5-7B, p = 0.038) and absent in models below 400M parameters and in the base-only Pythia-6.9B. We position these findings as a clean negative result for the dominant probing-as-detection-and-control research direction and as initial evidence that probe-based methods occupy a complementary deployment niche, namely pre-generation flagging, rather than competing with output-based detectors on raw accuracy.

Dip Roy, Rajiv Misra, Sanjay Kumar Singh +1

Department of Computer Science and Engineering, Indian Institute of Technology Patna, India · Department of Computer Science, Rajarshi School of Management and Technology, Varanasi, India · Department of Electronics and Communication Engineering, Jaypee Institute of Information Technology, Noida, India

LLM Hallucination DetectionHidden-State ProbingActivation Steering
AskPDF