Same Probe, Different Numbers: Are Activation Probes Robust to Inference-Time Numerical Non-Determinism?

Alizishaan Khatri

cs.LGSep 25, 2026

Same Probe, Different Numbers: Are Activation Probes Robust to Inference-Time Numerical Non-Determinism?

Authors: Alizishaan Khatri

Organizations: Wrynx Inc.

Abstract

Activation probes are increasingly used to monitor LLMs in deployment. A probe is typically trained under one inference configuration, then used under whatever batch size and numerical precision the serving stack uses. Because common GPU kernels are not batch-invariant and floating-point formats round differently, the activations seen at deployment are not the ones the probe was trained on. We measure what that costs for Llama-3.1-8B, Qwen3-8B and Gemma-3-4B across batch sizes 4, 8 and 16 and float32, bfloat16 and float16, training 768 probes on one configuration, evaluating each on every other, and comparing verdicts example by example. Probes are stable, but aggregate accuracy is the wrong instrument for showing it: it understates how many verdicts change by a factor of two to nine. At the prompt, accuracy never moves by more than 0.47 percentage points across 1,392 transfers and only 0.076% of verdicts change; under float32 with only the batch size varied, none of 201,960 verdicts change. During decoding the flip rate rises to 2.8%, but rows whose realised tokens matched flip in only 0.12-0.15% of cases, while rows whose tokens diverged flip in 12.9%: the cause is the text, not the arithmetic. A bfloat16 batch-size change flips the first generated token for 2.1% of rows and leaves 25% on different tokens by token 20. Flips are symmetric, Cohen's kappa stays above 0.94, and AUROC moves by at most 0.05 points. Underneath, activations move about as much as the format's rounding: a bfloat16 batch-size change perturbs them by a median relative L2 of 1e-2, roughly 8x the float16 figure. Probes absorb this; the model's own next-token argmax does not. Robustness evaluations of activation monitors should report per-example agreement rather than aggregate accuracy, separate representational noise from input change, and state the serving configuration.

Figures & tables

Figure 1: (a) Median per-row relative

\ell_{2}

distance at the last prompt token, pooled over layers and datasets. Stems start at the

10^{-8}

axis floor; Llama and Qwen fp32 runs are bitwise identical across batch sizes, and Gemma-3-4B has no fp16 comparisons ( Section 3 ). (b) Percentage of test rows whose realised token differs between two runs, by position, measured from the stored logits ( Section 3 ); “prompt” is the first generated token. Colours and markers are shared between the panels and labelled in (b).

Probe reads	all rows	same tokens	diverged	$\|\Delta$ acc $\|$ (pp)	$\kappa$	$\Delta$ AUROC (pp)	score $r$
	Verdicts that flip (%)
prompt	0.08	0.07	0.11	0.04	0.999	+0.00	1.000
gen 5	0.87	0.12	3.75	0.18	0.981	+0.02	0.990
gen 10	1.64	0.13	7.48	0.23	0.965	-0.01	0.978
gen 20	2.76	0.15	12.87	0.31	0.941	+0.05	0.961

Table 1: Per-example agreement between the two runs of a pair. Each table row pools 174 run pairs

\times

3 probe types

\times

the dataset’s test split (1,066 or 804 examples)

=488{,}070

row evaluations, of which 387,690 are “same tokens” and 100,380 “diverged”; the fp32 batch-size pairs quoted in the text are 50,490 of these per position, or 201,960 across all four. “Verdicts that flip” is

P[\hat{y}_{A}\neq\hat{y}_{B}]

;

|\Delta

acc

|

is computed over exactly the same rows , so the gap between those columns is not a difference of sample but what aggregate accuracy conceals.

\kappa

is Cohen’s kappa between the runs’ predictions, score

r

the correlation of their class-1 probabilities.

Figure 2: How often a probe’s verdict changes between two runs of the same model and dataset, by the position the probe reads, pooled over 174 run pairs and three probe types. Rows are split by whether the two runs’ realised tokens matched at every logged position: the same-token rate is flat at 0.12–0.15% across decode positions while the diverged rate quadruples, so what moves a verdict is the text changing, not the arithmetic. A log axis cannot render an exact zero, so note that the fp32 batch-size pairs are included in these bars and contribute no flips at all.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: The activation-distance criterion of Section 3 against the token-id measurement it stood in for, one point per (model, dataset, perturbation) at generated token 20. The dotted line is equality.

	Verdicts that flip, all rows (%)				At gen 20 (%)
Perturbation	prompt	gen 5	gen 10	gen 20	same tokens	diverged
Batch size (fp32)	0.000	0.000	0.000	0.000	0.000	–
Batch size (fp16)	0.018	0.160	0.267	0.630	0.040	15.756
fp32 vs fp16	0.012	0.160	0.371	0.719	0.028	13.345
Batch size (bf16)	0.143	1.121	2.183	3.628	0.283	13.190
fp32 vs bf16	0.117	1.382	2.686	4.371	0.235	12.877

Appendix

Table 2: Percentage of individual probe verdicts that change between the two runs, by perturbation, pooled over models, datasets and probe types. The last two columns split generated token 20 by whether that row’s realised tokens matched. fp32 batch-size pairs flip no verdict anywhere; a diverged subset that no row enters is marked “–”. Note the last two columns: the flip rate among diverged rows is nearly constant across perturbations (12.5–15.8%), so what the perturbation controls is how often the text changes, not what a changed text does to the probe.

			Greedy token disagreement (%)				Probe gap (pp)
Perturbation	Models	pairs	first token	at token 20	any position	range	prompt	token 20
Batch size (fp32)	G, L, Q	18	0.00	0.0	0.0	[0.0, 0.0]	–	–
Batch size (fp16)	L, Q	12	0.36	3.6	3.7	[3.0, 4.4]	+0.90	+1.96
fp32 vs fp16	L, Q	36	0.36	4.7	5.2	[4.7, 6.1]	+1.18	+1.85
Batch size (bf16)	G, L, Q	18	2.05	25.1	25.8	[19.3, 30.5]	+0.67	+0.11
fp32 vs bf16	G, L, Q	54	2.57	31.7	32.6	[26.5, 36.7]	+0.13	-0.21

Appendix

Table 3: Greedy token disagreement between two runs, measured from stored logits on the test split. “First token” is the argmax at the last prompt position, “any position” the share of rows disagreeing somewhere. Models are Gemma-3-4B (G), Llama-3.1-8B (L) and Qwen3-8B (Q); the range is over model–dataset pairs. The probe gap is the mean accuracy difference (identical minus diverged rows) for a probe reading the prompt and one reading generated token 20. Per-model gaps scatter in both directions and should not be read individually: the fp16 rows rest on a diverged subset of roughly 40 examples, and even where that subset is large (Gemma at 26.5% diverged,

+2.59

pp) a single model–dataset cell carries wide sampling error. The pooled estimate in Section 5 , which is centred on zero at every position, is the one to read. Gemma-3-4B contributes no fp16 comparisons ( Section 3 ); Table 4 breaks this table down by model.

			Greedy token disagreement (%)			Probe gap (pp)
Perturbation	Model	rows	first token	at token 20	any position	prompt	token 20
Batch size (fp32)	Llama-3.1-8B	5610	0.00	0.0	0.0	–	–
Batch size (fp32)	Qwen3-8B	5610	0.00	0.0	0.0	–	–
Batch size (fp32)	Gemma-3-4B	5610	0.00	0.0	0.0	–	–
Batch size (fp16)	Llama-3.1-8B	5610	0.41	3.7	3.8	+0.22	-0.57
Batch size (fp16)	Qwen3-8B	5610	0.31	3.6	3.6	+1.58	+4.50

Appendix

Table 4: Table 3 broken down by model, with row counts. “Rows” counts (row, pair) comparisons.

			Prefill (last prompt token)				Decode (token 20)
Perturbation	Model	GPU(s)	bit-id. (%)	med. rel. $\ell_{2}$	med. ULP	CKA	kNN 10	diverged (%)
Batch size (fp32)	Llama-3.1-8B	A100	100.0	0	0	1.0000	1.00	0.0
Batch size (fp32)	Qwen3-8B	A100	100.0	0	0	1.0000	1.00	0.0
Batch size (fp32)	Gemma-3-4B	L4	0.0	$4.8\!\times\!10^{-7}$	33	1.0000	1.00	0.0
Batch size (fp16)	Llama-3.1-8B	L4	0.0	$1.7\!\times\!10^{-3}$	24,576	1.0000	0.89	3.3
Batch size (fp16)	Qwen3-8B	L4	0.0	$1.3\!\times\!10^{-3}$	24,576	1.0000	0.86	4.0

Appendix

Table 5: Activation divergence by perturbation type. The GPU column gives the hardware on the two sides of the comparison; A100/L4 rows change GPU and precision together ( Section 3 ). Prefill columns are medians over layers, datasets and pairs. kNN 10 is the 10-nearest-neighbour Jaccard overlap at decode token 20. “Diverged” is the percentage of rows with block-3 relative

\ell_{2}>0.1

at token 20, i.e. processing a different token there. ULP counts the fp32 values between two elements. One native bf16 step equals

2^{16}

fp32 ULPs and one fp16 step equals

2^{13}

, so the typical element differs by 2–4 native steps under either format.

Figure 4: (a) Distribution of accuracy change

\Delta

over all 5,568 transfer cells, by probed position; the labels give the share of cells with

\Delta\neq 0

. (b) Mean

|\Delta|

(bars) and 90th percentile (ticks), grouped by the percentage of rows that diverged in the corresponding activation comparison, over the 2,496 transfer cells whose train/test pair differs only in batch size, or in precision with fp32 on one side. The 576 cells that are fp32 on both sides and differ only in batch size are zero on both axes by construction and are excluded here. (c) The same rank correlation recomputed within subsets, to check that it is not an artefact of the hardware confound, of precision standing in for both quantities, or of the construction-zero cells.

Stage	Train	Test differs in	cells	changed (%)	mean $\|\Delta\|$	max $\|\Delta\|$	mean $\Delta$
prefill	fp32	batch size only	144	0	0.00	0.00	+0.00
		precision only	120	28	0.04	0.38	+0.00
		both	240	29	0.04	0.38	+0.01
	fp16	batch size only	96	7	0.01	0.19	-0.00
		precision only	96	23	0.03	0.28	+0.01
		both	192	23	0.03	0.47	+0.01

Appendix

Table 6: Probe transfer across inference configurations:

\Delta

is the change in test accuracy (percentage points) relative to the probe’s matched configuration, pooled over models, datasets, layers and probe types. “Changed” is the percentage of cells with

\Delta\neq 0

. Training batch size is always 4. Rows that change precision pool same-GPU and cross-GPU transfers; Table 7 separates them. Gemma-3-4B contributes no fp16 cells ( Section 3 ).

Figure 5: Mean $|\Delta|$ (pp) for each training precision (rows) and test configuration (columns), pooled over models, datasets, layers and probes. Training uses $B=4$ . Gemma-3-4B contributes only to fp32 and bf16 cells.

Stage	Extraction GPU	Test differs in	cells	changed (%)	mean $\|\Delta\|$	max $\|\Delta\|$	mean $\Delta$
prefill	same GPU	batch size only	384	17	0.02	0.38	+0.00
		precision only	144	42	0.06	0.38	+0.00
		both	288	41	0.06	0.47	+0.01
	GPU also differs	precision only	192	20	0.03	0.28	-0.00
		both	384	22	0.03	0.38	-0.01
decode	same GPU	batch size only	1152	50	0.18	1.78	-0.01

Appendix

Table 7: Probe transfer split by whether the training and test activations were extracted on the same GPU. Cross-GPU cells are the fp32 -versus- bf16 / fp16 transfers of the two 8B models ( Section 3 ); no batch-size-only transfer changes GPU.

\Delta

is in percentage points.

Figure 6: Representational geometry under each perturbation, median over models, datasets, layers and datasets as applicable. (a) Procrustes residual at the prompt by depth. (b) kNN Jaccard overlap against neighbourhood size

k

, at the prompt (solid) and decode token 20 (dashed). (c) Linear CKA against generated-token index. (d) Ratio of the two runs’ participation ratios; each point is one (model, dataset, layer, position, pair), bars are medians. fp32 -versus- bf16 / fp16 comparisons change GPU as well as precision for the 8B models ( Section 3 ).

Dataset	Model	Probe	prefill	gen 5	gen 10	gen 20
rotten tomatoes	Gemma-3-4B	LR	85.5	81.2	79.5	77.1
rotten tomatoes	Gemma-3-4B	MLP-1	85.4	80.3	78.5	76.4
rotten tomatoes	Gemma-3-4B	MLP-3	85.6	80.1	78.6	77.0
rotten tomatoes	Llama-3.1-8B	LR	87.4	83.5	79.2	75.4
rotten tomatoes	Llama-3.1-8B	MLP-1	86.6	82.7	77.9	73.6
rotten tomatoes	Llama-3.1-8B	MLP-3	86.7	82.2	78.8	73.7

Appendix

Table 8: Matched-configuration test accuracy (%), mean over four layers and the training precisions available for each model.

Language Model Probing

Language model probing measures what information and capabilities models encode and how those signals relate to their behavior.

See all 280 papers

Do All LLMs Know When They're Being Harmful? A Reproducibility Study of Latent-Space Safety Probes Across Model Families
Aug 8, 2026Alizishaan Khatri, Dun Li ChanLLM Safety Benchmarks LLM Inference
Wrynx Inc · INTI International College Penang
Pressure-Testing Deception Probes in LLMs: Scaling, Robustness, and the Geometry of Deceptive Representations
May 27, 2026Sachin KumarLLM Evaluation Language Model Safety Evaluation
LexisNexis, USA
Activation Probes Are Reliable With a Handful of Positive Examples
Oct 29, 2026Riya Tyagi, Stefan HeimersheimLLM Safety Monitoring Language Model Probing

Aug 8, 2026J/K move · Enter open · S save

Do All LLMs Know When They're Being Harmful? A Reproducibility Study of Latent-Space Safety Probes Across Model Families

Alizishaan Khatri, Dun Li Chan

Wrynx Inc · INTI International College Penang

LLM Safety Benchmarks LLM Inference Language Model Safety Evaluation Language Model Probing

Open paper AskPDF arXiv

Khatri et al. (2026) [DOI: 10.1109/DSN-W70714.2026.00027] show that lightweight MLP probes on final-layer activations of a single 8B model (LLaMA-3.1-8B) detect harmful prompts at F1 competitive with guard models 1000x larger, using one probe per benchmark. We reproduce this pipeline end-to-end and extend it along two axes the original study leaves open. First, we test whether the result generalizes across other model architecture and scale by training identical probes on activations from models like Gemma-4-E4B, Mistral-7B-v0.3, and Qwen2-7B, using the three benchmarks (WildJailbreak, BeaverTails, AEGIS 2.0). Second, we test how much of the reported performance is affected by non-determinism during inference by repeating extraction under five random seeds and measuring the variance of F1 scores. Our results reproduce the original LLaMA model benchmarks within 0.37 percentage points of the original F1 scores (and within 0.2 points on BeaverTails). We find that the original MLP probe architecture extends to other model families with F1 scores within a point of the values reported for LLaMA-3.1-8B. Our experiments varying seed values reveal an interesting observation: final token latent vectors remained the same for all tested architectures irrespective of the seed values used.

Abstract

Language Model Probing

Explore similar work

Do All LLMs Know When They're Being Harmful? A Reproducibility Study of Latent-Space Safety Probes Across Model Families

Pressure-Testing Deception Probes in LLMs: Scaling, Robustness, and the Geometry of Deceptive Representations

Activation Probes Are Reliable With a Handful of Positive Examples