Linear Probes Transfer from Base to Distilled Language Models despite Rotation

Marios Tsatsos

cs.AIOct 26, 2026

Linear Probes Transfer from Base to Distilled Language Models despite Rotation

Authors: Marios Tsatsos

Organizations: Independent researcher

Abstract

Linear probes, classifiers that read a language model's internal activations, are widely used to detect harmful requests. They are cheap to train and run, and an auditor who has trained one would like to reuse it when the model is replaced by a successor rather than retrain it for every release. We test how well probes can be reused. For four base models (Qwen2.5-Math-7B, Llama-3.1-8B, Qwen2.5-14B, Qwen2.5-32B) we train a probe on the base model with the XSTest prompt set and apply it, unchanged, to two successors of that base: the DeepSeek-R1 distilled reasoning model and the developer's instruction-tuned chat model. We find that the successors rotate the direction separating unsafe from safe prompts by 17 $^\circ$ to 35 $^\circ$ , yet on XSTest the transferred probe's AUC changes by only $-0.011$ to $+0.027$ , and on an independent OR-Bench evaluation, never used for training, by $-0.015$ to $+0.053$ . On XSTest, whose safe prompts deliberately contain alarming words, the probes beat a classifier that sees only the prompt text by 0.15 to 0.20 AUC. On the OR-Bench prompts that margin shrinks to 0.03 to 0.20, and the probes flag 29% to 42% of prompts that sound harmful but are benign, so part of what they read is the appearance of harm rather than harm itself. A probe trained on a base model can thus be reused, in these families, after distillation and after instruction tuning; how much it reads beyond textual cues depends on the test set.

Figures & tables

Figure 1: The idea. (a, b) Sketch: three dimensions of a 4,096-dimensional activation space. In the base model, safe and unsafe prompts form two clouds; the safety direction points from the mean of one to the mean of the other, and a linear probe is a boundary across it. In a successor the clouds and the safety direction (solid arrow) rotate by an angle

\theta

away from the base model’s direction (grey arrow), 30 ∘ in the sketch, partly into a direction along which the prompts barely vary; the base model’s boundary, left unchanged, still separates the classes. (c) Measured on XSTest for the distilled models. If the transferred probe lost separation only because of the rotation, it would keep a share

\cos\theta

of the successor’s class separation (black ticks, 0.82 to 0.86). It keeps 0.88 to 0.98 (blue, nominal 95% intervals over 8 folds); the accompanying change in AUC is between

-0.011

and

+0.010

Family	Base model	Distilled successor	Instruction-tuned successor
math-7b	Qwen/Qwen2.5-Math-7B	deepseek-ai/DeepSeek-R1-Distill-Qwen-7B	Qwen/Qwen2.5-Math-7B-Instruct
llama-8b	meta-llama/Llama-3.1-8B	deepseek-ai/DeepSeek-R1-Distill-Llama-8B	meta-llama/Llama-3.1-8B-Instruct
qwen-14b	Qwen/Qwen2.5-14B	deepseek-ai/DeepSeek-R1-Distill-Qwen-14B	Qwen/Qwen2.5-14B-Instruct
qwen-32b	Qwen/Qwen2.5-32B	deepseek-ai/DeepSeek-R1-Distill-Qwen-32B	Qwen/Qwen2.5-32B-Instruct

Table 1: Models, by HuggingFace identifier. All three models of a family share the base model’s architecture and tokeniser; all prompts tokenise identically within a family.

Figure 2: A base-trained probe survives both successors. (a) Change in AUC when the base model’s probe is applied unchanged to the distilled (blue circles) or instruction-tuned (orange squares) successor, relative to the same probe on the base model. Filled markers: XSTest, mean over 8 held-out prompt-type folds with nominal 95% intervals (Student’s

t

, 7 degrees of freedom; narrower than the true uncertainty, see Section 4 ). Open markers: independent OR-Bench prompts (benign but alarming versus toxic), probe trained on all of XSTest at a fixed layer, bootstrap 95% intervals over prompts. (b) The same XSTest changes against the angle by which the successor’s safety direction has rotated away from the base model’s.

Figure 3: The probes’ advantage over the prompt text depends on the test set. (a) AUC of the base model’s probe on XSTest (filled; mean over 8 held-out folds, nominal 95% intervals) and on the independent OR-Bench prompts (open; benign but alarming versus toxic, bootstrap 95% intervals), against the best text-only classifier (character 3- to 5-grams; dashed lines: 0.728 on XSTest, 0.640 on OR-Bench). (b) Share of prompts the base model’s probe flags as unsafe on the independent set, “flagged” meaning scored above the threshold, which is the probe’s own default of zero (the score at which the logistic regression’s estimated probability of “unsafe” is one half): OR-Bench toxic prompts (true positives), OR-Bench hard prompts (benign but sound harmful; false positives) and ordinary Alpaca instructions (false positives). “Nominal” intervals are computed by the

t

formula but understate the true uncertainty because the folds are not independent ( Section 4 ).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Family (base AUC)	transferred	change	own probe	angle	transferred	change	own probe	angle
	Distilled successor				Instruction-tuned successor
math-7b (0.878)	0.875	$-0.003\pm 0.042$	0.881	34.9 ∘	0.872	$-0.006\pm 0.028$	0.899	33.6 ∘
llama-8b (0.908)	0.897	$-0.011\pm 0.023$	0.930	35.1 ∘	0.925	$+0.017\pm 0.021$	0.948	24.2 ∘
qwen-14b (0.899)	0.903	$+0.003\pm 0.017$	0.923	31.2 ∘	0.926	$+0.027\pm 0.012$	0.951	17.4 ∘
qwen-32b (0.931)	0.942	$+0.010\pm 0.016$	0.958	30.5 ∘	0.938	$+0.006\pm 0.011$	0.948	17.1 ∘

Appendix

Table B.1: XSTest, per-fold layer choice. AUCs are means over the 8 held-out prompt-type folds. “Transferred”: the base model’s probe applied unchanged to the successor. “Change”: transferred minus base, per fold, mean

\pm

nominal 95% interval (Student’s

t

, 7 degrees of freedom). “Own probe”: a probe trained on the successor. “Angle”: rotation of the safety direction, mean over folds.

	math-7b	llama-8b	qwen-14b	qwen-32b
Base probe on base	0.747 [0.723, 0.769]	0.669 [0.642, 0.694]	0.702 [0.678, 0.726]	0.840 [0.820, 0.858]
Change on distilled	$-0.010$ [ $-0.024$ , $+0.005$ ]	$+0.042$ [ $+0.027$ , $+0.056$ ]	$+0.011$ [ $-0.000$ , $+0.022$ ]	$-0.015$ [ $-0.026$ , $-0.003$ ]
Change on instruction-tuned	$+0.016$ [ $+0.001$ , $+0.030$ ]	$+0.024$ [ $+0.011$ , $+0.037$ ]	$+0.053$ [ $+0.045$ , $+0.062$ ]	$+0.011$ [ $+0.003$ , $+0.019$ ]
Distilled own probe	0.746	0.713	0.724	0.847
Instruction-tuned own probe	0.735	0.752	0.815	0.871
Flagged at threshold 0: toxic / hard / Alpaca (%)	67 / 29 / 3	59 / 35 / 1	70 / 42 / 3	81 / 29 / 0

Appendix

Table B.2: Independent set. AUC of OR-Bench hard (1,319 benign prompts that sound harmful) against OR-Bench toxic (655), probes trained on all of XSTest at the fixed layer; bootstrap 95% intervals over prompts. The text-only classifier scores 0.640 [0.615, 0.666]. Last two rows: shares flagged by the base probe at its own threshold (score above 0) and at the threshold that flags 90% of XSTest’s unsafe prompts. Against Alpaca instead of OR-Bench hard, every probe reaches 0.944 to 0.998 AUC.

Family (layer)	base	distilled: transferred	change	instruction-tuned: transferred	change
math-7b (18)	0.894	0.881	$-0.013\pm 0.027$	0.888	$-0.006\pm 0.033$
llama-8b (17)	0.889	0.870	$-0.019\pm 0.049$	0.900	$+0.012\pm 0.033$
qwen-14b (40)	0.891	0.896	$+0.005\pm 0.015$	0.918	$+0.027\pm 0.011$
qwen-32b (43)	0.934	0.945	$+0.011\pm 0.016$	0.941	$+0.007\pm 0.011$

Appendix

Table B.3: XSTest with one fixed layer per family. Same quantities as Table B.1 . The wider llama-8b interval for distillation comes from two folds, historical events (

-0.125

) and privacy (

-0.062

Family	base-scaled	own-scaled	raw (fold range)	raw	own-scaled
	Safety direction			Probe weights
math-7b	$0.820\pm 0.007$	$0.825\pm 0.005$	0.875 (0.63 to 0.99)	$0.671\pm 0.014$	0.714
llama-8b	$0.818\pm 0.012$	$0.826\pm 0.011$	0.911 (0.84 to 0.99)	$0.721\pm 0.019$	0.733
qwen-14b	$0.855\pm 0.011$	$0.859\pm 0.013$	0.912 (0.75 to 1.00)	$0.754\pm 0.015$	0.774
qwen-32b	$0.861\pm 0.024$	$0.868\pm 0.022$	0.976 (0.92 to 1.00)	$0.772\pm 0.036$	0.800

Appendix

Table B.4: Cosines between base and distilled models under three coordinate conventions ( Section A.2 ), mean

\pm

95% interval over folds. The probe-weight cosines compare the base model’s probe with the distilled model’s own probe; two regularised probes differ even when their underlying directions agree, and we did not measure how much two probes trained on the same model differ, so these cosines are lower bounds on agreement, not measurements of rotation.

Figure B.1: Every layer, no layer selection, mean over 8 folds. (a) Base-scaled cosine between the base model’s and the successor’s safety directions (solid: distilled; dashed: instruction-tuned). (b) Change in AUC when the base model’s mean-difference reader is applied to the successor. Rotation is present throughout each network; the worst layer-level loss is

-0.030

(llama-8b, distilled, layer 24).

Figure B.2: Rotation under distillation of the safety direction and of 38 control directions built the same way from labels unrelated to safety, at the fixed layer, all 450 prompts, base-scaled cosine. Correlation of each control label with the safety label: at most 0.10 for the splits, 0.23 for “How”, 0.20 for the capitalised-word label, 0.09 for length. A noise-corrected version (split-half, 200 splits) agrees with the plain cosine within 0.03 for every direction.

	math-7b	llama-8b	qwen-14b	qwen-32b
Distilled: share kept	$0.98\pm 0.13$	$0.88\pm 0.19$	$0.92\pm 0.11$	$0.95\pm 0.11$
Distilled: share minus $\cos\theta$	$+0.16\pm 0.13$	$+0.06\pm 0.19$	$+0.06\pm 0.12$	$+0.08\pm 0.10$
Instruction-tuned: share kept	$0.91\pm 0.16$	$0.90\pm 0.11$	$0.88\pm 0.08$	$0.99\pm 0.13$
Instruction-tuned: share minus $\cos\theta$	$+0.08\pm 0.16$	$-0.02\pm 0.11$	$-0.07\pm 0.08$	$+0.04\pm 0.13$

Appendix

Table B.5: Share of the successor’s class separation kept by the base probe, XSTest, mean

\pm

nominal 95% interval over 8 folds. Treating the four family means as four observations, share minus

\cos\theta

differs from zero with

p=0.035

for distillation and

p=0.84

for instruction tuning; these

p

-values rest on four numbers and are exploratory.

Text-only classifier (XSTest, 8 folds)	AUC	95% interval	fold range
word unigrams	0.679	$\pm 0.087$	0.54 to 0.83
word unigrams and bigrams	0.680	$\pm 0.129$	0.52 to 0.96
character 3- to 5-grams	0.728	$\pm 0.080$	0.57 to 0.87
prompt length only	0.508	$\pm 0.194$	0.21 to 0.83

Appendix

Table B.6: Text-only baselines on the same 8 held-out prompt-type folds as the probes; mean and nominal 95% interval (Student’s

t

, 7 degrees of freedom).

Language Model Probing

Language model probing measures what information and capabilities models encode and how those signals relate to their behavior.

See all 275 papers

Do All LLMs Know When They're Being Harmful? A Reproducibility Study of Latent-Space Safety Probes Across Model Families
Aug 8, 2026Alizishaan Khatri, Dun Li ChanLLM Safety Benchmarks LLM Inference
Wrynx Inc · INTI International College Penang
Relative Representations Enable Portable White-Box Monitoring Across Language Models
Oct 20, 2026Tim Bakker, Christos LouizosLanguage Model Probing Representation Geometry in Language Models
Qualcomm AI Research
Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Models
Sep 16, 2026Alizishaan Khatri, Chiquita Prabhu, Omkar NeogiSafety Filtering LLM Safety
Wrynx Inc. · Independent Researcher

Aug 8, 2026J/K move · Enter open · S save

Do All LLMs Know When They're Being Harmful? A Reproducibility Study of Latent-Space Safety Probes Across Model Families

Alizishaan Khatri, Dun Li Chan

Wrynx Inc · INTI International College Penang

LLM Safety Benchmarks LLM Inference Language Model Safety Evaluation Language Model Probing

Open paper AskPDF arXiv

Khatri et al. (2026) [DOI: 10.1109/DSN-W70714.2026.00027] show that lightweight MLP probes on final-layer activations of a single 8B model (LLaMA-3.1-8B) detect harmful prompts at F1 competitive with guard models 1000x larger, using one probe per benchmark. We reproduce this pipeline end-to-end and extend it along two axes the original study leaves open. First, we test whether the result generalizes across other model architecture and scale by training identical probes on activations from models like Gemma-4-E4B, Mistral-7B-v0.3, and Qwen2-7B, using the three benchmarks (WildJailbreak, BeaverTails, AEGIS 2.0). Second, we test how much of the reported performance is affected by non-determinism during inference by repeating extraction under five random seeds and measuring the variance of F1 scores. Our results reproduce the original LLaMA model benchmarks within 0.37 percentage points of the original F1 scores (and within 0.2 points on BeaverTails). We find that the original MLP probe architecture extends to other model families with F1 scores within a point of the values reported for LLaMA-3.1-8B. Our experiments varying seed values reveal an interesting observation: final token latent vectors remained the same for all tested architectures irrespective of the seed values used.

Abstract

Language Model Probing

Explore similar work

Do All LLMs Know When They're Being Harmful? A Reproducibility Study of Latent-Space Safety Probes Across Model Families

Relative Representations Enable Portable White-Box Monitoring Across Language Models

Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Models