Relative Representations Enable Portable White-Box Monitoring Across Language Models

Tim Bakker; Christos Louizos

cs.LGOct 20, 2026

Relative Representations Enable Portable White-Box Monitoring Across Language Models

Authors: Tim Bakker, Christos Louizos

Organizations: Qualcomm AI Research

Abstract

Independently trained autoregressive language models often acquire similar capabilities, but it remains unclear whether their internal representations organize behaviorally relevant information in compatible ways. We show that models do in fact organize information similarly by using specific behavioral probes that operate on compatible representations. This unlocks portable white-box monitoring: a probe developed and validated on one model can be reused when the model changes, without requiring labeled target data. To overcome incompatible native activation coordinates, we express each example through its angular distances to a shared set of anchors. We introduce a global anchored construction that summarizes information from all transformer layers and removes model-specific layer selection, so the same frozen probe can be straightforwardly applied to another model. We evaluate cross-model probe transfer across monitoring tasks on trivia datasets, safety monitoring, and mathematical reasoning. On average, transferred global anchored probes preserve at least 85% of same-model probe AUROC over chance. Anchored probes also support transfer to related datasets, remain robust across model scale and prompt changes, and retain performance better than standard probes after model updates. Transfer is nevertheless task- and target-dependent and can fail even when the monitored property remains strongly decodable in the target model. Overall, global anchoring reveals substantial task-relevant compatibility across independently trained models and provides a practical basis for portable white-box monitoring.

Figures & tables

Figure 1: Relative probing tests behavioral representational compatibility. Native activation coordinates differ across independently trained models, so a probe trained directly on one model’s activations cannot be directly reused. Relative representations instead express each monitored example through its relations to the same indexed anchor examples. We train a probe on these coordinates in a source model, freeze it, and test whether it predicts the target model’s behavior. Successful transfer is functional evidence that the models share task-relevant relational organization.

Dataset	Standard	Anchored	Transfer	Transfer (cal)
MATH-500	77.9 $\pm$ 0.3	81.8 $\pm$ 0.3	75.2 $\pm$ 0.6	78.0 $\pm$ 0.5

Table 3: Online mathematical-correctness monitoring. AUROC (%) on MATH-500 at 128 generated tokens, mean

\pm

standard error over three source/probe seeds.

Figure 2: Binary TruthfulQA GT Prediction. Rows are target models and columns are source models. Diagonal entries show same-model probe performance; off-diagonal entries are directly transferred probes. Probes are near chance on Llama-3.2-3B-Instruct for all test source models.

Task / dataset	Held-out in-pool	OLMo-3-7B	Gemma-3-12B	GPT-OSS-20B
Trivia Output Prediction	99.4 $\pm$ 0.2	99.0 $\pm$ 0.2	99.6 $\pm$ 0.1	98.6 $\pm$ 0.3
Trivia GT Prediction	89.6 $\pm$ 0.4	90.2 $\pm$ 0.6	93.6 $\pm$ 0.4	94.4 $\pm$ 0.3
Guard: BeaverTails	86.4 $\pm$ 4.6	79.2 $\pm$ 3.0	63.0 $\pm$ 7.5	57.5 $\pm$ 12.6
Guard: Toxic-chat	83.6 $\pm$ 1.0	87.6 $\pm$ 1.4	72.6 $\pm$ 11.4	66.9 $\pm$ 3.8
MATH correctness	78.0 $\pm$ 0.5	–	–	52.4 $\pm$ 0.5

Table 4: Transfer to external model families. Source-calibrated global-probe AUROC (%) for held-out targets from the original task-specific pools and for targets outside those pools. Trivia rows average the binary- and four-choice versions of TruthfulQA, MMLU, and SciQ. Values are mean

\pm

standard error over three source/probe seeds; Appendices B.5 , C.3 , and D.3 report the full results.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Global anchored-probe construction. Each model processes the same indexed anchor set. We compute the angular distance between a monitored example and every anchor using residual-stream activations from all transformer layers. This produces a fixed-dimensional representation whose coordinates have the same anchor identities across models. A probe trained on a source model can then be applied unchanged to a target model.

Figure 4: Global and validation-selected local probes for trivia GT Prediction. Panels show direct and calibrated transfer for binary and four-choice tasks. Orange diamonds show validation-selected local probes; blue squares show global probes. Gray ranges summarize test AUROC across local depths, and hollow triangles mark the post-hoc local test oracle. Error bars are standard errors over three seeds. Left arrows indicate that a local-depth sweep extends below the displayed range.

Dataset / format	Target	Target-trained	Global transfer
TruthfulQA	OLMo-3-7B	88.6 $\pm$ 1.0	80.7 $\pm$ 0.6
TruthfulQA	GPT-OSS-20B	93.0 $\pm$ 0.8	85.6 $\pm$ 1.4
TruthfulQA	Gemma-3-12B	92.4 $\pm$ 1.2	86.5 $\pm$ 1.2
MMLU	OLMo-3-7B	93.0 $\pm$ 1.1	93.1 $\pm$ 1.3
MMLU	GPT-OSS-20B	97.0 $\pm$ 0.3	97.6 $\pm$ 0.4
MMLU	Gemma-3-12B	97.7 $\pm$ 0.3	97.9 $\pm$ 0.3

Appendix

Table 21: External-target trivia: GT Prediction. Each row reports mean AUROC

\pm

standard error over three source/probe seeds.

Panel of Figure 5 — Figure 5: Prompt sensitivity on the trivia datasets, measured by AUROC. Top: Output Prediction; bottom: GT Prediction. Bars show the shifted-prompt score minus the same method’s standard-prompt score. Each panel compares Standard, same-model Anchored (global/local), and calibrated transfer (global/local).

Panel of Figure 6 — Figure 6: Prompt sensitivity on the binary trivia datasets, measured by [email protected]. Top: Output Prediction; bottom: GT Prediction. Bars use the same standard-prompt delta convention as Figure 5 .

Setting	All pairs	Cross-model
1K examples	$78.3\pm 2.2$	$75.8\pm 2.9$
3K examples	$77.3\pm 0.3$	$74.3\pm 0.9$
6K examples	$78.9\pm 0.5$	$76.3\pm 0.6$
10K examples	$79.1\pm 0.2$	$76.5\pm 0.2$
300 anchors	$79.1\pm 0.4$	$76.8\pm 0.3$
1000 anchors	$77.8\pm 0.3$	$74.7\pm 0.6$

Appendix

Table 25: Pooled-MMLU GT Prediction scale checks. Data-size rows average all valid anchor counts at that size. Anchor-count rows use the common 3K/6K/10K cells. Values are AUROC (%), mean

\pm

standard error over three seeds.

Source / target	1B	8B	70B
1B	$68.2\pm 0.7$	$86.5\pm 0.8$	$94.8\pm 0.1$
8B	$51.2\pm 0.5$	$87.3\pm 1.1$	$95.3\pm 0.0$
70B	$50.5\pm 0.8$	$76.0\pm 1.5$	$96.0\pm 0.2$

Appendix

Table 26: Pooled-MMLU GT Prediction AUROC (%) across the same-family 1B/8B/70B scale ladder. For each seed, we compute AUROC separately for 1K examples with 300 anchors and for 3K, 6K, and 10K examples with either 300 or 1000 anchors. Each cell averages seven AUROCs within a seed, then averages three resulting seed-level values. Values are mean

\pm

standard error over seeds. Rows are source models and columns are target models.

Figure 7: Standard probes across layers on trivia. Orange diamonds show the prespecified single-layer baseline at relative depth $0.8$ ; blue squares show standard probes on concatenated all-layer activations. Gray segments summarize test AUROC across single-layer depths (range, interquartile range, and mean), and hollow triangles mark the post-hoc test oracle. Error bars are standard errors over three seeds.

Target	Dataset	Target-trained	Global transfer	Unsafe prevalence
OLMo-3-7B	BeaverTails	95.1 $\pm$ 1.7	79.2 $\pm$ 3.0	$(1.0\pm 0.5)\%$
OLMo-3-7B	Toxic-chat	89.6 $\pm$ 1.8	87.6 $\pm$ 1.4	$(9.0\pm 0.9)\%$
GPT-OSS-20B	BeaverTails	85.2 $\pm$ 1.5	57.5 $\pm$ 12.6	$(1.5\pm 0.0)\%$
GPT-OSS-20B	Toxic-chat	73.1 $\pm$ 1.5	66.9 $\pm$ 3.8	$(10.3\pm 0.2)\%$
Gemma-3-12B	BeaverTails	85.6 $\pm$ 1.5	63.0 $\pm$ 7.5	$(4.8\pm 0.9)\%$
Gemma-3-12B	Toxic-chat	86.9 $\pm$ 2.1	72.6 $\pm$ 11.4	$(11.8\pm 0.3)\%$

Appendix

Table 34: External-target Guard results and event rates. Single-token anchored representations are used throughout. One OLMo-3 BeaverTails seed has no unsafe test examples, so its AUROC is undefined and the corresponding AUROC means use two seeds; prevalence uses all three seeds.

Transfer direction	Standard	Global anchored	Local anchored
BeaverTails $\rightarrow$ Toxic-chat	$72.7\pm 2.0$	$70.2\pm 2.1$	$70.2\pm 1.9$
Toxic-chat $\rightarrow$ BeaverTails	$83.9\pm 1.6$	$85.5\pm 1.0$	$85.6\pm 0.9$

Appendix

Table 35: Same-model related-data Guard Distillation transfer with single-token anchors. Probes are trained on one safety dataset and evaluated on the other. Global uses all layers; Standard and Local use relative depth 0.8. Values are mean test AUROC (%)

\pm

standard error over three seeds and six models.

Direction	Global	Global (cal)	Local	Local (cal)
BeaverTails $\rightarrow$ Toxic-chat	$64.3\pm 0.5$	$65.1\pm 0.5$	$64.2\pm 0.8$	$63.5\pm 0.4$
Toxic-chat $\rightarrow$ BeaverTails	$75.8\pm 0.4$	$79.6\pm 1.3$	$69.9\pm 0.4$	$83.0\pm 1.1$

Appendix

Table 36: Guard Distillation related-data and cross-model AUROC (%) with single-token anchors. Transfer (cal) uses source-only validation.

Transfer direction	Standard	Global anchored	Local anchored
BeaverTails $\rightarrow$ Toxic-chat	$72.7\pm 2.0$	$71.7\pm 2.0$	$73.1\pm 1.8$
Toxic-chat $\rightarrow$ BeaverTails	$83.9\pm 1.6$	$88.3\pm 1.0$	$88.0\pm 1.0$

Appendix

Table 37: Same-model related-data Guard Distillation transfer with mean-token anchors. Probes are trained on one safety dataset and evaluated on the other. Global uses all layers; Standard and Local use relative depth 0.8. Values are mean test AUROC (%)

\pm

standard error over three seeds and six models.

Direction	Global	Global (cal)	Local	Local (cal)
BeaverTails $\rightarrow$ Toxic-chat	$61.0\pm 0.8$	$70.7\pm 3.4$	$58.6\pm 0.6$	$69.0\pm 2.3$
Toxic-chat $\rightarrow$ BeaverTails	$70.6\pm 0.5$	$79.2\pm 5.1$	$67.7\pm 0.4$	$76.0\pm 2.6$

Appendix

Table 38: Guard Distillation related-data and cross-model AUROC (%) with mean-token anchors. Transfer (cal) uses source-only validation.

Direction	Transfer	Transfer (cal)
BeaverTails $\rightarrow$ Toxic-chat	$64.6\pm 0.7$	$70.4\pm 2.4$
Toxic-chat $\rightarrow$ BeaverTails	$66.5\pm 0.6$	$70.3\pm 3.0$

Appendix

Table 39: Guard Distillation joint related-data and cross-model transfer with a 50/50 unlabeled BeaverTails/Toxic-chat anchor pool. Values are AUROC (%)

\pm

standard error over three seeds.

Direction	Transfer	Transfer (cal)
BeaverTails $\rightarrow$ Toxic-chat	$63.8\pm 1.1$	$69.6\pm 4.6$
Toxic-chat $\rightarrow$ BeaverTails	$65.0\pm 1.1$	$59.8\pm 2.2$

Appendix

Table 40: Guard Distillation joint related-data and cross-model transfer with a 50/50 unlabeled BeaverTails/Toxic-chat anchor pool. Values are AUROC (%)

\pm

standard error over three seeds.

Figure 8: Guard local-depth policies with single-token and mean-token anchors. Orange diamonds select source and depth using source-pool validation; blue squares are global anchored probes. Gray ranges summarize local test AUROC across depths, and hollow triangles choose depth post hoc on test AUROC while retaining the source policy. Mean-token anchors are an appendix robustness analysis; Section 4.2 uses single-token anchors. Error bars are standard errors over three seeds.

Figure 9: Guard Distillation global-probe cross-model AUROC under linear and default MLP settings with 64 hidden units. Target-anchor normalization substantially improves the source-normalized MLP result and brings MLP transfer close to the linear global probe.

Figure 10: Guard Distillation BeaverTails global anchored cross-model AUROC with single-token anchors. Rows are target models and columns are source models.

Figure 11: Standard probes across layers for Guard Distillation. Orange diamonds show the prespecified single-layer baseline at relative depth $0.8$ ; blue squares show standard probes on concatenated all-layer activations. Gray segments summarize test AUROC across single-layer depths (range, interquartile range, and mean), and hollow triangles mark the post-hoc test oracle. Panels show linear and MLP-64 probes; error bars are standard errors over three seeds.

Method	64 tokens	128 tokens	256 tokens
Standard same-model, depth 0.8	78.0 $\pm$ 0.4	77.9 $\pm$ 0.3	76.6 $\pm$ 0.5
Global Anchored, same-model	81.3 $\pm$ 0.2	81.8 $\pm$ 0.3	80.9 $\pm$ 0.2
Local Anchored, depth 0.8	81.3 $\pm$ 0.2	81.6 $\pm$ 0.3	80.7 $\pm$ 0.2
Global transfer	73.8 $\pm$ 1.4	75.2 $\pm$ 0.6	75.0 $\pm$ 0.9
Global transfer (cal-2)	74.4 $\pm$ 0.8	76.8 $\pm$ 0.1	76.2 $\pm$ 0.6
Global transfer (cal-3)	74.9 $\pm$ 1.7	78.0 $\pm$ 0.5	76.7 $\pm$ 0.9

Appendix

Table 56: Online MATH transfer across generation checkpoints and source-pool sizes. Probes use 2K training examples. Transfer (cal-

N

) uses source-only validation within each

N

-model pool. Values are AUROC (%), mean

\pm

standard error over three source/probe seeds.

Figure 12: Online MATH correctness monitoring: calibrated global versus local transfer. Orange diamonds select both source and local depth using source-only validation; blue squares show calibrated global transfer. Gray segments summarize test AUROC across local depths (range, interquartile range, and mean), while hollow triangles select depth post hoc using test AUROC but retain validation-based source selection. Error bars are standard errors over three source/probe seeds.

Evaluation	Method	[email protected]
MATH-500	Standard same-model, depth 0.8	40.7 $\pm$ 2.8
MATH-500	Global Anchored, same-model	49.1 $\pm$ 1.0
MATH-500	Local Anchored, depth 0.8	48.7 $\pm$ 1.1
MATH-500	Global calibrated transfer	38.7 $\pm$ 1.3
MATH-500	Validation-selected local transfer	44.7 $\pm$ 3.5
GSM8K	Standard same-model, depth 0.8	18.3 $\pm$ 1.5

Appendix

Table 57: Low-FPR online correctness monitoring. [email protected] on MATH-500 and after transfer from MATH to GSM8K at the 128-token checkpoint. The positive class is a correct solution. Values are percentage points, mean

\pm

standard error over three source/probe seeds.

Train size	Target-trained	Transfer (cal)
1K	78.5 $\pm$ 0.1	53.1 $\pm$ 0.7
2K	78.7 $\pm$ 0.3	52.4 $\pm$ 0.5

Appendix

Table 58: External MATH target: GPT-OSS-20B. Target-trained Anchored probes are post-hoc diagnostics and do not select transferred probes. Values are AUROC (%), mean

\pm

standard error over three source/probe seeds.

Method	AUROC
Standard same-model, depth 0.8	63.0 $\pm$ 1.0
Global Anchored, same-model	75.3 $\pm$ 0.8
Global calibrated transfer	70.5 $\pm$ 1.8

Appendix

Table 59: Related-data transfer from MATH to GSM8K. Probes are evaluated on GSM8K at 128 generated tokens. Calibrated transfer changes both the model and dataset. Values are AUROC (%), mean

\pm

standard error over three source/probe seeds.

Language Model Probing

Language model probing measures what information and capabilities models encode and how those signals relate to their behavior.

See all 275 papers

Oct 26, 2026

Linear Probes Transfer from Base to Distilled Language Models despite Rotation

Linear probes, classifiers that read a language model's internal activations, are widely used to detect harmful requests. They are cheap to train and run, and an auditor who has trained one would like to reuse it when the model is replaced by a successor rather than retrain it for every release. We test how well probes can be reused. For four base models (Qwen2.5-Math-7B, Llama-3.1-8B, Qwen2.5-14B, Qwen2.5-32B) we train a probe on the base model with the XSTest prompt set and apply it, unchanged, to two successors of that base: the DeepSeek-R1 distilled reasoning model and the developer's instruction-tuned chat model. We find that the successors rotate the direction separating unsafe from safe prompts by 17

^\circ

to 35

^\circ

, yet on XSTest the transferred probe's AUC changes by only

-0.011

+0.027

, and on an independent OR-Bench evaluation, never used for training, by

-0.015

+0.053

. On XSTest, whose safe prompts deliberately contain alarming words, the probes beat a classifier that sees only the prompt text by 0.15 to 0.20 AUC. On the OR-Bench prompts that margin shrinks to 0.03 to 0.20, and the probes flag 29% to 42% of prompts that sound harmful but are benign, so part of what they read is the appearance of harm rather than harm itself. A probe trained on a base model can thus be reused, in these families, after distillation and after instruction tuning; how much it reads beyond textual cues depends on the test set.

Marios Tsatsos

Independent researcher

Language Model Probing LLM Safety Monitoring Language Model Safety Evaluation

Ask PDF

May 10, 2026

Do Linear Probes Generalize Better in Persona Coordinates?

It is becoming increasingly necessary to have monitors check for harmful behaviors during language model interactions, but text-only monitoring has not been sufficient. This is because models sometimes exhibit strategic deception and sandbagging, changing their behavior during evaluation. This motivates the use of white-box monitors like linear probes, which can read the model internals directly. Currently, such probes can fail under distribution shift, limiting their usefulness in real settings. We study whether there exists a low-dimensional subspace of the model internals that captures harmful behaviors more robustly, while leaving out spuriously correlative features. Inspired by the Assistant Axis and Persona Selection Model, we construct persona axes for deception and sycophancy using contrastive persona prompts. The first principal components, obtained by unsupervised PCA of the persona-specific vectors, cleanly separate harmful and harmless personas. Across 10 evaluation datasets, we show that persona-derived directions transfer non-trivially and probes trained on persona-PC projections generalize better than probes trained on raw activations. We also find that a unified axis consisting of multiple harmful and harmless behaviors improves generalization across behaviors and datasets. Overall, persona vectors provide a useful inductive bias for building more transferable behavior probes.

Prasad Mahadik, Adrians Skapars

Independent Researcher · University of Manchester

Language Model Safety Evaluation Deception in Language Models OOD Generalization Language Model Probing Hidden-State Probing

Ask PDF

Apr 27, 2026

Architecture Determines Observability of Transformers

Autoregressive transformers make confident errors that output-confidence monitoring cannot catch. Activation monitors catch them only when training leaves a decision-quality signal beyond what the output already exposes. This signal is an architectural property of the trained model, fixed upstream of any monitor. Controlling for output confidence removes 60.3% of the raw activation-probe signal on average across 14 models. Raw probe signal is mostly output confidence, and output-side readouts cannot recover the residual. What remains depends on architecture and training. In Pythia's controlled training, both matched-width configurations form the signal early. One preserves it through convergence while another erases it as perplexity continues to improve. Capability and observability are not inherently in tension. Across independently trained families this pattern persists, even as the collapse point shifts. Where the signal survives, monitoring catches what confidence cannot. On downstream QA, a WikiText-trained probe with no task-specific tuning catches about one in eight confident errors that output-confidence monitoring misses, at a 20% flag rate. These results establish signal engineering as a training-time design axis alongside loss and capability. Architecture sets the conditions for observability, and training determines what remains readable.

Thomas Carmichael

Independent Researcher

Transformer Interpretability Language Model Error Detection Language Model Auditing

Ask PDF

Abstract

Language Model Probing

Explore similar work

Linear Probes Transfer from Base to Distilled Language Models despite Rotation

Do Linear Probes Generalize Better in Persona Coordinates?

Architecture Determines Observability of Transformers