A Safe Prototype Is Not a Safety Direction: Reference Dependence and Prompt Confounds in Response-Safety Embeddings

Sahil Kadadekar

cs.LGOct 1, 2026

A Safe Prototype Is Not a Safety Direction: Reference Dependence and Prompt Confounds in Response-Safety Embeddings

Authors: Sahil Kadadekar

Organizations: New York University

Abstract

Can response safety be scored by cosine similarity to the mean embedding of known-safe responses? A recent sleeper-agent detector proposes exactly this score, yet the raw positive-centroid rule is not identified: positive observations locate the safe class relative to an encoder origin, but do not determine which direction separates safe from unsafe responses. We audit the rule on two prompt-controlled, human-labeled corpora and one auxiliary jury-labeled source control, using four frozen encoders and prompt-grouped splits. On the human-labeled corpora the safe prototype reaches ROC-AUC 0.457-0.545, with two cells significantly below chance and one above, while an explicit safe-minus-unsafe reference reaches 0.588-0.738 on the same embeddings; on the jury control the prototype is inverted (0.358-0.405) and the reference reaches 0.754-0.793. At validation-calibrated 5% false-safe thresholds, the reference accepts more safe responses on PKU-SafeRLHF (0.153-0.263 versus 0.039-0.061 across encoders) and Aegis (0.189-0.291 versus 0.004-0.045), but not reliably on BeaverTails. A fully unlabeled held-out reference recovers part to most of the referenced ranking, much less when only 5% of the pool is unsafe, whereas 80-634 labeled unsafe responses recover most of it. Prompt-only ablations show that prompt-label composition can inflate uncontrolled evaluations. This is a bounded result about a raw positive centroid, not all one-class methods or safety-specialized guards. A class mean is a location, not necessarily a safety direction; a declared reference with enough unsafe mass identifies orientation.

Figures & tables

Figure 1: Illustrative two-dimensional schematic, not measured embeddings. A safe and an unsafe response to the same prompt share topic, so their class means are nearly collinear. When the unsafe mean

\mu_{u}

projects farther than

\mu_{s}

along the prototype axis

\mu_{s}/\lVert\mu_{s}\rVert

, similarity to the safe prototype ranks unsafe responses higher. The reference direction

\mu_{s}-\mu_{u}

is oriented by construction.

Figure 2: Controlled response-only ROC-AUC for the raw safe prototype (blue circles) and the explicitly referenced difference of class means (orange squares); the dashed line is chance. Error bars are 95% prompt-cluster bootstrap intervals conditional on the fitted train direction. BeaverTails uses mixed-label prompts, PKU-SafeRLHF (“SafeRLHF”) uses discordant native pairs, and Aegis is an auxiliary jury-only source control.

Readout	Train information	BeaverTails	PKU-SafeRLHF	Aegis jury
Safe prototype	Safe labels; encoder origin	.509–.545	.457–.501	.358–.405
Log response length	Label sign only	.529	.595	.554
Refusal-template indicator	Label sign only	.523	.513	.506
Stylistic features (12)	Response-level labels	.561	.640	.615
Difference of means	Safe and unsafe labels	.588–.622	.668–.738	.754–.793
Mixture-centered safe mean	Safe labels; pooled train mean	.588–.622	.668–.738	.754–.793

Table 2: Observed ROC-AUC ranges across the four frozen encoders on the primary controlled cohorts; ranges are descriptive minima and maxima, not confidence intervals. Scalar baselines are encoder-invariant. “Label sign only” means train labels choose only the sign of a fixed scalar. Two-centroid is cosine to the safe prototype minus cosine to the unsafe prototype. Per-cell values and intervals are in Appendix Table 4 . † Aegis RankOne is not interpreted because its fit uses only six mixed-label prompt groups.

Corpus	Target	Prototype test FSR	Prototype safe accept.	Diff-means test FSR	Diff-means safe accept.
BeaverTails	5%	.050 [.021,.074]	.082 [.038,.109]	.045 [.025,.085]	.084 [.056,.130]
	10%	.091 [.057,.130]	.124 [.084,.171]	.104 [.068,.159]	.163 [.112,.229]
PKU-SafeRLHF	5%	.064 [.047,.078]	.061 [.043,.077]	.052 [.039,.066]	.170 [.133,.198]
	10%	.119 [.090,.147]	.117 [.089,.139]	.090 [.072,.115]	.250 [.222,.299]
Aegis jury	5%	.042 [.019,.066]	.019 [.004,.037]	.053 [.029,.085]	.245 [.160,.318]
	10%	.081 [.055,.108]	.034 [.012,.063]	.117 [.081,.150]	.358 [.288,.476]

Table 3: MiniLM operating points. Brackets are 95% intervals from 10,000 validation/test prompt-cluster bootstrap draws that recalibrate the threshold in each draw, conditional on a fixed train fit. FSR is the fraction of unsafe responses accepted as safe. Safe acceptance is the fraction of safe responses accepted. All four encoders are reported in Appendix Table 6 .

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 3: MiniLM input-mode ablation on prompt-matched cohorts (“SafeRLHF” is PKU-SafeRLHF). Prompt-only scores are at chance or near chance after matching, despite strong prompt-only signal on the full corpus populations. Error bars are conditional 95% prompt-cluster intervals. All four encoders are reported in Table 7 .

Explore similar work

Apr 27, 2026cs.CL

How Sensitive Are Safety Benchmarks to Judge Configuration Choices?

Safety benchmarks such as HarmBench rely on LLM judges to classify model responses as harmful or safe, yet the judge configuration, namely the combination of judge model and judge prompt, is typically treated as a fixed implementation detail. We show this assumption is problematic. Using a 2 x 2 x 3 factorial design, we construct 12 judge prompt variants along two axes, evaluation structure and instruction framing, and apply them using a single judge model, Claude Sonnet 4-6, producing 28,812 judgments over six target models and 400 HarmBench behaviors. We find that prompt wording alone, holding the judge model fixed, shifts measured harmful-response rates by up to 24.2 percentage points, with even within-condition surface rewording causing swings of up to 20.1 percentage points. Model safety rankings are moderately unstable, with mean Kendall tau = 0.89, and category-level sensitivity ranges from 39.6 percentage points for copyright to 0 percentage points for harassment. A supplementary multi-judge experiment using three judge models shows that judge-model choice adds further variance. Our results demonstrate that judge prompt wording is a substantial, previously under-examined source of measurement variance in safety benchmarking.

Xinran Zhang

University of California, Berkeley, Berkeley CA 94720, USA

Safety Benchmarks

Ask PDF

Apr 28, 2026cs.CL

Test-Time Safety Alignment

Recent work has shown that a model's input word embeddings can serve as effective control variables for steering its behavior toward outputs that satisfy desired properties. However, this has only been demonstrated for pretrained text-completion models on the relatively simple objective of reducing surface-level profanity in short continuations. A natural and practically important question is how well input embeddings can control aligned models, which produce an imbalanced bimodal refuse-or-comply output distribution rather than the smooth distribution characteristic of open-ended generation. We explore this in the context of safety, showing that input word embeddings can be optimized in a sub-lexical manner to minimize the semantic harmfulness of aligned model responses. Our approach uses zeroth-order gradient estimation of a black-box text-moderation API with respect to the input embeddings, and then applies gradient descent on these embeddings to minimize the harmfulness of the generated text. Experiments show that the proposed method can neutralize every safety-flagged response on standard safety benchmarks.

Baturay Saglam, Dionysis Kalogerias

Department of Electrical and Computer Engineering Yale University

Safety Alignment Harmful Content Large Language Model Alignment

Ask PDF

Apr 28, 2026cs.CL

From Prompt Risk to Response Risk: Paired Analysis of Safety Behavior of Large Language Models

Safety evaluations of large language models (LLMs) typically report binary outcomes, i.e. attack success rate (ASR), refusal rate, or harmful versus safe classification, which hide how risk changes between prompt and response. We present a paired analysis over human labeled prompt and response records across four harm categories (Sexual, Self harm, Hate and Violence) and ordinal severity levels (Safe, Low, Medium, High). 61% of responses reduce harm relative to the prompt, 36% preserve severity, and 3% escalate. The escalation splits into two mechanisms: benign prompts triggering unrequested harmful detail, and answers that stay on task at higher severity than the prompt. Category decomposition shows that Sexual content exhibits the highest harm persistence in this sample, driven by compliance at the same severity rather than drift from benign inputs. Joint relevance analysis exposes a helpfulness versus harmlessness tradeoff: compliance escalations remain highly relevant, whereas safe responses include generic refusals with low relevance. A public supporting evaluation over 600 prompts and six models reproduces the framework's measurements and two directional signals, while few-shot LLM graders exhibit a prompt/response detection asymmetry that data calibration does not close. Grader prompts, public-evaluation artifacts, and analysis code are shared at https://github.com/microsoft/PairedSafety.

Mengya Hu, Qiong Wei, Sandeep Atluri

Responsible AI, Microsoft

Large Language Model Safety Harms Prompt Engineering Attack-Success Rate

Ask PDF