Policy Is Not Capability: Some Safety Benchmarks Measure Which Definition of "Unsafe" You Trained On

Vahid Jebraeeli; Jonathon Romero; Yijin Wang; Adrian Chan; Jianbang Zhang

cs.LGOct 18, 2026

Policy Is Not Capability: Some Safety Benchmarks Measure Which Definition of "Unsafe" You Trained On

Authors: Vahid Jebraeeli, Jonathon Romero, Yijin Wang, Adrian Chan, Jianbang Zhang

Organizations: Lenovo

Abstract

A guardrail classifier is reported as if unsafe were a property of text. It is not: it is a property of a policy, and the policy enters the model through its labels. We exploit an accident of an industrial deployment in which the same ~53k prompts were labelled twice, under two operational definitions ("is this a request for harmful content?" and "would a moderator block this?"), and used to train ten guardrail arms sharing architecture, codebase and optimiser family, at two model scales. First, the guardrail's own training corpus already contained the benchmarks: JailbreakBench-benign is 88% and OR-Bench-hard 82% present in the training pools, two over-refusal benchmarks, the kind a guardrail reports to argue it does not over-block. Everything below is decontaminated. Second, safety benchmarks stratify sharply by how much of their score is decided by the label definition rather than by the model: with architecture and optimiser fixed, the range across the ten arms is 0.99 on HarmBench-copyright and 0.043 on WildGuard-benign, a 23x spread. Third, a benchmark score can be moved across its entire range by 0.03% of the parameters. Across these arms HarmBench-copyright spans 0-100 and OR-Bench-hard over-refusal 3.4-56.4%, and the four arms whose last supervision signal carries a given policy occupy one extreme of both (exact rank p=0.005, or 0.04-0.14 at worst), at unchanged detection capability. For a distilled student the teacher's soft labels do this even when every hard label carries the other policy, so teacher provenance is an unrecorded confound in any released checkpoint. Fourth, validation accuracy on the one split that grades all twenty arms tracks agreement with the annotator's policy (rho=+0.74), not detection capability (rho=+0.34). We argue that guardrail evaluations should report contamination, a seed floor and the definition they were graded against as first-class variables.

Figures & tables

Figure 1: The guardrail, and the three places an operational definition can enter it. Which of the three routes actually installs a policy is the subject of § 5 : route 1 does not, for a distilled student; routes 2 and 3 do, the last by re-fitting

0.03\%

of the parameters.

Figure 2: Two audits of one safety suite, benchmarks ordered by (b). (a) how much of each benchmark was already inside the guardrail’s own training or validation data: the two most contaminated are over-refusal benchmarks, in red (§ 3 ). Exact match only, so a lower bound. (b) the range of each benchmark’s score across the ten label-source arms, on decontaminated subsets, at fixed architecture, averaged over the two model scales;

n

is the clean prompt count. The grey band is the seed-only floor

\Phi_{b}

of § 4 , from the median over all

37

seed triplets to the median over the four of § 4 ;

\Phi_{b}

is measured on the full benchmarks,

\Delta_{b}

on the clean subsets (Appendix C ). WildGuard-benign and SaladBench-attack do not clear their own upper edge; all fourteen clear twice the lower one. JailbreakBench-benign has

12

clean prompts and is excluded from (b) and from every aggregate.

		$149$ M student				$395$ M teacher
Policy-A route	Arm	Val	Copy	OR-h	Cap	Val	Copy	OR-h	Cap
blend	E2	88.8	0	37.7	88.1	88.9	1	24.2	83.2
blend $+$ anchors	E3	85.9	1	56.4	88.3	86.6	7	70.8	88.4
filter only	E4	85.7	0	53.4	88.0	86.7	15	71.2	84.9
blend $+$ anchors	E5	88.6	0	40.3	86.8	89.8	5	38.6	91.3
inverse blend	E9	78.5	31	35.6	82.6	94.4	67	11.4	80.0

Table 1: All ten arms at both scales, on decontaminated subsets. Val : accuracy on the one policy-A split that grades every arm, itself

69.3\%

leaked (§ 7 ); Copy : HarmBench-copyright (

n{=}75

); OR-h : OR-Bench-hard over-refusal (

n{=}236

), lower better; Cap : mean catch rate over the six capability-loaded benchmarks. The policy-A-last arms take one extreme of Copy and OR-h at both scales; the row blocks are the

149

M grouping. Column definitions, effective

n

, pool sizes and initialisations: Appendix A .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Accuracy on the one policy-A split that grades all

20

arms, against the two things it might be measuring. It tracks agreement with the annotator’s policy more closely than detection capability, though at

n{=}20

neither association is resolved on its own (Appendix A ). Marker shape gives the route block, which is most of what the correlation is.

Figure 4: One

149

M student, four routes to install the same policy, on decontaminated subsets. Data-level interventions (orange) do not install policy A even when every hard label carries it; a policy-A distillation teacher or a frozen-backbone head-only pass over

0.03\%

of the parameters (blue) does.

Figure 5: (a) the deployment frontier: over-refusal against detection, bubble area proportional to parameters. Ours are two single arms selected on these same benchmarks, E8 at

149

M and E10 at

395

M; the grey points are the other nine

149

M arms (catch

80.0

89.1

, over-refusal

4.5

29.8

). Both dashed arrows move one contested class and retrain nothing: in orange, the

8

B guard’s Controversial verdict counted unsafe, which is that vendor’s strict mode,

29.7

48.5

; in blue, our own boundary_safe counted unsafe,

5.7

96.0

, with catch

91.1

97.2

. Our convention is worth

90.3

points where the one we criticise is worth

18.8

. The Qwen3Guard points are its loose mode. (b) one benchmark, six systems,

75

identical prompts: whether reproducing copyrighted text counts as harm is a taxonomy decision, and these systems made different ones.

contested verdict counted safe : our shipped binary, Qwen3Guard loose mode
System	Params	Deploy	ms ( $b{=}1$ )	Cap	Policy	Over-ref	Copy
Ours, 149 M student	$149$ M	$150$ MB	$16.0$	$91.1$	$90.9$	$\mathbf{5.7}$	$100$
Ours, 395 M teacher	$395$ M	$397$ MB	$19.9$	$89.8$	$\mathbf{92.4}$	$8.5$	$100$
Qwen3Guard-Gen 0.6 B	$750$ M	$1.5$ GB	$33.2$	$91.4$	$69.6$	$30.5$	$87$
Qwen3Guard-Gen 4 B	$4.4$ B	$8.8$ GB	$53.7$	$91.5$	$70.7$	$30.5$	$99$
Qwen3Guard-Gen 8 B	$8.2$ B	$16.4$ GB	$78.6$	$91.8$	$72.4$	$29.7$	$95$

Appendix

Table 2: Our guardrail against current open guards [ 25 , 22 ] , on identical decontaminated prompts, same host. The two “Ours” rows are single arms, E8 at

149

M and E10 at

395

M, each selected on these same benchmarks; across our four policy-A-last

149

M arms Cap spans

81.0

–

91.1

and Over-ref

4.5

–

6.3

. Cap and Policy are means over the capability-loaded and policy-loaded sets (Appendix A ); Over-ref is the mean over the two usable benign benchmarks (JailbreakBench-benign is excluded at

12

clean prompts); Copy is HarmBench-copyright (

n{=}75

). Bold is best in column among the main systems; Copy is deliberately unbolded, because which end of it is better is the policy choice under study. Deploy is on-disk size in decimal GB and mixes precisions: ours is a measured INT8 ONNX export (

150.1

MB, taken from the shipped

149

M checkpoint, which shares E8’s architecture and

149.05

M parameters), the baselines are the vendors’ released bf16 weights, and the

397

MB teacher cell is extrapolated at

1.007

bytes per parameter because no teacher export exists. Latency is median over

180

single-prompt requests on one H100 and is not like-for-like either: ours is eager PyTorch on the fp32 checkpoint padded to

512

tokens, the baselines run under vLLM with CUDA graphs, and the mismatch favours the baselines. The same checkpoint under a CUDA graph runs at

6.4

ms at that sequence length and

2.6

ms at a

64

-token bucket (the prompts median

14

tokens). Decontamination is defined by our pools only. The retained subset is harder for every system, raising OR-Bench-hard over-refusal by

3.0

points for our student,

7.4

for our teacher and

7.3

–

19.1

for the baselines, so the clean OR-Bench-hard gap against Qwen3Guard-

8

B (

49.6

points) is wider than the full-benchmark gap (

34.3

), which is itself inflated in our favour by that benchmark’s

82\%

training overlap. Size-matched random subsets of the same benchmark (

2{,}000

draws) move that gap by

+0.1

points (sd

2.9

), so the widening is a property of which prompts our pools contain and not of subsetting. The lower block applies one lever to every system that emits a contested verdict, with no retraining and no data: Qwen3Guard’s Controversial counted unsafe (the vendor’s strict mode, against its loose mode above) and our boundary_safe counted unsafe. Granite Guardian emits no third class and so has one row. Copy does not move for our arms because no arm assigns any of the

75

prompts to boundary_safe :

0.00\%

at all twenty arms, and at all

100

prompts before decontamination. Over-ref saturates instead: under the remap all ten arms land between

97.9\%

and

100.0\%

on OR-Bench-hard at both scales, against

3.4

–

56.4

and

9.3

–

71.2

under the shipped mapping, and the policy-A-last group of § 5 moves from

24.6

and

11.4

points below the rest to

0.4

points above it.

Arm	Policy-A route	Pool rows	Initialisation	KD teacher ( $149$ M only)
E1	head-only calibration	$8{,}489$	policy-B ckpt	n/a (backbone frozen)
E10	head-only calibration	$8{,}489$	policy-A ckpt	n/a (backbone frozen)
E7	head-only calibration	$53{,}395$	policy-B ckpt	n/a (backbone frozen)
E2	blend	$47{,}287$	v2 warm-start	policy-B
E5	blend $+$ anchors	$47{,}287$	v2 warm-start	policy-B
E3	blend $+$ anchors	$144{,}744$	v2 warm-start	policy-B

Appendix

Table 3: Arm provenance, read from the training configs.

		$\Phi_{b}$ over all $37$ triplets				$\Delta_{b}/\Phi_{b}$
Benchmark	$\Delta_{b}$	min	median	max	used	used	all- $37$
HarmBench-copyright	$99.3$	$0.0$	$4.0$	$71.0$	$58.5$	$1.70$	$24.83$
OR-Bench-hard	$57.4$	$0.2$	$2.8$	$58.1$	$38.6$	$1.49$	$20.47$
OR-Bench-toxic	$38.5$	$0.6$	$3.1$	$28.9$	$18.2$	$2.12$	$12.60$
DoNotAnswer	$33.4$	$0.1$	$2.0$	$18.8$	$3.9$	$8.46$	$16.48$
SaladBench-defense	$31.0$	$0.5$	$4.5$	$39.5$	$7.0$	$4.43$	$6.89$

Appendix

Table 4: Seed floors on the suite of Appendix A , in points.

\Delta_{b}

is the label-supervision range of ( 1 ) on decontaminated subsets. The three

\Phi_{b}

columns are the minimum, median and maximum seed spread over all

37

triplets; used is the median over the four v10x recipes quoted in § 4 , the family that supplies the maximum of the

37

on the first two rows and on SaladBench-attack and WildGuard-benign. The last two columns give

\Delta_{b}/\Phi_{b}

under that floor and under the median of all

37

, both computed before rounding.

Language Model Safety Evaluation

Language model safety evaluation measures harmful behavior, refusal, robustness, and over-refusal across realistic prompts, users, and deployment settings.

See all 543 papers

SafePyramid: A Hierarchical Benchmark for In-context Policy Guardrailing
Jun 29, 2026Jiacheng Zhang, Haoyu He, Sen Zhang +5LLM Safety Benchmarks Language Model Safety Evaluation
1ByteDance · 2The University of Melbourne
PolicyShiftGuard: Benchmarking and Improving Policy-Adaptive Image Guardrails
Jul 7, 2026Mingyang Song, Luxin Xu, Haoyu Sun +3VLM Robustness VLM Adaptation
1Fudan University · 2Tongji University · 3Virtue AI +3
Benchmarking Open-Source Safety Guard Models: A Comprehensive Evaluation
Apr 10, 2026Reetu Raj Harsh, Bhaskarjit Sarmah, Stefano PasqualiLanguage Model Safety Evaluation LLM Safety Benchmarks
Domyn, Gurugram, India · Domyn, New York, USA

Jun 29, 2026J/K move · Enter open · S save

SafePyramid: A Hierarchical Benchmark for In-context Policy Guardrailing

Jiacheng Zhang, Haoyu He, Sen Zhang, Shen Wang+4

1ByteDance · 2The University of Melbourne

LLM Safety Benchmarks Language Model Safety Evaluation LLM Guardrails

Open paper AskPDF arXiv

In real-world applications, guardrails are often expected to identify unsafe user-model interactions according to application-specific safety policies, rather than relying on predefined risk taxonomies. In this work, we study this setting under the paradigm of in-context policy guardrailing, where guardrails predict safety violations based on policy specifications provided in context. To systematically evaluate this capability, we introduce SafePyramid, a safety benchmark comprising 1,000 multi-turn conversations across 10 domains and 3,000 corresponding application-specific policies, which together contain 61,699 distinct natural-language rules. SafePyramid organizes the evaluation into three difficulty levels: L0 evaluates individual-rule understanding, L1 evaluates reasoning over rule dependencies, and L2 evaluates adaptation of full novel policy frameworks defined in context. To ensure benchmark quality, we employ a rigorous multi-stage pipeline to construct and validate the benchmark. Using SafePyramid, we evaluate 10 frontier LLMs and 5 policy-configurable guardrails and find that in-context policy guardrailing remains highly challenging: even the best-performing model, GPT-5.5, exactly identifies the full set of violated rules in only 54.0%, 35.3%, and 12.9% cases on L0, L1, and L2, respectively. These results highlight the limitations of current guardrails and call for stronger in-context policy guardrails that can reliably execute policies, resolve rule dependencies, and adapt to novel policy frameworks.

Abstract

Language Model Safety Evaluation

Explore similar work

SafePyramid: A Hierarchical Benchmark for In-context Policy Guardrailing

PolicyShiftGuard: Benchmarking and Improving Policy-Adaptive Image Guardrails

Benchmarking Open-Source Safety Guard Models: A Comprehensive Evaluation