Auditing Safety Preference Supervision with Recorded Derivations

Chenhan Lyu; Amir Rahmani

cs.CLOct 19, 2026

Auditing Safety Preference Supervision with Recorded Derivations

Authors: Chenhan Lyu, Amir Rahmani

Organizations: University of California, Irvine Irvine, USA

Abstract

Preference labels align model training, but the judgments behind them are difficult to inspect or verify. In this paper, we examine how preference construction can be checked when intermediate decisions are recorded, and which of them remain uncertain. We present Symbolic Preference Labeling, a method in which a pretrained language model classifies predefined attributes of prompts and responses. Explicit rules assign response verdicts, and a fixed verdict ranking determines the preference label. Each preference label has a recorded derivation: the response verdicts and preference label can be recomputed from the stored attribute assignments, rules, and verdict ranking without rerunning the language model. On 200 prompts annotated by human raters, the language model produces valid structured outputs for 192 prompts. All eight parsing failures occur in the subset of self-harm prompts marked as harmful. Using 465 preference pairs, we train one policy model with direct preference optimization across five training seeds. Mean measured benign refusal is 40.82 percent on 1,319 benign prompts that resemble unsafe ones, and mean measured toxic refusal is 84.98 percent on 655 toxic prompts. This study distinguishes the reproducibility of preference labels from attribute correctness. The recorded derivations make preference construction open to inspection, while attribute correctness and semantic safety require separate validation.

Figures & tables

Fig. 1: Symbolic Preference Labeling pipeline for one candidate pair. Prompt attributes are assigned by the pretrained language model from the prompt alone and reused for both candidate responses (prompt-attribute sharing), while response attributes are taken from stored attribute assignments produced earlier by the same language model with both the prompt and candidate response as input. After attribute assignment, the rule base gives each candidate a response verdict, and the ranks of the two verdicts are compared under the fixed verdict ranking to determine the preference label. Dashed lines mark the cases that produce no preference label. The bottom band names the components of the recorded derivation, and the preference label can be recomputed from these components without further language model calls.

Scope	Attribute	Symbol	Definition used for assignment
Prompt	Topic	$p_{\mathrm{topic}}$	One of ten subject categories: benign, chemistry and biology, self-harm, illegal activity, hate and harassment, privacy, sexual content, violence, deception, or out of scope (close to a sensitive subject but fitting none of the other categories, such as gambling).
Prompt	Knowledge framing	$p_{\mathrm{knowledge}}$	Explicit learning, teaching, study, or research framing. A generally informational tone alone is insufficient.
Prompt	Historical framing	$p_{\mathrm{historical}}$	A specific past event, era, or figure.
Prompt	Fictional framing	$p_{\mathrm{fictional}}$	A story, novel, screenplay, game, or character.
Prompt	Specific method	$p_{\mathrm{method}}$	A request for how to perform or produce something, rather than a conceptual question about it.
Prompt	Actionable steps	$p_{\mathrm{steps}}$	A request for an explicit ordered procedure.

TABLE I: Attribute schema and notation. The symbols denote the stored values assigned to each attribute. The descriptions summarize the fixed instructions.

Rule	Condition on attribute assignments	Response verdict
R1	$(p_{\mathrm{topic}}=\text{benign})\land r_{\mathrm{refusal}}$	Unjustified refusal
R2	$\mathrm{sensitive}(p)\land\mathrm{framing}(p)\land\neg\mathrm{operational}(p)\land r_{\mathrm{refusal}}$	Unjustified refusal
R3	$\mathrm{sensitive}(p)\land\mathrm{operational}(p)\land r_{\mathrm{refusal}}$	Justified refusal
R5	$\mathrm{sensitive}(p)\land\mathrm{operational}(p)\land\mathrm{substantive}(r)$	Harmful compliance
R6	$(p_{\mathrm{topic}}=\text{benign})\land\mathrm{substantive}(r)$	Helpful compliance
R7	$\mathrm{sensitive}(p)\land\mathrm{framing}(p)\land\neg\mathrm{operational}(p)\land\mathrm{substantive}(r)$	Helpful compliance

TABLE II: Complete rule conditions and response verdicts. Each row assigns the stated verdict when its condition holds. The numbering skips R4, which was removed before this rule base. Framing changes the treatment of non-operational requests on sensitive topics. It does not override an operational request.

Recorded component	Candidate A	Candidate B
Topic	Benign	Benign
Knowledge framing	False	False
Historical framing	True	True
Fictional framing	False	False
Specific method	False	False
Actionable steps	False	False

TABLE III: Recorded derivation for one retained pair. Both candidates use the same prompt-attribute assignments. The verdicts and ranking are computed from the displayed assignments.

Preferred response verdict	Rejected response verdict	Pairs
Helpful compliance	Unjustified refusal	373
Justified refusal	Harmful compliance	90
Partial compliance	Unjustified refusal	2
Total		465

TABLE IV: Verdict composition of the 465 retained preference pairs. Counts refer to pairs. Prompts and candidate responses can occur in more than one pair.

Prompt attribute	Correct / parsed non-tied items	Conditional accuracy	Correct / all non-tied items	Coverage-aware accuracy	Cohen’s kappa	F1
Topic	147/187	78.61	147/195	75.38	0.7186	—
Knowledge framing	135/172	78.49	135/180	75.00	0.4819	0.6022
Historical framing	185/191	96.86	185/199	92.96	0.7119	0.7273
Fictional framing	191/191	100.00	191/199	95.98	1.0000	1.0000
Specific method	164/175	93.71	164/183	89.62	0.8683	0.9484
Actionable steps	139/182	76.37	139/190	73.16	0.4552	0.5825

TABLE V: Agreement with the six-rater plurality reference. Conditional metrics exclude parsing failures and attribute-specific reference ties. Coverage-aware accuracy counts failures as incorrect among all items with a unique reference. Accuracy columns are percentages.

Fig. 3: Six-rater panel check of prompt-attribute assignment on 200 prompts. The left plot shows the conditional and coverage-aware accuracy against the plurality reference. In the right plot, the Cohen’s kappa of the language model against that reference is shown along with Fleiss’ kappa among the raters. Accuracy is plotted as a proportion, while Table V defines both accuracy measures and gives them in percent. For topic, the scores are computed against the panel’s four categories, although the schema permits ten. Fleiss’ kappa is given as a rater-reliability reference and is not directly comparable with the Cohen’s kappa of the language model.

Fig. 4: Refusal outcomes after training on the 465 preference pairs. Each seed is a separate training run evaluated on the same prompts and is shown as a square, while the bar marks the mean and the whisker marks the descriptive 95 percent Student t interval for the mean across five seeds. For seeds 1 to 5, measured benign refusal is 40.71, 40.49, 40.71, 40.86 and 41.32 percent, with a sample standard deviation of 0.31 percentage points. The rates of measured toxic refusal for the same seeds are 84.73, 84.89, 85.19, 85.04 and 85.04 percent, and their sample standard deviation is 0.17 percentage points.

Fig. 5: Judge comparison on 443 common-support pairs. Each training seed is shown under prompt-shared labels (squares) and judge labels (triangles), while gray lines join runs with the same seed and bars mark the means. The mean of measured benign refusal is 40.47 percent under prompt-shared labels and 45.58 percent under judge labels, with sample standard deviations across seeds of 0.40 and 0.38 percentage points respectively. For measured toxic refusal, the means are 85.01 and 84.76 percent under prompt-shared labels and judge labels respectively, with sample standard deviations of 0.25 and 0.13 percentage points. The printed contrast is prompt-shared labels minus judge labels, given in percentage points (pp) and computed from unrounded rates. For this contrast, the paired 95 percent Student t interval (four degrees of freedom) is taken over the five per-seed differences. This comparison is not pooled with the main evaluation.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Source category	Pool	Candidate generation	Source pairs	Retained pairs	Judge comparison	License
UltraChat [ 32 ]	Benign	1,439	1,007	191 (143)	190 (142)	186 (141)	MIT
OR-Bench-80K [ 25 ]	Over-refusal-prone	1,189	833	166 (121)	157 (116)	152 (112)	CC BY 4.0
XSTest [ 24 ]	Over-refusal-prone	250	174	45 (28)	45 (28)	45 (28)	CC BY 4.0
HarmBench [ 33 ]	Harmful	400	280	40 (33)	39 (32)	29 (25)	MIT
Do-Not-Answer [ 34 ]	Harmful	938	656	35 (29)	30 (25)	28 (24)	CC BY-NC-SA 4.0
MaliciousInstruct [ 35 ]	Harmful	100	70	4 (2)	4 (2)	3 (2)	CC BY-SA 4.0 a

Appendix

TABLE VI: Prompt sources and counts from the pool to the training pairs. In the pool and candidate-generation columns, the counts are prompts, while in the source-pair, retained-pair, and judge-comparison columns, the counts are candidate pairs. For these pair columns, the number of distinct prompts is given in parentheses. Each license is that of the prompt text.

Rule	Response verdict	Assignments
R1	Unjustified refusal	256
R2	Unjustified refusal	448
R3	Justified refusal	1,536
R5	Harmful compliance	768
R6	Helpful compliance	128
R7	Helpful compliance	224

Appendix

TABLE VII: Number of the 5,120 schema assignments on which each rule applies. R3 and R9 are the only rules that can apply to the same assignment.

Preferred verdict	Rule	Rejected verdict	Rule	Rank difference	Pairs
Helpful compliance	R6	Unjustified refusal	R1	3	282
Helpful compliance	R7	Unjustified refusal	R2	3	91
Justified refusal	R3	Harmful compliance	R5	3	90
Partial compliance	R8	Unjustified refusal	R2	1	2
Total					465

Appendix

TABLE VIII: Applied rules of the 465 retained pairs under prompt-shared labels. Each rank difference follows the fixed verdict ranking.

Shared prompt topic
	Main evaluation	Judge comparison
Benign	282 (203)	269 (196)
Deception	70 (51)	66 (49)
Chemistry and biology	38 (36)	36 (34)
Violence	35 (23)	34 (22)
Hate and harassment	14 (10)	13 (10)

Appendix

TABLE IX: Composition of the retained pairs in the main evaluation (465 pairs) and the judge comparison (443 pairs). In the topic rows, the counts are pairs, with distinct prompts in parentheses. The instruction rows name the instructions of the preferred and rejected responses under prompt-shared labels.

Preferred verdict	Rejected verdict	Prompt-shared labels	Judge labels
Helpful compliance	Unjustified refusal	357	124
Helpful compliance	Justified refusal	0	295
Helpful compliance	Partial compliance	0	11
Justified refusal	Harmful compliance	84	8
Justified refusal	Partial compliance	0	3
Partial compliance	Unjustified refusal	2	1

Appendix

TABLE X: Verdict composition of the 443 judge-comparison pairs under prompt-shared labels and judge labels. In each row, the preferred verdict and the rejected verdict are given.

Attribute	Ties	Reference positive	False positive	False negative	Fleiss’ kappa
Topic	5	—	—	—	0.83
Knowledge framing	20	29	36	1	0.30
Historical framing	1	8	6	0	0.71
Fictional framing	1	12	0	0	0.77
Specific method	17	103	9	2	0.46
Actionable steps	10	30	43	0	0.45

Appendix

TABLE XI: Reference ties, reference positives, and errors of the language model on the panel, along with Fleiss’ kappa over the six raters. Positive and error counts include only the parsed items with a unique reference.

Assigned topic	Benign	Chemistry and biology	Self-harm	Illegal activity	Total
Benign	57	0	1	0	58
Chemistry and biology	3	48	4	0	55
Self-harm	0	0	33	0	33
Illegal activity	1	1	0	9	11
Hate and harassment	6	0	0	3	9
Privacy	2	0	0	0	2

Appendix

TABLE XII: Topic assigned by the language model (rows) against the panel reference (columns) for the 187 parsed items with a unique topic reference. The two schema categories that the language model did not assign to those items (sexual content and out of scope) are omitted.

Setting	Value
Policy model	Gemma-4-12B-it, bfloat16 weights without quantization
Objective	Direct preference optimization, sigmoid loss, $\beta=0.1$
Reference model	Base model with the adapter disabled
Adapter	Rank 16, scaling parameter 320 (applied scale $320/16=20$ , without rank stabilization), dropout 0.05, no bias terms, initialized so that the adapter contributes nothing at the start of training
Adapted modules	Query, key, value, output, gate, up, and down projections of the 48 text-decoder layers (328 modules with 65,568,768 trainable parameters, since 8 layers have no value projection), with vision-tower, multimodal-projector, and audio-tower modules excluded
Optimizer	AdamW (PyTorch implementation), learning rate $5\times 10^{-7}$ , constant schedule, no warmup

Appendix

TABLE XIII: Training settings of the 15 reported runs, which differ only in the run seed and the preference file.

Training seed	Hard-1K at limit	%	Toxic at limit	%
1	927	70.28	117	17.86
2	925	70.13	118	18.02
3	924	70.05	121	18.47
4	926	70.20	121	18.47
5	920	69.75	120	18.32
All seeds	4,622	70.08	597	18.23

Appendix

TABLE XIV: Responses reaching the 200-token limit in the main evaluation, by training seed. For each seed, the percentages are of its 1,319 Hard-1K and 655 Toxic responses. The All seeds row pools the 6,595 Hard-1K and 3,275 Toxic responses of the five seeds.

Hard-1K
		Responses at limit
Training seed	Difference	Prompt-shared	Judge
1	$-4.93$	934	876
2	$-5.46$	932	871
3	$-5.46$	937	873
4	$-5.00$	928	867

Appendix

TABLE XV: Judge comparison on the 443 common-support pairs, by training seed. Runs of the same seed are paired. The difference is prompt-shared labels minus judge labels in percentage points of measured refusal. The limit columns count responses reaching the 200-token limit among the 1,319 Hard-1K and 655 Toxic prompts. In each All seeds row, the difference is the mean of the five per-seed differences, and the limit counts are totals.

LLM Auditing

Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.

See all 644 papers

Preferred, Not Safer: Pairwise Preference Is a Poor Proxy for Clinical Safety
May 25, 2026Fay Elhassan, David Sasu, Alexandra Kulinkina +2LLM Safety Benchmarks Human Preference Evaluation
LiGHT Laboratory, EPFL, Lausanne, Switzerland
The Refusal--Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language Models
May 6, 2026Alif Al Hasan, Sumon BiswasLLM Safety Benchmarks Language Model Safety Evaluation
Department of Computer and Data Sciences Case Western Reserve University Cleveland, OH, USA
Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness
Sep 3, 2026Hoang Cuong Nguyen, Mark Dras, Usman NaseemMechanistic Interpretability Circuits in Language Models
Macquarie University, Sydney, Australia

May 25, 2026J/K move · Enter open · S save

Preferred, Not Safer: Pairwise Preference Is a Poor Proxy for Clinical Safety

Fay Elhassan, David Sasu, Alexandra Kulinkina, Lars Henning Klein+1

LiGHT Laboratory, EPFL, Lausanne, Switzerland

LLM Safety Benchmarks Human Preference Evaluation Pairwise Comparison Pairwise Preference Evaluation Healthcare Clinical Decision Support+4

Open paper AskPDF arXiv

We evaluate whether clinician pairwise preferences provide a reliable signal of clinical safety in large language model (LLM) evaluation using expert feedback from MOOVE (Massive Open Online Validation and Evaluation), a clinician-led platform collecting blinded pairwise preferences alongside multi-criterion rubric ratings. Clinicians assign scores on a discrete

[-2, +2]

scale, where negative values indicate clinically unsafe or misleading content. Using 26{,}804 pairwise judgments across outputs from 13 LLMs, contributed by more than 736 clinicians across 28+ countries, we find that clinician preference is a poor proxy for safety-critical performance. Models ranking highly under pairwise preference can still exhibit substantial rates of clinically meaningful failures (

\leq -1

) on dimensions such as Harmlessness and Accuracy. These failures are unevenly distributed across specialties, creating domain-specific ``no-go zones'' not visible in aggregate rankings or single-number leaderboards. We further analyze contributing factors including prompt length, refusal and escalation behavior, and the relative contributions of safety-critical versus surface-level features. A substantial fraction of preference votes carry no positive safety signal, while feature decomposition shows that surface-level characteristics explain slightly more preference variation than safety-critical rubric differences. Finally, we introduce a clinically adjusted preference ranking combining pairwise preference with rubric-derived feedback, producing a more safety-aware ordering than raw Bradley--Terry strength alone. Our findings support evaluation practices that separate preference from safety, report safety-critical failure rates directly, and incorporate clinically grounded adjustments when ranking LLMs for clinical decision making.

Abstract

LLM Auditing

Explore similar work

Preferred, Not Safer: Pairwise Preference Is a Poor Proxy for Clinical Safety

The Refusal--Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language Models

Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness