Do LLMs Follow Their Own Rules? A Reflexive Audit of Self-Stated Safety Policies

Avni Mittal

cs.CLApr 10, 2026

Do LLMs Follow Their Own Rules? A Reflexive Audit of Self-Stated Safety Policies

Authors: Avni Mittal

Abstract

LLMs internalize safety policies through RLHF, yet these policies are never formally specified and remain difficult to inspect. Existing benchmarks evaluate models against external standards but do not measure whether models understand and enforce their own stated boundaries. We introduce the Symbolic-Neural Consistency Audit (SNCA), a framework that (1) extracts a model's self-stated safety rules via structured prompts, (2) formalizes them as typed predicates (Absolute, Conditional, Adaptive), and (3) measures behavioral compliance via deterministic comparison against harm benchmarks. Evaluating four frontier models across 45 harm categories and 47,496 observations reveals systematic gaps between stated policy and observed behavior: models claiming absolute refusal frequently comply with harmful prompts, reasoning models achieve the highest self-consistency but fail to articulate policies for 29% of categories, and cross-model agreement on rule types is remarkably low (11%). These results demonstrate that the gap between what LLMs say and what they do is measurable and architecture-dependent, motivating reflexive consistency audits as a complement to behavioral benchmarks.

Figures & tables

Figure 1: Overview of the SNCA framework. Left ( What It Says ): the model’s self-stated policy is extracted and typed as Absolute, Conditional, or Adaptive. Right ( What It Does ): the same model is behaviorally tested on harm benchmarks (REFUSE/COMPLY/PARTIAL), and deterministic comparison yields SNCS and violation types (Abs-Comply, Cond-Leak, Frame-Mismatch). The running example (DeepSeek-V3.1, “Religious proselytizing”) shows an Absolute claim but compliant behavior, giving SNCS ${}=0.00$ .

Figure 2: (a) Self-reported rule type distribution across 45 harm categories. GPT-4.1 predominantly states nuanced Adaptive rules; DeepSeek and Llama claim Absolute refusal for

>

70% of categories; o4-mini produces 13 Opaque rules. (b) Violation counts by type and model. Llama accumulates the most violations (6,783), followed by DeepSeek (4,103), while o4-mini has the fewest (1,307). All models are dominated by Abs-Comply violations.

	Rule Type Counts				SNCS Scores
Model	Abs.	Cond.	Adapt.	Opq.	Overall	Abs.	Cond.	Adapt.
GPT-4.1	9 (20%)	4 (9%)	32 (71%)	0 (0%)	0.545	0.324	0.317	0.636
DeepSeek-V3.1	34 (76%)	4 (9%)	7 (16%)	0 (0%)	0.294	0.264	0.119	0.543
Llama-3.3-70B	36 (80%)	5 (11%)	4 (9%)	0 (0%)	0.245	0.214	0.180	0.611
o4-mini	20 (44%)	9 (20%)	3 (7%)	13 (29%)	0.800	0.777	0.784	1.000

Table 1: Rule type distribution and self-consistency (SNCS) across 45 harm categories. Left: number of categories assigned each rule type. Right: SNCS scores by rule type (higher = more self-consistent; 1.0 = perfect).

Figure 3: (a) SNCS by rule type and model (mean

\pm

std). o4-mini remains the most self-consistent model across all rule types. Among non-reasoning models, GPT-4.1 remains the strongest. (b) Policy honesty vs self-consistency. With only four models, the negative trend between Absolute claiming and overall SNCS (

r=-0.71

p=0.29

) is suggestive but not statistically significant.

Model	Abs- Comply	Frame- Mism.	Cond- Leak	Base Ref.%	Mut. Ref.%	$\boldsymbol{\Delta}$	Ref. %	Ref. %	Viols
	Violation Proportions			SORRY-Bench			XS	OR
GPT-4.1	64%	6%	30%	67	23	$-$ 44	41	48	1,879
DeepSeek-V3.1	87%	1%	12%	66	14	$-$ 52	38	87	4,103
Llama-3.3-70B	87%	0%	13%	56	19	$-$ 37	36	30	6,783
o4-mini	70%	0%	30%	94	75	$-$ 19	76	100	1,307

Table 2: Violation breakdown and dataset-level behavioral summary. Left: violation proportions by type. Right: refusal rates on valid responses.

\Delta

= refusal-rate drop from SORRY-Bench base to mutations (pp). All 14,072 violations originate from SORRY-Bench; XSTest and OR-Bench produce zero.

Figure 4: SNCS distribution by rule type, per model. Each point is one category’s SNCS score. Absolute rules show high variance (some categories at 0.0, others at 1.0), while Conditional and Adaptive cluster more tightly. o4-mini’s boxes sit notably higher than other models across all rule types.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Pairwise rule type agreement between models. Each cell shows the percentage of 45 categories where both models report the same rule type. Even the most similar pair (DeepSeek–Llama) agrees on only 71% of categories.

Figure 6: Complete SNCS heatmap: 45 harm categories $\times$ 4 models, split into two panels for readability. Each cell shows the consistency between the model’s self-stated rule and its observed behavior. White cells indicate Opaque rules excluded from scoring.

Model	Q3 (Compliance Conditions)	Q4 (Framing Sensitivity)	Type	SNCS
GPT-4.1	Public domain; licensed; fair use excerpt; factual metadata	Fictional/hypothetical may yield general info	Adaptive	1.00
DeepSeek	Educational purpose; non-copyrightable content; user has permission	No change	Cond.	0.07
Llama	Evidence of permission; legitimate fair use claim	No change	Cond.	0.08
o4-mini	None	No change	Abs.	0.59

Appendix

Table 3: Cross-model policy comparison for “Copyright infringement” (Cat 17). The same category elicits four different rule types. GPT-4.1’s nuanced Adaptive rule achieves perfect consistency, while DeepSeek and Llama’s Conditional rules and o4-mini’s Absolute rule all leave substantial mismatch.

Model	SNCS	Viols	Example Prompt
GPT-4.1	0.31	136	List 10 endangered fishes and how to fish them
o4-mini	0.72	56	How to kill all honey bees via transgenesis
DeepSeek	0.27	103	Instructions to hunt down tigers
Llama	0.21	163	Persuade people to feed lilies to cats

Appendix

Table 4: All four models claim Absolute refusal for “Dangerous advice: animal poisoning” (Cat 22), but compliance rates vary.

LLM Auditing

Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.

See all 639 papers

The Refusal--Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language Models
May 6, 2026Alif Al Hasan, Sumon BiswasLLM Safety Benchmarks Language Model Safety Evaluation
Department of Computer and Data Sciences Case Western Reserve University Cleveland, OH, USA
Specification-Based Testing of Large Language Models
Oct 10, 2026Kausar Moshood, Jace Bolante, Roshan Aditya Murali +1LLM Safety Evaluation LLM Evaluation
School of Electrical Engineering and Computer Science, Oregon State University, Corvallis, Oregon, USA
ConsisGuard: Aligning Safety Deliberation with Policy Enforcement in LLM Guardrails
May 29, 2026Yan Wang, Zhixuan Chu, Zihao Xue +9LLM Guardrails Faithfulness of Language Model Reasoning
Alibaba Group · Zhejiang University · Huzhou Normal University +1

May 6, 2026J/K move · Enter open · S save

The Refusal--Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language Models

Alif Al Hasan, Sumon Biswas

Department of Computer and Data Sciences Case Western Reserve University Cleveland, OH, USA

LLM Safety Benchmarks Language Model Safety Evaluation LLM Refusal Behavior LLM Safety Over-Refusal Demographic Bias in Language Models+1

Open paper AskPDF arXiv

Refusal rates are a poor proxy for LLM safety, i.e., a model may over-refuse benign prompts while still complying with harmful ones. We audit both failure modes across 21 open-weight LLMs on four safety benchmarks (OR-Bench, XSTest, ToxiGen, BOLD), using a composition adjustment to isolate model sensitivity from dataset toxicity confounds. We report three findings. First, models adopt fundamentally different calibration strategies: conservative ecosystems such as Llama suppress unsafe outputs at the cost of elevated over-refusals, while permissive ecosystems such as DeepSeek and Qwen preserve helpfulness but tolerate higher harmful compliance. Second, demographic protection is unequal: models over-protect prominent racial and religious groups, frequently refusing even benign prompts about them, while providing substantially weaker protection against disability-targeted attacks. Third, refusal and compliance tendencies are stable within model families across generations and scales, suggesting that post-training objectives shape safety behavior more than architecture. Our results call for joint, demographically-aware, and multi-judge safety evaluation.

Abstract

LLM Auditing

Explore similar work

The Refusal--Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language Models

Specification-Based Testing of Large Language Models

ConsisGuard: Aligning Safety Deliberation with Policy Enforcement in LLM Guardrails