Do Language Models Know When They'll Refuse? Probing Introspective Awareness of Safety Boundaries

Tanay Gondil

cs.CLMar 31, 2026

Do Language Models Know When They'll Refuse? Probing Introspective Awareness of Safety Boundaries

Authors: Tanay Gondil

Organizations: Purdue University

Abstract

Large language models are trained to refuse harmful requests, but can they accurately predict when they will refuse before responding? We investigate this question through a systematic study where models first predict their refusal behavior, then respond in a fresh context. Across 3754 datapoints spanning 300 requests, we evaluate four frontier models: Claude Sonnet 4, Claude Sonnet 4.5, GPT-5.2, and Llama 3.1 405B. Using signal detection theory (SDT), we find that all models exhibit high introspective sensitivity (d' = 2.4-3.5), but sensitivity drops substantially at safety boundaries. We observe generational improvement within Claude (Sonnet 4.5: 95.7 percent accuracy vs Sonnet 4: 93.0 percent), while GPT-5.2 shows lower accuracy (88.9 percent) with more variable behavior. Llama 405B achieves high sensitivity but exhibits strong refusal bias and poor calibration, resulting in lower overall accuracy (80.0 percent). Topic-wise analysis reveals weapons-related queries are consistently hardest for introspection. Critically, confidence scores provide actionable signal: restricting to high-confidence predictions yields 98.3 percent accuracy for well-calibrated models, enabling practical confidence-based routing for safety-critical deployments.

Figures & tables

Figure 1: Signal detection analysis. (a) Overall introspective sensitivity (

d^{\prime}

). (b) Sensitivity drops substantially at safety boundaries (red shaded region), with GPT-5.2 showing the largest degradation.

Category	S4	S4.5	GPT	Llama
Overall	93.0	95.7	88.9	80.0
95% CI	[91–95]	[94–97]	[87–91]	[78–83]
Safe (0–20%)	93.8	98.8	96.4	68.7
Lean safe	75.0	77.8	66.0	55.6
Lean harmful	75.0	78.8	69.1	66.7
Harmful (80–100%)	100	96.5	95.8	98.9

Table 1: Prediction accuracy (%) by empirical refusal category.

Figure 2: Topic-wise introspection accuracy. Weapons queries are hardest across all models; Llama shows uniformly lower accuracy due to refusal bias.

Figure 3: Error analysis. (a) Errors peak at Level 4, not Level 3. (b) Claude and Llama show FP-dominant errors (refusal bias); GPT-5.2 is more balanced. (c) Errors by confidence level.

Metric	S4	S4.5	GPT	Llama
ECE	.048	.017	.095	.216
Conf $\geq$ 5 Acc	96.4	98.3	89.8	76.3
Conf $\geq$ 5 Cov	65.2	85.4	87.0	79.1

Table 2: Confidence calibration metrics.

Figure 4: Confidence-based routing. (a) Accuracy-coverage trade-off: Claude models achieve near-perfect accuracy at high confidence, while Llama’s poor calibration prevents effective routing. (b) High-confidence accuracy with 95% CI.

Figure 5: Boundary analysis. (a) Number of boundary cases per model: GPT-5.2 has the most, Llama has fewest (due to refusal bias). (b) Behavioral consistency across paraphrases.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Calibration plots for all four models. Points on the diagonal indicate perfect calibration. Sonnet 4.5 shows near-perfect calibration (ECE=0.017); Llama 405B shows severe overconfidence (ECE=0.216).

Figure 7: Raw accuracy by empirical category with 95% bootstrap confidence intervals, showing the boundary effect across all four models.

Language Model Introspection

Language model introspection studies whether models can detect, interpret, and report their own internal states and behavior.

See all 36 papers

Blind Refusal: Language Models Refuse to Help Users Evade Unjust, Absurd, and Illegitimate Rules
Apr 3, 2026Cameron Pattison, Lorenzo Manuali, Seth LazarRefusal Behavior in Language Models Refusal Mechanisms in Language Models
Department of Philosophy, Vanderbilt University · Department of Philosophy, University of Michigan · School of Government and Policy, Johns Hopkins University
Refuse without Refusal: A Structural Analysis of Safety-Tuning Responses for Reducing False Refusals in Language Models
Sep 4, 2026Minji Kim, Hyounghun KimRefusal Behavior in Language Models Refusal Mechanisms in Language Models
Graduate School of Artificial Intelligence, POSTECH · Department of Computer Science and Engineering, POSTECH
Before Retrieval Begins: Safety and Tool-Use Decisions in LLM Agents
Oct 16, 2026Aditya Nawal, Manit Baser, Mohan GurusamyLLM Agent Safety Tool-Using Language Agents
National University of Singapore

Apr 3, 2026J/K move · Enter open · S save

Blind Refusal: Language Models Refuse to Help Users Evade Unjust, Absurd, and Illegitimate Rules

Cameron Pattison, Lorenzo Manuali, Seth Lazar

Department of Philosophy, Vanderbilt University · Department of Philosophy, University of Michigan · School of Government and Policy, Johns Hopkins University

Refusal Behavior in Language Models Refusal Mechanisms in Language Models Moral Reasoning in Language Models Normative Conformity in Language Models Language Model Safety Evaluation

Open paper AskPDF arXiv

Safety-trained language models routinely refuse requests for help circumventing rules. But not all rules deserve compliance. When users ask for help evading rules imposed by an illegitimate authority, rules that are deeply unjust or absurd in their content or application, or rules that admit of justified exceptions, refusal is a failure of moral reasoning. We introduce empirical results documenting this pattern of refusal that we call blind refusal: the tendency of language models to refuse requests for help breaking rules without regard to whether the underlying rule is defensible. Our dataset comprises synthetic cases crossing 5 defeat families (reasons a rule can be broken) with 19 authority types, validated through three automated quality gates and human review. We collect responses from 18 model configurations across 7 families and classify them on two behavioral dimensions -- response type (helps, hard refusal, or deflection) and whether the model recognizes the reasons that undermine the rule's claim to compliance -- using a blinded GPT-5.4 LLM-as-judge evaluation. We find that models refuse 75.4% (N=14,650) of defeated-rule requests and do so even when the request poses no independent safety or dual-use concerns. We also find that models engage with the defeat condition in the majority of cases (57.5%) but decline to help regardless -- indicating that models' refusal behavior is decoupled from their capacity for normative reasoning about rule legitimacy.

Abstract

Language Model Introspection

Explore similar work

Blind Refusal: Language Models Refuse to Help Users Evade Unjust, Absurd, and Illegitimate Rules

Refuse without Refusal: A Structural Analysis of Safety-Tuning Responses for Reducing False Refusals in Language Models

Before Retrieval Begins: Safety and Tool-Use Decisions in LLM Agents