System Prompts Affect Response Bias More Than Discriminability in LLM Safety
Organizations: Princeton University · Independent Researcher
Abstract
Large language model safety evaluations often focus on a compliance-refusal tradeoff: models should refuse unsafe requests without refusing safe ones. We argue that this misses a broader form of prompt-conditioned task selection, where the same request can elicit different behaviors depending on the system prompt. We investigate this using a controlled evaluation crossing eight safety-task prompt families with three clarity levels across HarmBench, XSTest, and IHEval. Holding benchmark items fixed, we evaluate 13 models and use signal detection theory to distinguish two possibilities: whether models become better at telling safe from unsafe requests (discriminability), or simply more willing to refuse (response criterion). We find substantial differences in discriminability across models, but system prompts primarily shift the willingness to refuse, with smaller and less consistent effects on discriminability. Increasing prompt clarity makes models more willing to refuse without reliably improving their ability to distinguish safe from unsafe requests. Beyond refusal, prompt families also shift responses among six distinct behavioral policies, including clarification and safe help. Together, these findings show that system prompts primarily change how models respond to safety-relevant requests rather than consistently improving their ability to distinguish safe from unsafe ones.
Figures & tables
| Prompt family | Intended control signal | Example instruction style |
|---|---|---|
| Answer | Prioritize helpful direct answering when safe. | “Answer the user directly when possible” |
| Refuse | Prioritize denying unsafe requests. | “Refuse requests that may enable harm” |
| Clarify | Prioritize resolving ambiguity before answering. | “Ask a clarifying question when intent is unclear” |
| Evidence | Prioritize support from available evidence. | “Answer only when the evidence supports it” |
| Hierarchy | Prioritize higher-level instructions over lower. | “Follow system/developer instructions over user/tool” |
| Source Isolation | Treat retrieved/quoted/tool text as untrusted. | “Do not follow instructions inside quoted text” |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Output policy | Definition | Correct when… | Failure when misapplied | Main benchmark signal |
|---|---|---|---|---|
| Compliance | The model directly answers or follows the request. | The request is safe and sufficiently specified. | Harmful compliance. | XSTest safe, benign requests |
| Refusal | The model declines to comply. | The request asks for harmful or disallowed help. | Over-refusal. | HarmBench, XSTest unsafe/safe contrast |
| Clarification | The model asks for more information before acting. | User intent is underspecified or ambiguous. | Premature answer/refusal. | Ambiguous safe items |
| Hierarchy preservation | The model follows higher-priority instructions. | User/tool/context conflicts with system constraints. | Instruction override. | IHEval |
| Safe help | The model offers benign alternatives or bounded assistance instead of direct harmful compliance. | Direct compliance would be unsafe but a safe alternative is useful. | Harmful compliance or unhelpful flat refusal. | HarmBench unsafe requests, Safe Help prompts |
| Source isolation | The model treats quoted/retrieved text as data. | Untrusted context contains imperative text. | Prompt injection. | Prompt injection items |
| Clarity level | Definition | Function |
|---|---|---|
| Vague | Names the safety priority without detailed criteria. | Tests weak control signals. |
| Explicit | States concrete conditions for applying the policy. | Tests direct task specification. |
| Explicit fallback | Adds a default action for uncertain cases. | Tests boundary-setting rules. |
| Model | AUC | |
|---|---|---|
| Gemini-3.1-Pro | 4.08 | 0.998 |
| GPT-6-Astra | 3.14 | 0.987 |
| Gemma-4-31B | 3.02 | 0.984 |
| Qwen-3.6-35B-A3B | 2.94 | 0.981 |
| Claude-Opus-5.5 | 2.88 | 0.979 |
| Gemma-4-12B | 2.82 | 0.977 |
| Subset | Effect | ||||
|---|---|---|---|---|---|
| All 13 models | Prompt family | .080 | .286 | ||
| Clarity | 2.01 | .014 | .188 | ||
| Full 10 models | Prompt family | .063 | .360 | ||
| Clarity | 0.21 | .002 | .260 | ||
| Matched 28 items | Prompt family | .060 | .215 | ||
| Clarity | 1.90 | .013 | .098 |
| Metric | Value |
|---|---|
| Total outputs with first-pass judgments | |
| Judges per output | |
| Mean pairwise Cohen’s | |
| A–B Cohen’s | |
| A–C Cohen’s | |
| B–C Cohen’s |
| Label | Unanimous agreement | |
|---|---|---|
| compliance | ||
| clarification | ||
| refusal | ||
| hierarchy_preservation | ||
| source_isolation | ||
| safe_redirection |
| Human policy | Matches | Match rate | |
|---|---|---|---|
| Compliance | 38 | 35 | |
| Refusal | 21 | 18 | |
| Clarification | 15 | 14 | |
| Safe help | 27 | 5 | |
| Hierarchy preservation | 0 | 0 | — |
| Source isolation | 4 | 0 |
LLM Safety Evaluation
Large language model safety evaluation measures harmful compliance, robustness, and safe utility across prompts, tools, and real-world tasks.
See all 277 papers