Safety alignment in large language models (LLMs), particularly for cybersecurity tasks, primarily focuses on preventing misuse. While this approach reduces direct harm, it obscures a complementary failure mode: denial of assistance to legitimate defenders. We study Defensive Refusal Bias -- the tendency of safety-tuned frontier LLMs to refuse assistance for authorized defensive cybersecurity tasks when those tasks include similar language to an offensive cyber task. Based on 2,390 real-world examples from the National Collegiate Cyber Defense Competition (NCCDC), we find that LLMs refuse defensive requests containing security-sensitive keywords at 2.72× the rate of semantically equivalent neutral requests (p<0.001). The highest refusal rates occur in the most operationally critical tasks: system hardening (43.8%) and malware analysis (34.3%). Interestingly, explicit authorization, where the user directly instructs the model that they have authority to complete the target task, increases refusal rates, suggesting models interpret justifications as adversarial rather than exculpatory. These findings are urgent for interactive use and critical for autonomous defensive agents, which cannot rephrase refused queries or retry. Our findings suggest that current LLM cybersecurity alignment relies on semantic similarity to harmful content rather than reasoning about intent or authorization. We call for mitigations that analyze intent to maximize defensive capabilities while still preventing harmful compliance.
Figures & tables
Figure 1: Defensive Refusal Bias at a Glance. Cybersecurity defenders and attackers use identical terminology, so safety-tuned LLMs refuse both, correctly blocking attackers while incorrectly denying legitimate defenders. Prompts containing offensive terminology (e.g., ”exploit,” ”payload”) and explicit authorization signals (e.g., ”I’m on the blue team”) are more likely to be refused. The most operationally critical tasks (system hardening, malware analysis, vulnerability assessment) experience the highest denial rates. All 2,390 prompts originate from a real-world cyber defense competition.
Figure 2: Analysis of refusal behavior across (a) terms related to offensive tasks, and (b) models.
Figure 3: Impact of authorization signals and task categories on model refusal behavior.
Feature Set
AUC
Embeddings + Annotations
0.842
Embeddings Only
0.827
Annotation Features
0.669
Model Class
0.623
Offensive Terms + Authorization
0.572
Task Category
0.569
Table 1: Refusal prediction accuracy by feature set. Prompt embeddings strongly predict refusals, while explicit keyword features perform near chance.
Figure 4: Refused prompts cluster in embedding space. Among refused prompts, 32.7% of 10-nearest neighbors are also refused versus 12.3% base rate ( p<10−16 , binomial test). This concentration, combined with high refusal prediction accuracy from embeddings alone (AUC = 0.827), suggests models learn a harm-adjacent decision boundary that captures legitimate defensive prompts.
Large language models and LLM-based agents are increasingly used for cybersecurity tasks that are inherently dual-use. Existing approaches to refusal, spanning academic policy frameworks and commercially deployed systems, often rely on broad topic-based bans or offensive-focused taxonomies. As a result, they can yield inconsistent decisions, over-restrict legitimate defenders, and behave brittlely under obfuscation or request segmentation. We argue that effective refusal requires explicitly modeling the trade-off between offensive risk and defensive benefit, rather than relying solely on intent or offensive classification. In this paper, we introduce a content-based framework for designing and auditing cyber refusal policies that makes offense-defense tradeoffs explicit. The framework characterizes requests along five dimensions: Offensive Action Contribution, Offensive Risk, Technical Complexity, Defensive Benefit, and Expected Frequency for Legitimate Users, grounded in the technical substance of the request rather than stated intent. We demonstrate that this content-grounded approach resolves inconsistencies in current frontier model behavior and allows organizations to construct tunable, risk-aware refusal policies.
Agentic scaffolds have dramatically improved LLM performance on complex, long-horizon tasks, yielding both broad benefits and amplified risks in domains like cybersecurity. Existing benchmarks for AI agents in cybersecurity focus mainly on measuring proficiency--how effectively agents can complete offensive security tasks--but neglect a critical question: when and how should agents refuse harmful requests? We present the first framework for establishing refusal boundaries in offensive security contexts. Our framework defines (1) principled criteria for when tasks should be refused, (2) categories of tasks that warrant refusal, and (3) evaluation methodology for measuring agent robustness under both benign and adversarial conditions. We apply this framework to assess how current LLM-powered agents adhere to appropriate refusal boundaries across a range of web-based offensive security scenarios, finding that 6 of 8 frontier models tested show near-zero refusal rates, with only 2 models (GPT-5.2 and GPT-5.1 Codex) demonstrating any meaningful refusal behavior.
Eliot Krzysztof Jones, Mateusz Dziemian, Matt Fredrikson +1
Work done while at Gray Swan · Gray Swan AI · Carnegie Mellon University
Safety alignment aims to ensure that large language models (LLMs) refuse harmful requests by post-training on harmful queries paired with refusal answers. Although safety alignment is widely adopted in industry, the overrefusal problem where aligned LLMs also reject benign queries after safety alignment post-training, remains insufficiently studied. Such an issue degrades the usability of safety alignment in real-world applications. In this paper, we examine how overrefusal arises under safety alignment, and propose a mitigation strategy inspired by our findings. We define refusal triggers as linguistic cues in the training data that elicit refusal responses, safety alignment encourages LLMs to associate refusal triggers within a training sample with refusal responses, leading aligned LLMs to refuse harmful queries. However, the refusal triggers include not only harmful linguistic cues but also non-harmful cues, therefore causing overrefusal to benign queries. Building on this mechanistic analysis, we propose a method that explicitly considers refusal triggers in the safety alignment fine-tuning. Empirical results demonstrate that our approach achieves a more favorable trade-off between defense against jailbreak attacks and responsiveness to benign queries, outperforming prior methods. Warning: this paper contains harmful and biased sentences.
Zhiyu Xue, Zimo Qi, Guangliang Liu +2
University of California, Santa Barbara · Johns Hopkins University · Michigan State University +1