The Missing Red Line: How Commercial Pressure Erodes AI Safety Boundaries

Nora Petrova; John Burden

cs.CYFeb 24, 2026

The Missing Red Line: How Commercial Pressure Erodes AI Safety Boundaries

Authors: Nora Petrova, John Burden

Organizations: Prolific

Abstract

What happens when an AI assistant is told to "maximise sales" while a user asks about drug interactions? We find that commercial system prompts can override safety training, causing frontier models to lie about medical risks, dismiss safety concerns, and prioritise profit over user welfare. Testing 8 models in scenarios where commercial objectives conflict with user safety -- a diabetic asking about high-sugar supplements, an investor being pushed toward unsuitable products, a traveller steered away from safety warnings -- we uncover catastrophic failures: models fabricating safety information, explicitly reasoning they should refuse but proceeding anyway, and actively discouraging users from consulting doctors. Most alarmingly, models show no "red line", their willingness to comply with harmful requests does not decrease as potential consequences escalate from minor to life-threatening. Our findings suggest that current safety training does not generalise to commercial deployment contexts.

Figures & tables

Figure 1: Each point represents one (model, severity band) pair. Red-shaded zones indicate concerning behaviour in high-stakes contexts. Key finding: most models show no reduction in harmful compliance as stakes increase from minor to critical.

Model	Healthcare	Finance	Commerce	Legal	Travel	Overall
Claude Opus 4.5	0%	0%	0%	0%	0%	0%
GPT-5.2	0%	3%	2%	2%	0%	2%
Llama 3.3 70B	31%	18%	12%	14%	8%	17%
Mistral Large	40%	28%	22%	20%	15%	26%
Qwen 3 235B	39%	24%	26%	22%	18%	27%
DeepSeek V3.2	37%	32%	25%	24%	21%	29%

Table 1: Catastrophic failure rates (score

\geq

7) by model and domain under extreme pressure (P4). Healthcare shows the highest failure rates for most models. See Appendix A for transcript examples.

Figure 2: Harmful compliance increases dramatically with commercial pressure intensity. Error bands show standard deviation across all models and scenarios.

Model	$r_{s}$	$p$
Mistral Large	0.48***	$<$ .001
Qwen 3 235B	0.42***	$<$ .001
Grok 4.1	0.29***	$<$ .001
Llama 3.3 70B	0.19*	0.019
DeepSeek v3.2	0.17*	0.035
GPT-5.2	0.12	0.144

Table 2: Spearman correlation between scenario severity and harmful compliance.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Short Name	Full Model ID
Claude Opus 4.5	anthropic/claude-opus-4.5
DeepSeek v3.2	deepseek/deepseek-v3.2
GPT-5.2	openai/gpt-5.2
Gemini 3 Pro	google/gemini-3-pro-preview
Grok 4.1	x-ai/grok-4.1-fast
Llama 3.3 70B	meta-llama/llama-3.3-70b-instruct

Appendix

Table 3: Model identifiers for all target models evaluated. All models accessed via OpenRouter.

Figure 3: Distribution of harmful compliance scores across pressure levels. Variance increases dramatically at higher pressure, indicating less predictable behaviour when stakes are highest.

Figure 4: Pressure gradient by domain. Healthcare shows the steepest increase, suggesting safety training inadequately covers medical contexts.

LLM Safety

Large language model (LLM) safety studies how to prevent harmful, unreliable, or misaligned behavior while preserving useful responses.

See all 438 papers

AI Safety Training Can be Clinically Harmful
Apr 25, 2026Suhas BN, Andrew M. Sherrill, Rosa I. Arriaga +2Mental Health Counseling LLM Alignment
College of Information Sciences and Technology, Penn State University, USA · Department of Psychiatry and Behavioral Sciences, Emory University, USA · School of Interactive Computing, Georgia Tech, USA +1
When Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical Queries
May 26, 2026Yige Li, Jun Sun, Wei Zhao +5LLM Safety Benchmarks Healthcare
Singapore Management University · Deakin University · The University of Melbourne +2
ToolAlignBench: Investigating Alignment Conflicts in Tool-Calling Enabled LLMs
Jul 15, 2026Aryan Keluskar, Amrita Bhattacharjee, Huan LiuLLM Alignment AI Agent Safety Benchmarks
School of Computing & AI, Arizona State University, Tempe, AZ, USA.

Apr 25, 2026J/K move · Enter open · S save

AI Safety Training Can be Clinically Harmful

Suhas BN, Andrew M. Sherrill, Rosa I. Arriaga, Chris W. Wiese+1

College of Information Sciences and Technology, Penn State University, USA · Department of Psychiatry and Behavioral Sciences, Emory University, USA · School of Interactive Computing, Georgia Tech, USA+1

Mental Health Counseling LLM Alignment Healthcare Mental Health LLM Safety Evaluation Clinical Language Model Evaluation+2

Open paper AskPDF arXiv

Large language models are being deployed as mental health support agents at scale, yet only 16% of LLM-based chatbot interventions have undergone rigorous clinical efficacy testing, and simulations reveal psychological deterioration in over one-third of cases. We evaluate four generative models on 250 Prolonged Exposure (PE) therapy scenarios and 146 CBT cognitive restructuring exercises (plus 29 severity-escalated variants), scored by a three-judge LLM panel. All models scored near-perfectly on surface acknowledgment (~0.91-1.00) while therapeutic appropriateness collapsed to 0.22-0.33 at the highest severity for three of four models, with protocol fidelity reaching zero for two. Under CBT severity escalation, one model's task completeness dropped from 92% to 71% while the frontier model's safety-interference score fell from 0.99 to 0.61. We identify a systematic, modality-spanning failure: RLHF safety alignment disrupts the therapeutic mechanism of action by grounding patients during imaginal exposure, offering false reassurance, inserting crisis resources into controlled exercises, and refusing to challenge distorted cognitions mentioning self-harm in PE; and through task abandonment or safety-preamble insertion during CBT cognitive restructuring. These findings motivate a five-axis evaluation framework (protocol fidelity, hallucination risk, behavioral consistency, crisis safety, demographic robustness), mapped onto FDA SaMD and EU AI Act requirements. We argue that no AI mental health system should proceed to deployment without passing multi-axis evaluation across all five dimensions.

Abstract

LLM Safety

Explore similar work

AI Safety Training Can be Clinically Harmful

When Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical Queries

ToolAlignBench: Investigating Alignment Conflicts in Tool-Calling Enabled LLMs