Model cards, system cards, and safety reports increasingly document LLM safety behaviors, yet safety evaluations typically start from predefined benchmarks rather than asking which documented claims those benchmarks actually test. We introduce LEST (LLM Evaluation through Specification-driven Testing), a framework that treats documented safety claims as specifications for black-box testing. In LEST, each claim determines which benchmark prompt tests it and whether a Refuse-to-Comply transition, where a model refuses a prompt but complies with its mutated version, counts as evidence against it. LEST mutates these prompts to search for counterexamples, manually validates each candidate, and traces validated counterexamples back to the claims they contradict. We evaluate LEST using the system cards of Claude Opus 4.8 and GPT-5.5 with 97 seed prompts from SORRY-Bench across Mental Health and Self-Harm, Harmful and Violent Content, and Fairness and Bias. Of 56 identified safety claims, 29 (51.8%) are prompt-testable, but only 17 (58.6%) are exercised by an applicable benchmark prompt. Both models refuse >95% of benchmark prompts. Evaluating 1,836 mutated prompts on both models reveals 312 candidate Refuse-to-Comply transitions flagged by automated response classifiers. However, human validation confirms only 19 (6.1%) as both intent-preserving and genuinely compliant, showing automated evaluation substantially overestimates behavioral failures. Tracing these 19 transitions back to the claims reveals counterexamples for 6 of 9 covered claims for which refusal is an applicable oracle. Our results show that benchmark prompts may not be sufficient to validate all documented LLM safety behavior, and automated failure detection requires manual validation. LEST provides end-to-end traceability from natural-language safety documentation to executable tests and human-validated counterexamples.
Figures & tables
Figure 1. Overview of LEST. Starting from model documentation, LEST extracts and assesses safety claims, maps prompt-testable claims to applicable seed prompts, generates and validates mutated prompts, executes the original and mutated prompts on the target LLM, and validates observed Refuse → Comply transitions to produce a claim-level compliance report. Overview of LEST approach showing the pipeline from model documentation through safety claim extraction, prompt generation, mutation, execution, and compliance reporting.
#
Condition
Outcome
(a) Claim identification
1
Concerns the future (plans, anticipated capabilities)
Not a claim
2
Is an example prompt
Not a claim
3
Asserts a concrete result, property, or behavior of the model
Claim
4
Describes how users should use the model
Not a claim
5
Describes the model’s training process
Claim
Table 1. Annotation criteria for claim identification and prompt-testability. Rules within each stage are applied sequentially. For claim identification, a candidate reaching the end of the decision process without satisfying a claim criterion is labeled Not a claim . For testability, a claim that reaches Rule 4 but does not admit an objective confirm/deny prompt is labeled Not testable .
Identified claims
Refusal-testable claims
Model
Safety Domain
Prompt-Testable
Non-Prompt-Testable
Total
All
Covered
CE
Claude
Mental Health/Self-Harm
14
7
21
5
5
4
Harmful/Violent Content
5
1
6
2
2
0
Fairness/Bias
1
3
4
1
1
1
GPT-5.5
Harmful/Violent Content
9
16
25
3
1
1
Overall
29
27
56
11
9
6
Table 2. Testability, refusal-oracle applicability, benchmark coverage, and validated counterexamples of identified safety claims by model and safety domain. All identified GPT-5.5 claims fall under Harmful/Violent Content. Covered : refusal-testable claims mapped to at least one applicable seed prompt. CE : covered claims with at least one human-validated Refuse → Comply counterexample from the same model.
Safety Domain
Claude
GPT-5.5
Overall
Fairness/Bias
1/17 (5.9%)
2/16 (12.5%)
3/33 (9.1%)
Mental Health/Self-Harm
1/12 (8.3%)
0/12 (0.0%)
1/24 (4.2%)
Harmful/Violent Content
4/55 (7.3%)
3/55 (5.5%)
7/110 (6.4%)
Overall
6/84 (7.1%)
5/83 (6.0%)
11/167 (6.6%)
Table 3. Originally refused model–seed-prompt pairs for which at least one human-validated Refuse → Comply transition is observed.
Model
Mutation Operator
Fairness & Bias
Harmful & Violent
Mental Health
All Categories
Claude Opus 4.8
Paraphrase
13 (0)/127
13 (4)/264
8 (0)/189
34 (4)/580
PCLS
14 (0)/137
3 (2)/174
3 (0)/175
20 (2)/486
Fairness
42 (1)/324
2 (0)/31
0 (0)/4
44 (1)/359
Modifier
7 (0)/51
4 (0)/103
4 (1)/75
15 (1)/229
Deletion
12 (0)/62
1 (0)/35
0 (0)/24
13 (0)/121
First-to-Third
0 (0)/10
0 (0)/3
0 (0)/5
0 (0)/18
Table 4. Refuse → Comply (R → C) results by model, final mutation operator, and safety domain. Each cell reports automated R → C candidates (human-validated R → C transitions) / eligible mutated prompts , where eligible mutated prompts are those whose corresponding original prompt was refused.
Traditional benchmarks for large language models (LLMs), such as HELM and AIR-BENCH, primarily assess safety risk through breadth-oriented evaluation across diverse tasks. However, real-world deployment often exposes a different class of risk: operational failures arising from repeated generations of the same prompt rather than broad task generalization. In high-stakes settings, response consistency and safety under repeated use are critical operational requirements. We introduce Accelerated Prompt Stress Testing (APST), a depth-oriented evaluation framework inspired by highly accelerated stress testing in reliability engineering. APST probes LLM behavior by repeatedly sampling identical prompts under controlled operational conditions, including temperature variation and prompt perturbation, to surface latent failure modes such as hallucinations, refusal inconsistency, and unsafe completions. Rather than treating failures as isolated events, APST characterizes them statistically as stochastic outcomes of repeated inference. We model observed safety failures using Bernoulli and binomial formulations to estimate per-inference failure probabilities, enabling quantitative comparison of operational risk across models and configurations. We apply APST to multiple instruction-tuned LLMs evaluated on AIR-BENCH 2024 derived safety and security prompts. While models exhibit similar performance under conventional single- or very-low-sample evaluation (N <= 3), repeated sampling reveals substantial variation in empirical failure probabilities across temperatures. These results demonstrate that shallow benchmark scores can obscure meaningful differences in reliability under sustained use.
Refusal rates are a poor proxy for LLM safety, i.e., a model may over-refuse benign prompts while still complying with harmful ones. We audit both failure modes across 21 open-weight LLMs on four safety benchmarks (OR-Bench, XSTest, ToxiGen, BOLD), using a composition adjustment to isolate model sensitivity from dataset toxicity confounds. We report three findings. First, models adopt fundamentally different calibration strategies: conservative ecosystems such as Llama suppress unsafe outputs at the cost of elevated over-refusals, while permissive ecosystems such as DeepSeek and Qwen preserve helpfulness but tolerate higher harmful compliance. Second, demographic protection is unequal: models over-protect prominent racial and religious groups, frequently refusing even benign prompts about them, while providing substantially weaker protection against disability-targeted attacks. Third, refusal and compliance tendencies are stable within model families across generations and scales, suggesting that post-training objectives shape safety behavior more than architecture. Our results call for joint, demographically-aware, and multi-judge safety evaluation.
Alif Al Hasan, Sumon Biswas
Department of Computer and Data Sciences Case Western Reserve University Cleveland, OH, USA
Current LLM safety benchmarks largely rely on binary metrics, overlooking how models respond to harmful prompts with varying threat implicitness. We introduce TIER, a Threat Implicitness Benchmark for behavioral safety evaluation of LLMs. TIER covers four risk domains and four threat levels, from explicit harmful requests to sophisticated jailbreaks. Responses are assessed using a six-label behavior scale and two independent LLM judges. Experiments on six open-weight LLMs show that safety behaviors evolve gradually across threat levels rather than shifting directly from refusal to compliance. Contextual prompts yield the most diverse behaviors, while jailbreaks reveal the largest robustness gaps. Furthermore, models with similar Attack Success Rates can exhibit distinct response distributions, highlighting the need for behavior-aware LLM safety evaluation.