LLM-based information systems need safety guardrails. Most guardrails classify the prompt alone. If the prompt looks harmful, the system rejects it before any answer is generated. This is costly. Many rejected prompts would have received a safe response. Users lose help they should have received from the information system. This cost grows as conversational and generative search becomes common. Prompt-level labels treat harm as a property of the query. But harm depends on what the model actually writes. Existing input-output guards do look at responses. But they judge a finished response, so the decision comes after full generation. However, we observe that safety of model interaction with the input prompt can be inferred early with only partial output. So the system can start generating, watch the partial response, and stop only if it turns harmful. We test this idea in three steps. First, we build a dataset of 41,944 prompt-response pairs with seven open weights LLMs from three families: Llama, Gemma, and Qwen. Second, we show that prompt harmfulness and interaction safety differ. Prompt labels and response labels agree on only 50.46% of items (Cohen's κ = 0.06). Of the prompts that make any model respond harmfully, only 3.5% do so for all seven LLMs. Third, we show that partial responses carry most of the signal. Fine-tuned classifiers predict response-level harm well (best AUPRC 0.885, F1 0.807). About one third of the response tokens recover almost full-response AUPRC.
Figures & tables
Figure 2: Where safety decisions are made. Our monitor reads the response while it is generated and stops it if it turns harmful.
Train
Validation
Test
Source
Benign
Malicious
Benign
Malicious
Benign
Malicious
AdvBench
0
352
0
48
0
120
Arena
893
0
165
0
166
0
deepset
302
169
40
31
56
58
xTRam
852
1683
148
317
200
400
Total
2047
2204
353
396
422
578
Table 1: Prompt-level composition before response generation.
Pair
Agreement (%)
Cohen’s κ
PABAK
Human 1 – Human 2
92.57
0.664
0.851
Human 1 – Gemini
95.43
0.825
0.909
Human 2 – Gemini
92.57
0.667
0.851
Prompt label – Gemini
62.76
0.272
0.255
Prompt label – Human 1
61.05
0.238
0.221
Prompt label – Human 2
57.43
0.170
0.149
Table 2: Agreement on the 1,050 prompt-response pair audit set.
Classifier
Setting
AUPRC
Prec.
Rec.
F1
DeBERTa-v3
ZS, prompt only
0.071
0.084
0.743
0.151
ZS, prompt + resp.
0.076
0.082
0.614
0.145
FT (standard), prompt only
0.444
0.465
0.610
0.528
FT (standard), prompt + resp.
0.683
0.644
0.755
0.695
Prompt-Guard- 2-86M
ZS, prompt only
0.302
0.380
0.446
0.410
ZS, prompt + resp.
0.357
0.467
0.426
0.445
Table 3: Results on the test set (6,998 pairs, 3.56% harmful), harmful class. ZS is zero-shot and FT is fine-tuned, with the strategy in brackets. FT classifiers are trained on the prompt and the full response. In the “prompt only” rows, the response is withheld at test time. For the FT classifiers, precision, recall and F1 use a threshold chosen on the validation set for the same input as the row: no response for “prompt only”, and the full response for “prompt + resp.”
Figure 3: Test-set AUPRC, precision, recall and F1 of the fine-tuned classifiers as the prefix grows from 0% to 100% of the response tokens, in steps of 10%. The imbalance strategy chosen for each classifier is in brackets. At 0% the classifier sees only the prompt. Precision, recall and F1 use the threshold fixed on the validation set at the full response.
Paradigm
Decision point
Decision based on
Examples or focus
Prompt-level guards
Before generation
The prompt
ProtectAI detector; xTRam1, deepset, AdvBench
Input-output guards
After the full response
Prompt and response
Llama Guard, ShieldGemma, WildGuard
Streaming monitors
During generation
Prompt and partial response, trained for it
Streaming content monitor, Qwen3Guard-Stream, CARE
Model refusal benchmarks
No decision; models are ranked
Responses of many models to unsafe or safe-looking prompts
SORRY-Bench, HarmBench, XSTest, OR-Bench
Query harmfulness prediction
Before retrieval
The query
QHP
Ours
During generation
Prompt and partial response, with guards trained on full responses
Gap between prompt label and interaction outcome, across seven models
Table 4: Paradigms of safety work, and where our study sits.
Autonomous systems increasingly rely on Large Language Models (LLMs) yet the safety infrastructure surrounding these models introduces latency and compute overhead. This limits utility in resource-constrained, time-critical deployments. Existing external guardrail models remain blind to the model's internal workings, creating a fundamental assurance gap. We ask: does the model already know when the content is harmful? We extract activations from LLaMA-3.1-8B and train lightweight MLP classifier probes (12.6M parameters) to detect harmful prompts. Evaluated on WildJailbreak, Beavertails, and AEGIS 2.0, our probes achieve F1 scores of 99%, 83%, and 84%, respectively competitive with 1000x larger guard models while cutting latency and compute costs.
Safety evaluations of large language models (LLMs) typically report binary outcomes, i.e. attack success rate (ASR), refusal rate, or harmful versus safe classification, which hide how risk changes between prompt and response. We present a paired analysis over human labeled prompt and response records across four harm categories (Sexual, Self harm, Hate and Violence) and ordinal severity levels (Safe, Low, Medium, High). 61% of responses reduce harm relative to the prompt, 36% preserve severity, and 3% escalate. The escalation splits into two mechanisms: benign prompts triggering unrequested harmful detail, and answers that stay on task at higher severity than the prompt. Category decomposition shows that Sexual content exhibits the highest harm persistence in this sample, driven by compliance at the same severity rather than drift from benign inputs. Joint relevance analysis exposes a helpfulness versus harmlessness tradeoff: compliance escalations remain highly relevant, whereas safe responses include generic refusals with low relevance. A public supporting evaluation over 600 prompts and six models reproduces the framework's measurements and two directional signals, while few-shot LLM graders exhibit a prompt/response detection asymmetry that data calibration does not close. Grader prompts, public-evaluation artifacts, and analysis code are shared at https://github.com/microsoft/PairedSafety.
Large Language Models (LLMs) are increasingly deployed in real-world applications, yet they remain vulnerable to generating harmful content. From adversarial jailbreaks that bypass safety filters to implicit hate that evades detection, the range of risks these models pose continues to grow. While both specialized content moderators and general-purpose LLMs are being used as safety layers, the question of which model is best suited for which type of harmful content remains unanswered. We present the most comprehensive evaluation of LLM safety capabilities to date, systematically testing 53 models across 11 datasets that we organize into four distinct categories. Our evaluation under both prompt-only and prompt-response settings uncovers critical blind spots: large frontier models that lead on one category fall significantly behind smaller, specialized alternatives on others, and real-world conversational safety remains largely unsolved across all model families. These findings challenge the assumption that scale alone ensures safety, and provide the community with a structured framework for informed model selection.