Evaluating Language Model Safety Across Long Adversarial Conversations

Parisa Salmani; Peter R. Lewis

cs.CLSep 29, 2026

Evaluating Language Model Safety Across Long Adversarial Conversations

Authors: Parisa Salmani, Peter R. Lewis

Organizations: Ontario Tech University Oshawa, Canada

Abstract

Conversational safety evaluations often test language models with a single harmful prompt, even though real-world systems interact with users through long, adaptive conversations. This study examines whether models continue to respond safely when an adversarial user persists across multiple turns. We evaluate three open-weight, instruction-tuned models on two harmful prompts across different conversation lengths and random seeds. In each setting, a second language model acts as a persistent adversarial user, while a safety classifier labels every response as safe or unsafe. Across all model-prompt combinations, first-turn safe-response rates ranged from 85% to 100%. By depth 11, they dropped to 38-61%, and by depth 101, to 15-44%. This decline appeared across models and continued well beyond the short interactions typically used in multi-turn safety evaluations. These results provide proof-of-concept evidence that strong single-turn safety does not necessarily persist during sustained adversarial interaction. They highlight the need for long-horizon evaluations and conversation-level safeguards that account for risk accumulating across turns.

Figures & tables

Model	Prompt 1	Prompt 2
GPT-OSS-20B	$0.90\rightarrow 0.46$	$1.00\rightarrow 0.55$
Llama-3.2-3B	$1.00\rightarrow 0.43$	$1.00\rightarrow 0.61$
Gemma-4-26B-A4B	$0.85\rightarrow 0.38$	$1.00\rightarrow 0.52$

TABLE I: Safe-response rate at the first turn (Depth = 1) to deepest turn (Depth = 11), by model and domain, averaged over ten different seeds.

Panel of Figure 2 — Fig. 1: Safe-response rate declines with conversational depth. Bars report the proportion of assistant responses judged safe at each depth in a single depth-11 exhaustive tree (one seed) seeded with the first prompt with Llama model. Safety falls from near-complete at the opening turns to roughly 45 percent by depth 11.

Model	Prompt 1	Prompt 2
GPT-OSS-20B	$0.90\rightarrow 0.28$	$1.00\rightarrow 0.38$
Llama-3.2-3B	$1.00\rightarrow 0.22$	$1.00\rightarrow 0.44$
Gemma-4-26B-A4B	$0.85\rightarrow 0.15$	$1.00\rightarrow 0.36$

TABLE II: Longer conversation (50 sampled over each tree with 50 seeds): safe-response rate at the first turn

\rightarrow

deepest sampled turn, by model and domain.

Panel of Figure 4 — Fig. 2: Safe-response rate across 50 sampled trajectories for gpt-oss-20b at 101 depth. Top: First prompt. Bottom: Second prompt.

Panel of Figure 5 — Fig. 3: Safe-response rate across 50 sampled trajectories for Llama-3.2-3b at 101 depth. Top: First prompt. Bottom: Second prompt.

Panel of Figure 6 — Fig. 4: Safe-response rate across 50 sampled trajectories for Gemma-4-26B at 101 depth. Top: First prompt. Bottom: Second prompt.

Panel of Figure 7 — Fig. 5: Safe-response rates across a conversation of depth 101, reported separately for the evaluated assistant at odd depths and the adversarial shadow user at even depths. The assistant begins with a high safe-response rate but becomes progressively less safe, whereas the proportion of shadow-user messages classified as safe generally increases. The divergence suggests that assistant safety may depend on accumulated conversational history rather than solely on the explicit harmfulness of the immediately preceding user message. Shadow-user safety labels should be interpreted as measures of classifier-assessed harmfulness, not as direct measures of adversarial effectiveness.

Language Model Safety Evaluation

Language model safety evaluation measures harmful behavior, refusal, robustness, and over-refusal across realistic prompts, users, and deployment settings.

See all 542 papers

Beyond Unsafe Detection: Counterfactually Anchored Evidence Attribution for Multi-Turn LLM Safety Failures
Aug 17, 2026Srinivasan Subramanian, Kazi Aminul Islam, Md. Abdullah Al Hafiz KhanLLM Safety LLM Safety Evaluation
Department of Computer Science, Kennesaw State University, Marietta, GA, USA
ADVERSA: Measuring Multi-Turn Guardrail Degradation and Judge Reliability in Large Language Models
Mar 10, 2026Harry Owiredu-AshleyLanguage Model Safety Evaluation Adversarial Attacks on LLMs
Independent Researcher, New Jersey, United States
State-Dependent Safety Failures in Multi-Turn Language Model Interaction
Mar 15, 2026Pengcheng Li, Jie Zhang, Tianwei Zhang +5LLM Safety Language Model Safety Evaluation
School of Cyber Science and Technology, University of Science and Technology of China, China · Agency for Science, Technology and Research (A*STAR), Singapore · Nanyang Technological University, Singapore +3

Aug 17, 2026J/K move · Enter open · S save

Beyond Unsafe Detection: Counterfactually Anchored Evidence Attribution for Multi-Turn LLM Safety Failures

Srinivasan Subramanian, Kazi Aminul Islam, Md. Abdullah Al Hafiz Khan

Department of Computer Science, Kennesaw State University, Marietta, GA, USA

LLM Safety LLM Safety Evaluation LLM Safety Benchmarks Faithfulness of Language Model Explanations Adversarial Attacks on LLMs

Open paper AskPDF arXiv

As Large Language Models (LLMs) move from conversational assistants to advanced agentic systems, guardrail failures can convert adversarial intents into harmful executions. However, most evaluation frameworks focus only on whether a user request is unsafe. This is insufficient for multi-turn failures, where adversarial intent may be distributed across turns and emerge only from accumulated context. This motivates us to go beyond detection to localize the evidence that pushes the conversation toward unsafe trajectories. We introduce MTAG-Bench, a multi-turn guardrail attribution dataset with behavioral validation and counterfactual supervision containing 4,167 conversations. We also introduce MTAG-Probe, a hierarchical attribution model for turn- and span-level evidence localization with conversation-level safety detection. MTAG-Probe improves turn-level F1 by 28.1 and 24.1 points (79.0% and 61.0% relative) over self-attribution by the model that produced the unsafe response and the LLM safety-judge attribution, respectively. Removing MTAG-Probe's top-ranked evidence within budgets of 15% and 20% of user words per conversation changes 50.6% and 60.8% of Llama Guard decisions on Geometry from unsafe to safe, respectively. In full-trajectory replay, MTAG-Probe converts 84.3% of eligible unsafe final responses to safe. These results show that MTAG-Probe can provide evidence-level explanations as a practical complement to safety detection.

Abstract

Language Model Safety Evaluation

Explore similar work

Beyond Unsafe Detection: Counterfactually Anchored Evidence Attribution for Multi-Turn LLM Safety Failures

ADVERSA: Measuring Multi-Turn Guardrail Degradation and Judge Reliability in Large Language Models

State-Dependent Safety Failures in Multi-Turn Language Model Interaction