Refusal Is Not Robustness: Measuring Harmful Reachability in Autoregressive Language Models

Parv Kapoor; Teresa Fortes; Claudia Mamede; Rui Abreu; Jonathan Aldrich; Eunsuk Kang

cs.LGOct 15, 2026

Refusal Is Not Robustness: Measuring Harmful Reachability in Autoregressive Language Models

Authors: Parv Kapoor, Teresa Fortes, Claudia Mamede, Rui Abreu, Jonathan Aldrich, Eunsuk Kang

Organizations: Carnegie Mellon University, Pittsburgh, PA, USA. · General Robotics · University of Porto, Porto, Portugal.

Abstract

LLM safety evaluations via jailbreaking typically index on attack success rate (ASR): whether a generated response violates safeguards. However, ASR only evaluates sampled outputs and provides little information on the underlying generation process. Autoregressive language models define distributions over possible outputs, and a sampled refusal can coexist with substantial probability mass on harmful generation paths. We study this hidden structure through bounded continuation analysis (BCA), which models autoregressive generation as a stochastic process over token prefixes and computes the finite-horizon probability that generation reaches a harmful continuation. BCA constructs a bounded continuation tree, labels terminal continuations with a semantic harm evaluator, and estimates harmful reachability using exact probabilistic model checking. BCA complements ASR by measuring how much harmful behavior remains accessible beneath a sampled safe response. This makes it useful for distinguishing models that look identical under measured ASR values, and allocating red-team effort toward regions of the generation process with greater hidden risk. Across four instruction-tuned LLMs and 100 JailbreakBench behaviors, we show that BCA effectively quantifies hidden harmful behavior and provides a useful signal to guide red teaming.

Figures & tables

Figure 1: Bounded Continuation Analysis. Each node represents a partial continuation of the prompt and edge weights are the model’s next-token probabilities. The greedy decode (black path) ends in a safe refusal, so a one-sample evaluation such as ASR marks the prompt safe. However, alternative branches retain substantial harmful mass: Aggregating over the tree gives bounded harm reachability

\mathbb{P}[\mathbf{F}\,\texttt{harm}]=0.28

, which greedy decoding alone does not reveal.

Target model	No Attacker		Under 18-Template Attack
Target model	Greedy ASR	GAS@0	Empirical ASR@18	GAS@18
Qwen2.5-1.5B-Instruct	0%	0.004	100.0%	0.906
Qwen2.5-7B-Instruct	0%	0.004	100.0%	0.907
Mistral-7B-Instruct-v0.3	0%	0.211	100.0%	0.782
Llama-3.1-8B-Instruct	0%	0.002	76.7%	0.144

Table 1: Identical observed refusals conceal large differences in harmful reachability, and empirical ASR saturates where GAS still differentiates. All four models show 0% greedy ASR (unshaded) and GAS@0 still separates them by two orders of magnitude. Under 18 jailbreak templates, empirical ASR (fraction of behaviors where any template ever elicits a judged-harmful completion) saturates at 100% for three of four targets, erasing any distinction between them;

\mathrm{GAS@18}

(the maximum bounded harmful mass over the same 18 templates) still resolves a clear gap between them. Color intensity is normalized within each shaded column; bold denotes the column maximum.

Target	$n$ pairs	Spearman $\rho$	Pearson $r$	Top-1 lift
Qwen2.5-1.5B	540	0.716	0.74	4.93 $\times$
Qwen2.5-7B	540	0.733	0.83	7.43 $\times$
Mistral-7B	540	0.611	0.59	2.56 $\times$
Llama-3.1-8B	540	0.534	0.79	6.73 $\times$

Table 2: Bounded harmful reachability predicts independently judged harmful generations. Bounded reachability uses Qwen2.5-7B as the BCA leaf evaluator; empirical ASR is independently sampled and scored by HarmBench-Mistral-7B. Spearman’s

\rho

and Pearson’s

r

measure rank and linear agreement between the two; Top-1 lift is the empirical-ASR gain from picking each behavior’s highest-mass template over a uniform one. Darker shading denotes larger values within each metric; bold denotes the maximum

Language Model Safety Evaluation

Language model safety evaluation measures harmful behavior, refusal, robustness, and over-refusal across realistic prompts, users, and deployment settings.

See all 543 papers

LLMs Encode Harmfulness and Refusal Separately
Jul 16, 2025Jiachen Zhao, Jing Huang, Zhengxuan Wu +2LLM Interpretability LLM Refusal Behavior
Northeastern University · Stanford University
Different Paths to Harmful Compliance: Behavioral Side Effects and Mechanistic Divergence Across LLM Jailbreaks
Apr 20, 2026Md Rysul Kabir, Zoran TiganjLanguage Model Safety Evaluation LLM Interpretability
Department of Computer Science Luddy School of Informatics, Computing, and Engineering Indiana University Bloomington
When Prompt Optimization Becomes Jailbreaking: Adaptive Red-Teaming of Large Language Models
Feb 21, 2026Zafir Shamsi, Nikhil Chekuru, Zachary Guzman +1Adversarial Attacks on LLMs Language Model Safety Evaluation
Algoverse AI Research

Jul 16, 2025J/K move · Enter open · S save

LLMs Encode Harmfulness and Refusal Separately

Jiachen Zhao, Jing Huang, Zhengxuan Wu, David Bau+1

Northeastern University · Stanford University

LLM Interpretability LLM Refusal Behavior LLM Safety LLM Jailbreak Attacks Causal Interventions in Language Models Refusal Behavior in Language Models

Open paper AskPDF arXiv

LLMs are trained to refuse harmful instructions, but do they truly understand harmfulness beyond just refusing? Prior work has shown that LLMs' refusal behaviors can be mediated by a one-dimensional subspace, i.e., a refusal direction. In this work, we identify a new dimension to analyze safety mechanisms in LLMs, i.e., harmfulness, which is encoded internally as a separate concept from refusal. There exists a harmfulness direction that is distinct from the refusal direction. As causal evidence, steering along the harmfulness direction can lead LLMs to interpret harmless instructions as harmful, but steering along the refusal direction tends to elicit refusal responses directly without reversing the model's judgment on harmfulness. Furthermore, using our identified harmfulness concept, we find that certain jailbreak methods work by reducing the refusal signals without reversing the model's internal belief of harmfulness. We also find that adversarially finetuning models to accept harmful instructions has minimal impact on the model's internal belief of harmfulness. These insights lead to a practical safety application: The model's latent harmfulness representation can serve as an intrinsic safeguard (Latent Guard) for detecting unsafe inputs and reducing over-refusals that is robust to finetuning attacks. For instance, our Latent Guard achieves performance comparable to or better than Llama Guard 3 8B, a dedicated finetuned safeguard model, across different jailbreak methods. Our findings suggest that LLMs' internal understanding of harmfulness is more robust than their refusal decision to diverse input instructions, offering a new perspective to study AI safety.

Abstract

Language Model Safety Evaluation

Explore similar work

LLMs Encode Harmfulness and Refusal Separately

Different Paths to Harmful Compliance: Behavioral Side Effects and Mechanistic Divergence Across LLM Jailbreaks

When Prompt Optimization Becomes Jailbreaking: Adaptive Red-Teaming of Large Language Models