Quantifying Behavioral Tails in Black-Box Language Models

Elsayed Eshra; Ali Al-Lawati; Dongwon Lee; Suhang Wang

cs.LGSep 27, 2026

Quantifying Behavioral Tails in Black-Box Language Models

Authors: Elsayed Eshra, Ali Al-Lawati, Dongwon Lee, Suhang Wang

Organizations: The Pennsylvania State University University Park, PA 16802, USA

Abstract

We introduce RareTrap, a framework for estimating the probability of severe behaviors in black box large language models (LLMs). A key challenge for probability estimation is defining a tractable distribution over the input space. To accomplish that, RareTrap uses a surrogate LLM and constructs a geometry-aware mapping from a lower-dimensional latent reference space into its token-embedding space to induce an explicit and reproducible distribution over input prompts. A response-level performance function is utilized on the response to quantify behavior severity. This enables sequential rare event simulation that concentrates evaluations on progressively more severe behaviors while preserving probability under the induced prompt distribution, which would otherwise be prohibitive to measure. Across 10 open-weight and two frontier models (GPT-5.4 and Claude Sonnet 4.6), we find that RareTrap successfully induces severe resource consumption behaviors and computes their probability with as few as 200 evaluations. RareTrap provides model developers a principled approach for evaluating language models under a common distribution, and prioritizing alignment effort to improve safety and mitigate risks.

Figures & tables

		Over-generation ( $r_{\mathrm{len}}\geq 20{,}000$ )				Degenerate repetition ( $r_{\mathrm{rep}}\geq 0.99$ )
Target model	Surrogate	$\widehat{p}_{\mathcal{F}}$	Evals	Levels	Threshold trajectory	$\widehat{p}_{\mathcal{F}}$	Evals	Levels	Threshold trajectory
DeepSeek-R1-Distill-Llama-8B	Self	$0.350$	200	1	$20{,}000$	$0.455$	200	1	$0.990$
	Qwen3-0.6B	$0.160$	200	1	$20{,}000$	$0.225$	200	1	$0.990$
Phi-4-reasoning	Self	$1.000$	200	1	$20{,}000$	$1.000$	200	1	$0.990$
	Qwen3-0.6B	$1.000$	200	1	$20{,}000$	$1.000$	200	1	$0.990$
Qwen3.5-9B	Self	$0.995$	200	1	$20{,}000$	$0.990$	200	1	$0.990$

Table 1: Behavioral-tail estimates across eight targets under self and Qwen3-0.6B surrogates.

\widehat{p}_{\mathcal{F}}

is the measured probability, Evals is the number of evaluations required to measure the probability, and Threshold trajectory reports the severity threshold defining each level. Levels include the initial set

\mathcal{Z}_{0}

. Trajectories terminate at the prescribed target threshold.

		Over-generation ( $r_{\mathrm{len}}\geq 20{,}000$ )				Degenerate repetition ( $r_{\mathrm{rep}}\geq 0.99$ )
Target model	Surrogate	$\widehat{p}_{\mathcal{F}}$	Evals	Levels	Threshold trajectory	$\widehat{p}_{\mathcal{F}}$	Evals	Levels	Threshold trajectory
NVIDIA-Nemotron-3.5-Lightning	Qwen3-0.6B	$1.31{\times}10^{-3}$	2800	3	$2{,}035\rightarrow 3{,}652\rightarrow 20{,}000$	$5.73{\times}10^{-2}$	1900	2	$0.914\rightarrow 0.990$
	Qwen2.5-0.5B-Instruct	$6.68{\times}10^{-3}$	2800	3	$1638\rightarrow 3056\rightarrow 20{,}000$	$6.14{\times}10^{-3}$	2800	3	$0.685\rightarrow 0.792\rightarrow 0.990$
Qwen3.8-27B	Qwen3-0.6B	$7.21{\times}10^{-2}$	1900	2	$7{,}070\rightarrow 20{,}000$	$4.96{\times}10^{-2}$	1900	2	$0.892\rightarrow 0.990$
	Qwen2.5-0.5B-Instruct	$8.63{\times}10^{-2}$	1900	2	$13{,}089\rightarrow 20{,}000$	$8.78{\times}10^{-4}$	3700	4	$0.761\rightarrow 0.855\rightarrow 0.926\rightarrow 0.990$

Table 2: Behavioral-tail estimates for two recent high-capability models.

Target	Surrogate	$\tau_{\mathrm{len}}$	$\widehat{p}_{\mathcal{F}}$	Evals	Levels	Threshold trajectory
Sonnet 4.6 (medium)	Qwen3-0.6B	6,000	$2.33\times 10^{-4}$	3,700	4	$492\rightarrow 1{,}022\rightarrow 2{,}243\rightarrow 6{,}000$
Sonnet 4.6 (high)	Qwen3-0.6B	20,000	$3.35\times 10^{-4}$	3,700	4	$878\rightarrow 1{,}719\rightarrow 8{,}118\rightarrow 20{,}000$
GPT-5.4 (xhigh)	Qwen3-0.6B	20,000	$1.31\times 10^{-2}$	1,900	2	$3{,}706\rightarrow 20{,}000$

Table 3: Over-generation estimates for frontier models across two thresholds

		Direct Monte Carlo	RareTrap
Metric	Target / surrogate	P25 / Median / P75	Level 1 P25 / Median / P75	$\bm{\rightarrow}\;\shortstack{Level 2\\ P25 / Median / P75}$	$\bm{\rightarrow}\;\shortstack{Level 3\\ P25 / Median / P75}$
Length	GPT-OSS-20B / Qwen3-0.6B	$231/329/527$	$238/323/514$	$1,890/3,240/11,044$	–
Length	Nemotron-9B / Qwen3-0.6B	$309/386/510$	$304/375/502$	$678/739/886$	$20,003/20,003/20,003$
Length	OLMo-3-7B / Self	$48/66/132$	$49/69/159$	$553/651/784$	$1,077/4,159/19,739$
Repetition	GPT-OSS-20B / Qwen3-0.6B	$.206/.281/.423$	$.216/.289/.464$	$.978/.996/1.000$	–
Repetition	OLMo-3-7B / Self	$.000/.018/.056$	$.000/.022/.122$	$.354/.383/.435$	$.998/1.000/1.000$

Table 4: Response-severity distributions under independent sampling and successive RareTrap conditioning at equal target-model evaluation counts. Entries report P25/median/P75. Level 1 is

\mathcal{Z}_{0}

, while quantiles for subsequent levels are computed using only newly accepted proposals.

Model	Projection	Over-generation ( $r_{\mathrm{len}}$ ) P50 / P75 / P95	Degenerate repetition ( $r_{\mathrm{rep}}$ ) P50 / P75 / P95
DeepSeek-8B	Geometry-aware	$950/\emph{cap}/\emph{cap}$	$.781/1.000/1.000$
	Embedding-agnostic	$927/1{,}222/1{,}938$	$.430/.514/.697$
Qwen3.5-9B	Geometry-aware	$\emph{cap}/\emph{cap}/\emph{cap}$	$1.000/1.000/1.000$
	Embedding-agnostic	$2{,}369/2{,}841/\emph{cap}$	$.632/.688/.999$
OLMo-3-7B	Geometry-aware	$70/134/591$	$.022/.078/.353$
	Embedding-agnostic	$125/251/811$	$.042/.137/.304$

Table 5: Geometry-aware ablation using self surrogate. P50, P75, and P95, are the 50th (median), 75th, and 95th percentiles, and cap is the 20,003-token generation limit.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Repository identifier
Target panel
DeepSeek-R1-Distill-Llama-8B ( DeepSeek-AI, 2025 )	deepseek-ai/DeepSeek-R1-Distill-Llama-8B
GPT-OSS-20B ( OpenAI, 2025 )	openai/gpt-oss-20b
Phi-4-reasoning ( Abdin et al., 2025 )	microsoft/Phi-4-reasoning
Mistral-7B-Instruct-v0.3 ( Jiang et al., 2023 )	mistralai/Mistral-7B-Instruct-v0.3
Qwen3-14B ( Yang et al., 2025 )	Qwen/Qwen3-14B

Appendix

Table 6: Language models used in this work, with their public repository identifiers.

			Over-generation $(r_{\mathrm{len}}\geq 20{,}000)$				Degenerate repetition $(r_{\mathrm{rep}}\geq 0.99)$
Target model	Surrogate	$\mathbf{L}$	$\widehat{p}_{\mathcal{F}}$	Evals	Levels	Threshold trajectory	$\widehat{p}_{\mathcal{F}}$	Evals	Levels	Threshold trajectory
DeepSeek-R1- Distill-Llama-8B	self	20	0.355	200	1	20,000	0.380	200	1	0.99
		40 †	0.350	200	1	20,000	0.455	200	1	0.99
		80	0.355	200	1	20,000	0.420	200	1	0.99
	Qwen3-0.6B	20	0.135	200	1	20,000	0.165	200	1	0.99
		40 †	0.160	200	1	20,000	0.225	200	1	0.99

Appendix

Table 7: Sensitivity of

\widehat{p}_{\mathcal{F}}

to the prompt length

L

(a) and the shared latent dimension

D

(b) around the main configuration

L{=}40

D{=}200

( bold † ). The trajectory lists the selected threshold at each level, so it has one entry per level (consecutive repeats indicate a level that did not advance the threshold).

Target	Surrogate	$\tau_{\mathrm{len}}$	$\widehat{p}_{\mathcal{F}}$	Evals	Levels	Threshold trajectory
DeepSeek-8B	Qwen3-0.6B	100,000	$0.115$	200	1	$100{,}000$
Nemotron-9B	Qwen3-0.6B	100,000	$1.50\times 10^{-2}$	1,900	2	$655\rightarrow 100{,}000$
OLMo-3-7B	Self	65,000	$2.25\times 10^{-3}$	2,800	3	$389\rightarrow 1{,}115\rightarrow 65{,}000$
Qwen3-14B	Qwen3-0.6B	32,768	$7.57\times 10^{-3}$	2,800	3	$323\rightarrow 607\rightarrow 32{,}768$

Appendix

Table 8: Extreme over-generation estimates.

\tau_{\mathrm{len}}

is raised to 100,000 tokens, or to the model’s maximum sequence length where that is lower.

Pattern	Representative model	Observed behavior
Ignored stopping signal	DeepSeek / GPT-OSS / Phi-4	The generated text names a reason to halt—confusion, repetition, an unresolved policy check—but generation continues, looping on a revised interpretation, the same deliberation, or a single repeated sentence.
Overridden fallback	Nemotron-9B	The model identifies clarification as the appropriate action but replaces it with additional open-ended analysis.
Recurrent finalization	Qwen3.5	The response is selected and explicitly labeled final, yet finalization itself becomes another step that repeats without terminating.
Fixed frame, changing content	Qwen3.5 / Mistral	A stable discourse frame repeats while the words inserted into it keep changing, or the model invents a new task, including both sides of a conversation, that can sustain generation.
Single-symbol collapse	OLMo-3-7B / Nemotron-3.5 / Qwen3-14B	The generation stops producing new material and emits one surface form—an echoed glyph, an emoji, a model-introduced symbol, or a control marker—to the generation cap.
Unfixed breakdown position	Nemotron-3.5	Severe repetition occupies either the generated reasoning or the eventual answer, so the quality of the answer does not reveal whether it occurred.

Appendix

Table 9: Qualitatively distinct failure patterns observed among selected samples in and near the target event. The categories describe examples rather than estimate their frequency within the tail.

Explore similar work

Estimating Tail Risks in Language Model Output Distributions
Apr 24, 2026Rico Angell, Raghav Singhal, Zachary Horvitz +4Large Language Model Safety Large Language Model Alignment
Center for Data Science, New York University · Department of Computer Science, New York University · Columbia University
Estimating Rare Events in Language Models with Proper Evaluation
Jul 20, 2026Nikita Y. Parulekar, Anqi Liu
Johns Hopkins University
When Behavioral Safety Evaluation Fails: A Representation-Level Perspective
Jun 6, 2026Enyi Jiang, Anders Gjølbye, Yibo Jacky Zhang +1Large Language Model Safety Representation-Level Misalignment
1Stanford University · University of Illinois Urbana-Champaign · 3Technical University of Denmark

Apr 24, 2026, cs.LGJ/K move · Enter open · S save

Estimating Tail Risks in Language Model Output Distributions

Rico Angell, Raghav Singhal, Zachary Horvitz, Zhou Yu+3

Center for Data Science, New York University · Department of Computer Science, New York University · Columbia University

Large Language Model Safety Large Language Model Alignment Bad

Open paper AskPDF arXiv

Language models are increasingly capable and are being rapidly deployed on a population-level scale. As a result, the safety of these models is increasingly high-stakes. Fortunately, advances in alignment have significantly reduced the likelihood of harmful model outputs. However, when models are queried billions of times in a day, even rare worst-case behaviors will occur. Current safety evaluations focus on capturing the distribution of inputs that yield harmful outputs. These evaluations disregard the probabilistic nature of models and their tail output behavior. To measure this tail risk, we propose a method to efficiently estimate the probability of harmful outputs for any input query. Instead of naive brute-force sampling from the target model, where harmful outputs could be rare, we operationalize importance sampling by creating unsafe versions of the target model. These unsafe versions enable sample-efficient estimation by making harmful outputs more probable. On benchmarks measuring misuse and misalignment, these estimates match brute-force Monte Carlo estimates using 10-20x fewer samples. For example, we can estimate probability of harmful outputs on the order of 10^-4 with just 500 samples. Additionally, we find that these harmfulness estimates can reveal the sensitivity of models to perturbations in model input and predict deployment risks. Our work demonstrates that accurate rare-event estimation is both critical and feasible for safety evaluations. Code is available at https://github.com/rangell/LMTailRisk

Abstract

Explore similar work

Estimating Tail Risks in Language Model Output Distributions

Estimating Rare Events in Language Models with Proper Evaluation

When Behavioral Safety Evaluation Fails: A Representation-Level Perspective