Prioritizing Repeated LLM Evaluation for Hidden Failure Discovery

Keita Broadwater; Akin Broadwater

cs.LGSep 26, 2026

Prioritizing Repeated LLM Evaluation for Hidden Failure Discovery

Authors: Keita Broadwater, Akin Broadwater

Organizations: Independent Researcher San Jose, CA USA · Bellarmine College Prepatory San Jose, CA USA

Abstract

Large language models are commonly evaluated by generating a small number of stochastic responses for each prompt in a benchmark. Because inference budgets are limited, this shallow evaluation may fail to observe low-probability but operationally important failures. A prompt that produces no failures in a small sample may therefore appear reliable despite having a nonzero latent probability of failure under repeated inference. We formulate LLM reliability evaluation as a budget-constrained discovery problem in which each prompt is associated with an unknown per-generation failure probability. We propose a budgeted discovery framework that first performs shallow evaluation across the prompt set and then uses trial-level failure outcomes together with prompt-derived representations to learn a feature-based ranking of failure propensity. The resulting scores prioritize prompts with zero observed shallow failures for deeper evaluation, concentrating the deep-evaluation budget where hidden failures are more likely to be discovered. We evaluate this approach on AIRBench and StrongREJECT across multiple model and system-prompt conditions. The central empirical test asks whether models fit without access to deep-evaluation outcomes can rank prompts with zero observed shallow failures according to their likelihood of producing failures under deeper evaluation. On AIRBench, the highest-ranked 10% of unresolved prompts achieves 2.54x hidden-failure lift for Qwen 2.5 7B and 1.87x for Gemma 3n E4B, recovering 25.4% and 18.7% of subsequently observed hidden failures, respectively, compared with 10% expected under random allocation. Semantic-neighborhood and feature-ablation analyses further show that this predictive signal can be recovered from multiple representations of prompt content and relationships.

Figures & tables

Figure 1: Two-stage budgeted discovery. Shallow stochastic outcomes and prompt-derived representations are used to rank prompts with no observed failures; the deep-evaluation budget is then concentrated on higher-ranked prompts. Deep-evaluation outcomes are withheld from model fitting.

	Qwen 2.5 7B	Gemma 3n E4B
Unsafe-response rate, $N=5$	7.50%	0.63%
Unsafe-response rate, $N=150$	7.54%	0.63%
Positive prompts, $N=5$	771	84
Positive prompts, $N=150$	1,554	290
Unresolved prompts at $N=5$	4,923	5,610
Hidden failures	790	214

Table 1: AIRBench failure discovery under shallow (

N=5

) and deep (

N=150

) evaluation. Aggregate response-level failure rates remain nearly unchanged while deeper sampling reveals substantially more failure-capable prompts.

Figure 2: AIRBench hidden-failure rates by predicted-risk decile for Qwen 2.5 7B and Gemma 3n E4B. Deciles are computed within the population of prompts with zero observed failures at

N=5

, using grouped out-of-fold embedding + text XGBoost scores. Hidden failures are identified through

N=150

evaluation. Dashed lines show unresolved-population prevalence; error bars show 95% confidence intervals.

Figure 3: AIRBench hidden-failure recovery versus follow-up evaluation budget. Unresolved prompts are selected in descending order of grouped out-of-fold embedding + text XGBoost scores. The horizontal axis gives the fraction of unresolved prompts selected for deep evaluation; the vertical axis gives the fraction of hidden failures observed at

N=150

recovered. The dashed diagonal shows expected recovery under random selection.

Ranking method	Qwen	Gemma
Random allocation	1.00 $\times$	1.00 $\times$
Prompt length (longest first)	0.82 $\times$	1.64 $\times$
Mean top-5 similarity to shallow positives	2.40 $\times$	1.68 $\times$
20-NN shallow-positive prevalence	2.31 $\times$	1.82 $\times$
Logistic: embedding + text	1.63 $\times$	1.54 $\times$
XGBoost: embedding + text	2.54 $\times$	1.87 $\times$

Table 2: Hidden-failure lift at a 10% deep-evaluation budget. Predictors use only

N=5

information and are ranked within the unresolved population.

Condition	Endpoint	$n$	$H$	$h/K$	Rate	Lift	Recall
Basic safety prompt	Binary safety	309	2	1/31	3.2%	4.98 $\times$	50.0%
Basic safety prompt	StrongREJECT-positive	305	8	2/31	6.5%	2.46 $\times$	25.0%
No safety prompt	Binary safety	277	21	3/28	10.7%	1.41 $\times$	14.3%
No safety prompt	StrongREJECT-positive	279	16	2/28	7.1%	1.25 $\times$	12.5%

Table 3: Within-unresolved top-10% discovery on StrongREJECT.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	Human 0	Human 0.5	Human 1
Judge 0	14	3	6
Judge 0.5	57	3	17
Judge 1	15	4	81

Appendix

Table 4: Automated versus human AIRBench scores in the primary validation sample (

n=200

Condition	Agreement	Cohen’s $\kappa$
Basic safety prompt	98.8%	0.387
No system prompt	95.8%	0.634

Appendix

Table 5: Response-level agreement between the binary safety outcome and StrongREJECT judgment for Qwen.

Learned models
	Qwen		Gemma
Ranking / representation	$h/K$	Lift	$h/K$	Lift
Random allocation	–	1.00 $\times$	–	1.00 $\times$
XGBoost: embedding + text	201/493	2.54 $\times$	40/561	1.87 $\times$
XGBoost: full representation	198/493	2.50 $\times$	44/561	2.06 $\times$
XGBoost: embedding only	193/493	2.44 $\times$	37/561	1.73 $\times$

Appendix

Table 6: Complete AIRBench top-10% ranking comparison within the unresolved population. All methods use the common model-specific evaluation populations defined above; comparisons among methods are descriptive.

Condition	Endpoint	$N{=}5$ responses	$N{=}5$ prompts	$N{=}25$ responses	$N{=}25$ prompts
Basic safety	Binary safety	10/1,565	4/313	46/7,825	6/313
Basic safety	StrongREJECT	16/1,565	8/313	102/7,825	16/313
No safety prompt	Binary safety	93/1,565	36/313	485/7,825	57/313
No safety prompt	StrongREJECT	93/1,565	34/313	466/7,825	50/313

Appendix

Table 7: Repeated evaluation on StrongREJECT. Response counts report positive individual generations; prompt counts report prompts with at least one positive response by the indicated depth.

Condition	Endpoint	$n$	$H$	$K$	$h$	Selected	Lift	Recall
Basic safety	Binary safety	309	2	31	1	3.23%	4.98 $\times$	50.0%
Basic safety	StrongREJECT	305	8	31	2	6.45%	2.46 $\times$	25.0%
No safety prompt	Binary safety	277	21	28	3	10.71%	1.41 $\times$	14.3%
No safety prompt	StrongREJECT	279	16	28	2	7.14%	1.25 $\times$	12.5%

Appendix

Table 8: Within-unresolved top-10% targeted discovery on StrongREJECT.

H

is the number of hidden failures in the unresolved population and

h

the number discovered among the selected

K

prompts.

Explore similar work

Evaluating Reliability Gaps in Large Language Model Safety via Repeated Prompt Sampling
Mar 10, 2026Keita BroadwaterLLM Evaluation Language Model Safety Evaluation
Independent Researcher, San Jose, USA
Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarks
May 11, 2026Eungyeup Kim, Chenchen Gu, Vashisth Tiwari +1LLM Evaluation LLM Reliability
Carnegie Mellon University
How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure
Sep 24, 2026Dipankar SarkarLLM Evaluation LLM Auditing
Skelf Research

Mar 10, 2026J/K move · Enter open · S save

Evaluating Reliability Gaps in Large Language Model Safety via Repeated Prompt Sampling

Keita Broadwater

Independent Researcher, San Jose, USA

LLM Evaluation Language Model Safety Evaluation LLM Reliability LLM Safety Benchmarks

Open paper AskPDF arXiv

Traditional benchmarks for large language models (LLMs), such as HELM and AIR-BENCH, primarily assess safety risk through breadth-oriented evaluation across diverse tasks. However, real-world deployment often exposes a different class of risk: operational failures arising from repeated generations of the same prompt rather than broad task generalization. In high-stakes settings, response consistency and safety under repeated use are critical operational requirements. We introduce Accelerated Prompt Stress Testing (APST), a depth-oriented evaluation framework inspired by highly accelerated stress testing in reliability engineering. APST probes LLM behavior by repeatedly sampling identical prompts under controlled operational conditions, including temperature variation and prompt perturbation, to surface latent failure modes such as hallucinations, refusal inconsistency, and unsafe completions. Rather than treating failures as isolated events, APST characterizes them statistically as stochastic outcomes of repeated inference. We model observed safety failures using Bernoulli and binomial formulations to estimate per-inference failure probabilities, enabling quantitative comparison of operational risk across models and configurations. We apply APST to multiple instruction-tuned LLMs evaluated on AIR-BENCH 2024 derived safety and security prompts. While models exhibit similar performance under conventional single- or very-low-sample evaluation (N <= 3), repeated sampling reveals substantial variation in empirical failure probabilities across temperatures. These results demonstrate that shallow benchmark scores can obscure meaningful differences in reliability under sustained use.

Abstract

Explore similar work

Evaluating Reliability Gaps in Large Language Model Safety via Repeated Prompt Sampling

Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarks

How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure