Benchmarking the Benchmarks: Evaluating Automated Safety Benchmarks for Small Language Models

Nyamtulla Shaik; Fengjun Li; Bo Luo

cs.AIAug 17, 2026

Benchmarking the Benchmarks: Evaluating Automated Safety Benchmarks for Small Language Models

Authors: Nyamtulla Shaik, Fengjun Li, Bo Luo

Organizations: EECS/I2S, University of Kansas, Lawrence, KS, 66045, USA

Abstract

Small Language Models (SLMs) are increasingly deployed in resource-constrained, privacy-sensitive settings, where safety and bias failures can cause security and societal risks. However, existing AI safety\slash security\slash compliance benchmarks are designed for large language models that may not transfer reliably to SLMs. We therefore ask: Can these benchmarks effectively and reliably evaluate SLMs? To answer this question, we conduct a large-scale assessment of the effectiveness and robustness of these automated pipelines by evaluating five widely used benchmark suites across 26 open-source SLMs under a unified judging rubric, which assigns a score of 0, 1, or 0.5 to harmful, safe, or ambiguous/irrelevant responses, respectively. Across the benchmarks, ambiguous judgments dominate and correlate with prompt complexity and model architecture, indicating that {\em LLM-centric safety benchmarks are insufficient as standalone evidence for SLM safety assessment}. In general, the ambiguity rate increases with lexical density, output perplexity, and output length and decreases with lexical sophistication, self-coherence, and reply-prompt similarity. This reveals a capability-safety confound that mixes model capability with apparent safety. Since ambiguity is prevalent, aggregate mean-score leaderboards are mathematically brittle: model rankings change significantly under reasonable ambiguity treatments, even when the underlying outputs remain unchanged.

Figures & tables

Figure 1: Safety rankings for 26 SLMs in 14 model families across 5 benchmark suites: higher mean scores indicate the models exhibit safer behavior.

Panel of Figure 2 — Figure 2: The benchmark scores and proportions of the scores per model for (a) SLM runs and (b) LLM results reported in the literature [ 24 , 57 ] .

Figure 3: Score distributions for SLMs vs. LLMs on the HELM-safety suite. For SALAD, the raw panel shows MD-Judge unsafe probabilities and the ternary panel shows the derived $\{0,0.5,1\}$ ambiguity scale used in the joint analysis.

Figure 4: Rank sensitivity to ambiguity handling. Each line traces a model’s rank (1 is best) under different treatments of ambiguity. AirBench, HarmBench and SALAD-Bench exhibit substantial reordering across scenarios, while Simple Safety shows relatively stable rankings, consistent with its lower ambiguity mass.

Figure 5: Prompt-level metric associations with ambiguity rate across all combined safety suites. Each point is one prompt, with AR computed as the fraction of SLMs receiving score 0.5 for that prompt. The first row shows the three strongest FDR-significant positive Pearson associations with AR, and the second row shows the three strongest FDR-significant negative associations.

Benchmark	Best classifier	Acc.	Bal. Acc.	ROC-AUC	Test rows
AirBench	Random forest	0.786	0.773	0.855	29,614
HarmBench	CatBoost	0.786	0.789	0.863	2,080
SALAD	XGBoost	0.608	0.616	0.664	106,600

Table 1: Benchmark-specific ambiguity prediction using grouped 80/20 train/test splits by instance_id . The binary target is whether the judge score is ambiguous (

0.5

) versus decisive (

0

1

); the table reports the best classifier by balanced accuracy for each benchmark.

Source	Target	Best classifier	Acc.	Bal. Acc.	ROC-AUC	Test rows
AirBench	HarmBench	CatBoost	0.701	0.707	0.810	10,400
AirBench	SALAD	Random forest	0.607	0.568	0.578	532,950
HarmBench	AirBench	CatBoost	0.744	0.746	0.794	148,044
HarmBench	SALAD	Random forest	0.551	0.553	0.563	532,950
SALAD	AirBench	CatBoost	0.667	0.604	0.651	148,044
SALAD	HarmBench	CatBoost	0.559	0.576	0.634	10,400

Table 2: Cross-benchmark ambiguity-prediction transfer. Trains on the source benchmark and evaluates on the target benchmark; the table reports the best classifier by balanced accuracy for each source–target pair.

Benchmark	Best classifier	Acc.	Bal. Acc.	ROC-AUC	Test rows
AirBench	CatBoost	0.809	0.803	0.877	34,164
HarmBench	LightGBM	0.830	0.832	0.905	2,400

Table 3: Aggregate ambiguity prediction under unseen-SLM transfer. Classifiers are trained on 20 SLMs and evaluated on six unseen SLMs selected to balance model size, family coverage, and tuning type.

Model metric	Description	$r_{\mathrm{AR}}$	$p_{\mathrm{AR}}$	$r_{\mathrm{score}}$	$p_{\mathrm{score}}$
Instruction tuned	Instruction-tuned checkpoint	-0.49	0.011	0.36	0.068
Heads	Attention head count	-0.42	0.035	-0.25	0.21
Hidden size	Internal representation width	-0.34	0.09	-0.05	0.796
RMSNorm	RMS-based normalization	-0.30	0.141	0.24	0.244
GQA	Grouped-query attention	-0.27	0.18	0.17	0.405
Params	Total model size	-0.27	0.183	0.10	0.641

Table 4: Pearson correlations between model metadata and (i) ambiguity rate (AR) and (ii) mean safety score (mean ternary score) across all 26 SLMs.

Suite	AR (GPT)	AR (Llama)	$\Delta$ AR	Exact agree	$\kappa_{w}$	$0.5\!\leftrightarrow\!$ $\{0,1\}$	$0\!\leftrightarrow\!1$	Rank- range (GPT)	Rank- range (Llama)
HarmBench	0.55	0.47	0.08	0.75	0.56	0.21	0.04	7.44	4.50
Simple Safety	0.06	0.03	0.04	0.81	0.70	0.08	0.11	0.75	1.19

Table 5: Judge robustness summary (GPT vs. Llama) for HarmBench and Simple Safety Tests. Ambiguity rates (AR), agreement metrics, contradiction types, and rank–range under ambiguity mappings.

	HarmBench ( $N=6400$ )			Simple Safety Tests ( $N=1600$ )
	Llama 0	Llama 0.5	Llama 1	Llama 0	Llama 0.5	Llama 1
GPT 0	234 (3.66%)	196 (3.06%)	2 (0.03%)	640 (40.00%)	17 (1.06%)	92 (5.75%)
GPT 0.5	502 (7.84%)	2594 (40.53%)	437 (6.83%)	45 (2.81%)	10 (0.62%)	47 (2.94%)
GPT 1	261 (4.08%)	233 (3.64%)	1941 (30.33%)	85 (5.31%)	13 (0.81%)	651 (40.69%)

Table 6: Confusion matrices between GPT and Llama judges. Cells report count (percent of suite instances). The main pattern is that disagreement concentrates around the ambiguous boundary (

0.5\leftrightarrow\{0,1\}

), especially on HarmBench, rather than direct

0\leftrightarrow 1

flips. Simple Safety shows stronger diagonal concentration overall, indicating higher cross-judge stability on that suite.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

	AirBench						SALAD						HarmBench
Model	$S_{\mathrm{M}}$	HCR	AR	SRR	$S_{adj}$	$\Delta$ R	$S_{\mathrm{M}}$	HCR	AR	SRR	$S_{adj}$	$\Delta$ R	$S_{\mathrm{M}}$	HCR	AR	SRR	$S_{adj}$	$\Delta$ R
GPT-2 Small	0.59	0.07	0.69	0.24	0.18	$\downarrow$ 14	0.58	0.22	0.39	0.38	0.35	$\downarrow$ 2	0.56	0.01	0.86	0.13	0.08	$\downarrow$ 11
OPT-125M	0.49	0.11	0.81	0.08	0.09	$\downarrow$ 16	0.61	0.13	0.52	0.35	0.29	$\downarrow$ 15	0.58	0.01	0.82	0.17	0.11	$\downarrow$ 10
Smollm2 135m IT	0.33	0.46	0.43	0.11	0.19	$\uparrow$ 2	0.46	0.47	0.14	0.39	0.39	$\uparrow$ 11	0.51	0.12	0.74	0.14	0.13	$\uparrow$ 2
Gemma 3-270M IT	0.48	0.19	0.66	0.16	0.17	$\downarrow$ 11	0.71	0.21	0.15	0.64	0.61	$\uparrow$ 2	0.68	0.01	0.61	0.38	0.26	$\downarrow$ 1
OPT-350M	0.48	0.15	0.74	0.11	0.12	$\downarrow$ 10	0.58	0.20	0.43	0.36	0.33	$\downarrow$ 5	0.57	0.01	0.84	0.15	0.09	$\downarrow$ 11

Appendix

Table A1: Per-model outcome summary for AirBench, SALAD, and HarmBench. We report the raw ternary-score mean

S_{\mathrm{M}}

, harmful-completion rate (HCR; score

=0

), ambiguity rate (AR; score

=0.5

), safe-refusal rate (SRR; score

=1

), and the ambiguity-adjusted score

S_{adj}=S_{\mathrm{M}}\cdot(1-\mathrm{AR})

. We also report the implied rank change (

\Delta

R) relative to ranking by

S_{\mathrm{M}}

. In

\Delta

\uparrow

\downarrow

indicates direction, and the number is the absolute number of rank positions moved.

Vendor, year / Model	Params / Ctx	Architecture (reported)	Training data (reported)	Objective (reported)
OpenAI 2019 GPT-2 Small [ 41 ]	124M 1024	Decoder-only transformer; 12 layers; 12 attention heads; 768 hidden; 3072 FFN; GELU; absolute position embeddings; vocab 50257	WebText $\sim$ 40GB from 8M web pages; cutoff Dec 2017	Causal language modeling
Meta 2022 OPT-125M [ 33 ]	125M 2048	Decoder-only transformer (OPT/GPT-style); 12 layers; 12 heads; 768 hidden; 3072 FFN; ReLU; max pos 2048; vocab 50272	180B tokens; BookCorpus; Common Crawl; Reddit; predominantly English	Causal language modeling
HF 2024 SmolLM2-135M IT [ 21 ]	135M 8192	Transformer decoder (Llama-family); 30 layers; 9 heads; 3 KV heads; 576 hidden; 1536 FFN; vocab 49152	2T-token curated pre-training mix; instruction-tuning data	Pre-training + SFT + DPO
Google 2025 Gemma3-270M IT [ 17 ]	270M 32768	Decoder-only; 270M total (170M embedding from 256K vocab + 100M transformer blocks)	2T tokens; web; code; math; 140+ languages	Pre-training + instruction tuning
Meta 2022 OPT-350M [ 34 ]	350M 2048	Decoder-only transformer (OPT/GPT-style); 24 layers; 16 heads; 1024 hidden; 4096 FFN; ReLU; max pos 2048; vocab 50272	180B tokens; BookCorpus; Common Crawl; Reddit; predominantly English	Causal language modeling
OpenAI 2019 GPT-2 Medium [ 43 ]	355M 1024	Decoder-only transformer; 24 layers; 16 attention heads; 1024 hidden; 4096 FFN; GELU; absolute position embeddings; vocab 50257	WebText $\sim$ 40GB from 8M web pages; cutoff Dec 2017	Causal language modeling

Appendix

Table A2: SLMs evaluated in this study (26 models spanning 124M–4.0B parameters) and the model metadata. The table highlights substantial heterogeneity in size, context length, tuning, and architectural choices, motivating family- and metadata-aware interpretation of benchmark outcomes.

Language Model Safety Evaluation

Language model safety evaluation measures harmful behavior, refusal, robustness, and over-refusal across realistic prompts, users, and deployment settings.

See all 543 papers

No One Model Catches Every Harm: Benchmarking Content Moderation Across Safety Scenarios
Aug 22, 2026Afshin Orojlooyjadid, Hitesh PatelLLM Safety Evaluation LLM Safety Benchmarks
Oracle AI
SLMJury: Can Small Language Models Judge as Well as Large Ones?
Jun 5, 2026Anish Laddha, Nitesh Pradhan, Gaurav SrivastavaLLM-as-a-Judge Language Model Generation Evaluation
Department of Computer Science and Engineering, LNMIIT, Jaipur, India · Department of Computer Science, Virginia Tech, Blacksburg, VA, USA
Can Safety Emerge from Weak Supervision? A Systematic Analysis of Small Language Models
Mar 7, 2026Punyajoy Saha, Sudipta Halder, Debjyoti Mondal +1LLM Alignment Multi-Objective Language Model Alignment
Samsung Research Institute Bangalore, India

Aug 22, 2026J/K move · Enter open · S save

No One Model Catches Every Harm: Benchmarking Content Moderation Across Safety Scenarios

Afshin Orojlooyjadid, Hitesh Patel

Oracle AI

LLM Safety Evaluation LLM Safety Benchmarks Content Moderation LLM Safety Monitoring

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly deployed in real-world applications, yet they remain vulnerable to generating harmful content. From adversarial jailbreaks that bypass safety filters to implicit hate that evades detection, the range of risks these models pose continues to grow. While both specialized content moderators and general-purpose LLMs are being used as safety layers, the question of which model is best suited for which type of harmful content remains unanswered. We present the most comprehensive evaluation of LLM safety capabilities to date, systematically testing 53 models across 11 datasets that we organize into four distinct categories. Our evaluation under both prompt-only and prompt-response settings uncovers critical blind spots: large frontier models that lead on one category fall significantly behind smaller, specialized alternatives on others, and real-world conversational safety remains largely unsolved across all model families. These findings challenge the assumption that scale alone ensures safety, and provide the community with a structured framework for informed model selection.

Abstract

Language Model Safety Evaluation

Explore similar work

No One Model Catches Every Harm: Benchmarking Content Moderation Across Safety Scenarios

SLMJury: Can Small Language Models Judge as Well as Large Ones?

Can Safety Emerge from Weak Supervision? A Systematic Analysis of Small Language Models