cs.CRMar 3, 2026

Benchmark of Benchmarks: Unpacking Influence and Code Repository Quality in LLM Safety Benchmarks

Authors: Junjie Chu, Xinyue Shen, Ye Leng, Michael Backes, Yun Shen, Yang Zhang

Organizations: CISPA Helmholtz Center for Information Security · University of Waterloo · Flexera

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The rapid expansion of research in LLM safety presents challenges in tracking advancements, making benchmarks important evaluation infrastructures for identifying key trends and facilitating systematic comparisons. Yet no systematic assessment exists of their code quality and runnability, nor of what factors are associated with the community's adoption of certain benchmarks over others. To address this gap, we conduct a systematic measurement study of 31 LLM safety benchmarks (covering prompt injection, jailbreak, and hallucination) with 382 non-benchmark papers as a control group, combining automated static analysis, human runnability testing (220+ person-hours), and bibliometric analysis. We find that only 39% of benchmark repositories can run without modification, only 16% provide flawless installation guides, and a mere 6% include ethical considerations despite containing potentially harmful content. These deficiencies persist across the study period with no significant improvement. Analyzing adoption factors, we find that benchmark adoption correlates with author prominence and code runnability, but not with code quality standards such as Pylint score and maintainability, suggesting that the community's benchmark selection does not reward higher coding standards. Based on these results, we identify potential safety and reliability concerns. Some safety benchmark repositories openly expose harmful content, such as successful jailbreak responses, without any ethical warning or access control, effectively serving as unguarded attack resources. Furthermore, when benchmarks require ad-hoc modifications to run, downstream safety evaluations across different papers may not be comparable. We present case studies illustrating these concrete consequences and propose a targeted checklist to help benchmark contributors improve code quality, documentation, and ethical practices.

Figures & tables

Figure 1: Cumulative count of LLM and LLM safety papers since ChatGPT was released (data from Semantic Scholar).
Figure 2: Data collection pipeline.
 BenchmarkNon-Benchmark
Collection Period[November 30, 2022 – November 1, 2024]
# Papers31382
# Repositories27168
# Jailbreak12246
# Hallucination13103
# Prompt Injection633
Table 1: Summary of the Data Under Study
AspectDescription
Code QualityIs the necessary dataset available in the repository?
 Are the provided example scripts runnable?
 Are there any extra modifications required (such as fixing bugs) besides those mentioned in the guides when trying to run the example scripts?
Supplementary Material QualityDoes the repository provide good install guides?
 Does the repository provide useful data guides?
 Does the repository contain essential ethical considerations?
Table 2: An outline of the metrics measured during the human-based evaluation.
DimensionCohen’s κ\kappaGwet’s AC1PABAK
Runnable Demo0.7040.8850.833
Install Guide0.4670.9020.833
Ethical Consideration1.0001.0001.000
Data Available1.0001.0001.000
Data Guidance1.0001.0001.000
Table 3: Inter-rater agreement for human-based evaluation.
Metricpp -valueCliff’s δ\delta95% CIEffect Size
Pylint Score0.031−- 0.276[ −- 0.525, −- 0.014]Small
Cyclomatic Complexity0.649−- 0.055[ −- 0.296, 0.195]Negligible
Maintainability Index0.244−- 0.140[ −- 0.388, 0.113]Negligible
Number of Static Errors0.783−- 0.014[ −- 0.132, 0.078]Negligible
Reply Time (Hours)0.044−- 0.239[ −- 0.426, −- 0.040]Small
Last Commit Time (Days)0.4910.083[ −- 0.138, 0.299]Negligible
Table 4: Code quality M-W U test results with 95% bootstrap CIs. Positive δ\delta : non-benchmark dominates; negative: benchmark dominates.
Metric∣δ∣obs|\delta|_{\text{obs}}pp -valuePower
Pylint Score*0.2760.0310.774
Cyclomatic Complexity0.0550.6490.822
Maintainability Index0.1400.2440.822
Number of Static Errors0.0140.7830.822
Reply Time (Hours)*0.2390.0440.822
Last Commit Time (Days)0.0830.4910.822
Table 5: Sensitivity power analysis for code quality metrics at fixed target ∣δ∣=0.33|\delta|=0.33 . Metrics marked with * are statistically significant ( p<0.05p<0.05 ).
Figure 3: Human-based evaluation results of code quality.
Figure 4: Human-based evaluation results of supplementary materials. Repositories without code or unavailable ones are labeled “Not Applicable.” For install guides: repositories are categorized as “No” (no guide), “Partial” (guide with setup issues), or “Yes” (fully functional guide); if a guide includes all required Python/library/package versions, it is labeled “Yes: no flaws”; if version information is missing but the guide is still runnable, it is “Yes: lack versions.”
PeriodNNPylintRunnableEthicalInstall Guide
2023H124.91100%0%50%
2023H296.2887.5%0%100%
2024H1165.9183.3%13.3%100%
2024H234.06100%0%100%
K-W pp -value0.8280.8190.4990.007
Table 6: Temporal stratification of benchmark code quality by half-year period.
AspectFactorDescriptionType
AuthorAuthor NumberThe number of the authors of a paper.Quantitative
Author Citation Count (Top-1)The top-1 (highest) citation count among all authors in a paper.Quantitative
Author H-Index (Top-1)The top-1 (highest) h-index among all authors in a paper.Quantitative
InstitutionInstitution NumberThe number of institutions a paper is affiliated with.Quantitative
Industry Involvement StatusWhether organizations from industry participate in a paper.Qualitative
Institution CSRankings (Top-1)The top-1 (highest) ranking in CSRankings among all institutions a paper is affiliated with.Quantitative
Table 7: Summary of the 11 factors from five dimensions studied in this paper.
Metricpp -valueCliff’s δ\delta95% CIEffect Size
GitHub Star Density0.012−- 0.301[ −- 0.501, −- 0.091]Small
GitHub Star Count0.004−- 0.347[ −- 0.536, −- 0.151]Medium
Citation Density0.309−- 0.112[ −- 0.310, 0.090]Negligible
Citation Count0.237−- 0.130[ −- 0.320, 0.068]Negligible
Scientific Field Count0.632−- 0.052[ −- 0.247, 0.143]Negligible
Table 8: Adoption M-W U test results with 95% bootstrap CIs. Positive Cliff’s δ\delta : non-benchmark dominates; negative: benchmark dominates.
Metric∣δ∣obs|\delta|_{\text{obs}}pp -valuePower
GitHub Star Density*0.3010.0120.822
GitHub Star Count*0.3470.0040.822
Citation Density0.1120.3090.881
Citation Count0.1300.2370.882
Scientific Field Count0.0520.6320.882
Table 9: Sensitivity power analysis for adoption metrics at fixed target ∣δ∣=0.33|\delta|=0.33 . Metrics marked with * are statistically significant ( p<0.05p<0.05 ).
Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: PRISMA-style flow diagram for benchmark selection.
Figure 6: Average values of five influence-related metrics on benchmark and non-benchmark papers.
Figure 7: Distribution of the scientific fields that the LLM safety papers influence.
Figure 8: Average values of eight metrics related to the code repository quality on benchmark and non-benchmark papers. We have flipped the axes as needed, so now all points/lines on the upper side indicate better quality or performance.
PredictorCoef.Std. Err.t-valuep-value
Intercept-2.97e-170.143-0.00021.000
Author Number-0.08020.172-0.4660.646
Institution Number0.51970.1782.9190.008
Area Number-0.33470.182-1.8390.079
Author H-Index (Top-1)0.98270.2853.4520.002
Author Citation Count (Top-1)-0.79930.284-2.8150.010
Appendix
Table 10: OLS Regression Results: Author & Institution Factors
PredictorCoef.Std. Err.t-valuep-value
Intercept-2.97e-170.159-0.00021.000
Pylint Score-0.18970.182-1.0400.310
Cyclomatic Complexity0.34650.2571.3500.191
Maintainability Index-0.02400.178-0.1350.894
Number of Static Errors0.44530.1652.6990.013
Server Reply Time (Hours)0.18090.1671.0810.291
Appendix
Table 11: OLS Regression Results: Code Quality Factors
Figure 9: Average time to successfully run the example scripts in the repositories.
DimensionMetric
Academic CommunityCitation Count
 Citation Density
Open-Source CommunityGitHub Star Count
 GitHub Star Density
Cross-DisciplinaryScientific Field Count
Appendix
Table 12: Summary of the metrics used in adoption evaluation.
Figure 10: GitHub repository availability proportions.
Figure 11: A typical example of the general pattern we identify. After the paper is publicly available, its GitHub star count increases rapidly.
Figure 12: Spearman correlation ρ\rho matrix of the adoption metrics (those with p≥0.05p\geq 0.05 are omitted).
Figure 13: Spearman correlation matrices between the adoption metrics and the potential quantitative factors. The unadjusted p-values on the left can be interpreted exploratively.
Figure 14: Spearman correlation matrices between the adoption metrics and the code repository quality metrics. The unadjusted p-values on the left can be interpreted exploratively.
Figure 15: Spearman correlation matrices between the code repository quality metrics and the potential quantitative factors. The unadjusted p-values on the left can be interpreted exploratively.
Figure 16: Box plots of citation density by group. The red dashed lines represent the means.
Figure 17: Box plots of citation density and the status of extra modifications and runnable code. The red dashed lines represent the mean values.
Figure 18: Box plots of citation density and various potential qualitative factors. The red dashed lines in the box plots represent the mean values. The further to the right the x-axis tick is, the higher the mean value of the corresponding box.
Figure 19: Box plots of Pylint score and various potential qualitative factors. The red dashed lines in the box plots represent the mean values. The further to the right the x-axis tick is, the higher the mean value of the corresponding box.
More in this topic

Benchmark Reproducibility

See all 21 papers

Explore similar work

CardsList
Jan 18, 2025

Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility

Code-related benchmarks play a critical role in evaluating large language models (LLMs), yet their quality fundamentally shapes how the community interprets model capabilities. In the past few years, awareness of benchmark quality has grown. Yet, after a decade-scale (2014-2025) survey over 672 code benchmarks, we observed a lag between growing awareness and actual practice. For example, in 2025 alone, the number of benchmarks that ignore code coverage when providing test cases nearly matches the total count accumulated across the previous ten years. In response, we take a clear position: Code benchmarks must prioritize rigor in benchmark construction, reliability in evaluation, and reproducibility in release. To operationalize this position, we introduce a code benchmark guideline HOW2BENCH with 55 checklists. Finally, our further human study also exposed that the current issues not only stem from the significant effort required, but also from a lack of awareness regarding their importance.

Jialun Cao, Yuk-Kit Chan, Zixuan Ling +12

The Hong Kong University of Science and Technology · Guangzhou HKUST Fok Ying Tung Research Institute · The Chinese University of Hong Kong +6

Benchmark ConstructionBenchmark DesignCode Generation EvaluationML ReproducibilityBenchmark Reproducibility
AskPDF
Apr 14, 2026

AISafetyBenchExplorer: A Metric-Aware Catalogue of AI Safety Benchmarks Reveals Fragmented Measurement and Weak Benchmark Governance

The rapid expansion of large language model (LLM) safety evaluation has produced a substantial benchmark ecosystem, but not a correspondingly coherent measurement ecosystem. We present AISafetyBenchExplorer, a structured catalogue of 195 AI safety benchmarks released between 2018 and 2026, organized through a multi-sheet schema that records benchmark-level metadata, metric-level definitions, benchmark-paper metadata, and repository activity. This design enables meta-analysis not only of what benchmarks exist, but also of how safety is operationalized, aggregated, and judged across the literature. Using the updated catalogue, we identify a central structural problem: benchmark proliferation has outpaced measurement standardization. The current landscape is dominated by medium-complexity benchmarks (94/195), while only 7 benchmarks occupy the Popular tier. The workbook further reports strong concentration around English-only evaluation (165/195), evaluation-only resources (170/195), stale GitHub repositories (137/195), stale Hugging Face datasets (96/195), and heavy reliance on arXiv preprints among benchmarks with known venue metadata. At the metric level, the catalogue shows that familiar labels such as accuracy, F1 score, safety score, and aggregate benchmark scores often conceal materially different judges, aggregation rules, and threat models. We argue that the field's main failure mode is fragmentation rather than scarcity. Researchers now have many benchmark artifacts, but they often lack a shared measurement language, a principled basis for benchmark selection, and durable stewardship norms for post publication maintenance. AISafetyBenchExplorer addresses this gap by providing a traceable benchmark catalogue, a controlled metadata schema, and a complexity taxonomy that together support more rigorous benchmark discovery, comparison, and meta-evaluation.

Abiodun A. Solanke

LLM Safety BenchmarksBenchmark Auditing
AskPDF
Apr 11, 2026

Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents

The rapid deployment of LLM-based autonomous agents has introduced safety risks that extend far beyond traditional LLM concerns, prompting a proliferation of safety benchmarks since late 2023. However, these benchmarks have developed independently, with inconsistent threat models, incompatible metrics, and overlapping yet incomplete risk coverage. We present the first systematic analysis dedicated to agent safety benchmarks as evaluation instruments. We catalog 40 behavioral agent-safety benchmarks (2023-2026), plus 5 adjacent evaluator, defense, and dataset artifacts, propose a six-axis taxonomy of benchmark evaluation methodology, and apply it across the corpus to characterize how methodological choices shape safety conclusions. A coverage matrix reveals broad risk coverage but limited methodological convergence, while the taxonomy analysis shows a behavioral-benchmark core concentrated in sandboxed, constrained, and often safety-only evaluation. Across the landscape, we find that benchmark choice can yield contradictory safety conclusions, coverage counts often overstate evaluation depth, environment fidelity systematically shapes reported safety, the field disproportionately tests externally imposed rather than agent-internal risks, metric fragmentation limits comparison, and robustness remains effectively unbenchmarked. We ground these claims with a cross-benchmark consistency check, with 95% confidence intervals and Kendall's W concordance analysis, finding no evidence of ranking concordance across evaluation dimensions (W = 0.10, p = 0.94). We release structured metadata, full taxonomy codings, risk annotations, and all experimental artifacts, and propose minimum reporting standards for future benchmarks.

Miles Q. Li, Benjamin C. M. Fung, Boyang Li +2

School of Information Studies, McGill University, Canada · Department of Computer Science, Kean University, USA · College of Technological Innovation, Zayed University, UAE

AI Agent Safety BenchmarksAI Agent Evaluation
AskPDF