The rapid expansion of research in LLM safety presents challenges in tracking advancements, making benchmarks important evaluation infrastructures for identifying key trends and facilitating systematic comparisons. Yet no systematic assessment exists of their code quality and runnability, nor of what factors are associated with the community's adoption of certain benchmarks over others. To address this gap, we conduct a systematic measurement study of 31 LLM safety benchmarks (covering prompt injection, jailbreak, and hallucination) with 382 non-benchmark papers as a control group, combining automated static analysis, human runnability testing (220+ person-hours), and bibliometric analysis. We find that only 39% of benchmark repositories can run without modification, only 16% provide flawless installation guides, and a mere 6% include ethical considerations despite containing potentially harmful content. These deficiencies persist across the study period with no significant improvement. Analyzing adoption factors, we find that benchmark adoption correlates with author prominence and code runnability, but not with code quality standards such as Pylint score and maintainability, suggesting that the community's benchmark selection does not reward higher coding standards. Based on these results, we identify potential safety and reliability concerns. Some safety benchmark repositories openly expose harmful content, such as successful jailbreak responses, without any ethical warning or access control, effectively serving as unguarded attack resources. Furthermore, when benchmarks require ad-hoc modifications to run, downstream safety evaluations across different papers may not be comparable. We present case studies illustrating these concrete consequences and propose a targeted checklist to help benchmark contributors improve code quality, documentation, and ethical practices.
Figures & tables
Figure 1: Cumulative count of LLM and LLM safety papers since ChatGPT was released (data from Semantic Scholar).
Figure 2: Data collection pipeline.
Benchmark
Non-Benchmark
Collection Period
[November 30, 2022 – November 1, 2024]
# Papers
31
382
# Repositories
27
168
# Jailbreak
12
246
# Hallucination
13
103
# Prompt Injection
6
33
Table 1: Summary of the Data Under Study
Aspect
Description
Code Quality
Is the necessary dataset available in the repository?
Are the provided example scripts runnable?
Are there any extra modifications required (such as fixing bugs) besides those mentioned in the guides when trying to run the example scripts?
Supplementary Material Quality
Does the repository provide good install guides?
Does the repository provide useful data guides?
Does the repository contain essential ethical considerations?
Table 2: An outline of the metrics measured during the human-based evaluation.
Dimension
Cohen’s κ
Gwet’s AC1
PABAK
Runnable Demo
0.704
0.885
0.833
Install Guide
0.467
0.902
0.833
Ethical Consideration
1.000
1.000
1.000
Data Available
1.000
1.000
1.000
Data Guidance
1.000
1.000
1.000
Table 3: Inter-rater agreement for human-based evaluation.
Metric
p -value
Cliff’s δ
95% CI
Effect Size
Pylint Score
0.031
− 0.276
[ − 0.525, − 0.014]
Small
Cyclomatic Complexity
0.649
− 0.055
[ − 0.296, 0.195]
Negligible
Maintainability Index
0.244
− 0.140
[ − 0.388, 0.113]
Negligible
Number of Static Errors
0.783
− 0.014
[ − 0.132, 0.078]
Negligible
Reply Time (Hours)
0.044
− 0.239
[ − 0.426, − 0.040]
Small
Last Commit Time (Days)
0.491
0.083
[ − 0.138, 0.299]
Negligible
Table 4: Code quality M-W U test results with 95% bootstrap CIs. Positive δ : non-benchmark dominates; negative: benchmark dominates.
Metric
∣δ∣obs
p -value
Power
Pylint Score*
0.276
0.031
0.774
Cyclomatic Complexity
0.055
0.649
0.822
Maintainability Index
0.140
0.244
0.822
Number of Static Errors
0.014
0.783
0.822
Reply Time (Hours)*
0.239
0.044
0.822
Last Commit Time (Days)
0.083
0.491
0.822
Table 5: Sensitivity power analysis for code quality metrics at fixed target ∣δ∣=0.33 . Metrics marked with * are statistically significant ( p<0.05 ).
Figure 3: Human-based evaluation results of code quality.
Figure 4: Human-based evaluation results of supplementary materials. Repositories without code or unavailable ones are labeled “Not Applicable.” For install guides: repositories are categorized as “No” (no guide), “Partial” (guide with setup issues), or “Yes” (fully functional guide); if a guide includes all required Python/library/package versions, it is labeled “Yes: no flaws”; if version information is missing but the guide is still runnable, it is “Yes: lack versions.”
Period
N
Pylint
Runnable
Ethical
Install Guide
2023H1
2
4.91
100%
0%
50%
2023H2
9
6.28
87.5%
0%
100%
2024H1
16
5.91
83.3%
13.3%
100%
2024H2
3
4.06
100%
0%
100%
K-W p -value
0.828
0.819
0.499
0.007
Table 6: Temporal stratification of benchmark code quality by half-year period.
Aspect
Factor
Description
Type
Author
Author Number
The number of the authors of a paper.
Quantitative
Author Citation Count (Top-1)
The top-1 (highest) citation count among all authors in a paper.
Quantitative
Author H-Index (Top-1)
The top-1 (highest) h-index among all authors in a paper.
Quantitative
Institution
Institution Number
The number of institutions a paper is affiliated with.
Quantitative
Industry Involvement Status
Whether organizations from industry participate in a paper.
Qualitative
Institution CSRankings (Top-1)
The top-1 (highest) ranking in CSRankings among all institutions a paper is affiliated with.
Quantitative
Table 7: Summary of the 11 factors from five dimensions studied in this paper.
Metric
p -value
Cliff’s δ
95% CI
Effect Size
GitHub Star Density
0.012
− 0.301
[ − 0.501, − 0.091]
Small
GitHub Star Count
0.004
− 0.347
[ − 0.536, − 0.151]
Medium
Citation Density
0.309
− 0.112
[ − 0.310, 0.090]
Negligible
Citation Count
0.237
− 0.130
[ − 0.320, 0.068]
Negligible
Scientific Field Count
0.632
− 0.052
[ − 0.247, 0.143]
Negligible
Table 8: Adoption M-W U test results with 95% bootstrap CIs. Positive Cliff’s δ : non-benchmark dominates; negative: benchmark dominates.
Metric
∣δ∣obs
p -value
Power
GitHub Star Density*
0.301
0.012
0.822
GitHub Star Count*
0.347
0.004
0.822
Citation Density
0.112
0.309
0.881
Citation Count
0.130
0.237
0.882
Scientific Field Count
0.052
0.632
0.882
Table 9: Sensitivity power analysis for adoption metrics at fixed target ∣δ∣=0.33 . Metrics marked with * are statistically significant ( p<0.05 ).
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: PRISMA-style flow diagram for benchmark selection.
Figure 6: Average values of five influence-related metrics on benchmark and non-benchmark papers.
Figure 7: Distribution of the scientific fields that the LLM safety papers influence.
Figure 8: Average values of eight metrics related to the code repository quality on benchmark and non-benchmark papers. We have flipped the axes as needed, so now all points/lines on the upper side indicate better quality or performance.
Figure 11: A typical example of the general pattern we identify. After the paper is publicly available, its GitHub star count increases rapidly.
Figure 12: Spearman correlation ρ matrix of the adoption metrics (those with p≥0.05 are omitted).
Figure 13: Spearman correlation matrices between the adoption metrics and the potential quantitative factors. The unadjusted p-values on the left can be interpreted exploratively.
Figure 14: Spearman correlation matrices between the adoption metrics and the code repository quality metrics. The unadjusted p-values on the left can be interpreted exploratively.
Figure 15: Spearman correlation matrices between the code repository quality metrics and the potential quantitative factors. The unadjusted p-values on the left can be interpreted exploratively.
Figure 16: Box plots of citation density by group. The red dashed lines represent the means.
Figure 17: Box plots of citation density and the status of extra modifications and runnable code. The red dashed lines represent the mean values.
Figure 18: Box plots of citation density and various potential qualitative factors. The red dashed lines in the box plots represent the mean values. The further to the right the x-axis tick is, the higher the mean value of the corresponding box.
Figure 19: Box plots of Pylint score and various potential qualitative factors. The red dashed lines in the box plots represent the mean values. The further to the right the x-axis tick is, the higher the mean value of the corresponding box.
School of Information Studies, McGill University, Canada · Department of Computer Science, Kean University, USA · College of Technological Innovation, Zayed University, UAE