Between Rigor and Reality: How AI Safety Benchmark Developers Understand Benchmark Use and Maintenance
Organizations: Carnegie Mellon University Pittsburgh, PA
Abstract
Safety benchmarks are cited as the primary tool to understand the risks posed by generative AI (GenAI) systems, yet growing evidence suggests they often fail to meet the needs of real-world safety evaluation. We present findings from five interviews with AI safety benchmark developers in academia and industry. We find a disconnect between perceptions of the intended usefulness of safety benchmarks and their practical usefulness in safety evaluations of deployed systems. Our participants offered perspectives on two challenges contributing to this disconnect: (1) the difficulty of achieving inter-rater reliability on safety constructs, and (2) a lack of clarity regarding how to address persistent threats to external validity. Based on our analysis, we argue that safety benchmarks must not only be grounded in deployment contexts, but actively integrated with them. To facilitate this, we outline pathways for more transparent communication between academic and industry AI safety stakeholders.
Figures & tables
| ID | Professional Context | Focus | YoE |
|---|---|---|---|
| P1 | Industry | Evaluation methodology | 10+ |
| P2 | Academia | Mental health | 1-5 |
| P3 | Industry* | Bias, Toxicity | 1-5 |
| P4 | Industry | Human well-being | 1-5 |
| P5 | Industry* | Pluralism | 5-10 |
AI Safety Evaluation
AI Safety Evaluation studies how to test whether AI systems avoid harmful behavior and remain reliable across users, tasks, and deployment settings.
See all 160 papers