Benchmark Auditing

Benchmark auditing studies whether test items, labels, scoring rules, and evaluation setups support the claims made from model scores. Auditors use interventions and matched controls to detect shortcuts, contamination, and information leakage; they also check labels with human review or item-response models and test software verifiers through replay and mutation. Automated LLM and coding-agent tools can help inspect large benchmarks, but their findings still need validation against independent evidence. A central challenge is that audits themselves can be underpowered or incomplete, while scores and rankings may change with prompts, splits, references, and runtime conditions.