Overview

Benchmark auditing studies whether test items, labels, scoring rules, and evaluation setups support the claims made from model scores. Auditors use interventions and matched controls to detect shortcuts, contamination, and information leakage; they also check labels with human review or item-response models and test software verifiers through replay and mutation. Automated LLM and coding-agent tools can help inspect large benchmarks, but their findings still need validation against independent evidence. A central challenge is that audits themselves can be underpowered or incomplete, while scores and rankings may change with prompts, splits, references, and runtime conditions.

Latest papers

CardsList

109 papers16 papers in the last four weeks, down 16% on the four weeks before. 0.1% of all new papers.

  1. Encoding, Decodability, and Use: A Five-Layer Framework for Benchmark Measurement Audits with Evidence from UAV Intent Recognition

    Nov 17, 2026Zhirong Su, Tian ZhangBenchmark AuditingBenchmark Validity

  2. When Does an AUROC Comparison Survive a Change in Population Mix?

    Nov 15, 2026Mahtab Movahhedrad, Hamidreza Hasani Balyani, Seyed Pouyan Mousavi DavoudiBenchmark AuditingUncertainty Quantification

  3. BO-Arena: An Evolving Benchmark for High-Dimensional Bayesian Optimisation

    Nov 6, 2026Colin Doumont, Thomas Christie, Henry Moss +1Bayesian OptimizationBenchmark Design

  4. Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks

    Nov 5, 2026E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman +1Coding Agent BenchmarksBenchmark Validity

  5. AStockArena: Beyond Returns, Can We Entrust Real Capital to the Decisions of Financial Agents?

    Oct 31, 2026Zeyu Liu, Sing Kwong Lai, Xuanzhi Feng +4Financial Language Model EvaluationAI Agent Auditing

  6. Right Click, Wrong Box: Grading GUI Grounding by What a Click Does

    Oct 27, 2026Xiaokang Yuan, Binhui Yao, Luwei Yang +2GUI GroundingAutomated Evaluation

  7. Your Benchmark Is an Empirical Measure Over Difficulty

    Oct 21, 2026Yifan Sun, Naicheng Yu, Jiawen Gong +2Benchmark DesignBenchmark Auditing

  8. ExeBench-FHD: Addressing Fidelity Issues in Decompiler Evaluation

    Oct 20, 2026Akarsh Ashok, Dinil Mon Divakaran, Vivek BalachandranBenchmark DesignSoftware Engineering Benchmarks

  9. Evidence Sufficiency and Model Comparisons in MexRegQA: An Item-Level Audit of Mexican-Spanish Regulatory QA

    Oct 19, 2026Fred Hu, Yeqi Wang, Xing ZhouBenchmark AuditingLegal QA

  10. When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark

    Oct 12, 2026Junjie ChenBenchmark AuditingLLM Agent Evaluation

  11. How to Build a Benchmark That Actually Measures Taste

    Oct 10, 2026Xiaofan Wu, Xi Zeng, Miaoxia Chen +8Benchmark DesignBenchmark Validity

  12. Certified Corruption Budgets: Anytime-Valid Leaderboard Claims under Adaptive Rigging

    Oct 6, 2026Hamed Khosravi, Xiaoming HuoCorruption RobustnessAnytime-Valid Inference

  13. Measurement-First Auditing of Agentic Leaderboards: Contamination Susceptibility, Matched-Control Re-evaluation, and Scorer Validation

    Oct 5, 2026Dishu Yang, Qi Su, Hongbo Qin +1Benchmark AuditingAI Agent Auditing

  14. Protocol-Sensitive Evaluation of Log Anomaly Detection: Component Costs and Target-Access Sensitivity on HDFS and BGL

    Oct 5, 2026Hang Xiao, Janet Sung, Zhaoyi Li +2Benchmark AuditingAnomaly Detection

  15. SALUS: Automated Auditing of NL-to-SQL Benchmarks through Weak Supervision of Multi-Agent Output

    Oct 4, 2026Shiyuan Zhou, Ashwin Gerard Colaco, Sainyam Galhotra +1Benchmark AuditingText-to-SQL

  16. Cheap to Draw, Expensive to Trust: Certifying Test-Time Scaling Curves

    Sep 30, 2026Sohail, Sarkar, Shakuntala BaichooBenchmark AuditingTest-Time Scaling

  17. Do Agent Benchmarks Do What They Say? An Executable-Contract Audit of Tool-Using Agent Environments

    Sep 29, 2026Rohith Reddy Bellibatlu, Zichong Wang, Wenbin ZhangTool-Use EvaluationAutomated Software Testing