Overview

Benchmark validity studies whether evaluation tasks measure the intended capability and whether their scores justify claims about real-world performance. Researchers audit task coverage, labels, answer keys, contamination, scoring rules, and agent trajectories, using expert review, controlled interventions, item-level analysis, and psychometric methods such as item response theory. They also test robustness to prompts, data splits, evaluation settings, and benchmark composition, and ask whether rankings predict performance on new tasks or in deployment. A central tension is that standardized benchmarks enable comparison but can reward shortcuts or miss real-world needs, so no single score can establish broad capability without evidence tied to a specific claim.

Latest papers

CardsList

185 papers10 papers in the last four weeks, down 63% on the four weeks before. 0.1% of all new papers.

  1. Encoding, Decodability, and Use: A Five-Layer Framework for Benchmark Measurement Audits with Evidence from UAV Intent Recognition

    Nov 17, 2026Zhirong Su, Tian ZhangBenchmark AuditingBenchmark Validity

  2. Where Does Hindsight Enter Clinical Agent Benchmarks? An Audit of Simulators and Case Construction

    Nov 17, 2026Meilu Zhu, Xiaohan Li, Axiu MaoLLM Agent EvaluationBenchmark Validity

  3. Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks

    Nov 5, 2026E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman +1Coding Agent BenchmarksBenchmark Validity

  4. Gold vs.~Translation: Are We Measuring Language, Knowledge, or Locality in Translated Benchmarks?

    Oct 28, 2026Klaudia-Doris Thellmann, Bernhard Stadler, Dan Saattrup Nielsen +2Benchmark ValidityMultilingual Language Model Evaluation

  5. EvoBench: An Evolving, Diagnostic Benchmark Framework for Environmental Modeling

    Oct 26, 2026Yiming Sun, Yuan Zhang, Qi Cheng +4Benchmark DesignBenchmark Validity

  6. Your Benchmark Is an Empirical Measure Over Difficulty

    Oct 21, 2026Yifan Sun, Naicheng Yu, Jiawen Gong +2Benchmark DesignBenchmark Auditing

  7. Evidence Sufficiency and Model Comparisons in MexRegQA: An Item-Level Audit of Mexican-Spanish Regulatory QA

    Oct 19, 2026Fred Hu, Yeqi Wang, Xing ZhouBenchmark AuditingLegal QA

  8. Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks

    Oct 16, 2026Yuanzhe Liu, Bastian Lipka, Avraham Shinnar +3Benchmark DesignSoftware Engineering Benchmarks

  9. SWEPatchTest: Evaluating Software Tests Beyond Reproduction

    Oct 13, 2026Hua Lu, Pengcheng Wang, Han Li +4Software Engineering BenchmarksAutomated Test Generation

  10. How to Build a Benchmark That Actually Measures Taste

    Oct 10, 2026Xiaofan Wu, Xi Zeng, Miaoxia Chen +8Benchmark DesignBenchmark Validity

  11. Provenance-grouped protocol for evaluating machine learning in perovskite photovoltaics

    Oct 8, 2026Qiwei Han, Chi Zhouperovskite solar cellsBenchmark Validity

  12. On the estimation and validity of AI time horizons---a statistical look at the METR plot

    Oct 8, 2026Drew T. Nguyen, William FithianBenchmark Validity

  13. Clinician use of language models diverges from how the models are evaluated

    Oct 8, 2026Krithik Vishwanath, Haitong Lin, Anton Alyakin +19Clinical Language Model EvaluationMedical LLMs

  14. On Hyperparameter Tuning on the Test Set

    Oct 5, 2026Matteo Fregonara, Tom Viering, Jan van GemertModel SelectionBenchmark Validity

  15. What Does a Harness Repair? Four Preregistered Studies Around a Longevity-Tuned Language Model

    Oct 4, 2026Bowen Xu, Boyu ChenLLM EvaluationBenchmark Validity

  16. TARE: Weigh a Never-Poisoned Twin Before Reading Backdoor-Defense Costs

    Oct 4, 2026Ruizhi Xu, Wei Xu, Sibo ZhuBackdoor DefenseBenchmark Validity

  17. Frozen Scenes, Shifting Winners: Configuration Fragility in Text-to-3D Evaluation

    Sep 30, 2026Anson Y. Lam, Shuqing Li, Michael R. LyuVLM EvaluationBenchmark Validity

  18. Your Benchmark Is Not Saturated: Reviving Multiple-Choice Evaluation with Answer Pooling

    Sep 28, 2026Mohamed Eltahir, Abobaker Ahmed, Nawaf Barebood +3Multiple-Choice Question AnsweringBenchmark Validity