Benchmark Validity

Ask about this topic

Overview

Benchmark validity studies whether evaluation tasks measure the intended capability and whether their scores justify claims about real-world performance. Researchers audit task coverage, labels, answer keys, contamination, scoring rules, and agent trajectories, using expert review, controlled interventions, item-level analysis, and psychometric methods such as item response theory. They also test robustness to prompts, data splits, evaluation settings, and benchmark composition, and ask whether rankings predict performance on new tasks or in deployment. A central tension is that standardized benchmarks enable comparison but can reward shortcuts or miss real-world needs, so no single score can establish broad capability without evidence tied to a specific claim.

Latest papers

CardsList

175 papers20 papers in the last four weeks, down 9% on the four weeks before. 0.1% of all new papers.

  1. Identity or Evidence? Auditing Evidence Dependence in Agentic Life-Science Benchmarks

    Oct 27, 2026Danai Brilli, Maria GeorgilaScientific Agent BenchmarksEvidence-Grounded Reasoning

    Zo Labs

  2. EvoBench: An Evolving, Diagnostic Benchmark Framework for Environmental Modeling

    Oct 26, 2026Yiming Sun, Yuan Zhang, Qi Cheng +4Benchmark DesignBenchmark Validity

    Rutgers University, New Brunswick · Apple Inc. · University of Alabama +1

  3. Your Benchmark Is an Empirical Measure Over Difficulty

    Oct 21, 2026Yifan Sun, Naicheng Yu, Jiawen Gong +2Benchmark DesignBenchmark Auditing

    University of Illinois Urbana-Champaign · University of California San Diego · New York University

  4. Evidence Sufficiency and Model Comparisons in MexRegQA: An Item-Level Audit of Mexican-Spanish Regulatory QA

    Oct 19, 2026Fred Hu, Yeqi Wang, Xing ZhouBenchmark AuditingLegal QA

    Huawei Technologies de México, Mexico · Northeastern University, USA

  5. Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks

    Oct 16, 2026Yuanzhe Liu, Bastian Lipka, Avraham Shinnar +3Benchmark DesignSoftware Engineering Benchmarks

    Rensselaer Polytechnic Institute · IBM

  6. OpenTWBench (Open Taiwan Benchmark): Taiwan-Domain Knowledge Evaluation for Large Language Models, and How the Measurement Apparatus Shapes the Score

    Oct 14, 2026Liang-Hsun Huang, Min-Yi Chen, Chao-Chun Chuang +3LLM EvaluationBenchmark Validity

    Twinkle AI · APMIC · Open Taiwan Benchmark +4

  7. Whose Data, Whose Benchmark? A User-Centric Framework for Auditing Evaluation Resources

    Oct 13, 2026Mir Tafseer Nayeem, Davood RafieiBenchmark AuditingBenchmark Design

    University of Alberta

  8. SWEPatchTest: Evaluating Software Tests Beyond Reproduction

    Oct 13, 2026Hua Lu, Pengcheng Wang, Han Li +4Software Engineering BenchmarksAutomated Test Generation

    Baidu Inc. · Nanjing University · The Chinese University of Hong Kong

  9. How to Build a Benchmark That Actually Measures Taste

    Oct 10, 2026Xiaofan Wu, Xi Zeng, Miaoxia Chen +8Benchmark DesignBenchmark Validity

    Chance AI

  10. Provenance-grouped protocol for evaluating machine learning in perovskite photovoltaics

    Oct 8, 2026Qiwei Han, Chi ZhouBenchmark ValidityData Leakage

    Department of Chemistry, Duke University, Durham, NC 27708, USA · Department of Computer Science, Georgia Institute of Technology, Atlanta, GA 30332, USA

  11. On the estimation and validity of AI time horizons---a statistical look at the METR plot

    Oct 8, 2026Drew T. Nguyen, William FithianBenchmark Validity

    Department of Statistics UC Berkeley

  12. Clinician use of language models diverges from how the models are evaluated

    Oct 8, 2026Krithik Vishwanath, Haitong Lin, Anton Alyakin +19Clinical Language Model EvaluationMedical LLMs

    Department of Neurosurgery, NYU Langone Health, New York, NY, USA · Department of Technology Management, NYU Tandon School of Engineering, New York University, New York, NY, USA · Washington University School of Medicine, St. Louis, MO, USA +18

  13. On Hyperparameter Tuning on the Test Set

    Oct 5, 2026Matteo Fregonara, Tom Viering, Jan van GemertModel SelectionBenchmark Validity

    Delft University of Technology

  14. What Does a Harness Repair? A Preregistered Study of Visibility, Baseline Adequacy and Evaluation Defects

    Oct 4, 2026Bowen Xu, Boyu ChenLLM EvaluationBenchmark Validity

    Institute of Health Informatics, University College London, London, UK

  15. TARE: Weigh a Never-Poisoned Twin Before Reading Backdoor-Defense Costs

    Oct 4, 2026Ruizhi Xu, Wei Xu, Sibo ZhuBackdoor DefenseBenchmark Validity

    School of Mathematics University of Edinburgh · Beijing Jinhe Technology Co., Ltd. · Inno Asset Management

  16. Same Output, Different Gold: Measuring How Reference Choice Moves a Multilingual Benchmark Score

    Oct 2, 2026Parth Kulshreshtha, Shivali Dalmia, Abhishek MukherjiBenchmark ValidityBenchmark Auditing

    Centific

  17. SimplexUQ: An Evaluation Framework and Benchmark for Conformal Uncertainty on Simplex-Valued Predictions

    Sep 30, 2026Liang You, Hengyu Shi, Dongwen OuGroup-Conditional Conformal PredictionConformal Prediction

    University of Pittsburgh · Duke University

  18. Frozen Scenes, Shifting Winners: Configuration Fragility in Text-to-3D Evaluation

    Sep 30, 2026Anson Y. Lam, Shuqing Li, Michael R. LyuVLM EvaluationBenchmark Validity

    Department of Computer Science and Engineering The Chinese University of Hong Kong Hong Kong, China

  19. Your Benchmark Is Not Saturated: Reviving Multiple-Choice Evaluation with Answer Pooling

    Sep 28, 2026Mohamed Eltahir, Abobaker Ahmed, Nawaf Barebood +3Multiple-Choice Question AnsweringBenchmark Validity

    King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · Department of Computer Science, Edge Hill University, Ormskirk, England

  20. The Last Mile Is the File: OfficeEditBench for Preservation-Aware Office Editing

    Sep 28, 2026Zhiwen Wu, Chengxu WuBenchmark ValiditySpreadsheet Automation

    Peking University

  21. Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes

    Sep 28, 2026Weijun Luo, Kelvin Luu, Xinyi Liu +6Reward HackingBenchmark Auditing

    Scale AI

  22. Frozen Judges, Moving Agents: Version-Dependent LLM-Judge Error and the Limits of Judge-Assisted Agent Evaluation

    Sep 28, 2026Jiapeng LiLLM-as-a-JudgeAI Agent Auditing

  23. Are Benchmarks Reliable? Toward Structural Diagnosis via Item-Level Capability Boundaries

    Sep 27, 2026Haiquan Hu, Yuzhu Liang, Weicheng Tang +4LLM EvaluationBenchmark Validity

    Beijing Normal University

  24. Are You Sure You're Sure? Two Confounds in a Sycophancy Benchmark

    Sep 26, 2026Atharv Gupta, Akshat Jindal, Lavanya Nigam +1LLM SycophancyLanguage Model Generation Evaluation

    Indian Institute of Technology Roorkee

  25. Rank Confidence Sequences:Anytime-valid Leaderboards

    Sep 26, 2026Hamed Khosravi, Xiaoming HuoAnytime-Valid InferenceBenchmark Validity

    H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology, Atlanta, GA 30332, USA

  26. How Reusable Are Benchmarks with Richer Feedback?

    Sep 26, 2026Youssef Allouah, John DuchiBenchmark ValidityLLM Evaluation

    Department of Computer Science Stanford University · Departments of Statistics and Electrical Engineering Stanford University

  27. Recipe-Matching, Not Equivalence

    Sep 25, 2026Ali Habibullah, Mohammad Alshiekh, Yazan Alshoibi +2Benchmark ValidityIR Evaluation

    KAUST Academy Computer, Electrical & Mathematical Sciences & Engineering (CEMSE) King Abdullah University of Science and Technology (KAUST) Thuwal, Saudi Arabia · KAUST Academy & CEMSE, KAUST Lady Margaret Hall, University of Oxford

  28. When 10,000 Windows Are Not 10,000 Tests: Auditing Statistical Confidence in Sliding-Window Time-Series Classification

    Sep 25, 2026Xinze Shi, Litian Zhang, Binrui ShiTime Series ClassificationBenchmark Validity

    Beijing University of Posts and Telecommunications Beijing, China

  29. TopU-LBVS: A Realistic Multi Target Benchmark for Ligand Based Virtual Screening

    Sep 24, 2026Surbhi Kumar, Yuhe Zhou, Varun Shiralkar +2Benchmark DesignVirtual Screening

    UT Dallas, Mathematical Sciences Richardson, TX, USA

  30. Certified Against Which Oracle? Execution Labels Set the Reported Risk of Conformal Abstention for Text-to-SQL

    Sep 22, 2026Jiamiao Liu, Dewen Qiao, Yu Zhang +1Probability CalibrationBenchmark Auditing

    Department of Information, Xinqiao Hospital Army Medical University (Third Military Medical University) Chongqing 400037, China