Overview

Benchmark validity studies whether evaluation tasks measure the intended capability and whether their scores justify claims about real-world performance. Researchers audit task coverage, labels, answer keys, contamination, scoring rules, and agent trajectories, using expert review, controlled interventions, item-level analysis, and psychometric methods such as item response theory. They also test robustness to prompts, data splits, evaluation settings, and benchmark composition, and ask whether rankings predict performance on new tasks or in deployment. A central tension is that standardized benchmarks enable comparison but can reward shortcuts or miss real-world needs, so no single score can establish broad capability without evidence tied to a specific claim.

Latest papers

CardsList

185 papers10 papers in the last four weeks, down 63% on the four weeks before. 0.1% of all new papers.

  1. fev-bench: A Realistic Benchmark for Time Series Forecasting

    Sep 30, 2025Oleksandr Shchur, Abdul Fatir Ansari, Caner Turkmen +5Benchmark DesignTime Series Forecasting

  2. Who Benchmarks the Benchmarks? Towards Comprehensive Evaluation of Commonsense Reasoning Benchmarks

    Apr 10, 2025Pavel Chizhov, Anton Changalidis, Vishnu Prasad Vijaya Kumar +4Commonsense ReasoningBenchmark Auditing

  3. Is Your Benchmark Still Useful? Dynamic Benchmarking for Code Language Models

    Mar 9, 2025Batu Guan, Xiao Wu, Yuanyuan Yuan +1Benchmark DesignData Contamination in Language Models

  4. GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methods

    Feb 24, 2025Ruixuan Huang, Xunguang Wang, Zongjie Li +2Language Model Safety EvaluationLLM Jailbreak Attacks