Overview

Benchmark validity studies whether evaluation tasks measure the intended capability and whether their scores justify claims about real-world performance. Researchers audit task coverage, labels, answer keys, contamination, scoring rules, and agent trajectories, using expert review, controlled interventions, item-level analysis, and psychometric methods such as item response theory. They also test robustness to prompts, data splits, evaluation settings, and benchmark composition, and ask whether rankings predict performance on new tasks or in deployment. A central tension is that standardized benchmarks enable comparison but can reward shortcuts or miss real-world needs, so no single score can establish broad capability without evidence tied to a specific claim.

Latest papers

185 papers10 papers in the last four weeks, down 63% on the four weeks before. 0.1% of all new papers.

Sep 30, 2025

fev-bench: A Realistic Benchmark for Time Series Forecasting

Benchmark quality is critical for meaningful evaluation and sustained progress in time series forecasting, particularly with the rise of pretrained models. Existing benchmarks often have limited domain coverage or overlook real-world settings such as tasks with covariates. Their aggregation procedures frequently lack statistical rigor, making it unclear whether observed performance differences reflect true improvements or random variation. Many benchmarks lack consistent evaluation infrastructure or are too rigid for integration into existing pipelines. To address these gaps, we propose fev-bench, a benchmark of 100 forecasting tasks across seven domains, including 46 with covariates. Supporting the benchmark, we introduce fev, a lightweight Python library for forecasting evaluation emphasizing reproducibility and integration with existing workflows. Using fev, fev-bench employs principled aggregation with bootstrapped confidence intervals to report performance along two dimensions: win rates and skill scores. We report results on fev-bench for pretrained, statistical, and baseline models and identify promising future research directions.
Sep 27, 2025

WirelessMathBench-XL: An Auditable Benchmark for Wireless Mathematical Reasoning

Benchmarks built from arXiv papers can overlap the arXiv text used in LLM pretraining, yet technical-domain releases rarely ship the per-item evidence needed to check this. We introduce a reverse-probe n-gram audit that indexes the benchmark side and streams the corpus past it, so memory scales with the benchmark rather than the corpus and flags on the distributed problems can be inspected and recomputed. We instantiate it in WirelessMathBench-XL, a 4,027-problem benchmark for wireless mathematical reasoning built from 836 arXiv papers across 20 subfields, with multiple-choice, progressive fill-in, and full-equation-completion formats. Streaming 12.9 B RedPajama-arXiv 13-grams yields a strict zero-hit view S0\mathcal{S}_0 covering 95.7% of the problems, and shows that the Dolma and ProofPile-2 arXiv slices repackage the same RedPajama dump. Filtering to S0\mathcal{S}_0 moves every evaluated model by less than 1 pp. With 30/800 flagged test items, this comparison bounds detected-overlap inflation at 0.31-0.51 pp for frontier models rather than testing for contamination, and it does not cover paraphrase, target-answer, alternate-version, post-training, or closed-corpus exposure. Frontier models reach 86.5-91.3%, and GRPO on the released training split lifts a 7B model 25.1 pp above a size-matched general-math control. We release source-paper identifiers, per-problem audit and threshold metadata, license-aware data files, audit and bootstrap scripts, Croissant metadata, and a Datasheet.
Apr 10, 2025

Who Benchmarks the Benchmarks? Towards Comprehensive Evaluation of Commonsense Reasoning Benchmarks

Commonsense reasoning is a key language model capability, as it is purportedly a prerequisite for many basic tasks, unlike specific factual knowledge. It is often measured with multiple-choice questions (MCQ) benchmarks, e.g. HellaSwag and PIQA. Some of these benchmarks, however, are outdated and contain numerous validity issues. We illustrate some typical validity issues with a case study on HellaSwag, one of the most popular and problematic benchmarks for commonsense reasoning. The issues we find range from basic ungrammaticality and numerous typos to misleading prompts or equally correct options. We show that if we remove question prompts or replace them with "Lorem ipsum dolor...", about 68% of model predictions do not change. We argue that this occurs due to inner flaws in the benchmark, not mere contamination that might be present in some models. Since benchmark scores are an essential part of model selection in both research and commercial applications, these issues can have severe consequences. Based on our findings, we propose BenCheck, a package for benchmark validity analysis that encapsulates the main checks performed in our case study and can be used to audit commonsense reasoning benchmarks. We apply these checks to PIQA, Global PIQA, and Winogrande.
Mar 9, 2025

Is Your Benchmark Still Useful? Dynamic Benchmarking for Code Language Models

In this paper, we tackle a critical challenge in model evaluation: how to keep code benchmarks useful when models might have already seen them during training. We introduce a novel solution, dynamic benchmarking framework, to address this challenge. Given a code understanding or reasoning benchmark, our framework dynamically transforms each input, i.e., programs, with various semantic-preserving mutations to build a syntactically new while semantically identical benchmark. We evaluated 10 popular language models on our dynamic benchmarks. Our evaluation reveals several interesting or surprising findings: (1) all models perform significantly worse than before, (2) the ranking between some models shifts dramatically, and (3) dynamic benchmarks can resist against the data contamination problem.
Feb 24, 2025

GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methods

Despite the growing interest in jailbreaks as an effective red-teaming tool for building safe and responsible large language models (LLMs), flawed evaluation system designs have led to significant discrepancies in their effectiveness assessments. With a systematic measurement study based on 37 jailbreak studies since 2022, we find that existing evaluation systems lack case-specific criteria, resulting in misleading conclusions about their effectiveness and safety implications. In this paper, we introduce GuidedBench, a novel benchmark comprising a curated harmful question dataset and GuidedEval, an evaluation system integrated with detailed case-by-case evaluation guidelines. Experiments demonstrate that GuidedBench offers more accurate evaluations of jailbreak performance, enabling meaningful comparisons across methods. GuidedEval reduces inter-evaluator variance by at least 76.03%, ensuring reliable and reproducible evaluations. We reveal why existing jailbreak benchmarks fail to evaluate accurately and suggest better evaluation practices.