BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity

Harshita Diddee; Gregory Yauney; Swabha Swayamdipta; Daphne Ippolito

cs.CLFeb 25, 2026

BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity

Authors: Harshita Diddee, Gregory Yauney, Swabha Swayamdipta, Daphne Ippolito

Organizations: Language Technologies Institute, Carnegie Mellon University, Pittsburgh (PA), United States · Thomas Lord Department of Computer Science, University of Southern California, Los Angeles (CA), United States

Abstract

Do language model benchmarks actually measure what practitioners intend them to ? High-level metadata is too coarse to convey the granular reality of benchmarks: a "poetry" benchmark may never test for haikus, while "instruction-following" benchmarks will often test for an arbitrary mix of skills. This opacity makes verifying alignment with practitioner goals a laborious process, risking an illusion of competence even when models fail on untested facets of user interests. We introduce BenchBrowser, a retriever that surfaces evaluation items relevant to natural language use cases over 20 benchmark suites. Validated by a human study confirming high retrieval precision, BenchBrowser generates evidence to help practitioners diagnose low content validity (narrow coverage of a capability's facets) and low convergent validity (lack of stable rankings when measuring the same capability). BenchBrowser, thus, helps quantify a critical gap between practitioner intent and what benchmarks actually test.

Figures & tables

Figure 1: BenchBrowser retrieves use-case-relevant items from 20+ benchmarks to help diagnose validity failures. For a “Writing Programming Functions” use-case: Python-skewed coverage (left; content validity gap) and unstable model ranks across subsets (right; low convergent validity) make conclusions unreliable. $m_{1}$ to $m_{k}$ are mid-sized decoder models. See Appendix A for details.

Figure 2: BenchBrowser pipeline: a practitioner submits a use-case of interest to BenchBrowser ; the use-case is a rewritten into anchors for more diverse retrieval, which are then b embedded and used to retrieve c relevant examples from a suite of 20+ benchmarks. The retrieved evidence can be used by the practitioner in multiple ways (images show snapshots of BenchBrowser ’s website UI): $d_{1}$ Comparing the representation of the use-case along different facets. For example, a user interested in assessing “reasoning skills” can observe the different ways (or lack thereof) in which reasoning competence is evaluated (linguistics fundamentals versus math problems) and $d_{2}$ Compare the performance and stability of models evaluated on existing benchmarks (e.g., ARC AI2 Reasoning) for the task as well as subsets of the retrieved examples.

Original:

understanding geopolitical tensions in Southeast Asia

Rephrasing:

South China Sea territorial disputes and maritime tensions • ASEAN diplomacy, regional security mechanisms, and conflict prevention • US–China strategic competition

Example Synthesis:

Discuss the strategic factors driving contemporary geopolitical tensions in Southeast Asia • MCQ: Which factor most significantly contributes to territorial disputes in the South China Sea? […] • Fill in the blank: ASEAN’s principle of ___________ often limits collective action […]

Shorthand:

Table 1: We compare four methods for producing retrieval anchors. Here we show how each handles an example practitioner use-case: “ understanding geopolitical tensions in Southeast Asia ”.

Name	Origin	$\bm{N}$	Examples
Automatically synthesized
Topics	BBH, MMLU	58	Marketing, Law, HS Maths
Skills	Clio	50	creative writing, navigation
Applications	Clio	50	Tips for Winter Gardening
Manually curated
Known Valid.	LM Eval	20	Riddles, Calorie Counting

Table 2: Use-case query sets for evaluating the retrieval quality of BenchBrowser . Automatically synthesized sets derive from benchmarks; manually curated sets are hand-crafted for validation.

Figure 3: Variation in the percentage of Relevant examples across different facets of the same broad use-case. For each skill family, we show the facet for which the highest and lowest number of examples are retrieved. Most capabilities show sharp disparities: certain facets are heavily over-represented in our database of 70k benchmark examples, while others are nearly absent from benchmarks.

Use-case	Mean H	Mean LLM	${\rho}$	${\kappa}$
Topics	$0.96$	$0.92$ $\ddagger$	$0.15$ $\dagger$	$0.10$
Skills	$0.80$	$0.91$ $\ddagger$	$0.17$ $\dagger$	$0.11$
Applications	$0.74$	$0.75$	$0.50$ $\ddagger$	$0.29$
Known Valid.	$0.58$	$0.88$ $\ddagger$	$0.34$ $\ddagger$	$0.29$
Novel	$0.70$	$0.67$	$0.27$ $\ddagger$	$0.13$

Table 4: Mean relevance of retrieved examples by human annotators vs. the LLM judge across use-case query sets. Superscripts indicate significance:

\dagger

p<0.05

;

\ddagger

p<0.0001

. Markers on Mean LLM are significance markers for paired

t

-test for the means.

\rho

is Spearman’s rho and

\kappa

is Fleiss’s kappa.

Code generation		Creative writing
Programming language		Literary genre
“Write a function in $\ldots$ ”		“Write a $\ldots$ story”
Facet	$\text{Prec}_{\text{ sys}}@k$	Facet	$\text{Prec}_{\text{ sys}}@k$
Python	0.867	fantasy	0.759
C++	0.741	science fiction	0.679
Rust	0.478	mystery	0.654

Table 5: Different facets of the same skill family can have very different

\text{Prec}_{\text{ sys}}@k

when using BenchBrowser with Example Synthesis anchors and the LLM judge (relevant examples only). Most skill families show substantial spread: some variants (green) have many Relevant examples, while others (red) are barely represented. For complete results for all skill families, see Appendix J .

Figure 4: Kendall’s

\tau

correlations of model rankings on gold test sets vs. retrieved sets for known validation set use-cases with mean human relevance

\geq 0.5

. Wider bars correspond to greater rank divergence

\Delta=\tau_{\text{gold}}-\tau_{\text{ret}}

n

: number of retrieved examples.

Root Cause 1: Rubric Shift

Example Query: “Hindu Religion”

HellaSwag: Philosophy and Religion: How to learn about ancient hinduism …

BIG-bench Hard (Hindu Knowledge): Which of the following gods are associated with a flying elephant

Root Cause 2: Divergent Operationalization

Example Query: “Solving Puzzles”

BIG-bench Hard (Colored Objects): On the desk, you see items in a row: a gold textbook, a purple puzzle, a teal necklace, and a silver pencil. How many non-gold items do you see to the right of the pencil?

Table 6: Root causes of ranking inversions in BenchBrowser ; gold highlight indicates example is from the gold set, and blue indicates it is from the retrieved set.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Test Case	Shorthand
mathqa	if $x$ is a number such that $x^{2}+2x-24=0$ and $x^{2}-5x+4=0$ , then $x=?$ Options: a) 4, b) -4, c) -3, d) -6, e) 1	equation_solving & quadratic_equations & solution_id
triviaqa	Who invaded Europe from Mongolia and Turkey over 300 years, beginning in the 13th century?	factual_recall & historical_figures & mongols
alpacaeval	Provide a formula for computing the nth term in the given sequence: 5, 14, 23, 32, 41, …	equation_solving & sequence_formula & nth_term
arc_c	Scientists often change conclusions when new information becomes available. Which statement was changed? Options: Moon is in outer space / Plants make own food / Sun revolves around Earth	reading_comp & analysis & scientific_conclusions

Appendix

Table 10

use-case	Relevant example
Verify scientific claims	Claim: AdaBERT achieves inferior performance while significantly worsening the effiency by 12.7x. Evidence: The dataset contains 3772 word pairs. The accuracy of ConVecs 70.02% is not significantly different from the accuracy of SimDiffs (72.4%). Label: Negative/Refuted.
Blogging	Write a blog announcing the opening of a new mall in the town of Serenity.
Unit Testing Code	Question: An anagram is the result of rearranging the letters of a word to produce a new word. Note: anagrams are case insensitive Complete the function to return true if the two arguments given are anagrams of each other; return false otherwise. Examples: foefet is an anagram of toffee. ut_id: 0 code - import unittest class TestAreAnagramsFunction(unittest.TestCase): This class contains unit tests for the are_anagrams function. def test_anagram(self): Test that two anagrams return True. self.assertTrue(are_anagrams(”foefet”, ”toffee”)) def test_not_anagram(self): Test that two non-anagrams return False.
Star Gazing	The Big Dipper is a ______ constellation.
Resume for Machine Learning Engineer	I am a software engineer with 5 years of experience in the industry. Can you help me write a resume?

Appendix

Table 11

Retriever Type	$\text{Prec}_{\text{ sys}}@k$	$\text{Recall}@k$	NDCG@10 / MRR	Processing Time (s)
bge-base-V1.5	0.208	0.004	0.291	0.048
bge-large-V1.5	0.332	0.004	0.353	0.044
llm-embedder	0.488	0.012	0.537	0.029
bge-en-icl	0.694	0.218	0.684	1.040
bge-reranker	0.664	0.176	0.632	14.030

Appendix

Table 9: Retriever Performance Comparison: bge-en-icl achieves the highest

\text{Prec}_{\text{ sys}}@k

\text{Recall}@k

and

\text{NDCG}@k

with reasonable latency.

Benchmark	Citation	Benchmark	Citation
IFEval	Zhou et al. (2023)	logiqa	Liu et al. (2021)
MMLU	Hendrycks et al. (2021)	mathqa	Amini et al. (2019)
TruthfulQA	Lin et al. (2022)	openbookqa	Mihaylov et al. (2018)
HellaSwag	Zellers et al. (2019)	qasper	Dasigi et al. (2021)
ARC	Chollet (2019)	triviaqa	Joshi et al. (2017)
Winogrande	Sakaguchi et al. (2020)	webqs	Berant et al. (2013)

Appendix

Table 10: Benchmarks Used for Example Retrieval

Usecase	Gold Test Set
Learn how to measure my carbohydrate intake and make my meals more balanced	nutribench ( Hua et al., 2024 )
Test my knowledge of different sports and their respective players	bb_sports
Help me learn more about the Hindu religion	bb_hindu
Test me on different topics in science	sciq ( Welbl et al., 2017 )
Help me answer questions for my biomedical research	pubmed ( Jin et al., 2019 )
Help me mitigate different biological, cyber, and chemical attacks	wmdp ( Li et al., 2024 )

Appendix

Table 11: Prompts with Corresponding Gold Datasets used for the Known Validation Set. All datasets prefixed with “bb” are included in the Big Bench Dataset ( Srivastava et al., 2023 ) .

Figure 5: We observe that performance on the retrieved testset and the gold test set (BBH and MMLU for the topics set set) show reasonable to high correlation.

Figure 6:

\text{intersection}@k

in approximated recall setup: Gray region indicates precision for fewer than

90\%

of queries. For each method we compute

K_{common}

: the largest k for which

\geq 90\%

of tasks contribute a value at every position

\leq

k (i.e., their lists are at least that long). The area up to

K_{common}

under that method’s curve is filled green (trusted, high task support); the tail beyond

K_{common}

is filled grey (lower support). A dot marks (

K_{common}

, Precision at K common.)

Figure 7: System precision (

\text{Prec}_{\text{ sys}}@k

) as a function of

k

for topics set, skills set, and applications set. For topics set and skills set, precision remains high as

k

increases, while applications set exhibits a sharper decline.

Category	Score	$\text{Prec}_{\text{ sys}}@k$ [k=10]	$\text{Prec}_{\text{ sys}}@k$ [k=20]
Overall	0.629 ‡	0.700 ‡	0.795 ‡
Applications	0.613 ‡	0.738 ‡	0.725 ‡
Known Validation	0.549 ‡	0.485 ‡	0.604 ‡
Skills	0.603 ‡	0.626 ‡	0.747 ‡
Topics	0.576 ‡	0.562 ‡	0.622 ‡

Appendix

Table 12: Spearman correlation coefficients. ‡ indicates

p<0.001

; † indicates

p<0.05

Figure 8: Marginal relevance-label distributions assigned by GPT-5-mini (designated judge) and Gemini-2.5-flash-lite over the same evaluated top- $k$ retrieved items.

Panel of Figure 9 — Figure 9: Latency of the BenchBrowser web demo. (Left) Empirical CDF of request latency over 210 runs of the public BenchBrowser demo ( $k{=}35$ ), with percentile markers (p50/p90/p95/p99). Median latency is 4.47s; tail latency remains below 8.95s at p99 (0 failures). (Right) Average time spent in each pipeline stage, showing testcase generation (1.67s; 31.7%), embedding generation (0.99s; 18.8%), and sample scoring (0.59s; 11.2%) as the dominant contributors; the remaining time corresponds to end-to-end overhead.

Sample Annotator Comment
use-case: Learn Differential Equations
Prompt: Antiderivar la siguiente función f(x)=(x+1)(2x-1)	A says: “solving DEs requires computing antiderivatives;” B says: “The question is testing both translation and maths;within maths, differential equation is not explicitly tested so on average I this isn’t relevant”.
use-case: Detecting emotional support in conversations
Group chat excerpt: Owner: @Team - Aren’t you too unpunctual? We agreed on 10 o’clock yesterday ….. Prompt : Determine the emotion of the owner’s last message. Choose from [negative, neutral, positive].	A says : The task is sentiment classification which is relevant to detecting emotions. B says: The task description is specifically asking for cases that test emotional support, while the question tests emotion classification with predefined categories which aren’t relevant to emotional support.
use-case: Identifying the underlying concept for a given set of items
All of the following support the endosymbiotic theory …. EXCEPT • Mitochondria and chloroplasts divide … • Mitochondria and chloroplasts have ribosomes ….	Annotator B: Given the broad nature of ”concepts, factual knowledge,” technically anything in a domain could share underlying concepts. This seems too abstract and insufficiently related to the specific task so I can consider this partially relevant at best”. Annotator A marks as ’relevant’ but does not justify in the comments.

Appendix

Table 13: Comments from Annotators over samples where we observe disagreement: Annotator A is always the annotator assigning a high relevance (1) and Annotator B is always the annotator assigning a low relevance (0).

Panel of Figure 10 — Figure 10: Intra and Inter-Label distributions between LLM and human annotators on the examples evaluated from the topics, skills and applications set. Most disagreements occur between partially relevant and relevant categories rather than polar opposites, implying calibration rather than conceptual divergence.

Figure 11: BM25 retrieval results for calibrating the variation in content validity as reported by BenchBrowser

Category	Key Used	Processors / Benchmarks
Custom Metric	Varied	complexbench (point_judges), infobench (DRFR), ifeval (inst_level_strict_acc), llmbar (gold label)
Accuracy	acc	nutribench, truthfulQA, arc, mmlu, winogrande, hellaswag, headqa, logiqa, mathqa, openbookqa, sciq, pubmed, wmdp, moral_stories, big-bench tasks
	exact_match	triviaqa
	pass@1	humaneval
Exact Match	exact_match	bbh, gsm8k, webqs
F1 Score	f1/f1_abstractive	qasper

Appendix

Table 14: Summary of evaluation metrics used across benchmarks.

Figure 12:

\tau_{\mathrm{sanity}}

and

\tau_{\mathrm{gold}}

provide gold-only stability controls;

\tau_{\mathrm{ret}}

measures additional divergence introduced by retrieval.

Benchmark Validity

Benchmark validity concerns whether evaluation tasks, scores, and rankings support the capability claims researchers make about AI systems.

See all 170 papers

Who Benchmarks the Benchmarks? Towards Comprehensive Evaluation of Commonsense Reasoning Benchmarks
Apr 10, 2025Pavel Chizhov, Anton Changalidis, Vishnu Prasad Vijaya Kumar +4Benchmark Auditing Reasoning Evaluation
1CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt · 2PleIAs, Paris, France
BenchBench: Benchmarking Automated Benchmark Generation
Mar 21, 2026Yandan Zheng, Haoran Luo, Zhenghong Lin +2Benchmark Design Synthetic Benchmark Generation
Nanyang Technological University Singapore, Singapore
Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHub
May 31, 2025Eunsu Kim, Haneul Yoo, Guijin Son +3LLM Evaluation
KAIST · Yonsei University · OnelineAI +1

Apr 10, 2025J/K move · Enter open · S save

Who Benchmarks the Benchmarks? Towards Comprehensive Evaluation of Commonsense Reasoning Benchmarks

Pavel Chizhov, Anton Changalidis, Vishnu Prasad Vijaya Kumar, Yannick Detrois+3

1CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt · 2PleIAs, Paris, France

Benchmark Auditing Reasoning Evaluation Benchmark Design Benchmark Validity

Open paper AskPDF arXiv

Commonsense reasoning is a key language model capability, as it is purportedly a prerequisite for many basic tasks, unlike specific factual knowledge. It is often measured with multiple-choice questions (MCQ) benchmarks, e.g. HellaSwag and PIQA. Some of these benchmarks, however, are outdated and contain numerous validity issues. We illustrate some typical validity issues with a case study on HellaSwag, one of the most popular and problematic benchmarks for commonsense reasoning. The issues we find range from basic ungrammaticality and numerous typos to misleading prompts or equally correct options. We show that if we remove question prompts or replace them with "Lorem ipsum dolor...", about 68% of model predictions do not change. We argue that this occurs due to inner flaws in the benchmark, not mere contamination that might be present in some models. Since benchmark scores are an essential part of model selection in both research and commercial applications, these issues can have severe consequences. Based on our findings, we propose BenCheck, a package for benchmark validity analysis that encapsulates the main checks performed in our case study and can be used to audit commonsense reasoning benchmarks. We apply these checks to PIQA, Global PIQA, and Winogrande.

Abstract

Benchmark Validity

Explore similar work

Who Benchmarks the Benchmarks? Towards Comprehensive Evaluation of Commonsense Reasoning Benchmarks

BenchBench: Benchmarking Automated Benchmark Generation

Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHub