Organizations: Language Technologies Institute, Carnegie Mellon University, Pittsburgh (PA), United States · Thomas Lord Department of Computer Science, University of Southern California, Los Angeles (CA), United States
Do language model benchmarks actually measure what practitioners intend them to ? High-level metadata is too coarse to convey the granular reality of benchmarks: a "poetry" benchmark may never test for haikus, while "instruction-following" benchmarks will often test for an arbitrary mix of skills. This opacity makes verifying alignment with practitioner goals a laborious process, risking an illusion of competence even when models fail on untested facets of user interests. We introduce BenchBrowser, a retriever that surfaces evaluation items relevant to natural language use cases over 20 benchmark suites. Validated by a human study confirming high retrieval precision, BenchBrowser generates evidence to help practitioners diagnose low content validity (narrow coverage of a capability's facets) and low convergent validity (lack of stable rankings when measuring the same capability). BenchBrowser, thus, helps quantify a critical gap between practitioner intent and what benchmarks actually test.
Figures & tables
Figure 1: BenchBrowser retrieves use-case-relevant items from 20+ benchmarks to help diagnose validity failures. For a “Writing Programming Functions” use-case: Python-skewed coverage (left; content validity gap) and unstable model ranks across subsets (right; low convergent validity) make conclusions unreliable. m1 to mk are mid-sized decoder models. See Appendix A for details.
Figure 2: BenchBrowser pipeline: a practitioner submits a use-case of interest to BenchBrowser ; the use-case is a rewritten into anchors for more diverse retrieval, which are then b embedded and used to retrieve c relevant examples from a suite of 20+ benchmarks. The retrieved evidence can be used by the practitioner in multiple ways (images show snapshots of BenchBrowser ’s website UI): d1 Comparing the representation of the use-case along different facets. For example, a user interested in assessing “reasoning skills” can observe the different ways (or lack thereof) in which reasoning competence is evaluated (linguistics fundamentals versus math problems) and d2 Compare the performance and stability of models evaluated on existing benchmarks (e.g., ARC AI2 Reasoning) for the task as well as subsets of the retrieved examples.
Original:
understanding geopolitical tensions in Southeast Asia
Rephrasing:
South China Sea territorial disputes and maritime tensions • ASEAN diplomacy, regional security mechanisms, and conflict prevention • US–China strategic competition
Example Synthesis:
Discuss the strategic factors driving contemporary geopolitical tensions in Southeast Asia • MCQ: Which factor most significantly contributes to territorial disputes in the South China Sea? […] • Fill in the blank: ASEAN’s principle of ___________ often limits collective action […]
Shorthand:
Table 1: We compare four methods for producing retrieval anchors. Here we show how each handles an example practitioner use-case: “ understanding geopolitical tensions in Southeast Asia ”.
Name
Origin
N
Examples
Automatically synthesized
Topics
BBH, MMLU
58
Marketing, Law, HS Maths
Skills
Clio
50
creative writing, navigation
Applications
Clio
50
Tips for Winter Gardening
Manually curated
Known Valid.
LM Eval
20
Riddles, Calorie Counting
Table 2: Use-case query sets for evaluating the retrieval quality of BenchBrowser . Automatically synthesized sets derive from benchmarks; manually curated sets are hand-crafted for validation.
Figure 3: Variation in the percentage of Relevant examples across different facets of the same broad use-case. For each skill family, we show the facet for which the highest and lowest number of examples are retrieved. Most capabilities show sharp disparities: certain facets are heavily over-represented in our database of 70k benchmark examples, while others are nearly absent from benchmarks.
Use-case
Mean H
Mean LLM
ρ
κ
Topics
0.96
0.92‡
0.15†
0.10
Skills
0.80
0.91‡
0.17†
0.11
Applications
0.74
0.75
0.50‡
0.29
Known Valid.
0.58
0.88‡
0.34‡
0.29
Novel
0.70
0.67
0.27‡
0.13
Table 4: Mean relevance of retrieved examples by human annotators vs. the LLM judge across use-case query sets. Superscripts indicate significance: †p<0.05 ; ‡p<0.0001 . Markers on Mean LLM are significance markers for paired t -test for the means. ρ is Spearman’s rho and κ is Fleiss’s kappa.
Code generation
Creative writing
Programming language
Literary genre
“Write a function in … ”
“Write a … story”
Facet
Prec sys@k
Facet
Prec sys@k
Python
0.867
fantasy
0.759
C++
0.741
science fiction
0.679
Rust
0.478
mystery
0.654
Table 5: Different facets of the same skill family can have very different Prec sys@k when using BenchBrowser with Example Synthesis anchors and the LLM judge (relevant examples only). Most skill families show substantial spread: some variants (green) have many Relevant examples, while others (red) are barely represented. For complete results for all skill families, see Appendix J .
Figure 4: Kendall’s τ correlations of model rankings on gold test sets vs. retrieved sets for known validation set use-cases with mean human relevance ≥0.5 . Wider bars correspond to greater rank divergence Δ=τgold−τret . n : number of retrieved examples.
Root Cause 1: Rubric Shift
Example Query: “Hindu Religion”
HellaSwag: Philosophy and Religion: How to learn about ancient hinduism …
BIG-bench Hard (Hindu Knowledge): Which of the following gods are associated with a flying elephant
Root Cause 2: Divergent Operationalization
Example Query: “Solving Puzzles”
BIG-bench Hard (Colored Objects): On the desk, you see items in a row: a gold textbook, a purple puzzle, a teal necklace, and a silver pencil. How many non-gold items do you see to the right of the pencil?
Table 6: Root causes of ranking inversions in BenchBrowser ; gold highlight indicates example is from the gold set, and blue indicates it is from the retrieved set.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Test Case
Shorthand
mathqa
if x is a number such that x2+2x−24=0 and x2−5x+4=0 , then x=? Options: a) 4, b) -4, c) -3, d) -6, e) 1
Who invaded Europe from Mongolia and Turkey over 300 years, beginning in the 13th century?
factual_recall & historical_figures & mongols
alpacaeval
Provide a formula for computing the nth term in the given sequence: 5, 14, 23, 32, 41, …
equation_solving & sequence_formula & nth_term
arc_c
Scientists often change conclusions when new information becomes available. Which statement was changed? Options: Moon is in outer space / Plants make own food / Sun revolves around Earth
reading_comp & analysis & scientific_conclusions
Appendix
Table 10
use-case
Relevant example
Verify scientific claims
Claim: AdaBERT achieves inferior performance while significantly worsening the effiency by 12.7x. Evidence: The dataset contains 3772 word pairs. The accuracy of ConVecs 70.02% is not significantly different from the accuracy of SimDiffs (72.4%). Label: Negative/Refuted.
Blogging
Write a blog announcing the opening of a new mall in the town of Serenity.
Unit Testing Code
Question: An anagram is the result of rearranging the letters of a word to produce a new word. Note: anagrams are case insensitive Complete the function to return true if the two arguments given are anagrams of each other; return false otherwise. Examples: foefet is an anagram of toffee. ut_id: 0 code - import unittest class TestAreAnagramsFunction(unittest.TestCase): This class contains unit tests for the are_anagrams function. def test_anagram(self): Test that two anagrams return True. self.assertTrue(are_anagrams(”foefet”, ”toffee”)) def test_not_anagram(self): Test that two non-anagrams return False.
Star Gazing
The Big Dipper is a ______ constellation.
Resume for Machine Learning Engineer
I am a software engineer with 5 years of experience in the industry. Can you help me write a resume?
Appendix
Table 11
Retriever Type
Prec sys@k
Recall@k
NDCG@10 / MRR
Processing Time (s)
bge-base-V1.5
0.208
0.004
0.291
0.048
bge-large-V1.5
0.332
0.004
0.353
0.044
llm-embedder
0.488
0.012
0.537
0.029
bge-en-icl
0.694
0.218
0.684
1.040
bge-reranker
0.664
0.176
0.632
14.030
Appendix
Table 9: Retriever Performance Comparison: bge-en-icl achieves the highest Prec sys@k , Recall@k and NDCG@k with reasonable latency.
Benchmark
Citation
Benchmark
Citation
IFEval
Zhou et al. (2023)
logiqa
Liu et al. (2021)
MMLU
Hendrycks et al. (2021)
mathqa
Amini et al. (2019)
TruthfulQA
Lin et al. (2022)
openbookqa
Mihaylov et al. (2018)
HellaSwag
Zellers et al. (2019)
qasper
Dasigi et al. (2021)
ARC
Chollet (2019)
triviaqa
Joshi et al. (2017)
Winogrande
Sakaguchi et al. (2020)
webqs
Berant et al. (2013)
Appendix
Table 10: Benchmarks Used for Example Retrieval
Usecase
Gold Test Set
Learn how to measure my carbohydrate intake and make my meals more balanced
nutribench ( Hua et al., 2024 )
Test my knowledge of different sports and their respective players
bb_sports
Help me learn more about the Hindu religion
bb_hindu
Test me on different topics in science
sciq ( Welbl et al., 2017 )
Help me answer questions for my biomedical research
pubmed ( Jin et al., 2019 )
Help me mitigate different biological, cyber, and chemical attacks
wmdp ( Li et al., 2024 )
Appendix
Table 11: Prompts with Corresponding Gold Datasets used for the Known Validation Set. All datasets prefixed with “bb” are included in the Big Bench Dataset ( Srivastava et al., 2023 ) .
Figure 5: We observe that performance on the retrieved testset and the gold test set (BBH and MMLU for the topics set set) show reasonable to high correlation.
Figure 6: intersection@k in approximated recall setup: Gray region indicates precision for fewer than 90% of queries. For each method we compute Kcommon : the largest k for which ≥90% of tasks contribute a value at every position ≤ k (i.e., their lists are at least that long). The area up to Kcommon under that method’s curve is filled green (trusted, high task support); the tail beyond Kcommon is filled grey (lower support). A dot marks ( Kcommon , Precision at K common.)
Figure 7: System precision ( Prec sys@k ) as a function of k for topics set, skills set, and applications set. For topics set and skills set, precision remains high as k increases, while applications set exhibits a sharper decline.
Figure 8: Marginal relevance-label distributions assigned by GPT-5-mini (designated judge) and Gemini-2.5-flash-lite over the same evaluated top- k retrieved items.
Figure 9: Latency of the BenchBrowser web demo. (Left) Empirical CDF of request latency over 210 runs of the public BenchBrowser demo ( k=35 ), with percentile markers (p50/p90/p95/p99). Median latency is 4.47s; tail latency remains below 8.95s at p99 (0 failures). (Right) Average time spent in each pipeline stage, showing testcase generation (1.67s; 31.7%), embedding generation (0.99s; 18.8%), and sample scoring (0.59s; 11.2%) as the dominant contributors; the remaining time corresponds to end-to-end overhead.
Sample Annotator Comment
use-case: Learn Differential Equations
Prompt: Antiderivar la siguiente función f(x)=(x+1)(2x-1)
A says: “solving DEs requires computing antiderivatives;” B says: “The question is testing both translation and maths;within maths, differential equation is not explicitly tested so on average I this isn’t relevant”.
use-case: Detecting emotional support in conversations
Group chat excerpt: Owner: @Team - Aren’t you too unpunctual? We agreed on 10 o’clock yesterday ….. Prompt : Determine the emotion of the owner’s last message. Choose from [negative, neutral, positive].
A says : The task is sentiment classification which is relevant to detecting emotions. B says: The task description is specifically asking for cases that test emotional support, while the question tests emotion classification with predefined categories which aren’t relevant to emotional support.
use-case: Identifying the underlying concept for a given set of items
All of the following support the endosymbiotic theory …. EXCEPT • Mitochondria and chloroplasts divide … • Mitochondria and chloroplasts have ribosomes ….
Annotator B: Given the broad nature of ”concepts, factual knowledge,” technically anything in a domain could share underlying concepts. This seems too abstract and insufficiently related to the specific task so I can consider this partially relevant at best”. Annotator A marks as ’relevant’ but does not justify in the comments.
Appendix
Table 13: Comments from Annotators over samples where we observe disagreement: Annotator A is always the annotator assigning a high relevance (1) and Annotator B is always the annotator assigning a low relevance (0).
Figure 10: Intra and Inter-Label distributions between LLM and human annotators on the examples evaluated from the topics, skills and applications set. Most disagreements occur between partially relevant and relevant categories rather than polar opposites, implying calibration rather than conceptual divergence.
Figure 11: BM25 retrieval results for calibrating the variation in content validity as reported by BenchBrowser
Commonsense reasoning is a key language model capability, as it is purportedly a prerequisite for many basic tasks, unlike specific factual knowledge. It is often measured with multiple-choice questions (MCQ) benchmarks, e.g. HellaSwag and PIQA. Some of these benchmarks, however, are outdated and contain numerous validity issues. We illustrate some typical validity issues with a case study on HellaSwag, one of the most popular and problematic benchmarks for commonsense reasoning. The issues we find range from basic ungrammaticality and numerous typos to misleading prompts or equally correct options. We show that if we remove question prompts or replace them with "Lorem ipsum dolor...", about 68% of model predictions do not change. We argue that this occurs due to inner flaws in the benchmark, not mere contamination that might be present in some models. Since benchmark scores are an essential part of model selection in both research and commercial applications, these issues can have severe consequences. Based on our findings, we propose BenCheck, a package for benchmark validity analysis that encapsulates the main checks performed in our case study and can be used to audit commonsense reasoning benchmarks. We apply these checks to PIQA, Global PIQA, and Winogrande.
Pavel Chizhov, Anton Changalidis, Vishnu Prasad Vijaya Kumar +4
1CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt · 2PleIAs, Paris, France
Benchmarks are the de facto standard for tracking progress in large language models (LLMs), yet static test sets can rapidly saturate, become vulnerable to contamination, and are costly to refresh. Scalable evaluation of open-ended items often relies on LLM judges, introducing additional sources of bias and prompt sensitivity. We argue that evaluation must extend beyond how well models answer benchmarks to how well models design them. We introduce BenchBench, a three-stage pipeline and dataset for benchmarking automated benchmark generation: (i) extract structured domain cards from seed benchmarks, (ii) prompt multiple designer LLMs to generate quota-controlled suites, and (iii) validate items with a multi-model answerer panel using exact/numeric/symbolic verifiers when possible and rubric-guided judging otherwise, yielding designer--answerer matrices with item-level quality flags and psychometric diagnostics. Across nine variants spanning computer science, mathematics, medicine, and theory-of-mind reasoning (including multilingual and multimodal settings), we generate 16.7K items, retain ~15K core items post-filtering, and produce ~152K graded model--item responses. BenchBench shows that benchmark-design ability is only moderately correlated with answer-time strength (Spearman rho 0.37), invalidity is negatively associated with discrimination (Pearson r0.62), and the resulting designer--answerer matrices enable scalable audits of format/modality/language fidelity and suite-dependent self/family interactions. The project is available at: https://github.com/koanatakiyo/BenchBench.
Yandan Zheng, Haoran Luo, Zhenghong Lin +2
Nanyang Technological University Singapore, Singapore
LLM benchmarks are often treated as coherent measurement units, yet they are heterogeneous collections of instances spanning diverse domains, skills, formats, and contexts. As a result, aggregate benchmark scores can conflate model capability with benchmark composition, obscuring what existing benchmarks actually cover. We introduce BenchHub, a composition-aware evaluation framework that represents benchmarks as distributions over instance-level attributes. BenchHub integrates 54 benchmarks comprising 839K samples across 10 languages. It enables researchers to inspect benchmark contents, uncover reusable coverage hidden behind benchmark names, transparently compare benchmarks, and construct controllable evaluation sets for application-aligned model selection. Using BenchHub, we show that similarly motivated benchmarks can differ substantially in internal composition, existing benchmarks often contain reusable coverage beyond their stated purposes, models exhibit fine-grained category-level performance variation hidden by aggregate scores, and model rankings can shift under different reweighting and resampling configurations. Our results motivate evaluation practices that make benchmark composition explicit, inspectable, and controllable.