Whose Data, Whose Benchmark? A User-Centric Framework for Auditing Evaluation Resources

Mir Tafseer Nayeem; Davood Rafiei

cs.CLOct 13, 2026

Whose Data, Whose Benchmark? A User-Centric Framework for Auditing Evaluation Resources

Authors: Mir Tafseer Nayeem, Davood Rafiei

Organizations: University of Alberta

Abstract

Benchmarks shape how model capabilities are measured and interpreted, yet their scores can obscure whose conventions, contexts, and judgments define successful performance. We introduce a user-centric audit framework that distinguishes users who contribute benchmark content from those whose needs evaluation results are intended to represent. The framework operationalizes seven dimensions across benchmark content, metadata, documentation, and scoring procedures. Across 16 resources, we find that American-centered conventions account for 62.4-94.4% of matched signals in 11 text benchmarks, while geographic visibility in five image-text resources is concentrated and strongly associated with national GDP share (mean Pearson r = 0.780; Spearman p = 0.778). Documentation reveals a systematic source-target gap: all 16 resources operationalize source-user context, but none operationalize target-user context. Paired scoring experiments further show that meaning-preserving answer variants reduce DROP exact-match accuracy from 100.0% to 33.3%, while TruthfulQA exhibits metric-dependent reductions in reference similarity. Our framework and documentation checklist make user assumptions auditable, clarifying what benchmark scores represent and where evaluation claims are supported.

Figures & tables

Audit dimension	User-centric question	Operational signals in this paper
Linguistic and locale conventions	Whose spelling, formatting, and measurement conventions are treated as default?	Spelling variants (e.g., color/colour), date formats, time notation (e.g., 7 pm/19:00), and measurement units (e.g., miles/kilometres) in benchmark questions, passages, answer options, and gold answers.
Geographic visibility	Which countries and regions are visible in image–text resource descriptions?	Explicit country references in captions and textual metadata, resolved using country names, demonyms (e.g., Japanese), capitals, and selected cities (e.g., Paris).
Socioeconomic structure	How does geographic visibility relate to economic and socioeconomic indicators?	Associations with GDP share, population share, GDP per capita, Internet usage, and tourism arrivals.
Source-user context	Where does benchmark content originate, and whose contexts shape its collection?	Documented sources (e.g., examinations, Wikipedia, Reddit, CommonCrawl) and provenance metadata, such as website domains (e.g., bbc.co.uk , reddit.com ) and country-code TLDs (e.g., .uk , .ca , .jp ).
Target-user assumptions	Which populations and settings are evaluation claims intended to represent?	Documented intended users (e.g., students, clinicians), evaluation audiences, deployment contexts, and conditions for interpreting benchmark results.
Annotator visibility	Whose judgments determine labels, references, and annotation quality?	Documented annotator expertise (e.g., medical professionals), demographics, locale (e.g., country, language variety), recruitment (e.g., crowdsourcing), compensation, annotation protocols, and quality-control procedures (e.g., inter-annotator agreement).

Table 1: User-centric audit taxonomy for evaluation resources. Seven dimensions connect user-relevant assumptions to measurable and inspectable evidence across benchmark content, metadata, documentation, and scoring procedures.

Figure 1: Linguistic and locale convention signals across 11 text benchmarks. Bars compare American-centered and other user-valid matches in spelling, dates, time, and measurement units; rightmost columns show aggregate shares. American-centered conventions predominate, with variation across benchmarks and convention types. Appendix Figure 4 provides the breakdown.

Benchmark	Coverage (%)	AmE excl. spelling (%)
MMLU ( Hendrycks et al., 2021 )	26.60	66.9
GSM8K ( Cobbe et al., 2021 )	25.40	83.3
HellaSwag ( Zellers et al., 2019 )	38.11	71.2
DROP ( Dua et al., 2019 )	78.08	72.3
WinoGrande ( Sakaguchi et al., 2021 )	5.97	77.2
ARC-Challenge ( Clark et al., 2018 )	18.73	22.5

Table 2: Convention coverage and spelling-excluded robustness across 11 text benchmarks. Coverage is the share of items containing an audited signal; the final column reports the American-centered share among non-spelling matches.

Benchmark	Metric	Orig.	Var.	$\Delta$ [95% CI]
DROP	EM	100.0	33.3	$-66.7$ [ $-92.3,-36.4$ ]
DROP	F1	100.0	77.2	$-22.8$ [ $-41.6,-5.2$ ]
TQA	BLEU correct.	97.0	97.0	$0.0$ [ $0.0,0.0$ ]
TQA	ROUGE-1 correct.	97.0	97.0	$0.0$ [ $0.0,0.0$ ]
TQA	ROUGE-2 correct.	100.0	93.9	$-6.1$ [ $-16.1,0.0$ ]
TQA	ROUGE-1 sim.	100.0	88.9	$-11.1$ [ $-13.5,-9.0$ ]

Table 3: Scorer sensitivity to answer variants. DROP uses official EM/F1; TQA (TruthfulQA) uses BLEU/ROUGE reference-ranking rules. Changes are in percentage points for EM, F1, and correctness, and on a 0–100 scale for similarity. CIs use question-clustered bootstrap resampling.

Figure 2: Caption geography is concentrated in a few countries. Bars show the top five country shares from caption-derived assignments in each resource. The US leads four resources, while Iceland leads CC12M, revealing source-specific variation. Countries are identified through deterministic text matching, without image geolocation.

Figure 3: Country visibility is strongly associated with GDP share. Each point represents a country with at least one million people. GDP and caption-derived country shares are normalized over the same retained countries and plotted on log scales. Fitted trends and Spearman correlations summarize the association across resources.

Criterion	Score	Audit question
Source-user context	0–2	Does the documentation identify who produced the content, including source populations, platforms (e.g., Reddit, Wikipedia), institutions, or collection processes?
Target-user or evaluation context	0–2	Does it specify intended users (e.g., students, clinicians), evaluation audiences, deployment settings, or the contexts in which benchmark results are expected to apply?
Annotator visibility	0–2	Does it describe annotator expertise, demographics, locale, recruitment (e.g., crowdsourcing), compensation, quality control, or the absence of human annotation?
Locale assumptions	0–2	Does it document spelling (e.g., color/colour), date, time, measurement, currency, numeric-format, language-variety, or source-language conventions?
Evaluation normalization	0–2	Does it document how scoring procedures handle valid alternative answers, formatting conventions (e.g., date formats), or locale-specific equivalents?
Source/geographic metadata	0–2	Does it expose source URLs, website domains (e.g., bbc.co.uk , reddit.com ), top-level domains (e.g., .uk , .ca , .jp ), language tags (e.g., en , fr , ja ), country or region fields, or other metadata supporting provenance and geographic auditing?

Table 4: Documentation audit rubric across six user-relevant criteria. Each criterion is scored as 0 (absent), 1 (mentioned explicitly), or 2 (operationalized through inspectable data, metadata, or evaluation procedures).

Criterion	Mean score	Ment. (%)	Oper. (%)
Source-user context	2.00	100.0	100.0
Target/eval. context	1.00	100.0	0.0
Annotator visibility	0.56	56.2	0.0
Locale assumptions	0.31	25.0	6.2
Evaluation normalization	1.44	75.0	68.8
Source/geographic metadata	0.69	37.5	31.2

Table 5: Documentation Audit results across 16 resources. Mean scores range from 0 (absent) to 2 (operationalized). “Ment.” indicates resources scoring 1 or 2; “Oper.” indicates those scoring 2, % are computed across all 16 resources.

Field	Minimum documentation question	Why it matters for evaluation
Source-user context	Who produced the content, and through which platform, institution, task, or collection process?	Identifies whose language, knowledge, experiences, and judgments are represented.
Target-user context	Which users, settings, or evaluation claims is the benchmark intended to represent?	Clarifies the populations and contexts to which benchmark results may apply.
Annotator context	Who provided labels or reference answers, and what is known about their expertise, locale, recruitment, compensation, and quality control?	Makes visible whose judgments determine correctness, relevance, or acceptability.
Locale conventions	Which spelling, date, time, unit, currency, numeric-format, or language-variety conventions appear in the data or scoring procedure?	Identifies regional assumptions that may influence benchmark content and evaluation outcomes.
Evaluation normalization	How does the scorer handle valid alternative answers, including spelling, formatting, measurement, and locale-specific equivalents?	Reveals whether equivalent responses receive consistent scores across user conventions.
Source/geographic metadata	Which URLs, website domains, top-level domains, language tags, or geographic fields are available for auditing?	Supports tracing content sources and examining geographic visibility without conflating source provenance with content location.

Table 6: Minimal user-centric documentation checklist for evaluation resources. The six fields translate the audit taxonomy into practical reporting questions that clarify benchmark provenance, intended users, and evaluation assumptions.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Task	Category	Short description
MMLU ( Hendrycks et al., 2021 )	Multiple-choice QA over 57 subjects	Broad knowledge and reasoning	Academic and professional exam questions spanning humanities, STEM, and professional domains.
GSM8K ( Cobbe et al., 2021 )	Grade-school math word problems	Mathematical reasoning	Linguistically diverse elementary math problems with free-form reasoning and numeric answers.
HellaSwag ( Zellers et al., 2019 )	Sentence / ending completion	Commonsense reasoning	Adversarially filtered everyday-event completion benchmark with multiple-choice endings.
DROP ( Dua et al., 2019 )	Reading comprehension with arithmetic	Reading + discrete reasoning	Paragraph-based QA requiring counting, comparison, addition, subtraction, and date/number tracking.
WinoGrande ( Sakaguchi et al., 2021 )	Pronoun / referent resolution	Commonsense coreference	Large-scale adversarial Winograd-style benchmark for commonsense pronoun resolution.
ARC-Challenge ( Clark et al., 2018 )	Science multiple-choice QA	Science reasoning	Difficult subset of grade-school science questions from the AI2 Reasoning Challenge.

Appendix

Table 7: Overview of the benchmarks used in our evaluation, summarizing each benchmark’s task formulation, capability category, and intended reasoning or knowledge focus.

Convention	American-centered example	Other user-valid example	Signal type
Spelling	color , center	colour , centre	Orthographic
Numeric date	04/21/2025	21/04/2025	Date order
Written date	April 21, 2025	21 April 2025	Date order
Time notation	7:00 pm	19:00	Time format
Measurement	5 miles	8.05 kilometres	Unit convention
Temperature	68 degrees Fahrenheit	20 degrees Celsius	Unit convention

Appendix

Table 8: Illustrative linguistic and locale conventions used in the text benchmark audit. The examples show alternative valid forms; the detectors measure their occurrence in benchmark text rather than assigning correctness or user identity.

Figure 4: Benchmark-level distribution of linguistic and locale convention signals across 11 text benchmarks. Each panel shows, for a convention type, the proportions of American-centered and other user-valid matches within each benchmark. American-centered forms predominate in many benchmarks, with substantial variation in the direction of matches across spelling, dates, time, and measurement units. Date and time signals are generally sparser, revealing distinct convention profiles across resources.

Benchmark	Scoring rule	Pairs	Questions	Original	Variant	Change [95% CI]
DROP	Official EM	12	11	100.0%	33.3%	$-66.7$ [ $-92.3,-36.4$ ]
DROP	Official F1	12	11	100.0%	77.2%	$-22.8$ [ $-41.6,-5.2$ ]
TruthfulQA	BLEU correctness	33	25	97.0%	97.0%	$0.0$ [ $0.0,0.0$ ]
TruthfulQA	ROUGE-1 correctness	33	25	97.0%	97.0%	$0.0$ [ $0.0,0.0$ ]
TruthfulQA	ROUGE-2 correctness	33	25	100.0%	93.9%	$-6.1$ [ $-16.1,0.0$ ]
TruthfulQA	ROUGE-Lsum correctness	33	25	100.0%	100.0%	$0.0$ [ $0.0,0.0$ ]

Appendix

Table 9: Detailed paired scorer sensitivity to meaning-preserving answer variants. DROP uses its official EM/F1 scorer; TruthfulQA reproduces the released BLEU/ROUGE correct-versus-incorrect reference-ranking procedure. EM, F1, and correctness changes are percentage points; reference similarities are reported on a 0–100 scale. Confidence intervals are calculated using 10,000 question-clustered bootstrap replicates. Benchmark results are not pooled.

Benchmark	Source / formulation	Role in this study	Short description
DataComp1B ( Gadre et al., 2023 )	Web-scale image–text data curation benchmark.	Core web training-corpus benchmark.	Broad, noisy web image–text source used to test whether country mentions in caption metadata reflect the geographic skew of large-scale multimodal training data.
WIT English ( Srinivasan et al., 2021 )	Wikipedia image–text dataset with captions and structured page context.	Curated encyclopedia-style control.	English view of WIT, using caption and contextual text fields from Wikipedia. It provides a more structured source domain than web-crawl alt-text corpora.
RedCaps ( Desai et al., 2021 )	Reddit-sourced image–text pairs from curated subreddits.	Social-web comparison benchmark.	User-generated social-media caption source. It tests whether geographic concentration persists outside Wikipedia and conventional web alt-text datasets.
CC12M ( Changpinyo et al., 2021 )	Web-collected image–alt-text pairs extending Conceptual Captions.	Classic alt-text caption benchmark.	Large web caption corpus designed to broaden Conceptual Captions toward long-tail visual concepts. It gives a canonical vision–language pretraining comparison point.
COYO-700M ( Byeon et al., 2022 )	CommonCrawl-derived image–text pairs with additional metadata and filtering scores.	Web-scale expansion benchmark.	Open web-scale image–text corpus included to test whether the observed geographic patterns remain consistent beyond the smaller benchmark and control datasets.

Appendix

Table 10: Overview of the five image–text resources included in the geographic visibility audit. The table summarizes their data sources, characteristics, and roles in the analysis. Geographic visibility is measured using explicit country references resolved from captions and associated textual metadata. Country assignments reflect textual references rather than image geolocation or the geographic origin of the content.

Figure 5: The high-frequency tail remains geographically concentrated. Extending Figure 2 to the top ten countries shows that resolved caption geography often centers on recurring countries in North America, Europe, and large Asian economies. CC12M is a benchmark-specific exception in which Iceland is the top resolved country, suggesting that source-pipeline artifacts or repeated content clusters may shape some country-frequency patterns. Rankings are benchmark-specific and do not require cross-benchmark overlap.

Rank	DataComp1B	WIT English	RedCaps	CC12M	COYO-700M
1	Belarus (2)	Angola (2)	Anguilla (2)	Afghanistan (2)	Argentina (2)
2	Belize (2)	Bahrain (2)	Canada (2)	Bahamas (2)	Bangladesh (2)
3	Bolivia (2)	Belize (2)	China (2)	Belize (2)	Bulgaria (2)
4	Congo (2)	Benin (2)	Germany (2)	Brunei Darussalam (2)	Ethiopia (2)
5	Dominica (2)	Cameroon (2)	India (2)	Denmark (2)	Holy See (2)
6	Guam (2)	Dominica (2)	Italy (2)	Falkland Islands (2)	Jamaica (2)

Appendix

Table 11: Benchmark-wise low-frequency countries with at least two samples. Each column lists the ten lowest-share countries within a benchmark, ranked independently over resolved country assignments. Countries are included only when they have at least two samples in that benchmark; counts in parentheses denote those caption-derived assignments. The resulting lists include small island territories, Caribbean and Pacific countries, and countries from Africa, Asia, Europe, and the Americas. This pattern suggests a broad long tail of sparsely represented geographies rather than a single region-specific effect.

Benchmark	Countries	Pearson $r$	Spearman $\rho$	Weighted $r$	OLS $R^{2}$
DataComp1B	118	0.816	0.786	0.907	0.667
WIT English	146	0.851	0.823	0.874	0.724
RedCaps	30	0.647	0.735	0.809	0.419
CC12M	74	0.770	0.772	0.833	0.593
COYO-700M	109	0.817	0.775	0.897	0.667
Mean	–	0.780	0.778	0.864	0.614

Appendix

Table 12: GDP-share association with country visibility across five image–text resources. Countries with populations below one million are excluded. Pearson correlations use log GDP share and log dataset share; weighted correlations use resolved country counts.

Model	Mean OLS $R^{2}$	Weighted $R^{2}$
GDP share only	0.614	0.748
Population share only	0.252	0.374
GDP per capita only	0.210	0.254
Internet users (%) only	0.111	0.124
Tourism arrivals only	0.347	0.396
GDP share + population share	0.617	0.763

Appendix

Table 13: Mean explanatory fit of socioeconomic indicators across the five resources. Results follow the population threshold and complete-case requirements of the geographic analysis.

Measure	English	Balanced non-English
U.S. share (%)	20.0	8.5
Top-10 GDP share (%)	–	53.5
Pearson $r$	–	0.752
Spearman $\rho$	–	0.744

Appendix

Table 14: Geographic visibility in the WIT language-composition stress test. The balanced non-English sample includes French, German, Spanish, and Japanese. Country shares use resolved assignments; GDP correlations and top-ten concentration are reported for the non-English setting.

Criterion	Exact (%)	$\kappa$	Disagr.
Source-user context	100.0	–	0
Target/eval. context	100.0	–	0
Annotator visibility	100.0	1.000	0
Locale assumptions	100.0	1.000	0
Evaluation normalization	100.0	1.000	0
Source/geographic metadata	93.8	0.881	1

Appendix

Table 15: Inter-annotator agreement across 16 resources before adjudication. Exact agreement and Cohen’s

\kappa

are reported for each criterion. Dashes indicate undefined

\kappa

when both annotators assign a constant score.

Resource	Family	Primary documentation source
MMLU	Text QA	Hendrycks et al. (2021) , paper .
GSM8K	Text QA	Cobbe et al. (2021) , paper and dataset release .
HellaSwag	Text QA	Zellers et al. (2019) , paper and supplement .
DROP	Text QA	Dua et al. (2019) , paper .
WinoGrande	Text QA	Sakaguchi et al. (2021) , paper and project page .
ARC-Challenge	Text QA	Clark et al. (2018) , paper .

Appendix

Table 16: Resources included in the documentation audit and the public documentation sources used for scoring. Each link points to the benchmark paper, official dataset card, or official repository consulted during the audit. All documentation sources were accessed in April 2026.

Resource	Family	Source	Target/eval.	Annot.	Locale	Eval.	Src./Geo.	Total
MMLU	Text QA	2	1	0	0	2	0	5/12
GSM8K	Text QA	2	1	1	0	2	0	6/12
HellaSwag	Text QA	2	1	1	0	2	0	6/12
DROP	Text QA	2	1	1	0	2	0	6/12
WinoGrande	Text QA	2	1	1	0	2	0	6/12
ARC-Challenge	Text QA	2	1	0	0	2	0	5/12

Appendix

Table 17: Benchmark-wise documentation audit scores across 16 resources. Each criterion is scored as 0 (absent), 1 (mentioned), or 2 (operationalized), with a maximum total of 12. Source = source-user context; Target/eval. = target-user or evaluation context; Annot. = annotator visibility; Locale = locale assumptions; Eval. = evaluation normalization; Src./Geo. = source/geographic metadata.

Benchmark Auditing

Benchmark auditing checks whether evaluation datasets, metrics, and infrastructure measure the capabilities and risks they claim to assess.

See all 97 papers

Feb 25, 2026

BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity

Do language model benchmarks actually measure what practitioners intend them to ? High-level metadata is too coarse to convey the granular reality of benchmarks: a "poetry" benchmark may never test for haikus, while "instruction-following" benchmarks will often test for an arbitrary mix of skills. This opacity makes verifying alignment with practitioner goals a laborious process, risking an illusion of competence even when models fail on untested facets of user interests. We introduce BenchBrowser, a retriever that surfaces evaluation items relevant to natural language use cases over 20 benchmark suites. Validated by a human study confirming high retrieval precision, BenchBrowser generates evidence to help practitioners diagnose low content validity (narrow coverage of a capability's facets) and low convergent validity (lack of stable rankings when measuring the same capability). BenchBrowser, thus, helps quantify a critical gap between practitioner intent and what benchmarks actually test.

Harshita Diddee, Gregory Yauney, Swabha Swayamdipta +1

Language Technologies Institute, Carnegie Mellon University, Pittsburgh (PA), United States · Thomas Lord Department of Computer Science, University of Southern California, Los Angeles (CA), United States

Benchmark Validity Benchmark Design

Ask PDF

Jul 30, 2026

Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation

Benchmark datasets are central to evaluating Large Language Models (LLMs), yet they are typically conceived as monolithic tasks, obscuring substantial variation in the demands of individual samples. We introduce a dataset-centric meta-evaluation framework that audits benchmark datasets at the sample level along five latent dimensions: 1. Cognitive and Knowledge Demands, 2. Language and Content Quality, 3. Task Properties, 4. Context, and 5. Ethics, Safety, and Fairness. Applying this framework, we annotate five influential benchmarks -- MMLU, ARC, WinoGrande, HellaSwag, and TruthfulQA -- revealing pronounced internal heterogeneity that is not captured by aggregate accuracy scores. We show how these annotations enable criterion-driven orchestration of composite benchmark subsets across datasets, supporting targeted evaluation of model capabilities such as Reasoning Depth or Ethical Sensitivity. This approach reframes benchmark evaluation as dataset introspection, providing a principled methodology for analyzing and re-composing existing benchmarks to better reflect diverse evaluation needs.

Philipp D. Siedler, Jordan Sassoon

Aleph Alpha Research, Heidelberg, Germany.

LLM Evaluation Benchmark Auditing Benchmark Construction LLM Auditing

Ask PDF

May 31, 2025

Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHub

LLM benchmarks are often treated as coherent measurement units, yet they are heterogeneous collections of instances spanning diverse domains, skills, formats, and contexts. As a result, aggregate benchmark scores can conflate model capability with benchmark composition, obscuring what existing benchmarks actually cover. We introduce BenchHub, a composition-aware evaluation framework that represents benchmarks as distributions over instance-level attributes. BenchHub integrates 54 benchmarks comprising 839K samples across 10 languages. It enables researchers to inspect benchmark contents, uncover reusable coverage hidden behind benchmark names, transparently compare benchmarks, and construct controllable evaluation sets for application-aligned model selection. Using BenchHub, we show that similarly motivated benchmarks can differ substantially in internal composition, existing benchmarks often contain reusable coverage beyond their stated purposes, models exhibit fine-grained category-level performance variation hidden by aggregate scores, and model rankings can shift under different reweighting and resampling configurations. Our results motivate evaluation practices that make benchmark composition explicit, inspectable, and controllable.

Eunsu Kim, Haneul Yoo, Guijin Son +3

KAIST · Yonsei University · OnelineAI +1

LLM Evaluation

Ask PDF

Abstract

Benchmark Auditing

Explore similar work

BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity

Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation

Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHub