LLM benchmarks are often treated as coherent measurement units, yet they are heterogeneous collections of instances spanning diverse domains, skills, formats, and contexts. As a result, aggregate benchmark scores can conflate model capability with benchmark composition, obscuring what existing benchmarks actually cover. We introduce BenchHub, a composition-aware evaluation framework that represents benchmarks as distributions over instance-level attributes. BenchHub integrates 54 benchmarks comprising 839K samples across 10 languages. It enables researchers to inspect benchmark contents, uncover reusable coverage hidden behind benchmark names, transparently compare benchmarks, and construct controllable evaluation sets for application-aligned model selection. Using BenchHub, we show that similarly motivated benchmarks can differ substantially in internal composition, existing benchmarks often contain reusable coverage beyond their stated purposes, models exhibit fine-grained category-level performance variation hidden by aggregate scores, and model rankings can shift under different reweighting and resampling configurations. Our results motivate evaluation practices that make benchmark composition explicit, inspectable, and controllable.
Figures & tables
Figure 1: Conceptual overview of BenchHub . BenchHub automatically merges existing benchmark datasets and annotates questions at the instance level with a unified taxonomy, making the contents of existing benchmarks explicit and reusable beyond their dataset names. This enables users to construct objective-aligned test sets and conduct fine-grained, controllable model evaluation.
Figure 2: Data distribution of all datasets used in BenchHub by coarse-grained subjects, targets, and tasks. The English and Korean data include 183k and 137k questions, respectively.
Figure 3: Subject coverage distribution and alignment in English benchmarks.
Figure 4: Subject distributions of existing evaluation benchmarks.
Figure 5: LLM evaluation rankings using BenchHub (English).
Figure 6: Model pairs with similar aggregate accuracy can show distinct category-level profiles. Each point is an unordered model pair. The shaded region marks pairs within 5 %p differences in overall accuracy; red highlighted points (points A, B, C) indicate examples shown in the profile plots.
Figure 7: LLM ranking according to four sampling methods.
Rank
(a) Legal chatbot with advanced reasoning (EN)
(b) Docent agent for Korean traditional art (KO)
Customized
Stratified
Customized
Stratified
1
Qwen3-30B-A3B-Instruct-2507
Qwen3-Next-80B-A3B-Instruct
Qwen3-Next-80B-A3B-Instruct
Qwen3-Next-80B-A3B-Instruct
2
Qwen3-Next-80B-A3B-Instruct
Qwen3-30B-A3B-Instruct-2507
Qwen3-32B
Qwen3-32B
3
Olmo-3.1-32B-Instruct
Qwen3-32B
Mistral-Small-3.2-24B-Instruct-2506
Qwen3-30B-A3B-Instruct-2507
4
gemma-3-12b-it
Olmo-3.1-32B-Instruct
Meta-Llama-3-70B-Instruct
Meta-Llama-3-70B-Instruct
5
Meta-Llama-3-70B-Instruct
Meta-Llama-3-70B-Instruct
gemma-3-12b-it
Qwen3-14B
Table 1: Top-5 LLMs evaluated by BenchHub in real-world application scenarios.
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: User Interface of BenchHub Web Demo.
Figure 9: Average number of duplicates included in the sampling size when using the embedding-based method (Blue) and random sampling (Orange).
Hyperparameter
Value
Sequence Length
8,192
Learning Rate
2×10−5
Global Batch (Effective)
256
Learning Rate Scheduler
Cosine Decay
Warmup Ratio
0.05
Training Epochs
3
Appendix
Table 2: SFT configuration details.
Sample-level Attribute
Accuracy
Subject
0.871
Skill
0.967
Cultural-specificity
0.986
Appendix
Table 3: Accuracy of fine-tuned categorizer on Qwen-2.5-7b.
Noise rate
0%
5%
10%
15%
20%
25%
30%
Random
65.07
66.07 ± 0.02
67.11 ± 0.02
68.11 ± 0.01
69.11 ± 0.01
70.10 ± 0.03
71.08 ± 0.02
Systematic
65.07
65.70 ± 0.03
66.33 ± 0.02
67.01 ± 0.04
67.65 ± 0.04
68.37 ± 0.03
68.97 ± 0.08
Appendix
Table 4: Proportion (%) of instances outside the advertised scope under random and systematic label noise.
Noise type
0%
5%
10%
15%
20%
25%
30%
Random
1.000
0.995
0.991
0.979
0.975
0.964
0.956
Systematic
1.000
0.997
0.988
0.980
0.974
0.970
0.960
Appendix
Table 5: Spearman’s ρ between original and perturbed model rankings, averaged across the six subject domains.
Language
G-MMLU
M-MMLU
Arabic
0.765
0.767
German
0.789
0.833
Indonesian
0.800
0.808
Kyrgyz
0.681
–
Nepali
0.709
–
Dutch
0.804
–
Appendix
Table 6: Categorizer accuracy on G-MMLU and M-MMLU.
Dataset
Reference
Cultural Specificity
Lang.
# of Samples
License
ARC
[ 10 ]
General
EN
3,548
cc-by-sa 4.0
SocialIQA
[ 63 ]
General
EN
1,954
CC-BY-4.0
WinoGrande
[ 62 ]
General
EN
1,767
Apache-2.0
Natural Questions (open)
[ 33 ]
General
EN
1,769
Apache-2.0
NarrativeQA
[ 30 ]
General
EN
10,557
Apache-2.0
TruthfulQA
[ 38 ]
General
EN
817
Apache-2.0
Appendix
Table 7: Benchmarks Included in BenchHub .
Dataset
Reference
Cultural Specificity
# of Samples
License
Language: AR
G-MMLU
[ 67 ]
General/Local
14,042
apache-2.0
ArabLegalEval
[ 20 ]
Local
15,311
-
ArabicMMLU
[ 31 ]
General/Local
14,455
cc-by-nc-sa-4.0
Language: DE
G-MMLU
[ 67 ]
General/Local
14,042
apache-2.0
Appendix
Table 8: Benchmarks Included in BenchHub -multilingual.
Figure 10: Fine-grained subject distribution of all datasets used in BenchHub .
Figure 11: Subject distribution of MMLU series in English, Korean, Japanese, Indonesian, and Chinese, respectively.
Figure 12: Skill and cultural specificity distributions of the Holistic benchmarks and MMLU series in English, Korean, Japanese, Indonesian, and Chinese.
Figure 13: Subject distribution by benchmarks in korean benchmarks (%, p(subject∣benchmark) ).
Figure 14: Cross-benchmark overlap at cosine thresholds τ∈{0.70,0.80,0.90} . Rows index the source benchmark A ; columns index the target benchmark B . Cell (A,B) reports the fraction of items in A whose nearest neighbor in B has cosine similarity at least τ .
Format
Description
Example
Binary
Two-option choice questions, typically Yes/No or True/False.
“Is the Earth flat?” → “No”
Multiple-choice QA (MCQA)
Multiple-choice question answering format.
“What is the capital of France? (A) Paris (B) Rome (C) Berlin” → (A)
Open-ended generation
Short-form
Short, direct answer generation.
“What is 2+2?” → “4”
Free-form
Extended, explanatory answer generation.
“Explain the theory of relativity.” → “The theory of relativity, developed by Albert Einstein…”
Open-ended
Answer generation without a definitive correct response, involving free imagination or storytelling.
“Tell a story about a journey to the moon.” → “Once upon a time, a brave astronaut set off on a magical voyage beyond the stars…”
Comparison
Subjective or preference-based evaluation of responses, typically involving agreement, helpfulness, or safety.
“Which response is more helpful?” A: “Yes.” B: “Sure, here’s how you can do that…” → (B)
Appendix
Table 9: Problem format types, descriptions, and examples.
Benchmark
Entity Being Evaluated
Problem Format
ToolHop [ 82 ]
Final answer
Short-form
ToolQA [ 87 ]
Final answer
Short-form
GPT4Tools [ 81 ]
Tool calling within output reasoning
Open-ended
ToolBench [ 15 ]
Tool calling
Short-form
Appendix
Table 10: Tool usage datasets, entity being evaluated, and problem format.
Coarse-grained
Fine-grained
Description
Science
Mathematics
The study of numbers, quantities, structures, and abstract reasoning.
Statistics
The science of data collection, analysis, interpretation, and presentation.
Physics
The study of matter, energy, and the fundamental forces of nature.
Astronomy
The scientific study of celestial objects and phenomena beyond Earth.
Chemistry
The study of substances, their properties, and how they interact and change.
Biology
The study of living organisms and their vital processes.
Appendix
Table 11: Subject types and descriptions.
Skill
Description
Example
Knowledge
Questions that seek factual information, definitions, or explanations. Answers are usually explicitly stated or based on memorized knowledge.
"What is the capital of France?"
Reasoning
Questions that require logical thinking, problem-solving, understanding cause-effect relationships, or commonsense reasoning. Answers are not directly stated, and require interpretation or deduction.
“There are 12 apples in a box. If 4 people share them equally, how many apples does each person get?”
Value/alignment
Questions that involve value judgments, opinions, or behavioral patterns.
"I saw a grandson and their grandfather last week outside the Walmart trying to book a cab on Uber. Who was not comfortable using the phone?" (Example from [ 50 ] )
Appendix
Table 12: Skill types, descriptions, and examples.
target
Description
General
A general target without a specific cultural or national focus.
Local
A specific target toward a certain culture ( e.g ., US, KO).
Appendix
Table 13: Cultural specificity types and descriptions.
Model
% boxed
% rule-based
% other
Gemma3-4b-it
2.01
54.60
43.40
Gemma3-12b-it
93.40
5.01
1.59
Meta-Llama-3-70B-Instruct
92.51
0.00
7.49
Ministral-8B-Instruct-2410
68.94
16.23
14.84
Mistral-Small-3.2-24B-Instruct-2506
93.14
0.06
6.80
Mixtral-8x7B-Instruct-v0.1
98.81
0.56
0.64
Appendix
Table 14: Per-response extraction source on Korean datasets (each column sums to 100%). Other : no boxed substring and no rule-based parse.
Model
Before
After
gemma-3-4b-it
97.99%
1.14%
Olmo-3-1025-7B
66.77%
14.81%
Olmo-3-1125-32B
12.64%
9.28%
Appendix
Table 15: % of extraction-failure cases before and after improving the Korean extraction procedure.
Rank
(c) STEM knowledge evaluation
(d) Math teaching agent
(e) Counseling agent
1
Qwen3-Next-80B-A3B-Instruct
Qwen3-Next-80B-A3B-Instruct
Qwen3-32B
2
Qwen3-30B-A3B-Instruct-2507
Qwen3-32B
Qwen3-Next-80B-A3B-Instruct
3
Qwen3-32B
Qwen3-30B-A3B-Instruct-2507
Qwen3-30B-A3B-Instruct-2507
4
Qwen3-14B
Qwen3-14B
Qwen3-14B
5
Meta-Llama-3-70B-Instruct
Qwen3-4B
Qwen3-8B
Appendix
Table 16: Top 5 LLMs evaluated by customized BenchHub across three scenarios.
Figure 15: LLM evaluation rankings using BenchHub (Korean).
Model
Subject Type
Target Type
Task Type
Art & Sports
Culture
HASS
Science
Social Intel.
Tech.
Cultural
General
Knowledge
Reasoning
Value/Align.
Meta-Llama-3-70B-Instruct
54.1
62.9
68.7
72.4
54.6
70.1
60.0
68.4
72.7
56.6
63.2
Ministral-8B-Instruct-2410
30.6
38.9
37.6
49.6
27.5
44.2
36.9
41.1
43.2
39.1
29.2
Mistral-Small-3.2-24B-Instruct-2506
61.1
63.6
68.2
71.4
47.7
66.6
65.0
66.4
73.8
53.2
53.5
Mixtral-8x7B-Instruct-v0.1
58.8
59.0
63.0
65.3
45.9
60.9
54.6
61.9
68.1
47.6
53.7
OLMo-3-1025-7B
15.0
22.3
31.5
42.6
26.1
38.0
10.1
34.9
34.7
27.8
30.0
Appendix
Table 17: English accuracy (%) by model and category. Best performance in each category is bolded.
Model
Subject Type
Target Type
Task Type
Art & Sports
Culture
HASS
Science
Social Intel.
Tech.
Cultural
General
Knowledge
Reasoning
Value/Align.
Meta-Llama-3-70B-Instruct
68.8
49.9
61.1
51.6
77.7
53.4
58.5
86.0
53.7
46.1
83.2
Ministral-8B-Instruct-2410
61.1
46.2
49.1
23.9
53.9
25.4
42.4
75.4
41.3
20.4
57.9
Mistral-Small-3.2-24B-Instruct-2506
68.5
52.7
63.3
48.1
68.6
47.8
56.9
75.4
54.5
42.0
73.8
Mixtral-8x7B-Instruct-v0.1
65.0
47.7
53.7
35.0
64.1
36.3
48.9
75.0
46.1
31.3
68.6
OLMo-3-1025-7B
23.5
18.4
20.3
9.9
41.0
12.1
21.3
67.3
16.7
11.8
43.6
Appendix
Table 18: Korean accuracy (%) by model and category. Best performance in each category is bolded.
Category
Meta-Llama-3-70B-Instruct
Ministral-8B-Instruct-2410
Mistral-Small-3.2-24B-Instruct-2506
Mixtral-8x22B-Instruct-v0.1
Mixtral-8x7B-Instruct-v0.1
OLMo-3-1025-7B
OLMo-3-1125-32B
OLMo-3.1-32B-Instruct
Qwen3-14B
Qwen3-30B-A3B-Instruct-2507
Qwen3-32B
Qwen3-4B
Qwen3-8B
Qwen3-Next-80B-A3B-Instruct
Gemma3-12b-it
Gemma3-4b-it
Art & Sports
Architecture
81.5
64.8
72.2
83.3
74.1
37.0
57.4
77.8
63.0
66.7
64.8
48.1
55.6
74.1
64.8
53.7
Fashion
59.1
54.5
59.1
59.1
59.1
22.7
59.1
63.6
59.1
50.0
50.0
63.6
50.0
72.7
68.2
72.7
Festivals
62.5
50.0
68.8
62.5
56.2
12.5
50.0
56.2
50.0
68.8
62.5
43.8
56.2
75.0
81.2
62.5
Music
38.3
17.7
56.2
54.9
63.0
13.0
24.2
46.2
34.5
50.8
42.7
30.2
36.4
60.9
43.2
24.2
Painting
84.6
48.1
94.2
80.8
92.3
48.1
67.3
86.5
78.8
86.5
88.5
82.7
80.8
92.3
78.8
67.3
Appendix
Table 19: English accuracy (%) by model and fine-grained subject category. Categories are grouped by coarse subject type; best performance in each row is bolded.
Category
Meta-Llama-3-70B-Instruct
Ministral-8B-Instruct-2410
Mistral-Small-3.2-24B-Instruct-2506
Mixtral-8x7B-Instruct-v0.1
OLMo-3-1025-7B
OLMo-3-1125-32B
OLMo-3.1-32B-Instruct
Qwen3-14B
Qwen3-30B-A3B-Instruct-2507
Qwen3-32B
Qwen3-4B
Qwen3-8B
Qwen3-Next-80B-A3B-Instruct
Gemma3-12b-it
Gemma3-4b-it
Art & Sports
Architecture
58.4
28.2
54.6
34.5
10.9
34.0
37.4
50.8
60.9
58.4
45.0
47.5
65.1
45.4
17.6
Fashion
43.8
17.6
37.1
33.3
7.6
24.8
29.5
41.9
50.5
51.4
33.3
36.2
53.3
39.5
23.3
Festivals
97.3
97.3
97.3
97.3
27.0
97.3
97.3
97.3
97.3
97.3
97.3
97.3
97.3
97.3
13.5
Music
89.7
85.3
90.6
88.2
28.3
85.0
87.6
90.6
90.6
91.2
86.4
88.2
92.6
89.7
18.3
Painting
91.4
89.4
96.7
93.3
38.5
92.0
91.0
93.7
94.1
96.7
91.4
92.7
96.0
93.1
24.4
Appendix
Table 20: Korean accuracy (%) by model and fine-grained subject category. Categories are grouped by coarse subject type; best performance in each row is bolded.
Model
Random
Stratified
Chatbot Arena
MixEval
Qwen3-4B
0.600
0.559
0.592
0.577
Qwen3-8B
0.624
0.577
0.601
0.598
Qwen3-14B
0.662
0.609
0.638
0.635
Qwen3-32B
0.683
0.642
0.672
0.660
Qwen3-30B-A3B-Instruct-2507
0.726
0.693
0.744
0.707
Qwen3-Next-80B-A3B-Instruct
0.734
0.704
0.727
0.720
Appendix
Table 21: Evaluation results of 15 open LLMs in English across four different sampling strategies.
Model
Random
Stratified
Chatbot Arena
MixEval
Qwen3-4B
0.557
0.550
0.497
0.533
Qwen3-8B
0.577
0.570
0.521
0.550
Qwen3-14B
0.617
0.607
0.566
0.595
Qwen3-32B
0.671
0.666
0.633
0.650
Qwen3-30B-A3B-Instruct-2507
0.642
0.639
0.606
0.623
Qwen3-Next-80B-A3B-Instruct
0.706
0.705
0.673
0.684
Appendix
Table 22: Evaluation results of 15 open LLMs in Korean across four different sampling strategies.
Model
(a)
(b)
(c)
(d)
(e)
Qwen3-4B
0.553
0.677
0.555
0.413
0.593
Qwen3-8B
0.556
0.690
0.584
0.401
0.637
Qwen3-14B
0.580
0.704
0.629
0.420
0.658
Qwen3-32B
0.605
0.795
0.687
0.468
0.684
Qwen3-30B-A3B-Instruct-2507
0.707
0.739
0.703
0.451
0.665
Qwen3-Next-80B-A3B-Instruct
0.698
0.796
0.717
0.469
0.678
Appendix
Table 23: Evaluation results of 15 open LLMs using customized BenchHub across five use cases.
Rank
BenchHub (Ours)
MATH [ 19 ]
1
Qwen/Qwen3-8B (0.35)
OLMoE-Contaminated (0.84)
2
gemma-3-4b-it (0.30)
Qwen/Qwen3-8B (0.31)
3
Llama-3.1-8B-Instruct (0.26)
gemma-3-4b-it (0.28)
4
OLMoE-Contaminated (0.22)
Llama-3.1-8B-Instruct (0.13)
5
OLMoE-base (0.16)
OLMoE-base (0.10)
Appendix
Table 24: Model ranking and accuracy across BenchHub customized for math evaluation and MATH [ 19 ] for simulation study on data contamination
Benchmark scores increasingly influence the development, marketing, and selection of large language models (LLMs). Yet an overall score is interpretable only in relation to the system tested, the questions included, and the conditions of evaluation. This perspective examines five connected limitations of general LLM rankings: differences between evaluated and publicly available systems; commercial incentives and dependencies in external evaluation; benchmark saturation, defective tests, and data contamination; models exploiting scoring procedures; and the limited relevance of general scores to users' tasks. Documented cases illustrate why these problems require different responses. I argue for evaluation procedures that disclose the tested configuration, validate questions and successful task completion, report performance alongside cost and execution time, and make the scope of generalization explicit. I then discuss Isotanta, a crowdsourced benchmarking platform, as a practical example of contributed questions and repeated evaluation. A larger question pool may improve task coverage, while repeated sampling can improve the stability of estimates on that pool; neither guarantees validity or personalization. The paper distinguishes the platform's current shared ranking from proposed task-specific and user-provided evaluations. Its central argument is that model selection requires evidence about performance on the intended work, not simply a high position on a general leaderboard.
Benchmark datasets are central to evaluating Large Language Models (LLMs), yet they are typically conceived as monolithic tasks, obscuring substantial variation in the demands of individual samples. We introduce a dataset-centric meta-evaluation framework that audits benchmark datasets at the sample level along five latent dimensions: 1. Cognitive and Knowledge Demands, 2. Language and Content Quality, 3. Task Properties, 4. Context, and 5. Ethics, Safety, and Fairness. Applying this framework, we annotate five influential benchmarks -- MMLU, ARC, WinoGrande, HellaSwag, and TruthfulQA -- revealing pronounced internal heterogeneity that is not captured by aggregate accuracy scores. We show how these annotations enable criterion-driven orchestration of composite benchmark subsets across datasets, supporting targeted evaluation of model capabilities such as Reasoning Depth or Ethical Sensitivity. This approach reframes benchmark evaluation as dataset introspection, providing a principled methodology for analyzing and re-composing existing benchmarks to better reflect diverse evaluation needs.
Do language model benchmarks actually measure what practitioners intend them to ? High-level metadata is too coarse to convey the granular reality of benchmarks: a "poetry" benchmark may never test for haikus, while "instruction-following" benchmarks will often test for an arbitrary mix of skills. This opacity makes verifying alignment with practitioner goals a laborious process, risking an illusion of competence even when models fail on untested facets of user interests. We introduce BenchBrowser, a retriever that surfaces evaluation items relevant to natural language use cases over 20 benchmark suites. Validated by a human study confirming high retrieval precision, BenchBrowser generates evidence to help practitioners diagnose low content validity (narrow coverage of a capability's facets) and low convergent validity (lack of stable rankings when measuring the same capability). BenchBrowser, thus, helps quantify a critical gap between practitioner intent and what benchmarks actually test.
Language Technologies Institute, Carnegie Mellon University, Pittsburgh (PA), United States · Thomas Lord Department of Computer Science, University of Southern California, Los Angeles (CA), United States