How to Evaluate When You Don't Know How to Evaluate? Choosing Reliable Setups for Base Language Model Benchmarking

Shenbin Qian; David Samuel; Jindřich Helcl; Andrey Kutuzov; Stephan Oepen

cs.CLOct 13, 2026

How to Evaluate When You Don't Know How to Evaluate? Choosing Reliable Setups for Base Language Model Benchmarking

Authors: Shenbin Qian, David Samuel, Jindřich Helcl, Andrey Kutuzov, Stephan Oepen

Organizations: Department of Informatics, University of Oslo

Abstract

Evaluation of large language models (LLMs) remains largely pre-scientific: it involves many design decisions, with little guidance on how to make them. These choices can substantially affect measured performance and, consequently, decisions about subsequent pretraining. To identify effective evaluation setups, we investigate 8 aspects, such as task formulation or the number of in-context examples. We compare the setups using three widely-used signal criteria, repurposed as normative meta-measures: monotonicity, non-randomness, and ranking consistency of the evaluation scores across intermediate checkpoints. In an early pretraining scenario, we evaluate these setups on 26 monolingual 2B-parameter models trained on different types of data, using 80 tasks in four languages. We also examine the setups on three larger LLMs ( $\geq$ 7B) in the same four languages and explore how the preferred setups change with model scale. Our results yield concrete recommendations: some choices, such as the use of multiple prompts, are consistently useful, whereas others, such as task formulation, depend on model scale. More generally, our framework serves as a recipe for the choice of evaluation setups in new scenarios.

Figures & tables

Figure 1: Average score of six Norwegian 2B models, pretrained on different corpora under two evaluation setups for NorEval ( Mikhailov et al., 2025 ) classification tasks. Both use 5-shot prompting and the maximum over multiple prompts. Setup A : random-baseline rescaling, mean over task categories, cloze formulation (CF), no log-probability normalization, accuracy. Setup B : z -score rescaling, mean over tasks, multiple-choice formulation (MCF), character normalization, answer probability. At the end of training, the two setups rank the models largely in reverse order (Kendall’s

\tau=-0.6

). MADLAD-400 ranks highest under setup A but lowest under setup B, whereas the two machine-translated corpora, OPUS and Tower-9B, shift from the bottom to the top of the ranking. The choice of evaluation setup can lead to different conclusions about which training corpus is preferable.

	CF	MCF	Hybrid
Context $x$	Question: {question} Answer:	Question: {question} A: {answer 1} … D: {answer 4} Answer:	Question: {question} - {answer 1} … - {answer 4} Answer:
Options $a_{i}$	{answer 1}, …, {answer 4}	A, B, C, D	{answer 1}, …, {answer 4}

Table 1: Example prompt templates and answer options for the three task formulations of a multiple-choice task.

Figure 2: Per-option win rates of the 2B models (circle ) and larger LLMs (diamond ) pooled over languages and task types; win rates are computed per meta-measure, then averaged. Higher win rate (red) is better. The decoding aspect is only available for the 2B models.

Tier	Aspect (top option)	2B	Larger
High-impact	Metric (AnsProb)	4/4	3/4
	Logprob. Norm. (char)	3/4	3/4
	Shots (5-shot)	4/4	4/4
	Formul. (CF)	2/2	1/2
Low-impact	Prompts (multi-max)	0/4	1/4
	Task agg. (task-level mean)	2/4	3/4

Table 2: Number of languages in which all three setup choice methods select the option with the highest pooled win rate ( Figure 2 ). For rescaling, the top option is baseline for the 2B models and none for the larger LLMs. Formulation: Finnish and Norwegian; decoding: Norwegian, 2B models only.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Aspect	Options	Finnish	French	Norwegian	Spanish
Number of shots	0-shot, 5-shot	10	11	14	11
Prompts	single, multi-mean, multi-max	10	11	14	11
Rescaling	baseline, z -score, min-max, none	10	11	14	11
Task aggregation	task-level mean, category mean	10	11	14	11
Formulation	CF / MCF / hybrid	10 / 10 / –	–	14 / 14 / 14	–
Accuracy	none / char / PMI	10 / 10 / 10	11 / 11 / –	14 / 14 / 14	11 / 11 / 5

Appendix

Table C.1: Options of each evaluation aspect in classification tasks and the number of tasks per language on which they are evaluated (–: unavailable). Where the number of tasks differs between options, the numbers are separated by slashes in the order of the options. Metric and log-probability normalization are given as one row per metric (Accuracy, CondProb, AnsProb) with the three normalizations. Finnish and Norwegian setups cover the multiple-choice tasks with formulation variants (10 of 12 and 14 of 17), while French and Spanish have no formulation variants. Generation tasks (omitted in the table) are compared for the 18 Norwegian tasks only, with both options of greedy decoding and random sampling.

		Pooled			Finnish			French			Norwegian			Spanish
		B	O	W	B	O	W	B	O	W	B	O	W	B	O	W
Metric	2B	AP	AP	AP	AP	AP	AP	AP	AP	AP	AP	AP	AP	AP	AP	AP
	Larger	AP	AP	AP	AP	AP	AP	AP	AP	AP	AP	AP	Acc	AP	AP	AP
Norm.	2B	char	char	char	char	char	char	char	char	char	char	char	char	char	char	PMI
	Larger	char	char	char	char	char	none	char	char	char	char	char	char	char	char	char
Shots	2B	5	5	5	5	5	5	5	5	5	5	5	5	5	5	5

Appendix

Table D.1: Options selected by the three methods (B: best complete setup, O: option ranking, W: per-option win rate) for classification tasks, pooled over languages and per language, for the 2B models and the larger LLMs. Bold : the method selects a different option than the best complete setup. Grey: the larger LLMs get a different option than the 2B models from the same method. AP: answer probability, CP: conditional answer probability, Acc: accuracy; 5, 0: number of shots; max, mean, sgl: multi-max, multi-mean and single prompt; base, z, mm: baseline, z -score and min-max rescaling; mean: task-level mean, cat: category mean; hyb: hybrid; –: no choice (no formulation variants).

Aspect	Option	2B	Larger
Metric	AnsProb	0.77	0.61
	CondProb †	0.51	0.50
	Acc	0.38	0.46
Norm.	char	0.59	0.58
	PMI	0.41	0.32
	none	0.41	0.50

Appendix

Table D.2: Option-ranking scores (mean overall score of all setups with the option, over all combinations of the other aspects), averaged over languages, for the 2B models and the larger LLMs. Options are ordered by the 2B model scores. Bold marks the highest score per aspect and model scale, before rounding into two decimals. † Norwegian only. ‡ Finnish and Norwegian.

Figure D.1: Per-option win rates of the 2B models (circle ) and larger LLMs (diamond ) pooled over languages and task types; win rates are computed for each meta-measure are visualized in the three columns: monotonicity, non-randomness and ranking consistency.

Aspect	Mono	NonRand	RankCons
Classification tasks
Formulation	CF 97	CF 69	CF 63
Metric	AnsProb 92	AnsProb 95	AnsProb 92
Normalization	char 64	PMI 66	none 57
Shots	5-shot 84	5-shot 61	5-shot 63
Prompts	multi-mean 67	multi-max 82	tie (53)

Appendix

Table E.1: Option with the highest win rate (%) per meta-measure, pooled over languages, for the 2B models.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,330 papers

Anticipatory Evaluation of Language Models: Predicting LLM Benchmark Scores from Descriptions
Sep 25, 2025Jungsoo Park, Ethan Mendes, Gabriel Stanovsky +1Language Model Performance Prediction LLM Evaluation
Georgia Institute of Technology, Atlanta, Georgia · The Hebrew University of Jerusalem, Jerusalem, Israel
Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleading
Apr 30, 2026Nicholas Sadjoli, Tim Siefken, Atin Ghosh +2LLM Evaluation Language Model Generation Evaluation
SAP · Stanford University
Do Not Trust the Benchmark: Limitations of General LLM Rankings and a Case for Task-Specific Evaluation
Sep 19, 2026Danial AminLLM Evaluation Benchmark Validity

Sep 25, 2025J/K move · Enter open · S save

Anticipatory Evaluation of Language Models: Predicting LLM Benchmark Scores from Descriptions

Jungsoo Park, Ethan Mendes, Gabriel Stanovsky, Alan Ritter

Georgia Institute of Technology, Atlanta, Georgia · The Hebrew University of Jerusalem, Jerusalem, Israel

Language Model Performance Prediction LLM Evaluation

Open paper AskPDF arXiv

Modern ML research increasingly faces an evaluation bottleneck: building benchmarks and running experiments is costly, slowing iteration. We ask how much can be inferred about evaluation outcomes before experiments, given only a natural-language description of the task and setup. We call this setting anticipatory evaluation. We introduce EVALCAST, a literature-grounded resource of 2,290 description-performance pairs from 1,519 arXiv papers, including a test set of papers published after the model's knowledge cutoff. We evaluate two tasks: predicting absolute performance through regression and comparing benchmark difficulty through pairwise prediction. Experiments with frontier LLMs show that accurate regression remains challenging, while pairwise predictions can recover relative performance ordering under matched settings. Comparable performance on post-cutoff and streaming evaluations suggests that the observed signal is not solely attributable to previously reported results. Overall, EVALCAST provides a resource and evaluation framework for studying anticipatory evaluation and its potential use in prioritizing benchmarks.

Abstract

LLM Evaluation

Explore similar work

Anticipatory Evaluation of Language Models: Predicting LLM Benchmark Scores from Descriptions

Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleading

Do Not Trust the Benchmark: Limitations of General LLM Rankings and a Case for Task-Specific Evaluation