Who Benchmarks the Benchmarks? A Case Study of LLM Evaluation in Icelandic

Finnur Ágúst Ingimundarson; Steinunn Rut Friðriksdóttir; Bjarki Ármannsson; Iris Edda Nowenstein; Steinþór Steingrímsson

cs.CLMar 17, 2026

Who Benchmarks the Benchmarks? A Case Study of LLM Evaluation in Icelandic

Authors: Finnur Ágúst Ingimundarson, Steinunn Rut Friðriksdóttir, Bjarki Ármannsson, Iris Edda Nowenstein, Steinþór Steingrímsson

Organizations: University of Zürich · University of Iceland · The Árni Magnússon Institute for Icelandic Studies

Abstract

This paper evaluates current Large Language Model (LLM) benchmarking for Icelandic, identifies problems, and calls for improved evaluation methods in low/medium-resource languages in particular. We show that benchmarks that include synthetic or machine-translated data that have not been verified in any way, commonly contain severely flawed test examples that are likely to skew the results and undermine the tests' validity. We warn against the use of such methods without verification in low/medium-resource settings as the translation quality can, at best, only be as good as MT quality for a given language at any given time. Indeed, the results of our quantitative error analysis on existing benchmarks for Icelandic show clear differences between human-authored/-translated benchmarks vs. synthetic or machine-translated benchmarks.

Figures & tables

Benchmark	Samples	Total size	Synth	IRR	CI	p-value
ARC-IS (EuroEval)	112	1,116	x	0.39	(0.28, 0.51)	$<.001$
ScaLA-IS	102	1,024		0.52	(0.36, 0.67)	$<.001$
IceEC	250	58,239		0.63	(0.55, 0.71)	$<.001$
IcelandicQA	200	2,000		0.54	(0.39, 0.68)	$<.001$
Belebele-IS	90	900		0.49	(0.36, 0.63)	$<.001$
MultiWikiQA-IS	250	5,004	x	0.71	(0.66, 0.77)	$<.001$

Table 1: Overview of benchmarks, sample size, total size, inter-rater reliability (IRR) scores along with their confidence intervals and p-values

Figure 1: Mean proportion and 95% confidence interval for each of the labels across annotations and benchmarks, arranged by OK proportion in descending order

Ref	Original	MT / Our translation
1	Louis Pasteur created a process that reduced the amount of bacteria in milk. How does this process most likely benefit people?	M: Louis Guðmundsson fann upp aðferð sem dró úr fjölda baktería í mjólk. Hvernig nýtist þessi aðferð fólki líklega best?
2a	Which protist is a bi-flagellate autotroph?	M : Hvaða frumvera er tvíflagella sjálfnærandi lífvera?
2b	-	EE : Hvaða mótefni er bi-flagellate autotroph?
3	Hvers vegna erum við ekki með lögin þannig að hver sem opinberlega hæðist að, rógber, smánar og ógnar manni og svo framvegis án tillits Forseti hringir. til nokkurs skapaðar hlutar sæti refsingu?	Why are our laws not written such that anyone who publicly mocks, defames, humiliates and threatens someone etc., with no regard to Speaker rings. anything at all, is punished?
4	Varðandi smábátahlutfallið í þessu frumvarpi verð ég að segja ég hef ekki skoðað skiptinguna á því mjög „grundigt“.	Regarding the proportion of small fishing boats in this bill , I have to say that I have not looked into the distribution very thoroughly.

Table 2: Examples of flaws in Icelandic benchmarks. For space reasons we use the same column for either machine-translated examples from the benchmarks or, in italics, our translation of Icelandic examples for the foreign reader. ( M : Miðeind, EE : EuroEval.)

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Label proportions per rater across the evaluated benchmarks

Explore similar work

Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation
May 24, 2026Klaudia-Doris Thellmann, Bernhard Stadler, Michael Färber +1Multilingual Language Model Evaluation MT Evaluation
TUD Dresden University of Technology · 2InfAI e.V. · 3Amazon +1
KyrgyzLLM-Bench: Benchmarking Kyrgyz Language Understanding
Jul 19, 2026Timur Turatali, Aida Turdubaeva, Rustem Izmailov +2Multilingual Language Model Evaluation LLM Reliability
1The Cramer Project · Institute of IT, Kyrgyz State Technical University named after I. Razzakov · School of Computer Science, University of Windsor +2
Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets
Feb 25, 2026Hanna Yukhymenko, Anton Alexandrov, Martin VechevMultilingual Language Model Evaluation MT Evaluation
INSAIT, Sofia University "St. Kliment Ohridski" · ETH Zurich

May 24, 2026J/K move · Enter open · S save

Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation

Klaudia-Doris Thellmann, Bernhard Stadler, Michael Färber, Jens Lehmann

TUD Dresden University of Technology · 2InfAI e.V. · 3Amazon+1

Multilingual Language Model Evaluation MT Evaluation

Open paper AskPDF arXiv

Machine-translated benchmarks are widely used to assess the multilingual capabilities of large language models (LLMs), yet translation errors in these benchmarks remain underexplored, raising concerns about the reliability and comparability of multilingual evaluation. We address two practical gaps: (i) how well automatic MQM-style error spans from LLM judges and a span-aware QE baseline (xCOMET-XXL) match expert human span annotations on benchmark translations, and (ii) how strongly translation errors (as opposed to source-side issues in the English original) explain accuracy drops on translated benchmarks. We find that span agreement is non-trivial on naturally occurring benchmark translations, and that target-side translation errors are consistently associated with measurable, percentage-point drops in translated accuracy even after controlling for English correctness and source-side anomalies.

Abstract

Explore similar work

Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation

KyrgyzLLM-Bench: Benchmarking Kyrgyz Language Understanding

Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets