From Global Benchmarks to Local Evaluations: Benchmarking LLMs for the German Public Sector

Camilla Dalerci; Thilo Michael; Robin Schaefer; Daniel Weinland

cs.CLAug 18, 2026

From Global Benchmarks to Local Evaluations: Benchmarking LLMs for the German Public Sector

Authors: Camilla Dalerci, Thilo Michael, Robin Schaefer, Daniel Weinland

Organizations: Innovations Department, Bundesdruckerei GmbH, Berlin, Germany

Abstract

Public institutions face a persistent challenge in selecting LLMs suited to their specific context. Existing benchmarks, however, are of limited use as they primarily reflect English-language and US-centric settings, and often only evaluate task performance. In this paper, we present first results of MÖVE, a holistic evaluation framework for the German public sector, examining three rarely considered governance dimensions: energy consumption, provider transparency, and knowledge of German-party positions. Our results reveal significant trade-offs, with no single model excelling across all dimensions: estimated energy consumption varies more than 60-fold and is not explained by model size alone, information disclosure varies systematically across providers, and European models do not exhibit stronger knowledge of German party positions. Model selection for public institutions thus cannot rely on performance rankings alone. Instead, evaluations should also reflect the governance requirements of the deployment context.

Figures & tables

Dataset	Task	Type	Size
Eur-Lex-Sum Aumiller et al. (2022)	Summarization	Existing	850
Swiss Leading Decision Summarization Rasiah et al. (2023)	Summarization	Existing	1,530
KIKC Summary	Summarization	Private	40
German Ministry Publications (Summaries)	Summarization	Private	1,530
German-QuAD Möller et al. (2021)	QA	Existing	4,710
KIKC QA	QA	Private	72

Table 1: Overview of datasets used for calculating energy consumption during model evaluation. Type indicates whether a dataset is pre-existing or specifically constructed for the benchmark. The latter are not released to the public. Dataset size is reported with respect to the evaluation unit used in each task, e.g., summaries in summarization tasks.

Figure 1: Mean transparency score (%) by domain for EU AI Act Code of Practice signatories vs. non-signatories.

Rank	Model	Organisation	Region	Size	Score
1	DeepSeek R1	DeepSeek	China	Large	0.671
1	GPT-4o Mini	OpenAI	USA	Small	0.671
3	GPT-OSS 120B	OpenAI	USA	Large	0.667
4	Llama 3.3 70B	Meta	USA	Large	0.663
5	Mistral Small 3.1	Mistral	Europe	Medium	0.645
6	GPT-4o	OpenAI	USA	Large	0.624

Table 2: Top-10 models for party-position classification, ranked by mean accuracy. All top-performing models surpass the majority baseline.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Transparency scores for all 39 models, sorted by domain and total score. Each segment represents the score in one of the seven transparency documentation domains. The pronounced gap between well-documented domains (Model Identification, Architecture, Distribution & Access) and poorly-documented ones (Use & Deployment, Training & Data, Compute & Energy) is visible across nearly all models.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

MÖVE: A Holistic LLM Benchmark for the German Public Sector
Jun 11, 2026Camilla Dalerci, Thilo Michael, Robin Schaefer +1Multilingual Language Model Evaluation LLM Evaluation
Innovations Department, Bundesdruckerei GmbH, Berlin, Germany.
From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch
Aug 10, 2026Laurens Samson, Iva Gornishka, Gossa Lô +2Multilingual Language Model Evaluation LLM Evaluation
City of Amsterdam · Socially Intelligent Artificial Systems Group, University of Amsterdam · Fundamental AI Lab, University of Technology Nuremberg
Do Not Trust the Benchmark: Limitations of General LLM Rankings and a Case for Task-Specific Evaluation
Sep 19, 2026Danial AminLLM Evaluation Benchmark Validity

Jun 11, 2026J/K move · Enter open · S save

MÖVE: A Holistic LLM Benchmark for the German Public Sector

Camilla Dalerci, Thilo Michael, Robin Schaefer, Daniel Weinland

Innovations Department, Bundesdruckerei GmbH, Berlin, Germany.

Multilingual Language Model Evaluation LLM Evaluation LLM Auditing Language Modeling

Open paper AskPDF arXiv

We present MÖVE (Modelle für die Öffentliche Verwaltung Evaluieren), a holistic benchmark for evaluating large language models (LLMs) in the context of the German public sector. While LLMs are increasingly adopted in public administration, model selection remains largely ad hoc, and existing benchmarks offer limited guidance: they are predominantly English-centric, US-centric in content, and focus exclusively on task performance. MÖVE addresses these gaps by evaluating 39 models across two complementary dimensions. Performance criteria cover summarization, question answering, and topic extraction. Governance criteria assess hallucination tendencies, energy consumption, provider transparency, and alignment with German constitutional values and knowledge about positions by German political parties. In total, we utilize ten German-language datasets, including gold- and silverstandard datasets that we constructed to reflect public-administration domains. We employ a multi-metric evaluation strategy combining classical NLP metrics, embedding-based methods, and LLM-as-a-judge approaches. Our results show that no single model dominates across all criteria: top performers differ between tasks, and model size alone is a poor predictor of quality. We further evaluate the benchmark itself, analyzing its statistical precision, LLM judge reliability, the impact of our private datasets on model rankings, the sensitivity of our results to prompt formulation, and the validity of our energy consumption estimates. MÖVE is designed as a living benchmark under active development; results are publicly available at https://moeve.bundesdruckerei.de/.

Abstract

LLM Evaluation

Explore similar work

MÖVE: A Holistic LLM Benchmark for the German Public Sector

From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch

Do Not Trust the Benchmark: Limitations of General LLM Rankings and a Case for Task-Specific Evaluation