Generating and Evaluating Sustainable Procurement Criteria for the Swiss Public Sector using In-Context Prompting with Large Language Models

Yingqiang Gao; Veton Matoshi; Luca Rolshoven; Tilia Ellendorff; Judith Binder; Jeremy Austin Jann; Gerold Schneider; Matthias Stürmer

cs.SEMar 23, 2026

Generating and Evaluating Sustainable Procurement Criteria for the Swiss Public Sector using In-Context Prompting with Large Language Models

Authors: Yingqiang Gao, Veton Matoshi, Luca Rolshoven, Tilia Ellendorff, Judith Binder, Jeremy Austin Jann, Gerold Schneider, Matthias Stürmer

Organizations: University of Zurich Zurich, Switzerland · Bern University of Applied Sciences Bern, Switzerland · University of Bern Bern, Switzerland

Abstract

Public procurement refers to the process by which public sector institutions, such as governments, municipalities, and publicly funded bodies, acquire goods and services. Swiss law requires the integration of ecological, social, and economic sustainability requirements into tender evaluations in the format of criteria that have to be fulfilled by a bidder. However, translating high-level sustainability regulations into concrete, verifiable, and sector-specific procurement criteria (such as selection criteria, award criteria, and technical specifications) remains a labor-intensive and error-prone manual task, requiring substantial domain expertise in several groups of goods and services and considerable manual effort. This paper presents a configurable, LLM-assisted pipeline that is presented as a software supporting the systematic generation and evaluation of sustainability-oriented procurement criteria catalogs for Switzerland. The system integrates in-context prompting, interchangeable LLM backends, and automated output validation to enable auditable criteria generation across different procurement sectors. As a proof of concept, we instantiate the pipeline using official sustainability guidelines published by the Swiss government and the European Commission, which are ingested as structured reference documents. We evaluate the system through a combination of automated quality checks, including an LLM-based evaluation component, and expert comparison against a manually curated gold standard. Our results demonstrate that the proposed pipeline can substantially reduce manual drafting effort while producing criteria catalogs that are consistent with official guidelines. We further discuss system limitations, failure modes, and design trade-offs observed during deployment, highlighting key considerations for integrating generative AI into public sector software workflows.

Figures & tables

Figure 1. Writing sustainable procurement criteria for the Swiss public sector is traditionally a labor-intensive process, requiring substantial human effort and specialized expertise. Our approach leverages state-of-the-art large language models as domain experts to automatically generate sustainable procurement criteria in an end-to-end manner, significantly reducing time and cognitive workload. We highlight the generation of award criteria as an example.

Figure 2. End-to-end pipeline of SwissSPC for sustainable procurement criteria generation and evaluation. SwissSPC integrates API configuration and model selection, prompt templating with optional document references, and XLSX-based criteria generation, followed by LLM-as-a-Judge evaluation to assess generation quality along multiple dimensions.

AA	AA-ID	C-ID	CC	SPC	AL: Basis	AL: Good Practice	AL: Exemplary
Chemical management	AA-1	C-01	TS	The supplier must implement a chemical management system that minimizes the use of hazardous substances. Evidence must be provided through system descriptions and inspection reports.	Proof of a chemical management system.	Proof of a chemical management system with annual review and update.	Proof of a chemical management system with complete elimination of substances of very high concern (SVHC).
Wood origin and certification	AA-2	C-02	TS	The wood used must come from verifiably sustainable forestry. Certificates such as FSC or PEFC must be provided as proof. The traceability of the wood supply chain must be demonstrated through appropriate documentation.	At least 70% of the wood used is FSC or PEFC certified.	At least 90% of the wood used is FSC or PEFC certified; complete supply chain documentation.	100% of the wood used is FSC or PEFC certified; additionally, proof of chain of custody for all suppliers.

Table 1. Structure of an LLM-generated SPC table (other rows are dropped for compact demonstration). Abbreviations: AA (Area of Action), C-ID(Criterion-ID), CC (Criterion Category), SPC (Sustainable Procurement Criterion), AL (Ambition Level).

Figure 3. A snippet of the structured prompt used in SwissSPC (translated from German to English) for generating SPCs . The prompt is decomposed into semantically distinct components: Background, Problem, Task, Requirement, Inputs, etc. (the full prompt contains 875 words), to explicitly encode sector context, motivation, generation objectives, and output constraints.

Sector	EU-GPP	Toolbox	Expert	# Area	# Criteria
NT	✓	✗	✓	15	79
FN	✓	✓	✓	26	60
LM	✓	✓	✓	52	101
TS	✓	✓	✓	69	106
CS	✓	✓	✓	48	144
PS	✓	✓	✓	36	60

Table 2. Collected sources for sustainable procurement criteria which serve as references to the LLM in-context learning.

Figure 4. SwissSPC user interface for AI-assisted generation of Sustainable Procurement Criteria. The workflow guides users through (1) generation settings such as model and sector selection, (2) in-context prompting via configurable prompt templates, and (3) optional upload of credited reference PDFs, producing structured, sector-specific SPCs catalogs exported as XLSX files.

Sector	Model	AA (Area of Action)			SPC (Sustainable Procurement Criteria)
Sector	Model	Coverage	Quality	Harmonic	Coverage	Quality	Harmonic
NT	gpt-4.1	0.6329	0.2159	0.3220	0.6359	0.6540	0.6433
	gpt-4o	0.1519	0.7013	0.2497	0.1519	0.6648	0.2473
	o4-mini	0.5063	0.2770	0.3581	0.5063	0.6465	0.5679
FN	gpt-4.1	0.8333	0.3892	0.5305	0.8333	0.6619	0.7378
	gpt-4o	0.2500	0.8435	0.3857	0.2500	0.7514	0.3752

Table 3. Automatic evaluation results. Generated tables are evaluated against reference sources, with results based on EU-GPP recommendations shown in blue and those based on Toolbox in red . Cases with identical performance across both sources are marked in black. For each instance, only the reference source yielding the better performance is reported.

Figure 5. LLM-as-a-Judge evaluation scores for SPCs . We report the better performing scores using either EU-GPP or Toolbox when generating the outputs. Abbreviations: PR (Precision & Relevance), CC (Completeness & Coverage), DS (Differentiation & Scalability), LF (Language & Formality).

Language Model Generation Evaluation

Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.

See all 395 papers

From Global Benchmarks to Local Evaluations: Benchmarking LLMs for the German Public Sector
Aug 18, 2026Camilla Dalerci, Thilo Michael, Robin Schaefer +1LLM Evaluation
Innovations Department, Bundesdruckerei GmbH, Berlin, Germany
MÖVE: A Holistic LLM Benchmark for the German Public Sector
Jun 11, 2026Camilla Dalerci, Thilo Michael, Robin Schaefer +1Multilingual Language Model Evaluation LLM Evaluation
Innovations Department, Bundesdruckerei GmbH, Berlin, Germany.
A Cascaded Unsupervised-Supervised NLP Pipeline for Detecting Accusatory Language in Public Procurement
Aug 12, 2026Bryan Torres, Daniel Riofrío, José Vega-Sánchez +4Text Clustering
Colegio de Ciencias e Ingenierías “El Politécnico”, Universidad San Francisco de Quito (USFQ), Diego de Robles S/N, Quito 170157 · Faculty of Engineering and Applied Sciences, Telecommunications Engineering, ETEL Research Group, Universidad de Las Américas (UDLA), 170503 Quito, Ecuador · Departamento de Estudios Organizacionales y Desarrollo Humano, Escuela Politécnica Nacional, Quito 170525, Ecuador +1

Aug 18, 2026J/K move · Enter open · S save

From Global Benchmarks to Local Evaluations: Benchmarking LLMs for the German Public Sector

Camilla Dalerci, Thilo Michael, Robin Schaefer, Daniel Weinland

Innovations Department, Bundesdruckerei GmbH, Berlin, Germany

LLM Evaluation

Open paper AskPDF arXiv

Public institutions face a persistent challenge in selecting LLMs suited to their specific context. Existing benchmarks, however, are of limited use as they primarily reflect English-language and US-centric settings, and often only evaluate task performance. In this paper, we present first results of MÖVE, a holistic evaluation framework for the German public sector, examining three rarely considered governance dimensions: energy consumption, provider transparency, and knowledge of German-party positions. Our results reveal significant trade-offs, with no single model excelling across all dimensions: estimated energy consumption varies more than 60-fold and is not explained by model size alone, information disclosure varies systematically across providers, and European models do not exhibit stronger knowledge of German party positions. Model selection for public institutions thus cannot rely on performance rankings alone. Instead, evaluations should also reflect the governance requirements of the deployment context.

Abstract

Language Model Generation Evaluation

Explore similar work

From Global Benchmarks to Local Evaluations: Benchmarking LLMs for the German Public Sector

MÖVE: A Holistic LLM Benchmark for the German Public Sector

A Cascaded Unsupervised-Supervised NLP Pipeline for Detecting Accusatory Language in Public Procurement