cs.SEMar 23, 2026

Generating and Evaluating Sustainable Procurement Criteria for the Swiss Public Sector using In-Context Prompting with Large Language Models

Authors: Yingqiang Gao, Veton Matoshi, Luca Rolshoven, Tilia Ellendorff, Judith Binder, Jeremy Austin Jann, Gerold Schneider, Matthias Stürmer

Organizations: University of Zurich Zurich, Switzerland · Bern University of Applied Sciences Bern, Switzerland · University of Bern Bern, Switzerland

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Public procurement refers to the process by which public sector institutions, such as governments, municipalities, and publicly funded bodies, acquire goods and services. Swiss law requires the integration of ecological, social, and economic sustainability requirements into tender evaluations in the format of criteria that have to be fulfilled by a bidder. However, translating high-level sustainability regulations into concrete, verifiable, and sector-specific procurement criteria (such as selection criteria, award criteria, and technical specifications) remains a labor-intensive and error-prone manual task, requiring substantial domain expertise in several groups of goods and services and considerable manual effort. This paper presents a configurable, LLM-assisted pipeline that is presented as a software supporting the systematic generation and evaluation of sustainability-oriented procurement criteria catalogs for Switzerland. The system integrates in-context prompting, interchangeable LLM backends, and automated output validation to enable auditable criteria generation across different procurement sectors. As a proof of concept, we instantiate the pipeline using official sustainability guidelines published by the Swiss government and the European Commission, which are ingested as structured reference documents. We evaluate the system through a combination of automated quality checks, including an LLM-based evaluation component, and expert comparison against a manually curated gold standard. Our results demonstrate that the proposed pipeline can substantially reduce manual drafting effort while producing criteria catalogs that are consistent with official guidelines. We further discuss system limitations, failure modes, and design trade-offs observed during deployment, highlighting key considerations for integrating generative AI into public sector software workflows.

Figures & tables

Figure 1. Writing sustainable procurement criteria for the Swiss public sector is traditionally a labor-intensive process, requiring substantial human effort and specialized expertise. Our approach leverages state-of-the-art large language models as domain experts to automatically generate sustainable procurement criteria in an end-to-end manner, significantly reducing time and cognitive workload. We highlight the generation of award criteria as an example.
Figure 2. End-to-end pipeline of SwissSPC for sustainable procurement criteria generation and evaluation. SwissSPC integrates API configuration and model selection, prompt templating with optional document references, and XLSX-based criteria generation, followed by LLM-as-a-Judge evaluation to assess generation quality along multiple dimensions.
AAAA-IDC-IDCCSPCAL: BasisAL: Good PracticeAL: Exemplary
Chemical managementAA-1C-01TSThe supplier must implement a chemical management system that minimizes the use of hazardous substances. Evidence must be provided through system descriptions and inspection reports.Proof of a chemical management system.Proof of a chemical management system with annual review and update.Proof of a chemical management system with complete elimination of substances of very high concern (SVHC).
Wood origin and certificationAA-2C-02TSThe wood used must come from verifiably sustainable forestry. Certificates such as FSC or PEFC must be provided as proof. The traceability of the wood supply chain must be demonstrated through appropriate documentation.At least 70% of the wood used is FSC or PEFC certified.At least 90% of the wood used is FSC or PEFC certified; complete supply chain documentation.100% of the wood used is FSC or PEFC certified; additionally, proof of chain of custody for all suppliers.
Table 1. Structure of an LLM-generated SPC table (other rows are dropped for compact demonstration). Abbreviations: AA (Area of Action), C-ID(Criterion-ID), CC (Criterion Category), SPC (Sustainable Procurement Criterion), AL (Ambition Level).
Figure 3. A snippet of the structured prompt used in SwissSPC (translated from German to English) for generating SPCs . The prompt is decomposed into semantically distinct components: Background, Problem, Task, Requirement, Inputs, etc. (the full prompt contains 875 words), to explicitly encode sector context, motivation, generation objectives, and output constraints.
SectorEU-GPPToolboxExpert# Area# Criteria
NT✓✗✓1579
FN✓✓✓2660
LM✓✓✓52101
TS✓✓✓69106
CS✓✓✓48144
PS✓✓✓3660
Table 2. Collected sources for sustainable procurement criteria which serve as references to the LLM in-context learning.
Figure 4. SwissSPC user interface for AI-assisted generation of Sustainable Procurement Criteria. The workflow guides users through (1) generation settings such as model and sector selection, (2) in-context prompting via configurable prompt templates, and (3) optional upload of credited reference PDFs, producing structured, sector-specific SPCs catalogs exported as XLSX files.
SectorModelAA (Area of Action) SPC (Sustainable Procurement Criteria)
CoverageQualityHarmonic CoverageQualityHarmonic
NTgpt-4.10.63290.21590.3220 0.63590.65400.6433
gpt-4o0.15190.70130.2497 0.15190.66480.2473
o4-mini0.50630.27700.3581 0.50630.64650.5679
FNgpt-4.10.83330.38920.5305 0.83330.66190.7378
gpt-4o0.25000.84350.3857 0.25000.75140.3752
Table 3. Automatic evaluation results. Generated tables are evaluated against reference sources, with results based on EU-GPP recommendations shown in blue and those based on Toolbox in red . Cases with identical performance across both sources are marked in black. For each instance, only the reference source yielding the better performance is reported.
Figure 5. LLM-as-a-Judge evaluation scores for SPCs . We report the better performing scores using either EU-GPP or Toolbox when generating the outputs. Abbreviations: PR (Precision & Relevance), CC (Completeness & Coverage), DS (Differentiation & Scalability), LF (Language & Formality).
More in this topic

Language Model Generation Evaluation

Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.

See all 395 papers

Explore similar work

CardsList
Aug 18, 2026

From Global Benchmarks to Local Evaluations: Benchmarking LLMs for the German Public Sector

Public institutions face a persistent challenge in selecting LLMs suited to their specific context. Existing benchmarks, however, are of limited use as they primarily reflect English-language and US-centric settings, and often only evaluate task performance. In this paper, we present first results of MÖVE, a holistic evaluation framework for the German public sector, examining three rarely considered governance dimensions: energy consumption, provider transparency, and knowledge of German-party positions. Our results reveal significant trade-offs, with no single model excelling across all dimensions: estimated energy consumption varies more than 60-fold and is not explained by model size alone, information disclosure varies systematically across providers, and European models do not exhibit stronger knowledge of German party positions. Model selection for public institutions thus cannot rely on performance rankings alone. Instead, evaluations should also reflect the governance requirements of the deployment context.

Camilla Dalerci, Thilo Michael, Robin Schaefer +1

Innovations Department, Bundesdruckerei GmbH, Berlin, Germany

LLM Evaluation
AskPDF
Jun 11, 2026

MÖVE: A Holistic LLM Benchmark for the German Public Sector

We present MÖVE (Modelle für die Öffentliche Verwaltung Evaluieren), a holistic benchmark for evaluating large language models (LLMs) in the context of the German public sector. While LLMs are increasingly adopted in public administration, model selection remains largely ad hoc, and existing benchmarks offer limited guidance: they are predominantly English-centric, US-centric in content, and focus exclusively on task performance. MÖVE addresses these gaps by evaluating 39 models across two complementary dimensions. Performance criteria cover summarization, question answering, and topic extraction. Governance criteria assess hallucination tendencies, energy consumption, provider transparency, and alignment with German constitutional values and knowledge about positions by German political parties. In total, we utilize ten German-language datasets, including gold- and silverstandard datasets that we constructed to reflect public-administration domains. We employ a multi-metric evaluation strategy combining classical NLP metrics, embedding-based methods, and LLM-as-a-judge approaches. Our results show that no single model dominates across all criteria: top performers differ between tasks, and model size alone is a poor predictor of quality. We further evaluate the benchmark itself, analyzing its statistical precision, LLM judge reliability, the impact of our private datasets on model rankings, the sensitivity of our results to prompt formulation, and the validity of our energy consumption estimates. MÖVE is designed as a living benchmark under active development; results are publicly available at https://moeve.bundesdruckerei.de/.

Camilla Dalerci, Thilo Michael, Robin Schaefer +1

Innovations Department, Bundesdruckerei GmbH, Berlin, Germany.

Multilingual Language Model EvaluationLLM EvaluationLLM AuditingLanguage Modeling
AskPDF
Aug 12, 2026

A Cascaded Unsupervised-Supervised NLP Pipeline for Detecting Accusatory Language in Public Procurement

Public procurement involves the allocation of substantial financial resources; therefore, continuous oversight through audits, controls, and monitoring mechanisms is essential. However, stakeholder comments and publicly available government data are often underutilized, despite their potential to reveal procedural irregularities. To address this gap, this paper analyzes metadata from Ecuador's Sistema Oficial de Contratación Pública (SOCE, Official Public Procurement System), with particular emphasis on participant comments generated during the pre-contractual phase. We propose a hybrid modeling framework that integrates unsupervised clustering and supervised classification within a natural language processing (NLP) pipeline to uncover latent patterns and detect potentially irregular procurement processes. Semantic embeddings are generated using Word2Vec, LLaMA, and RoBERTa, followed by Gaussian Mixture Models (GMMs) for unsupervised clustering. A supervised classification stage is then applied to identify accusatory or whistleblowing-style comments. Experimental results show that the combination of domain-trained Word2Vec embeddings, GMM-based clustering, and a Random Forest classifier achieves high precision and recall, even under severe class imbalance. These findings demonstrate that lightweight, domain-adapted NLP architectures can effectively support risk identification and enhance transparency in public procurement systems without requiring large-scale computational infrastructure.

Bryan Torres, Daniel Riofrío, José Vega-Sánchez +4

Colegio de Ciencias e Ingenierías “El Politécnico”, Universidad San Francisco de Quito (USFQ), Diego de Robles S/N, Quito 170157 · Faculty of Engineering and Applied Sciences, Telecommunications Engineering, ETEL Research Group, Universidad de Las Américas (UDLA), 170503 Quito, Ecuador · Departamento de Estudios Organizacionales y Desarrollo Humano, Escuela Politécnica Nacional, Quito 170525, Ecuador +1

Text Clustering
AskPDF