Swiss-Bench SBP-002: A Frontier Model Comparison on Swiss Legal and Regulatory Tasks

Fatih Uenal

cs.CLMar 24, 2026

Swiss-Bench SBP-002: A Frontier Model Comparison on Swiss Legal and Regulatory Tasks

Authors: Fatih Uenal

Organizations: University of Colorado Boulder

Abstract

While recent work has benchmarked large language models on Swiss legal translation (Niklaus et al., 2025) and academic legal reasoning from university exams (Fan et al., 2025), no existing benchmark evaluates frontier model performance on applied Swiss regulatory compliance tasks. I introduce Swiss-Bench SBP-002, a trilingual benchmark of 395 expert-crafted items spanning three Swiss regulatory domains (FINMA, Legal-CH, EFK), seven task types, and three languages (German, French, Italian), and evaluate ten frontier models from March 2026 using a structured three-dimension scoring framework assessed via a blind three-judge LLM panel (GPT-4o, Claude Sonnet 4, Qwen3-235B) with majority-vote aggregation and weighted kappa = 0.605, with reference answers validated by an independent human legal expert on a 100-item subset (73% rated Correct, 0% Incorrect, perfect Legal Accuracy). Results reveal three descriptive performance clusters: Tier A (35-38% correct), Tier B (26-29%), and Tier C (13-21%). The benchmark proves difficult: even the top-ranked model (Qwen 3.5 Plus) achieves only 38.2% correct, with 47.3% incorrect and 14.4% partially correct. Task type difficulty varies widely: legal translation and case analysis yield 69-72% correct rates, while regulatory Q&A, hallucination detection, and gap analysis remain below 9%. Within this roster (seven open-weight, three closed-source), an open-weight model leads the ranking, and several open-weight models match or outperform their closed-source counterparts. These findings provide an initial empirical reference point for assessing frontier model capability on Swiss regulatory tasks under zero-retrieval conditions.

Figures & tables

Domain	DE	FR	IT	Total
FINMA	80	62	36	178
Legal-CH	40	74	55	169
EFK (Reg. QA)	30	12	6	48
Total	150	148	97	395

Table 1: Swiss-Bench v3 dataset composition (395 main items). 18 English experimental items excluded from main analysis.

Model	Provider	Category	API Identifier
Claude Sonnet 4	Anthropic	Closed-source	anthropic/claude-sonnet-4
Gemini 2.5 Flash	Google	Closed-source	google/gemini-2.5-flash
GPT-4o	OpenAI	Closed-source	openai/gpt-4o
GLM 5	Zhipu AI	Open-weight	z-ai/glm-5
Qwen 3.5 Plus	Alibaba	Open-weight	qwen/qwen3.5-plus-02-15
MiMo-V2-Flash	Xiaomi	Open-weight	xiaomi/mimo-v2-flash

Table 2: Models evaluated in SBP-002.

Figure 1: Inter-rater reliability improvement from grade-mode to structured scoring. Dashed lines indicate Landis & Koch interpretation thresholds. Structured scoring moves the panel average from moderate ( $\kappa=0.512$ ) to substantial ( $\kappa=0.605$ ) agreement.

Rank	Model	Category	C%	95% CI	P%	I%	Tier
1	Qwen 3.5 Plus	Open-weight	38.2	[33.6, 43.1]	14.4	47.3	A
2	Gemini 2.5 Flash	Closed	35.4	[30.9, 40.3]	16.5	48.1	A
3	GLM 5	Open-weight	28.6	[24.4, 33.3]	14.7	56.7	B
4	Claude Sonnet 4	Closed	26.1	[22.0, 30.6]	14.7	59.2	B
5	MiniMax M2.5	Open-weight	20.8	[17.1, 25.0]	13.7	65.6	C
6	MiMo-V2-Flash	Open-weight	19.0	[15.4, 23.1]	10.1	70.9	C

Table 3: Overall model rankings by correct rate (C%) on 395 items with 95% Wilson score confidence intervals. P = partially correct, I = incorrect. Tier assignment based on natural breaks in the C% distribution.

Figure 2: Overall model ranking by correct rate on 395 items. Colors indicate tier: dark (A), medium (B), light (C).

Model	FINMA	Legal-CH	EFK (Reg. QA)
Qwen 3.5 Plus	21.9	63.9	8.3
Gemini 2.5 Flash	16.3	62.7	10.4
GLM 5	11.2	53.3	6.2
Claude Sonnet 4	8.4	49.7	8.3
MiniMax M2.5	4.5	40.8	10.4
MiMo-V2-Flash	2.2	40.8	4.2

Table 4: Correct rate (C%) by model and regulatory domain.

Figure 3: Correct rate by regulatory domain. Legal-CH is much easier than FINMA and EFK (Reg. QA) across all models. Note: EFK results are confounded with the Regulatory Q&A task type (all 48 EFK items belong to this single task type).

Model	Reg.	Halluc.	Gap	Jurisd.	Stat.	Case	Legal
	Q&A	Det.	Anal.	Discr.	Interp.	Anal.	Transl.
Qwen 3.5 Plus	13.5	30.2	16.9	69.0	30.4	88.6	76.7
Gemini 2.5 Flash	13.5	15.9	16.9	65.5	28.3	94.3	73.3
GLM 5	9.6	9.5	11.9	58.6	15.2	82.9	66.7
Claude Sonnet 4	9.6	6.3	8.5	41.4	17.4	74.3	86.7
MiniMax M2.5	8.7	4.8	1.7	31.0	8.7	71.4	73.3

Table 5: Correct rate (C%) by task type. Models sorted by overall rank.

Figure 4: Heatmap of correct rate (%) by model and task type. Case analysis and legal translation are easiest; regulatory Q&A and gap analysis are hardest.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model	API Identifier	Access
Claude Sonnet 4	anthropic/claude-sonnet-4	OpenRouter
Gemini 2.5 Flash	google/gemini-2.5-flash	Google AI API
GPT-4o	openai/gpt-4o	OpenRouter
GLM 5	z-ai/glm-5	OpenRouter
Qwen 3.5 Plus	qwen/qwen3.5-plus-02-15	OpenRouter
MiMo-V2-Flash	xiaomi/mimo-v2-flash	OpenRouter

Appendix

Table 6: Model identifiers and access details.

Domain	Task Type	DE	FR	IT	Total
FINMA	Regulatory Q&A	35	13	8	56
	Hallucination Det.	25	24	14	63
	Gap Analysis	20	25	14	59
Legal-CH	Jurisdiction Discr.	20	24	14	58
	Statutory Interp.	10	25	11	46
	Case Analysis	10	15	10	35

Appendix

Table 7: Item counts by domain, task type, and language.

Figure 5: Correct rate by language (descriptive only). Italian items yield higher C% across all models, but this is heavily confounded by compositional bias: 30.9% of Italian items are case analysis or legal translation vs. 6.7% for German. See Section 5 and Table 7 .

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Swiss-Bench 003: Evaluating LLM Reliability and Adversarial Security for Swiss Regulatory Contexts
Apr 7, 2026Fatih UenalLLM Evaluation LLM Reliability
Department of Computer Science, University of Colorado Boulder
Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions
May 28, 2026Volodymyr OvcharovMultilingual Language Model Evaluation Legal NLP
SecondLayer
EuroExec: Frontier Language Models Fall Short of Expert Judgment on European Executive Decision Tasks
Aug 5, 2026Pau Arnal, Khaled Denfir, Danylo Smahliuk +2Open-Ended Generation LLM Evaluation
Sovrano AI

Apr 7, 2026J/K move · Enter open · S save

Swiss-Bench 003: Evaluating LLM Reliability and Adversarial Security for Swiss Regulatory Contexts

Fatih Uenal

Department of Computer Science, University of Colorado Boulder

LLM Evaluation LLM Reliability Language Model Safety Evaluation Adversarial Robustness of Language Models Multilingual Language Model Evaluation Privacy Leakage in Language Models

Open paper AskPDF arXiv

The deployment of large language models (LLMs) in Swiss financial and regulatory contexts demands empirical evidence of both production reliability and adversarial security, dimensions not jointly operationalized in existing Swiss-focused evaluation frameworks. This paper introduces Swiss-Bench 003 (SBP-003), extending the HAAS (Helvetic AI Assessment Score) from six to eight dimensions by adding D7 (Self-Graded Reliability Proxy) and D8 (Adversarial Security). I evaluate ten frontier models across 808 Swiss-specific items in four languages (German, French, Italian, English), comprising seven Swiss-adapted benchmarks (Swiss TruthfulQA, Swiss IFEval, Swiss SimpleQA, Swiss NIAH, Swiss PII-Scope, System Prompt Leakage, and Swiss German Comprehension) targeting FINMA Guidance 08/2024, the revised Federal Act on Data Protection (nDSG), and OWASP Top 10 for LLMs. Self-graded D7 scores (73-94%) exceed externally judged D8 security scores (20-61%) by a wide margin, though these dimensions use non-comparable scoring regimes. System prompt leakage resistance ranges from 24.8% to 88.2%, while PII extraction defense remains weak (14-42%) across all models. Qwen 3.5 Plus achieves the highest self-graded D7 score (94.4%), while GPT-oss 120B achieves the highest D8 score (60.7%) despite being the lowest-cost model evaluated. All evaluations are zero-shot under provider default settings; D7 is self-graded and does not constitute independently validated accuracy. I provide conceptual mapping tables relating benchmark dimensions to FINMA model validation requirements, nDSG data protection obligations, and OWASP LLM risk categories.

Abstract

LLM Evaluation

Explore similar work

Swiss-Bench 003: Evaluating LLM Reliability and Adversarial Security for Swiss Regulatory Contexts

Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions

EuroExec: Frontier Language Models Fall Short of Expert Judgment on European Executive Decision Tasks