Swiss-Bench SBP-002: A Frontier Model Comparison on Swiss Legal and Regulatory Tasks

Fatih Uenal

cs.CLMar 24, 2026

Swiss-Bench SBP-002: A Frontier Model Comparison on Swiss Legal and Regulatory Tasks

Authors: Fatih Uenal

Organizations: University of Colorado Boulder

Abstract

While recent work has benchmarked large language models on Swiss legal translation (Niklaus et al., 2025) and academic legal reasoning from university exams (Fan et al., 2025), no existing benchmark evaluates frontier model performance on applied Swiss regulatory compliance tasks. I introduce Swiss-Bench SBP-002, a trilingual benchmark of 395 expert-crafted items spanning three Swiss regulatory domains (FINMA, Legal-CH, EFK), seven task types, and three languages (German, French, Italian), and evaluate ten frontier models from March 2026 using a structured three-dimension scoring framework assessed via a blind three-judge LLM panel (GPT-4o, Claude Sonnet 4, Qwen3-235B) with majority-vote aggregation and weighted kappa = 0.605, with reference answers validated by an independent human legal expert on a 100-item subset (73% rated Correct, 0% Incorrect, perfect Legal Accuracy). Results reveal three descriptive performance clusters: Tier A (35-38% correct), Tier B (26-29%), and Tier C (13-21%). The benchmark proves difficult: even the top-ranked model (Qwen 3.5 Plus) achieves only 38.2% correct, with 47.3% incorrect and 14.4% partially correct. Task type difficulty varies widely: legal translation and case analysis yield 69-72% correct rates, while regulatory Q&A, hallucination detection, and gap analysis remain below 9%. Within this roster (seven open-weight, three closed-source), an open-weight model leads the ranking, and several open-weight models match or outperform their closed-source counterparts. These findings provide an initial empirical reference point for assessing frontier model capability on Swiss regulatory tasks under zero-retrieval conditions.

Figures & tables

Domain	DE	FR	IT	Total
FINMA	80	62	36	178
Legal-CH	40	74	55	169
EFK (Reg. QA)	30	12	6	48
Total	150	148	97	395

Table 1: Swiss-Bench v3 dataset composition (395 main items). 18 English experimental items excluded from main analysis.

Model	Provider	Category	API Identifier
Claude Sonnet 4	Anthropic	Closed-source	anthropic/claude-sonnet-4
Gemini 2.5 Flash	Google	Closed-source	google/gemini-2.5-flash
GPT-4o	OpenAI	Closed-source	openai/gpt-4o
GLM 5	Zhipu AI	Open-weight	z-ai/glm-5
Qwen 3.5 Plus	Alibaba	Open-weight	qwen/qwen3.5-plus-02-15
MiMo-V2-Flash	Xiaomi	Open-weight	xiaomi/mimo-v2-flash

Table 2: Models evaluated in SBP-002.

Figure 1: Inter-rater reliability improvement from grade-mode to structured scoring. Dashed lines indicate Landis & Koch interpretation thresholds. Structured scoring moves the panel average from moderate ( $\kappa=0.512$ ) to substantial ( $\kappa=0.605$ ) agreement.

Rank	Model	Category	C%	95% CI	P%	I%	Tier
1	Qwen 3.5 Plus	Open-weight	38.2	[33.6, 43.1]	14.4	47.3	A
2	Gemini 2.5 Flash	Closed	35.4	[30.9, 40.3]	16.5	48.1	A
3	GLM 5	Open-weight	28.6	[24.4, 33.3]	14.7	56.7	B
4	Claude Sonnet 4	Closed	26.1	[22.0, 30.6]	14.7	59.2	B
5	MiniMax M2.5	Open-weight	20.8	[17.1, 25.0]	13.7	65.6	C
6	MiMo-V2-Flash	Open-weight	19.0	[15.4, 23.1]	10.1	70.9	C

Table 3: Overall model rankings by correct rate (C%) on 395 items with 95% Wilson score confidence intervals. P = partially correct, I = incorrect. Tier assignment based on natural breaks in the C% distribution.

Figure 2: Overall model ranking by correct rate on 395 items. Colors indicate tier: dark (A), medium (B), light (C).

Model	FINMA	Legal-CH	EFK (Reg. QA)
Qwen 3.5 Plus	21.9	63.9	8.3
Gemini 2.5 Flash	16.3	62.7	10.4
GLM 5	11.2	53.3	6.2
Claude Sonnet 4	8.4	49.7	8.3
MiniMax M2.5	4.5	40.8	10.4
MiMo-V2-Flash	2.2	40.8	4.2

Table 4: Correct rate (C%) by model and regulatory domain.

Figure 3: Correct rate by regulatory domain. Legal-CH is much easier than FINMA and EFK (Reg. QA) across all models. Note: EFK results are confounded with the Regulatory Q&A task type (all 48 EFK items belong to this single task type).

Model	Reg.	Halluc.	Gap	Jurisd.	Stat.	Case	Legal
	Q&A	Det.	Anal.	Discr.	Interp.	Anal.	Transl.
Qwen 3.5 Plus	13.5	30.2	16.9	69.0	30.4	88.6	76.7
Gemini 2.5 Flash	13.5	15.9	16.9	65.5	28.3	94.3	73.3
GLM 5	9.6	9.5	11.9	58.6	15.2	82.9	66.7
Claude Sonnet 4	9.6	6.3	8.5	41.4	17.4	74.3	86.7
MiniMax M2.5	8.7	4.8	1.7	31.0	8.7	71.4	73.3

Table 5: Correct rate (C%) by task type. Models sorted by overall rank.

Figure 4: Heatmap of correct rate (%) by model and task type. Case analysis and legal translation are easiest; regulatory Q&A and gap analysis are hardest.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model	API Identifier	Access
Claude Sonnet 4	anthropic/claude-sonnet-4	OpenRouter
Gemini 2.5 Flash	google/gemini-2.5-flash	Google AI API
GPT-4o	openai/gpt-4o	OpenRouter
GLM 5	z-ai/glm-5	OpenRouter
Qwen 3.5 Plus	qwen/qwen3.5-plus-02-15	OpenRouter
MiMo-V2-Flash	xiaomi/mimo-v2-flash	OpenRouter

Appendix

Table 6: Model identifiers and access details.

Domain	Task Type	DE	FR	IT	Total
FINMA	Regulatory Q&A	35	13	8	56
	Hallucination Det.	25	24	14	63
	Gap Analysis	20	25	14	59
Legal-CH	Jurisdiction Discr.	20	24	14	58
	Statutory Interp.	10	25	11	46
	Case Analysis	10	15	10	35

Appendix

Table 7: Item counts by domain, task type, and language.

Figure 5: Correct rate by language (descriptive only). Italian items yield higher C% across all models, but this is heavily confounded by compositional bias: 30.9% of Italian items are case analysis or legal translation vs. 6.7% for German. See Section 5 and Table 7 .

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Apr 7, 2026

Swiss-Bench 003: Evaluating LLM Reliability and Adversarial Security for Swiss Regulatory Contexts

The deployment of large language models (LLMs) in Swiss financial and regulatory contexts demands empirical evidence of both production reliability and adversarial security, dimensions not jointly operationalized in existing Swiss-focused evaluation frameworks. This paper introduces Swiss-Bench 003 (SBP-003), extending the HAAS (Helvetic AI Assessment Score) from six to eight dimensions by adding D7 (Self-Graded Reliability Proxy) and D8 (Adversarial Security). I evaluate ten frontier models across 808 Swiss-specific items in four languages (German, French, Italian, English), comprising seven Swiss-adapted benchmarks (Swiss TruthfulQA, Swiss IFEval, Swiss SimpleQA, Swiss NIAH, Swiss PII-Scope, System Prompt Leakage, and Swiss German Comprehension) targeting FINMA Guidance 08/2024, the revised Federal Act on Data Protection (nDSG), and OWASP Top 10 for LLMs. Self-graded D7 scores (73-94%) exceed externally judged D8 security scores (20-61%) by a wide margin, though these dimensions use non-comparable scoring regimes. System prompt leakage resistance ranges from 24.8% to 88.2%, while PII extraction defense remains weak (14-42%) across all models. Qwen 3.5 Plus achieves the highest self-graded D7 score (94.4%), while GPT-oss 120B achieves the highest D8 score (60.7%) despite being the lowest-cost model evaluated. All evaluations are zero-shot under provider default settings; D7 is self-graded and does not constitute independently validated accuracy. I provide conceptual mapping tables relating benchmark dimensions to FINMA model validation requirements, nDSG data protection obligations, and OWASP LLM risk categories.

Fatih Uenal

Department of Computer Science, University of Colorado Boulder

LLM Evaluation LLM Reliability Language Model Safety Evaluation Adversarial Robustness of Language Models Multilingual Language Model Evaluation Privacy Leakage in Language Models

Ask PDF

May 28, 2026

Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions

Legal NLP benchmarks overwhelmingly evaluate a single language or aggregate tasks that differ fundamentally across jurisdictions, making cross-lingual comparison impossible. We introduce Multi-Legal-Bench, the first cross-jurisdictional legal benchmark that evaluates identical tasks across six countries (Ukraine, France, Netherlands, Poland, Czech Republic, Lithuania), four language families, and 134 million court decisions. The benchmark defines five tasks court-type classification, judgment form classification, case-outcome prediction, legal norm extraction, and cause category prediction mapped to structured metadata from national court registries, forming a deliberately sparse 5x6 task-jurisdiction matrix (20 of 30 cells filled). We evaluate 7 frontier LLMs under zero-shot and 3-shot prompting via AWS Bedrock, with 4 additional small/medium models (3-12B) for scaling analysis. Our results reveal that: (1) task-dependent few-shot effects discovered in Ukrainian replicate across all jurisdictions; (2) no single model dominates any language rankings shift with both task and jurisdiction; (3) cross-lingual few-shot transfer does not follow language proximity: UA->FR (Romance, -2.1 pp) transfers better than UA->PL (Slavic, -13.7 pp), with label-set alignment predicting transfer quality better than language family; and (4) tokenizer fertility, despite a 2.3x spread, does not significantly predict cross-lingual accuracy (r=-0.27, p=0.14), suggesting that model architecture and pretraining data dominate tokenizer efficiency. We release all data, prompts, and model predictions.

Volodymyr Ovcharov

SecondLayer

Multilingual Language Model Evaluation Legal NLP LLM Evaluation Cross-Lingual Transfer

Ask PDF

Aug 5, 2026

EuroExec: Frontier Language Models Fall Short of Expert Judgment on European Executive Decision Tasks

Frontier LLMs are increasingly put to use on open-ended complex questions, different in nature from the ones they are typically evaluated on. We dedicate more than 4,000 human expert hours to evaluate a selection of six frontier LLMs on a member of this class of problems: EuroExec, our introduced human expert-based benchmark composed of 413 open-ended long-form European executive tasks authored by 47 vetted domain experts, each question drawn from experience in a real case. Every response is manually evaluated through a multi-attribute rubric, an item-specific checklist of requirements, and a preference rank ordering, extracting an aggregate metric "Solve Rate". The strongest model solves only 56.9% of tasks, while expert-written reference answers judged blindly are solved at near-ceiling levels and are preferred over every model response in 74% of direct rankings, placing frontier generative systems well below the professional standard of work they are already used for. We see that the best way to extract this kind of conclusion is by employing human evaluators, carefully checking their consistency through rigorous statistical analysis, and observe that automatic measurements also fall short when evaluating on this case of real-world open-ended problems with a subjective ground truth.

Pau Arnal, Khaled Denfir, Danylo Smahliuk +2

Sovrano AI

Open-Ended Generation LLM Evaluation Language Model Generation Evaluation LLM Decision-Making

Ask PDF

Abstract

LLM Evaluation

Explore similar work

Swiss-Bench 003: Evaluating LLM Reliability and Adversarial Security for Swiss Regulatory Contexts

Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions

EuroExec: Frontier Language Models Fall Short of Expert Judgment on European Executive Decision Tasks