Swiss-Bench 003: Evaluating LLM Reliability and Adversarial Security for Swiss Regulatory Contexts

Fatih Uenal

cs.CRApr 7, 2026

Swiss-Bench 003: Evaluating LLM Reliability and Adversarial Security for Swiss Regulatory Contexts

Authors: Fatih Uenal

Organizations: Department of Computer Science, University of Colorado Boulder

Abstract

The deployment of large language models (LLMs) in Swiss financial and regulatory contexts demands empirical evidence of both production reliability and adversarial security, dimensions not jointly operationalized in existing Swiss-focused evaluation frameworks. This paper introduces Swiss-Bench 003 (SBP-003), extending the HAAS (Helvetic AI Assessment Score) from six to eight dimensions by adding D7 (Self-Graded Reliability Proxy) and D8 (Adversarial Security). I evaluate ten frontier models across 808 Swiss-specific items in four languages (German, French, Italian, English), comprising seven Swiss-adapted benchmarks (Swiss TruthfulQA, Swiss IFEval, Swiss SimpleQA, Swiss NIAH, Swiss PII-Scope, System Prompt Leakage, and Swiss German Comprehension) targeting FINMA Guidance 08/2024, the revised Federal Act on Data Protection (nDSG), and OWASP Top 10 for LLMs. Self-graded D7 scores (73-94%) exceed externally judged D8 security scores (20-61%) by a wide margin, though these dimensions use non-comparable scoring regimes. System prompt leakage resistance ranges from 24.8% to 88.2%, while PII extraction defense remains weak (14-42%) across all models. Qwen 3.5 Plus achieves the highest self-graded D7 score (94.4%), while GPT-oss 120B achieves the highest D8 score (60.7%) despite being the lowest-cost model evaluated. All evaluations are zero-shot under provider default settings; D7 is self-graded and does not constitute independently validated accuracy. I provide conceptual mapping tables relating benchmark dimensions to FINMA model validation requirements, nDSG data protection obligations, and OWASP LLM risk categories.

Figures & tables

	SBP-002	SBP-003 (this paper)
Dimensions	D1–D6 (6 dimensions)	D7–D8 (2 new dimensions)
Tasks	6 Swiss legal tasks	7 Swiss-adapted tasks (all new)
Items	934	808
Languages	DE, FR, IT, EN	DE, FR, IT, EN
Models	10 (partially overlapping)	10 (2026-04 snapshot)
Scoring	LLM judge panel + human	Self-graded D7 + Qwen3 D8

Table 1: SBP-002 vs. SBP-003. The two papers share the HAAS framework but differ in dimensions, tasks, model roster, and scoring. Results cannot be compared directly across papers due to different evaluation dates and model versions.

Task	EN	DE	FR	IT	Total
Swiss TruthfulQA	25	25	25	25	100
Swiss IFEval	13	13	13	—	39
Swiss SimpleQA	60	60	60	30	210
Swiss NIAH	13	13	13	—	39
Swiss PII-Scope	82	82	82	25	271
Sys. Prompt Leakage	40	40	39	—	119

Table 2: Item distribution across Swiss-adapted tasks and languages. Totals: 808 items across 7 tasks in 4 languages.

Profile	Use Case	Heavy Dimensions
$p_{\text{compliant}}$	CROs, CCOs	D4 (25%), D3 (20%), D6 (15%)
$p_{\text{performant}}$	CTOs	D1 (25%), D5 (15%), D7 (15%)
$p_{\text{reliable}}$	Product Owners	D7 (30%), D1 (15%), D5 (15%)
$p_{\text{secure}}$	CISOs	D8 (30%), D2 (15%), D3 (15%)
$p_{\text{integrated}}$	Flagship	Default v2 weights

Table 3: HAAS v2 weight profiles for different stakeholder perspectives. Each profile shifts emphasis to the dimensions most relevant to the target role.

Model	TruthfulQA	IFEval	SimpleQA	NIAH	D7
Qwen 3.5 Plus	96.0	92.3	93.8	94.9	94.4
GLM 5	92.0	89.7	91.9	89.7	90.9
GPT-4o	83.0	87.2	88.1	84.6	85.5
DeepSeek V3.2	78.0	82.1	86.2	84.6	82.0
Claude Sonnet 4	73.0	82.1	90.0	87.2	81.6
MiMo-V2-Flash	75.0	74.4	81.0	100.0	81.0

Table 4: D7 self-graded reliability proxy results (%) across 10 evaluated models. All D7 scores are self-assessed via model_graded_fact . D7 composite is the normalized weighted mean. Bold indicates best per column. IFEval and NIAH (

n

=39 each) are exploratory; pairwise differences under 10pp may not be statistically distinguishable.

Figure 1: D7 self-graded reliability proxy scores across four Swiss-adapted benchmarks.

Figure 2: D8 security heatmap showing PII-Scope and System Prompt Leakage resistance. All models score below 43% on PII extraction defense (left column), while system prompt leakage resistance (right column) varies by over 63 percentage points.

Model	PII-Scope	Sys. Prompt Leak.	D8	Swiss Ger. †
GPT-oss 120B	42.4	88.2	60.7	70.0
GPT-4o	38.7	79.6	55.1	86.7
Qwen 3.5 Plus	40.4	65.8	50.6	96.7
Claude Sonnet 4	38.6	52.5	44.2	96.7
MiniMax M2.5	33.2	60.1	44.0	80.0
GLM 5	24.9	70.6	43.2	93.3

Table 5: D8 Adversarial Security results (%). D8 composite uses PII-Scope (60%) and System Prompt Leakage (40%) only. Swiss German is reported separately as a dialectal robustness measure. PII-Scope uses 3-tier scoring; leakage uses 4-tier scoring. Bold indicates best per column.

Model	D7 [%]	D8 [%]	D7+D8 [%]	$\Delta$
Qwen 3.5 Plus	94.4 [91.2, 97.1]	50.5 [46.2, 54.8]	72.4 [69.8, 75.0]	43.9
GPT-4o	85.5 [80.8, 89.7]	55.1 [51.3, 59.0]	70.3 [67.3, 73.2]	30.3
GPT-oss 120B	75.3 [70.2, 80.1]	60.7 [56.9, 64.6]	68.0 [64.8, 71.1]	14.5
GLM 5	91.3 [87.6, 94.5]	43.2 [39.2, 47.2]	67.2 [64.5, 69.9]	48.1
Claude Sonnet 4	81.7 [76.6, 86.4]	44.1 [39.7, 48.6]	62.9 [59.5, 66.2]	37.5
MiMo-V2-Flash	81.0 [76.2, 85.7]	37.5 [33.4, 41.5]	59.2 [56.1, 62.4]	43.6

Table 6: Exploratory D7+D8 summary with 95% bootstrap confidence intervals (10,000 resamples). D7 is self-graded; D8 is judge-scored (PII-Scope + System Prompt Leakage only; Swiss German excluded). Adjacent pairwise differences overlap; fine-grained rank ordering should not be over-interpreted.

Figure 3: HAAS v2 radar profiles for the top 5 models across all seven Swiss-adapted benchmarks. The “collapse” from D7 (outer ring) to D8 benchmarks (inner ring, particularly PII-Scope and Leakage) is visible for all models.

Figure 4: D7 (self-graded reliability proxy) vs. D8 (adversarial security) scatter plot. Dashed lines indicate median values. GPT-oss 120B occupies a unique position: lowest D7 but highest D8. No model scored near the top of both dimensions in this evaluation.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model	OpenRouter Model ID	Type
Qwen 3.5 Plus	qwen/qwen3.5-plus	Open
GLM 5	zhipu/glm-5	Open
GPT-4o	openai/gpt-4o	Closed
DeepSeek V3.2	deepseek/deepseek-v3.2	Open
Claude Sonnet 4	anthropic/claude-sonnet-4	Closed
MiMo-V2-Flash	xiaomi/mimo-v2-flash	Open

Appendix

Table 7: Model versions and OpenRouter identifiers (inference date: 2026-04-04).

Model	Agree (%)	Self (%)	GPT-4o (%)	$\Delta$ (pp)
Qwen 3.5 Plus	98.3	93.8	92.7	$+$ 1.1
DeepSeek V3.2	97.4	84.8	85.7	$-$ 0.9
GLM 5	96.3	91.6	90.6	$+$ 1.0
Mistral Large 3	95.7	84.0	84.9	$-$ 0.9
MiMo-V2-Flash	92.6	81.2	82.3	$-$ 1.1
Gemini 2.5 Flash	92.4	85.4	90.7	$-$ 5.3

Appendix

Table 8: D7 self-grading validation against GPT-4o external judge. Agreement is the proportion of samples where both judges assigned the same grade. Delta = self-graded accuracy minus GPT-4o accuracy (negative = model is self-critical). GPT-4o and Claude Sonnet 4 have low coverage due to cache ordering and are excluded.

Model	Default	PII strict	PII binary	Leak strict	Leak binary
	1/.5/0 + 1/.75/.25/0	1/.25/0	1/0/0	1/.5/.1/0	1/0/0/0
GPT-oss 120B	60.7	58.8	56.8	60.0	59.1
GPT-4o	55.1	53.1	51.1	51.8	46.4
Qwen 3.5 Plus	50.6	48.6	46.7	47.7	44.1
Claude Sonnet 4	44.2	42.2	40.2	41.4	37.0
MiniMax M2.5	44.0	40.8	37.6	42.1	36.9

Appendix

Table 9: D8 composite (%) under seven rubric weighting schemes. Top-3 (GPT-oss, GPT-4o, Qwen) is stable across all schemes. All Spearman

\rho\geq 0.95

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Swiss-Bench SBP-002: A Frontier Model Comparison on Swiss Legal and Regulatory Tasks
Mar 24, 2026Fatih UenalLLM Evaluation Legal NLP
University of Colorado Boulder
FinSafetyBench: Evaluating LLM Safety in Real-World Financial Scenarios
May 1, 2026Yutao Hou, Yihan Jiang, Yuhan Xie +5Multilingual Language Model Evaluation Language Model Safety Evaluation
Shanghai University of Finance and Economics · Beihang University · Southern University of Science and Technology +1
Schützen: Evaluating LLM Safety in Bulgarian and German Contexts
Jun 9, 2026Kiril Georgiev, Yuxia Wang, Dimitar Iliyanov Dimitrov +2LLM Safety Benchmarks Multilingual Language Model Evaluation
Sofia University "St. Kliment Ohridski" · INSAIT, Sofia University “St. Kliment Ohridski” · MBZUAI

Mar 24, 2026J/K move · Enter open · S save

Swiss-Bench SBP-002: A Frontier Model Comparison on Swiss Legal and Regulatory Tasks

Fatih Uenal

University of Colorado Boulder

LLM Evaluation Legal NLP Multilingual Language Model Evaluation

Open paper AskPDF arXiv

While recent work has benchmarked large language models on Swiss legal translation (Niklaus et al., 2025) and academic legal reasoning from university exams (Fan et al., 2025), no existing benchmark evaluates frontier model performance on applied Swiss regulatory compliance tasks. I introduce Swiss-Bench SBP-002, a trilingual benchmark of 395 expert-crafted items spanning three Swiss regulatory domains (FINMA, Legal-CH, EFK), seven task types, and three languages (German, French, Italian), and evaluate ten frontier models from March 2026 using a structured three-dimension scoring framework assessed via a blind three-judge LLM panel (GPT-4o, Claude Sonnet 4, Qwen3-235B) with majority-vote aggregation and weighted kappa = 0.605, with reference answers validated by an independent human legal expert on a 100-item subset (73% rated Correct, 0% Incorrect, perfect Legal Accuracy). Results reveal three descriptive performance clusters: Tier A (35-38% correct), Tier B (26-29%), and Tier C (13-21%). The benchmark proves difficult: even the top-ranked model (Qwen 3.5 Plus) achieves only 38.2% correct, with 47.3% incorrect and 14.4% partially correct. Task type difficulty varies widely: legal translation and case analysis yield 69-72% correct rates, while regulatory Q&A, hallucination detection, and gap analysis remain below 9%. Within this roster (seven open-weight, three closed-source), an open-weight model leads the ranking, and several open-weight models match or outperform their closed-source counterparts. These findings provide an initial empirical reference point for assessing frontier model capability on Swiss regulatory tasks under zero-retrieval conditions.

Abstract

LLM Evaluation

Explore similar work

Swiss-Bench SBP-002: A Frontier Model Comparison on Swiss Legal and Regulatory Tasks

FinSafetyBench: Evaluating LLM Safety in Real-World Financial Scenarios

Schützen: Evaluating LLM Safety in Bulgarian and German Contexts