norregel: a preregistered benchmark of dated numeric values in Norwegian law

Eirik Botten Nicolaysen

cs.CLOct 21, 2026

norregel: a preregistered benchmark of dated numeric values in Norwegian law

Authors: Eirik Botten Nicolaysen

Organizations: EcoDeco AS

Abstract

norregel asks language models for numeric values in Norwegian statutes, regulations and parliamentary decisions (NOK amounts, percentages, multiples of the National Insurance basic amount (G) and age limits), each at a stated date. The 916 items were extracted mechanically from Lovdata's open data, and each carries the dated history of its provision, so an answer can be classed as correct, outdated, wrong or uninterpretable. We report a first and a second preregistered run on the same items. In the first run (five models, listed publicly only after local analysis), H1, that outdated answers are more frequent for values that took effect after a model's cutoff, could not be tested in substance, and H2 and H3 were inconclusive; Opus 5.5 and Sonnet 4.6 answered 46.7% and 40.2% of the 880 primary items correctly. The second run was preregistered and published before any of its calls, and eight models were analysed. Its pooled primary test supported H1 under the preregistered decision rule (d = 17.8 percentage points, 95% interval [7.7, 28.9], p = 0.0001). In the preregistered sensitivity analysis S8, which counts as outdated only the most recent value that differs from the key, d was 3.9 [-3.6, 12.7] (p = 0.1754), an interval that contains 0. H1 was supported in every other preregistered sensitivity analysis, but S8 is the only one that gives both groups one target value, so we do not regard the support for H1 as robust. The share of outdated answers equal to that most recent value (H2) was 56.1% [46.6, 65.0], an interval that contains 0.5, and the pooled test of a lower share correct in Nynorsk than in Bokmål (H3) was inconclusive. Several models, among them Haiku 4.5, gave mostly uninterpretable answers. Items and keys are withheld; item IDs, per-answer classes, code and results are on Zenodo.

Figures & tables

Set type	Items (all)	Items (primary)	Series (primary)
permanent provisions	265	235	48
yearly regulations	92	92	21
basic amount G	26	26	1
named rates	533	527	26
all	916	880	96

Table 1: Items per set type (the field sett ), all items and primary set, and series in the primary set. Not preregistered as a table; computed after review (supplement A2).

Model	Inspect model id	Cutoff	Reasoning	Run
Haiku 4.5	anthropic/claude-haiku-4-5-20251001	2025-07-31	no	API
Sonnet 4.6	anthropic/claude-sonnet-4-6	2026-01-31	no	API
Opus 5.5	anthropic/claude-opus-5-5	2026-06-30	yes	API
NB-Llama 8B	ollama/nb-llama-3.1-8b-instruct-cardtmpl:latest	unknown	no	local (Ollama)
NorMistral 7B	ollama/normistral-7b-warm-instruct-cardtmpl:latest	unknown	no	local (Ollama)

Table 2: Models. Cutoff is the provider’s stated training-data cutoff; a local model whose card states no date has an unknown cutoff and is not tested in H1.

Model	Correct	Outdated	Wrong	Uninterpretable
Haiku 4.5	0.5 [0.0, 1.1]	0.1 [0.0, 0.4]	5.5 [3.2, 7.7]	94.0 [91.5, 96.4]
Sonnet 4.6	40.2 [29.3, 49.7]	6.5 [3.7, 9.6]	40.9 [32.4, 51.0]	12.4 [9.0, 16.7]
Opus 5.5	46.7 [35.4, 56.2]	6.5 [4.0, 9.3]	46.4 [37.1, 57.4]	0.5 [0.1, 1.0]
NB-Llama 8B	8.5 [3.4, 14.1]	1.2 [0.3, 2.5]	21.6 [14.9, 28.3]	68.6 [60.0, 78.0]
NorMistral 7B	0.0 [0.0, 0.0]	0.0 [0.0, 0.0]	0.2 [0.0, 0.6]	99.8 [99.4, 100.0]

Table 3: Primary set, scorer v2: share of all answers (%) with cluster-bootstrap CI in brackets. Percentages are rounded to one decimal, half to even (Python string formatting), so a row need not add up exactly. Figure 1 shows the shares.

Figure 1: Class shares per model on the primary set (scorer v2), from Table 3 .

H1 : $d$ = share outdated after minus before the cutoff (pp)
Model	$n$ after / before	$d$	CI	$p$ (one-sided)	$H_{0}$ rejected (Holm)
Haiku 4.5	56 / 511	0.0	[0.0, 0.0]	1.0000	no
Sonnet 4.6	9 / 558	$-$ 9.1	[ $-$ 13.6, $-$ 5.1]	n.t. a	–
Opus 5.5	1 / 566	$-$ 9.2	[ $-$ 13.5, $-$ 5.6]	n.t. a	–
NB-Llama 8B	not tested (no cutoff)
NorMistral 7B	not tested (no cutoff)

Table 4: Preregistered tests, primary analysis (Holm within each hypothesis [ 16 ] ). n.t. = not tested: a below the H1 minimum:

d

and its interval are reported without a test, and with no outdated answer after the cutoff the interval is that of minus the share outdated before, conditional on drawing a series with an item after the cutoff (Section 7 ); it is not evidence against H1. b below the H2 minimum; c

b+c=0

. – = not computed. Percentages and percentage points are rounded to one decimal, half to even.

Model	Outdated	Q1	Median	Q3	CI for median
Haiku 4.5	1	12.00	12.00	12.00	[12.00, 12.00]
Sonnet 4.6	57	1.00	1.00	3.00	[1.00, 2.00]
Opus 5.5	57	1.00	1.00	4.00	[1.00, 3.00]
NB-Llama 8B	11	1.50	3.00	4.50	[1.50, 5.00]
NorMistral 7B	0	–	–	–	–

Table 5: FS2 (descriptive, no test): age in years of outdated answers, primary set.

Model	S1 H1	S1 H2	S1 H3	S2 H1	S2 H3	S3 H1	S3 H2	S3 H3
Haiku 4.5	1.0000	n.t.	0.5000	1.0000	0.5000	1.0000	n.t.	0.5000
Sonnet 4.6	1.0000	0.0738	0.0730	n.t.	0.0730	n.t.	0.1018	0.0730
Opus 5.5	n.t.	0.4157	0.9673	n.t.	0.9673	n.t.	0.5027	0.9673
NB-Llama 8B	–	n.t.	0.0625	–	0.3438	–	n.t.	0.0625
NorMistral 7B	–	n.t.	n.t.	–	n.t.	–	n.t.	n.t.

Table 6: Sensitivity analyses: nominal one-sided

p

-values, no Holm correction. S1 = all items and pairs; S2 = scorer v1 (H2 is not defined under v1); S3 = primary set without the flagged items. – = no cutoff; n.t. = not tested (below a minimum or

b+c=0

Inspect model id	Role	Cutoff	Status
anthropic/claude-haiku-4-5-20251001	v1, H1	2025-07-31	analysed
anthropic/claude-opus-4-5-20251101	H1	2025-08-31	analysed
anthropic/claude-opus-4-6	H1	2025-08-31	analysed
anthropic/claude-sonnet-4-6	v1	2026-01-31	analysed
google/gemini-2.5-flash	H1	2025-01-31	dropped (A-V2-5)
google/gemini-2.5-flash-lite	H1	2025-01-31	dropped (A-V2-3)

Table 7: Second run: the model list as locked (PREREG_V2.md, section 5) and the status of each model in the run. Deviations are listed in Table 14 .

Model	$n$	Correct	Outdated	Wrong	Uninterpretable
Haiku 4.5	880	0.5 [0.0, 1.3]	0.1 [0.0, 0.4]	6.4 [3.6, 9.3]	93.1 [89.7, 96.1]
Opus 4.5	880	45.1 [35.2, 53.6]	8.8 [5.8, 11.8]	37.3 [29.7, 46.5]	8.9 [6.8, 11.5]
Opus 4.6	880	43.9 [34.1, 52.3]	11.2 [8.0, 14.5]	44.4 [36.0, 54.7]	0.5 [0.1, 1.0]
Sonnet 4.6	880	39.5 [28.9, 48.9]	6.7 [3.5, 10.1]	40.9 [32.7, 50.6]	12.8 [9.3, 17.5]
Gemma 4 26B	880	7.6 [3.8, 11.8]	4.0 [1.7, 6.7]	50.0 [44.5, 55.3]	38.4 [32.9, 44.3]
Gemma 4 31B	880	11.7 [6.0, 18.4]	5.5 [2.9, 8.6]	81.6 [74.1, 88.7]	1.2 [0.5, 2.1]

Table 8: Second run, primary set, run 1, scorer v2: share of all answers (%) with cluster-bootstrap CI in brackets (RESULTS_V2.md, T3).

Model	$n$ after	Series after	$n$ before	$d$	95% interval	$p$	State
Primary analysis (H1), interpretable answers
Pooled	115	55	1877	17.8	[7.7, 28.9]	0.0001	supported
Haiku 4.5	0	0	40	–	–	–	not tested (Holm: –)
Opus 4.5	6	6	492	36.4	[ $-$ 12.4, 85.3]	–	not tested (Holm: –)
Opus 4.6	35	35	528	34.2	[17.5, 50.9]	0.0001	supported (Holm: yes)
Gemma 4 26B	15	13	318	1.0	[ $-$ 8.4, 17.9]	–	not tested (Holm: –)

Table 9: Second run, H1 (RESULTS_V2.md, T8).

d

is the share outdated after minus before the cutoff among interpretable answers, in percentage points, with a stratified cluster-bootstrap 95% interval;

p

is one-sided. The pooled primary test decides H1; per-model decisions use Holm. S8 is a preregistered sensitivity analysis with nominal

p

. A model is tested only with at least 30 interpretable answers in each group.

Analysis	Definition	$d$	95% interval	$p$	State
S1	«Alle 916 items; H3 har de samme 880 parene» ( docs/PREREG_V2.md:271 )	10.6	[2.9, 19.7]	0.0032	supported
S2	«Scorer v1 (H1 og H3; H2 er ikke definert under v1)» ( docs/PREREG_V2.md:272 )	11.9	[2.2, 22.6]	0.0071	supported
S3	«Uten de 7 flaggede itemene» ( docs/PREREG_V2.md:273 )	18.4	[8.2, 29.3]	0.0005	supported
S4	«H1 med alle svar som nevner» ( docs/PREREG_V2.md:274 )	5.2	[0.7, 10.0]	0.0115	supported
S5	«H1 med «reliable knowledge cutoff» der leverandøren oppgir den, ellers samme cutoff» ( docs/PREREG_V2.md:275 )	14.0	[5.0, 24.0]	0.0007	supported
S6, run 2	«Samme analyse på kjøring 2 og 3 for modellene med R = 3 som er komplette i den kjøringen» ( docs/PREREG_V2.md:276 )	10.7	[3.5, 19.5]	0.0009	supported

Table 10: Second run: all preregistered sensitivity analyses of H1, pooled over the H1 models, with nominal one-sided

p

-values (RESULTS_V2.md, T16; S8 from T8). The definitions are quoted verbatim from the preregistration (PREREG_V2.md, section 7.5).

d

in percentage points with a 95% interval.

p

is one-sided and the interval two-sided, so a test can reject at 0.05 while its interval contains 0 (S9).

	After the cutoff			Before the cutoff
Model	Items	Interpretable	Uninterpretable	Items	Interpretable	Uninterpretable
Haiku 4.5	56	0	56	511	40	471
Opus 4.5	36	6	30	531	492	39
Opus 4.6	36	35	1	531	528	3
Gemma 4 26B	64	15	49	503	318	185
Gemma 4 31B	64	59	5	503	499	4

Table 11: Second run, H1 set: answers after and before each H1 model’s cutoff. Items and uninterpretable answers are from RESULTS_V2.md, T19; interpretable answers are the

n

of the primary H1 analysis (T8). Uninterpretable is the scorer-v2 class utolket : no answer line, no number, or more than one number.

Model	Outdated	Series	$s$	95% interval	Interval vs 0.5
Pooled	287	33	56.1	[46.6, 65.0]	contains 0.5
Haiku 4.5	0	0	fewer than 30	–	–
Opus 4.5	70	23	65.7	[50.5, 81.0]	above 0.5
Opus 4.6	88	30	58.0	[46.6, 68.9]	contains 0.5
Sonnet 4.6	52	19	53.8	[38.2, 75.8]	contains 0.5
Gemma 4 26B	30	12	50.0	[20.7, 78.6]	contains 0.5

Table 12: Second run, H2 as estimation (RESULTS_V2.md, T9):

s

is the share (%) of outdated answers that equal the most recent value that differs from the key, with a cluster-bootstrap 95% interval; no test.

Model	$b$	$c$	Pairs	$D$	95% interval	$p$	State	Uninterpretable (%)
Pooled	173	146	7040	$-$ 0.4	[ $-$ 1.1, 0.3]	0.1495	inconclusive	–
Haiku 4.5	11	0	880	$-$ 1.2	[ $-$ 3.6, 0.0]	0.0005	supported (Holm: yes)	93.4
Opus 4.5	24	35	880	1.2	[ $-$ 0.9, 3.4]	0.9413	no effect of interest (Holm: no)	8.4
Opus 4.6	14	26	880	1.4	[ $-$ 0.2, 3.2]	0.9808	no effect of interest (Holm: no)	0.3
Sonnet 4.6	39	26	880	$-$ 1.5	[ $-$ 4.0, 0.7]	0.0680	inconclusive (Holm: no)	12.6
Gemma 4 26B	16	21	880	0.6	[ $-$ 1.1, 2.4]	0.8380	inconclusive (Holm: no)	37.3

Table 13: Second run, H3 (RESULTS_V2.md, T10):

b

pairs with only Bokmål correct,

c

pairs with only Nynorsk correct;

D=(c-b)/\mathrm{pairs}

in percentage points with a cluster-bootstrap 95% interval;

p

is one-sided for a lower share correct in Nynorsk. Per-model decisions use the exact McNemar test with Holm. The last column is the share of uninterpretable answers among all answers of the H3 task, run 1 (descriptive count written after the analysis, anc/H3_UTOLKET_V2.md).

Code	Content	Source
D1	the log directory differs from the addendum’s command	docs/RUN_PROTOCOL.md:211
D2	the local models ran with a single connection	docs/RUN_PROTOCOL.md:212
D3	the years-and-months rule is applied in the analysis, not in the locked scorer	docs/RUN_PROTOCOL.md:214
D4	the provider client libraries were installed in the existing environment	docs/RUN_PROTOCOL.md:215
A1	replaced client library versions, before any request	docs/RUN_PROTOCOL.md:216
D5	the model list was bound by a local commit during the runs	docs/RUN_PROTOCOL.md:217

Table 14: All deviations and amendments of the first (D, A) and second run (A-V2). The source is the repository file and line; D13 is recorded in the results file.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Uninterpretable	(a)	(c)	(d)	(e)	Sample
Haiku 4.5	827	30	0	0	0	30
Sonnet 4.6	109	30	0	0	0	30
NB-Llama 8B	604	13	9	4	4	30

Appendix

Table 15: Exploratory, not preregistered: categories of a random sample of uninterpretable answers. (a) refusal; (b) no value; (c) one value in a format the scorer does not read; (d) several values or ambiguous; (e) other.

Model	Correct	Outdated	Wrong	Uninterpretable	Rewritten
Haiku 4.5	0.5	0.1	5.5	94.0	0
Sonnet 4.6	40.2	6.5	40.9	12.4	0
Opus 5.5	46.7	6.5	46.4	0.5	0
NB-Llama 8B	11.5	1.7	31.0	55.8	113
NorMistral 7B	0.5	0.2	0.8	98.5	11

Appendix

Table 16: Exploratory, not preregistered: primary set re-scored after a tolerant parser rewrote answers in three formats found in category (c). Shares (%) and number of rewritten answers.

Model	Group	$n$	Correct	Outdated	Wrong	Uninterpretable
Haiku 4.5	X1	5	0	0	0	5
Haiku 4.5	X2	23	0	0	0	23
Haiku 4.5	X3	6	0	0	0	6
Haiku 4.5	X4	2	0	0	0	2
Sonnet 4.6	X1	5	1	1	3	0
Sonnet 4.6	X2	23	0	0	0	23

Appendix

Table 17: Excluded items per exclusion group (counts, scorer v2), as preregistered in the addendum. X1 wrong key for the question date; X2 question date after the corpus retrieval; X3 ambiguous value in the provision; X4 amendments of the annual tax decision.

Start (UTC)	Model	Status	Samples	Errors	Classes	Reason
2026-10-16 16:30:54	anthropic/claude-haiku-4-5-20251001	success	20	20	0 / 0 / 0 / 20	pilot, A1
2026-10-16 16:31:39	anthropic/claude-haiku-4-5-20251001	success	20	0	0 / 0 / 1 / 19	pilot
2026-10-16 16:32:18	anthropic/claude-opus-5-5	success	20	0	10 / 1 / 9 / 0	pilot
2026-10-16 16:32:01	anthropic/claude-sonnet-4-6	success	20	0	9 / 1 / 8 / 2	pilot
2026-10-16 16:41:54	ollama/hf.co/ltg/normistral-7b-warm-instruct:Q4_K_M	success	20	0	0 / 0 / 0 / 20	pilot
2026-10-16 17:52:39	ollama/nb-llama-3.1-8b-instruct-cardtmpl:latest	success	20	0	1 / 1 / 4 / 14	pilot

Appendix

Table 18: Every run that is not a primary log, all on the main task: start (UTC) from the log header, status, samples in the log, samples in error, and classes under scorer v2 (correct / outdated / wrong / uninterpretable) of the samples present. None of them counts in the results (supplement A1).

Model	Interpretable / all	Correct	Outdated	Wrong
Haiku 4.5	53 / 880	7.5	1.9	90.6
Sonnet 4.6	771 / 880	45.9	7.4	46.7
Opus 5.5	876 / 880	46.9	6.5	46.6
NB-Llama 8B	276 / 880	27.2	4.0	68.8
NorMistral 7B	2 / 880	0.0	0.0	100.0

Appendix

Table 19: Exploratory, after review: classes among interpretable answers (correct, outdated or wrong), primary set (supplement A3). Shares in %.

Model	Cutoff	$n$ after / before	Uninterpretable after	Uninterpretable before	Classes after
Haiku 4.5	2025-07-31	56 / 511	56 (100.0)	475 (93.0)	0 / 0 / 0 / 56
Sonnet 4.6	2026-01-31	9 / 558	8 (88.9)	74 (13.3)	0 / 0 / 1 / 8
Opus 5.5	2026-06-30	1 / 566	0 (0.0)	4 (0.7)	1 / 0 / 0 / 0

Appendix

Table 20: Exploratory, after review: uninterpretable answers in the H1 set after and before the cutoff (count and % of all answers in the group), and the classes after the cutoff (correct / outdated / wrong / uninterpretable); supplement A3.

Model	$b$	$c$	Difference	Interval from $b$ , $c$	Bootstrap interval
Haiku 4.5	1	0	$-$ 0.5	[ $-$ 0.5, 0.5]	[ $-$ 1.6, 0.0]
Sonnet 4.6	9	3	$-$ 3.2	[ $-$ 5.7, 0.9]	[ $-$ 8.6, 1.1]
Opus 5.5	3	8	2.6	[ $-$ 1.3, 5.1]	[ $-$ 1.0, 6.6]
NB-Llama 8B	6	1	$-$ 2.6	[ $-$ 3.7, 0.6]	[ $-$ 6.7, 0.5]
NorMistral 7B	0	0	0.0	not defined	[0.0, 0.0]

Appendix

Table 21: Exploratory, after review: H3 difference (Nynorsk

-

Bokmål, pp) with an exact interval from

b

and

c

alone (Clopper–Pearson for

c/(b+c)

, converted to

(c-b)/n

given

b+c

; it ignores clustering and is narrower), next to the preregistered bootstrap interval; supplement A4.

Model	Correct	Outdated	Wrong	Uninterpretable
S1 (all items)
Haiku 4.5	0.4	0.1	5.2	94.2
Sonnet 4.6	39.1	6.3	40.2	14.4
Opus 5.5	45.1	6.4	46.2	2.3
NB-Llama 8B	8.2	1.2	21.2	69.4
NorMistral 7B	0.0	0.0	0.2	99.8

Appendix

Table 22: Sensitivity analyses: share of all answers (%) per class. Percentages are rounded to one decimal, half to even.

Model	NOK amount ( $n=553$ )	Percentage ( $n=319$ )	Age ( $n=8$ )
Haiku 4.5	0.2	0.6	12.5
Sonnet 4.6	27.5	63.3	0.0
Opus 5.5	32.9	70.8	37.5
NB-Llama 8B	3.1	17.9	12.5
NorMistral 7B	0.0	0.0	0.0

Appendix

Table 23: Share correct (%) by value type, primary set (counts and shares only, no intervals or tests). Percentages are rounded to one decimal, half to even.

Model	Wrong	Later value	Not in history
Haiku 4.5	48	5	43
Sonnet 4.6	360	41	319
Opus 5.5	408	49	359
NB-Llama 8B	190	10	180
NorMistral 7B	2	0	2

Appendix

Table 24: Wrong answers split into a later value and a value not in the item’s history (counts, primary set).

LLM Reliability

Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.

See all 789 papers

Aug 10, 2026

Temporal Misgrounding in Legal RAG: A Versioned-Corpus Benchmark for French Tax Law

We identify and quantify temporal misgrounding: the systematic retrieval and citation of the currently in-force version of a legal article when the applicable version is an earlier or future one. Standard legal RAG treats the corpus as static; we argue legal question answering is a temporally-indexed retrieval problem. We introduce FiscalQA Pro, pairing a versioned corpus of 32,436 article-versions of the French tax code (93 years, 1938-2031) with an all-model-hard temporal-reasoning track: 209 scored, expert-reviewed questions across 33 CGI articles (221 released; twelve flagged out of the answerable scope). At selection time, no evaluated model recovered its date-applicable answer closed-book in any of four sampling draws, and the currently in-force text lacks the gold value for all but one of the scored questions. Answers are scored deterministically via atomic ground-truth "nuggets" (regex and numeric-with-tolerance), never LLM-as-judge: an LLM judge would inherit the temporal bias it is meant to score. Across eleven models (five frontier closed-API systems plus Gemini 2.5 Pro as a substitute entry, and five open-weight), parametric knowledge yields 3.0% mean strict accuracy and RAG over a static current-version corpus 2.7%. Static RAG retrieves the date-applicable version 0% of the time, confidently citing a real but inapplicable version. Our end-to-end retriever over a multi-version index, with no oracle, reaches 98.3% mean strict; an oracle-article ablation reaches 99.1%, locating the residual gap in first-stage recall, not version selection. We additionally release a version-aware jurisprudence dataset of 69,208 citation links, together with the corpus, benchmark, model responses, and pipeline code.

Rose Cymbler, Daniel Guez, Laurent Fabre

Talia, Paris, France · Databricks

Temporal IR Legal QA RAG Evaluation Temporal QA

Ask PDF

May 22, 2026

Asking For An Old Friend: Diagnosing and Mitigating Temporal Failure Modes in LLM-based Statutory Question Answering

Large language models are increasingly used for legal research, yet their fixed training cutoffs and reliance on static parametric knowledge are at odds with the evolving nature of statutory law. We study two temporal failure modes: post-cutoff staleness, where models apply superseded rules after legislative amendments, and recency bias, where models prefer newer provisions even when a historical version governs the fact pattern. To this end, we present a benchmark of 312 expert-validated, time-sensitive German statutory QA pairs spanning three categories: Post-Cutoff Amendment Questions, Pre-Amendment Questions, and Multi-Provision Pre-Amendment Questions. We evaluate five LLMs by OpenAI, Anthropic and DeepSeek under four inference settings: Vanilla, Web-search, and two retrieval-augmented variants that enforce temporal validity via a fact date extraction and version filtering. Using an LLM-as-a-judge validated against human expert ratings, we find severe degradation in the Vanilla post-cutoff setting. Both RAG approaches substantially improve performance across all question types, while web search yields unstable gains and exhibits a marked recency bias on historically anchored tasks. Our results indicate that reliable legal QA requires treating temporal validity as a hard constraint.

Max Prior, Andreas Schultz, Matthias Grabmair

Technical University of Munich Garching near Munich, Germany

Retrieval-Augmented Generation Temporal IR Legal QA Temporal QA

Ask PDF

Mar 24, 2026

Swiss-Bench SBP-002: A Frontier Model Comparison on Swiss Legal and Regulatory Tasks

While recent work has benchmarked large language models on Swiss legal translation (Niklaus et al., 2025) and academic legal reasoning from university exams (Fan et al., 2025), no existing benchmark evaluates frontier model performance on applied Swiss regulatory compliance tasks. I introduce Swiss-Bench SBP-002, a trilingual benchmark of 395 expert-crafted items spanning three Swiss regulatory domains (FINMA, Legal-CH, EFK), seven task types, and three languages (German, French, Italian), and evaluate ten frontier models from March 2026 using a structured three-dimension scoring framework assessed via a blind three-judge LLM panel (GPT-4o, Claude Sonnet 4, Qwen3-235B) with majority-vote aggregation and weighted kappa = 0.605, with reference answers validated by an independent human legal expert on a 100-item subset (73% rated Correct, 0% Incorrect, perfect Legal Accuracy). Results reveal three descriptive performance clusters: Tier A (35-38% correct), Tier B (26-29%), and Tier C (13-21%). The benchmark proves difficult: even the top-ranked model (Qwen 3.5 Plus) achieves only 38.2% correct, with 47.3% incorrect and 14.4% partially correct. Task type difficulty varies widely: legal translation and case analysis yield 69-72% correct rates, while regulatory Q&A, hallucination detection, and gap analysis remain below 9%. Within this roster (seven open-weight, three closed-source), an open-weight model leads the ranking, and several open-weight models match or outperform their closed-source counterparts. These findings provide an initial empirical reference point for assessing frontier model capability on Swiss regulatory tasks under zero-retrieval conditions.

Fatih Uenal

University of Colorado Boulder

LLM Evaluation Legal NLP Multilingual Language Model Evaluation

Ask PDF

Abstract

LLM Reliability

Explore similar work

Temporal Misgrounding in Legal RAG: A Versioned-Corpus Benchmark for French Tax Law

Asking For An Old Friend: Diagnosing and Mitigating Temporal Failure Modes in LLM-based Statutory Question Answering

Swiss-Bench SBP-002: A Frontier Model Comparison on Swiss Legal and Regulatory Tasks