NitiBench+: A Temporal Thai Legal Question Answering Benchmark Across 17 Domains
Organizations: FourCorners Team, VISAI AI, Thailand · VISTEC, Thailand
Abstract
Law changes over time. A court decides a case under the law in force on that day, and legal QA benchmarks grade answers against today's law. We checked 156,650 decisions of four Thai courts. Up to 67% of them cite law that has since been amended or repealed. We introduce Nitibench+, the sequel of our NitiBench, a Thai legal QA benchmark with 1,670 items across 17 domains, built from decisions of three courts. For each cited section it holds both the text in force on the decision date and the current text, and its retrieval corpus holds every dated version of every Thai statute section. Answers are graded on the Thai Bar Council rubric, the Bar-standard score, and on the overlap between the cited sections and the court's. The report has two parts. Part I measures the cost of the wrong text on 13 open-weight models. Reading the current text of amended law drops the Bar-standard score by 0.25 and missing repealed law drops it by 0.79. The penalty comes from rule changes, 0.45 on the 428 rule-changed items and none on the 331 wording-only items, and Citation F1 does not see it. The text in force is the fair setting. Part II benchmarks today's models on that setting. A public leaderboard ranks 33 models on a held-out test split of 500 items stratified by domain, with a validation split of the same shape released for public use, from memory alone, with the court's sections provided, and as a retrieval agent over the law in force, with the cost per item and the hallucinated citations. The best model scores 7.78 out of 10 and the best open-weight model 6.39. Without a search tool only five open-weight models recall any Thai law, and with it 27 of the 33 models gain and six lose. Leaderboard will take new models as they are released. Legal benchmarks must track dates, and on the right text today's models still leave a wide gap on Thai law.
Figures & tables
| Court | Decisions | Current | Amended | Repealed | Drifted |
|---|---|---|---|---|---|
| Supreme Court | 51,566 | 32.9 | 56.1 | 11.0 | 67.1 |
| Constitutional Court | 2,025 | 37.6 | 2.7 | 59.8 | 62.4 |
| Administrative Courts | 101,315 | 57.4 | 26.2 | 16.4 | 42.6 |
| Court of Appeal for Specialised Cases | 1,744 | 76.7 | 19.3 | 3.9 | 23.3 |
| Released | Bench. | Cites statute | Item date | Version | Snapshot |
|---|---|---|---|---|---|
| before 2024 | 20 | 5 (+11) | 2 (+6) | 1 | 2 (+2) |
| 2024–2025 | 19 | 9 (+5) | 1 (+5) | 3 | 3 (+2) |
| 2026 | 14 | 6 (+8) | 7 (+4) | 5 (+1) | 5 (+1) |
| All | 53 | 20 (+24) | 10 (+15) | 9 (+1) | 10 (+5) |
| Benchmark | Year | Jurisdiction | Cites | Date | Ver. | Snap. |
| before 2024 | ||||||
| CAIL2018 [ 88 ] | 2018 | China | ✓ | – | – | – |
| JEC-QA [ 97 ] | 2020 | China | – | – | – | – |
| SARA [ 41 ] | 2020 | US | ✓ | ✓ | – | – |
| CaseHOLD [ 95 ] | 2021 | US | ( ) | – | – | – |
| COLIEE statute law tasks [ 80 ] | 2021 | Japan | ✓ | ( ) | – | ✓ |
| Benchmark | Questions | Amended ever | Stale |
|---|---|---|---|
| NitiBench-CCL | 3,729 | 25% | 2.3% |
| NitiBench-Tax | 50 | 72% | 0.0% |
| Drift band (items) | Sub-questions by type (court’s own + four added kinds) | |||||||||
| Domain | Items | Sub-q. | No drift | Amended | Repealed | Court’s own | Governs | Conseq. | Combine | Lower ct. |
| Family & Succession | 146 | 326 | 60 | 85 | 1 | 153 | 15 | 61 | 57 | 40 |
| Criminal | 146 | 352 | 13 | 106 | 27 | 157 | 39 | 52 | 43 | 61 |
| Civil & Commercial | 144 | 351 | 63 | 60 | 21 | 153 | 31 | 63 | 56 | 48 |
| Land & Property | 117 | 284 | 43 | 42 | 32 | 123 | 22 | 56 | 50 | 33 |
| Labor | 116 | 239 | 41 | 63 | 12 | 117 | 12 | 37 | 35 | 38 |
| \headrow | Reasoning quality | ||||
|---|---|---|---|---|---|
| \headrow Final answer | Unusable | Some | Acceptable | Good | Very good |
| matches gold | T4: – | T5: – | T6: – | T7: – | T8: – |
| differs | T1: | T2: – | T3: – | ||
| Citation F1 | Bar-standard (0–10) | ||||||||||
| Model | Param. | GC Cur. | GC InF. | RAG Cur. | RAG InF. | Param. | GC Cur. | GC InF. | RAG Cur. | RAG InF. | |
| General | |||||||||||
| Gemma 4 31B | 0.304 | 0.645 | 0.778 | 0.432 | 0.512 | 3.68 | 4.60 | 5.11 | 4.20 | 4.55 | |
| Gemma 4 26B-A4B | 0.272 | 0.667 | 0.791 | 0.228 | 0.262 | 3.42 | 4.37 | 4.79 | 3.61 | 3.71 | |
| Nemotron 3.5 Lightning 30B-A3B | 0.028 | 0.655 | 0.770 | 0.287 | 0.354 | 1.76 | 3.08 | 3.44 | 2.26 | 2.47 | |
| Muse Glimmer 30B | 0.198 | 0.742 | 0.878 | 0.456 | 0.551 | 3.34 | 5.18 | 5.75 | 4.39 | 4.94 | |
| Gold-cited | RAG | ||||
| Model | Amend. | Repeal. | Amend. | Repeal. | |
| General | |||||
| Gemma 4 31B | |||||
| Gemma 4 26B-A4B | |||||
| Nemotron 3.5 Lightning 30B-A3B | |||||
| Muse Glimmer 30B | |||||
| Amendment | Median | Bar-standard | Citation F1 |
|---|---|---|---|
| Rule changed (428 items) | 0.34 | +0.45 [+0.38, +0.53] | +0.04 [+0.03, +0.06] |
| Wording only (331 items) | 0.10 | -0.00 [-0.03, +0.03] | -0.00 [-0.01, +0.00] |
| Agrees with court (%) | Score if agreeing | |||
|---|---|---|---|---|
| Band | Current | In force | Current | In force |
| No drift | 60.9 | 61.2 | 5.07 | 5.06 |
| Amended | 57.6 | 59.2 | 4.83 | 5.12 |
| Repealed | 53.9 | 58.4 | 3.88 | 4.83 |
| Amendment size | Bar-standard | Citation F1 |
|---|---|---|
| 1 character-trigram Jaccard | +0.35 [+0.28, +0.41] | +0.29 [+0.22, +0.37] |
| Normalized Levenshtein distance | +0.34 [+0.27, +0.41] | +0.28 [+0.20, +0.35] |
| Parametric | Gold-cited In-force | RAG In-force | |||||||
| Model | Full | Test | Val. | Full | Test | Val. | Full | Test | Val. |
| General | |||||||||
| Gemma 4 31B | 3.68 | 3.66 | 3.74 | 5.11 | 5.16 | 5.07 | 4.55 | 4.57 | 4.52 |
| Gemma 4 26B-A4B | 3.42 | 3.44 | 3.43 | 4.79 | 4.84 | 4.76 | 3.71 | 3.68 | 3.74 |
| Nemotron 3.5 Lightning 30B-A3B | 1.76 | 1.73 | 1.82 | 3.44 | 3.49 | 3.46 | 2.47 | 2.47 | 2.47 |
| Muse Glimmer 30B | 3.34 | 3.35 | 3.40 | 5.75 | 5.83 | 5.72 | 4.94 | 4.93 | 4.95 |
| Bar-standard | ||||||
| Model | Param. | RAG InF. | GC InF. | Gain \up | Gap \down | Searches |
| Proprietary | ||||||
| Claude Fable 5.1 | \best 7.40 | \best 7.78 | \best 8.04 | +0.38 | +0.26 | 8.6 |
| Gemini 4 Carbon | 6.64 | \second 7.68 | 7.62 | +1.04 | \best 0.06 | 19.1 |
| Claude Fable 5.2 | \second 7.19 | 7.55 | \second 7.89 | +0.36 | +0.34 | 9.8 |
| Gemini 4 Argon | 6.17 | 7.54 | 7.63 | +1.37 | +0.09 | 7.6 |
| Bar-standard | In RAG In-force | ||||
|---|---|---|---|---|---|
| Model | Param. | GC InF. | RAG InF. | Citation F1 | Searches |
| Nemotron 3 Nano 30B-A3B | 1.15 | 2.71 | 1.62 | 0.251 | 2.9 |
| SEA-LION v4.8 30B-A3B | 1.66 (+0.51) | 3.53 (+0.82) | 2.45 (+0.83) | 0.335 (+0.084) | 4.7 (+1.8) |
| Nemotron 3 Nano Omni 30B-A3B | 1.23 (+0.08) | 2.78 (+0.07) | 1.45 ( 0.17) | 0.194 ( 0.057) | 2.9 (+0.0) |
| OpenThai 2.0 Legal 30B-A3B | 1.18 ( 0.05) | 1.92 ( 0.86) | 0.87 ( 0.58) | 0.005 ( 0.189) | 0.0 ( 2.9) |
| Nemotron 3 Super 120B-A12B | 2.05 | 4.29 | 3.23 | 0.424 | 6.5 |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Checkpoint (Hugging Face) | Params | TP | Window | Reasoning | Tool parser |
| General | ||||||
| Gemma 4 31B | google/gemma-4-31B-it | 31B | 2 | 262k | on ( gemma4 ) | gemma4 |
| Gemma 4 26B-A4B | google/gemma-4-26B-A4B-it | 26B / 4B | 2 | 262k | on ( gemma4 ) | gemma4 |
| Nemotron 3.5 Lightning 30B-A3B | nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 | 32B / 3B | 2 | 262k | on ( nemotron_v3 ) | qwen3_xml |
| Muse Glimmer 30B | meta-models/Muse-Glimmer-30B | 30B | 2 | 131k | on ( muse_glimmer ) | muse_glimmer |
| Nemotron 3 bases and their adaptations | ||||||
| Tokens (M) | Reasoning | Searches per item | GPU | |||
| Model | In | Out | share | RAG Cur. | RAG InF. | hours |
| General | ||||||
| Gemma 4 31B | 416 | 126.4 | 84% | 3.2 | 3.3 | 315 |
| Gemma 4 26B-A4B | 193 | 202.8 | 91% | 1.3 | 1.2 | 200 |
| Nemotron 3.5 Lightning 30B-A3B | 1563 | 424.2 | 92% | 5.7 | 5.1 | 380 |
| Muse Glimmer 30B | 3599 | 165.7 | 84% | 16.4 | 15.6 | 455 |
| Bar-standard score | Cited sections (RAG In-force) | |||||||
|---|---|---|---|---|---|---|---|---|
| # | Model | Access | RAG InF. | Parametric | GC InF. | F1 | Precision | Recall |
| 1 | Claude Fable 5.1 | Proprietary | 7.78 | 7.40 | 8.04 | 0.515 | 0.394 | 0.880 |
| 2 | Gemini 4 Carbon | Proprietary | 7.68 | 6.64 | 7.62 | 0.516 | 0.406 | 0.868 |
| 3 | Claude Fable 5.2 | Proprietary | 7.55 | 7.19 | 7.89 | 0.595 | 0.494 | 0.872 |
| 4 | Gemini 4 Argon | Proprietary | 7.54 | 6.17 | 7.63 | 0.629 | 0.549 | 0.857 |
| 5 | Gemini 4 Hydrogen | Proprietary | 7.53 | 6.63 | 7.58 | 0.480 | 0.361 | 0.878 |
| # | Model | Bar-standard | Cost (USD) | Output tokens (K) | Searches | Hallucinated citations |
|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | 7.78 | 1.155 | 8.3 | 8.6 | 0.0% |
| 2 | Gemini 4 Carbon | 7.68 | 0.773 | 7.7 | 19.1 | 0.0% |
| 3 | Claude Fable 5.2 | 7.55 | 1.736 | 12.8 | 9.8 | 0.0% |
| 4 | Gemini 4 Argon | 7.54 | 0.338 | 5.2 | 7.6 | 0.0% |
| 5 | Gemini 4 Hydrogen | 7.53 | 0.893 | 42.6 | 45.5 | 0.0% |
| 6 | GPT-6.1 Sol | 7.28 | 0.134 | 4.1 | 6.3 | 0.0% |
| \headrow | Mean Bar-standard | Best (RAG InF.) | |||||
|---|---|---|---|---|---|---|---|
| \headrow Domain | Items | Param. | GC InF. | RAG InF. | Model | Score | Lowest |
| Family & Succession | 146 | 2.02 | 3.99 | 2.72 | Muse Glimmer 30B | 5.09 | 0.90 |
| Intellectual Property | 109 | 2.31 | 3.66 | 2.70 | Muse Glimmer 30B | 5.30 | 1.04 |
| Public Law | 98 | 2.26 | 3.73 | 2.69 | Muse Glimmer 30B | 5.49 | 0.85 |
| Natural Resources & Licensing | 84 | 2.07 | 3.82 | 2.66 | Muse Glimmer 30B | 5.39 | 0.93 |
| Traffic & Transport | 63 | 1.88 | 3.57 | 2.62 | Muse Glimmer 30B | 5.45 | 0.88 |
| \headrow Model | All | Family & Succession | Intellectual Property | Public Law | Natural Resources & Licensing | Traffic & Transport | Labor | Civil & Commercial | Criminal | Tax & Revenue | Securities & Finance | Civil Procedure | Administrative Acts | Land & Property | State Liability | Narcotics | Administrative Contracts | Public Personnel |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Muse Glimmer 30B | 4.94 | 5.09 | 5.30 | 5.49 | 5.39 | 5.45 | 5.10 | 4.87 | 4.98 | 4.98 | 4.88 | 5.00 | 4.94 | 4.45 | 4.73 | 4.70 | 4.46 | 4.22 |
| Gemma 4 31B | 4.55 | 5.05 | 4.83 | 4.85 | 4.75 | 4.59 | 4.76 | 4.92 | 4.73 | 4.59 | 4.64 | 4.50 | 4.37 | 4.46 | 4.20 | 4.16 | 4.23 | 3.79 |
| Gemma 4 26B-A4B | 3.71 | 3.98 | 4.04 | 4.08 | 3.78 | 3.77 | 3.85 | 3.94 | 3.92 | 3.76 | 3.75 | 3.74 | 3.68 | 3.58 | 3.47 | 3.33 | 3.51 | 2.97 |
| Nemotron 3 Super | 3.31 | 3.70 | 3.45 | 3.52 | 3.52 | 3.79 | 3.48 | 3.37 | 3.39 | 3.25 | 3.32 | 3.31 | 3.13 | 3.20 | 3.00 | 3.21 | 2.93 | 2.63 |
| SEA-LION v4.8 120B | 2.94 | 3.19 | 3.10 | 3.43 | 3.47 | 3.33 | 3.01 | 2.93 | 2.84 | 2.98 | 2.99 | 2.86 | 2.80 | 2.79 | 2.71 | 2.73 | 2.56 | 2.21 |
| SEA-LION v4.8 30B | 2.50 | 2.84 | 2.82 | 2.64 | 2.81 | 2.66 | 2.62 | 2.52 | 2.35 | 2.46 | 2.51 | 2.31 | 2.54 | 2.55 | 2.53 | 2.28 | 2.08 | 1.96 |
| Model | Land & Property | Intellectual Property | Family & Succession | Tax & Revenue | Civil & Commercial | Criminal | Labor | Securities & Finance | Narcotics | Civil Procedure | Natural Resources & Licensing | Traffic & Transport | Public Law | Administrative Acts | Administrative Contracts | Public Personnel | State Liability |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Fable 5.1 | 7.83 | 8.08 | 8.09 | 7.28 | 7.81 | 7.41 | 8.06 | 7.83 | 7.59 | 7.75 | 8.02 | 8.13 | 8.17 | 8.10 | 7.16 | 7.21 | 7.80 |
| Gemini 4 Carbon | 7.69 | 7.72 | 7.32 | 7.26 | 7.72 | 7.45 | 8.54 | 8.25 | 7.60 | 8.31 | 7.72 | 7.64 | 7.57 | 7.68 | 7.08 | 7.23 | 7.77 |
| Claude Fable 5.2 | 7.37 | 7.78 | 7.68 | 6.93 | 7.87 | 7.38 | 8.21 | 7.73 | 7.24 | 8.25 | 7.80 | 7.45 | 7.93 | 7.21 | 6.86 | 7.00 | 7.69 |
| Gemini 4 Argon | 7.23 | 7.57 | 7.62 | 6.76 | 7.68 | 7.51 | 8.28 | 7.65 | 6.39 | 7.82 | 8.04 | 7.62 | 8.16 | 7.67 | 7.00 | 7.45 | 7.71 |
| Gemini 4 Hydrogen | 7.45 | 7.31 | 7.52 | 6.75 | 7.93 | 7.23 | 8.25 | 7.66 | 7.08 | 8.16 | 7.72 | 7.60 | 7.66 | 7.91 | 6.94 | 7.17 | 7.70 |
| GPT-6.1 Sol | 7.01 | 7.68 | 6.96 | 6.38 | 7.55 | 7.31 | 7.75 | 7.70 | 6.50 | 7.52 | 7.77 | 7.39 | 7.84 | 7.84 | 6.40 | 6.98 | 7.19 |
| Citation F1, best@3 | Bar-standard, best@3 | ||||||||||
| Model | Param. | GC Cur. | GC InF. | RAG Cur. | RAG InF. | Param. | GC Cur. | GC InF. | RAG Cur. | RAG InF. | |
| General | |||||||||||
| Gemma 4 31B | 0.373 | 0.715 | 0.861 | 0.506 | 0.604 | 4.38 | 5.46 | 6.05 | 4.93 | 5.33 | |
| Gemma 4 26B-A4B | 0.341 | 0.712 | 0.850 | 0.352 | 0.414 | 4.11 | 5.07 | 5.53 | 4.37 | 4.52 | |
| Nemotron 3.5 Lightning 30B-A3B | 0.054 | 0.717 | 0.842 | 0.377 | 0.466 | 2.28 | 3.81 | 4.21 | 2.95 | 3.18 | |
| Muse Glimmer 30B | 0.271 | 0.773 | 0.914 | 0.521 | 0.632 | 4.00 | 5.94 | 6.58 | 5.16 | 5.78 | |
| Citation F1, SD over samples | Bar-standard, SD over samples | ||||||||||
| Model | Param. | GC Cur. | GC InF. | RAG Cur. | RAG InF. | Param. | GC Cur. | GC InF. | RAG Cur. | RAG InF. | |
| General | |||||||||||
| Gemma 4 31B | 0.003 | 0.005 | 0.004 | 0.002 | 0.007 | 0.00 | 0.02 | 0.02 | 0.01 | 0.02 | |
| Gemma 4 26B-A4B | 0.005 | 0.001 | 0.004 | 0.002 | 0.004 | 0.04 | 0.02 | 0.03 | 0.02 | 0.02 | |
| Nemotron 3.5 Lightning 30B-A3B | 0.002 | 0.003 | 0.004 | 0.001 | 0.002 | 0.00 | 0.02 | 0.01 | 0.01 | 0.03 | |
| Muse Glimmer 30B | 0.003 | 0.003 | 0.002 | 0.002 | 0.002 | 0.01 | 0.01 | 0.02 | 0.02 | 0.04 | |
| Gold-cited, In-force minus Current | RAG, In-force minus Current | ||||||
| Model | No drift | Amended | Repealed | No drift | Amended | Repealed | |
| General | |||||||
| Gemma 4 31B | [ , ] | [ , ] | [ , ] | [ , ] | [ , ] | [ , ] | |
| Gemma 4 26B-A4B | [ , ] | [ , ] | [ , ] | [ , ] | [ , ] | [ , ] | |
| Nemotron 3.5 Lightning 30B-A3B | [ , ] | [ , ] | [ , ] | [ , ] | [ , ] | [ , ] | |
| Muse Glimmer 30B | [ , ] | [ , ] | [ , ] | [ , ] | [ , ] | [ , ] | |
| Model | Param. | GC Cur. | GC InF. | RAG Cur. | RAG InF. |
|---|---|---|---|---|---|
| Citation precision | |||||
| General | |||||
| Gemma 4 31B | 0.411 | 0.761 | 0.881 | 0.551 | 0.647 |
| Gemma 4 26B-A4B | 0.375 | 0.791 | 0.904 | 0.319 | 0.363 |
| Nemotron 3.5 Lightning 30B-A3B | 0.035 | 0.763 | 0.871 | 0.351 | 0.430 |
| Muse Glimmer 30B | 0.206 | 0.791 | 0.898 | 0.458 | 0.548 |
Legal QA
Legal question answering uses language models and legal sources to provide accurate, jurisdiction- and time-specific, verifiable answers.
See all 63 papers