XLaw-JP-27B: A Lightweight Open Model Specialized for Japanese Law

Andrew Shin

cs.CLOct 29, 2026

XLaw-JP-27B: A Lightweight Open Model Specialized for Japanese Law

Authors: Andrew Shin

Organizations: Keio University · X-Law, Inc.

Abstract

We present XLaw-JP-27B, an open-weight model specialized for Japanese law that runs on a single 48,GB GPU under the Apache2.0 licence. It adapts Qwen3.8-27B with hybrid retrieval over all current statutes and 31k case summaries, a light QLoRA fine-tune on its own verified answers to past bar-exam questions (self-distillation), a contract-specific LoRA adapter, and four-sample voting. We evaluate it on eight Japanese legal tasks---the multiple-choice section of the bar examination in 2024, 2025 and 2026, statement-level bar questions (JBE-QA), statute questions (LawQA-JP), the patent attorney exam, and contract review and drafting built from government model contracts---against open models of 12--33B under identical prompts and retrieval. XLaw-JP-27B ranks first on six tasks, ties on one and is third on one (mean rank 1.3 against 2.8--3.1 for the baselines). It is the only system that passes the bar exam in all three years (129, 120 and 130 of 175, meeting every subject minimum; no test-year material was used in training), while open Japanese 32--33B models fail at least one year. A controlled study of each component shows that retrieval, voting and self-distillation help, and that thinking mode, retrieval-grounded synthetic tuning, self-verification, per-statement decomposition and more self-distillation do not. The largest remaining gaps to frontier models are in contract review and patent law. The model is available as a free web service at https://x-law.jp/ with the weights available at https://huggingface.co/X-LawJapan/xlaw-jp-27b under Apache2.0.

Figures & tables

Benchmark	Items	Metric	Test material
司法試験短答式 2024	77	points/175	held out
司法試験短答式 2025	77	points/175	held out
司法試験短答式 2026	77	points/175	held out
JBE-QA	3,464	MCC, F1	2015–24
LawQA-JP	140	accuracy	statutes
Benrishi-MC (弁理士)	120	accuracy	2024–25

Table 1: Evaluation suite. Training uses only 2019–2023 exam answers produced by the model itself, contract material from the training half of the OI contracts, and other government model contracts. JBE-QA overlaps the 2019–2023 papers (and contains 2024), so we report it for comparison with the literature, not as a clean test.

	2019	2020	2021	2022	2023	2024	2026
closed-book	94	79	89	86	69	79	88
+ rag5	111	104	131	104	111	113	104

Table 2: Untuned Qwen3.8-27B by exam year (seed 0, /175). No trend towards older, longer-public papers.

	rag4	rag5
Statute recall (silver set)	0.395	0.502
Questions with $\geq$ 1 silver article	0.844	0.921
Statutes in context per question	3.0	4.9
2026: mean of 4 runs / worst / vote@4	103.8 / 94 / 126	108.3 / 104 / 118
2024: mean of 4 runs / worst / vote@4	96.8 / 88 / 108	111.8 / 103 / 123
Unanimous questions (2026)	26/77	35/77

Table 3: Retriever ablation (untuned model). rag5 raises every single run above the pass mark; its runs agree more, so voting adds less, and the two years’ votes move in opposite directions—a reminder that single-paper comparisons of one vote are noisy.

	Bar exam (points /175)			Benrishi-MC	LawQA-JP	JBE-QA §	OI-Contract		Mean
Model (same prompts and rag5 retrieval)	2024	2025	2026	acc.	acc.	MCC	Review	Draft	rank
XLaw-JP-27B (vote@4; contract adapter)	129	120	130	0.517	0.821	0.537	0.240	0.829	1.3
single run (mean of 4)	115.0	105.0	115.8	0.473	0.818	0.537	0.240	0.829	1.4
Qwen3.8-27B base (vote@4)	123	–	118	0.483	0.800	0.515	0.240	0.816
Gemma 4 12B	90 ‡	87	101	0.350	0.821	0.534	0.228	0.794	2.8
LLM-jp-4.1 33B (thinking)	100	88	92 ‡	0.383	0.793	0.600	0.169	0.759	2.8

Table 4: Eight Japanese legal tasks, same prompts and retrieval for every open model; baselines one run each. Mean rank over the eight task columns among the four systems above the line (ties averaged). ‡ below the pass mark or a subject minimum. § the same 200 JBE-QA items. ∗ no retrieval. Differences of a few points on one exam paper, and the OI-Contract Review gap to Gemma, are within run-to-run noise (§ 3 ).

System (one run unless noted)	2024	2025	2026
Pass mark	93	81	99
Gemma 4 12B, closed-book	69	–	76
Gemma 4 12B + rag4	83	–	78
Gemma 4 12B + rag5	90	87	101
LLM-jp-4.1 33B (thinking) + rag5	100	88	92
Qwen3-Swallow-32B (thinking) + rag5	96	85 †	82

Table 5: Bar exam multiple-choice section, points out of 175, official scoring. All Qwen rows share the base model. Our final system passes all three years including every subject minimum. 2025 was added for the final system and the baselines only (–: not run). † fails a subject minimum.

Figure 1: Four single runs (dots) and their vote (stars) per system. Seed spread is 10–16 points; voting adds 3–22 points over the mean run, least for systems whose runs already agree (E17).

Figure 2: Share of points by subject, 2024 and 2026 combined (all systems with rag5; baselines one run).

	JBE-QA MCC	JBE-QA F1	LawQA-JP	弁理士	判例 ROUGE-L	JSQuAD EM
Qwen3.8-27B, closed-book	0.307	0.634	0.580 †	0.300	–	0.875
Qwen3.8-27B + rag5, vote@4	0.586	0.777	0.800	0.483	0.469 ∗	–
E8 + rag5, vote@4 (ours)	0.615	0.785	0.821	0.517	0.514 ∗	0.877 ∗
E8 + gold articles (oracle)	–	–	0.943	–	–	–
Gemma 4 12B + rag5	0.598	0.786	0.821	0.350
LLM-jp-4.1 33B (thinking) + rag5	0.600 §	0.800 §	0.793	0.383

Table 6: Other benchmarks. † mean of 4 seeds; ∗ seed 0, no retrieval. 弁理士 is accuracy over 2024–25 (pass

\approx

0.65). JBE-QA closed-book uses all 3,464 items; retrieval rows the 3,460 items in the rag5 cache. § the same random 200 JBE-QA items (thinking baselines are 10–20

\times

slower). Baselines: one run, same prompts and contexts as ours. Literature rows are for orientation only: their setting differs (no retrieval).

Held-out (OI-Contract)	Review	Draft
Qwen3.8-27B	0.240	0.816
E8	0.220	0.769
Qwen3.8-27B + statute retrieval	0.270
E8 + statute retrieval	0.241
Issue-injection SFT (E17)	0.242	0.810
E8 + contract adapter (E17c)	0.240	0.829

Table 7: Contract review (checklist recall) and drafting (term-sheet row score). Opus also grades, so its own scores may be inflated by self-preference.

Technique	Measured on	without	with
Thinking mode (16k)	2026 + rag4, one run	88	58
	JBE-QA + rag4 (400), MCC	0.606	0.710
Retrieval-grounded SFT (15k)	2026 + rag4, one run	105	70
Self-verification	vote@4, 2026 / 2024	126 / 108	126 / 102
Statement decomposition	E8, 2026 / 2024	130 / 129 ∗	116 / 120
6 $\times$ self-distillation	vote@4, 2026 / 2024	130 / 129	123 / 123

Table 8: Negative results. Points /175 unless noted. ∗ vote@4; decomposition uses five samples per statement, comparable compute.

Legal NLP

Legal natural language processing (NLP) develops methods for analyzing and generating statutes, cases, contracts, and legal advice.

See all 112 papers

Expert Evaluation of LLM's Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Task
Apr 26, 2026Jungmin Choi, Keisuke Sakaguchi, Hiroaki YamadaLLM Evaluation Hallucination in Language Models
RIKEN · Tohoku University · Tokyo Metropolitan University
TW-LegalBench: Measuring Taiwanese Legal Understanding
Jun 17, 2026Fei-Yueh Chen, Chun Huang Lin, Chan Wei Hsu +4LLM Evaluation Legal Judgment Prediction
University of Rochester Rochester, NY, USA · National Taiwan University Taipei, Taiwan · NVIDIA Santa Clara, CA, USA
LexRubric: A Rubric-Guided Diagnostic Benchmark for Open-Ended Legal Tasks
Jun 8, 2026Yifan Chen, Haitao Li, Yiran Hu +6LLM Evaluation Rubric-Based Evaluation
1Beijing University of Posts and Telecommunications · 2Tsinghua University · University of Waterloo +1

Apr 26, 2026J/K move · Enter open · S save

Expert Evaluation of LLM's Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Task

Jungmin Choi, Keisuke Sakaguchi, Hiroaki Yamada

RIKEN · Tohoku University · Tokyo Metropolitan University

LLM Evaluation Hallucination in Language Models Legal Reasoning

Open paper AskPDF arXiv

Large language models (LLMs) have shown strong performance on legal benchmarks, including multiple-choice components of bar exams. However, their capacity for generating open-ended legal reasoning in realistic scenarios remains insufficiently explored. Notably, to our best knowledge, there are no prior studies or datasets addressing this issue in the Japanese context. This study presents the first dataset designed to evaluate the open-ended legal reasoning performance of LLMs within the Japanese jurisdiction. The dataset is based on the writing component of the Japanese bar examination, which requires examinees to identify multiple legal issues from long narratives and to construct structured legal arguments in free text format. Our key contribution is the manual evaluation of LLMs' generated responses by legal experts, which reveals limitations and challenges in legal reasoning. Moreover, we conducted a manual analysis of hallucinations to characterize when and how the models introduce content not supported by precedent or law. Our real exam questions, model-generated responses, and expert evaluations reveal the milestones of current LLMs in the Japanese legal domain. Our dataset and relevant resources will be available online.

Abstract

Legal NLP

Explore similar work

Expert Evaluation of LLM's Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Task

TW-LegalBench: Measuring Taiwanese Legal Understanding

LexRubric: A Rubric-Guided Diagnostic Benchmark for Open-Ended Legal Tasks