Legal language understanding requires reasoning over semi-structured regulatory documents, yet standard pre-training objectives do not explicitly target this structural capability. Because current legal language models rely solely on masked language modeling, they treat statutory text as flat token sequences and ignore the underlying hierarchical constraints. To bridge this gap, we present a multi-task pre-training approach for German traffic law (StVO). Our approach combines three complementary objectives: Next Sentence Prediction (NSP), Knowledge Graph (KG) Guided Clustering (CLU), and Knowledge Graph (KG) Aware Paragraph Classification (CLS). This approach explicitly encodes structured legal knowledge directly into the model weights. This encoding enables zero-shot legal reasoning at inference time and removes the need for discrete graph traversal. We evaluate our approach across nine Transformer-based encoders and 324 experimental runs. On synthetic StVO question answering, our models demonstrate consistent relative improvements between 10.3% and 155.0% over baselines. Furthermore, relative to the established LexGLUE baselines (Chalkidis et al., 2022), our models improve over prior results on four of seven legal benchmarks. It also achieves consistent zero-shot transfer gains between 3.4% and 15.8% on Austrian driving regulations. These experimental results indicate that our training approach generalizes successfully beyond the initial training legal domain.
Figures & tables
KG embedding
m-F1
TransE
0.864
RotatE
0.532
ComplEx
0.392
DistMult
0.370
Table 1: Downstream StVO QA performance (m-F1) of the KG-guided model on bert-base-echr , with the pre-training configuration fixed and only the KG embedding varied. TransE yields the most robust downstream performance, motivating its use for the CLU soft targets.
Figure 1: Architecture of the proposed KG-guided multi-task pre-training pipeline. German StVO legal text is parsed, translated via an LLM translator, and structurally aligned with a corresponding English-based knowledge graph to inject explicit relational constraints into model reasoning. A detailed visualization of the generated graph topology is provided in Appendix A .
Statistic
Train
Test
QA pairs
10,245
2,309
Unique paragraphs
49
10
Positive samples
50.1%
49.4%
Negative samples
49.9%
50.6%
Table 2: Synthetic StVO QA dataset statistics.
Multi-task Pre-training on StVO (Zero-shot evaluations)
Fine-tuning
Model
Base
PT
PT
PT
PT
PT
PT
FT
FT
m-F1
m-F1
Gain
μ-F1
Bal. Acc.
P
R
m-F1
Gain
Top Performers ( m-F1>0.8 )
bert-base-echr
0.566
0.864
+ 0.298
0.865
0.864
0.937
0.779
0.912
+ 0.048
bert-base-eurlex
0.338
0.862
+ 0.524
0.863
0.864
0.812
0.939
0.909
+ 0.047
bert-base-contracts
0.475
0.845
+ 0.370
0.846
0.848
0.784
0.950
0.905
+ 0.060
Table 3: Model performance on the synthetic StVO Q&A benchmark. Base : m-F1 with no StVO knowledge; PT : after multi-task pre-training; FT : after fine-tuning on a few downstream samples from the synthetic StVO Q&A benchmark. PT and FT Gain report absolute m-F1 improvement over Base and PT scores, respectively.
Multi-task Pre-training on StVO (Zero-shot evaluations)
Fine-tuning
Model
Base
PT
PT
PT
PT
PT
PT
FT
FT
m-F1
m-F1
Gain
μ-F1
Bal. Acc.
P
R
m-F1
Gain
Successful Cross-Lingual Transfer
bert-base-eurlex
0.338
0.837
+ 0.499
0.837
0.837
0.832
0.839
0.869
+ 0.032
google-bert-base
0.425
0.773
+ 0.347
0.775
0.777
0.719
0.893
0.871
+ 0.099
bert-base-contracts
0.475
0.694
+ 0.219
0.701
0.699
0.776
0.555
0.878
+ 0.184
Table 4: The objectives (NSP+CLU) are pre-trained on non-translated German StVO . Models are evaluated on the synthetic StVO Q&A benchmark. Base : m-F1 before pre-training. PT columns report zero-shot performance after pre-training. PT Gain = PT m-F1− Base. FT columns report performance after fine-tuning on a few downstream samples from the synthetic StVO Q&A. FT Gain = FT − PT. Bold marks the higher of Base and PT m-F1 per model.
Figure 2: m - F1 gain of StVO multi-task pre-training over each base model on Austrian and Irish zero-shot driving exams. Austrian gains are consistently larger, reflecting greater regulatory overlap with German StVO . Legal-heBERT is the only model showing negative transfer on Irish ( − 0.049).
Model
ECtHR-A
ECtHR-B
UNFAIR-ToS
CaseHOLD
μ-F1
m-F1
μ-F1
m-F1
μ-F1
m-F1
F1
SOTA Published Baselines
TFIDF-SVM
62.6
48.9
73.0
63.8
94.7
75.0
22.4
BERT
71.2
63.6
79.7
73.4
95.6
81.3
70.8
RoBERTa
69.2
59.0
77.3
68.9
95.2
79.2
71.4
DeBERTa
70.0
60.8
78.8
71.0
95.5
80.3
72.6
Table 5: LexGLUE results on the four tasks where StVO pre-training improves over the published baselines ( Chalkidis et al., 2022a ) . (full seven-task results in Appendix D ). Scores are F1 (%): macro ( m ) and micro ( μ ). Bold marks the best score per column; † marks the improvement over the baselines of Chalkidis et al. (2022a) . CaseHOLD is reported as a single F1 ( μ=m ).
Figure 3: Task scaling across nine models. bert-base-echr ( ⋆ ) peaks at full-task synergy (0.864). ( † ) models show negative CLU interference when added to NSP+CLS. Legal-heBERT ( ‡ ) degrades monotonically.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Partial visual topology of the generated StVO knowledge graph (554 nodes, 945 edges). Every node has tuples that correspond to its unique IDs, and the node content incorporates the related legal knowledge
Hyperparameter
Value
Scoring function
−∥eh+r−et∥2
Embedding dimension d
500
Distance norm p
2 (L2)
Margin γ
1.0
Loss
Margin ranking
Negative sampling
Uniform (head + tail)
Appendix
Table 6: TransE hyperparameters for the StVO KG paragraph embedding.
Hyperparameter
Value
Modeling type
Masked (NSP core)
KG embedding source
TransE, d=500
Optimizer
AdamW
Learning rate
5×10−5
LR scheduler
Linear w/ warmup
warmup ratio
0.10
Appendix
Table 7: Multi-task pre-training hyperparameters, shared across all base encoders.
Base Model
ALL
NSP+CLS
CLU+CLS
NSP+CLU
CLS only
CLU only
NSP only
Δ (w/o CLS)
bert-base-echr
0.864
0.688
0.826
0.336
0.587
0.331
0.336
− 0.528
bert-base-eurlex
0.862
0.833
0.831
0.457
0.675
0.336
0.336
− 0.405
bert-base-contracts
0.797
0.845
0.805
0.390
0.643
0.331
0.331
− 0.407
italian-legal-bert
0.813
0.806
0.651
0.500
0.537
0.331
0.331
− 0.313
custom-legalbert
0.820
0.684
0.435
0.349
0.522
0.336
0.336
− 0.471
google-bert-base
0.620
0.630
0.559
0.422
0.480
0.331
0.336
− 0.198
Appendix
Table 8: Complete ablation: 9 models × 7 task configurations. Bold = best per model. Δ (w/o CLS) = ALL − NSP+CLU.
Figure 5: Ablation study: CLS dominates (avg. drop =0.277 ). bert-base-contracts and google-bert-base show negative CLU interference, confirming task synergy is architecture dependent.
Model
ECtHR-A
ECtHR-B
SCOTUS
EUR-LEX
LEDGAR
UNFAIR-ToS
CaseHOLD
μ-F1
m-F1
μ-F1
m-F1
μ-F1
m-F1
μ-F1
m-F1
μ-F1
m-F1
μ-F1
m-F1
μ /m -F1
Published Baselines ( Chalkidis et al., 2022a )
TFIDF-SVM
62.6
48.9
73.0
63.8
74.0
64.4
63.4
47.9
87.0
81.4
94.7
75.0
22.4
BERT
71.2
63.6
79.7
73.4
68.3
58.3
71.4
57.2
87.6
81.8
95.6
81.3
70.8
RoBERTa
69.2
59.0
77.3
68.9
71.6
62.0
71.9
57.9
87.9
82.3
95.2
79.2
71.4
DeBERTa
70.0
60.8
78.8
71.0
71.1
62.7
72.1
57.4
88.2
83.1
95.5
80.3
72.6
Appendix
Table 9: Full LexGLUE results across all seven tasks: our StVO -pretrained models vs. the published baselines of Chalkidis et al. (2022a) . Scores are F1 (%), micro ( μ ) and macro ( m ); CaseHOLD is a single F1 ( μ=m ). Bold marks the best score per column; † marks an increase over the published baselines. Our models improve on four tasks (ECtHR-A, ECtHR-B, UNFAIR-ToS, CaseHOLD). The models show negative transfer on SCOTUS, EUR-LEX, and LEDGAR.
Small language models (SLMs) are promising for real-world deployment due to their efficiency and low operational cost. However, their limited capacity struggles with high-stakes legal reasoning tasks that require coherent statute interpretation and logically consistent deduction. Furthermore, training SLMs for such tasks demands high-quality, concise reasoning trajectories, which are prohibitively expensive to manually collect and difficult to curate via standard rejection sampling, lacking granularity beyond final verdicts. To address these challenges, we propose {LegalDrill}, a diagnosis-driven synthesis framework that extracts and iteratively refines reasoning trajectories from a capable teacher via fine-grained prompting, then a self-reflective verification is employed to adaptively select the most effective data for the SLM student. The resulting data empower SLM training through supervised fine-tuning and direct preference optimization. Extensive experiments on several legal benchmarks demonstrate that {LegalDrill} significantly bolsters the legal reasoning capabilities of representative SLMs while bypassing the need for scarce expert annotations, paving a scalable path toward practical legal reasoning systems.
Tianchun Li, Haochen Liu, Vishwa Pardeshi +5
Purdue University, West Lafayette, IN, USA · Fidelity Investments, Boston, MA, USA
Large language models have demonstrated remarkable capabilities across a wide range of natural language processing tasks, yet their application in the legal domain remains challenging due to the specialized terminology, complex reasoning requirements, and rapidly evolving legal knowledge involved. In this paper, we present WisdomInterrogatory (LuWen), an open-source Chinese legal language model built upon the Baichuan foundation model through three key techniques: continual pre-training on a large-scale legal corpus, supervised fine-tuning with carefully curated legal instruction data, and retrieval-augmented generation integrated with a comprehensive legal knowledge base. We evaluate LuWen on five representative legal tasks spanning both prediction and generation settings, including legal judgment prediction, judicial examination, legal text summarization, law article question answering, and judicial decision reasoning. Experimental results show that LuWen outperforms several strong baselines, demonstrating the effectiveness of our approach in adapting general-purpose language models to the legal domain.
Neural machine translation (NMT) in the legal domain is a linguistically and conceptually demanding task, primarily due to the complexity of legal language and the high level of precision it requires. The recent emergence of reasoning-capable language models opens new possibilities for tackling such challenges. They add to a set of other previously proposed techniques to enhance the translation quality, which includes supervised fine-tuning and reinforcement learning. In this work, we perform a comparison between these various approaches. More particularly, we evaluate small language models such as Qwen3.5 4B, Qwen3.5 9B, and Gemma 3 12B enhanced with various re-training paradigms and compare their performances against frontier reasoning models. We focus on the Swiss legal system, which -- with its unique multilingual statutes -- offers a particularly challenging testbed for reasoning-augmented models. Our results show that the quality of small ``base'' models can be greatly enhanced, and that reinforcement learning with verifiable rewards can be applied to NMT in the legal domain and surpasses the translation quality of supervised fine-tuning. The performance of enhanced small models is close to the one of state-of-the-art reasoning models yet remains inferior. We also note that re-training paradigms yield diminishing returns as model size increase. The code and models are publicly available at https://github.com/aixiuxiuxiu/Legal-MT-SFT-RL.
Aixiu An, Michael Jungo, Eloi Eynard +4
iCoSys, University of Applied Sciences and Arts Western Switzerland, Fribourg, Switzerland · AIBEX, University of Fribourg, Fribourg, Switzerland · Neur.on, Fribourg, Switzerland +1