Enhancing Legal Reasoning in Pre-trained Language Models via Knowledge Graph-Guided Multi-Task Pre-training

Ibrahim Siddig; Munir Georges

cs.CLOct 9, 2026

Enhancing Legal Reasoning in Pre-trained Language Models via Knowledge Graph-Guided Multi-Task Pre-training

Authors: Ibrahim Siddig, Munir Georges

Organizations: AImotion Bavaria, Technische Hochschule Ingolstadt

Abstract

Legal language understanding requires reasoning over semi-structured regulatory documents, yet standard pre-training objectives do not explicitly target this structural capability. Because current legal language models rely solely on masked language modeling, they treat statutory text as flat token sequences and ignore the underlying hierarchical constraints. To bridge this gap, we present a multi-task pre-training approach for German traffic law (StVO). Our approach combines three complementary objectives: Next Sentence Prediction (NSP), Knowledge Graph (KG) Guided Clustering (CLU), and Knowledge Graph (KG) Aware Paragraph Classification (CLS). This approach explicitly encodes structured legal knowledge directly into the model weights. This encoding enables zero-shot legal reasoning at inference time and removes the need for discrete graph traversal. We evaluate our approach across nine Transformer-based encoders and 324 experimental runs. On synthetic StVO question answering, our models demonstrate consistent relative improvements between 10.3% and 155.0% over baselines. Furthermore, relative to the established LexGLUE baselines (Chalkidis et al., 2022), our models improve over prior results on four of seven legal benchmarks. It also achieves consistent zero-shot transfer gains between 3.4% and 15.8% on Austrian driving regulations. These experimental results indicate that our training approach generalizes successfully beyond the initial training legal domain.

Figures & tables

KG embedding	m-F1
TransE	0.864
RotatE	0.532
ComplEx	0.392
DistMult	0.370

Table 1: Downstream StVO QA performance (m-F1) of the KG-guided model on bert-base-echr , with the pre-training configuration fixed and only the KG embedding varied. TransE yields the most robust downstream performance, motivating its use for the CLU soft targets.

Panel of Figure 1 — Figure 1: Architecture of the proposed KG-guided multi-task pre-training pipeline. German StVO legal text is parsed, translated via an LLM translator, and structurally aligned with a corresponding English-based knowledge graph to inject explicit relational constraints into model reasoning. A detailed visualization of the generated graph topology is provided in Appendix A .

Statistic	Train	Test
QA pairs	10,245	2,309
Unique paragraphs	49	10
Positive samples	50.1%	49.4%
Negative samples	49.9%	50.6%

Table 2: Synthetic StVO QA dataset statistics.

		Multi-task Pre-training on StVO (Zero-shot evaluations)						Fine-tuning
Model	Base	PT	PT	PT	PT	PT	PT	FT	FT
	$m\text{-}F_{1}$	$m\text{-}F_{1}$	Gain	$\mu\text{-}F_{1}$	Bal. Acc.	P	R	$m\text{-}F_{1}$	Gain
Top Performers ( $m\text{-}F_{1}>0.8$ )
bert-base-echr	0.566	0.864	$+$ 0.298	0.865	0.864	0.937	0.779	0.912	$+$ 0.048
bert-base-eurlex	0.338	0.862	$+$ 0.524	0.863	0.864	0.812	0.939	0.909	$+$ 0.047
bert-base-contracts	0.475	0.845	$+$ 0.370	0.846	0.848	0.784	0.950	0.905	$+$ 0.060

Table 3: Model performance on the synthetic StVO Q&A benchmark. Base :

m\text{-}F_{1}

with no StVO knowledge; PT : after multi-task pre-training; FT : after fine-tuning on a few downstream samples from the synthetic StVO Q&A benchmark. PT and FT Gain report absolute

m\text{-}F_{1}

improvement over Base and PT scores, respectively.

		Multi-task Pre-training on StVO (Zero-shot evaluations)						Fine-tuning
Model	Base	PT	PT	PT	PT	PT	PT	FT	FT
	$m\text{-}F_{1}$	$m\text{-}F_{1}$	Gain	$\mu\text{-}F_{1}$	Bal. Acc.	P	R	$m\text{-}F_{1}$	Gain
Successful Cross-Lingual Transfer
bert-base-eurlex	0.338	0.837	$+$ 0.499	0.837	0.837	0.832	0.839	0.869	$+$ 0.032
google-bert-base	0.425	0.773	$+$ 0.347	0.775	0.777	0.719	0.893	0.871	$+$ 0.099
bert-base-contracts	0.475	0.694	$+$ 0.219	0.701	0.699	0.776	0.555	0.878	$+$ 0.184

Table 4: The objectives (NSP+CLU) are pre-trained on non-translated German StVO . Models are evaluated on the synthetic StVO Q&A benchmark. Base :

m\text{-}F_{1}

before pre-training. PT columns report zero-shot performance after pre-training. PT Gain

=

m\text{-}F_{1}-

Base. FT columns report performance after fine-tuning on a few downstream samples from the synthetic StVO Q&A. FT Gain

=

-

PT. Bold marks the higher of Base and PT

m\text{-}F_{1}

per model.

Panel of Figure 2 — Figure 2: $m$ - $F_{1}$ gain of StVO multi-task pre-training over each base model on Austrian and Irish zero-shot driving exams. Austrian gains are consistently larger, reflecting greater regulatory overlap with German StVO . Legal-heBERT is the only model showing negative transfer on Irish ( $-$ 0.049).

Model	ECtHR-A		ECtHR-B		UNFAIR-ToS		CaseHOLD
Model	$\mu\text{-}F_{1}$	$m\text{-}F_{1}$	$\mu\text{-}F_{1}$	$m\text{-}F_{1}$	$\mu\text{-}F_{1}$	$m\text{-}F_{1}$	$F_{1}$
SOTA Published Baselines
TFIDF-SVM	62.6	48.9	73.0	63.8	94.7	75.0	22.4
BERT	71.2	63.6	79.7	73.4	95.6	81.3	70.8
RoBERTa	69.2	59.0	77.3	68.9	95.2	79.2	71.4
DeBERTa	70.0	60.8	78.8	71.0	95.5	80.3	72.6

Table 5: LexGLUE results on the four tasks where StVO pre-training improves over the published baselines ( Chalkidis et al., 2022a ) . (full seven-task results in Appendix D ). Scores are

F_{1}

(%): macro (

m

) and micro (

\mu

). Bold marks the best score per column;

\dagger

marks the improvement over the baselines of Chalkidis et al. (2022a) . CaseHOLD is reported as a single

F_{1}

(

\mu{=}m

Panel of Figure 3 — Figure 3: Task scaling across nine models. bert-base-echr ( $\star$ ) peaks at full-task synergy (0.864). ( $\dagger$ ) models show negative CLU interference when added to NSP+CLS. Legal-heBERT ( $\ddagger$ ) degrades monotonically.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Partial visual topology of the generated StVO knowledge graph (554 nodes, 945 edges). Every node has tuples that correspond to its unique IDs, and the node content incorporates the related legal knowledge

Hyperparameter	Value
Scoring function	$-\lVert\mathbf{e}_{h}+\mathbf{r}-\mathbf{e}_{t}\rVert_{2}$
Embedding dimension $d$	500
Distance norm $p$	2 (L2)
Margin $\gamma$	1.0
Loss	Margin ranking
Negative sampling	Uniform (head + tail)

Appendix

Table 6: TransE hyperparameters for the StVO

KG

paragraph embedding.

Hyperparameter	Value
Modeling type	Masked (NSP core)
KG embedding source	TransE, $d=500$
Optimizer	AdamW
Learning rate	$5\times 10^{-5}$
LR scheduler	Linear w/ warmup
warmup ratio	0.10

Appendix

Table 7: Multi-task pre-training hyperparameters, shared across all base encoders.

Base Model	ALL	NSP+CLS	CLU+CLS	NSP+CLU	CLS only	CLU only	NSP only	$\Delta$ (w/o CLS)
bert-base-echr	0.864	0.688	0.826	0.336	0.587	0.331	0.336	$-$ 0.528
bert-base-eurlex	0.862	0.833	0.831	0.457	0.675	0.336	0.336	$-$ 0.405
bert-base-contracts	0.797	0.845	0.805	0.390	0.643	0.331	0.331	$-$ 0.407
italian-legal-bert	0.813	0.806	0.651	0.500	0.537	0.331	0.331	$-$ 0.313
custom-legalbert	0.820	0.684	0.435	0.349	0.522	0.336	0.336	$-$ 0.471
google-bert-base	0.620	0.630	0.559	0.422	0.480	0.331	0.336	$-$ 0.198

Appendix

Table 8: Complete ablation: 9 models

\times

7 task configurations. Bold = best per model.

\Delta

(w/o CLS) = ALL

-

NSP+CLU.

Panel of Figure 5 — Figure 5: Ablation study: CLS dominates (avg. drop $=0.277$ ). bert-base-contracts and google-bert-base show negative CLU interference, confirming task synergy is architecture dependent.

Model	ECtHR-A		ECtHR-B		SCOTUS		EUR-LEX		LEDGAR		UNFAIR-ToS		CaseHOLD
Model	$\mu\text{-}F_{1}$	$m\text{-}F_{1}$	$\mu\text{-}F_{1}$	$m\text{-}F_{1}$	$\mu\text{-}F_{1}$	$m\text{-}F_{1}$	$\mu\text{-}F_{1}$	$m\text{-}F_{1}$	$\mu\text{-}F_{1}$	$m\text{-}F_{1}$	$\mu\text{-}F_{1}$	$m\text{-}F_{1}$	$\mu$ /m $\text{-}F_{1}$
Published Baselines ( Chalkidis et al., 2022a )
TFIDF-SVM	62.6	48.9	73.0	63.8	74.0	64.4	63.4	47.9	87.0	81.4	94.7	75.0	22.4
BERT	71.2	63.6	79.7	73.4	68.3	58.3	71.4	57.2	87.6	81.8	95.6	81.3	70.8
RoBERTa	69.2	59.0	77.3	68.9	71.6	62.0	71.9	57.9	87.9	82.3	95.2	79.2	71.4
DeBERTa	70.0	60.8	78.8	71.0	71.1	62.7	72.1	57.4	88.2	83.1	95.5	80.3	72.6

Appendix

Table 9: Full LexGLUE results across all seven tasks: our StVO -pretrained models vs. the published baselines of Chalkidis et al. (2022a) . Scores are

F_{1}

(%), micro (

\mu

) and macro (

m

); CaseHOLD is a single

F_{1}

(

\mu{=}m

). Bold marks the best score per column;

\dagger

marks an increase over the published baselines. Our models improve on four tasks (ECtHR-A, ECtHR-B, UNFAIR-ToS, CaseHOLD). The models show negative transfer on SCOTUS, EUR-LEX, and LEDGAR.

Explore similar work

LegalDrill: Diagnosis-Driven Synthesis for Legal Reasoning in Small Language Models
Apr 26, 2026Tianchun Li, Haochen Liu, Vishwa Pardeshi +5Synthetic Data Generation Small Language Models
Purdue University, West Lafayette, IN, USA · Fidelity Investments, Boston, MA, USA
WisdomInterrogatory (LuWen): An Open-Source Legal Large Language Model Technical Report
Apr 8, 2026Yiquan Wu, Yuhang Liu, Yifei Liu +4Legal NLP Legal Judgment Prediction
Zhejiang University, Hangzhou, China
Reasoning Before Translation: Enhancing Legal Machine Translation with Structured Reasoning
Jul 21, 2026Aixiu An, Michael Jungo, Eloi Eynard +4RL for Language Models Legal NLP
iCoSys, University of Applied Sciences and Arts Western Switzerland, Fribourg, Switzerland · AIBEX, University of Fribourg, Fribourg, Switzerland · Neur.on, Fribourg, Switzerland +1

Apr 26, 2026J/K move · Enter open · S save

LegalDrill: Diagnosis-Driven Synthesis for Legal Reasoning in Small Language Models

Tianchun Li, Haochen Liu, Vishwa Pardeshi, Xingchen Wang+4

Purdue University, West Lafayette, IN, USA · Fidelity Investments, Boston, MA, USA

Synthetic Data Generation Small Language Models Language Model Distillation Synthetic Data Generation for Language Models Legal Reasoning

Open paper AskPDF arXiv

Small language models (SLMs) are promising for real-world deployment due to their efficiency and low operational cost. However, their limited capacity struggles with high-stakes legal reasoning tasks that require coherent statute interpretation and logically consistent deduction. Furthermore, training SLMs for such tasks demands high-quality, concise reasoning trajectories, which are prohibitively expensive to manually collect and difficult to curate via standard rejection sampling, lacking granularity beyond final verdicts. To address these challenges, we propose {LegalDrill}, a diagnosis-driven synthesis framework that extracts and iteratively refines reasoning trajectories from a capable teacher via fine-grained prompting, then a self-reflective verification is employed to adaptively select the most effective data for the SLM student. The resulting data empower SLM training through supervised fine-tuning and direct preference optimization. Extensive experiments on several legal benchmarks demonstrate that {LegalDrill} significantly bolsters the legal reasoning capabilities of representative SLMs while bypassing the need for scarce expert annotations, paving a scalable path toward practical legal reasoning systems.

Abstract

Explore similar work

LegalDrill: Diagnosis-Driven Synthesis for Legal Reasoning in Small Language Models

WisdomInterrogatory (LuWen): An Open-Source Legal Large Language Model Technical Report

Reasoning Before Translation: Enhancing Legal Machine Translation with Structured Reasoning