Legal2LogicICL: Improving Generalization in Transforming Legal Cases to Logical Formulas via Diverse Few-Shot Learning

Jieying Xue; Phuong Minh Nguyen; Ha Thanh Nguyen; May Myo Zin; Ken Satoh

cs.CLApr 13, 2026

Legal2LogicICL: Improving Generalization in Transforming Legal Cases to Logical Formulas via Diverse Few-Shot Learning

Authors: Jieying Xue, Phuong Minh Nguyen, Ha Thanh Nguyen, May Myo Zin, Ken Satoh

Organizations: Center for Juris-Informatics, ROIS-DS, Tokyo, Japan · Japan Advanced Institute of Science and Technology, Ishikawa, Japan

Abstract

This work aims to improve the generalization of logic-based legal reasoning systems by integrating recent advances in NLP with legal-domain adaptive few-shot learning techniques using LLMs. Existing logic-based legal reasoning pipelines typically rely on fine-tuned models to map natural-language legal cases into logical formulas before forwarding them to a symbolic reasoner. However, such approaches are heavily constrained by the scarcity of high-quality annotated training data. To address this limitation, we propose a novel LLM-based legal reasoning framework that enables effective in-context learning through retrieval-augmented generation. Specifically, we introduce Legal2LogicICL, a few-shot retrieval framework that balances diversity and similarity of exemplars at both the latent semantic representation level and the legal text structure level. In addition, our method explicitly accounts for legal structure by mitigating entity-induced retrieval bias in legal texts, where lengthy and highly specific entity mentions often dominate semantic representations and obscure legally meaningful reasoning patterns. Our Legal2LogicICL constructs informative and robust few-shot demonstrations, leading to accurate and stable logical rule generation without requiring additional training. In addition, we construct a new dataset, named Legal2Proleg, which is annotated with alignments between legal cases and PROLEG logical formulas to support the evaluation of legal semantic parsing. Experimental results on both open-source and proprietary LLMs demonstrate that our approach significantly improves accuracy, stability, and generalization in transforming natural-language legal case descriptions into logical representations, highlighting its effectiveness for interpretable and reliable legal reasoning. Our code is available at https://github.com/yingjie7/Legal2LogicICL.

Figures & tables

Content Example - (denotaion)
Legal Issue
The borrower’s use of the assets in a manner that harms the lender’s interests or reputation has damaged the lender’s rights and reputation.
Legal Case - ( $l^{query}$ )
Medical supplies were given to the hospital by the health organization as part of a supply agreement, intended for patient treatment. Instead, the hospital redistributed the supplies to external clinics without consent. This resulted in a shortage of supplies during a critical time, affecting patient care. These actions were discovered on 2023/08/20. Does the health organization have grounds for legal action to protect their reputation?
Entities Set (or Slot Holders) - $(\mathcal{E}(l^{query}))$
{ "Borrower" : "The hospital" , "Object" : "medical supplies" , "Lender" : "the health organization" , "Agreement" : "a supply agreement" , "Harm" : "a shortage of supplies during a critical time, affecting patient care" , "T_discovery" : "2023/08/20" }

Table 1. An Example Legal Case and Its Structured Representation in the Dataset Legal2Proleg

Dataset	LegalCaseNER	Legal2ProlegV0	Legal2Proleg
Source	Zin et al. (2023)	Phuong et al. (2026)	(this work)
#Samples ( $\times 1000$ )	$\approx$ 6.3	$\approx$ 5	$\approx$ 1
#Templates	947	240	1057
#Template Vocab.	264	907	2897
#Entity Types	13	27	45
#Legal Issues	4	9	24

Table 2. Statistics of the Legal2Proleg dataset and others.

Figure 1. Overview of the proposed diversity-aware hybrid few-shot in-context learning framework.

### You are an expert in the Semantic parsing task, which maps from legal cases to logical formulas (Note: following the exact function name defined in the fewshot samples).

### Input: {{ Selected legal case,

l_{i}\in\mathcal{S}_{few}

}}

### Logical Formulas Template: {{

\textrm{template}(f_{i})

where

f_{i}\in\mathcal{S}_{few}

}}

### Output: {{ Logical facts,

f_{i}\in\mathcal{S}_{few}

}}

…

### Input: {{ Query legal case,

l^{query}

}}

Table 3. Prompting template combining case-based and template-based demonstrations. The blue text indicates content that is replaced with data from the selected few-shot set,

\mathcal{S}_{\text{few}}

, while the red text denotes the content to be generated by the LLM.

Panel of Figure 2 — Figure 2. Performance comparison between NER-based method LegalCaseNER and few-shot learning methods Legal2LogicICL on the Legal2Proleg dataset with respect to the portion of seen data. Results are averaged over five random seeds, corresponding to the shaded lines.

Panel of Figure 3 — Figure 3. Performance comparison on the effect of the diversity control parameter ( $\lambda$ ) across diverse LLMs under identical ICL experimental settings (seen data rate $=0.6$ ). Results are averaged over five random seeds for each setting.

$k$ -shot	$\lambda$	Qwen3-8B	Llama-3.1-8B	Qwen3-14B	Phi-4
3c	0.6	0.7075	0.7824	0.8437	0.8894
5c	1.0	0.7899	0.8171	0.8930	0.9141
5c	0.6	0.8005	0.8246	0.8955	0.9216
3c+3t	0.6	0.8472	0.8327	0.9216	0.9412

Table 4. Ablation study on the impact of different

k

-shot strategies under identical seen data rate (

=0.6

Seen data	Legal2LogicICL	Legal2LogicICL	(SFT) LegalCaseNER
	( Qwen3-8B )	( Qwen3-14B )	( Roberta-large )
40%	0.7973	0.9000	0.9957
60%	0.8353	0.8975	0.9988
80%	0.8466	0.8981	0.9953

Table 5. Impact of entity name bias on model performance.

Figure 4. Visualization of our ranking method (DiverseSim) with

\lambda=0.6

. Each data point represents a sample in the Legal2Proleg dataset. The highlighted points indicate the few-shot exemplars selected by DiverseSim and by cosine similarity with respect to the query, respectively.

Case	Content
Query Case	Artist Jordan admires musician Alex and habitually listened to Alex’s composition. Jordan released a new composition by adjusting the vocals while maintaining the hook, sharing it as his own on SoundCloud. Eventually, Alex acknowledged, “What an inventive version!”, causing the composition to exceed one million plays.
Exemplar	Semantic-based Retrieved Exemplars	Template-based Retrieved Exemplars
Example_1	Inspired by Alex, Jordan who habitually listened to Alex’s composition, created a new melody by adjusting the vocals while maintaining the hook. Posting it on his Facebook page, Alex acknowledged, “What an inventive version!”, it soon attracted a million likes.	Musician V is admired by person U and often vibed to V’s rhythm. U crafted a fresh tune by preserving the tempo but altering the arrangement, subsequently sharing it on Instagram as U’s composition. Eventually, V said, “This is captivating,”, leading the track to amass over two million streams.
Example_2	Musician Casey has a fan named Chris who played regularly Casey’s tune. Chris created a new track by changing the melody slightly but keeping the hook and uploaded it as Chris’s own on Instagram. Subsequently, Casey responded, “Amazing interpretation!”, leading to the track amassing a million streams.	T is a fan of musician U and heavily sampled U’s piece. T crafted a new tune while keeping the vocal style retained but changed the tempo. T shared this on his SoundCloud as original.
Example_3	Artist Jamie has a follower in Sam who practiced daily Jamie’s single. Sam composed a new melody by modifying the chords but keeping the pre-chorus and published it on TikTok as Sam’s piece. Finally, Jamie expressed, “Fantastic transformation!”, resulting in the track garnering over three million plays.	Inspired by Alex, Jordan who habitually listened to Alex’s composition, created a new melody by adjusting the vocals while maintaining the hook. Posting it on his Facebook page, Alex acknowledged, “What an inventive version!”, it soon attracted a million likes.

Table 6. Qualitative comparison of few-shot exemplars retrieved by semantic-only and template-aware strategies.

(Prediction)

(Lucas)[Lender] provided (the)[Asset] conference room to (Mia)[Borrower] under (conference-rental8)[Agreement] for enhancing their product offerings. However, (Mia)[Borrower] did not utilize …, (Lucas’s)[Lender] reputation was compromised as clients began avoiding (the venue)[Object] due to the unexpected activities. The actions of (Mia)[Borrower] came to light on (2023/06/30)[T_due] .

(Gold Ouput)

(Lucas)[Lender] provided (the conference room)[Object] to (Mia)[Borrower] under (conference-rental8)[Agreement] for enhancing their product offerings. However, (Mia)[Borrower] did not utilize …, (Lucas’s reputation was compromised as clients began avoiding the venue due to the unexpected activities)[Harm] . The actions of (Mia)[Borrower] came to light on (2023/06/30)[T_discovery] .

Table 7. Representative Errors of LegalCaseNER (incorrect predictions in red, corresponding gold entities in blue)

(Prediction)

borrower(’Mason’). lender(’Emma’). owned_by( ’the laptop’ , ’Emma’). borrowing_agreement(’lease78’). damage_fact(’Mason’, ’the laptop’ ). repair_payment_request_fact(’Mason’,’Emma’, ’the laptop’ ). repair_request_fact(’Mason’,’Emma’, ’the laptop’ , ’2024/01/10’). demob :- block(right_to_dispute_repair_demand(’Emma’,’Mason’, ’the laptop’ , ’lease78’)).

(Gold)

borrower(’Mason’). lender(’Emma’). owned_by( ’a laptop’ , ’Emma’). borrowing_agreement(’lease78’). damage_fact(’Mason’, ’a laptop’ ). repair_payment_request_fact(’Mason’,’Emma’, ’a laptop’ ). repair_request_fact(’Mason’,’Emma’, ’a laptop’ , ’2024/01/10’). demob :- block(right_to_dispute_repair_demand(’Emma’,’Mason’, ’a laptop’ , ’lease78’)).

Table 8. Representative Errors of Legal2LogicICL (incorrect entities in red, corresponding gold entities in blue)

Eval.Metric	Qwen3-8B	Llama-3.1-8B	Qwen3-14B	Phi-4
Exact-Match Acc.	84.72	83.27	92.16	94.12
Soft-Match Acc.	92.18 (+7.46)	97.68 (+14.44)	97.52 (+4.91)	98.00 (+3.88)

Table 9. Comparison on two evaluation metrics, Exact-Match Acc. and Soft-Match Acc., under settings of

3c+3t

and

\lambda=0.6

Explore similar work

LegalDrill: Diagnosis-Driven Synthesis for Legal Reasoning in Small Language Models
Apr 26, 2026Tianchun Li, Haochen Liu, Vishwa Pardeshi +5Synthetic Data Generation Small Language Models
Purdue University, West Lafayette, IN, USA · Fidelity Investments, Boston, MA, USA
LEGO: Synergizing Expert GraphRAG and Expert Chain-of-Thought for Legal Reasoning
Sep 22, 2026Qingjing Chen, Junkai Zhang, Shaochun Wang +7LLM Reasoning with Graphs Graph Retrieval-Augmented Generation
Alma AI, University of Bologna, Italy · Tsinghua University, China · Modelbest Inc. +2
NOWJ@COLIEE 2026: Adaptive Pipelines for Legal Retrieval and Reasoning
Jul 18, 2026Thuong-Hieu Ngo, Hoang-Trung Nguyen, Huu-Dong Nguyen +5Legal NLP Legal IR
VNU University of Engineering and Technology Hanoi, Vietnam · Center for Juris-Informatics, ROIS-DS College of Engineering & Computer Science, VinUniversity Hanoi, Vietnam

Apr 26, 2026J/K move · Enter open · S save

LegalDrill: Diagnosis-Driven Synthesis for Legal Reasoning in Small Language Models

Tianchun Li, Haochen Liu, Vishwa Pardeshi, Xingchen Wang+4

Purdue University, West Lafayette, IN, USA · Fidelity Investments, Boston, MA, USA

Synthetic Data Generation Small Language Models Language Model Distillation Synthetic Data Generation for Language Models Legal Reasoning

Open paper AskPDF arXiv

Small language models (SLMs) are promising for real-world deployment due to their efficiency and low operational cost. However, their limited capacity struggles with high-stakes legal reasoning tasks that require coherent statute interpretation and logically consistent deduction. Furthermore, training SLMs for such tasks demands high-quality, concise reasoning trajectories, which are prohibitively expensive to manually collect and difficult to curate via standard rejection sampling, lacking granularity beyond final verdicts. To address these challenges, we propose {LegalDrill}, a diagnosis-driven synthesis framework that extracts and iteratively refines reasoning trajectories from a capable teacher via fine-grained prompting, then a self-reflective verification is employed to adaptively select the most effective data for the SLM student. The resulting data empower SLM training through supervised fine-tuning and direct preference optimization. Extensive experiments on several legal benchmarks demonstrate that {LegalDrill} significantly bolsters the legal reasoning capabilities of representative SLMs while bypassing the need for scarce expert annotations, paving a scalable path toward practical legal reasoning systems.

Abstract

Explore similar work

LegalDrill: Diagnosis-Driven Synthesis for Legal Reasoning in Small Language Models

LEGO: Synergizing Expert GraphRAG and Expert Chain-of-Thought for Legal Reasoning

NOWJ@COLIEE 2026: Adaptive Pipelines for Legal Retrieval and Reasoning