This work aims to improve the generalization of logic-based legal reasoning systems by integrating recent advances in NLP with legal-domain adaptive few-shot learning techniques using LLMs. Existing logic-based legal reasoning pipelines typically rely on fine-tuned models to map natural-language legal cases into logical formulas before forwarding them to a symbolic reasoner. However, such approaches are heavily constrained by the scarcity of high-quality annotated training data. To address this limitation, we propose a novel LLM-based legal reasoning framework that enables effective in-context learning through retrieval-augmented generation. Specifically, we introduce Legal2LogicICL, a few-shot retrieval framework that balances diversity and similarity of exemplars at both the latent semantic representation level and the legal text structure level. In addition, our method explicitly accounts for legal structure by mitigating entity-induced retrieval bias in legal texts, where lengthy and highly specific entity mentions often dominate semantic representations and obscure legally meaningful reasoning patterns. Our Legal2LogicICL constructs informative and robust few-shot demonstrations, leading to accurate and stable logical rule generation without requiring additional training. In addition, we construct a new dataset, named Legal2Proleg, which is annotated with alignments between legal cases and PROLEG logical formulas to support the evaluation of legal semantic parsing. Experimental results on both open-source and proprietary LLMs demonstrate that our approach significantly improves accuracy, stability, and generalization in transforming natural-language legal case descriptions into logical representations, highlighting its effectiveness for interpretable and reliable legal reasoning. Our code is available at https://github.com/yingjie7/Legal2LogicICL.
Figures & tables
Content Example - (denotaion)
Legal Issue
The borrower’s use of the assets in a manner that harms the lender’s interests or reputation has damaged the lender’s rights and reputation.
Legal Case - ( lquery )
Medical supplies were given to the hospital by the health organization as part of a supply agreement, intended for patient treatment. Instead, the hospital redistributed the supplies to external clinics without consent. This resulted in a shortage of supplies during a critical time, affecting patient care. These actions were discovered on 2023/08/20. Does the health organization have grounds for legal action to protect their reputation?
Entities Set (or Slot Holders) - (E(lquery))
{ "Borrower" : "The hospital" , "Object" : "medical supplies" , "Lender" : "the health organization" , "Agreement" : "a supply agreement" , "Harm" : "a shortage of supplies during a critical time, affecting patient care" , "T_discovery" : "2023/08/20" }
Table 1. An Example Legal Case and Its Structured Representation in the Dataset Legal2Proleg
Dataset
LegalCaseNER
Legal2ProlegV0
Legal2Proleg
Source
Zin et al. (2023)
Phuong et al. (2026)
(this work)
#Samples ( ×1000 )
≈ 6.3
≈ 5
≈ 1
#Templates
947
240
1057
#Template Vocab.
264
907
2897
#Entity Types
13
27
45
#Legal Issues
4
9
24
Table 2. Statistics of the Legal2Proleg dataset and others.
Figure 1. Overview of the proposed diversity-aware hybrid few-shot in-context learning framework.
### You are an expert in the Semantic parsing task, which maps from legal cases to logical formulas (Note: following the exact function name defined in the fewshot samples).
### Input: {{ Selected legal case, li∈Sfew }}
### Logical Formulas Template: {{ template(fi) where fi∈Sfew }}
### Output: {{ Logical facts, fi∈Sfew }}
…
### Input: {{ Query legal case, lquery }}
Table 3. Prompting template combining case-based and template-based demonstrations. The blue text indicates content that is replaced with data from the selected few-shot set, Sfew , while the red text denotes the content to be generated by the LLM.
Figure 2. Performance comparison between NER-based method LegalCaseNER and few-shot learning methods Legal2LogicICL on the Legal2Proleg dataset with respect to the portion of seen data. Results are averaged over five random seeds, corresponding to the shaded lines.
Figure 3. Performance comparison on the effect of the diversity control parameter ( λ ) across diverse LLMs under identical ICL experimental settings (seen data rate =0.6 ). Results are averaged over five random seeds for each setting.
k -shot
λ
Qwen3-8B
Llama-3.1-8B
Qwen3-14B
Phi-4
3c
0.6
0.7075
0.7824
0.8437
0.8894
5c
1.0
0.7899
0.8171
0.8930
0.9141
5c
0.6
0.8005
0.8246
0.8955
0.9216
3c+3t
0.6
0.8472
0.8327
0.9216
0.9412
Table 4. Ablation study on the impact of different k -shot strategies under identical seen data rate ( =0.6 ).
Seen data
Legal2LogicICL
Legal2LogicICL
(SFT) LegalCaseNER
( Qwen3-8B )
( Qwen3-14B )
( Roberta-large )
40%
0.7973
0.9000
0.9957
60%
0.8353
0.8975
0.9988
80%
0.8466
0.8981
0.9953
Table 5. Impact of entity name bias on model performance.
Figure 4. Visualization of our ranking method (DiverseSim) with λ=0.6 . Each data point represents a sample in the Legal2Proleg dataset. The highlighted points indicate the few-shot exemplars selected by DiverseSim and by cosine similarity with respect to the query, respectively.
Case
Content
Query Case
Artist Jordan admires musician Alex and habitually listened to Alex’s composition. Jordan released a new composition by adjusting the vocals while maintaining the hook, sharing it as his own on SoundCloud. Eventually, Alex acknowledged, “What an inventive version!”, causing the composition to exceed one million plays.
Exemplar
Semantic-based Retrieved Exemplars
Template-based Retrieved Exemplars
Example_1
Inspired by Alex, Jordan who habitually listened to Alex’s composition, created a new melody by adjusting the vocals while maintaining the hook. Posting it on his Facebook page, Alex acknowledged, “What an inventive version!”, it soon attracted a million likes.
Musician V is admired by person U and often vibed to V’s rhythm. U crafted a fresh tune by preserving the tempo but altering the arrangement, subsequently sharing it on Instagram as U’s composition. Eventually, V said, “This is captivating,”, leading the track to amass over two million streams.
Example_2
Musician Casey has a fan named Chris who played regularly Casey’s tune. Chris created a new track by changing the melody slightly but keeping the hook and uploaded it as Chris’s own on Instagram. Subsequently, Casey responded, “Amazing interpretation!”, leading to the track amassing a million streams.
T is a fan of musician U and heavily sampled U’s piece. T crafted a new tune while keeping the vocal style retained but changed the tempo. T shared this on his SoundCloud as original.
Example_3
Artist Jamie has a follower in Sam who practiced daily Jamie’s single. Sam composed a new melody by modifying the chords but keeping the pre-chorus and published it on TikTok as Sam’s piece. Finally, Jamie expressed, “Fantastic transformation!”, resulting in the track garnering over three million plays.
Inspired by Alex, Jordan who habitually listened to Alex’s composition, created a new melody by adjusting the vocals while maintaining the hook. Posting it on his Facebook page, Alex acknowledged, “What an inventive version!”, it soon attracted a million likes.
Table 6. Qualitative comparison of few-shot exemplars retrieved by semantic-only and template-aware strategies.
(Prediction)
(Lucas)[Lender] provided (the)[Asset] conference room to (Mia)[Borrower] under (conference-rental8)[Agreement] for enhancing their product offerings. However, (Mia)[Borrower] did not utilize …, (Lucas’s)[Lender] reputation was compromised as clients began avoiding (the venue)[Object] due to the unexpected activities. The actions of (Mia)[Borrower] came to light on (2023/06/30)[T_due] .
(Gold Ouput)
(Lucas)[Lender] provided (the conference room)[Object] to (Mia)[Borrower] under (conference-rental8)[Agreement] for enhancing their product offerings. However, (Mia)[Borrower] did not utilize …, (Lucas’s reputation was compromised as clients began avoiding the venue due to the unexpected activities)[Harm] . The actions of (Mia)[Borrower] came to light on (2023/06/30)[T_discovery] .
Table 7. Representative Errors of LegalCaseNER (incorrect predictions in red, corresponding gold entities in blue)
Small language models (SLMs) are promising for real-world deployment due to their efficiency and low operational cost. However, their limited capacity struggles with high-stakes legal reasoning tasks that require coherent statute interpretation and logically consistent deduction. Furthermore, training SLMs for such tasks demands high-quality, concise reasoning trajectories, which are prohibitively expensive to manually collect and difficult to curate via standard rejection sampling, lacking granularity beyond final verdicts. To address these challenges, we propose {LegalDrill}, a diagnosis-driven synthesis framework that extracts and iteratively refines reasoning trajectories from a capable teacher via fine-grained prompting, then a self-reflective verification is employed to adaptively select the most effective data for the SLM student. The resulting data empower SLM training through supervised fine-tuning and direct preference optimization. Extensive experiments on several legal benchmarks demonstrate that {LegalDrill} significantly bolsters the legal reasoning capabilities of representative SLMs while bypassing the need for scarce expert annotations, paving a scalable path toward practical legal reasoning systems.
Tianchun Li, Haochen Liu, Vishwa Pardeshi +5
Purdue University, West Lafayette, IN, USA · Fidelity Investments, Boston, MA, USA
Large language models are increasingly applied to high-risk domains such as law, yet complex legal reasoning remains limited by two structural challenges. First, existing RAG and GraphRAG methods emphasize lexical or semantic similarity while overlooking normative relations among legal provisions. Second, vanilla Chain-of-Thought prompting may generate plausible rationales without enforcing the normative structure of legal reasoning. To deal with the bottleneck of pipelines in the legal reasoning domain, we propose LEGO, a dual-module framework that synergizes Legal Expert GraphRAG and expert Chain-of-thought for complex legal reasoning. ExpertGraphRAG uses an expert-annotated civil code graph encoding these normative relations with a greedy normative-coverage retrieval algorithm to dynamically extract instance-specific provision subgraphs, while ExpertCoT organizes the retrieved provisions and case facts into structured Provision-Fact-Conclusion reasoning. With a Qwen3-8B backbone, LEGO achieves 40.53% exact-match accuracy on LawExamQA_Civil, outperforming the evaluated RAG and CoT baselines and performing comparably to the evaluated larger models, while remaining robust on multi-hop questions. It also achieves the best results among the evaluated baselines on the open-ended benchmarks. Ablation studies confirm the individual and complementary contributions of both modules, demonstrating LEGO's effectiveness in improving LLMs' complex legal reasoning ability. Code and dataset can be found in the link: https://github.com/BLK-WHT/LEGO
Qingjing Chen, Junkai Zhang, Shaochun Wang +7
Alma AI, University of Bologna, Italy · Tsinghua University, China · Modelbest Inc. +2
This paper presents the methodologies and results of the NOWJ team's participation across all five tasks of the COLIEE 2026 competition. For Task 1 (Legal Case Retrieval), we propose a four-stage pipeline comprising candidate filtering, dense retrieval with complementary embedding models, cross-encoder reranking via fine-tuned generative rerankers and MLP-based pairwise classification, and adaptive per-query cutoff prediction. For Task 2 (Legal Case Entailment), we combine BM25 filtering, T5-based reranking, and LLM-based entailment verification with consensus ensemble. For Task 3 (Statute Law Retrieval and Entailment), we adopt a retrieval-augmented generation framework with dense retrieval, attention-based reranking, and few-shot-prompted LLM reasoning. For Task 4 (Legal Textual Entailment), we introduce a dynamic routing pipeline that classifies query difficulty and dispatches cases to either a balanced few-shot solver or a structured zero-shot chain-of-thought solver. For the Pilot Task (Legal Judgment Prediction), we combine hierarchical transformers with CRF layers, argument relation mining, and probabilistic argumentation graph reasoning.
VNU University of Engineering and Technology Hanoi, Vietnam · Center for Juris-Informatics, ROIS-DS College of Engineering & Computer Science, VinUniversity Hanoi, Vietnam