Enhancing Legal LLMs through Metadata-Enriched RAG Pipelines and Direct Preference Optimization

Suyash Maniyar; Deepali Singh; Rohith Reddy

cs.CLFeb 25, 2026

Enhancing Legal LLMs through Metadata-Enriched RAG Pipelines and Direct Preference Optimization

Authors: Suyash Maniyar, Deepali Singh, Rohith Reddy

Abstract

Large Language Models (LLMs) perform well in short contexts but degrade on long legal documents, often producing hallucinations such as incorrect clauses or precedents. In the legal domain, where precision is critical, such errors undermine reliability and trust. Retrieval Augmented Generation (RAG) helps ground outputs but remains limited in legal settings, especially with small, locally deployed models required for data privacy. We identify two failure modes: retrieval errors due to lexical redundancy in legal corpora, and decoding errors where models generate answers despite insufficient context. To address this, we propose Metadata Enriched Hybrid RAG to improve document level retrieval, and apply Direct Preference Optimization (DPO) to enforce safe refusal when context is inadequate. Together, these methods improve grounding, reliability, and safety in legal language models.

Figures & tables

Figure 1: Overview of the chunking strategy evaluation pipeline. Legal documents undergo recursive chunking with optional metadata enhancement, are indexed using FAISS, and evaluated using comprehensive retrieval metrics across multiple chunking and retrieval configurations.

Panel of Figure 2 — Figure 2: End-to-end RAG performance comparison across three legal datasets. The left column shows Document Retrieval Mismatch (DRM) rates, where lower scores indicate better performance (less hallucinated/incorrect retrieval). The right column shows Span Recall @ K, where higher scores indicate better performance . Our Metadata-Enhanced (orange) approach consistently outperforms the baseline (blue) across all metrics and datasets.

Document Retrieval Mismatch Rate (%)
Metric	k=1	k=2	k=4	k=8	k=16	k=32	k=64
Baseline	18.1 (9.2–24.6)	26.4 (19.9–34.1)	38.7 (30.9–48.0)	50.7 (43.9–60.1)	65.0 (58.4–73.1)	76.8 (70.5–83.2)	86.3 (82.3–90.7)
Enhanced	18.1 (16.7–19.4)	22.2 (20.8–26.0)	25.2 (23.6–27.6)	33.8 (31.7–36.3)	48.0 (43.2–50.8)	64.1 (60.2–67.2)	77.6 (75.0–80.4)
Delta	+0.0	-4.2	-13.5	-16.9	-17.0	-12.7	-8.7
Span Recall
Baseline	0.194 (0.139–0.297)	0.354 (0.306–0.438)	0.542 (0.478–0.581)	0.764 (0.647–0.857)	0.903 (0.812–0.970)	1.014 (0.947–1.081)	1.069 (1.028–1.135)

Table 1: Comprehensive performance analysis on Australian Legal QA dataset comparing baseline recursive chunking against metadata-enhanced approach.(For DRM the lower the score the better the method is performing)

Context Set	Zero-shot	Post-DPO
Correct Context	53.2%	1.5%
Incorrect Context	87.3%	99.3%

Table 2: Refusal percentage on the test set before and after DPO alignment.

Setting	Mean BERTScore (F1)
Zero-shot	0.8526
Post-DPO	0.9074 (+0.0548 improvement)

Table 3: BERTScore (F1) comparison between zero-shot and DPO-aligned models. The improvement over the baseline is shown in parentheses.

Mean improvement: 0.0178
System	Mean F1 (with 95% CI)
Baseline (No Meta-Enhanced Chunking, No DPO)	0.8479
Ours (Meta-Enhanced Chunking + DPO)	0.8657
95% CI (delta): [0.0113, 0.0244]

Table 4: End-to-end evaluation of RAG + Answer Generation. The context is constructed from the top 4 retrieved chunks. Statistical details are shown below the mean F1 score.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Document Retrieval Mismatch Rate (%)
Metric	k=1	k=2	k=4	k=8	k=16	k=32	k=64
Baseline	16.5	17.8	18.0	23.6	33.0	60.3	78.7
Enhanced	11.4 (6.3–18.8)	11.4 (6.3–18.8)	11.4 (6.3–18.8)	14.1 (9.1–21.9)	24.0 (19.4–33.2)	58.3 (54.2–64.3)	79.2 (77.1–82.2)
Delta	-5.1	-6.4	-6.6	-9.5	-9.0	-2.0	+0.5
Span Recall
Baseline	0.198	0.366	0.564	0.810	1.036	1.119	1.140

Appendix

Table 5: Performance analysis on PrivacyQA dataset comparing baseline against metadata-enhanced approach. Enhanced results show Mean (Min–Max) across bootstrap samples.

Document Retrieval Mismatch Rate (%)
Metric	k=1	k=2	k=4	k=8	k=16	k=32	k=64
Baseline	34.7	36.6	48.8	65.0	76.2	83.3	87.9
Enhanced	8.5 (7.1–10.9)	9.0 (7.8–10.7)	9.9 (8.7–11.4)	10.7 (9.6–11.8)	11.1 (9.9–12.4)	11.6 (10.4–12.8)	12.8 (11.2–13.9)
Delta	-26.2	-27.6	-38.9	-54.3	-65.1	-71.7	-75.1
Span Recall
Baseline	0.010	0.021	0.038	0.067	0.113	0.167	0.249

Appendix

Table 6: Performance analysis on MAUD dataset comparing baseline against metadata-enhanced approach. Enhanced results show Mean (Min–Max) across bootstrap samples.

Explore similar work

NeuroSymbolic AI for Legal AI-TRISM: Trustworthy, Reliable, Interpretable, Safe Models
Apr 5, 2026Deepa Tilwani, Yash Saxena, Ankur Padia +2LLM Reliability Neuro-Symbolic AI
Department of Computer Science, AI Institute, University of South Carolina, Columbia, South Carolina, USA · Department of Computer Science and Electrical Engineering, University of Maryland, Baltimore County, Baltimore, Maryland, USA · Department of Computer Science and Engineering, The Ohio State University, Columbus, Ohio, USA
Can Small Models Reason About Legal Documents? A Comparative Study
Mar 26, 2026Snehit VaddiSmall Language Models Legal Reasoning
Independent Researcher
Building Legal Reward Models for Grounding and Abstention
Sep 13, 2026Rilton Franzone, Valentin Noël, Puyu Wang +2Reward Modeling LLM Grounding
Devoteam

Apr 5, 2026J/K move · Enter open · S save

NeuroSymbolic AI for Legal AI-TRISM: Trustworthy, Reliable, Interpretable, Safe Models

Deepa Tilwani, Yash Saxena, Ankur Padia, Srinivasan Parthasarathy+1

Department of Computer Science, AI Institute, University of South Carolina, Columbia, South Carolina, USA · Department of Computer Science and Electrical Engineering, University of Maryland, Baltimore County, Baltimore, Maryland, USA · Department of Computer Science and Engineering, The Ohio State University, Columbus, Ohio, USA

LLM Reliability Neuro-Symbolic AI Legal NLP LLM Interpretability LLM Grounding Retrieval-Augmented Generation

Open paper AskPDF arXiv

Large Language Models (LLMs) have transformed natural language processing, but their lack of interpretable reasoning and tendency to hallucinate pose significant challenges for legal applications. While LLMs show promise for legal text analysis and generation, they struggle with accurate citation attribution and precedent verification. For example, in legal contexts, a single incorrect precedent can jeopardize a case. Current approaches to improve LLM reliability in legal domains suffer from two key limitations: inadequate integration of structured legal knowledge during training or fine-tuning, and insufficient verification mechanisms for generated legal content. To address these challenges, we propose the TRISM (Trustworthy, Reliable, Interpretable, Safe Models) framework, which integrates NeuroSymbolic AI principles with LLMs to leverage both neural learning capabilities and symbolic reasoning over structured legal knowledge. The TRISM approach addresses the above limitations while maintaining interpretable decision pathways. Our framework formalizes the extraction of symbolic knowledge from legal textual documents and incorporates Retrieval-Augmented Generation (RAG) as a core component for grounding LLM outputs in verified legal sources. In this position paper, we make the following contributions: (1) An analysis of the limitations of AI in law; (2) Introduce RASOR RAG which creates foundations for neurosymbolic RAG by generating explicit interpretable rationales that could be formalized into symbolic representations; (3) A formalized methodology for creating symbolic legal knowledge bases that support both interpretable reasoning and output verification in LLMs; and (4) The TRISM framework for integrating symbolic legal knowledge with LLMs.

Abstract

Explore similar work

NeuroSymbolic AI for Legal AI-TRISM: Trustworthy, Reliable, Interpretable, Safe Models

Can Small Models Reason About Legal Documents? A Comparative Study

Building Legal Reward Models for Grounding and Abstention