Organizations: Department of Computer Science, AI Institute, University of South Carolina, Columbia, South Carolina, USA · Department of Computer Science and Electrical Engineering, University of Maryland, Baltimore County, Baltimore, Maryland, USA · Department of Computer Science and Engineering, The Ohio State University, Columbus, Ohio, USA
Large Language Models (LLMs) have transformed natural language processing, but their lack of interpretable reasoning and tendency to hallucinate pose significant challenges for legal applications. While LLMs show promise for legal text analysis and generation, they struggle with accurate citation attribution and precedent verification. For example, in legal contexts, a single incorrect precedent can jeopardize a case. Current approaches to improve LLM reliability in legal domains suffer from two key limitations: inadequate integration of structured legal knowledge during training or fine-tuning, and insufficient verification mechanisms for generated legal content. To address these challenges, we propose the TRISM (Trustworthy, Reliable, Interpretable, Safe Models) framework, which integrates NeuroSymbolic AI principles with LLMs to leverage both neural learning capabilities and symbolic reasoning over structured legal knowledge. The TRISM approach addresses the above limitations while maintaining interpretable decision pathways. Our framework formalizes the extraction of symbolic knowledge from legal textual documents and incorporates Retrieval-Augmented Generation (RAG) as a core component for grounding LLM outputs in verified legal sources. In this position paper, we make the following contributions: (1) An analysis of the limitations of AI in law; (2) Introduce RASOR RAG which creates foundations for neurosymbolic RAG by generating explicit interpretable rationales that could be formalized into symbolic representations; (3) A formalized methodology for creating symbolic legal knowledge bases that support both interpretable reasoning and output verification in LLMs; and (4) The TRISM framework for integrating symbolic legal knowledge with LLMs.
Figures & tables
Figure 1: SaulLM & GPT-4 lack context. Tested on July 08, 2025
Figure 2: RASOR Architecture
Figure 3: RASOR’s dynamic K-value selection (K=23,17,12 based on query complexity) outperforms Contriever’s fixed approach (K=16), enabling LLAMA-8B to generate superior responses across all context lengths. K represents the number of top-ranked chunks.
Figure 4: NeuroSymbolic RAG Pipeline integrating TRISM principles. The framework combines an LLM with a Legal Knowledge Graph (KG), leveraging query enrichment and retrieval techniques (e.g., FAISS ( Douze et al., 2024 ) ) for scalable and secure document retrieval. It incorporates trust, responsibility, and security through enriched rule-based reasoning in the KG, ensuring robust query-answering in legal contexts.
Figure 5: A representation of a Legal Knowledge Graph (KG) showcasing the relationships in a legal case, including entities such as a lawsuit, claims of violation of the Consumer Privacy Act and unauthorized data collection, the involved parties (an individual and Tech Corp), the year (2023), and the associated claim of $5 million in damages.
Figure 6: A 10-step process to automate the process of updating Legal KG using LLMs. The architecture highlights (a) the transformation of legal KG to embedding, (b) the identification of new legal events, (c) the detection of knowledge gaps, and (d) addressing the knowledge gaps with new candidate triples. A pre-trained LLM powers these major phases through three key mechanisms: (a) tuning with synthetic negative samples to establish clear decision boundaries and validate existing knowledge in the KG, (b) identification of unseen instances to explore and fill gaps in the current legal knowledge representation, and (c) dynamic prompt improvement to optimize hypothesis generation. The updated legal KG is then used to refine the LLM and even reduce hallucinations.
Task
Category
Knowledge Encoding
Reference
KG Construction
Construction
–
Kondo et al. (2024) , Hlyzov et al. (2017) , Filtz (2017) , Tang et al. (2020) , Song and de la Clergerie (2020)
Charge/Judgement Prediction
Classification
Explicit
Bi et al. (2024) , Gan et al. (2021)
Implict
He et al. (2023) , Wang et al. (2024b)
Legal Statute Identification
Explicit
Nguyen et al. (2023)
Question Answer
Question Answer
Explicit
Jiang et al. (2024)
Implicit
Louis et al. (2024) , McElvain et al. (2019)
Table 1: Neural-based approaches, with and without the use of graphs, for various downstream legal tasks. Here, ‘-’ indicates “not applicable,” “Explicit“ denotes that knowledge is represented in the form of a graph, and “Implicit” signifies that knowledge encoded in a neural network is used, either by pre-training a language model on legal data or by relying on the knowledge present in an off-the-shelf language model.
The growing adoption of large language models in legal practice brings both significant promise and serious risk. Legal professionals stand to benefit from AI that can reason over contracts, draft documents, and analyze sources at scale, yet the high-stakes nature of legal work demands a level of rigor that current AI systems do not provide. The central problem is not simply that LLMs hallucinate facts and references; it is that they systematically draw inferences that go beyond what the source text actually supports, presenting assumption-laden conclusions as if they were logically grounded. This proposal presents a neuro-symbolic approach to legal AI that combines the expressive power of large language models with the rigor of formal verification, aiming to make AI-assisted legal reasoning both capable and trustworthy, thus reducing the burden of manual verification without sacrificing the accountability that legal practice demands.
Large Language Models (LLMs) perform well in short contexts but degrade on long legal documents, often producing hallucinations such as incorrect clauses or precedents. In the legal domain, where precision is critical, such errors undermine reliability and trust. Retrieval Augmented Generation (RAG) helps ground outputs but remains limited in legal settings, especially with small, locally deployed models required for data privacy. We identify two failure modes: retrieval errors due to lexical redundancy in legal corpora, and decoding errors where models generate answers despite insufficient context. To address this, we propose Metadata Enriched Hybrid RAG to improve document level retrieval, and apply Direct Preference Optimization (DPO) to enforce safe refusal when context is inadequate. Together, these methods improve grounding, reliability, and safety in legal language models.
Recent advances in large language models (LLMs) have significantly enhanced automated legal reasoning. Yet, it remains unclear whether their performance reflects genuine legal reasoning ability or artifacts of data contamination. We present a comprehensive empirical study of tax law reasoning approaches and implement a contamination detection protocol to rigorously assess LLM reliability. We show that performance can be inflated by contamination. Building on this analysis, we conduct a systematic evaluation, comparing monolithic LLMs with hybrid systems that translate statutory text into formal representations and delegate inference to symbolic solvers. We build a novel test suite designed to probe generalization to unseen documents via case and rule variations. Our findings indicate that legal reasoning is inherently compositional and that neuro-symbolic frameworks offer a more reliable and robust foundation for legal AI, as well as improved generalization to unobserved situations.