Verifiable Retrieval for Legal Reasoning through Structure-driven Knowledge Graph

Ibrahim Siddig; Sviatoslav Tugeev; Munir Georges

cs.CLOct 9, 2026

Verifiable Retrieval for Legal Reasoning through Structure-driven Knowledge Graph

Authors: Ibrahim Siddig, Sviatoslav Tugeev, Munir Georges

Organizations: Technische Hochschule Ingolstadt, Esplanade 10, 85049 Ingolstadt, Germany

Abstract

Any automation within reliability-critical contexts such as medicine or law has to be transparent and verifiable. Despite their advantages, current LLM-based systems do not meet these requirements due to their inherent black-box nature. Retrieval Augmented Generation (RAG) attempts to ground the generative process in the relevant interpretable documents, thereby reducing the risk of AI 'hallucinations' and potentially enabling verification of the initial generation context. However, as the number and size of potential sources increases, naive retrieval approaches tend to perform worse, ignoring critical context elements. In this paper we aim to tackle this problem in the context of German Legal Code. Using structure-driven parsing to create a Knowledge Graph (KG) for the source document navigation and retrieval, we compare several KG-informed chunking and retrieval strategies and evaluate their effect on the accuracy of retrieved citations.

Figures & tables

Panel of Figure 1 — Figure 1: The Statute knowledge graph hierarchy includes $58$ federal codes, $168{,}469$ nodes, and $240{,}507$ edges. Column headers display each level’s name and total node count, noting that not all Sätze contain a Nummer or Buchstabe . Edge labels provide the average number of child nodes per branching parent at that level. Cross-reference edges marked cites_* , drawn in blue. Citations originate at a Satz node since they are tagged in sentence text and target a Satz ( $15{,}030$ ), Nummer ( $6{,}805$ ) or Buchstabe ( $418$ ) node within the same code. The final graph is a forest of 58 disjoint code graphs, connected to the empty parent root node.

Figure 2: After the initial chunking, whether naive or KG-driven, the resulting chunks are embedded using KaLM-Embedding-Gemma3-12B. For each entry in the OpenLegalData corpus, the Tatbestand section, which describes the facts of the case, is embedded the same way to serve as pipeline input, and the top- k relevant chunks of the legal codes are retrieved. This list is then optionally expanded through a 1-hop expansion strategy using KG. The final structured list of retrieved references is compiled based on the IDs of associated nodes in KG and compared to the structured list of references from the corresponding section of the same corpus entry.

	Evaluation depth (strict match up to) [ToER $\uparrow$ ]
Granularity (Chunks)	Code	§	Absatz	Satz	Nummer	Buchst
Buchstabe	12.89	2.93	2.05	1.35	1.91	0.50
Nummer	14.23	3.25	2.27	1.49	2.17	0.28
Satz	13.59	3.20	2.28	1.53	2.79	0.52
Absatz	7.78	1.93	1.36	1.37	1.57	0.30
§ ( Paragraph )	3.06	0.85	0.92	0.90	0.94	0.16

Table 1: Depth-stratified ToER at top

k=5

retrieved chunks. Structure-aware chunking granularities ( Nummer and Satz ) yield the highest scores across all depths, outperforming both the Naive-100w baseline and paragraph-level (§) chunking strategies. Values in bold indicate the best-performing chunking granularity per evaluation depth.

Panel of Figure 3 — Figure 3: Impact of 1-hop context expansion on token efficiency (ToER) at top-5 retrieved chunks. Evaluation is conducted under matching at the Absatz evaluation depth across all chunking granularities for unexpanded retrieval and three expansion strategies. Granularities are ordered from higher to lower, and the y-axis is shared between figures. Figure 3(a) is uncharged ( $W{=}W_{\ell}$ ): expanded context is added to retrieval but not considered against the token budget. Figure 3(b) is charged ( $W{=}W_{\ell}^{m}$ ): the expansion tokens count toward $W$ , exposing the efficiency cost of each strategy.

Legal IR

Legal information retrieval finds and ranks relevant statutes, cases, patents, and other legal materials for queries and evidence-grounded systems.

See all 53 papers

LegalGraphRAG: Multi-Agent Graph Retrieval-Augmented Generation for Reliable Legal Reasoning
May 27, 2026Zerui Chen, Qinggang Zhang, Zhishang Xiang +5Multi-Agent LLM Systems Graph Retrieval-Augmented Generation
School of Informatics, Xiamen University · The Hong Kong Polytechnic University · Institute of Artificial Intelligence, Xiamen University
StrucChunk: Structure-Aware Retrieval for Legal Document RAG
Oct 19, 2026Surya Deep Singh, Ashutosh Tripathi, Gaurav Deep SinghLegal IR Document Chunking
Independent Researcher.
Beyond Probabilistic Similarity: Structural, Temporal, and Causal Limitations of Retrieval-Augmented Generation in the Legal Domain
Jun 8, 2026Hudson de MartimRetrieval-Augmented Generation Legal IR
Federal Senate of Brazil, Brasília, Brazil

May 27, 2026J/K move · Enter open · S save

LegalGraphRAG: Multi-Agent Graph Retrieval-Augmented Generation for Reliable Legal Reasoning

Zerui Chen, Qinggang Zhang, Zhishang Xiang, Zhimin Wei+4

School of Informatics, Xiamen University · The Hong Kong Polytechnic University · Institute of Artificial Intelligence, Xiamen University

Multi-Agent LLM Systems Graph Retrieval-Augmented Generation Legal Document Analysis Knowledge Graphs Legal Reasoning

Open paper AskPDF arXiv

Graph-based Retrieval-Augmented Generation (GraphRAG) advances flat document retrieval by structuring knowledge as relational graphs, enabling more coherent and effective reasoning. However, applying it to specific domains like legal reasoning faces critical challenges. (i) Legal corpora are heterogeneous, containing multi-granular knowledge from cases, articles and interpretations. A flat knowledge graph cannot adequately differentiate between factual details, applied rules, and abstract principles, limiting accurate retrieval. (ii) Reliable legal judgment demands transparent, evidence-based reasoning. Traditional RAG passes retrieved context directly to an LLM without verification, resulting in opaque, error-prone reasoning. To this end, we propose LegalGraphRAG, a framework designed for reliable legal reasoning. Our approach introduces two core components: a hierarchical legal graph that hierarchically organizes legal sources to enable retrieval at appropriate abstraction levels, and a multi-agent system for reliable legal reasoning, where a Researcher retrieves candidate evidence, an Auditor rigorously verifies its validity against source documents, and an Adjudicator synthesizes the set of verified evidence to render a final judgment. Extensive experiments show that LegalGraphRAG achieves the state-of-the-art performance, outperforming existing GraphRAG baselines in accurate and trustworthy legal analysis. Our code, datasets and implementation details are available at https://github.com/XMUDeepLIT/LegalGraphRAG.

Abstract

Legal IR

Explore similar work

LegalGraphRAG: Multi-Agent Graph Retrieval-Augmented Generation for Reliable Legal Reasoning

StrucChunk: Structure-Aware Retrieval for Legal Document RAG

Beyond Probabilistic Similarity: Structural, Temporal, and Causal Limitations of Retrieval-Augmented Generation in the Legal Domain