De Jure: Iterative LLM Self-Refinement for Structured Extraction of Regulatory Rules

Keerat Guliani; Deepkamal Gill; David Landsman; Nima Eshraghi; Krishna Kumar; Lovedeep Gondara

cs.AIApr 2, 2026

De Jure: Iterative LLM Self-Refinement for Structured Extraction of Regulatory Rules

Authors: Keerat Guliani, Deepkamal Gill, David Landsman, Nima Eshraghi, Krishna Kumar, Lovedeep Gondara

Organizations: The Vanguard Group, Inc.

Abstract

Regulatory documents encode legally binding obligations that LLM-based systems must respect. Yet converting dense, hierarchically structured legal text into machine-readable rules remains a costly, expert-intensive process. We present De Jure, a fully automated, domain-agnostic pipeline for extracting structured regulatory rules from raw documents, requiring no human annotation, domain-specific prompting, or annotated gold data. De Jure operates through four sequential stages: normalization of source documents into structured Markdown; LLM-driven semantic decomposition into structured rule units; multi-criteria LLM-as-a-judge evaluation across 19 dimensions spanning metadata, definitions, and rule semantics; and iterative repair of low-scoring extractions within a bounded regeneration budget, where upstream components are repaired before rule units are evaluated. We evaluate De Jure across four models on three regulatory corpora spanning finance, healthcare, and AI governance. On the finance domain, De Jure yields consistent and monotonic improvement in extraction quality, reaching peak performance within three judge-guided iterations. De Jure generalizes effectively to healthcare and AI governance, maintaining high performance across both open- and closed-source models. In a downstream compliance question-answering evaluation via RAG, responses grounded in De Jure extracted rules are preferred over prior work in 73.8% of cases at single-rule retrieval depth, rising to 84.0% under broader retrieval, confirming that extraction fidelity translates directly into downstream utility. These results demonstrate that explicit, interpretable evaluation criteria can substitute for human annotation in complex regulatory domains, offering a scalable and auditable path toward regulation-grounded LLM alignment.

Figures & tables

Figure 1: De Jure pipeline overview. Input documents are pre-processed into structured Markdown (Stage 1), parsed into JSON rule units via a domain-agnostic LLM prompt (Stage 2), scored by an LLM judge across 19 criteria in three stages (Stage 3), and iteratively repaired until the per-stage average score reaches 90% or the retry budget (max 3) is exhausted (Stage 4).

Dataset	Domain	Jurisdiction	Sections	Rule Density
SEC Advisers Act	Financial securities	United States	$\sim$ 50	High
EU AI Act	AI governance	European Union	113	Medium–High
HIPAA Privacy Rule	Healthcare privacy	United States	$\sim$ 30	High

Table 1: Summary of evaluation datasets.

Model	J1 (Metadata)	J2 (Definitions)	J3 (Per-Rule)	Overall
Llama-3.1-8B	4.93	$4.75$	$4.53$	$4.74$
Qwen3-VL-8B	4.93	4.87	$4.64$	$4.81$
Claude-3.5-Sonnet	4.99	$4.83$	$4.67$	$4.83$
GPT-5-mini	4.99	$4.83$	4.75	4.85

Table 2: Averaged judge (J) scores on the SEC Investment Advisers Act corpus (scale 1–5; higher is better). Per-criterion breakdowns for all three judges are provided in Appendix E (Tables 9 – 11 ).

Dataset	Model	J1 (Metadata)	J2 (Definitions)	J3 (Per-Rule)	Overall
SEC	GPT-5-mini	4.99	$4.83$	4.75	4.85
SEC	Qwen3-VL-8B	$4.93$	4.87	$4.64$	$4.82$
HIPAA	GPT-5-mini	4.93	$4.61$	4.75	4.76
HIPAA	Qwen3-VL-8B	$4.90$	4.71	$4.66$	$4.76$
EU AI Act	GPT-5-mini	4.72	$4.69$	4.71	$4.71$
EU AI Act	Qwen3-VL-8B	$4.65$	4.82	$4.65$	4.71

Table 3: Generalization across three regulatory corpora (scale 1–5; higher is better). De Jure is applied without domain-specific modification. Per-criterion breakdown details are provided in Appx. F (Tables 12 – 14 ).

Criterion	$k{=}1$	$k{=}5$	$k{=}10$
Completeness	78.00	80.50	83.50
Factual Grounding	80.50	76.00	85.50
Handling Ambiguity	53.50	66.50	84.00
Practical Actionability	78.00	80.50	84.00
Regulatory Precision	74.50	80.50	83.50
Overall Preference	78.00	80.50	83.50

Table 4: Pairwise win rates (%) of De Jure against Datla et al. (2025) on the HIPAA downstream QA task across three retrieval depths.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

$\theta$	Step 1	Step 2	Step 3	Total
0.6	4.86	4.12	4.47	4.48
0.7	4.86	4.21	4.49	4.52
0.8	4.92	4.31	4.58	4.61
0.9	4.92	4.42	4.65	4.67
$\Delta$ (0.6 $\to$ 0.9)	+0.06	+0.30	+0.18	+0.19

Appendix

Table 5: Average quality score per pipeline step across acceptance thresholds

\theta

(scale 1–5). Bold row denotes our default.

\Delta

reports the absolute gain from the weakest to the strongest threshold.

Figure 2: Average quality score per pipeline step as a function of retry budget

r

(scale 1–5; higher is better). Steps 1 and 3 remain largely flat throughout, while Step 2 exhibits a sharp threshold effect: negligible gain from

r{=}0

r{=}1

, followed by a 1.25-point (25%) recovery at

r{=}2

. The shaded region marks the negligible-gain zone. All steps saturate beyond

r{=}2

Chunking Strategy	Step 1	Step 2	Step 3	Total
Ours	4.97	4.47	4.60	4.68
Datla et al. (2025)	4.64	4.34	4.59	4.52
$\Delta$ (Ours $-$ comparison target)	+0.33	+0.13	+0.01	+0.16

Appendix

Table 6: Average quality score per pipeline step for each chunking strategy on the HIPAA subset (scale 1–5).

\Delta

reports the absolute gain of our strategy.

Max Retries ( $r$ )	Trigger	Step 1	Step 2	Step 3	Total
$r{=}3$	Avg (default)	4.92	4.42	4.65	4.66
$r{=}3$	Individual	4.92	4.44	4.63	4.66
$r{=}1$	Avg	4.87	3.16	4.49	4.18
$r{=}1$	Individual	4.87	3.21	4.43	4.17
$\Delta$ ( $r$ : 1 $\to$ 3, Avg)		+0.05	+1.26	+0.16	+0.48

Appendix

Table 7: Average quality score per pipeline step across trigger strategies and retry budgets.

\Delta

reports the gain from

r{=}1

r{=}3

under the default avg-trigger.

Judge	Evaluates	Criteria
Judge 1	Section Metadata	Completeness
		Fidelity to Source Text
		Non-Hallucination
		Title Quality
		Precision of Citations and Dates
		Meaningful Population of Optional Fields

Appendix

Table 8: Judgment criteria grouped by pipeline stage. Each judge targets the failure modes specific to its semantic layer: structural integrity (Judge 1), definitional precision (Judge 2), and fine-grained rule correctness and utility (Judge 3).

Figure 3: De Jure applied to HIPAA § 164.306. Panels (c)→(d)→(e) read left-to-right across the bottom row. Panels (a)–(b) show the raw PDF source and its pre-processed Markdown. Panel (c) shows the initial extraction with two field-level defects: an incomplete label and a misclassified rule type. Panel (d) shows the judge evaluation (avg. 0.55, fail ) with per-criterion scores and targeted critiques. Panel (e) shows the corrected extraction after a single repair iteration (avg. 0.90, pass ), with only deficient fields revised and all others preserved.

Model	Completeness	Fidelity	Non-Halluc.	Title Quality	Citation & Date	Opt.	Avg
Claude-3.5-Sonnet	4.95	5.00	5.00	5.00	5.00	5.00	4.99
Llama-3.1-8B	4.84	5.00	5.00	4.95	4.89	4.92	4.93
GPT-5-mini	4.95	5.00	5.00	5.00	5.00	4.97	4.99
Qwen3-VL-8B	4.85	5.00	5.00	5.00	4.97	4.77	4.99

Appendix

Table 9: Full per-criteria scores on the SEC Investment Advisers Act corpus for Judge 1 (Metadata) criteria (scale 1–5; higher is better). Abbreviations: Non-Halluc. = Non-Hallucination.; Opt. = Optional Field Population; Avg = Average Score

Model	Completeness	Fidelity	Non-Halluc.	Precision	Term Quality	Avg
Claude-3.5-Sonnet	4.59	4.90	5.00	4.77	4.90	4.83
Llama-3.1-8B	4.50	4.76	5.00	4.66	4.84	4.75
GPT-5-mini	4.54	4.95	5.00	4.69	4.95	4.83
Qwen3-VL-8B	4.69	4.90	5.00	4.85	4.92	4.87

Appendix

Table 10: Full per-criteria scores on the SEC Investment Advisers Act corpus for Judge 2 (Definitions) criteria (scale 1–5; higher is better). Abbreviations: Non-Halluc. = Non-Hallucination; Precision = Precision & Formatting.

Model	Comp.	Conc.	Accuracy	Cons.	Fidelity	Neut.	Actionab.	Non-Halluc.	Avg
Claude-3.5-Sonnet	4.11	4.68	4.80	4.95	4.43	5.00	4.42	5.00	4.67
Llama-3.1-8B	4.06	4.40	4.57	4.72	4.26	5.00	4.25	5.00	4.53
GPT-5-mini	4.27	4.49	4.95	4.98	4.74	5.00	4.54	5.00	4.75
Qwen3-VL-8B	4.10	4.48	4.81	4.93	4.43	5.00	4.36	5.00	4.64

Appendix

Table 11: Full per-criteria scores on the SEC Investment Advisers Act corpus for Judge 3 (Per-Rule) criteria (scale 1–5; higher is better). Abbreviations: Comp. = Completeness; Conc. = Conciseness; Cons. = Consistency; Neut. = Neutrality; Actionab. = Actionability; Non-Halluc. = Non-Hallucination.

Dataset	Model	Completeness	Fidelity	Non-Halluc.	Title Quality	Citation & Date	Opt.
SEC	Qwen3-VL-8B	4.85	5.00	5.00	5.00	4.97	4.77
SEC	GPT-5-mini	4.95	5.00	5.00	5.00	5.00	4.97
EU AI Act	Qwen3-VL-8B	3.89	5.00	5.00	5.00	4.67	4.33
EU AI Act	GPT-5-mini	4.00	5.00	5.00	5.00	4.78	4.56
HIPAA	Qwen3-VL-8B	4.85	4.95	4.93	5.00	4.90	4.76
HIPAA	GPT-5-mini	4.85	5.00	5.00	5.00	4.95	4.76

Appendix

Table 12: Generalization across three regulatory corpora for Judge 1 (Metadata) criteria (scale 1–5; higher is better). Abbreviations: Non-Halluc. = Non-Hallucination.; Opt. = Optional Field Population

Dataset	Model	Completeness	Fidelity	Non-Halluc.	Precision	Term Quality
SEC	Qwen3-VL-8B	4.69	4.90	5.00	4.85	4.92
SEC	GPT-5-mini	4.54	4.95	5.00	4.69	4.95
EU AI Act	Qwen3-VL-8B	4.67	4.89	5.00	4.67	4.89
EU AI Act	GPT-5-mini	4.44	4.78	5.00	4.44	4.78
HIPAA	Qwen3-VL-8B	4.51	4.71	5.00	4.63	4.68
HIPAA	GPT-5-mini	4.20	4.76	5.00	4.32	4.78

Appendix

Table 13: Generalization across three regulatory corpora for Judge 2 (Definitions) criteria (scale 1–5; higher is better). Abbreviations: Non-Halluc. = Non-Hallucination; Precision = Precision & Formatting.

Dataset	Model	Comp.	Conc.	Accuracy	Cons.	Fidelity	Neut.	Actionab.	Non-Halluc.
SEC	Qwen3-VL-8B	4.10	4.48	4.81	4.93	4.43	5.00	4.36	5.00
SEC	GPT-5-mini	4.27	4.49	4.95	4.98	4.74	5.00	4.54	5.00
EU AI Act	Qwen3-VL-8B	4.28	4.41	4.90	4.93	4.51	5.00	4.19	5.00
EU AI Act	GPT-5-mini	4.42	4.35	4.95	4.98	4.65	5.00	4.34	5.00
HIPAA	Qwen3-VL-8B	4.15	4.52	4.84	4.96	4.50	5.00	4.32	4.99
HIPAA	GPT-5-mini	4.31	4.52	4.95	4.99	4.76	5.00	4.46	5.00

Appendix

Table 14: Generalization across three regulatory corpora for Judge 3 (Per-Rule) criteria (scale 1–5; higher is better). Abbreviations: Comp. = Completeness; Conc. = Conciseness; Cons. = Consistency; Neut. = Neutrality; Actionab. = Actionability; Non-Halluc. = Non-Hallucination.

#	Question
1	Who may not use or disclose protected health information except as permitted or required by this subpart?
2	When is a covered entity permitted to use or disclose protected health information for treatment, payment, or health care operations?
3	What conditions must be met for a use or disclosure to be considered incident to a permitted use or disclosure?
4	What are the required disclosures of protected health information by a covered entity?
5	What are the limitations on a business associate’s use and disclosure of protected health information?
6	What is prohibited regarding the use and disclosure of genetic information for underwriting purposes?

Appendix

Table 15: Ten representative questions from the 100-question evaluation set used in Experiment 3. Each question is fully answerable from its source passage alone. The set spans five HIPAA topic areas and three question types, reflecting the diversity requirements specified in the generation prompt (Appendix G.4 ).

Document Information Extraction

Document Information Extraction turns text, tables, forms, and images into structured, searchable data.

See all 290 papers

Apr 13, 2026

REGREACT: Self-Correcting Multi-Agent Pipelines for Structured Regulatory Information Extraction

Extracting structured, machine-readable compliance criteria from regulatory documents remains an open challenge. Single-pass language models hallucinate structural elements, lose hierarchical relationships, and fail to resolve inter-document dependencies. We introduce RegReAct, a self-correcting multi-agent framework that decomposes regulatory information extraction into seven specialized stages, each with an Observe--Diagnose--Repair (ODR) loop that validates outputs against the source, correcting not only model hallucinations but also cross-reference errors in the regulations themselves. To ensure structural accuracy, RegReAct constructs a typed criterion graph; to ensure completeness, it resolves external dependencies by retrieving, summarizing, and embedding referenced legal content inline, producing self-contained outputs. Applying RegReAct to three EU Taxonomy Delegated Acts, we construct a dataset comprising 242 activities with over 4,800 hierarchical criteria, thresholds, and enriched source summaries. Evaluation against a GPT-4o single-pass baseline shows that RegReAct outperforms it across all structural and semantic metrics.

Mohammed Ali, Abdelrahman Abdallah, Adam Jatowt

University of Innsbruck, Austria

LLM Self-Correction Document Information Extraction Legal Document Analysis Multi-Agent Systems AI Regulatory Compliance

Ask PDF

May 24, 2025

Business as Rulesual: A Benchmark and Framework for Business Rule Flow Modeling with LLMs

Extracting structured procedural knowledge from unstructured business documents is a critical yet unresolved bottleneck in process automation. While prior work has focused on extracting linear action flows from instructional texts, such as recipes, it has insufficiently addressed the complex logical structures, including conditional branching and parallel execution, that are pervasive in real-world regulatory and administrative documents. Furthermore, existing benchmarks are limited by simplistic schemas and shallow logical dependencies, restricting progress toward logic-aware large language models.To bridge this Logic Gap, we introduce BREX, a carefully curated benchmark comprising 409 real-world business documents and 2,855 expert-annotated rules. Unlike prior datasets centered on narrow service scenarios, BREX spans over 30 vertical domains, covering scientific, industrial, administrative, and financial regulations. We further propose ExIde, a structure-aware reasoning framework that investigates five distinct prompting strategies, ranging from implicit semantic alignment to executable grounding via pseudo-code generation. This enables explicit modeling of rule dependencies and provides an out-of-the-box framework for different business customers without finetuning their own large language models. We benchmark ExIde using 13 state-of-the-art large language models. Our extensive evaluation reveals that executable grounding serves as a superior inductive bias, significantly outperforming standard prompts in rule extraction. In addition, reasoning-optimized models demonstrate a distinct advantage in tracing long-range and non-linear rule dependencies compared to standard instruction-tuned models.

Chen Yang, Ruping Xu, Ruizhe Li +2

Zhejiang University of Technology, China1 · Zhejiang Key Laboratory of Visual Information Intelligent Processing, China 2 · University of Aberdeen, UK +1

LLM Grounding LLM Prompting Document Information Extraction Business Process Modeling

Ask PDF

May 28, 2026

Citation-Closure Retrieval and Per-Rule Attribution for Real-World Regulatory Compliance Question Answering

Deploying Large Language Models (LLMs) for regulatory compliance demands rigorous traceability via comprehensive citations across multi-tiered authority structures. Unlike traditional multi-hop or legal QA, this task requires structured procedural lookups and evidence-set closure rather than entity resolution or case-law reasoning. Existing RAG systems struggle here due to flattened citation edges, fragmented retrieval expansions, and fragile post-hoc attribution. We formalize Regulatory Compliance QA with RegOps-Bench, a novel benchmark featuring an Operational Knowledge Graph derived from complex national R&D regulations. To address these bottlenecks, we propose RefWalk, a unified framework driven by a shared topic anchor. RefWalk traverses cross-document citations, fuses multi-view candidates via max-based aggregation, and enforces per-rule attribution to explicitly map claims to sources. We establish a strong baseline with substantial improvements in retrieval recall and citation accuracy. Finally, a contrastive evaluation on a U.S. health compliance dataset (HIPAA) reveals that existing systems exhibit saturation on flat-structure rules, underscoring the need for RegOps-Bench. Our code is available at https://github.com/yeongjoonJu/RefWalk.

Yeong-Joon Ju, Seong-Whan Lee

Department of Artificial Intelligence, Korea University

Graph Retrieval-Augmented Generation Legal QA Evidence Retrieval AI Regulatory Compliance

Ask PDF

Abstract

Document Information Extraction

Explore similar work

REGREACT: Self-Correcting Multi-Agent Pipelines for Structured Regulatory Information Extraction

Business as Rulesual: A Benchmark and Framework for Business Rule Flow Modeling with LLMs

Citation-Closure Retrieval and Per-Rule Attribution for Real-World Regulatory Compliance Question Answering