Empirical legal research often relies on turning research questions into structured data extracted from large collections of rulings and judgments. Designing the extraction schema and then extracting the data remain a manual, expertise-heavy bottleneck. We present schematize, an open-source multi-agent system that interactively turns a researcher's problem statement into a validated extraction schema that can later be used for autonomous extraction. Schematize couples (i) a clarification dialogue that elicits implicit expert intent, (ii) iterative schema generation, (iii) data-grounded refinement that tests the schema against documents, and (iv) chat-based post-editing. We evaluated the system with human legal professional, introducing our novel methodology, and schematize achieves top performance in most of tested configurations. While the system is designed to be domain-agnostic and applicable to any document collection, we tailor and evaluate it on legal research problems. We release schematize as a pip-installable Python package with full documentation.
Figures & tables
Figure 1: Overview of schematize ’s agentic pipeline. A user request is clarified, formalized into retrieval queries and an initial schema, refined via criteria-based (CBR, up to maxref iterations) and data-grounded (DGR, up to maxdref iterations) loops, and finalized through a summary and interactive chat. Blue ( ) marks automated agent steps, orange ( ) marks human-in-the-loop steps, violet marks the document store, and teal marks refinement loops.
Backbone
vanilla LLM
vanilla LLM + PDH
ScheMatiQ
ScheMatiQ + PDH
schematize
gpt-5.4
36.3% ± 11.7pp
64.3% ± 19.7pp
43.8% ± 15.6pp
64.3% ± 17.6pp
79.4% ± 12.6pp
gpt-5.4-mini
31.7% ± 0 7.2pp
55.5% ± 14.9pp
23.9% ± 0 7.9pp
57.6% ± 0 7.6pp
74.8% ± 0 7.2pp
gpt-5.4-nano
22.2% ± 18.0pp
59.8% ± 10.3pp
24.0% ± 13.8pp
42.5% ± 20.4pp
34.7% ± 35.0pp
claude-sonnet-4.6
37.1% ± 10.8pp
60.5% ± 12.9pp
49.6% ± 29.2pp
67.1% ± 14.8pp
70.0% ± 18.5pp
qwen3.6-35b-a3b
28.5% ± 0 8.7pp
51.6% ± 0 2.5pp
40.4% ± 15.6pp
67.2% ± 15.3pp
66.5% ± 16.3pp
gemma-4-e4b-it
0 3.8% ± 0 4.0pp
38.2% ± 14.8pp
34.5% ± 18.2pp
46.5% ± 0 4.8pp
53.4% ± 0 3.7pp
Table 1: Expert-question coverage (mean ± std, in pp) across backbones and configurations. “+ PDH” prepends the Problem-Definition-Helper stage to a baseline. Bold marks the best, underline the second-best, coverage per backbone.
Figure 2: Comparison of the schematize to all tested baselines. The dashed line marks the average inter-expert coverage ( Table 2 ).
Figure 3: Coverage change over subsequent iterations of schema refinements for three considered problems.
Figure 4: Results of the hyperparameter sensitivity study. We measure coverage of the tested configurations for three legal research problems.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Expert-based evaluation pipeline. Given a shared research problem, each domain expert writes an exhaustive question set the extracted data should answer, while the agentic system independently produces an extraction schema . An LLM-as-judge checks whether each expert question is covered by the schema, yielding per-expert and average coverage scores.
Problem
Questions/expert
Inter-expert coverage
Drug offences
28.3 ± 0 9.6
56.4% ± 0 7.9pp
Medical errors
33.0 ± 13.0
52.7% ± 19.0pp
Personal rights
26.0 ± 0 3.0
52.3% ± 17.1pp
Overall
29.1 ± 0 8.8
53.8% ± 14.6pp
Appendix
Table 2: Average number of questions written per expert and inter-expert coverage ( Section B.2 ) per research problem.
Drug offences
Medical errors
Personal rights
Backbone
vanilla LLM
vanilla LLM + PDH
Sche MatiQ
Sche MatiQ + PDH
schematize
vanilla LLM
vanilla LLM + PDH
Sche MatiQ
Sche MatiQ + PDH
schematize
vanilla LLM
vanilla LLM + PDH
Sche MatiQ
Sche MatiQ + PDH
schematize
gpt-5.4
49.0%
87.0%
59.3%
84.3%
94.0%
25.9%
53.4%
28.1%
57.2%
71.5%
33.9%
52.5%
43.9%
51.2%
72.8%
gpt-5.4-mini
38.0%
72.7%
15.9%
66.3%
83.2%
23.9%
47.9%
31.7%
54.1%
70.6%
33.3%
46.0%
24.1%
52.4%
70.6%
gpt-5.4-nano
41.2%
71.5%
0 9.2%
63.6%
0 1.9%
20.0%
52.3%
26.1%
22.8%
30.7%
0 5.5%
55.6%
36.6%
41.0%
71.4%
claude-sonnet-4.6
49.5%
74.9%
82.6%
84.1%
89.9%
29.7%
50.1%
27.3%
57.2%
53.5%
32.1%
56.4%
38.8%
59.9%
66.7%
qwen3.6-35b-a3b
27.9%
54.5%
47.3%
84.0%
83.9%
20.2%
50.1%
22.6%
54.0%
51.5%
37.5%
50.2%
51.4%
63.5%
64.0%
Appendix
Table 3: Per-problem expert-question coverage (%) across backbones and configurations; columns match Table 1 . Bold marks the best, underline the second-best, configuration per backbone and problem.
Ablation
Drug offences
Medical errors
Personal rights
Overall
Full ( schematize )
+ 83.9%
+ 51.5%
+ 64.0%
+ 66.5% ± 16.3pp
w/o PDH
-31.5pp
0 +5.5pp
-16.9pp
-14.3pp ± 18.7pp
w/o CBR
0 +0.1pp
0 +0.2pp
0 +4.4pp
0 +1.5pp ± 0 2.5pp
w/o DGR
0 +0.7pp
0 +3.9pp
0 -7.4pp
0 -1.0pp ± 0 5.8pp
w/o PDH+CBR
-36.2pp
0 -7.4pp
0 -4.3pp
-16.0pp ± 17.5pp
w/o PDH+DGR
-29.4pp
-13.7pp
-18.8pp
-20.6pp ± 0 8.0pp
Appendix
Table 4: Expert-question coverage after selectively ablating components of the schematize pipeline. Entries show the change in coverage (percentage points, pp) relative to the full system, averaged over the research problems.
Backbone
vanilla LLM
ScheMatiQ
schematize
gpt-5.4
0.024\pm$ 0.008
0.176\pm$ 0.076
3.785\pm$ 0.312
gpt-5.4-mini
0.005\pm$ 0.000
0.044\pm$ 0.020
0.828\pm$ 0.059
gpt-5.4-nano
0.002\pm$ 0.001
0.014\pm$ 0.007
0.234\pm$ 0.015
claude-sonnet-4.6
0.027\pm$ 0.006
0.243\pm$ 0.086
3.057\pm$ 0.510
qwen3.6-35b-a3b
0.005\pm$ 0.001
0.033\pm$ 0.007
0.183\pm$ 0.010
gemma-4-e4b-it
0.000\pm$ 0.000
0.016\pm$ 0.001
0.041\pm$ 0.001
Appendix
Table 5: The LLM inference cost in USD per research problem ( mean±std ), per backbone, for the vanilla LLM , ScheMatiQ , and schematize configurations.
Many disciplines pose natural-language research questions over large document collections whose answers typically require structured evidence, traditionally obtained by manually designing an annotation schema and exhaustively labeling the corpus, a slow and error-prone process. We introduce ScheMatiQ, which leverages calls to a backbone LLM to take a question and a corpus to produce a schema and a grounded database, with a web interface that lets steer and revise the extraction. In collaboration with domain experts, we show that ScheMatiQ yields outputs that support real-world analysis in law and computational biology. We release ScheMatiQ as open source with a public web interface, and invite experts across disciplines to use it with their own data. All resources, including the website, source code, and demonstration video, are available at: www.ScheMatiQ-ai.com
Shahar Levy, Eliya Habba, Reshef Mintz +3
School of Computer Science and Engineering, The Hebrew University of Jerusalem · Faculty of Law, The Hebrew University of Jerusalem · Allen Institute for AI
Legal information in India remains largely inaccessible due to the complexity of legal language and the sheer volume of legal documentation involved in research and case analysis. This paper presents NyayaAI, an AI-powered legal assistant that automates and simplifies legal workflows for lawyers, law students, and general users. The system combines Large Language Models with a Retrieval-Augmented Generation pipeline grounded in a curated Indian legal knowledge base comprising constitutional provisions, statutes, case laws, and judicial precedents. A multi-agent architecture orchestrated through the Mastra TypeScript framework coordinates a main agent with specialized sub-agents handling legal research, document summarization, case law retrieval, and drafting assistance. A compliance module validates all responses before delivery. Domain classification achieved 70% precision across test samples, with RAG retrieval precision at 74% and overall response accuracy at 72%, demonstrating that structured multi-agent LLM systems can meaningfully improve legal accessibility and workflow efficiency. The code\footnote{https://github.com/B97784/NyayaAI} is made publicly available for the benefit of the research community.
Enterprise workflows increasingly rely on agents for schema-guided extraction: given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata. We present ExtractBench, a benchmark for schema-guided extraction and, to our knowledge, the first to score value accuracy, record completeness at scale, grounding, and measured cost together. The evaluation system contains 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, with clear tags differentiating their challenge scenarios. The scalable schema and ground-truth curation pipeline combines independent-system agreement for real documents, known values for synthetic lists, and human verification for forms. We report order-insensitive value F1 for value accuracy, plus two grounding metrics for source traceability: word- and page-level F1. Commercial VLMs perform well on short documents but often truncate record lists on long ones, while coding agents retain higher accuracy at much higher cost. LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to coding agents at a fraction of the cost. Dataset and evaluation code are available on HuggingFace and GitHub.