Empirical legal research often relies on turning research questions into structured data extracted from large collections of rulings and judgments. Designing the extraction schema and then extracting the data remain a manual, expertise-heavy bottleneck. We present schematize, an open-source multi-agent system that interactively turns a researcher's problem statement into a validated extraction schema that can later be used for autonomous extraction. Schematize couples (i) a clarification dialogue that elicits implicit expert intent, (ii) iterative schema generation, (iii) data-grounded refinement that tests the schema against documents, and (iv) chat-based post-editing. We evaluated the system with human legal professional, introducing our novel methodology, and schematize achieves top performance in most of tested configurations. While the system is designed to be domain-agnostic and applicable to any document collection, we tailor and evaluate it on legal research problems. We release schematize as a pip-installable Python package with full documentation.
Figures & tables
Figure 1: Overview of schematize ’s agentic pipeline. A user request is clarified, formalized into retrieval queries and an initial schema, refined via criteria-based (CBR, up to maxref iterations) and data-grounded (DGR, up to maxdref iterations) loops, and finalized through a summary and interactive chat. Blue ( ) marks automated agent steps, orange ( ) marks human-in-the-loop steps, violet marks the document store, and teal marks refinement loops.
Backbone
vanilla LLM
vanilla LLM + PDH
ScheMatiQ
ScheMatiQ + PDH
schematize
gpt-5.4
36.3% ± 11.7pp
64.3% ± 19.7pp
43.8% ± 15.6pp
64.3% ± 17.6pp
79.4% ± 12.6pp
gpt-5.4-mini
31.7% ± 0 7.2pp
55.5% ± 14.9pp
23.9% ± 0 7.9pp
57.6% ± 0 7.6pp
74.8% ± 0 7.2pp
gpt-5.4-nano
22.2% ± 18.0pp
59.8% ± 10.3pp
24.0% ± 13.8pp
42.5% ± 20.4pp
34.7% ± 35.0pp
claude-sonnet-4.6
37.1% ± 10.8pp
60.5% ± 12.9pp
49.6% ± 29.2pp
67.1% ± 14.8pp
70.0% ± 18.5pp
qwen3.6-35b-a3b
28.5% ± 0 8.7pp
51.6% ± 0 2.5pp
40.4% ± 15.6pp
67.2% ± 15.3pp
66.5% ± 16.3pp
gemma-4-e4b-it
0 3.8% ± 0 4.0pp
38.2% ± 14.8pp
34.5% ± 18.2pp
46.5% ± 0 4.8pp
53.4% ± 0 3.7pp
Table 1: Expert-question coverage (mean ± std, in pp) across backbones and configurations. “+ PDH” prepends the Problem-Definition-Helper stage to a baseline. Bold marks the best, underline the second-best, coverage per backbone.
Figure 2: Comparison of the schematize to all tested baselines. The dashed line marks the average inter-expert coverage ( Table 2 ).
Figure 3: Coverage change over subsequent iterations of schema refinements for three considered problems.
Figure 4: Results of the hyperparameter sensitivity study. We measure coverage of the tested configurations for three legal research problems.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Expert-based evaluation pipeline. Given a shared research problem, each domain expert writes an exhaustive question set the extracted data should answer, while the agentic system independently produces an extraction schema . An LLM-as-judge checks whether each expert question is covered by the schema, yielding per-expert and average coverage scores.
Problem
Questions/expert
Inter-expert coverage
Drug offences
28.3 ± 0 9.6
56.4% ± 0 7.9pp
Medical errors
33.0 ± 13.0
52.7% ± 19.0pp
Personal rights
26.0 ± 0 3.0
52.3% ± 17.1pp
Overall
29.1 ± 0 8.8
53.8% ± 14.6pp
Appendix
Table 2: Average number of questions written per expert and inter-expert coverage ( Section B.2 ) per research problem.
Drug offences
Medical errors
Personal rights
Backbone
vanilla LLM
vanilla LLM + PDH
Sche MatiQ
Sche MatiQ + PDH
schematize
vanilla LLM
vanilla LLM + PDH
Sche MatiQ
Sche MatiQ + PDH
schematize
vanilla LLM
vanilla LLM + PDH
Sche MatiQ
Sche MatiQ + PDH
schematize
gpt-5.4
49.0%
87.0%
59.3%
84.3%
94.0%
25.9%
53.4%
28.1%
57.2%
71.5%
33.9%
52.5%
43.9%
51.2%
72.8%
gpt-5.4-mini
38.0%
72.7%
15.9%
66.3%
83.2%
23.9%
47.9%
31.7%
54.1%
70.6%
33.3%
46.0%
24.1%
52.4%
70.6%
gpt-5.4-nano
41.2%
71.5%
0 9.2%
63.6%
0 1.9%
20.0%
52.3%
26.1%
22.8%
30.7%
0 5.5%
55.6%
36.6%
41.0%
71.4%
claude-sonnet-4.6
49.5%
74.9%
82.6%
84.1%
89.9%
29.7%
50.1%
27.3%
57.2%
53.5%
32.1%
56.4%
38.8%
59.9%
66.7%
qwen3.6-35b-a3b
27.9%
54.5%
47.3%
84.0%
83.9%
20.2%
50.1%
22.6%
54.0%
51.5%
37.5%
50.2%
51.4%
63.5%
64.0%
Appendix
Table 3: Per-problem expert-question coverage (%) across backbones and configurations; columns match Table 1 . Bold marks the best, underline the second-best, configuration per backbone and problem.
Ablation
Drug offences
Medical errors
Personal rights
Overall
Full ( schematize )
+ 83.9%
+ 51.5%
+ 64.0%
+ 66.5% ± 16.3pp
w/o PDH
-31.5pp
0 +5.5pp
-16.9pp
-14.3pp ± 18.7pp
w/o CBR
0 +0.1pp
0 +0.2pp
0 +4.4pp
0 +1.5pp ± 0 2.5pp
w/o DGR
0 +0.7pp
0 +3.9pp
0 -7.4pp
0 -1.0pp ± 0 5.8pp
w/o PDH+CBR
-36.2pp
0 -7.4pp
0 -4.3pp
-16.0pp ± 17.5pp
w/o PDH+DGR
-29.4pp
-13.7pp
-18.8pp
-20.6pp ± 0 8.0pp
Appendix
Table 4: Expert-question coverage after selectively ablating components of the schematize pipeline. Entries show the change in coverage (percentage points, pp) relative to the full system, averaged over the research problems.
Backbone
vanilla LLM
ScheMatiQ
schematize
gpt-5.4
0.024\pm$ 0.008
0.176\pm$ 0.076
3.785\pm$ 0.312
gpt-5.4-mini
0.005\pm$ 0.000
0.044\pm$ 0.020
0.828\pm$ 0.059
gpt-5.4-nano
0.002\pm$ 0.001
0.014\pm$ 0.007
0.234\pm$ 0.015
claude-sonnet-4.6
0.027\pm$ 0.006
0.243\pm$ 0.086
3.057\pm$ 0.510
qwen3.6-35b-a3b
0.005\pm$ 0.001
0.033\pm$ 0.007
0.183\pm$ 0.010
gemma-4-e4b-it
0.000\pm$ 0.000
0.016\pm$ 0.001
0.041\pm$ 0.001
Appendix
Table 5: The LLM inference cost in USD per research problem ( mean±std ), per backbone, for the vanilla LLM , ScheMatiQ , and schematize configurations.
School of Computer Science and Engineering, The Hebrew University of Jerusalem · Faculty of Law, The Hebrew University of Jerusalem · Allen Institute for AI