Unlocking Criminal Court Decisions: Flexible Information Extraction for Empirical Legal Research
Authors: Tomáš Knap, Ivana Kvapilíková, Jan Černý, Klára Bendová, Jaromír Šavelka, Vojtěch Pour, Jakub Drápal
Organizations: Faculty of Law, Charles University, Prague, Czechia · Faculty of Mathematics and Physics, Charles University, Prague, Czechia · School of Computer Science, Carnegie Mellon University, Pittsburgh PA, USA
Criminal verdicts contain descriptions of criminal behavior that can support quantitative legal research, but structuring them requires attributes and extraction rules that cannot always be fully specified in advance. We present an LLM-based pipeline that builds a structured dataset through attribute exploration, text span extraction, and value normalization, guided by annotation instructions that legal experts can refine. We evaluate it on two traffic offenses: driving under the influence (DUI) and driving despite a ban (DDB). For DUI, revising the instructions was associated with higher extraction accuracy (from 52% to 70% of correctly extracted values) and higher agreement between human annotators. Repeated runs of the same model agreed with each other more than with the gold annotations, showing that consistent output is not necessarily correct, while disagreement between annotators and between models helped reveal unclear instructions. For DDB, where the instructions defined fixed categories and output formats, agreement was high for most attributes.
Figures & tables
Figure 1: English translation of a shortened factual sentence with selected attributes highlighted. The passages are edited for illustration purposes and conciseness (i.e., […]).
Figure 2: The extraction pipeline. The annotation instructions serve as the extraction schema for all three stages: (1) attribute exploration, (2) span extraction and (3) value normalization. Loop (A) induces new attributes every N statements; cycle (B) is the researcher’s revision of the annotation instructions.
Batch size 100
Batch size 200
Attributes
39
30
Mean population rate (%)
32.6
45.2
Median population rate (%)
10.4
48.0
Attributes < 5% population
13 (33%)
4 (13%)
Attributes < 10% population
18 (46%)
6 (20%)
Table 1: Attribute population rates for induction batch sizes 100 and 200 across the same 200 factual statements. Population rate denotes the percentage of sentences in which a value for the attribute was extracted.
Human: Schema 1
Human: Schema 2
LLM: Schema 2
Attribute
Po
κ
Po
κ
Internal
vs. Mini
vs. GPT-5
vs. gold
SubstanceUsed
0.714
0.38
0.859
0.64
0.86
0.40
0.78
0.84
MeasuringMethod
0.662
0.30
0.792
0.52
0.79
0.58
0.56
0.55
LicenseStatus
0.179
− 0.03
0.742
0.57
0.90
0.45
0.41
0.51
ReasonForCheck
0.223
0.01
0.595
0.47
0.67
0.17
0.20
0.12
RefusedTesting
0.464
0.04
0.556
0.29
0.49
0.48
0.16
0.29
Table 2: Agreement on DUI annotations. Human agreement is reported separately for Schema 1 (initial) and Schema 2 (revised), using observed agreement ( Po ) and Cohen’s κ . All LLM comparisons use Schema 2 and report κ : between two repeated runs of gpt-5.6-terra (Internal), between this model and GPT-5 mini or GPT-5, and between this model and gold annotations.
Schema 1
Schema 2
Schema 1
Schema 2
Attribute
span
value
span
value
Attribute
span
value
span
value
dateOfOffense
0.960
–
0.985
–
measuringMethod
0.453
0.557
0.583
0.639
timeOfOffense
0.874
–
0.975
–
substanceUsed
0.316
0.399
0.639
0.923
measuringTime
0.785
–
0.939
–
typeOfRoad
0.354
0.385
0.653
0.762
measuringDevice
0.748
–
0.824
–
refusedTesting
0.421
0.507
0.722
0.667
amountUsed
0.719
–
0.750
–
licenseStatus
0.125
0.282
0.345
0.655
Table 3: Automatic extraction accuracy against curated reference annotations using the initial and revised schemas. Span accuracy applies to all attributes; value accuracy is reported for the seven categorized attributes. Overall accuracy is the share of attribute values that match the reference annotations across all sentences.
Human
LLM
Cross-model
Cross-model
LLM vs.
Attribute
IAA
internal
(vs. mini)
(vs. gpt-5.2)
gold
DateOfOffense
0.789
1.000
1.000
1.000
0.980
TimeOfOffense
0.848
0.990
0.985
0.980
0.965
PlaceOfOffense
0.336
0.789
0.403
0.739
0.482
OffenseMunicipality
0.563
0.964
0.857
0.928
0.758
VehicleType
0.348
1.000
0.822
0.959
0.876
Table 4: Cohen’s κ for DDB across human, repeated-run, cross-model, and model-to-gold comparisons.
Large Language Models (LLMs) are increasingly applied to legal and criminal justice tasks, yet existing work focuses almost exclusively on post-arrest scenarios where the suspect's identity is already known, leaving the critical pre-arrest challenge of inferring suspect characteristics from incomplete evidence largely unexplored. To fill this gap, we introduce the Profiling, Investigation, and Judgment (PIJ), comprising 2,500 real homicide cases from five countries. PIJ evaluates LLMs across three tasks that span the entire criminal investigation pipeline: criminal profiling, which requires abductive reasoning to infer suspect attributes from fragmentary scene evidence, crime process reconstruction, which tests structured information extraction, and sentence prediction, which demands legal deductive reasoning. We evaluate 9 powerful LLMs and find that performance degrades systematically as tasks shift from explicit fact extraction to implicit reasoning over unknown suspect profiles. Categories requiring inferential reasoning, such as motivation and victim-offender relationships, remain the primary bottlenecks. Further analysis reveals substantial gaps between LLMs and human experts, along with pervasive biases in gender, age, and motive attribution. Our findings indicate that pre-arrest inference from incomplete evidence remains an open challenge.
Yutong Yao, Yanjie Cao, Guanhua Chen +5
NLP2CT Lab, Faculty of Information Science and Computing, University of Macau · Faculty of Social Sciences, University of Macau
Court judgments are central to legal practice and jurisprudence, yet discourse analysis of Hong Kong judgments has received limited attention, owing largely to the absence of expert-annotated corpora. We introduce the Hong Kong Judgment Discourse Dataset (HKJudge), the first sentence-level expert-annotated legal discourse corpus. HKJudge includes criminal judgments across all five levels of HK's court hierarchy, comprising ∼290k sentences and ∼6.5 million tokens, fully annotated by legal linguistics experts. We design a two-tier discourse schema that captures what facts a court finds, how it reasons, and what it rules. At the sentence level, each sentence is assigned one of 26 rhetorical roles. At the span level, sentences are further annotated with three sentencing elements (charge, imprisonment term, fine). Ten legal linguistics annotators produced the annotations with an inter-annotator agreement of κ=0.8. We formulate two tasks on HKJudge, termed rhetorical role classification and legal element extraction, and provide the first benchmark evaluation of four BERT-based models, two open-source LLMs under zero-shot and fine-tuning settings, and four commercial LLMs on both tasks. Our work demonstrates the value of sentence-level discourse annotation for modeling the structure of HK judgments and provides a rich data foundation for future work on legal judgment prediction. The HKJudge dataset and code are available at https://github.com/xuanxixi/HKJudge.
Xi Xuan, Wenxin Zhang, Yufei Zhou +2
City University of Hong Kong · University of Chinese Academy of Sciences
Legal Judgment Prediction (LJP) has become a core benchmark for evaluating AI in the criminal legal domain, but it only sees criminal cases that have already passed prosecutorial review and been formally indicted. As a result, LJP leaves a substantial blind spot in assessing criminal liability, overlooking cases involving insufficient evidence, no criminal liability, or guilt exempted from punishment. To fill this gap, we propose Prosecution Decision Prediction (PDP), the first Legal AI task built around prosecutorial review, which classifies each case into prosecution or one of three non-prosecution decisions and reflects legal AI's capabilities in evidence evaluation, legal subsumption, and value-based discretion. We further construct PDP-Bench, a benchmark of 4{,}630 real Chinese prosecutorial decisions spanning 190 charges. Extensive experiments show that state-of-the-art LLMs perform substantially worse on PDP than on LJP and that mainstream enhancement routes fail to close the gap. Moreover, controlled RLVR interventions show that simple outcome rewards fail to produce generalizable PDP discrimination.
Junyu Lu, Qi Wei, Peishuo Zheng +6
1Beijing Institute of Technology, Zhuhai · 2Osaka University · 3Xi’an University of Technology +2