cs.CLNov 2, 2026

Unlocking Criminal Court Decisions: Flexible Information Extraction for Empirical Legal Research

Authors: Tomáš Knap, Ivana Kvapilíková, Jan Černý, Klára Bendová, Jaromír Šavelka, Vojtěch Pour, Jakub Drápal

Organizations: Faculty of Law, Charles University, Prague, Czechia · Faculty of Mathematics and Physics, Charles University, Prague, Czechia · School of Computer Science, Carnegie Mellon University, Pittsburgh PA, USA

Abstract

Criminal verdicts contain descriptions of criminal behavior that can support quantitative legal research, but structuring them requires attributes and extraction rules that cannot always be fully specified in advance. We present an LLM-based pipeline that builds a structured dataset through attribute exploration, text span extraction, and value normalization, guided by annotation instructions that legal experts can refine. We evaluate it on two traffic offenses: driving under the influence (DUI) and driving despite a ban (DDB). For DUI, revising the instructions was associated with higher extraction accuracy (from 52% to 70% of correctly extracted values) and higher agreement between human annotators. Repeated runs of the same model agreed with each other more than with the gold annotations, showing that consistent output is not necessarily correct, while disagreement between annotators and between models helped reveal unclear instructions. For DDB, where the instructions defined fixed categories and output formats, agreement was high for most attributes.

Figures & tables

Explore similar work

CardsList
  1. Before the Arrest: Benchmarking LLMs on Criminal Profiling from Incomplete Evidence

    Sep 17, 2026Yutong Yao, Yanjie Cao, Guanhua Chen +5Abductive ReasoningLegal Judgment Prediction

  2. HKJudge: A Legal Discourse-Annotated Corpus for Interpreting What Courts Find, How They Reason, and What They Rule

    Jun 4, 2026Xi Xuan, Wenxin Zhang, Yufei Zhou +2Legal NLPDocument Information Extraction

  3. The Cases LJP Never Sees: Prosecution Decision Prediction for More Complete Criminal Liability Assessment

    May 27, 2026Junyu Lu, Qi Wei, Peishuo Zheng +6AI-Assisted Decision MakingLegal Reasoning