The Measurement Gap in the Automation of EU Law: Benchmarking Doctrinal Legal Reasoning under the EU AI Act

Michèle Finck

cs.CYJun 16, 2026

The Measurement Gap in the Automation of EU Law: Benchmarking Doctrinal Legal Reasoning under the EU AI Act

Authors: Michèle Finck

Abstract

Large language models now produce legal text of at least median quality, yet no existing benchmark can evaluate whether they perform doctrinal legal reasoning, which forms the interpretive core of legal work, rather than the ancillary, paralegal tasks that most current legal-AI evaluations measure. This measurement gap is not only methodological but legal: the EU AI Act makes "appropriate accuracy" a binding requirement for high-risk AI used in the judicial domain, yet that requirement cannot acquire operational content without the very doctrinal-reasoning benchmark the field lacks.

Explore similar work

Toward Human Rights Benchmarking for LLMs: A Pilot Methodology
Aug 10, 2026Savannah Thais, Wm. Matthew Kennedy, Abhigyan Acherjee +3Legal Domain Artificial Intelligence Evaluation
LP-Eval: Rubric and Dataset for Measuring the Quality of Legal Proposition Generation
May 19, 2026Shanshan Xu, Johan Lindholm, Amogh Raina +2Legal Domain Rubrics
Magis-Bench: Evaluating LLMs on Magistrate-Level Legal Tasks
May 8, 2026Ramon Pires, Thales Sales Almeida, Celio Larcher Junior +6Legal Domain Llm-As-A-Judge

Aug 10, 2026 · cs.LGJ/K move · Enter open · S save

Toward Human Rights Benchmarking for LLMs: A Pilot Methodology

Savannah Thais, Wm. Matthew Kennedy, Abhigyan Acherjee, Matilda Wysocki+2

Large language models (LLMs) increasingly mediate legal determinations over what human rights are realized, and how. Yet, no evaluation benchmark exists to assess whether they can reason correctly about human rights law. To this end, we report our efforts to develop a robust and scalable methodology for creating HumRightsBench: the first expert-validated, scenario-based benchmark for evaluating reasoning grounded in the obligation structure of international human rights law. We adapt the IRAC framework for legal reasoning to better suit the unique reasoning patterns of human rights work (substituting P, "proposing remedies," for C, "legal conclusion," yielding IRAP) to structure our evaluation heuristics. We also produce a pilot series of authentic scenarios designed to implicate the many dimensions of real-world human rights issues and annotated by human rights lawyers and professionals across the world. Ultimately, we find that model accuracy scores range considerably across legal reasoning tasks (overall model performance ranges from 0.339 to 0.577, task min-max ranges from 0.025 to 0.774), which strongly implies that HumRightsBench is a capable instrument for advancing this emerging subfield of AI evaluations science at a critical moment in its evolution.

Legal Domain Artificial Intelligence Evaluation Humanitarian Reports Rights Methodology

Open paper Ask about it PDF

Abstract

Explore similar work

Toward Human Rights Benchmarking for LLMs: A Pilot Methodology

LP-Eval: Rubric and Dataset for Measuring the Quality of Legal Proposition Generation

Magis-Bench: Evaluating LLMs on Magistrate-Level Legal Tasks