Can LLMs Reason in a Legally Meaningful Manner? A Small-scale Study on European Court of Human Rights Cases

Amogh Raina; Ilias Chalkidis; Daniel Hershcovich; Henrik Palmer Olsen

cs.CLAug 17, 2026

Can LLMs Reason in a Legally Meaningful Manner? A Small-scale Study on European Court of Human Rights Cases

Authors: Amogh Raina, Ilias Chalkidis, Daniel Hershcovich, Henrik Palmer Olsen

Organizations: Department of Computer Science, University of Copenhagen, Denmark · Faculty of Law, University of Copenhagen, Denmark

Abstract

Reasoning has become a standard technique and feature for contemporary LLMs; however, its application and quality in the context of demanding legal-oriented tasks, such as legal case forecasting, remain under explored. We investigate how LLMs reason in the context of legal case forecasting, using legal cases from the European Court of Human Rights (ECtHR) as a testbed. We evaluate OpenAI GPT 5.4, a recent top-tier LLM, by exploring alternative prompting strategies that are more or less suggestive of what counts as legally meaningful reasoning in the context of ECtHR jurisprudence. We present our findings derived from assessing the model's responses with both human and LLM evaluation. We find that the examined model scores far from ideal in legal reasoning, the model produces structurally complete but substantively shallow analyses, and that LLM-as-a-Judge evaluators are internally consistent yet align only weakly with our trained annotators, i.e., reliable but not a valid substitute for human evaluation. Overall, the expert-curated prompt leads to more comprehensive reasoning, which does not result in more accurate predictions compared to the other examined settings. Based on our findings, we urge the community not to rely solely on automated LLM-based evaluation and to avoid using task accuracy as an appropriate proxy for reasoning quality.

Figures & tables

Figure 1: Cartoonish abstract depiction of the real-life ECtHR case workflow from case lodging to the court’s judgment and its publishing; and of our LLM-driven legal judgment-forecasting experiment.

Figure 2: Appropriate reasoning strategy to assess ECHR cases related to Article 10 (Freedom of Expression).

Prompt Setting	Step Occurrence		Step Comprehensiveness		Overall Conciseness		Fact. Ref. F1	Accuracy
Prompt Setting	Human	Models	Human	Models	Human	Models	Fact. Ref. F1	Accuracy
A: Non-Curated	0.96 $\pm$ 0.19	0.92 $\pm$ 0.28	3.95 $\pm$ 1.08	3.61 $\pm$ 1.44	3.87 $\pm$ 0.75	3.83 $\pm$ 0.94	0.51 $\pm$ 0.19	0.82
B: Curated (Expert)	1.00 $\pm$ 0.06	1.00 $\pm$ 0.00	4.45 $\pm$ 0.79	4.13 $\pm$ 0.82	4.38 $\pm$ 0.67	4.49 $\pm$ 0.54	0.51 $\pm$ 0.19	0.77
C: Curated (Guide)	0.99 $\pm$ 0.11	0.99 $\pm$ 0.09	4.22 $\pm$ 0.88	4.00 $\pm$ 0.96	4.10 $\pm$ 0.71	3.92 $\pm$ 0.74	0.53 $\pm$ 0.20	0.77
Majority baseline (always “violation”)	–	–	–	–	–	–	–	0.77

Table 1: Results for each examined setting (A-C) across all metrics, averaged over all steps and the 22 distinct cases for GPT-5.4. For each setting, both Human (three annotators) and Models (LLM-as-a-Judge) ratings are reported. The always-“violation” majority baseline attains

0.77

accuracy; settings B and C match it exactly and A exceeds it by a single case, while reasoning-quality scores differ markedly across settings .i.e., reasoning quality does not track predictive accuracy (agreement with the court’s decision; see Section 4.5 ).

Figure 3: Results for step occurrence and comprehensiveness per step (1-4) across settings (A-C) .

Prompt Setting	Internal reasoning	Output	Total
A: Non-Curated	1709 $\pm$ 577	1341 $\pm$ 138	3050
B: Curated (Expert)	2016 $\pm$ 644	963 $\pm$ 109	2979
C: Curated (Guide)	3550 $\pm$ 1371	1230 $\pm$ 130	4780

Table 2: Mean billed tokens (from the API usage) for the model’s internal reasoning (stage (i): an internal chain of computation, not returned as text) and its disclosed output assessment (stage (iii)), with the total (completion tokens), per setting over the 22 cases. The internal reasoning is billed but not observable in the output; the model emits at most a brief, often empty, reasoning summary (stage (ii)), which we do not evaluate.

Figure 4: Results per Setting (A-C) across the three model judges (Claude Opus 4.7, DeepSeek V4 Pro, and GPT 5.5).

(a) Human inter-annotator agreement (3 annotators)
Criterion	Chance-corr.	Agreement
Step occurrence	$\kappa=0.14$	$97\%$ (exact)
Step comprehensiveness	$\alpha=0.09$	$87\%$ (within-1)
Overall conciseness	$\alpha=0.11$	$91\%$ (within-1)

Table 3: Reliability and validity of the evaluation. (a) Agreement among the three annotators: a chance-corrected coefficient (Fleiss’

\kappa

for binary occurrence; Krippendorff’s ordinal

\alpha

for the

1

–

5

criteria) alongside raw agreement (exact for occurrence, within-one-point for the

1

–

5

criteria). The low chance-corrected values are a prevalence / restricted-range artifact—raw agreement is high (see text). (b) Alignment of each LLM judge with the human consensus: Spearman

\rho

and mean signed bias (judge

-

human; positive

=

more lenient) for comprehensiveness and conciseness, and % agreement on occurrence.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Role	Name (main text)	API identifier (OpenRouter)	Reasoning effort	Release date
Generator	GPT-5.4	openai/gpt-5.4	medium	March 5, 2026
Evaluator	GPT-5.5	openai/gpt-5.5	high	April 23, 2026
Evaluator	Claude Opus 4.7	anthropic/claude-opus-4.7	high	April 16, 2026
Evaluator	DeepSeek V4 Pro	deepseek/deepseek-v4-pro	high	April 24, 2026

Appendix

Table 4: Exact model identifiers and settings. All models were accessed via the OpenRouter API. A, B, and C denote the Non-Curated, Curated (Expert), and Curated (Guide) prompting settings, respectively (Section 3.2 ).

Explore similar work

Toward Human Rights Benchmarking for LLMs: A Pilot Methodology
Aug 10, 2026Savannah Thais, Wm. Matthew Kennedy, Abhigyan Acherjee +3LLM Evaluation Moral Reasoning in Language Models
Hunter College, New York, USA · Oxford Internet Institute, University of Oxford, UK · King’s College London, UK +2
BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law
May 27, 2026Sebastian Nagl, Ann-Kristin Mayrhofer, Martin Heidebach +6Legal NLP LLM Evaluation
Technical University of Munich (TUM) · Ludwig Maximilian University of Munich (LMU) · University of Konstanz +1
LP-Eval: Rubric and Dataset for Measuring the Quality of Legal Proposition Generation
May 19, 2026Shanshan Xu, Johan Lindholm, Amogh Raina +2Legal NLP LLM Evaluation
University of Copenhagen · Umeå University

Aug 10, 2026J/K move · Enter open · S save

Toward Human Rights Benchmarking for LLMs: A Pilot Methodology

Savannah Thais, Wm. Matthew Kennedy, Abhigyan Acherjee, Matilda Wysocki+2

Hunter College, New York, USA · Oxford Internet Institute, University of Oxford, UK · King’s College London, UK+2

LLM Evaluation Moral Reasoning in Language Models Legal Reasoning

Open paper AskPDF arXiv

Large language models (LLMs) increasingly mediate legal determinations over what human rights are realized, and how. Yet, no evaluation benchmark exists to assess whether they can reason correctly about human rights law. To this end, we report our efforts to develop a robust and scalable methodology for creating HumRightsBench: the first expert-validated, scenario-based benchmark for evaluating reasoning grounded in the obligation structure of international human rights law. We adapt the IRAC framework for legal reasoning to better suit the unique reasoning patterns of human rights work (substituting P, "proposing remedies," for C, "legal conclusion," yielding IRAP) to structure our evaluation heuristics. We also produce a pilot series of authentic scenarios designed to implicate the many dimensions of real-world human rights issues and annotated by human rights lawyers and professionals across the world. Ultimately, we find that model accuracy scores range considerably across legal reasoning tasks (overall model performance ranges from 0.339 to 0.577, task min-max ranges from 0.025 to 0.774), which strongly implies that HumRightsBench is a capable instrument for advancing this emerging subfield of AI evaluations science at a critical moment in its evolution.

Abstract

Explore similar work

Toward Human Rights Benchmarking for LLMs: A Pilot Methodology

BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law

LP-Eval: Rubric and Dataset for Measuring the Quality of Legal Proposition Generation