Reasoning has become a standard technique and feature for contemporary LLMs; however, its application and quality in the context of demanding legal-oriented tasks, such as legal case forecasting, remain under explored. We investigate how LLMs reason in the context of legal case forecasting, using legal cases from the European Court of Human Rights (ECtHR) as a testbed. We evaluate OpenAI GPT 5.4, a recent top-tier LLM, by exploring alternative prompting strategies that are more or less suggestive of what counts as legally meaningful reasoning in the context of ECtHR jurisprudence. We present our findings derived from assessing the model's responses with both human and LLM evaluation. We find that the examined model scores far from ideal in legal reasoning, the model produces structurally complete but substantively shallow analyses, and that LLM-as-a-Judge evaluators are internally consistent yet align only weakly with our trained annotators, i.e., reliable but not a valid substitute for human evaluation. Overall, the expert-curated prompt leads to more comprehensive reasoning, which does not result in more accurate predictions compared to the other examined settings. Based on our findings, we urge the community not to rely solely on automated LLM-based evaluation and to avoid using task accuracy as an appropriate proxy for reasoning quality.
Figures & tables
Figure 1: Cartoonish abstract depiction of the real-life ECtHR case workflow from case lodging to the court’s judgment and its publishing; and of our LLM-driven legal judgment-forecasting experiment.
Figure 2: Appropriate reasoning strategy to assess ECHR cases related to Article 10 (Freedom of Expression).
Prompt Setting
Step Occurrence
Step Comprehensiveness
Overall Conciseness
Fact. Ref. F1
Accuracy
Human
Models
Human
Models
Human
Models
A: Non-Curated
0.96 ± 0.19
0.92 ± 0.28
3.95 ± 1.08
3.61 ± 1.44
3.87 ± 0.75
3.83 ± 0.94
0.51 ± 0.19
0.82
B: Curated (Expert)
1.00 ± 0.06
1.00 ± 0.00
4.45 ± 0.79
4.13 ± 0.82
4.38 ± 0.67
4.49 ± 0.54
0.51 ± 0.19
0.77
C: Curated (Guide)
0.99 ± 0.11
0.99 ± 0.09
4.22 ± 0.88
4.00 ± 0.96
4.10 ± 0.71
3.92 ± 0.74
0.53 ± 0.20
0.77
Majority baseline (always “violation”)
–
–
–
–
–
–
–
0.77
Table 1: Results for each examined setting (A-C) across all metrics, averaged over all steps and the 22 distinct cases for GPT-5.4. For each setting, both Human (three annotators) and Models (LLM-as-a-Judge) ratings are reported. The always-“violation” majority baseline attains 0.77 accuracy; settings B and C match it exactly and A exceeds it by a single case, while reasoning-quality scores differ markedly across settings .i.e., reasoning quality does not track predictive accuracy (agreement with the court’s decision; see Section 4.5 ).
Figure 3: Results for step occurrence and comprehensiveness per step (1-4) across settings (A-C) .
Prompt Setting
Internal reasoning
Output
Total
A: Non-Curated
1709 ± 577
1341 ± 138
3050
B: Curated (Expert)
2016 ± 644
963 ± 109
2979
C: Curated (Guide)
3550 ± 1371
1230 ± 130
4780
Table 2: Mean billed tokens (from the API usage) for the model’s internal reasoning (stage (i): an internal chain of computation, not returned as text) and its disclosed output assessment (stage (iii)), with the total (completion tokens), per setting over the 22 cases. The internal reasoning is billed but not observable in the output; the model emits at most a brief, often empty, reasoning summary (stage (ii)), which we do not evaluate.
Figure 4: Results per Setting (A-C) across the three model judges (Claude Opus 4.7, DeepSeek V4 Pro, and GPT 5.5).
(a) Human inter-annotator agreement (3 annotators)
Criterion
Chance-corr.
Agreement
Step occurrence
κ=0.14
97% (exact)
Step comprehensiveness
α=0.09
87% (within-1)
Overall conciseness
α=0.11
91% (within-1)
Table 3: Reliability and validity of the evaluation. (a) Agreement among the three annotators: a chance-corrected coefficient (Fleiss’ κ for binary occurrence; Krippendorff’s ordinal α for the 1 – 5 criteria) alongside raw agreement (exact for occurrence, within-one-point for the 1 – 5 criteria). The low chance-corrected values are a prevalence / restricted-range artifact—raw agreement is high (see text). (b) Alignment of each LLM judge with the human consensus: Spearman ρ and mean signed bias (judge − human; positive = more lenient) for comprehensiveness and conciseness, and % agreement on occurrence.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Role
Name (main text)
API identifier (OpenRouter)
Reasoning effort
Release date
Generator
GPT-5.4
openai/gpt-5.4
medium
March 5, 2026
Evaluator
GPT-5.5
openai/gpt-5.5
high
April 23, 2026
Evaluator
Claude Opus 4.7
anthropic/claude-opus-4.7
high
April 16, 2026
Evaluator
DeepSeek V4 Pro
deepseek/deepseek-v4-pro
high
April 24, 2026
Appendix
Table 4: Exact model identifiers and settings. All models were accessed via the OpenRouter API. A, B, and C denote the Non-Curated, Curated (Expert), and Curated (Guide) prompting settings, respectively (Section 3.2 ).
Large language models (LLMs) increasingly mediate legal determinations over what human rights are realized, and how. Yet, no evaluation benchmark exists to assess whether they can reason correctly about human rights law. To this end, we report our efforts to develop a robust and scalable methodology for creating HumRightsBench: the first expert-validated, scenario-based benchmark for evaluating reasoning grounded in the obligation structure of international human rights law. We adapt the IRAC framework for legal reasoning to better suit the unique reasoning patterns of human rights work (substituting P, "proposing remedies," for C, "legal conclusion," yielding IRAP) to structure our evaluation heuristics. We also produce a pilot series of authentic scenarios designed to implicate the many dimensions of real-world human rights issues and annotated by human rights lawyers and professionals across the world. Ultimately, we find that model accuracy scores range considerably across legal reasoning tasks (overall model performance ranges from 0.339 to 0.577, task min-max ranges from 0.025 to 0.774), which strongly implies that HumRightsBench is a capable instrument for advancing this emerging subfield of AI evaluations science at a critical moment in its evolution.
Savannah Thais, Wm. Matthew Kennedy, Abhigyan Acherjee +3
Hunter College, New York, USA · Oxford Internet Institute, University of Oxford, UK · King’s College London, UK +2
We introduce BenGER (Benchmark for German Law), a benchmark and dataset for evaluating LLM systems on subsumption-based legal reasoning in German law. The dataset combines 596 exam-style free-text legal case tasks across multiple levels of legal education and 531 short doctrinal reasoning tasks. It includes a controlled validation subset of timed human-written solutions under both unaided and human-AI co-creation conditions. We evaluate 12 contemporary LLM systems - closed flagship, efficiency-oriented, and open-weight - with a rubric-aligned LLM-as-a-Judge cross-validated against a multi-rater human-grading layer (three blind reviews per solution, six judge families benchmarked against the human pool). Closed-flagship systems lead the leaderboard across all three corpora, human-AI co-creation measurably improves on unaided human work, and the LLM judge tracks human grading at Pearson r=0.76 and Cohen's k=0.60. System rankings are stable across judge families and two judges from independent providers clear the Calderon single-reviewer replacement bar on human-authored solutions.
Sebastian Nagl, Ann-Kristin Mayrhofer, Martin Heidebach +6
Technical University of Munich (TUM) · Ludwig Maximilian University of Munich (LMU) · University of Konstanz +1
Legal proposition generation is central to legal reasoning and doctrinal scholarship, yet remain under-examined in Legal NLP. This paper investigates the automatic generation and evaluation of legal propositions from decisions of the Court of Justice of the European Union using large language models (LLMs). We introduce LP-Eval, a three-step evaluation rubric co-designed with legal experts that decomposes legal proposition quality into formal validity and substantive dimensions. Using this rubric, we release a dataset of two experts' annotations for 100 LLM-generated legal propositions. Our results show that LLMs can generate predominantly well-formed and high-quality propositions, while expert evaluations reveal higher quality for propositions derived from well established cases than from recent ones. We further examine LLMs as evaluators and find that rubric-guided LLM judgments align more closely with expert assessments than direct overall scoring, but remain insensitive to finer-grained distinctions captured by human experts.