AnnoCaseLaw: Legal Annotation Benchmark of Civil Negligence
Authors: Rilton Franzone, Magnus Sesodia, Alina Petrova, John Armour, Thomas Lukasiewicz, Oana-Maria Camburu, Puneet K. Dokania, Philip Torr, Christian Schroeder de Witt, Fabio J. Fehr
Organizations: University of Oxford · Thomson Reuters Labs · Vienna University of Technology · University College London
Language models offer a way to scale costly, labour-intensive expert annotation, but their ability to reproduce expert judgements remains uncertain. This is particularly challenging in law, where annotators must distinguish the reasoning supporting judicial decisions from other legally relevant discussion. We present AnnoCaseLaw as a densely annotated benchmark comprising 471 U.S. civil-negligence court opinions with more than 12,000 expert highlights across five annotation categories. We evaluate proprietary frontier, open-weight, legal-specialised, and decision models on their ability to reproduce these annotations. Models perform better on Facts and Outcome than on Application of Law to Facts, Procedural History, and Relevant Precedents. On Application of Law to Facts, proprietary frontier models achieve the strongest balance of reasoning coverage and precision, while the open-weight model Gemma 4 achieves comparable Macro-F1 to proprietary frontier models at a fraction of the estimated inference cost. Our findings highlight the importance of evaluating annotation coverage and cost alongside aggregate scores when assessing the feasibility of automated legal annotation.
Figures & tables
Figure 1: AnnoCaseLaw expert annotations. Five categories illustrated in St. Mary’s Hospital v. Bynum (1978). Sentence numbers refer to the full opinion; bracketed ellipses mark omitted text. Unhighlighted text was not annotated.
Table 1: Models perform better on Facts and Outcome than on nuanced legal categories. Per-opinion F1 (%); ± denotes the half-width of a 95% bootstrap confidence interval. Macro-F1 averages the five category scores. The select-all baseline labels every scorable item. Sizes indicate total/active parameters; † marks legal-specialised models. Bold and underline mark the highest and second-highest point estimates per column.
Figure 2: Proprietary frontier models achieve the strongest precision–recall balance on ALF. Recall, precision and F1 (%), averaged separately over opinions and ordered by F1. Mean F1 is not calculated from the displayed mean precision and recall.
Figure 3: Gemma 4 achieves comparable Macro-F1 to proprietary frontier models at lower estimated inference cost. Macro-F1 across five categories versus estimated inference cost for all 471 opinions. Small dots show values; lines connect them to model labels. Cost calculations are provided in Appendix C.4 .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Opinions
Spans
Items
Facts
470
3,906
10,848
PH
448
1,601
2,254
ALF
471
2,770
5,164
Outcome
462
927
1,015
Precedents
409
3,334
2,139
Appendix
Table 2: Annotation counts by category. Opinions are those with reference items; spans are original highlights. Items are distinct cited cases for Precedents and sentences for the other categories.
Model
Serving
T
p
k
Thinking
Output cap
Context
GPT-5.6 Sol
API
–
–
–
high
32,768
1,050,000
Claude Opus 5
API
–
–
–
adaptive/high
32,768
1,000,000
Qwen3.8 Max
API
0
–
–
–
32,768
1,000,000
Gemma 4 31B
API
0
–
–
–
8,192
256,000
DeepSeek V4 Pro
API
0
–
–
–
32,768
1,048,576
Kimi K3
API
1
–
–
high
32,768
1,000,000
Appendix
Table 3: Generation settings. Dashes denote no explicit override. Output caps and context sizes are in tokens. SaulLM and DISC-LawLLM use smaller output caps when needed to fit the full opinion; inputs are never truncated.
Model
Endpoint or checkpoint
Weights revision
Source
Sol
openai-gpt-5.6-sol
–
( OpenAI, 2026 )
Opus
anthropic-claude-opus-5
–
( Anthropic, 2026 )
Qwen
qwen3.8-max
–
( Qwen Team, 2026 )
Gemma
gemma-4-31B-it
–
( Gemma Team, 2026 )
DeepSeek
deepseek-v4-pro
–
( DeepSeek-AI, 2026 )
Kimi
kimi-k3
–
( Kimi Team, 2026 )
Appendix
Table 4: Model identifiers and references. Checkpoint links point to the evaluated revisions, fixing the weights and tokenizer; API aliases do not expose fixed revisions.
Model
Input
Cached
Output
Total
Opus
5
0.5
25
84.46
Sol
4
0.4
20
103.01
Qwen
2
0.2
6
13.30
Kimi
3
0.3
15
121.45
DeepSeek
1.74
0.348
3.48
9.32
Gemma
0.18
0.036
0.5
1.07
Appendix
Table 5: API prices (USD per million tokens) and estimated total benchmark cost (USD). Cached denotes cache reads; cache writes cost 5 for Sol and 6.25 for Opus. A dash denotes no separate rate.
Model
GPU
Hours
USD/h
Total (USD)
Thomson
H100
≥ 6.186
4.41
≥ 27.28
Snowdon
H100
3.598
4.41
15.87
SaulLM
H200
2.390
4.47
10.68
DISC-LawLLM
H100
1.653
4.41
7.29
Kev
B300
1.665
8.00
13.32
Clef
H100
5.732
4.41
25.28
Appendix
Table 6: GPU rental rates and estimated benchmark costs. Clef and Clef Flash use estimated H100 inference hours; other durations include setup and idle time. Thomson’s cost is a lower bound.
Model
Precision
Recall
Selections
Ratio
Opus
47.1
75.5
9,150
1.77
Sol
48.3
75.8
9,295
1.80
Qwen
47.7
69.7
8,025
1.55
Kimi
31.4
87.8
18,155
3.52
DeepSeek
29.9
78.6
19,442
3.76
Gemma
57.4
53.7
4,966
0.96
Appendix
Table 7: ALF precision and recall (%), averaged over 471 opinions. Selections count scorable sentences across these opinions; the ratio divides this count by the 5,164 expert-selected sentences.
Model and procedure
ALF
PH
Precedents
Facts
Outcome
Macro-F1
Gemma (joint selection)
47.5
53.0
60.5
77.0
79.6
63.5
Gemma (yes/no)
52.5
43.2
59.2
66.2
71.7
58.6
Jev (yes/no)
38.2
50.8
60.8
77.5
76.0
60.7
Kev (yes/no)
48.9
38.1
64.6
75.2
84.2
62.2
Clef 27B (yes/no)
45.4
42.0
62.1
77.6
73.7
60.2
Clef Flash 9B (yes/no)
32.1
45.9
58.3
77.0
77.5
58.2
Appendix
Table 8: Exploratory comparison of joint selection and yes/no annotation on the same 40 opinions. Gemma (generative) uses both procedures; Jev, Kev, Clef and Clef Flash use yes/no questions with a 0.5 cut-off. F1 (%) averages opinions with reference annotations: 40 for ALF and Facts, 36 for PH and Precedents, and 39 for Outcome.