Legal work, with its heavy reliance on processing large amounts of text, is often considered one of the domains most exposed to the use of LLMs. Contract ``scrubbing,'' the final review of transactional agreements for errors and inconsistencies, is a particularly suitable task for automation, because it is routine, painstaking work requiring detailed attention to long documents. Scrubbing also seems to align naturally with the general capabilities expected of frontier LLMs around long-context reasoning, consistency checking, and named entity recognition (NER). Despite the economic value and potential for automation, no formal evaluations of LLMs performing contract scrubbing have been conducted. We introduce ContractScrub, the first benchmark designed to evaluate contract scrubbing capabilities, comprising contracts hand-crafted by experienced lawyers over diverse error categories such as misuse of defined terms, incorrect references, and inconsistent language. Frontier models perform surprisingly poorly with only one model reaching 0.75 macro average recall despite strong performance on seemingly related general benchmarks, demonstrating the practical limits of current models and the importance of narrowly targeted, domain-specific benchmarks for measuring real-world impact.
Figures & tables
Figure 1: An example of errors that should be found in the scrubbing process.
Category
Explanation
Defined Terms
Words or phrases formally defined in the agreement with a designated meaning, often capitalized, underlined, bolded, italicized, or in quotations.
Undefined Capitalized Terms
Terms that are capitalized or otherwise treated as a defined term in an agreement but not formally defined.
Uncapitalized Defined Terms
Formally defined terms appearing in lowercase when they should be capitalized.
Incorrectly Capitalized Terms In Context
Terms that have a definition but are capitalized in a context where they are not being used as a defined term.
Unused Defined Terms
Defined terms that are not used outside of the specific instance where they are defined.
Terms Defined Multiple Times
Terms that are defined more than one time in an agreement.
Table 1: Scrub Review Categories. Each tested contract contains different categories of errors.
Figure 2: ContractScrub construction pipeline. Contracts are sourced and screened for existing structural issues (Stage 1), annotated for existing drafting errors (Stage 2), and augmented with targeted additional errors (Stage 3) to produce the gold answer. All steps are carried out by experienced lawyers.
Model
R
P
F1
Cost ($ | s)
GPT-5.5
0.750
0.580
0.632
1.38 | 533
Gemini 3.1 Pro
0.744
0.616
0.655
0.19 | 76
Claude Sonnet 4.6
0.686
0.620
0.637
1.52 | 258
Gemini 2.5 Pro
0.632
0.527
0.557
0.13 | 48
Claude Opus 4.7
0.616
0.644
0.621
0.66 | 23
GPT-5.2
0.589
0.540
0.553
0.32 | 192
Table 2: Main Results. Precision (P), Recall (R), F1 scores, and cost for all evaluated models. The best score in each column is in bold ; the second best is underlined . Cost column includes the mean price ($) and time (sec.) per contract.
Metric
Model
GPT-5.5
Gemini
Claude
Gemini
Claude
GPT-5.2
Claude
Qwen3.5
o4-mini
3.1 Pro
Sonnet 4.6
2.5 Pro
Opus 4.7
Haiku 4.5
(397B)
Overall Recall
0.750
0.744
0.686
0.632
0.616
0.589
0.445
0.438
0.409
Defined Terms
0.905
0.901
0.862
0.862
0.835
0.876
0.753
0.793
0.730
Undef. Capitalized Terms
0.514
0.440
0.367
0.319
0.447
0.543
0.141
0.216
0.171
Uncapitalized Defined Terms
0.868
0.868
0.703
0.662
0.596
0.628
0.306
0.331
0.227
Table 3: Recall score comparison across models and scrub categories. Color heatmap applied to all metric rows. Recall Overall shown without heatmap for reference. All models use reasoning variants.
Figure 6
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Total
Avg/Contract
Defined Terms
1,505
34.2
Undefined Capitalized Terms
689
15.7
Uncapitalized Defined Terms
317
7.2
Unused Defined Terms
202
4.6
Incorrect Section, Article, or Paragraph References
150
3.4
Incorrect Party References
130
3.0
Appendix
Table 4: Scrub error category counts in ContractScrub and per-contract averages
Undefined Capitalized Terms : Terms that are capitalized or otherwise treated as a defined term in an agreement but not formally defined. For example, an agreement provides: “Party A shall comply with all applicable requirements in the Approved Specification,” but Approved Specification is not defined. This matters because an undefined capitalized term indicates that a contracting party specifically intended a special contractual meaning. As a result, an ambiguity is created and a party may later dispute what specification was approved and whether it was applicable, whether a breach occurred, and whether outside evidence is admissible to contradict, vary, or add to the terms of the agreement.
Appendix
Table 8
Figure 4: Recall by categories for two models with and without reasoning. Solid lines show per-section recall across nine defined-term and reference-checking tasks; dashed horizontal lines indicate each condition’s mean.
Metric
Model
GPT-5.5
Gemini
Claude
Gemini
Claude
GPT-5.2
Claude
Qwen3.5
o4-mini
3.1 Pro
Sonnet 4.6 ‡
2.5 Pro
Opus 4.7
Haiku 4.5
(397B)
Overall Recall
0.799
0.793
0.746
0.698
0.661
0.631
0.478
0.529
0.466
Defined Terms
0.940
0.941
0.919
0.944
0.890
0.933
0.809
0.877
0.791
Undef. Capitalized Terms
0.552
0.460
0.392
0.347
0.505
0.582
0.165
0.253
0.205
Uncapitalized Defined Terms
0.924
0.915
0.776
0.776
0.656
0.707
0.369
0.533
0.297
Appendix
Table 5: Recall score comparison across models and scrub categories. Color heatmap applied to all metric rows. Recall Overall shown without heatmap for reference. All models use reasoning variants.
Metric
Model
GPT-5.5
Gemini
Claude
Gemini
Claude
GPT-5.2
Claude
Qwen3.5
o4-mini
3.1 Pro
Sonnet 4.6 ‡
2.5 Pro
Opus 4.7
Haiku 4.5
(397B)
Overall Precision
0.580
0.616
0.620
0.527
0.644
0.540
0.592
0.268
0.548
Defined Terms
0.878
0.909
0.901
0.860
0.916
0.899
0.890
0.575
0.906
Undef. Capitalized Terms
0.632
0.596
0.634
0.564
0.615
0.570
0.618
0.185
0.608
Uncapitalized Defined Terms
0.327
0.299
0.408
0.325
0.543
0.375
0.406
0.073
0.369
Appendix
Table 6: Precision score comparison across models and scrub categories. Color heatmap applied to all metric rows. Precision Overall shown without heatmap for reference.
Metric
Model
GPT-5.5
Gemini
Claude
Gemini
Claude
GPT-5.2
Claude
Qwen3.5
o4-mini
3.1 Pro
Sonnet 4.6 ‡
2.5 Pro
Opus 4.7
Haiku 4.5
(397B)
Overall F1
0.632
0.655
0.637
0.557
0.621
0.553
0.492
0.316
0.453
Defined Terms
0.891
0.905
0.881
0.861
0.873
0.888
0.816
0.667
0.809
Undef. Capitalized Terms
0.567
0.506
0.465
0.408
0.518
0.556
0.229
0.199
0.267
Uncapitalized Defined Terms
0.475
0.445
0.517
0.436
0.568
0.470
0.349
0.119
0.281
Appendix
Table 7: F1 score comparison across models and scrub categories. Color heatmap applied to all metric rows. F1 Overall shown without heatmap for reference.
Aggregate accuracy summarizes how often a model is correct, but not which judgments remain correct when requests change. We evaluate Jev and nine language models on ContractNLI, with repeated requests that preserve the contract and target while varying visible hypotheses, requested answers, and output order. A four-model follow-up adds an equal-call repetition control and coverage of the remaining test contracts. Corrections and regressions can cancel, leaving unchanged accuracy despite changed decisions; consistent answers can also preserve errors. Majority voting over recorded responses yields small, uncertain gains, with repeated wrong answers accounting for much of the observed error. Gold-class diagnostics further reveal different error profiles behind aggregate model rankings. Resource comparisons add distinct trade-offs: Jev is cheapest and fastest for the joint baseline, while hosted language models achieve higher accuracy. Together, these findings connect decision-level evaluation to whether additional calls improve correctness and what the resulting decisions cost.
Fan Zhang, Yankai Chen, Zhuohan Xie +11
The University of Tokyo · MBZUAI · McGill University +8
As large language models (LLMs) are increasingly applied to real-world legal tasks, evaluating the reliability of their open-ended legal responses has become essential. These tasks require context-sensitive answers and allow little room for error, motivating fine-grained and diagnostic evaluation that can identify specific sources of response quality failures. We introduce LexRubric, a rubric-based benchmark for evaluating open-ended Chinese legal tasks. LexRubric contains 649 instances from legal consultation and judicial examination, which reflect both everyday legal needs and professional legal reasoning and cover 14 legal scenarios. It further includes 12,337 expert-written atomic scoring criteria organized under a unified six-dimensional framework, enabling accurate evaluation and diagnostic analysis across tasks and evaluation dimensions. To validate the reliability of the evaluation, we test multiple judge models and compare model-based judgments with human judgments. We further evaluate 18 recent general and legal-domain LLMs on LexRubric. Results show that different models exhibit distinct capability profiles, and that open-ended legal tasks remain challenging for current LLMs. Data is available at: https://github.com/foggpoy/LexRubric.
Yifan Chen, Haitao Li, Yiran Hu +6
1Beijing University of Posts and Telecommunications · 2Tsinghua University · University of Waterloo +1
This paper evaluates whether a domain trained Small Language Model (SLM) can outperform frontier Large Language Models on structured contract extraction at radically lower cost. We test Olava Extract, a self hosted legal domain Mixture of Experts model, against five frontier models. Olava Extract achieved the strongest aggregate performance in the study, with a macro F1 of 0.812 and a micro F1 of 0.842, while reducing inference cost by 78% to 97% compared with the frontier models tested. It also achieved the highest precision scores, producing fewer hallucinated and unsupported extractions, an important distinction in legal workflows where hallucinations create operational risk and downstream review burden. The findings shows that high performing, human comparable legal AI no longer requires the largest externally hosted models. More broadly, they challenge the assumption that commercially valuable enterprise AI capability must remain tied to ever larger models, massive infrastructure expenditure, and centrally hosted providers.