Legal work, with its heavy reliance on processing large amounts of text, is often considered one of the domains most exposed to the use of LLMs. Contract ``scrubbing,'' the final review of transactional agreements for errors and inconsistencies, is a particularly suitable task for automation, because it is routine, painstaking work requiring detailed attention to long documents. Scrubbing also seems to align naturally with the general capabilities expected of frontier LLMs around long-context reasoning, consistency checking, and named entity recognition (NER). Despite the economic value and potential for automation, no formal evaluations of LLMs performing contract scrubbing have been conducted. We introduce ContractScrub, the first benchmark designed to evaluate contract scrubbing capabilities, comprising contracts hand-crafted by experienced lawyers over diverse error categories such as misuse of defined terms, incorrect references, and inconsistent language. Frontier models perform surprisingly poorly with only one model reaching 0.75 macro average recall despite strong performance on seemingly related general benchmarks, demonstrating the practical limits of current models and the importance of narrowly targeted, domain-specific benchmarks for measuring real-world impact.
Figures & tables
Figure 1: An example of errors that should be found in the scrubbing process.
Category
Explanation
Defined Terms
Words or phrases formally defined in the agreement with a designated meaning, often capitalized, underlined, bolded, italicized, or in quotations.
Undefined Capitalized Terms
Terms that are capitalized or otherwise treated as a defined term in an agreement but not formally defined.
Uncapitalized Defined Terms
Formally defined terms appearing in lowercase when they should be capitalized.
Incorrectly Capitalized Terms In Context
Terms that have a definition but are capitalized in a context where they are not being used as a defined term.
Unused Defined Terms
Defined terms that are not used outside of the specific instance where they are defined.
Terms Defined Multiple Times
Terms that are defined more than one time in an agreement.
Table 1: Scrub Review Categories. Each tested contract contains different categories of errors.
Figure 2: ContractScrub construction pipeline. Contracts are sourced and screened for existing structural issues (Stage 1), annotated for existing drafting errors (Stage 2), and augmented with targeted additional errors (Stage 3) to produce the gold answer. All steps are carried out by experienced lawyers.
Model
R
P
F1
Cost ($ | s)
GPT-5.5
0.750
0.580
0.632
1.38 | 533
Gemini 3.1 Pro
0.744
0.616
0.655
0.19 | 76
Claude Sonnet 4.6
0.686
0.620
0.637
1.52 | 258
Gemini 2.5 Pro
0.632
0.527
0.557
0.13 | 48
Claude Opus 4.7
0.616
0.644
0.621
0.66 | 23
GPT-5.2
0.589
0.540
0.553
0.32 | 192
Table 2: Main Results. Precision (P), Recall (R), F1 scores, and cost for all evaluated models. The best score in each column is in bold ; the second best is underlined . Cost column includes the mean price ($) and time (sec.) per contract.
Metric
Model
GPT-5.5
Gemini
Claude
Gemini
Claude
GPT-5.2
Claude
Qwen3.5
o4-mini
3.1 Pro
Sonnet 4.6
2.5 Pro
Opus 4.7
Haiku 4.5
(397B)
Overall Recall
0.750
0.744
0.686
0.632
0.616
0.589
0.445
0.438
0.409
Defined Terms
0.905
0.901
0.862
0.862
0.835
0.876
0.753
0.793
0.730
Undef. Capitalized Terms
0.514
0.440
0.367
0.319
0.447
0.543
0.141
0.216
0.171
Uncapitalized Defined Terms
0.868
0.868
0.703
0.662
0.596
0.628
0.306
0.331
0.227
Table 3: Recall score comparison across models and scrub categories. Color heatmap applied to all metric rows. Recall Overall shown without heatmap for reference. All models use reasoning variants.
Figure 6
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Total
Avg/Contract
Defined Terms
1,505
34.2
Undefined Capitalized Terms
689
15.7
Uncapitalized Defined Terms
317
7.2
Unused Defined Terms
202
4.6
Incorrect Section, Article, or Paragraph References
150
3.4
Incorrect Party References
130
3.0
Appendix
Table 4: Scrub error category counts in ContractScrub and per-contract averages
Undefined Capitalized Terms : Terms that are capitalized or otherwise treated as a defined term in an agreement but not formally defined. For example, an agreement provides: “Party A shall comply with all applicable requirements in the Approved Specification,” but Approved Specification is not defined. This matters because an undefined capitalized term indicates that a contracting party specifically intended a special contractual meaning. As a result, an ambiguity is created and a party may later dispute what specification was approved and whether it was applicable, whether a breach occurred, and whether outside evidence is admissible to contradict, vary, or add to the terms of the agreement.
Appendix
Table 8
Figure 4: Recall by categories for two models with and without reasoning. Solid lines show per-section recall across nine defined-term and reference-checking tasks; dashed horizontal lines indicate each condition’s mean.
Metric
Model
GPT-5.5
Gemini
Claude
Gemini
Claude
GPT-5.2
Claude
Qwen3.5
o4-mini
3.1 Pro
Sonnet 4.6 ‡
2.5 Pro
Opus 4.7
Haiku 4.5
(397B)
Overall Recall
0.799
0.793
0.746
0.698
0.661
0.631
0.478
0.529
0.466
Defined Terms
0.940
0.941
0.919
0.944
0.890
0.933
0.809
0.877
0.791
Undef. Capitalized Terms
0.552
0.460
0.392
0.347
0.505
0.582
0.165
0.253
0.205
Uncapitalized Defined Terms
0.924
0.915
0.776
0.776
0.656
0.707
0.369
0.533
0.297
Appendix
Table 5: Recall score comparison across models and scrub categories. Color heatmap applied to all metric rows. Recall Overall shown without heatmap for reference. All models use reasoning variants.
Metric
Model
GPT-5.5
Gemini
Claude
Gemini
Claude
GPT-5.2
Claude
Qwen3.5
o4-mini
3.1 Pro
Sonnet 4.6 ‡
2.5 Pro
Opus 4.7
Haiku 4.5
(397B)
Overall Precision
0.580
0.616
0.620
0.527
0.644
0.540
0.592
0.268
0.548
Defined Terms
0.878
0.909
0.901
0.860
0.916
0.899
0.890
0.575
0.906
Undef. Capitalized Terms
0.632
0.596
0.634
0.564
0.615
0.570
0.618
0.185
0.608
Uncapitalized Defined Terms
0.327
0.299
0.408
0.325
0.543
0.375
0.406
0.073
0.369
Appendix
Table 6: Precision score comparison across models and scrub categories. Color heatmap applied to all metric rows. Precision Overall shown without heatmap for reference.
Metric
Model
GPT-5.5
Gemini
Claude
Gemini
Claude
GPT-5.2
Claude
Qwen3.5
o4-mini
3.1 Pro
Sonnet 4.6 ‡
2.5 Pro
Opus 4.7
Haiku 4.5
(397B)
Overall F1
0.632
0.655
0.637
0.557
0.621
0.553
0.492
0.316
0.453
Defined Terms
0.891
0.905
0.881
0.861
0.873
0.888
0.816
0.667
0.809
Undef. Capitalized Terms
0.567
0.506
0.465
0.408
0.518
0.556
0.229
0.199
0.267
Uncapitalized Defined Terms
0.475
0.445
0.517
0.436
0.568
0.470
0.349
0.119
0.281
Appendix
Table 7: F1 score comparison across models and scrub categories. Color heatmap applied to all metric rows. F1 Overall shown without heatmap for reference.