Large language models (LLMs) are increasingly used to draft contractual language, yet conventional accuracy or preference-based evaluations are poorly matched to legal drafting. A clause may be fluent and stylistically polished while still omitting an essential carve-out, allocating risk in an unenforceable way, assuming an inapplicable jurisdiction, or exposing a party to regulatory liability. This paper presents a empirical study design and framework for evaluating LLM-generated contract clauses. The study evaluates four models - Claude Haiku 4.5, Gemini 2.5 Flash Lite, GPT 5.4 Nano, and Qwen 3.5 Flash, across 22 contract clause categories and 34 legally-motivated failure modes. We combine two evaluation frameworks: CLAUSE, which classifies prompts by legal function and failure target, and LENS-CRAFT, which scores outputs across nine legal-quality dimensions. Instead of averaging dimension scores, the study applies a Max Severity Principle so that a single legally decisive defect remains visible. The paper provides the evaluation protocol, taxonomy, analysis plan, and a results structure for reporting empirical findings. We argue that legal AI evaluation should move beyond aggregate accuracy toward clause-specific, failure-mode-driven, and risk-sensitive assessment.
Figures & tables
Code
Evaluation target
C
Compliance and Jurisdiction: regulatory compliance, governing law, venue, and jurisdictional adaptability.
L
Legal Competence and Reasoning: doctrine, terminology, enforceability, and contractual logic.
A
Accuracy and Trustworthiness: avoidance of hallucination, outdated assumptions, and unsupported legal claims.
U
Understanding Context: alignment with party roles, transaction facts, industry, and prompt constraints.
S
Structured Contract Handling: definitions, exceptions, cross-references, procedural triggers, and remedies.
E
Ethics, Privacy and Accountability: confidentiality, data protection, fairness, and responsible allocation of accountability.
Table 1: CLAUSE dimensions used for prompt classification.
Dimension
Evaluation question
Legal Soundness
Is the provision valid, enforceable, and consistent with active legal constraints?
Expressiveness
Is the drafting clear, precise, readable, and free from avoidable ambiguity?
Necessity and Completeness
Does the clause include required carve-outs, exclusions, notice mechanics, remedies, and procedural details?
Scenario Alignment
Does the clause fit the specified business model, party role, transaction scale, and industry context?
Compliance and Regulatory Alignment
Does the clause respect privacy, consumer, security, sectoral, and other regulatory obligations?
Risk Allocation
Is liability, indemnity, warranty, and operational exposure allocated in a commercially plausible and legally defensible way?
Table 2: LENS-CRAFT scoring dimensions. Each dimension is scored on a 1–5 severity scale.
Family
Clause categories
Data, privacy, and security
Confidentiality; Privacy and data protection; Security; Data use; Use of content to train the product.
Technology and IP controls
Intellectual property; Restrictions on reverse engineering; Restrictions on use; Advertising or publication rights.
Risk allocation and remedies
Indemnity; Warranty; Limitations of liability; Insurance; Safety.
Operational service terms
Outage, service interruption, and changes to service; Subcontracting; Third-party involvement; Termination.
Dispute and boilerplate terms
Disputes; Jurisdiction; Invalidity or severability; Waiver.
Table 3: Clause categories evaluated in the study.
Model
Level 1
Level 4–5
Summary pattern
GPT-5.4 Nano
42.6%
12.8%
Strongest legal robustness
Claude Haiku 4.5
30.0%
25.6%
Stable but conservative
Qwen 3.5 Flash
13.3%
25.5%
Operationally unstable
Gemini 2.5 Flash Lite
9.3%
27.9%
Fluent but ambiguous
Table 4: Aggregate model-level risk distribution.
Risk level
Meaning
Count
Share
Level 1
Production-ready
2,120
23.9%
Level 2
Generally safe
1,493
16.8%
Level 3
Ambiguous or risk-prone
3,222
36.3%
Level 4
Problematic or potentially unenforceable
2,027
22.9%
Level 5
Highly dangerous or litigious
7
<0.1 %
Table 5: Overall risk distribution across all evaluated clauses.
Context
L1
L2
L3
L4
L5
B2B
21.0%
17.8%
39.2%
22.0%
0.1%
B2C
19.0%
15.4%
38.3%
27.2%
0.1%
B2G
31.9%
17.4%
31.3%
19.3%
0.1%
Table 6: Normalized risk distribution by transactional context (%).
Model
Level 1
Level 4–5
Summary pattern
GPT-5.4 Nano
35.7%
14.7%
Strongest commercial robustness
Claude Haiku 4.5
26.1%
25.1%
Stable but generalized drafting
Qwen 3.5 Flash
13.1%
22.8%
Moderate operational instability
Gemini 2.5 Flash Lite
9.2%
25.7%
Fluent but commercially ambiguous
Table 7: Model-level risk distribution in B2B context.
Model
Level 1
Level 4–5
Summary pattern
GPT-5.4 Nano
47.9%
14.7%
Strongest consumer-oriented robustness
Claude Haiku 4.5
24.6%
29.9%
Structurally stable but conservative
Qwen 3.5 Flash
9.0%
31.3%
Highest consumer-context instability
Gemini 2.5 Flash Lite
5.7%
32.9%
Fluent but highly ambiguous under consumer protections
Table 8: Model-level risk distribution in B2C context.
Model
Level 1
Level 4–5
Summary pattern
GPT-5.4 Nano
55.5%
9.0%
Strongest public-sector legal robustness
Claude Haiku 4.5
39.4%
21.4%
Stable and compliance-oriented drafting
Qwen 3.5 Flash
18.4%
22.2%
Moderate public-sector instability
Gemini 2.5 Flash Lite
12.9%
24.8%
Fluent but weak under regulatory complexity
Table 9: Model-level risk distribution in B2G context.
Clause category
Dominant failure pattern
Limitation of liability
Overbroad exclusions and weak carve-outs
Indemnity
Asymmetrical or incomplete liability allocation
Intellectual property
Ownership and license-scope ambiguity
Outage/service interruption
Weak remedies and SLA specificity
Jurisdiction
Cross-border enforceability assumptions
Privacy and data protection
Compliance and consent incompleteness
Table 10: Recurring high-risk clause categories and dominant failure patterns.
Sub-component
Risk
Legal Standards
3.43
Nuance and Interpretation
3.35
Nuance Omission
3.33
Jurisdiction
3.23
Ambiguity Handling
3.18
Accuracy Rate
3.10
Table 11: Most fragile CLAUSE sub-components overall.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Clause Category
Primary Weakness
Dominant Failure Pattern
Example Problematic Language
Confidentiality Clauses
Ambiguity and weak procedural safeguards
Missing survival clauses, vague disclosure obligations, and incomplete incident procedures
Legal proposition generation is central to legal reasoning and doctrinal scholarship, yet remain under-examined in Legal NLP. This paper investigates the automatic generation and evaluation of legal propositions from decisions of the Court of Justice of the European Union using large language models (LLMs). We introduce LP-Eval, a three-step evaluation rubric co-designed with legal experts that decomposes legal proposition quality into formal validity and substantive dimensions. Using this rubric, we release a dataset of two experts' annotations for 100 LLM-generated legal propositions. Our results show that LLMs can generate predominantly well-formed and high-quality propositions, while expert evaluations reveal higher quality for propositions derived from well established cases than from recent ones. We further examine LLMs as evaluators and find that rubric-guided LLM judgments align more closely with expert assessments than direct overall scoring, but remain insensitive to finer-grained distinctions captured by human experts.
In this paper, we evaluate open-source generative LLMs on legal Natural Language Inference (NLI). Legal inspectorial processes take place in specific domains and often deal with confidential data. This creates a need for working with local models that do not require labeled training data. We evaluate our models on the ContractNLI benchmark and two NLI4Wills datasets. We successfully reproduce the baseline for the task (Span NLI BERT) and we evaluate multiple open-source LLMs on the same task. We analyze the invalid rate of the models, and their stability across temperature settings and domains. Among the generative models, Gemma-4 26B performs the best, reaching an accuracy of 81.2%, even outperforming the supervised model on one metric. On accuracy, it is not possible to beat the supervised model with zero-shot approaches. Qwen-3.6 35B performs well on both ContractNLI and additional datasets in the legal wills domain. Our findings indicate that zero-shot, open-source, generative LLMs are a viable alternative for real-world legal NLI when no supervised data is available. Our code is available at https://github.com/fbaratov/contractnli-llms.
As large language models (LLMs) are increasingly applied to real-world legal tasks, evaluating the reliability of their open-ended legal responses has become essential. These tasks require context-sensitive answers and allow little room for error, motivating fine-grained and diagnostic evaluation that can identify specific sources of response quality failures. We introduce LexRubric, a rubric-based benchmark for evaluating open-ended Chinese legal tasks. LexRubric contains 649 instances from legal consultation and judicial examination, which reflect both everyday legal needs and professional legal reasoning and cover 14 legal scenarios. It further includes 12,337 expert-written atomic scoring criteria organized under a unified six-dimensional framework, enabling accurate evaluation and diagnostic analysis across tasks and evaluation dimensions. To validate the reliability of the evaluation, we test multiple judge models and compare model-based judgments with human judgments. We further evaluate 18 recent general and legal-domain LLMs on LexRubric. Results show that different models exhibit distinct capability profiles, and that open-ended legal tasks remain challenging for current LLMs. Data is available at: https://github.com/foggpoy/LexRubric.
Yifan Chen, Haitao Li, Yiran Hu +6
1Beijing University of Posts and Telecommunications · 2Tsinghua University · University of Waterloo +1