cs.CLAug 22, 2026
SaveBeyond Accuracy and Surface Fluency: Risk-Sensitive Evaluation of LLMs for Legal Clause Generation
Organizations: AI Tech Ethics
Abstract
Large language models (LLMs) are increasingly used to draft contractual language, yet conventional accuracy or preference-based evaluations are poorly matched to legal drafting. A clause may be fluent and stylistically polished while still omitting an essential carve-out, allocating risk in an unenforceable way, assuming an inapplicable jurisdiction, or exposing a party to regulatory liability. This paper presents a empirical study design and framework for evaluating LLM-generated contract clauses. The study evaluates four models - Claude Haiku 4.5, Gemini 2.5 Flash Lite, GPT 5.4 Nano, and Qwen 3.5 Flash, across 22 contract clause categories and 34 legally-motivated failure modes. We combine two evaluation frameworks: CLAUSE, which classifies prompts by legal function and failure target, and LENS-CRAFT, which scores outputs across nine legal-quality dimensions. Instead of averaging dimension scores, the study applies a Max Severity Principle so that a single legally decisive defect remains visible. The paper provides the evaluation protocol, taxonomy, analysis plan, and a results structure for reporting empirical findings. We argue that legal AI evaluation should move beyond aggregate accuracy toward clause-specific, failure-mode-driven, and risk-sensitive assessment.
Figures & tables
| Code | Evaluation target |
|---|---|
| C | Compliance and Jurisdiction: regulatory compliance, governing law, venue, and jurisdictional adaptability. |
| L | Legal Competence and Reasoning: doctrine, terminology, enforceability, and contractual logic. |
| A | Accuracy and Trustworthiness: avoidance of hallucination, outdated assumptions, and unsupported legal claims. |
| U | Understanding Context: alignment with party roles, transaction facts, industry, and prompt constraints. |
| S | Structured Contract Handling: definitions, exceptions, cross-references, procedural triggers, and remedies. |
| E | Ethics, Privacy and Accountability: confidentiality, data protection, fairness, and responsible allocation of accountability. |
Table 1: CLAUSE dimensions used for prompt classification.
| Dimension | Evaluation question |
|---|---|
| Legal Soundness | Is the provision valid, enforceable, and consistent with active legal constraints? |
| Expressiveness | Is the drafting clear, precise, readable, and free from avoidable ambiguity? |
| Necessity and Completeness | Does the clause include required carve-outs, exclusions, notice mechanics, remedies, and procedural details? |
| Scenario Alignment | Does the clause fit the specified business model, party role, transaction scale, and industry context? |
| Compliance and Regulatory Alignment | Does the clause respect privacy, consumer, security, sectoral, and other regulatory obligations? |
| Risk Allocation | Is liability, indemnity, warranty, and operational exposure allocated in a commercially plausible and legally defensible way? |
Table 2: LENS-CRAFT scoring dimensions. Each dimension is scored on a 1–5 severity scale.
| Family | Clause categories |
|---|---|
| Data, privacy, and security | Confidentiality; Privacy and data protection; Security; Data use; Use of content to train the product. |
| Technology and IP controls | Intellectual property; Restrictions on reverse engineering; Restrictions on use; Advertising or publication rights. |
| Risk allocation and remedies | Indemnity; Warranty; Limitations of liability; Insurance; Safety. |
| Operational service terms | Outage, service interruption, and changes to service; Subcontracting; Third-party involvement; Termination. |
| Dispute and boilerplate terms | Disputes; Jurisdiction; Invalidity or severability; Waiver. |
Table 3: Clause categories evaluated in the study.
| Model | Level 1 | Level 4–5 | Summary pattern |
|---|---|---|---|
| GPT-5.4 Nano | 42.6% | 12.8% | Strongest legal robustness |
| Claude Haiku 4.5 | 30.0% | 25.6% | Stable but conservative |
| Qwen 3.5 Flash | 13.3% | 25.5% | Operationally unstable |
| Gemini 2.5 Flash Lite | 9.3% | 27.9% | Fluent but ambiguous |
Table 4: Aggregate model-level risk distribution.
| Risk level | Meaning | Count | Share |
|---|---|---|---|
| Level 1 | Production-ready | 2,120 | 23.9% |
| Level 2 | Generally safe | 1,493 | 16.8% |
| Level 3 | Ambiguous or risk-prone | 3,222 | 36.3% |
| Level 4 | Problematic or potentially unenforceable | 2,027 | 22.9% |
| Level 5 | Highly dangerous or litigious | 7 | % |
Table 5: Overall risk distribution across all evaluated clauses.
| Context | L1 | L2 | L3 | L4 | L5 |
|---|---|---|---|---|---|
| B2B | 21.0% | 17.8% | 39.2% | 22.0% | 0.1% |
| B2C | 19.0% | 15.4% | 38.3% | 27.2% | 0.1% |
| B2G | 31.9% | 17.4% | 31.3% | 19.3% | 0.1% |
Table 6: Normalized risk distribution by transactional context (%).
| Model | Level 1 | Level 4–5 | Summary pattern |
|---|---|---|---|
| GPT-5.4 Nano | 35.7% | 14.7% | Strongest commercial robustness |
| Claude Haiku 4.5 | 26.1% | 25.1% | Stable but generalized drafting |
| Qwen 3.5 Flash | 13.1% | 22.8% | Moderate operational instability |
| Gemini 2.5 Flash Lite | 9.2% | 25.7% | Fluent but commercially ambiguous |
Table 7: Model-level risk distribution in B2B context.
| Model | Level 1 | Level 4–5 | Summary pattern |
|---|---|---|---|
| GPT-5.4 Nano | 47.9% | 14.7% | Strongest consumer-oriented robustness |
| Claude Haiku 4.5 | 24.6% | 29.9% | Structurally stable but conservative |
| Qwen 3.5 Flash | 9.0% | 31.3% | Highest consumer-context instability |
| Gemini 2.5 Flash Lite | 5.7% | 32.9% | Fluent but highly ambiguous under consumer protections |
Table 8: Model-level risk distribution in B2C context.
| Model | Level 1 | Level 4–5 | Summary pattern |
|---|---|---|---|
| GPT-5.4 Nano | 55.5% | 9.0% | Strongest public-sector legal robustness |
| Claude Haiku 4.5 | 39.4% | 21.4% | Stable and compliance-oriented drafting |
| Qwen 3.5 Flash | 18.4% | 22.2% | Moderate public-sector instability |
| Gemini 2.5 Flash Lite | 12.9% | 24.8% | Fluent but weak under regulatory complexity |
Table 9: Model-level risk distribution in B2G context.
| Clause category | Dominant failure pattern |
|---|---|
| Limitation of liability | Overbroad exclusions and weak carve-outs |
| Indemnity | Asymmetrical or incomplete liability allocation |
| Intellectual property | Ownership and license-scope ambiguity |
| Outage/service interruption | Weak remedies and SLA specificity |
| Jurisdiction | Cross-border enforceability assumptions |
| Privacy and data protection | Compliance and consent incompleteness |
Table 10: Recurring high-risk clause categories and dominant failure patterns.
| Sub-component | Risk |
|---|---|
| Legal Standards | 3.43 |
| Nuance and Interpretation | 3.35 |
| Nuance Omission | 3.33 |
| Jurisdiction | 3.23 |
| Ambiguity Handling | 3.18 |
| Accuracy Rate | 3.10 |
Table 11: Most fragile CLAUSE sub-components overall.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Clause Category | Primary Weakness | Dominant Failure Pattern | Example Problematic Language |
|---|---|---|---|
| Confidentiality Clauses | Ambiguity and weak procedural safeguards | Missing survival clauses, vague disclosure obligations, and incomplete incident procedures | “reasonable efforts”; “appropriate safeguards” |
| Privacy & Data Protection Clauses | Compliance incompleteness | Missing lawful basis requirements, weak retention obligations, and inadequate cross-border safeguards | “Provider may process user data as necessary.” |
| Security Clauses | Operational vagueness | Undefined security standards, weak breach notification procedures, and missing escalation mechanisms | “industry-standard security measures” |
| Safety Clauses | Lack of operational accountability | Missing audit obligations, escalation procedures, and responsibility allocation | Generalized safety commitments without enforcement structures |
| Intellectual Property Clauses | Ownership ambiguity | Conflicting derivative-work provisions, overlapping rights structures, and licensing inconsistencies | Contradictory jointly developed IP ownership provisions |
| Use of Content to Train Product Clauses | Consent and governance failures | Failure to distinguish analytics usage, model training, and commercialization rights | “Provider may use submitted content to improve services.” |
Table 12: Clause-level comparative analysis of dominant legal-risk patterns across evaluated language models.
| CLAUSE component | Avg. risk | Most fragile sub-components | Dominant failure pattern |
|---|---|---|---|
| Compliance and Jurisdictional Adaptability | 3.04 | Legal Standards (3.43); Jurisdiction (3.23); Regulatory Compliance (3.08) | Cross-jurisdiction enforceability failures, governing-law ambiguity, missing statutory safeguards, vague regulatory language. |
| Legal Competence and Reasoning | 2.66 | Nuance and Interpretation (3.35); Time-Series Structuring (2.83); Legal Principle Application (2.77) | Weak handling of doctrinal nuance, temporal obligations, proportionality, survivability, causation, and effort standards. |
| Structured Contract Elements Handling | 2.59 | Nuance Omission (3.33) | Missing notice obligations, escalation procedures, survival clauses, exception handling, and fallback remedies. |
| Understanding Context | 2.60 | Ambiguity Handling (3.18) | Context-sensitive drafting instability under incomplete, multi-party, or hybrid transactional prompts. |
| Accuracy, Reliability, and Trustworthiness | 2.57 | Accuracy Rate (3.10); Prompt Sensitivity (3.06); Parametric Knowledge Reliance (2.55) | Legally plausible but unreliable drafting, unsupported assumptions, and instability under perturbation. |
| Ethics, Privacy, and Accountability | 2.01 | Accountability Allocation; Data Governance; Privacy Precision | Broad data rights, weak governance assignment, and incomplete privacy safeguards. |
Table 13: CLAUSE component and sub-component risk analysis.
More in this topic
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,408 papers