cs.CLApr 5, 2026
SaveAdaptive Cost-Efficient Evaluation for Reliable Patent Claim Generation
Organizations: Frontier AI Research Centre, Macquarie University · School of Computing, FSE, Macquarie University
Abstract
Automated patent claim validation demands low error tolerance. However, existing approaches face a rigidity-resource dilemma: lightweight encoders cannot track long-range legal dependencies, while exhaustive LLM verification incurs 4-5X higher overhead at million-claim scale. A naive confidence-based cascade cannot resolve this because binary validity scores fail to distinguish structurally distinct error types which require different reasoning depths. We propose a two-stage framework: Adaptive Cost-efficient Evaluation (ACE), which exploits the categorical structure of patent errors for uncertainty-aware routing. In the first stage, a fine-tuned encoder projects claims into a K+1 distribution over legal error types, whose predictive entropy serves as the routing signal. Claims exceeding an entropy threshold are escalated to the second stage, where an expert LLM executes a schema-constrained Chain-of-Patent-Thought (CoPT) protocol to map claim elements against 35 U.S.C. standards whose schema constraint reduces per-claim latency by 42% while producing legally grounded verdicts. We further present a 40,000-claim dataset ACE-40k with MPEP-grounded annotations, where ACE surpasses competitive baselines including a supervised 70B-parameter LLM while reducing costs by 78%. On real USPTO rejection data, the routing mechanism transfers without re-calibration, reducing inference time by 60% while maintaining competitive recall.
Figures & tables
Figure 1: High-stakes patent claim validation requires low error tolerance while demanding cost efficiency. Adaptive Cost-efficient Evaluation (ACE) leverages the throughput of encoder models and the inference accuracy of Chain-of-Patent-Thought (CoPT) reasoning.
| Category | Description | Count |
| Pass | Published patent | 20,000 |
| Fail: Antecedent | Lack of antecedent basis | 4,000 |
| Fail: Dependency | Invalid cross-reference | 4,000 |
| Fail: Logical | Impossible causality or circularity | 4,000 |
| Fail: Ambiguity | Indefinite terminology | 4,000 |
| Fail: Syntax | Structural format violations | 4,000 |
Table 1: Statistics of the constructed ACE dataset. We enforce a strictly balanced distribution of 1:1 between valid and invalid samples.
Figure 2: The ACE framework. An uncertainty-aware Gatekeeper routes patent claims based on predictive entropy: confident predictions ( ) are resolved via the Fast Path, while ambiguous claims ( ) are escalated to an instruction-tuned Expert LLM that performs Chain-of-Patent-Thought (CoPT) reasoning for statutory verification.
| Model / Method | Acc. (%) | F1 (%) | AUC (%) | Latency (s) | Est. Cost ($)/M |
| Sanity Check & General Baselines | |||||
| Vanilla BERT | 49.60 | 66.25 | 50.93 | 0.12 | $100 |
| Vanilla Longformer | 75.52 | 73.11 | 47.26 | 0.42 | $350 |
| Frozen BERT + Logistic Regression | 69.27 | 68.12 | 66.20 | 0.08 | $67 |
| TF-IDF + Logistic Regression | 73.38 | 72.23 | 83.00 | 0.01 | $1 |
| Domain-Specific Experts | |||||
Table 2: Main Performance Results. Domain-specific baselines use pre-trained representations without task-specific fine-tuning. ACE Hybrid processes 1M claims at 5,733). Cost estimated at $3.00/hr for NVIDIA H100. AUC is reported only for single-model classifier.
Figure 3: ROC curves for claim validity discrimination. The Gatekeeper (AUC=0.9716) substantially outperforms baselines.
| Model | Recall (%) | Time (s) |
| Llama-3-8B (zero-shot) | 13.2 | 103.1 |
| Llama-3-70B (zero-shot) | 8.3 | 355.3 |
| PatentBERT Standalone | 56.4 | 7.2 |
| Llama-3-70B + SFT + CoPT | 65.2 | 6,820.9 |
| ACE Hybrid (Ours) | 66.2 | 2,674.8 |
Table 3: Results on ACE-Real112b (204 real USPTO §112(b) rejections). All samples are genuine Office Action rejections; recall is reported as the transfer metric. No parameters were re-tuned on this set.
Figure 4: Fine-grained Gatekeeper performance across error types. Antecedent Basis records the lowest recall, confirming the long-range tracking bottleneck.
| Failure Pattern | Count | % |
| Cross-claim dependency / Nested antecedent | 38 | 55.1 |
| Claim-set semantic scope ambiguity | 17 | 24.6 |
| Examiner-specific subjective judgment | 14 | 20.3 |
Table 4: Breakdown of 69 missed cases on ACE-Real112b.
Figure 5: Risk-Coverage Trade-off. Retained F1 and operational cost as a function of escalation rate.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Label | Claim Text | Error Type |
| Pass | A semiconductor device comprising a substrate, a gate structure formed over the substrate, and a spacer adjacent to gate structure. | None |
| Fail | A semiconductor device comprising the gate structure formed over the substrate, wherein the substrate is configured to receive the gate structure of claim 1. | Antecedent |
| Fail | The system of claim 1, wherein the processor of claim 3 further executes the module of claim 7. | Dependency |
| Fail | A battery including a charging unit that generates electrical energy from the discharged battery cell. | Logical |
Table 5: Representative samples from the ACE dataset. Invalid claims contain single targeted errors injected based on MPEP guidelines.
| Parameter | Value | Rationale |
| Base Model | PatentBERT-base | Domain-Adapted Efficiency |
| Max Sequence Length | 512 | Standard BERT Limit |
| Batch Size | 32 | Stable Gradient Estimation |
| Learning Rate | Optimal for Fine-tuning | |
| Epochs | 5 | Sufficient Convergence |
| Weight Decay | L2 Regularization |
Table 6: Hyperparameters for Gatekeeper Training
| Parameter | Value | Rationale |
| Base Model | Meta-Llama-3-70B-Instruct | Instruction-Following Capability |
| Quantization | 4-bit (NF4) | Memory Efficiency (QLoRA) |
| LoRA Rank ( ) | 16 | Parameter Efficiency |
| LoRA Alpha ( ) | 32 | Scaling Factor ( ) |
| LoRA Dropout | 0.05 | Regularization |
| Learning Rate | Optimal for QLoRA Fine-tuning |
Table 7: Hyperparameters for Llama-3 Evaluator Training (QLoRA)
| Error Type | Recall | F1-Score | Nature |
| Semantic Reasoning Tasks | |||
| Ambiguity | 90.75% | 0.8985 | Semantic Interpretation |
| Logical | 86.45% | 0.8645 | Inferential Reasoning |
| Structural Verification Tasks | |||
| Dependency | 92.37% | 0.9130 | Claim Hierarchy |
| Antecedent | 82.77% | 0.8141 | Long-Range Reference |
Table 8: Fine-grained Performance of the Gatekeeper (Test Set). The model struggles with structural errors (Antecedent, Syntax), highlighting the specific areas where the LLM Evaluator is most needed.
| Role | Content |
| System | You are a Senior Patent Attorney. |
| Your task is to evaluate the validity of a patent claim based on specific error types. | |
| If ANY error exists, output {"verdict": "Fail"} . If NO error, output {"verdict": "Pass"} . | |
| User | Claim: { claim_text } |
| Assistant | (Model Generation Target) |
Table 9: The Standard SFT prompt template. This strictly follows the format used during the supervised fine-tuning stage to prevent distribution shift during inference.
| Role | Content |
| System | You are a Senior Patent Attorney. |
| Your task is to evaluate the validity of a patent claim based on specific error types. | |
| [Special Instruction]: To ensure strict adherence to 35 U.S.C. § 112, you must explicitly document your reasoning process using the Chain of Patent Thought (CoPT) protocol before concluding. | |
| 1. [Element Parsing] : Identify preamble, transition, and body elements. |
Table 10: The full prompt template used for the Stage 2 Specialist LLM (SFT + CoPT). The [Special Instruction] tag and 2-shot examples are designed to override the model’s training bias towards short responses, enforcing a detailed legal analysis path.
| Component | Content |
| Input Claim (Excerpt) | 1. (canceled) 2. The workpiece made according to the method of claim 8, further comprising an intermediate layer containing TiAlN provided between the TixSi1-xN layer and the substrate body of the workpiece. (...) 5. The workpiece made according to the method of claim 8, further comprising an intermediate layer containing CryAl1-yN provided between the TixSi1-xN layer and the substrate body of the workpiece (...) (...) 8. A method for coating a workpiece having a coating comprising at least one TixSi1-xN layer, wherein x 0.85 (...), the method comprising: using a sputtering process to produce the TixSi1-xN layer (...) |
| CoPT Output | {"reasoning": "Step 1: Parsed claim elements. Step 2: Antecedent Basis error: Claim 2 refers to claim 8, but claim 8 is not yet defined. Claim 5 refers to claim 8, but claim 8 is not yet defined. Step 3: No Definiteness errors. Step 4: Fail due to Antecedent Basis errors.", "verdict": "Fail"} |
| Ground Truth | Fail (§112(b) rejection in USPTO Office Action) |
Table 11: Example of CoPT protocol output on a real USPTO §112(b) case. The model identifies forward-reference dependencies (claims 2 and 5 referencing the later-defined claim 8) through structured step-by-step reasoning, providing actionable grounds for the rejection verdict.
| Configuration | F1 (%) | Cost ($/M) |
| Llama-3-8B SFT (Standalone) | 83.5 | $667 |
| ACE Hybrid w/ 8B Expert | 89.3 | $233 |
| ACE Gatekeeper (Stage 1 Only) | 91.58 | $100 |
| ACE Hybrid w/ 70B Expert | 94.95 | $1,247 |
Table 12: Effect of expert model scale on ACE Hybrid performance (ACE-40k test set). The 8B expert degrades below the Gatekeeper alone, confirming that 70B-class reasoning is necessary for the escalated high-uncertainty subset.
| Escal Ratio | Stage-2 Samples | Recall | Cost ($/M) |
| 10% | 20 | 60.8 | $673 |
| 20% | 40 | 65.7 | $1,247 |
| 30% | 61 | 72.1 | $1,820 |
| 40% | 81 | 78.9 | $2,393 |
| 50% | 101 | 86.3 | $2,967 |
Table 13: Threshold sensitivity on ACE-Real112b. The recall-cost trade-off is smooth and monotonic across escalation ratios.
More in this topic
Claim Verification
Claim verification assesses whether statements are supported, contradicted, or unresolved by relevant evidence.
See all 59 papers