Benchmarking Patent Drafting from Inventor-Style Disclosures

Lekang Jiang; Wenjun Sun; Stephan Goetz

cs.CLAug 21, 2026

Benchmarking Patent Drafting from Inventor-Style Disclosures

Authors: Lekang Jiang, Wenjun Sun, Stephan Goetz

Organizations: University of Cambridge · National Science Library, Chinese Academy of Sciences · Department of Information Resources Management, School of Economics and Management, University of Chinese Academy of Sciences

Abstract

While recent large language models (LLMs) have achieved promising results on individual patent drafting tasks, they fundamentally fail to investigate the core challenge of real-world patent drafting: generating a complete and legally coherent patent application directly from early-stage invention materials. Prior work predominantly assumes later-stage, highly structured, or already legalistic inputs. However, real patenting workflows begin with informal, de-legalized disclosures authored by inventors. To bridge the gap, we introduce Dis2Pat, a disclosure-to-patent dataset that reflects realistic patenting workflows by requiring the generation of complete patent applications directly from inventor-style, de-legalized disclosures. Given the inherent difficulty of long-form, legally constrained patent drafting and the strong privacy requirements, we further propose a strong baseline named Patent-MAF. It is a multi-agent framework for locally deployable patent drafting. Benchmark results reveal that current LLMs exhibit limitations in patent drafting, while Patent-MAF provides a strong baseline that consistently outperforms evaluated open-source models and remains competitive with large closed-source models.

Figures & tables

Dataset / Work	Size	Task	Input	Output
HUPD-DCG ( Jiang et al., 2025d )	$\sim$ 10K	Claim generation	Specification	Claims
EPD ( Jiang et al., 2025a )	$\sim$ 107K	Claim generation	Specification	Claims
Patent-CR ( Jiang et al., 2025b )	$\sim$ 20K	Claim revision	Rejected application claims	Granted claims
PatentFormer ( Wang et al., 2024a )	$\sim$ 13K	Specification generation	Claims + figure descriptions	Specification
PatentVision Yang et al. (2026)	$\sim$ 230K	Specification generation	Claims + figure	Specification
AutoSpec Shea and Yu (2025)	$\sim$ 1.3K	Specification generation	Claims	Specification

Table 1: Comparison of related datasets and works for LLM-based patent drafting.

Metric	Original Patent	Disclosure
Corpus Statistics
# Tokens	11207.4	1196.2
# Title Tokens	8.7	–
# Abstract Tokens	128.1	–
# Claim Tokens	1594.1	–
# Specification Tokens	9476.5	–

Table 2: Statistics and quality of the Dis2Pat dataset.

Figure 1: Overview of the Patent-MAF method. The manager agent structures the disclosure and coordinates specialized drafter agents, followed by a polisher agent to refine the final patent text.

Model / Setting	Text Overlap			Semantic Similarity		Content Quality
Model / Setting	BLEU	R-1	R-L	BERTScore	BERT-for-Patent	Completeness	Clarity	Style	Overall
Text-Only
Qwen3-32B	0.8	28.8	12.1	81.5	94.4	82.8	81.0	72.9	78.9
Qwen3-VL-32B	0.7	30.3	12.3	82.0	94.6	83.3	85.9	73.7	81.0
Qwen3-VL-72B	1.6	32.2	12.9	81.6	94.5	84.2	83.5	75.7	81.1
Llama-3.3-70B	0.3	26.1	11.3	81.6	94.6	72.0	74.7	59.6	68.8

Table 3: Model performance on patent specification. The highest score in each column is marked in bold .

Model / Setting	Text Overlap			Semantic Similarity		Content Quality
Model / Setting	BLEU	R-1	R-L	BERTScore	BERT-for-Patent	Scope	Clarity	Legality	Overall
Text-Only
Qwen3-32B	13.5	48.0	25.3	83.2	94.5	82.8	81.2	78.6	80.9
Qwen3-32B-FT	19.1	54.3	30.4	87.3	96.1	83.6	85.2	81.2	84.4
Qwen3-VL-32B	10.8	48.6	24.1	82.3	93.9	82.7	79.5	74.6	78.9
Qwen3-VL-72B	13.5	51.3	26.3	84.3	94.5	83.5	81.1	77.4	80.7

Table 4: Model performance on patent claims. The highest score in each column is marked in bold .

Figure 2: Pairwise human evaluation results that compare Patent-MAF against four strong baselines.

Task: Pseudo-Disclosure Generation

You will be given a patent title, claims, and full specification. Your task is to extract and rewrite the invention into a clear, human-readable disclosure using plain, non-legal English. Do not copy legal phrasing or claim-style language, and do not introduce any new technical information beyond the original patent.

Output Structure:

You must generate the disclosure using the following seven invention-centric sections:

1. Title of the Invention (Layperson Version): Summarize the invention in one simple sentence. If the original title is technical or obscure, rewrite it into an easy-to-understand name.

2. Why This Invention Is Needed (Problems in the Prior Art): Describe the limitations of existing technologies and explain why these problems matter, using simple and intuitive language.

Table 5: Prompt used to construct pseudo-disclosures for the Dis2Pat dataset.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Text Overlap			Semantic Similarity		Content Quality
Model	BLEU	R-1	R-L	BERTScore	BERT-for-Patent	Completeness	Clarity	Style	Overall
Baselines (Text-Only)
Llama-3.2-90B-Vision	0.5	27.3	11.8	81.5	94.5	68.7	73.6	58.1	66.8
Baselines (Text+Image)
Llama-3.2-90B-Vision	0.6	27.7	12.1	81.8	94.6	68.9	73.9	58.6	67.1
Ours (Based on Llama-3.2-90B-Vision)

Appendix

Table 6: Performance test on patent specification based on Llama-3.2-90B-Vision.

Instructions:

You will receive two patent specifications: (1) gold_spec : the gold-standard reference specification, and (2) candidate_spec : the specification to be evaluated. Your task is to evaluate how well the candidate specification matches the gold specification in terms of content coverage, logical clarity, and legal drafting quality.

You must assign a score from 0 to 100 for each evaluation dimension, where: 0 indicates extremely poor quality or entirely missing content, and 100 indicates excellent quality fully aligned with professional patent drafting standards.

Evaluation Criteria:

1. Completeness (0–100)

Appendix

Table 7: LLM-based evaluation prompt used for patent specification quality assessment.

Instructions:

You will receive two patent claim sets: (1) Gold Claims : the gold-standard reference claims, and (2) Candidate Claims : the claims to be evaluated.

Role:

You are a Senior Patent Examiner with experience in patent examination.

Objective:

Appendix

Table 8: LLM-based evaluation prompt used for patent claim quality assessment.

Human Evaluation Task: Overall Patent Quality Ranking

Evaluation Objective:

You are asked to manually evaluate and compare multiple patent texts generated by different models. Each patent consists of two parts: a Specification and a set of Claims . An officially published or granted patent ( Gold Patent ) is provided as the upper-bound quality reference. Your goal is to compare two candidate patents against the Gold Patent and rank them by quality.

Input Materials:

Each evaluation sample includes:

Appendix

Table 9: Expert human evaluation instructions for overall patent quality ranking.

Language Model Generation Evaluation

Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.

See all 395 papers

PatentAgentBench: Benchmarking LLM Agents with Professional Patent Task Suites
Oct 26, 2026Yijie Hu, Wei Li, Junjie Zhou +6LLM Agent Evaluation AI Agent Benchmarks
AI Eval & Alignment, PatSnap Ltd. Co.
LogicTree-RAG: Logic Tree-guided Retrieval-Augmented Generation for Long-form Patent Drafting
Sep 25, 2026Jiaqi Zhu, Naili Xing, Hexiang Pan +4Long-Form Document Generation Text Generation
National University of Singapore · Zhejiang University
Vibe Patenting: Evaluating LLM Judges for Professional Patent-Drafting Agents
Sep 11, 2026Toshiaki Koike-Akino, Vladislav Blaykhman, Ye Wang +2LLM Self-Refinement LLM-as-a-Judge
Mitsubishi Electric Research Laboratories (MERL) 202 Broadway, Cambrdige, MA 02139

Oct 26, 2026J/K move · Enter open · S save

PatentAgentBench: Benchmarking LLM Agents with Professional Patent Task Suites

Yijie Hu, Wei Li, Junjie Zhou, Zhiqi Jiang+5

AI Eval & Alignment, PatSnap Ltd. Co.

LLM Agent Evaluation AI Agent Benchmarks Information Retrieval

Open paper AskPDF arXiv

Patent practice lies at the intersection of technical, legal, and linguistic work, making it challenging yet crucial to technological innovation. Recently, rapid advances in large language models (LLMs) and agentic systems have demonstrated the potential to complete realistic business workflows and even patent works. This raises a question: can current AI systems perform patent work reliably? Existing benchmarks cover a limited range of patent tasks and simplify evidence acquisition by providing reference materials or candidate documents. In this paper, we introduce PatentAgentBench, a benchmark comprising 510 instances across eight task suites: novelty search, freedom-to-operate (FTO) search, design FTO, invalidity search, claim charting, patent drafting, office-action response, and patent translation. Curated from real patent records and professional materials, the benchmark combines task-specific metrics with expert-reviewed annotations and rubrics. Retrieval tasks such novelty search, FTO, etc., require agents to search and compare patent collections without a supplied candidate pool. The drafting and OAR tasks assess the quality of agents' work given technical disclosure and office action notice. We evaluate seven LLMs across four agent harnesses on PatentAgentBench, finding that current systems still struggle to identify novelty, assess risk, and understand practitioner intent throughout patent workflows.

Abstract

Language Model Generation Evaluation

Explore similar work

PatentAgentBench: Benchmarking LLM Agents with Professional Patent Task Suites

LogicTree-RAG: Logic Tree-guided Retrieval-Augmented Generation for Long-form Patent Drafting

Vibe Patenting: Evaluating LLM Judges for Professional Patent-Drafting Agents