Agent4Spec: An LLM-based Agentic System to Optimize API Specifications through Automated Validation and Market Analysis

Yongjian Tang; Costel Moraru; Doruk Tuncel; Hanya Elhashemy; Steffen Klepke; Thomas Runkler

cs.SEOct 9, 2026

Agent4Spec: An LLM-based Agentic System to Optimize API Specifications through Automated Validation and Market Analysis

Authors: Yongjian Tang, Costel Moraru, Doruk Tuncel, Hanya Elhashemy, Steffen Klepke, Thomas Runkler

Organizations: Siemens Research (RPD), Munich, Germany · Technical University of Munich, Munich, Germany

Abstract

API specifications serve as critical documentation for modern software development, directly impacting integration efficiency and client code generation. While recent advancements have leveraged Large Language Models (LLMs) to generate initial specifications from source code, existing approaches treat this process as a one-time event - overlooking the needs for continuous validation, long-term maintenance, and market-driven evolution of these artifacts. To address this issue, we present Agent4Spec, an autonomous multi-agent framework featuring two distinct pipelines: (1) a validation pipeline that detects and self-corrects OpenAPI syntax errors, and (2) a market analysis pipeline that conducts competitive research, identifies functional gaps against industry leaders, and automatically generates missing API-spec functionalities. We evaluate Agent4Spec on two curated datasets of API specifications spanning both general-purpose and industrial scenarios. Among 9 SOTA LLMs, the top-performer achieves 100% syntax error detection rates and over 95% implementation success for proposed corrections, while securing over 50% adoption rates for suggested endpoint improvements, representing a practical tool for API development.

Figures & tables

Fig. 1: Agent4Spec consists of four core components: an orchestrator agent $A_{o}$ acting as the central controller and communicator; a validation agent $A_{v}$ to detect and resolve specification syntax errors; multiple sequential agents $A_{d},A_{m},A_{g},A_{e}$ to explore current technology trends and identify gaps against market leaders for improvement; and finally a revision agent $A_{r}$ that automatically updates specifications based on developer-approved suggestions.

Notation	Description / Functional Definition
$S_{0}$	Original OpenAPI specification provided by the user
$S_{f}$	Final revised OpenAPI specification
$R_{v}$	Report of validation errors with proposed correction suggestions
$R_{d}$	Identified API domain and domain summary
$R_{m}$	Synthesized market analysis report
$R_{g}$	Gap analysis report with prioritized suggestions

TABLE I: Notation used in Agent4Spec.

Statistic	Guru-Spec	Industry-Spec
Number of specifications	$300$	$200$
Total dataset tokens	$3\,483\,222$	$1\,684\,932$
Maximum tokens in a single spec	$59\,836$	$36\,145$
Average tokens per spec	$11\,610.7$	$8424.7$
Average lines per spec	$1549.2$	$934.4$

TABLE II: Statistics of the curated datasets.

Company	Model	Size	Context Window	Modalities (input)	Openness
OpenAI	GPT-5.2	–	1M tokens	text/image	closed
	GPT-4.1	–	1M tokens	text/image	closed
	GPT-4o	–	128k tokens	text/image/audio/video	closed
Google	Gemini-3.1-Pro	–	1M tokens	text/image/audio/video	closed
Google	Gemma4-4B	4B	128k tokens	text/image/audio	open
Anthropic	Claude-Sonnet-4.6	–	1M tokens	text/image	closed

TABLE III: Basic information of selected LLMs.

Fig. 2: Evaluation on domain identification using LLMs and human ratings.

Model Class	Evaluated LLM	Guru-Spec Dataset ( $N=300$ APIs)						Industry-Spec Dataset ( $N=200$ APIs)
		Errors detected		Solutions proposed		Solutions correct		Errors detected		Solutions proposed		Solutions correct
		Baseline	Agent4Spec	Baseline	Agent4Spec	Baseline	Agent4Spec	Baseline	Agent4Spec	Baseline	Agent4Spec	Baseline	Agent4Spec
Large-scale Frontier	Claude-Sonnet-4.6	93.7%	100.0%	92.0%	100.0%	88.7%	99.3%	91.5%	100.0%	89.5%	100.0%	85.5%	99.5%
	Gemini-3.1-Pro	92.3%	100.0%	91.0%	100.0%	87.0%	99.0%	90.0%	100.0%	88.0%	100.0%	84.0%	99.0%
	GPT-5.2	90.7%	100.0%	89.3%	100.0%	84.7%	98.3%	87.5%	99.0%	85.5%	99.0%	81.0%	98.0%
	GPT-4.1	88.0%	99.7%	86.7%	99.7%	81.3%	97.7%	85.0%	99.0%	83.0%	99.0%	78.5%	97.5%

TABLE IV: Comparing the single-prompt baseline with Agent4Spec to detect syntax errors.

Model Class	Evaluated LLM	Guru-Spec Dataset ( $N=300$ APIs)				Industry-Spec Dataset ( $N=200$ APIs)
		Solutions correct		Solutions saved		Solutions correct		Solutions saved
		Baseline	Agent4Spec	Baseline	Agent4Spec	Baseline	Agent4Spec	Baseline	Agent4Spec
Large-scale Frontier	Claude-Sonnet-4.6	88.0%	96.3%	94.7%	100.0%	85.5%	97.5%	92.5%	100.0%
	Gemini-3.1-Pro	86.7%	98.3%	94.0%	100.0%	84.0%	98.5%	92.0%	100.0%
	GPT-5.2	82.3%	94.3%	92.0%	100.0%	78.5%	90.5%	89.0%	100.0%
	GPT-4.1	80.7%	94.0%	90.3%	100.0%	76.0%	89.5%	87.5%	100.0%

TABLE V: Comparing the single-prompt baseline with Agent4Spec on the first attempt to revise the entire API specifications.

Fig. 3: Evaluation on online search.

Fig. 4: Evaluation on gap analysis.

Fig. 5: Evaluation on endpoint suggestion.

Fig. 6: Endpoint suggestions approved by humans.

Fig. 7: Agent4Spec token consumption and inference latency per API specification across different LLMs.

Software Engineering Agents

Software Engineering Agents are AI systems that use code, tools, and feedback to perform software development tasks.

See all 349 papers

Doc2Spec: Synthesizing Formal Programming Specifications from Natural Language via Grammar Induction
Jan 30, 2026Shihao Xia, Mengting He, Haomin Jia +2Autoformalization Program Synthesis
The Pennsylvania State University · University of the Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences
Mitigating Errors in LLM-Generated Web API Invocations via Retrieval-Augmented Generation and Constrained Decoding
Jul 7, 2026Daniel Maninger, Leon Chemnitz, Jannis Brugger +3Retrieval-Augmented Generation Constrained Decoding
Technische Universität Darmstadt, Darmstadt, Germany · Hessian Center for Artificial Intelligence (hessian.AI), Darmstadt, Germany · Pariton AI, Berlin, Germany +1
AutoReSpec: A Framework for Generating Specification using Large Language Models
Apr 4, 2026Ragib Shahariar Ayon, Shibbir AhmedAutoformalization Formal Verification
Texas State University, San Marcos, TX, USA

Jan 30, 2026J/K move · Enter open · S save

Doc2Spec: Synthesizing Formal Programming Specifications from Natural Language via Grammar Induction

Shihao Xia, Mengting He, Haomin Jia, Xinyan Zhao+1

The Pennsylvania State University · University of the Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences

Autoformalization Program Synthesis Formal Verification LLM-Based Program Synthesis

Open paper AskPDF arXiv

Ensuring that API implementations and usage comply with natural language programming rules is critical for software correctness, security, and reliability. Formal verification can provide strong guarantees but requires precise specifications, which are difficult and costly to write manually. To address this challenge, we present Doc2Spec, a multi-agent framework that automatically induces a domain-specific grammar from natural-language API rules and uses it to guide specification generation. Doc2Spec fixes a domain-agnostic logical skeleton as a grammar template, prompts LLMs to infer domain-specific predicates and sorts, and formalizes each rule within the resulting grammar, turning an unreliable one-shot translation into a sequence of constrained, checkable steps. Across six benchmarks spanning Solidity and Rust, Doc2Spec improves precision by 0.28 and recall by 0.37 over baselines that lack grammar induction or perform it in one unstaged step, demonstrating the benefits of grammar-guided formalization and the staged pipeline. Moreover, its formalized rules enable symbolic-execution tools to uncover 142 previously unknown rule violations, confirming the rules' correctness and practical usefulness.

Abstract

Software Engineering Agents

Explore similar work

Doc2Spec: Synthesizing Formal Programming Specifications from Natural Language via Grammar Induction

Mitigating Errors in LLM-Generated Web API Invocations via Retrieval-Augmented Generation and Constrained Decoding

AutoReSpec: A Framework for Generating Specification using Large Language Models