Multi-Step Tool-Calling over Korean Open Public APIs: A Benchmark and a Data-Synthesis Recipe

Dain Kim; Eungi Cho; Kyumin Kim; Shinyeong Noh; Kyuseong Lim

cs.AISep 4, 2026

Multi-Step Tool-Calling over Korean Open Public APIs: A Benchmark and a Data-Synthesis Recipe

Authors: Dain Kim, Eungi Cho, Kyumin Kim, Shinyeong Noh, Kyuseong Lim

Organizations: LG CNS

Abstract

Data-sovereignty regulations increasingly require public institutions to deploy open-source, on-premise LLM agents that chain multiple tool-calls across live government APIs. However, open-source models consistently underperform in this multi-step setting, and no existing benchmark measures the gap. We introduce the Korean Open Public API Benchmark (KOPA-Bench), comprising 145 real-world tasks. To close this gap, we present EDGE, an Execution-grounded Dynamic Graph for tool-calling data synthEsis driven by live execution. EDGE builds a graph of how each tool's output can feed another's input, keeps only the links that succeed when actually called against the live APIs, and traverses these verified links to synthesize executable multi-step trajectories. Fine-tuned via GRPO on the resulting dataset, our 9B model nearly matches the untuned 27B model from the same family, improving substantially not only on KOPA-Bench but also on the BFCL benchmark.

Figures & tables

Figure 1: Comparison of tool-calling trajectories in Korean public APIs before and after fine-tuning. (a) Failure case (Before fine-tuning): The agent fails to retrieve the API key and, even after successfully retrieving a valid key, it answers incompletely based only on the first page of a paginated API response. (b) Success case (After fine-tuning): The agent successfully chains tool outputs, queries multiple pages, and aggregates the results to deliver an accurate answer.

Figure 2: Overview of EDGE. (a) Phase A constructs an execution-grounded dynamic graph: from a skeleton of LLM-proposed candidate edges, it revises both edge posteriors and its topology using live execution evidence, yielding a refined graph $G^{\star}$ . (b) Phase B assembles trajectories over $G^{\star}$ by junction type and generates a natural-language query for each, producing the training dataset.

	KOPA-Bench			BFCL
Model	pass@1	pass@4	Action	Multi- Turn	Single (non-live)	Single (live)
Proprietary
Claude Sonnet 4.6 ( Anthropic, 2026 )	0.4655	0.8207	0.4269	60.13	84.98	76.83
GPT-5.1 ( OpenAI, 2025 )	0.3706	0.4690	0.5881	38.12	82.96	65.21
Large open-source
Qwen3.5-27B	0.4482	0.5655	0.4213	65.38	89.65	83.35

Table 1: Results on KOPA-Bench and BFCL. KOPA-Bench — pass@

k

: fraction of the

145

tasks solved by at least one of

k

rollouts, so pass@1 is the fraction solved on a single attempt (averaged over the four rollouts) and pass@4 the fraction solved by at least one of the four; Action : tool-call accuracy over golden actions. BFCL scores are computed via the official evaluation repository. The bottom block reports our fine-tuned models.

Training objective	pass@1	pass@4	Action
Qwen3.5-4B (base)	0.1758	0.3103	0.2140
SFT (Ours)	0.2724	0.3586	0.3048
GRPO (Ours)	0.3094	0.4690	0.3462

Table 2: Training objective ablation, evaluated on KOPA-Bench with four rollouts per task; metrics are defined as in Table 1 . SFT and GRPO consume the identical

1{,}781

EDGE tasks and share the same base checkpoint.

Training dataset	pass@1	pass@4	Action
Qwen3.5-4B (base)	0.1758	0.3103	0.2140
pure-sequential	0.2327	0.3724	0.3648
parallel + Mixed	0.3080	0.4000	0.4070
full	0.3094	0.4690	0.3462

Table 3: Trajectory composition ablation, evaluated on KOPA-Bench with four rollouts per task; metrics are defined as in Table 1 . Mixed trajectories are retained in both parallel + Mixed and full ; pure-sequential contains only purely sequential trajectories. full uses the complete filtered dataset and corresponds to our final model, Qwen3.5-4B (Ours).

Figure 3: Effect of trajectory-level filtering on the dataset. Filtering improves every metric.

Figure 4: Execution-grounded refinement improves the graph and prunes the right edges. Edges are grouped as the pre-refinement Skeleton, the surviving graph $G^{\star}$ , and the Pruned set; light and dark bars count all tested edges ( $t_{e}\!\geq\!1$ ) and high-evidence edges ( $t_{e}\geq 10$ ); the $t_{e}\geq 10$ bars are not comparable across groups, since failing edges are pruned before accumulating trials.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: A representative task from the Finance domain of KOPA-Bench . Each task pairs a natural-language query with system state injected at runtime. The model calls the provided tools to solve the query and output a final answer, which is evaluated against the annotated ground truth (golden actions, expected response, and resulting environment state).

Model	pass@1	CI width
Qwen3.5-27B	0.4681 $\pm$ 0.0146	0.0292
gemma-4-26B-A4B-it	0.3672 $\pm$ 0.0151	0.0301
Qwen3.5-9B	0.3526 $\pm$ 0.0249	0.0498
EXAONE-4.5-33B	0.2224 $\pm$ 0.0122	0.0244
gemma-4-E4B-it	0.1845 $\pm$ 0.0110	0.0220
Qwen3.5-4B	0.1603 $\pm$ 0.0204	0.0407

Appendix

Table 4: Pass@1 on KOPA-Bench , reported as mean

\pm

95% confidence-interval half-width, computed over eight independent random seeds using the Student’s

t

-distribution (

\mathrm{df}=7

); the last column gives the full CI width. These eight-seed estimates complement the four-run averages reported in Table 1 .

Domain	Platforms	#Tasks
Traffic	Korea Expressway Corporation, Seoul Open Data Plaza	22
Finance	DART, KRX, Korea Deposit Insurance Corporation	29
Education	Educational Data Platform, HRD Korea	30
Law	Korean Law Info Center	22
Politics	Open Assembly	25
District Admin.	Gyeonggi Data Dream	17

Appendix

Table 5: Selected platforms grouped by domain.

Domain	#Tasks	Steps	Tool calls	Headroom	Avail. tools	Parallel %
Traffic	22	3.64	5.14	2.21	7.5	55%
Finance	29	3.69	5.65	2.24	6.6	83%
Education	30	2.93	4.30	2.79	7.7	57%
Law	22	3.00	3.55	3.63	6.7	27%
Politics	25	3.72	5.48	2.10	8.7	52%
District Administration	17	2.76	5.41	1.55	8.0	82%

Appendix

Table 6: Per-domain benchmark statistics. Means are weighted by the number of tasks per domain.

Outcome	Group	Update
Semantic success	—	$\alpha_{e}\mathrel{+}=1.0$
Generic-only success	—	$\alpha_{e}\mathrel{+}=0.3$
Schema/type mismatch	structural	$\beta_{e}\mathrel{+}=1.5$
Missing-field error	structural	$\beta_{e}\mathrel{+}=1.0$
Rate limit	environmental	$\beta_{e}\mathrel{+}=0.1$
Timeout	environmental	$\beta_{e}\mathrel{+}=0.3$

Appendix

Table 7: Outcome taxonomy and the corresponding Beta-posterior update. Successes increment

\alpha_{e}

, more for a semantic success (

1.0

) than for a generic-only success (

0.3

); structural failures increment

\beta_{e}

1.0

–

1.5

, environmental failures by only

0.1

–

0.8

. A success is semantic if at least one non-generic input parameter (i.e., not an API key, pagination, or format field) carried a value, and generic-only otherwise.

Figure 6: Execution success rises as unreliable edges are pruned. Trajectory pass rate and step success rate of sampled paths over the $100$ Phase A iterations; faint lines are per-iteration values and bold lines a moving average. As the loop prunes structurally failing edges and samples toward reliable ones, both rise steadily, indicating that refinement concentrates the graph on executable dependencies.

Figure 7: Pruning is governed by execution, not by the LLM prior. Each point is an admitted edge, plotting its feasibility prior $s_{e}$ against the converged posterior mean $\bar{\theta}_{e}$ . Retained and pruned edges separate along the posterior axis near $\tau=0.5$ rather than along the prior axis, so an edge can carry a high prior yet be pruned once execution contradicts it.

Stage	Filter Class	Representative Anomaly Example (User Query / Trajectory Flaw)
Stage 1	Real-time source	Trajectory components invoke APIs that fetch live highway gating status without a fixed reference timestamp, causing the ground-truth targets in the training dataset to dynamically change per execution (violating data determinism). Example: “실시간 영업소 진입조절 현황에 뜬 각 노선별로 휴게시설 경로 정보가 몇 건씩 있는지 확인해 줄 수 있어?” ( Gloss: “Can you check how many rest area route information items are available for each route listed in the real-time tollbooth entry control status?”)
Stage 1	Linguistic anomaly	Query contaminated with foreign tokens or mixed scripts during synthesis, violating standard format constraints. Example: “전월 대비总人口 증감 수치를 알려줘." or "먼저 확인한 뒤,そこに 포함된 각 영업소별로”. ( Gloss: “Tell me the change in 总人口 compared to the previous month." / "After checking first, そこに for each tollbooth included there.”)
Stage 2	Answer leakage	The query phrasing explicitly discloses the target intermediate parameter, rendering the associated tool call redundant. Example: “2023년 김포시에 있는 야구 가능한 공공체육시설 주소에서 시군명을 뽑아서, 그 시군별로 경기도 내 골프장이 몇 건씩 있는지 알려줄래?” ( Gloss: “Extract the municipality name from the address of baseball-available public sports facilities in Gimpo City in 2023, and tell me how many golf courses there are in Gyeonggi-do for that municipality?”)

Appendix

Table 8: Taxonomy and representative examples of filtered data anomalies, categorized by filtering stage, filter class, and associated query/trajectory flaws.

Symbol	Description	Value
Phase A: skeleton graph construction
$K$	Intra-domain candidate neighbors / source tool	$15$
$K_{\mathrm{x}}$	Cross-domain candidate neighbors / source tool	$10$
$s_{\min}$	Min. feasibility score to admit an edge	$0.3$
$\kappa$	Prior strength	$2.0$
	Skeleton-construction model	Qwen3.5-122B

Appendix

Table 9: EDGE hyperparameters and the values used in all reported runs.

Metric	Value
Calls per task (mean / med. / min / max)	4.13 / 4 / 2 / 36
Steps: $\leq$ 3 / 4–5 / $\geq$ 6	42.3 / 50.9 / 6.7 %
Calls: sequential / parallel	63.8 / 36.2 %

Appendix

Table 10: Training-data statistics over the

1{,}781

synthesized tasks: the distribution of calls per task, the share of tasks by step count, and the split of calls into sequential and parallel.

Dataset	Environment	one-to-many chaining (%)	chained cardinality
Dataset	Environment	one-to-many chaining (%)	median	p90	max
ToolBench-v1	RapidAPI	1.9	1	16	127
APIGen-MT	$\tau$ -bench	10.0	1	3	16
Nemotron ( NVIDIA, 2025 )	$\tau^{2}$ -bench	1.6	1	1	15
ToolACE	synthetic	4.5	1	5	10
Ours	KOPA-Bench	81.2	27	157	224,958

Appendix

Table 11: Comparison of chaining-related statistics across datasets. one-to-many chaining is the share of multi-step trajectories that contain at least one junction consuming a multi-record tool output. chained cardinality reports the distribution of records consumed at such junctions.

Type	Count	%
Mixed	654	36.7
Drv	350	19.7
Sem	303	17.0
Seq	215	12.1
Cmp	161	9.0
Cond	50	2.8

Appendix

Table 12: Distribution of task types in training set

Level	Value
Verbatim query leakage	0.0% (0/145)
Tool-universe overlap (functions)	62.2% (135/217)
Class-level held-out platforms	3/12
Tasks using only unseen functions	12.4% (18/145)
Mean per-task function coverage	62.9%

Appendix

Table 13: Train–eval contamination audit between KOPA-Bench (145 tasks) and the EDGE training corpus (1,781 tasks).

Domain	Held-out platform	$n$	Base	Ours
Traffic	Seoul Open Data Plaza	10	0.2000	0.6000
Finance	DART•KRX	21	0.3333	0.4762
Total		31	0.2903	0.5161

Appendix

Table 14: Performance on the 31 KOPA-Bench tasks grounded in platforms held out from synthesis, for Qwen3.5-4B before and after training on the EDGE corpus.

n

is the number of tasks per platform; values are pass@4 as defined in Table 1 .

Resolution	Eval	Train	Jaccard	Unseen
Function-pair	178	687	0.000	$100\%$
Adjacency	592	2,911	0.004	$95.6\%$

Appendix

Table 15: Overlap of edges between calls in KOPA-Bench and the EDGE training corpus. Eval and Train are edge occurrence counts without deduplication, while Jaccard is computed over the corresponding sets of unique edges. Function-pair is the dependency relation; adjacency links any two consecutive calls and serves as an upper bound. API key acquisition edges are excluded.

Tool-Augmented Language Model Agents

Tool-augmented language model agents use external tools, APIs, and environments to solve tasks beyond text generation alone.

See all 167 papers

E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios
Jul 26, 2026Weihuang Zheng, Tianyuan Zou, Eileen Ye +5Computer-Use Agent Benchmarks Tool-Use Evaluation
Hunyuan Team, Tencent · Institute for AI Industry Research, Tsinghua University · School of Computer Science and Engineering, Southeast University
Data Turnstile: A Scalable Open Framework for Function-Calling Data Generation
Jul 31, 2026Goutham Ramakrishnan, Megha SharmaTool-Augmented Language Model Agents Small Language Models
Amazon AGI Bengaluru, India
HyperTool: Beyond Step-Wise Tool Calls for Tool-Augmented Agents
Jun 11, 2026Yaxin Du, Yifan Zhou, Yujie Ge +7Tool-Augmented Language Model Agents LLM Agent Workflow Optimization
Shanghai Jiao Tong University · IQuest Research · Beijing University of Aeronautics and Astronautics

Jul 26, 2026J/K move · Enter open · S save

E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios

Weihuang Zheng, Tianyuan Zou, Eileen Ye, Alphet Liu+4

Hunyuan Team, Tencent · Institute for AI Industry Research, Tsinghua University · School of Computer Science and Engineering, Southeast University

Computer-Use Agent Benchmarks Tool-Use Evaluation Tool-Augmented Language Model Agents AI Agent Benchmarks Synthetic Benchmark Generation Tool Use

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly deployed as agents that interact with stateful environments over multiple steps: gathering hidden information, composing tool calls, and committing state changes. We refer to this capability as multi-step tool use. Existing benchmarks have advanced tool-use agent evaluation, but often focus on isolated API calls, short trajectories, or settings that are difficult to scale or control. We introduce E-Bench, a fully synthetic benchmark with 323 state-changing tasks across three product domains: Honor of Kings, QQ Music, and Tencent Meeting. E-Bench decouples environment synthesis from task synthesis: graph-guided database filling builds reusable, orphan-free product environments, while generator-solver asymmetry creates tasks with both an information gap and a tool gap, requiring agents to discover hidden data and compose multiple tool calls before changing state. Outcomes are graded deterministically by database-state diffs. Since both environments and tasks are synthetic, E-Bench is controllable at the environment level and scalable at the task level. Benchmarking 11 cutting-edge LLMs shows that multi-step tool use remains challenging: Pass^3 stays below 60% for the strongest models, and even with code execution in the E-Bench-Code extension, reliability (Pass^3) remains below 70%.

Abstract

Tool-Augmented Language Model Agents

Explore similar work

E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios

Data Turnstile: A Scalable Open Framework for Function-Calling Data Generation

HyperTool: Beyond Step-Wise Tool Calls for Tool-Augmented Agents