SchemaFill: Efficient LLM Tool Calling via Slot-Parallel Speculative Decoding

Zhi-Kai Chen; Song-Yan Li; De-Chuan Zhan; Han-Jia Ye

cs.LGOct 5, 2026

SchemaFill: Efficient LLM Tool Calling via Slot-Parallel Speculative Decoding

Authors: Zhi-Kai Chen, Song-Yan Li, De-Chuan Zhan, Han-Jia Ye

Organizations: School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China · Nanjing University, China

Abstract

LLM agents interact with external systems by generating structured tool calls. Given a user request, conversational context, and a catalog of tool schemas, a tool-calling model must select tools and generate their arguments, potentially producing multiple calls in a single response. Standard autoregressive decoding generates these calls token by token, incurring substantial latency for requests involving multiple calls or many argument fields. The explicit argument structure offers opportunities for parallel generation, but later argument values may depend on preceding fields and calls, so independently generated values can differ from the target model's output. We present SchemaFill, a framework for efficient LLM tool calling through slot-parallel speculative decoding. SchemaFill generates future slot values concurrently as candidates, without requiring advance knowledge of the actual call sequence or argument values. Candidates spanning multiple fields and calls are concatenated for verification by the target model under the actual output prefix. Only verified tokens are committed, and the target supplies corrections when candidates disagree. This applies target verification while exploiting parallelism across slots and calls. On Glaive and BFCL, SchemaFill achieves up to a 4.05 $\times$ improvement in end-to-end throughput over autoregressive decoding. Code is available at https://github.com/Czzzk/SchemaFill.

Figures & tables

Figure 1: SD denotes speculative decoding. Direct slot-parallel generation can raise throughput but may change the target response. SchemaFill combines parallel proposals with target verification. The direct-parallel baseline follows the skeleton-then-parallel-fill idea of Skeleton-of-Thought ( Ning et al., 2024 ) , with the correct skeleton supplied in advance.

Figure 2: AR deviation from independently generated slot values. Left: BFCL responses with at least eight parameters; right: at least three calls. Each curve follows a fixed cohort as more slots or calls are considered. Shading: 95% Wilson intervals ( Wilson, 1927 ) .

Figure 3: Target-KV cache lifecycle in SchemaFill. The public prefix and verified rows occupy a shared arena, while attention masks restrict branches to valid context and their own candidate paths. One target pass verifies branch scratch rows. Accepted rows extend their cache paths; rejected suffixes are discarded. Only verified tokens on the committed branch become output; a side-branch value must be verified again under the actual output prefix before it can be committed.

Dataset	Method	AL	E2E TPS	Speedup	AR agreement (%)
Glaive	AR	$1.00$	$91.21$ 0.18	1.00 $\times$	100.00
	EAGLE-3 ( Li et al., 2025 )	$3.24$	$225.04$ 0.91	2.47 $\times$	100.00
	ToolSpec ( Xia et al., 2026 )	$2.64$	$194.47$ 1.04	2.13 $\times$	100.00
	Parallel Fill	—	$219.79$ 0.72	2.41 $\times$	82.62
	SchemaFill	$\mathbf{6.21}$	340.60 2.96	$\mathbf{3.73}\bm{\times}$	100.00
BFCL v3	AR	$1.00$	$92.68$ 0.14	1.00 $\times$	100.00

Table 1: End-to-end results on the Glaive ( Glaive AI, 2023 ) and BFCL v3 ( Patil et al., 2025 ) test sets with Qwen2.5-7B-Instruct. All methods use temperature

0

. Acceptance length (AL) is committed tokens per target verification call; E2E TPS is committed tokens per second, shown as mean

\pm

sample standard deviation over three launches. Speedup is the TPS ratio to AR within each dataset.

Stage	Configuration	Accept Length	E2E TPS	Speedup
A	EAGLE-3	2.84	$200.99$	1.00 $\times$
B	A + schema-guided MAIN path	4.57	$297.61$	1.48 $\times$
C	B + SIDE speculative drafting	5.48	$330.46$	1.64 $\times$
D	C + cross-call packing (full SchemaFill)	6.42	$375.71$	1.87 $\times$

Table 2: Cumulative ablation on the BFCL test set. Each stage retains all mechanisms from the preceding stage. A is the EAGLE-3 baseline. B uses SchemaFill’s schema-guided MAIN path to forecast tool and slot positions and concatenate structural text with drafted values, without SIDE branches. C adds SIDE drafting of future slot values. D adds cross-call packing of prepared values and intervening structural text into one target-verification proposal, yielding full SchemaFill. Speedup is the TPS ratio to the EAGLE-3 baseline (A).

Ref. Calls ( $C$ )	Accept Length	E2E TPS	Nodes/ Verify
1	5.14	$265.95$	$23.43$
2	7.98	$438.62$	$35.76$
3	8.06	$443.54$	$39.29$
4	7.54	$410.72$	$42.71$

Table 3: Separate BFCL v3 call-count diagnostic (100 requests/group), using

B=C

. AL is committed tokens per verification; E2E TPS is tokens/s; Nodes/Verify is pooled tree size. All requests and initialization are included.

Figure 4: Target-verification workload during SchemaFill decoding. Candidate-node counts peak at slot initialization and drop sharply in subsequent rounds. Curves average requests reaching each round in the Table 3 diagnostic, excluding prompt tokens. Rounds 6–11 are compressed; endpoint labels mark the last rounds with at least 10 contributing requests.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Target model	KV-cache increase	Model-and-cache increase from extra KV
Qwen2.5-7B-Instruct	$+24.65\%$	$+0.28\%$

Appendix

Table 4: SchemaFill’s per-request effective KV-cache overhead with an EAGLE-3 drafter, BF16 weights, and batch size one. KV-cache growth is relative to greedy AR with the same target-KV layout. The model-and-cache comparison holds target and drafter weights fixed and measures the memory increase due to extra KV.

Phase	Total (s)	Mean (ms/request)	Share (%)
BM25 tool forecast	0.032	0.070	0.027
EAGLE-3 drafting	21.14	46.98	18.28
Target verification	86.94	193.19	75.17
Other generation	7.54	16.76	6.52
Total	115.65	256.99	100.00

Appendix

Table 5: SchemaFill time breakdown on the BFCL test set. Percentages use the sum of the four profiled phases.

Dataset	Method	Accept Length	E2E TPS	Speedup
Glaive	AR	$1.00$	$91.01$	1.00 $\times$
	EAGLE-3 ( Li et al., 2025 )	$3.12$	$203.27$	2.23 $\times$
	ToolSpec ( Xia et al., 2026 )	$2.62$	$190.64$	2.09 $\times$
	SchemaFill	$\mathbf{5.99}$	326.13	$\mathbf{3.58}\bm{\times}$
BFCL v3	AR	$1.00$	$92.22$	1.00 $\times$
	EAGLE-3	$2.75$	$188.13$	2.04 $\times$

Appendix

Table 6: End-to-end results on Glaive650 and BFCL450 at target temperature

1

, with one run per method and dataset. Learned drafters use temperature

0

. Speedup is the TPS ratio to AR at target temperature

1

Model	Method	Temperature $=0$			Temperature $=1$
Model	Method	TPS	AL	Speedup	TPS	AL	Speedup
Qwen2.5-7B	AR	92.51	1.00	1.00 $\times$	92.22	1.00	1.00 $\times$
	EAGLE-3	199.89	2.84	2.16 $\times$	188.13	2.75	2.04 $\times$
	ToolSpec	239.87	3.27	2.59 $\times$	238.57	3.28	2.59 $\times$
	SchemaFill	371.48	6.42	$\mathbf{4.02}\bm{\times}$	356.61	6.16	$\mathbf{3.87}\bm{\times}$
Qwen3-8B	AR	79.78	1.00	1.00 $\times$	79.33	1.00	1.00 $\times$

Appendix

Table 7: Cross-target evaluation on BFCL450 at target temperatures

0

and

1

. TPS is end-to-end throughput, and AL includes initialization. Speedup is the TPS ratio to AR at the corresponding temperature.

Explore similar work

OoO-Spec: Out-of-Order Semantic Speculation for Fast Tool Calling
Aug 1, 2026Zhiheng Zhang, Mujie Xu, Feiyu Sun +1Self-Speculative Tool Invocation
The University of Tokyo · Peking University · Nanjing University
Harness Engineering in LLM Tool Use via Agent-Native Reusable Tool Primitives
Sep 1, 2026Haibo Jin, Suijin Wang, Xucheng Yu +2Large Language Model Tool Use
School of Information Sciences University of Illinois at Urbana-Champaign Champaign, IL 61820 · Independent Researcher Starc Institute
LLM Agents Already Know When to Call Tools -- Even Without Reasoning
May 10, 2026Chung-En Sun, Linbo Liu, Ge Yan +2Large Language Model Tool Use Large Language Model Agents
University of California, San Diego

Aug 1, 2026, cs.CLJ/K move · Enter open · S save

OoO-Spec: Out-of-Order Semantic Speculation for Fast Tool Calling

Zhiheng Zhang, Mujie Xu, Feiyu Sun, Zhixin Zhang

The University of Tokyo · Peking University · Nanjing University

Self-Speculative Tool Invocation Schema

Open paper AskPDF arXiv

LLMs generate tool calls token by token, even though the function choice and argument values can often be predicted in parallel from the request and tool schema. ToolSpec reduces this cost by drafting schema tokens and retrieving earlier calls, but cannot propose request-specific values absent from either source. We present OoO-Spec, which computes these missing semantics out of order. At request arrival, a Qwen3-0.6B sidecar predicts the function choice and all schema-defined argument slots in one parallel request-level wave while the target begins ToolSpec decoding. The runtime joins the slot values, renders the resulting call as text, and exposes it to subsequent candidate-construction rounds. The target polls without blocking, re-tokenizes a ready hint with its own tokenizer, and remains the sole verifier and commit authority. The sidecar is trained once with LoRA on Qwen2.5-32B teacher traces and used unchanged across Qwen2.5, Qwen3, and Llama targets, without target-specific drafter training. Across seven fully ranked targets and three benchmarks under greedy batch-one decoding, OoO-Spec is fastest among all evaluated methods in all 21 target-benchmark cells, reaching 2.46x-5.34x over autoregressive decoding with an unweighted mean of 3.89x, versus 2.95x for ToolSpec. It also outperforms every evaluated released learned drafter in each comparable cell. Across Qwen3-4B, 8B, 14B, and 32B targets, the same sidecar improves on ToolSpec by 34.1% on average. Its compact semantic payload averages 85 bytes per request excluding protocol metadata, supporting effective split-GPU overlap.

Abstract

Explore similar work

OoO-Spec: Out-of-Order Semantic Speculation for Fast Tool Calling

Harness Engineering in LLM Tool Use via Agent-Native Reusable Tool Primitives

LLM Agents Already Know When to Call Tools -- Even Without Reasoning