Large language models (LLMs) on mobile devices can act as agents that call external software tools to complete complex tasks, but the major challenge of supporting on-device mobile agentic AI with LLMs is the long latency of LLM tool calling on mobile devices. In particular, the decode phase is the main contributor of such long latency, but is hard to speed up on mobile processors as it is bound by memory bandwidth rather than compute power. To address this limitation, in this paper we present SchemaWeaver, a new on-device mobile agentic AI technique that enforces schema-guided parallel decoding in the decode phase of LLM tool calling, to reduce its latency by producing multiple tokens in each decode pass. SchemaWeaver can be generically applied to the LLM inference engine on mobile devices without involving any training efforts or changes on LLMs, and our experiment results on major mobile SoCs and LLM families show that SchemaWeaver can reduce the LLM tool calling latency by up to 3.9 times, compared to the competitive baselines.
Figures & tables
Figure 1. Agentic AI systems based on LLM agents that call external software tools
Figure 2. The latency of LLM tool calling in on-device mobile agentic AI
Figure 3. Wall-clock time breakdown of LLM tool calling latency on an OnePlus 12R smartphone
Mobile Processor
Prefill
Decode
Hexagon NPU
685.4
16.4
Adreno GPU
93.1
11.3
CPU
145.7
22.6
Table 1. Prefill and decode throughputs (tokens/s) on mobile processors with Qwen3-1.7B-Q8 model.
Figure 4. Prefill and decode performance with different memory bus frequencies on mobile devices
Figure 5. Overview of SchemaWeaver design
# tokens in one pass
1
2
3
4
5
6
8
OnePlus 12R
1.00
1.61
2.15
1.20
1.91
2.49
2.26
Google Pixel 7
1.00
1.75
2.34
1.11
1.88
2.47
2.06
Table 2. Relative increase of the parallel decoding cost in one decode pass on different smartphone models
Figure 6. The tool skeleton in one prefill pass, with every argument value starting at the same position
Model
Serial decode (ms)
Jump-forward prefills (ms)
SchemaWeaver’s prefill (ms)
Qwen3-1.7B
1,201
577
314
Qwen3-1.7B 4-bit
999
615
335
Qwen3-0.6B
545
299
172
Llama-3.2-1B
743
407
222
Llama-3.2-3B
1,886
997
551
SmolLM2-1.7B
1,352
945
395
Table 3. Compute cost of tool skeleton decode per call
Figure 7. Parallel decoding of three argument values with rugged lengths being padded
Figure 8. Parallel decoding invalidates the KV cache entries of token sequences. The write-back reassembles the tool text and rebuilds memory of the whole call.
Figure 9. Handling optional arguments with a provisional call that only contains the required arguments
When the optional arguments are decided
Argument correctness
Tool call correctness
Latency (ms)
End of the call (serial decoding)
80%
76.0%
2,586
Before required argument values (in the skeleton)
63%
62.5%
1,105
Never (all put into the skeleton)
57%
70.8%
1,103
After required argument values (SchemaWeaver)
79%
74.0%
1,211
Table 4. Different choices of times when the optional arguments are decided, with the BFCL dataset ( Patil et al., 2025 ) and Qwen3-1.7B model
Qwen3-1.7B
Qwen3-1.7B 4-bit
Qwen3-0.6B
Qwen3-4B
Llama-3.2-1B
Llama-3.2-3B
SmolLM2-1.7B
Latency (ms)
12R
Pixel 7
12R
Pixel 7
12R
Pixel 7
12R
Pixel 7
12R
Pixel 7
12R
Pixel 7
12R
Pixel 7
No Grammar
2,602
3,565
2,242
2,482
1,271
1,569
5,605
24,004
1,854
2,707
4,137
6,145
3,749
5,664
Constrained decoding
2,586
3,561
2,173
2,456
1,183
1,569
5,574
23,871
1,636
2,415
4,009
5,852
3,541
5,619
Jump-forward decoding
1,921
2,723
1,807
1,794
1,071
1,346
3,867
18,563
1,304
1,842
2,719
4,365
2,396
3,536
Speculative, EAGLE-3
9,610
12,031
8,141
9,102
3,709
4,812
14,714
OOM
8,703
12,847
14,585
21,290
/
/
Speculative, DFlash-2
/
/
/
/
/
/
13,433
OOM
/
/
/
/
/
/
Table 5. The end-to-end latency and correctness of tool calls over different LLMs. BFCL-v4 dataset is used.
Figure 10. Implementation of SchemaWeaver inside the llama-server
Qwen3-1.7B
Llama-3.2-1B
SmolLM2-1.7B
Latency (ms)
12R
Pixel 7
12R
Pixel 7
12R
Pixel 7
Constrained
1,771
2,523
1,035
1,567
2,682
4,136
Jump-forward
804
1,080
461
693
1,115
1,630
SchemaWeaver
677
944
328
490
779
1,166
Correctness
Constrained
67.2%
58.4%
69.3%
Table 6. The end-to-end latency and correctness of tool calls over different LLMs. DroidCall dataset is used.
Figure 11. Tool call latency by phase on the OnePlus 12R
Figure 12. Tool call latency by the number of arguments of the call on the OnePlus 12R.
Figure 13. Tool call latency and speedup with different memory bus frequencies on the Pixel 7.
Setup
Latency (ms)
Correctness
Constrained decoding
2,586
76.0%
+ skip the fixed tokens (jump-forward)
1,421
76.0%
jump-forward with a provisional call
1,349
76.0%
+ skeleton and provisional call
1,369
75.0%
+ parallel value sequences
1,247
74.0%
+ padded value batch (= SchemaWeaver)
1,211
74.0%
Table 7. Latency and correctness on SchemaWeaver design variants
Phase
Passes
Tokens
Time (ms)
Constrained decoding
35
35
2,004
Tool selector
1
6
109
Skeleton prefill
1
19
268
Parallel decode
2
3
128
Provisional tool call
505
Write-back
1
18
238
Table 8. Cost breakdown for get_traffic_info tool, on the OnePlus 12R with Qwen3-1.7B.
Constrained decoding
No Grammar
Jump- forward
Speculative, EAGLE-3
SchemaWeaver
Peak memory
4.725 GB
4.725 GB
4.725 GB
6.926 GB
4.734 GB
Table 9. Peak resident memory of each scheme on the OnePlus 12R with Qwen3-1.7B.
Figure 14. Energy per tool call on the Google Pixel 7
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Step (ms)
Prefill
Phone
Core set
Threads
serial
4-sequence
Ratio
ms/token
12R
2 A715
2
65.0
112.3
1.73
20.1
12R
4 A715
4
54.2
65.9
1.22
9.8
12R
4 A715 + X3
5
43.2
50.8
1.18
5.6
12R
all 8
8
59.1
73.7
1.25
6.9
12R
3 A510 + X3
4
73.7
98.6
1.34
13.9
Appendix
Table 10. Cost of a serial decode step, a four-sequence step and a prefill token on different core sets of the two phones; Ratio is the four-sequence step over the serial step; the protocol’s core set in bold.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China · Nanjing University, China