SPECMEM: Accelerating Agentic Tool Calling via Live Memory Management

Zihe Song; Wenting Zhao; Ye Liu; Yuanjie Zhu; Ke Xu; Shanghao Li; Philip S. Yu

cs.CLOct 22, 2026

SPECMEM: Accelerating Agentic Tool Calling via Live Memory Management

Authors: Zihe Song, Wenting Zhao, Ye Liu, Yuanjie Zhu, Ke Xu, Shanghao Li, Philip S. Yu

Organizations: Department of Computer Science, University of Illinois Chicago · Salesforce Research

Abstract

Tool-calling agents are mostly optimized for accuracy; in deployment, however, their growing cost is latency: every tool call is decoded autoregressively. Speculative decoding accelerates this without changing the output, and retrieved past calls make strong drafts under real use, where each user's behavior tends to repeat. Yet existing systems keep no cross-session memory, or build the draft store once and never update it. We propose SpecMem, a tool-call memory kept alive and managed throughout deployment: verified calls are written back online into bounded per-user stores with LRU eviction, and a side-effect gate defers irreversible calls until verification confirms. On multi-session, multi-user query streams served by three open-weight models, a frozen store steadily loses acceptance while SpecMem holds it: a 27% gap in mean accepted tokens (+11% to +34% across benchmarks, tracking how much novel work arrives after the store is built) and up to 1.74x wall-clock speedup. Ablations attribute the gain almost entirely to keeping the store live; local per-user scoping adds no acceptance but closes a measured cross-user leakage channel, and gating on declared side effects, rather than on retrieval confidence, is what eliminates irreversible mistakes.

Figures & tables

Figure 1: A frozen ToolSpec -style datastore vs. SpecMem on a recurring call: user

u

checked Google’s stock the day before and now asks for Apple’s. Each arm drafts a candidate call for a draft-then-verify decoder over the fixed served model (acceptance

=

token longest-common-prefix). SpecMem retrieves

u

’s own GOOG call, accepted up to the changed ticker (

17

28

target tokens) with only the remainder decoded; the frozen ToolSpec -style store’s stale build-time entry accepts

9

, the schema-only draft

7

(full-run MAT

19.96

vs.

15.70

and

5.92

). Only the live store writes the verified target back, so its advantage grows across sessions.

Benchmark	Arm	MAT	MAT seed-std	Accepted frac	Exact rate
BFCL	No memory (schema draft)	5.92	$\pm 0.07$	0.190	0.00
	Static datastore (frozen)	15.70	$\pm 0.20$	0.492	0.314
	Ours (local, live, evict)	19.96	$\pm 0.05$	0.621	0.444
Seal-Tools	No memory (schema draft)	4.60	$\pm 0.06$	0.176	0.007
	Static datastore (frozen)	15.68	$\pm 0.27$	0.568	0.435
	Ours (local, live, evict)	21.04	$\pm 0.37$	0.753	0.640

Table 1: Post-warmup acceptance (sessions 1–11,

3

seeds; per-arm

n

: BFCL

6{,}814

, Seal-Tools

7{,}592

). Metrics as defined in § 4.1 ; exact rate

=

draft identical to target.

Figure 2: Per-session MAT on the BFCL stream (3 seeds). The frozen store and SpecMem are nearly tied at session 1; the frozen store then degrades while SpecMem holds and the no-memory floor stays flat.

Figure 3: Per-session MAT on

\tau^{2}

-bench (

3

domains,

3

seeds). At session 1 both stores tie near

31

MAT; the frozen ToolSpec /static store then degrades into a

20

–

25

band while SpecMem holds at

27

–

32

. Right: the advantage over the frozen store grows from

{\sim}1\%

(session 1) to a peak of

{\sim}37\%

(session 10;

34\%

at the final session 11) as novel per-user calls accumulate, an effect a single-session benchmark cannot expose.

Method	API-Bank	ToolAlp.	BFCLv4	ToolBench
Vanilla AR	$1.00\times$	$1.00\times$	$1.00\times$	$1.00\times$
ToolSpec (reimpl.)	$1.21\times$	$1.66\times$	$1.06\times$	$1.58\times$
$+$ SpecMem (ours)	$\mathbf{1.62\times}$	$\mathbf{1.74\times}$	$\mathbf{1.18\times}$	$\mathbf{1.69\times}$
SpecMem MAT	$19.1$	$20.9$	$20.5$	$21.0$
Live token-match MAT	$19.1$	$20.4$	$19.8$	$20.7$

Table 2: Wall-clock decode speedup vs. vanilla autoregressive decoding (p50) on ToolSpec ’s four benchmarks. Bottom rows: MAT for SpecMem and for a live token-match store at the same footprint (

3

seeds).

Arm	live?	per-user?	evicts?	MAT
Static (frozen)	—	—	—	$15.66{\scriptstyle\pm.11}$
Global, live+evict	✓	—	✓	$20.06{\scriptstyle\pm.35}$
Local, live, unbnd	✓	✓	—	$19.96{\scriptstyle\pm.29}$
Ours	✓	✓	✓	$\mathbf{19.91}{\scriptstyle\pm.29}$

Table 3: Ablation: post-warmup MAT as each lifecycle component is toggled (full scale, mean

{\pm}

std over

3

seeds).

Policy	Weighted cost	Bad actions	Rate/opp.	Worst sev.	Safe spec-exec
Naive spec-exec	800	124	0.75	10	255
Confidence gate	255	40	0.24	10	244
Side-effect gate (ours)	0	0	0.00	0	214

Table 4: Side-effect-gating simulation (

400

-call streams,

5

seeds; setup in § 5.2 ). Weighted cost

=\sum\sigma

over wrong irreversible speculative executions; rate

=

per drafted unsafe call; safe spec-exec

=

executions causing no irreversible harm.

Tool-Using Language Agents

Tool-using language agents combine language models with APIs, code, and other tools to answer questions and carry out multi-step tasks.

See all 59 papers

MemToolAgent: Leveraging Memory for Tool Using Agents Based on Environment and User Feedback
Jun 6, 2026Suleyman Armagan Er, Danilo Ribeiro, Yogesh Virkar +5Tool-Using Agents LLM Tool Use
AWS AI · University of Washington
Speculate with Memory: Lossless Acceleration for LLM Agents
Jul 14, 2026Yu Li, Qinyuan Ye, Prafulla Kumar Choubey +2Next-Action Prediction LLM Agent Memory
Salesforce Research
Stateful Inference for Low-Latency Multi-Agent Tool Calling
May 25, 2026Victor NorgrenLLM Inference Efficiency Multi-Agent LLM Systems
LayerScale, Inc.

Jun 6, 2026J/K move · Enter open · S save

MemToolAgent: Leveraging Memory for Tool Using Agents Based on Environment and User Feedback

Suleyman Armagan Er, Danilo Ribeiro, Yogesh Virkar, Surafel Lakew+4

AWS AI · University of Washington

Tool-Using Agents LLM Tool Use Agent Memory Management Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Modern large language model (LLM) agents can use external tools to help users solve complex tasks. However, for problems that require learning from long-term historical events or from previous agent-environment interactions, LLM agents are required to use memory mechanisms to store and retrieve experiences. While sophisticated memory systems exist for dialogue agents, few studies have empirically examined how to improve agents' tool-using capabilities through past user-agent conversations. We propose MemToolAgent, a framework that improves tool use through memory management. Our approach contains a memory extraction module that processes past experiences into structured memory entries, and a retrieval module that dynamically selects a subset of the stored memory entries. This enables more personalized and accurate responses aligned with user preferences and feedback without requiring LLM fine-tuning. In summary, this work has three main contributions: (1) a unified memory entry format that improves both general-purpose and personalized tool use without LLM fine-tuning, (2) a reflection-based memory extraction that uses environment and user feedback to distill wrong executions into critiques to store, and (3) a retrieval module that chooses how many past experiences to use based on the memory similarity distribution. MemToolAgent achieves 29%, 80%, and 17% relative improvements compared to strong baselines on the WorkBench, NESTFUL, and PEToolBench benchmarks, respectively.

Abstract

Tool-Using Language Agents

Explore similar work

MemToolAgent: Leveraging Memory for Tool Using Agents Based on Environment and User Feedback

Speculate with Memory: Lossless Acceleration for LLM Agents

Stateful Inference for Low-Latency Multi-Agent Tool Calling