LLM-powered agents execute tasks through a sequential loop of model generation and tool execution. Today's serving systems serialize this loop, leaving tool latency exposed on the task critical path. This paper presents PASTE, a tool-aware agent-serving system that predicts concrete future tool invocations from recurring agent patterns and executes them speculatively while the LLM is still generating. PASTE isolates speculative results until confirmed by the LLM and jointly schedules tool execution and returning LLM sessions to avoid shifting bottlenecks to the GPU. Across deep research, coding, and scientific-agent workloads, PASTE reduces average task completion time by 43.5% and lowers observed tool latency by 1.8x.
Figures & tables
Figure 1. Workflow of an LLM agent.
Figure 2. Example recurring tool-call patterns and latent data dependencies in agent traces.
Figure 3. Breakdown of tool execution and LLM generation.
Figure 4. Tool execution time histogram. Blue: args from prior outputs; orange: LLM-generated args.
Figure 5. LLM generation and tool execution have different load sensitivity under concurrent agent tasks.
Figure 6. Bursty GPU pressure from alternating LLM generation and tool execution.
Figure 7. System architecture of PASTE . PASTE places a session-level control plane across the agent runtime, tool executor, and LLM serving engine.
Figure 8. Lossless speculative tool execution. Workflow showing speculative tool calls running in parallel with LLM generation, followed by a match decision that either reuses the speculative result or runs the authoritative tool call normally before committing a tool result.
Figure 9. Gain-preserving LLM–tool co-scheduling keeps LLM load near the task-optimal region. Two timeline panels compare native execution and overlapped execution for four tasks. Native execution finishes later and shows LLM GPU load fluctuating above and below the optimal load, while overlapped execution finishes earlier and keeps GPU load close to the optimal level.
Component
Setting
Cluster
4 nodes
Per-node CPU
96 vCPUs, AMD EPYC 7V13
Per-node Memory
512 GB
Per-node GPU
8 GPU, NVIDIA A100 80GB
Total GPUs
32 × NVIDIA A100
LLM Serving Engine
vLLM
Table 1. Evaluation setup
Figure 10. Average E2E latency comparison against all baselines.
Figure 11. Average tool latency comparison against tool-side baselines.
Figure 12. CDF of tool latency for each task
Figure 13. Tool throughput under trace-driven arrivals.
Figure 14. CDF of per-request tool speedup over ORION/SpecFaaS.
Figure 15. Time breakdown across all baselines.
Figure 16. Scalability under multi-session concurrent agent requests: speedup compared with LLM-side baselines.
Figure 17. Task’s E2E latency in ablation evaluation.
Figure 18. Pattern prediction quality: Top-1 accuracy, Top-3 recall, and overall hit rate.