Agent workflows issue dependent model calls that share long context prefixes, so serving them efficiently depends on reusing the key-value (KV) cache of those prefixes. On accelerators that compile model graphs ahead of time, however, the device KV budget is fixed, and prefixes evicted under load must be recomputed. Restoring them from host memory avoids this recomputation, but the transfer occupies the same engine thread as prefill and decode, so its benefit is not guaranteed. In this paper, we study when restoring beats recomputing on a commercial static-graph neural processing unit (NPU) stack (Rebellions ATOM+, vllm-rbln 0.10.2) that serves a Qwen3-8B planner on two cards and a Qwen3-0.6B executor on one. We first show that repeated prefill dominates planner work when default serving collapses: 89.5% of the executed planner prompt tokens had been materialized before. We then build a synchronous host-DRAM tier on the runtime's public KV copy calls, together with a scheduler-aware event model whose predictions are frozen before each test cohort. Experimental results show that the host tier raises workflow service-level objective (SLO) attainment from 22.8% to 100% and from 9.4% to 99.8% in the two tested overload conditions, and endpoint confirmation on two new seeds supports conservative load gains of 11.8% (chain) and 18.2% (fork), while mixed and bursty workloads remain seed-sensitive. The frozen model predicts median and 99th-percentile (P99) latency within its 15% tolerance in all 24 common-load conditions, including the 16-18% P99 increase that the tier causes for fork workflows, but it misorders three policy contrasts and overestimates the benefit of a cheaper store path (11% predicted against 3.1% measured). The trade-off therefore depends on runtime scheduling and transfer cost as well as cache capacity.
Figures & tables
Figure 1: Planner work in the KV64 fork collapse run (seed 501, 0.55 workflows/s). Under stock serving (B0), 89.5% of the executed prompt tokens are repeated (a), and prefill calls made only of repeated tokens take 59.3% of the host-inclusive API time (b). Repeated tokens are classified from earlier runtime completions and block history. The panels use different numerators and denominators, and API time includes warmup and failed-client requests, so it is not divided by the client wall time in (c).
Figure 2: Overview of the serving path (top) and the prospective evaluation (bottom), with a chain workflow as the dependency example. On the planner, restore, prefill, store, and decode share one engine thread: the full host prefix is restored when the usable host prefix is longer than the local one ( Hh>Hl ), and missing host entries are stored after prefill. Boxes show the order of operations, not their durations. The event model makes predictions from the frozen trace and does not control serving.
Cohort
Seeds
KV
Client
collapse lineage
501
64
orig.
restore, fork 0.55
501
64
orig.
restore, burst 0.65
505
32
orig.
host capacity × load
506
32
orig.
calibration (480 wf)
701
32
orig.
common load, 24 cond.
702, 703
32
fixed
Table 1: Cohorts. Runs have 1,200 workflows unless noted. The collapse cohort also includes 48-workflow observer checks, screen probes have at least 480 workflows and 600 s of arrivals, and confirmation runs have at least 1,200 workflows and 1,800 s (more at high rates). Only the calibration cohort is used to fit the model. Client: original, or with the event-loop fix described in § 4.1 .
Figure 3: Recovery under KV32 mixed-burst overload (seed 505, 0.65 workflows/s, 1,200 arrivals per arm). Each point summarizes a 60 s arrival cohort; the horizontal axis is arrival time, not completion time. SLO attainment counts all workflows including errors, and latency quantiles use successful workflows only. With the host tier, both B0 and F6 keep SLO attainment near 100% and latency flat, whereas without it SLO attainment falls to zero and latency rises to hundreds of seconds. Error counts are in Table 2 .
Figure 4: Frozen common-load forecasts against measurements for 24 conditions (top) and predicted against observed host-tier effects at fixed F6 admission (bottom). Every P50 and P99 forecast falls within the frozen 15% tolerance (shaded band, not an uncertainty interval), while crosses mark the three host-tier effects whose sign was mispredicted. In the top panels, dark and pale fills are seeds 702 and 703.
Condition
Policy
SLO (%)
Errors
wf/s
KV64 fork 0.55
B0
22.8 → 100
536 → 0
0.269 → 0.549
F6
73.8 → 100
0 → 0
0.516 → 0.549
KV32 burst 0.65
B0
9.4 → 99.8
291 → 2
0.429 → 0.650
F6
34.8 → 100
0 → 0
0.581 → 0.651
Table 2: Workflow SLO attainment without → with the host tier, 1,200 workflows, one run each. Errors are failed workflows.
Figure 5: Capacity screen (seed 711, all 52 probes) and frozen endpoint confirmation (seeds 712/713, 48 runs) per workload and policy. Filled circles are finite-window passes and crosses are failures; dashed spans are brackets, not confidence intervals. C: the low endpoint passed and the high endpoint failed on both new seeds; U: both endpoints passed; S: seed-sensitive. The host-tier gain is confirmed for chain and fork only. Bounds in the panel titles are the frozen host-tier low endpoint over the B0 high endpoint, minus one.
Workload
P50 (%)
P99 (%)
J/wf vs. B0 (%)
chain
−19.9 / −19.7
−0.4 / +11.0
−3.7 / −3.9
fork
−20.1 / −19.6
+17.5 / +15.6
−6.5 / −6.9
mixed
+1.7 / −3.8
−3.4 / +2.3
−6.1 / −6.3
burst
−23.6 / −29.3
−36.5 / −40.5
−10.5 / −10.4
Table 3: F6 → F6+host at common load, seed 702 / 703. Last column: B0 → F6+host, so it includes the admission cap as well as the tier; gross board energy over the fixed offered window per offered workflow.
Workload
Host lower
B0 upper
Gain
chain
1.258 ✓
1.125 ×
≥ 11.8%
fork
0.629 ✓
0.532 ×
≥ 18.2%
mixed
0.769 mixed
0.582 ×
not established
burst
0.727 ✓
0.492 mixed
not established
Table 4: Confirmed capacity endpoints (workflows/s). ✓ : pass on both new seeds; × : fail on both; mixed: one of each.
Figure 6: Store path before and after the change (A0). (a) Copy-size diagnostic with identical packing; (b) the op gate, isolated total store time excluding pool setup (7.8 s); (c) the HTTP gate, serialized requests with copy and packing timed separately; (d) the serving panel, measured repeats against frozen predictions. A0 cuts isolated store time by 38%, but the measured fork P99 improves far less than predicted. The panels are separate measurements and are not combined.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Role
Phase
Active
u
o
Planner
prefill
1
57.876607
1.760586
Planner
decode
1
33.706571
0.117403
Planner
decode
2
34.441473
0.118596
Planner
decode
3
36.734474
0.135935
Planner
decode
4
37.070009
0.116141
Executor
prefill
1
14.958987
1.875139
Appendix
Table 5: Frozen runtime mean u and wrapper residual o , in milliseconds. Prefill u is per 128-token runtime call; prefill o is per model forward, not per chunk.