Agent sessions may involve frequent context edits, which effectively invalidate cached KV state and force requests to wait for re-prefill before decoding. We present Callo, a disaggregated serving system that leverages invalidated KV for speculative drafts to reduce this latency, while verifying drafts against the exact KV after prefill before committing tokens. Callo makes effective use of decode resources by scheduling speculations during periods of underutilization and preempting speculations when drafts are likely to not be accepted. We evaluated Callo on a two-GPU testbed using SWE-agent sessions across three model families, observing all-request speedups of 1.05-1.19x over vLLM at moderate load. We demonstrate that these gains are also realized at scale through cluster-level simulations driven by real-world traces.
Figures & tables
Figure 1: One edited request. (a) vLLM, like other conventional systems supporting P/D, waits on the D-instance for the exact KV. (b) Callo drafts on invalidated KV meanwhile, then verifies.
Figure 2: Drafts made on invalidated KV (Llama-3.3-70B-Instruct FP8, SWE-agent L5, 32-token window, offline). (a) Miss steps whose first k drafts are all accepted (240 steps), against drafts on the exact KV and n-gram drafts. (b) Agreement by the draft’s top-1 minus top-2 logit gap (1,400 steps); below, the share of positions in each bin. (c) Accepted drafts when a share f of the invalidated KV’s position–layer cells is replaced by exact KV before drafting: Callo ’s delivery order and two reference orders (240 steps; bands: 95% confidence intervals).
Per miss request
Workload
Prompt
Reply
Miss
Prefill
Reusable
mean
mean
rate
p50/p90
p50/p90
Llama L5
13.5k
117
79%
3.1k/8.9k
2.5k/7.4k
Llama L10
15.3k
121
58%
8.3k/18.3k
7.6k/16.8k
Gemma L5
15.4k
199
74%
5.2k/8.3k
4.2k/7.2k
Gemma L10
17.5k
189
53%
10.9k/15.5k
9.8k/14.5k
Table 1: Workloads recorded on SWE-bench Verified and replayed in the evaluation (§ 5.1 ). L n keeps the last n observations in full. For each miss request, Prefill counts tokens computed by P-instances, and Reusable counts those whose invalidated KV remains on D-instances.
Figure 3: Architecture of Callo and the life of a miss request. (a) Numbers give the order of events in this example: (1) the P-instance starts the re-prefill while the D-instance starts staging, (2) the D-instance registers the request’s blocks with the P-instance, (3) the P-instance’s connector starts writing each finished layer into them, (4) the D-instance starts drafting, (5) the P-instance signals completion, and (6) the D-instance starts verification. (b) The D-instance’s batch with a limit of four requests, of which SR1 and SR2 speculate: when R3 arrives, SR2 yields its slot but keeps its drafts for verification.
Property of invalidated KV
Design choice
Close to the exact KV
Inherit and reposition (§ 4.2 )
Near-exact acceptance (Fig. 2 a)
Target-model drafting (§ 4.1 )
Errors at near-ties (Fig. 2 b)
Cumulative early stop (§ 4.5 )
Heals layer by layer (Fig. 2 c)
Layer-wise streaming (§ 4.3 )
Mixing never hurts (Fig. 2 c)
Lock-free drafting (§ 4.3 )
Table 2: Properties of invalidated KV measured in § 3.2 , and the design choices of Callo they motivate.
Light (12%)
Moderate (52%)
Full (100%)
Workload
Miss
All
Miss
All
Miss
All
Llama L5
1.13
1.10
1.19
1.14
1.32
1.27
Llama L10
1.24
1.17
1.24
1.19
1.21
1.17
Gemma L5
1.07
1.05
1.06
1.05
1.10
1.09
Gemma L10
1.11
1.07
1.11
1.08
1.12
1.10
Mistral L5
1.15
1.14
1.16
1.14
1.16
1.15
Table 3: Mean request-latency ratios (vLLM/ Callo ) for miss requests and all requests. D-instance batch limit: 16. Ratios use requests completed by both systems.
Figure 4: All-request speedup (vLLM/ Callo ) across D-instance batch limits. Lower panels show baseline time with requests queued at the P-instance and with the D-instance full while requests wait. Error bars: 95% CIs. Hollow markers: either run left measured requests unissued or unfinished.
Figure 5: Contributions of streaming and speculation to all-request speedup. Dashed: measured Callo -Stream. Both curves use requests completed by all three systems. CIs and hollow markers follow Figure 4 .
Figure 6: Mean latency breakdown for 624 miss requests that speculate under Callo at moderate load. Whiskers: 95% CIs. Grey ends at exact-KV arrival, estimated for vLLM. Hatched: accepted drafts, in equivalent baseline decode time.
Figure 7: Miss-request speedup over Callo -Stream across P:D allocations in a 96-GPU cluster. Load is relative to each allocation’s vLLM capacity. Below: share of Azure’s week at or below each load; the dashed light (12%) and moderate (52%) loads are its 25th and 75th percentiles.
Variant
Folded
Speedup
Accepted
Callo (fixed-tie greedy)
49%
1.15
22.6
Callo -Stream
4.5%
1.01
–
any-max greedy verification
54%
1.17
24.4
speculative sampling, T=0.8
45%
1.14
21.4
n-gram drafts
18%
1.05
7.5
Table 4: Single-request ablations on Llama-3.3-70B and SWE-agent L5. Folded: net share of miss penalty hidden. Speedup: ratio of mean miss-request latencies (vLLM/variant). Accepted: mean accepted drafts per miss request, including zeros.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Sessions
Requests
Accepted (of 32)
Whole window
3 looping at the limit
183
30.2
81%
6 others at the limit
392
25.5
64%
86 below the limit
825
24.2
58%
All
1,400
25.4
62.5%
All but the 3 looping
1,217
24.6
59.7%
Appendix
Table 5: Offline acceptance on invalidated KV by session group (the 1,400 miss requests of Figure 2 b, fixed-tie greedy, 32-token window). Figure 2 a’s 65% whole-window share comes from a different sample of 240 requests (Appendix B.1 ).
Figure 8: Per-token relative KV difference on the 240 miss requests of Figure 2 : median and 90th percentile for keys (a) and values (b), with the numerical reference (“floor”) from the same prompt computed at shifted positions. Layer 0’s values are identical and omitted from the logarithmic plot.
Figure 9: Figure 2 a and b for Gemma-4-31B (a, b) and Mistral-Small-3.2-24B (c, d): 100 miss requests of each model’s own SWE-agent L5 recording, 32-token window, offline. In (b) and (d), the two smallest logit-gap bins hold 46–68 positions each.
Logit gap
< .25
.25–.5
.5–1
1–2
2–4
4–8
≥ 8
Llama-3.3-70B
0.63
0.69
0.89
0.97
1.00
1.00
1.00
Mistral-Small-3.2
0.70
0.92
0.99
1.00
1.00
1.00
1.00
Gemma-4-31B
0.57
0.78
0.86
0.95
1.00
1.00
1.00
Appendix
Table 6: Agreement tables of the early stop: agreement probability g by the draft’s top-1 minus top-2 logit gap, fitted per model on the held-out issues.
Refused
Load
Misses
Spec.
Budget
Evicted
Scratch
Slot
Cut
Light
280
274
6
0
0
0
1.8%
Moderate
1,247
1,070
173
4
0
0
4.3%
Full
1,103
755
344
4
0
0
4.0%
Past cap.
1,271
652
402
54
163
0
3.5%
Appendix
Table 7: Admission of miss requests on the main configuration (D-instance batch limit 16) by load, over every miss request of the run that reaches admission, measured or not; past capacity: 0.045 sessions/s; moderate load pools two arrival schedules. Spec.: miss requests that speculate; Evicted: a block of the previous request that staging needs was no longer cached; Cut: share of speculations cut short within three drafts because the exact KV arrived.
Moderate load
90% load
Setting
Miss
Others
Miss
Others
θ=0.1
1.162
1.024
1.193
1.071
θ=0.3
1.161
1.026
1.194
1.076
θ=0.5
1.155
1.024
1.195
1.083
No early stop
1.164
1.027
1.194
1.074
K=2,048
1.134
1.048
1.182
1.091
Appendix
Table 8: Parameter sweeps in the simulated 1:1 cluster (96 GPUs, three arrival schedules per setting). Values are latency speedups over vLLM. Others denotes non-speculating requests.
P:D
Load
Discarded
Total accepted
Accepted/spec.
reduced by
retained
default / disabled
1:2
Moderate
35.4%
98.1%
22.3 / 22.5
90%
34.1%
96.7%
26.0 / 26.7
1:1
Moderate
30.3%
99.1%
19.7 / 19.8
90%
28.5%
96.2%
22.7 / 23.2
2:1
Moderate
29.3%
94.5%
16.0 / 16.3
Appendix
Table 9: Confidence early stopping in 96-GPU clusters, against disabling it. Top: the default θ=0.3 per allocation and load; accepted/spec. gives mean accepted drafts per speculation (default / disabled). Bottom: pooled over the eight settings.
Figure 10: The main configuration (D-instance batch limit 16) up to and past its capacity of 0.04 sessions/s. Top: vLLM/ Callo mean request latency (bars: 95% confidence intervals). Bottom: share of time under vLLM that the P-instance has a waiting request and that the D-instance waits for room in its KV pool. Hollow and shaded: past capacity, where the run left measured requests unissued or unfinished.
0.045 sessions/s
0.04
vLLM
Callo
vLLM
P-instance cache hit rate
47.7%
60.4%
70.6%
P-instance tokens computed
7.64M
7.11M
5.20M
Sessions in flight
13 → 29
11 → 12
11 → 11
Measured sessions finished
23 of 27
26 of 27
24 of 24
Measured requests unissued
90
3
0
Appendix
Table 10: The P-instance at 0.045 sessions/s, one step past the main configuration’s capacity, against vLLM at capacity (0.04), one run each, from the end of the warm-up to the end of the run. Sessions in flight, measured or not: at the end of the arrival window → peak afterwards.
Figure 11: Mean request latency against the session arrival rate on the testbed’s configuration (one P-instance, one D-instance), measured and simulated, for vLLM and Callo at each D-instance batch limit (hollow: testbed points past the configuration’s capacity).
D-instance batch limit
16
2
4
6
vLLM latency error
2.2%
2.2%
3.2%
5.7%
Gain inside testbed CI
7/8
6/8
6/8
7/8
Gain error
+0.004
−0.001
+0.003
+0.004
Speculating requests
−1.1 %
+0.0 %
+0.5 %
+0.4 %
Accepted drafts
−3.9 %
−4.0 %
−1.6 %
−4.4 %
Appendix
Table 11: Simulator against the testbed, by D-instance batch limit. vLLM latency: largest per-request error over the load points up to capacity; gain inside the testbed’s 95% CI: out of all load points with a testbed CI; gain error: median over the points up to capacity; speculating requests and accepted drafts: error in their counts, pooled over the points up to capacity.
Figure 12: Miss-request speedup over P-instance utilization and the D-instance’s KV occupancy (“D utilization” on the axis), interpolated from nine P:D ratios. Dashed lines follow the four allocations of Figure 7 .
P:D
Load
vLLM+CD
Callo
Callo +CD
Callo +budget
1:2
12%
1.00
1.11
1.10
1.05
52%
1.03
1.16
1.17
1.15
90%
1.14
1.26
1.32
1.31
1:1
12%
1.00
1.12
1.09
1.05
52%
1.02
1.13
1.11
1.12
90%
1.08
1.18
1.19
1.21
Appendix
Table 12: Callo combined with prefilling short requests on the D-instance in 96-GPU clusters: all-request speedup over vLLM, mean of three arrival schedules. CD: Dynamo’s default rule; budget: rB≤K . Miss-request speedups differ from these by at most 0.04. † D-instances are overloaded, and not every session finishes.