Applications such as visual accessibility agents and robotics require answering questions about recent video content promptly, over a bounded window of past frames (e.g., the last two minutes). Hybrid vision-language models (VLMs), which interleave attention with linear recurrent layers such as Gated DeltaNet (GDN), are attractive for this setting, but maintaining a sliding window is expensive. The reason is that recurrent layers combine all past frames into a shared state, so expiring old frames normally requires re-running the entire window through the model. We observe that the recurrent update is affine in the state with input-dependent coefficients, so the effect of each frame can be captured as a transformation that composes associatively. We present Spool, a training-free framework that caches per-frame state transformations and composes them efficiently as the window advances, removing expired frames from each recurrent layer's state without replaying frames through the backbone. Removal is exact within each layer; as with key-value eviction in attention, expired frames can retain residual influence through the cached inputs computed by earlier layers. We design a dedicated data structure and GPU kernels to achieve low caching and composition overhead. Across five open-source hybrid VLMs and four video reasoning benchmarks, Spool achieves the lowest p95 time-to-first-token in all 20 model-benchmark tests, reducing it by a geometric mean of 6.6× over the faster of two recent vLLM baselines, while improving accuracy by 2.3 percentage points on average.
Figures & tables
Figure 1: Spool maintains a sliding video window without replaying retained frames. (a) Shifting the window retains four of five frames. (b) Prefix-cache invalidation forces the retained context to be rebuilt. (c) Spool composes cached recurrent transformations and reclaims key–value slots by moving surviving tail entries into holes left by eviction.
Figure 2: Architecture of the hybrid VLM. Multiple input frames are converted into visual tokens by a vision encoder and passed alongside text tokens into a unified backbone. This backbone interleaves recurrent layers, which update a fixed-size state for each token, with attention layers, which retain per-token key–value entries.
Figure 3: Cost of window rebuilding. Here we plot time for a frame to be processed by Qwen3.5-9B on 1 A100. Every time the window must be rebuilt from scratch, a 29.2x latency cost must be paid.
Figure 4: Efficient recurrent-state reconstruction with cached transformations. Spool caches each block’s affine state transformation and then replays them without the effect of a given block. A segment tree composes consecutive transformations, enabling reconstruction of the active window in O(logW) operations. In this example, frames 5–8 occupy cyclic slots 1–3 and 0, with frame 8 wrapping around to slot 0. Spool first applies F5 , then the combined transformation F6,7 , and finally F8 to the prefix state Sp . This recovers the window state using only three cached transformations, without reprocessing video tokens or updating the tree during reconstruction.
Figure 5: KV tail filling and RoPE rebasing. Top: surviving tokens 11 and 12 fill holes in slots 1 and 2, releasing the final page; P denotes the fixed prefix. Bottom: filling changes storage slots while preserving position coordinates. A separate rebase shifts all cached positions by a common offset ( −100 here) and rotates cached keys consistently.
Benchmark
Method
Qwen3.5
Nemotron-VL
Qwen3.5
Qwen3.6
Qwen3.5
9B
Nano-12B
27B
35B-A3B
122B-A10B
PhoStream
vLLM-Omni
31.53
5.52
12.11
8.13
5.57
vLLM-Incremental
12.22
2.23
5.79
2.44
2.02
Spool (Ours)
0.34 (×35.7)
0.17 (×13.0)
0.29 (×20.0)
0.91 (×2.7)
1.31 (×1.5)
StreamingBench
vLLM-Omni
31.59
5.55
9.64
11.74
8.02
vLLM-Incremental
11.70
2.24
4.51
1.84
1.99
Table 1: Online p95 TTFT (s, ↓ ). Spool can maintain real-time performance as live video streams in, achieving the lowest tail latency in all 20 model–benchmark tests: a geometric-mean speedup of 6.6× over the faster baseline and 33.8× over vLLM-Omni. Gains are largest on smaller models (0.34 s vs. 12.22 s on PhoStream with Qwen3.5-9B, 35.7× ) and remain clear at the largest scale (0.33 s vs. 1.99 s on StreamingBench with Qwen3.5-122B-A10B). (Parentheses are speedup over the faster baseline.)
Benchmark
Method
Qwen3.5
Nemotron-VL
Qwen3.5
Qwen3.6
Qwen3.5
9B
Nano-12B
27B
35B-A3B
122B-A10B
PhoStream
vLLM-Omni
54.06
55.72
67.16
67.66
70.15
vLLM-Incremental
61.25
57.71
71.64
69.65
71.14
Spool (Ours)
64.27 (+3.0%)
56.22 (−1.5%)
69.15 (−2.5%)
69.15 (−0.5%)
71.14 (+0.0%)
StreamingBench
vLLM-Omni
78.79
72.50
80.63
80.00
85.63
vLLM-Incremental
84.24
77.50
86.88
83.75
88.13
Table 2: Online accuracy (%, ↑ ). By answering on time, Spool drops fewer questions under live load and answers more correctly than both baselines where they fall behind. On BEYOND3D, it improves over the more accurate baseline by 3.6–18.4 points (mean 9.5), such as 30.32% vs. 11.94% with Nemotron-VL Nano-12B. On the other benchmarks, it stays within −2.5 to +3.6 points. Averaged over all tests, Spool reaches 52.8%, compared with 50.3% for vLLM-Incremental and 45.4% for vLLM-Omni. Questions dropped while the server is busy count as incorrect. (Parentheses are percentage points change over the more accurate baseline.)
Benchmark
Method
Qwen3.5
Qwen3.5
Nemotron-VL
Qwen3.5
Qwen3.6
Qwen3.5
0.8B
9B
Nano-12B
27B
35B-A3B
122B-A10B
PhoStream
vLLM-Omni
41.59
60.87
50.66
69.00
69.65
72.14
vLLM-Incremental
41.59
62.57
51.61
68.05
69.65
71.14
Spool (Ours)
41.97 (+0.4%)
63.33 (+0.8%)
51.04 (−0.6%)
69.94 (+0.9%)
68.66 (−1.0%)
71.14 (−1.0%)
StreamingBench
vLLM-Omni
72.32
83.64
81.41
86.06
80.63
86.88
vLLM-Incremental
73.13
85.66
83.43
88.08
83.13
88.13
Table 3: Offline accuracy (%, ↑ ). Spool’s windowed state maintenance preserves the backbone’s answer quality: without timing pressure, it stays within −1.6 to +1.8 points of the more accurate baseline in each of the 24 tests and has the highest mean accuracy, 54.4% vs. 54.1% for vLLM-Incremental and 53.7% for vLLM-Omni. Combined with the large online gains (Table 2 ), this shows that Spool’s online advantage comes from answering on time, not from changing what the model sees. (Parentheses are percentage points change over the more accurate baseline.)
Figure 6: Component ablation: latency per ingested frame (ms, ↓ ) as caching components are added, from a naive cache to the segment tree cache and KV filling.
Configuration
Qwen3.5
Nemotron-VL
0.8B
9B
Nano-12B
No Spool
5.53
27.33
27.06
+ Naive cache
8.52
30.93
31.06
+ Segment tree
10.77
33.93
35.43
+ KV Filling
9.43
32.16
35.05
Table 4: Peak GPU memory (GiB, ↓ ). Each of Spool’s components has a modest memory cost: the segment tree trades 2.3–4.4 GiB for 4.7–14.8% lower per-frame latency (Figure 6 ), and KV filling recovers 0.38–1.77 GiB of it while lowering latency further. Overall, the full system uses 9.2% more memory on average than the naive cache.
Figure 7: Effect of state reconstruction on output fidelity. (a) Output KL divergence from vLLM-Omni remains near 0.04 through 1024 expired frames with reconstruction, but accumulates without it. (b) Accuracy gain (points) from reconstruction on PhoStream’s backward questions, which target earlier video content, grouped by expired frames relative to window size. Both variants perform identically before expiration; reconstruction gains 2.4 points after half a window expires and 3.8–4.7 points beyond two windows.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Mean GDN state error. Relative Frobenius error averaged over each model’s GDN layers, comparing Spool with Spool without reconstruction against the vLLM reference. The evaluation uses all 9,000 Beyond3D questions.
Figure 9: Per-layer GDN state error for Qwen3.5-9B. Relative Frobenius error against the vLLM reference for all 24 GDN layers, averaged over questions within each expired-frame bin. The evaluation uses all 9,000 Beyond3D questions and a 64-frame retained window.
Benchmark
Method
Qwen3.5
Nemotron-VL
Qwen3.5
Qwen3.6
Qwen3.5
9B
Nano-12B
27B
35B-A3B
122B-A10B
PhoStream
vLLM-Omni
27.08
5.35
12.03
6.86
4.85
vLLM-Incremental
1.42
0.47
0.28
0.37
0.31
Spool (Ours)
0.33 (×4.3)
0.15 (×3.1)
0.28 (×1.0)
0.19 (×1.9)
0.28 (×1.1)
StreamingBench
vLLM-Omni
28.39
5.44
9.59
7.93
5.16
vLLM-Incremental
0.78
0.42
0.25
0.30
0.28
Appendix
Table 5: Online p50 TTFT (s, ↓ ). Spool’s latency advantage is concentrated in the tail: vLLM-Incremental answers typical questions quickly but stalls on a minority, which dominates its p95 (Table 1 ). Spool still has the lowest or tied-lowest p50 in 18 of 20 tests ( 1.9× geometric mean over the faster baseline, 32.2× over vLLM-Omni), with the largest gap on FastBench with Nemotron-VL Nano-12B (0.14 s vs. 2.66 s). vLLM-Incremental is marginally faster on StreamingBench with Qwen3.5-27B and Qwen3.5-122B-A10B, by 0.01–0.02 s. Parentheses: speedup over the faster baseline.
Figure 10: Empirical CDFs of online TTFT. Rows correspond to Qwen3.5-9B, Qwen3.5-122B-A10B, Qwen3.6-35B-A3B, Nemotron-VL Nano-12B, and Qwen3.5-27B. Columns correspond to PhoStream, StreamingBench, FastBench, and BEYOND3D.