Applications such as visual accessibility agents and robotics require answering questions about recent video content promptly, over a bounded window of past frames (e.g., the last two minutes). Hybrid vision-language models (VLMs), which interleave attention with linear recurrent layers such as Gated DeltaNet (GDN), are attractive for this setting, but maintaining a sliding window is expensive. The reason is that recurrent layers combine all past frames into a shared state, so expiring old frames normally requires re-running the entire window through the model. We observe that the recurrent update is affine in the state with input-dependent coefficients, so the effect of each frame can be captured as a transformation that composes associatively. We present Spool, a training-free framework that caches per-frame state transformations and composes them efficiently as the window advances, removing expired frames from each recurrent layer's state without replaying frames through the backbone. Removal is exact within each layer; as with key-value eviction in attention, expired frames can retain residual influence through the cached inputs computed by earlier layers. We design a dedicated data structure and GPU kernels to achieve low caching and composition overhead. Across five open-source hybrid VLMs and four video reasoning benchmarks, Spool achieves the lowest p95 time-to-first-token in all 20 model-benchmark tests, reducing it by a geometric mean of 6.6× over the faster of two recent vLLM baselines, while improving accuracy by 2.3 percentage points on average.
Figures & tables
Figure 1: Spool maintains a sliding video window without replaying retained frames. (a) Shifting the window retains four of five frames. (b) Prefix-cache invalidation forces the retained context to be rebuilt. (c) Spool composes cached recurrent transformations and reclaims key–value slots by moving surviving tail entries into holes left by eviction.
Figure 2: Architecture of the hybrid VLM. Multiple input frames are converted into visual tokens by a vision encoder and passed alongside text tokens into a unified backbone. This backbone interleaves recurrent layers, which update a fixed-size state for each token, with attention layers, which retain per-token key–value entries.
Figure 3: Cost of window rebuilding. Here we plot time for a frame to be processed by Qwen3.5-9B on 1 A100. Every time the window must be rebuilt from scratch, a 29.2x latency cost must be paid.
Figure 4: Efficient recurrent-state reconstruction with cached transformations. Spool caches each block’s affine state transformation and then replays them without the effect of a given block. A segment tree composes consecutive transformations, enabling reconstruction of the active window in O(logW) operations. In this example, frames 5–8 occupy cyclic slots 1–3 and 0, with frame 8 wrapping around to slot 0. Spool first applies F5 , then the combined transformation F6,7 , and finally F8 to the prefix state Sp . This recovers the window state using only three cached transformations, without reprocessing video tokens or updating the tree during reconstruction.
Figure 5: KV tail filling and RoPE rebasing. Top: surviving tokens 11 and 12 fill holes in slots 1 and 2, releasing the final page; P denotes the fixed prefix. Bottom: filling changes storage slots while preserving position coordinates. A separate rebase shifts all cached positions by a common offset ( −100 here) and rotates cached keys consistently.
Benchmark
Method
Qwen3.5
Nemotron-VL
Qwen3.5
Qwen3.6
Qwen3.5
9B
Nano-12B
27B
35B-A3B
122B-A10B
PhoStream
vLLM-Omni
31.53
5.52
12.11
8.13
5.57
vLLM-Incremental
12.22
2.23
5.79
2.44
2.02
Spool (Ours)
0.34 (×35.7)
0.17 (×13.0)
0.29 (×20.0)
0.91 (×2.7)
1.31 (×1.5)
StreamingBench
vLLM-Omni
31.59
5.55
9.64
11.74
8.02
vLLM-Incremental
11.70
2.24
4.51
1.84
1.99
Table 1: Online p95 TTFT (s, ↓ ). Spool can maintain real-time performance as live video streams in, achieving the lowest tail latency in all 20 model–benchmark tests: a geometric-mean speedup of 6.6× over the faster baseline and 33.8× over vLLM-Omni. Gains are largest on smaller models (0.34 s vs. 12.22 s on PhoStream with Qwen3.5-9B, 35.7× ) and remain clear at the largest scale (0.33 s vs. 1.99 s on StreamingBench with Qwen3.5-122B-A10B). (Parentheses are speedup over the faster baseline.)
Benchmark
Method
Qwen3.5
Nemotron-VL
Qwen3.5
Qwen3.6
Qwen3.5
9B
Nano-12B
27B
35B-A3B
122B-A10B
PhoStream
vLLM-Omni
54.06
55.72
67.16
67.66
70.15
vLLM-Incremental
61.25
57.71
71.64
69.65
71.14
Spool (Ours)
64.27 (+3.0%)
56.22 (−1.5%)
69.15 (−2.5%)
69.15 (−0.5%)
71.14 (+0.0%)
StreamingBench
vLLM-Omni
78.79
72.50
80.63
80.00
85.63
vLLM-Incremental
84.24
77.50
86.88
83.75
88.13
Table 2: Online accuracy (%, ↑ ). By answering on time, Spool drops fewer questions under live load and answers more correctly than both baselines where they fall behind. On BEYOND3D, it improves over the more accurate baseline by 3.6–18.4 points (mean 9.5), such as 30.32% vs. 11.94% with Nemotron-VL Nano-12B. On the other benchmarks, it stays within −2.5 to +3.6 points. Averaged over all tests, Spool reaches 52.8%, compared with 50.3% for vLLM-Incremental and 45.4% for vLLM-Omni. Questions dropped while the server is busy count as incorrect. (Parentheses are percentage points change over the more accurate baseline.)
Benchmark
Method
Qwen3.5
Qwen3.5
Nemotron-VL
Qwen3.5
Qwen3.6
Qwen3.5
0.8B
9B
Nano-12B
27B
35B-A3B
122B-A10B
PhoStream
vLLM-Omni
41.59
60.87
50.66
69.00
69.65
72.14
vLLM-Incremental
41.59
62.57
51.61
68.05
69.65
71.14
Spool (Ours)
41.97 (+0.4%)
63.33 (+0.8%)
51.04 (−0.6%)
69.94 (+0.9%)
68.66 (−1.0%)
71.14 (−1.0%)
StreamingBench
vLLM-Omni
72.32
83.64
81.41
86.06
80.63
86.88
vLLM-Incremental
73.13
85.66
83.43
88.08
83.13
88.13
Table 3: Offline accuracy (%, ↑ ). Spool’s windowed state maintenance preserves the backbone’s answer quality: without timing pressure, it stays within −1.6 to +1.8 points of the more accurate baseline in each of the 24 tests and has the highest mean accuracy, 54.4% vs. 54.1% for vLLM-Incremental and 53.7% for vLLM-Omni. Combined with the large online gains (Table 2 ), this shows that Spool’s online advantage comes from answering on time, not from changing what the model sees. (Parentheses are percentage points change over the more accurate baseline.)
Figure 6: Component ablation: latency per ingested frame (ms, ↓ ) as caching components are added, from a naive cache to the segment tree cache and KV filling.
Configuration
Qwen3.5
Nemotron-VL
0.8B
9B
Nano-12B
No Spool
5.53
27.33
27.06
+ Naive cache
8.52
30.93
31.06
+ Segment tree
10.77
33.93
35.43
+ KV Filling
9.43
32.16
35.05
Table 4: Peak GPU memory (GiB, ↓ ). Each of Spool’s components has a modest memory cost: the segment tree trades 2.3–4.4 GiB for 4.7–14.8% lower per-frame latency (Figure 6 ), and KV filling recovers 0.38–1.77 GiB of it while lowering latency further. Overall, the full system uses 9.2% more memory on average than the naive cache.
Figure 7: Effect of state reconstruction on output fidelity. (a) Output KL divergence from vLLM-Omni remains near 0.04 through 1024 expired frames with reconstruction, but accumulates without it. (b) Accuracy gain (points) from reconstruction on PhoStream’s backward questions, which target earlier video content, grouped by expired frames relative to window size. Both variants perform identically before expiration; reconstruction gains 2.4 points after half a window expires and 3.8–4.7 points beyond two windows.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Mean GDN state error. Relative Frobenius error averaged over each model’s GDN layers, comparing Spool with Spool without reconstruction against the vLLM reference. The evaluation uses all 9,000 Beyond3D questions.
Figure 9: Per-layer GDN state error for Qwen3.5-9B. Relative Frobenius error against the vLLM reference for all 24 GDN layers, averaged over questions within each expired-frame bin. The evaluation uses all 9,000 Beyond3D questions and a 64-frame retained window.
Benchmark
Method
Qwen3.5
Nemotron-VL
Qwen3.5
Qwen3.6
Qwen3.5
9B
Nano-12B
27B
35B-A3B
122B-A10B
PhoStream
vLLM-Omni
27.08
5.35
12.03
6.86
4.85
vLLM-Incremental
1.42
0.47
0.28
0.37
0.31
Spool (Ours)
0.33 (×4.3)
0.15 (×3.1)
0.28 (×1.0)
0.19 (×1.9)
0.28 (×1.1)
StreamingBench
vLLM-Omni
28.39
5.44
9.59
7.93
5.16
vLLM-Incremental
0.78
0.42
0.25
0.30
0.28
Appendix
Table 5: Online p50 TTFT (s, ↓ ). Spool’s latency advantage is concentrated in the tail: vLLM-Incremental answers typical questions quickly but stalls on a minority, which dominates its p95 (Table 1 ). Spool still has the lowest or tied-lowest p50 in 18 of 20 tests ( 1.9× geometric mean over the faster baseline, 32.2× over vLLM-Omni), with the largest gap on FastBench with Nemotron-VL Nano-12B (0.14 s vs. 2.66 s). vLLM-Incremental is marginally faster on StreamingBench with Qwen3.5-27B and Qwen3.5-122B-A10B, by 0.01–0.02 s. Parentheses: speedup over the faster baseline.
Figure 10: Empirical CDFs of online TTFT. Rows correspond to Qwen3.5-9B, Qwen3.5-122B-A10B, Qwen3.6-35B-A3B, Nemotron-VL Nano-12B, and Qwen3.5-27B. Columns correspond to PhoStream, StreamingBench, FastBench, and BEYOND3D.
Token reduction accelerates long-video vision--language models (VLMs), but existing methods target Transformers, where reduction is treated as token pruning. We study token reduction in hybrid Mamba--Transformer VLMs and find that it is stateful: Mamba layers maintain a recurrent state that accumulates information from earlier tokens, allowing discarded tokens to persist, so reduction behaves more like compression than dropping.We support this view with a representation-based probing method measuring how much information from discarded tokens is retained, and analyze layer-wise sparsity and cross-layer importance stability. Our findings show importance is sparse within layers but unstable across layers, making aggressive early pruning unreliable while hybrids remain robust to later reduction.Motivated by this, we propose a hybrid-aware token reduction framework with a low-to-high progressive schedule and a unified query-conditioned importance score for attention and Mamba layers. For Mamba, excluding the position-dependent decay from the recurrence produces a stronger selection signal. Across long-video benchmarks, our method achieves 3.8×--4.2× prefilling speedups at a 25% token budget while maintaining near-baseline accuracy and improving with light finetuning. Hybrid models benefit from aggressive reduction, improving both efficiency and accuracy, whereas Transformers exhibit the standard trade-off. Our method also outperforms prior baselines on the same hybrid backbone and combines effectively with visual redundancy reduction methods.
Jindong Jiang, Amala Sanjay Deshmukh, Kateryna Chumachenko +7
Processing long videos with Vision-Language Models (VLMs) is bottlenecked by the quadratic cost of visual tokens, making long-form inference prohibitively expensive. While single-axis compression methods mitigate this, they hit a hard efficiency floor because they treat spatial and temporal redundancy independently. We present VETO (Video Efficient Token Optimization for Vision-Language Models), a training-optional plug-in that eliminates this bottleneck through dual-axis compression: (i) an intra-frame compressor that merges semantically similar tokens within each frame via optimal-transport inspired matching, and (ii) an inter-frame compressor that identifies and merges temporally redundant frames. The key design insight is hierarchical ordering: by first compressing spatial dimensions, VETO drastically reduces the cost of subsequent global temporal matching, bypassing the efficiency wall of single-axis approaches, with an advantage that grows with modern fully-fused attention infrastructure. Empirically, VETO achieves up to 45% faster inference (e.g., on LLaVA-OneVision-7B) while preserving or improving accuracy. Under extreme token starvation (10% budget), VETO outperforms VFlowOpt (54.9%), VisionZip (52.6%), and FastV (47.9%) with 55.7% accuracy. We demonstrate universal applicability across LLaVA-OneVision, InternVL-2.5, and LongVA, with zero-shot accuracy preserved or improved in all cases.
Video vision-language models (VLMs) keep paying for visual state the stream already told us was stable. The factory wall did not move, but most VLM pipelines still hand the model dense RGB frames or a fresh prefix again. We study that waste as training-free anti-recomputation: reuse state when validation says it survives, and buy fresh evidence when the scene, query, or cache topology requires it. The largest measured win is after ingest. On frozen Qwen2.5-VL-7B-Instruct-4bit, adaptive same-video follow-up reuse preserves paired choices and correctness on a 93-query VideoMME breadth setting while reducing follow-up latency by 14.90-35.92x. The first query is still cold; the win starts when later questions reuse the same video state. Stress tests bound the result: repeated-question schedules hold through 50 turns, while dense-answer-anchored prompt variation separates conservative fixed K=1 repair from faster aggressive policies that drift. Fresh-video pruning is smaller but real. C-VISION skips timed vision-tower work before the first answer is generated. On Gemma 4-E4B-4bit, the clean 32f short cell reaches 1.316x first-query speedup with no paired drift or parse failures on 20 items; Qwen shows the fidelity/speed boundary. Stage-share ceiling (C-CEILING) is the accounting guardrail: a component speedup becomes an end-to-end speedup only in proportion to the wall-clock share it accelerates, so C-VISION and after-ingest follow-up reuse do not multiply. Candidate C-STREAM remains a native-rate target, not a headline result here. The broader direction is VLM-native media that expose change, motion, uncertainty, object state, sensor time, and active tiles directly, so models do not have to rediscover the world from dense RGB every frame.
JF Bastien, Sam D'Amico
Look, they said attention was all you needed, well we got your attention with this title.