cs.CVNov 17, 2026

Spool: Low-Latency Streaming Inference for Hybrid Vision-Language Models

Authors: Rui Ning, Om Chabra, Hari Balakrishnan

Organizations: Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA.

Abstract

Applications such as visual accessibility agents and robotics require answering questions about recent video content promptly, over a bounded window of past frames (e.g., the last two minutes). Hybrid vision-language models (VLMs), which interleave attention with linear recurrent layers such as Gated DeltaNet (GDN), are attractive for this setting, but maintaining a sliding window is expensive. The reason is that recurrent layers combine all past frames into a shared state, so expiring old frames normally requires re-running the entire window through the model. We observe that the recurrent update is affine in the state with input-dependent coefficients, so the effect of each frame can be captured as a transformation that composes associatively. We present Spool, a training-free framework that caches per-frame state transformations and composes them efficiently as the window advances, removing expired frames from each recurrent layer's state without replaying frames through the backbone. Removal is exact within each layer; as with key-value eviction in attention, expired frames can retain residual influence through the cached inputs computed by earlier layers. We design a dedicated data structure and GPU kernels to achieve low caching and composition overhead. Across five open-source hybrid VLMs and four video reasoning benchmarks, Spool achieves the lowest p95 time-to-first-token in all 20 model-benchmark tests, reducing it by a geometric mean of 6.6×6.6\times over the faster of two recent vLLM baselines, while improving accuracy by 2.3 percentage points on average.

Figures & tables

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Stateful Token Reduction for Long-Video Hybrid VLMs

    Feb 27, 2026Jindong Jiang, Amala Sanjay Deshmukh, Kateryna Chumachenko +7Hybrid Mamba-Transformer ArchitecturesEfficient VLM Inference

  2. VETO: Video Efficient Token Optimization for Vision Language Models

    Oct 1, 2026Gueter Josmy Faure, Hao Ping Wang, Min-Hung Chen +1Efficient VLM InferenceLarge Vision-Language Models

  3. VLMaxxing through FrameMogging Training-Free Anti-Recomputation for Video Vision-Language Models

    May 5, 2026JF Bastien, Sam D'AmicoEfficient VLM InferenceVideo-Language Models