Streaming Inference

Neural Processes (NPs), and specifically Transformer Neural Processes (TNPs), have demonstrated remarkable performance across tasks ranging from spatiotemporal forecasting to tabular data modelling. However, many of these applications are inherently sequential, involving continuous data streams such as real-time sensor readings or database updates. In such settings, models should support cheap, incremental updates rather than recomputing internal representations from scratch for every new observation -- a capability existing TNP variants lack. Drawing inspiration from Large Language Models, we introduce the Incremental TNP (incTNP). By leveraging causal masking, Key-Value (KV) caching, and a data-efficient autoregressive training strategy, incTNP matches the predictive performance of standard TNPs while reducing the computational cost of updates from quadratic to linear time complexity. We empirically evaluate our model on a range of synthetic and real-world tasks, including tabular regression and temperature prediction. Our results show that, surprisingly, incTNP delivers performance comparable to -- or better than -- non-causal TNPs while unlocking orders-of-magnitude speedups for sequential inference. Finally, we assess the consistency of the model's updates -- by adapting a metric of "implicit Bayesianness", we show that under a one-at-a-time streaming protocol, incTNP retains a prediction rule as implicitly Bayesian as standard non-causal TNPs, demonstrating that incTNP achieves the computational benefits of causal masking without sacrificing the consistency required for streaming inference.

Speech Language Models (SpeechLMs) are increasingly deployed for streaming speech generation, where audio playback must start quickly and never stall. Serving them requires running two very different model components on the same system. An autoregressive LLM backbone emits audio tokens, and a codec converts chunks of those tokens into waveforms. Existing systems run the two as separately scheduled stages, and therefore either serialize them, let them interfere, or strand capacity by placing them on separate devices. We observe that overlapping the backbone and the codec on separate parts of the GPU raises throughput. Although this increases codec latency, most chunks can tolerate it, as long as it neither lies on the critical path of time-to-first-audio (TTFA) nor causes a playback stall. Based on this insight, we present VoxServe, a SpeechLM serving system that allocates GPU resources according to the urgency of chunks. VoxServe executes work on three asynchronous lanes. The LM lane runs the backbone on a large partition of the GPU's streaming multiprocessors, the background codec lane runs on a small partition beside it, and the urgent lane uses the whole GPU. The scheduler queues each request to the LM lane and routes every chunk the backbone completes by its urgency. A request's first chunk and chunks close to their playback deadline go to the urgent lane, and the rest are batched on the background lane. On Qwen3-TTS, VoxServe sustains 1.5x the request rate of the best existing system at a 200ms TTFA target without playback stalls. The code of VoxServe is available at https://github.com/vox-serve/vox-serve.

We present StreamDEQ, a method that aims to infer frame-wise representations on videos with minimal per-frame computation. Conventional deep networks perform feature extraction from scratch at each frame in the absence of ad-hoc solutions. We instead aim to build streaming recognition models that can natively exploit temporal smoothness between consecutive video frames. We observe that the recently emerging implicit layer models provide a convenient foundation to construct such models, as they define representations as the fixed points of shallow networks, which need to be estimated using iterative methods. Our main insight is to distribute the inference iterations over the temporal axis by using the most recent representation as a starting point at each frame. This scheme effectively recycles the recent inference computations and greatly reduces the required processing time. Through extensive experimental analysis, we show that StreamDEQ is able to recover near-optimal representations within a few frames and maintain an up-to-date representation throughout the video duration. Our experiments on video semantic segmentation, video object detection, and human pose estimation in videos show that StreamDEQ achieves on-par accuracy with the baseline while providing 2-4x higher throughput.