cs.CLMar 4, 2026

From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Models

Authors: Junlong Tong, Zilong Wang, YuJie Ren, Peiran Yin, Hao Wu, Wei Zhang, Xiaoyu Shen

Organizations: Shanghai Jiao Tong University · Institute of Digital Twin, Eastern Institute of Technology, Ningbo

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Standard Large Language Models (LLMs) are predominantly designed for static inference with pre-defined inputs, which limits their applicability in dynamic, real-time scenarios. To address this gap, the streaming LLM paradigm has emerged. However, existing definitions of streaming LLMs remain fragmented, conflating streaming generation, streaming inputs, and interactive streaming architectures, while a systematic taxonomy is still lacking. This paper provides a comprehensive overview and analysis of streaming LLMs. First, we establish a unified definition of streaming LLMs based on data flow and dynamic interaction to clarify existing ambiguities. Building on this definition, we propose a systematic taxonomy of current streaming LLMs and conduct an in-depth discussion on their underlying methodologies. Furthermore, we explore the applications of streaming LLMs in real-world scenarios and outline promising research directions to support ongoing advances in streaming intelligence. We maintain a continuously updated repository of relevant papers at https://github.com/EIT-NLP/Awesome-Streaming-LLMs.

Figures & tables

Figure 1: Illustration of three types of streaming large language models (LLMs). ( Left ) Output-streaming LLM performs streaming generation after static reading. ( Middle ) Sequential-streaming LLM performs streaming generation after streaming reading. ( Right ) Concurrent-streaming LLM performs streaming generation while streaming reading.
Figure 2: Overview of streaming LLM paradigms and their key challenges. The figure contrasts Output-streaming, Sequential-streaming, and Concurrent-streaming LLMs, highlighting their core goals and corresponding research components. Concurrent-streaming builds on the first two and adds extra challenges in real-time streaming architecture adaptation and interaction policy learning.
Figure 4: Illustration of structural conflicts when adapting batch-oriented LLMs (left) to concurrent streaming (right), where indicates the token generation direction, denotes attention dependencies, blocks represent the input, and blocks represent the output. (1) Attention contention : Ambiguous causal dependency between the newly inserted streaming input and historical outputs. (2) Position-ID conflict : The new streaming input and generated output compete for the identical position ID.
Table 1: Comparison of concurrent-streaming architecture adaptation methods from the perspectives of attention (Attn.) and position (Pos.). indicates the token generation direction, while denotes attention dependencies. blocks represent the input stream, and blocks represent the output stream. pp indicates the corresponding position ID.
Figure 5: Illustration of interaction decision in concurrent streaming LLMs, where the model learns to dynamically schedule reading inputs and emitting outputs.
Streaming-InBoundInc.Cxt.Example methods
TextMemory-✓StreamingDialogue Macháček et al. (2023)
AudioCausal, Memory✓✓WhisperStreaming Li et al. (2024b)
VideoMemory-✓Timechat-online Yao et al. (2025)
Table 2: Summary of sequential streaming tasks. Incremental encoding ( Inc ) and context management ( Ctx ) are the key technical dimensions. The checkmark ( ✓ ) indicates the scope covered by existing research.
Task typeLevelModalityParadigmInteraction policyExample methods
InOutR.C.I.G.RuleSFTRL
TranslationX→Y\mathcal{X}\to\mathcal{Y}T/ST/S✓✓✓✓✓✓✓Seed LiveInterpret 2.0 Cheng et al. (2025b)
DetectionX→Y\mathcal{X}\to\mathcal{Y}T/S/VT--✓--✓-FineHarm Li et al. (2025h)
ASRX→Y\mathcal{X}\to\mathcal{Y}ST--✓✓✓✓-ReaLLM Seide et al. (2024) , Llama-omni Fang et al. (2024)
TTSX→Y\mathcal{X}\to\mathcal{Y}TS-✓-✓✓✓-Cosyvoice Du et al. (2024) , DSM Zeghidour et al. (2025)
QAX→Y\mathcal{X}\to\mathcal{Y}T/S/VT/S-✓✓--✓✓Qwen3-omni Xu et al. (2025c) , VideoLLM-online Chen et al. (2024a)
Table 3: Summary of concurrent streaming tasks and representative methods. Tasks are categorized by processing depth ( Level ), where X→Y\mathcal{X}\to\mathcal{Y} denotes direct mapping (perception) and X→Z→Y\mathcal{X}\to\mathcal{Z}\to\mathcal{Y} denotes intermediate processing with a latent state Z\mathcal{Z} (cognition). Modality: text ( T ), speech ( S ), vision ( V ). Streaming Paradigm: re-encoding ( R ), Concatenated ( C ), Interactive ( I ), Group ( G ). Interaction Policy: Rule-based ( Rule ), SFT-based ( SFT ), and RL-based ( RL ). The checkmark ( ✓ ) indicates the scope covered by existing research.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Typical SurveysPrimary FocusTypical Technologies CoveredDifferentiation in This Survey
Survey Category: Efficient LLMs
Li et al. (2024a)   
Dantas et al. (2025)   
Cheng et al. (2025a)   
Wu et al. (2026b)Compression/adaptation and memory bottlenecks.1) Compression: quantization, pruning, distillation, low-rank; and 2) KV-cache management: selection / eviction, cache compression, offloading, sliding-window / hierarchical cache.Prior surveys treat compression and KV-cache optimization as separate threads; we unify them under streaming interaction, highlighting online constraints and dynamic runtime budgeting.
Survey Category: Multimodal LLMs
Appendix
Table 1: Comparison between this survey and existing related surveys. We highlight the unique positioning of our work in the context of streaming interaction.
Streaming Generation
MechanismModality-OutMethods
TokenBlockRefinement  
✓--TGPT OpenAI (2023) , Gemini Team et al. (2023) , Qwen3 Yang et al. (2025a) , DeepSeek-V3 DeepSeek-AI et al. (2024) , InternVL Chen et al. (2024c) , ChatGLM GLM et al. (2024) , Gemma Gemma Team (2024)
✓--SAudioLM Borsos et al. (2023a) , SpeechGPT Zhang et al. (2023a) , AudioPaLM Rubenstein et al. (2023) , FireRedTTS Guo et al. (2024a) , Moshi Défossez et al. (2024) , Llama-omni2 Fang et al. (2025) , Qwen3-Omni Xu et al. (2025c) , StyLLE Hao et al. () , Llmvox Shikhar et al. (2025) , SpeakStream Bai et al. (2025)
✓--VDALLE Ramesh and others (2021) , VideoPoet Kondratyuk et al. (2024) , Chameleon Team (2024) , Emu3 Wang et al. (2024e) , Anole Chern et al. (2024) , Lumina-mGPT2.0 Xin et al. (2025) , Infinity Han et al. (2025)
-✓-TSAT Wang et al. (2018) , SoT Ning et al. (2023) , CtrlDiff Huang and Tang (2025) , PredSent Hwang et al. (2025) , Falcon Gao et al. (2025) , SSD-LM Han et al. (2023) , WeDLM Liu et al. (2025a) , Next-Block Tian et al. (2025) , Block Diffusion Arriola et al. (2025)
Appendix
Table 2: Summary of additional literature on output-streaming LLMs, complementing the discussion in Sec. 3 .
Incremental Encoding
TypeModality-InMethods
Fragmented EncodingAtomic Encoding  
✓-TSimulMT Wang et al. (2024c) , Moshi Défossez et al. (2024) , Codec Ye et al. (2025) , dmel Bai et al. (2024) , Lightweight Audio Segmentation Frohmann et al. (2024) , Semantic VAD Shi et al. (2023)
✓-SWhisper-Streaming Macháček et al. (2023) , SimulST Zhang and Feng (2023) , CTC Graves (2012) , Speechtokenizer Xin et al. (2024) , Moshi Défossez et al. (2024) , Codec Ye et al. (2025) , dmel Bai et al. (2024) , Lightweight Audio Segmentation Frohmann et al. (2024) , Semantic VAD Shi et al. (2023)
✓-VS-ViT Zhao et al. (2023)
-✓TSaT Frohmann et al. (2024) , SegFree Iranzo-Sánchez et al. (2024) , WtP Minixhofer et al. (2023) , subword regularization Kudo (2018) , SentencePiece Kudo and Richardson (2018) ,
Appendix
Table 3: Summary of additional literature on sequential-streaming LLMs, complementing the discussion in Sec. 4 .
Streaming Paradigm
ParadigmModalityMethods
R.C.I.G.InOut 
✓---TTSimul-LLM Agostinelli et al. (2024) , SiLLM Guo et al. (2024c) , TransLLaMA Koshkin et al. (2024b) , CAST Koshkin et al. (2024a) , RALCP Wang et al. (2024c)
✓---STCAST Koshkin et al. (2024a) , TransLLaMA Koshkin et al. (2024b)
-✓--TSLLMVoX Shikhar et al. (2025) , Mini-Omni Xie and Wu (2024)
-✓--SSMini-Omni Xie and Wu (2024)
Appendix
Table 4: Summary of additional literature on concurrent-streaming LLMs, complementing the discussion in Sec. 5 .
More in this topic

LLM Inference

Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.

See all 276 papers

Explore similar work

CardsList
May 30, 2026

ProactiveLLM: Learning Active Interaction for Streaming Large Language Models

Standard Large Language Models (LLMs) follow a read-then-generate paradigm, causing unnecessary latency and computation. Streaming LLMs alleviate this issue by generating while receiving inputs, but still struggle to decide when to interact with the stream. Existing methods either hard-code interaction timing or rely on costly external alignment signals, such as timing labels, reasoning trajectories, or stronger teachers. In this paper, we propose ProactiveLLM, which achieves active interaction by leveraging the model's endogenous states to guide interaction decisions. The model first learns to perceive semantic sufficiency from partial inputs through two complementary training mechanisms: mask-based streaming modeling and synchronized privileged self-distillation (SPSD). The former applies monotonic random masking to the input during training, simulating progressively revealed streaming inputs and enabling the model to learn local semantic dependencies from partial-input views. The latter aligns the partial-context student view with a full-context teacher view generated by the same evolving model, allowing privileged full-context evidence to guide the student's understanding under incomplete observations. Together, these mechanisms induce endogenous sufficiency cues without requiring external teachers or annotations, providing a versatile foundation for the plug-and-play integration of diverse decision heads. Extensive evaluation across text and speech streaming tasks confirms that ProactiveLLM significantly reduces interaction latency while maintaining quality, validating its capacity for dynamic and active interaction. Code is publicly available at https://github.com/EIT-NLP/StreamingLLM/tree/main/ProactiveLLM.

Junlong Tong, Yao Zhang, Anhao Zhao +3

Eastern Institute of Technology, Ningbo · Shanghai Jiao Tong University · Hong Kong Polytechnic University +1

LLM InferenceLLM Inference Acceleration
AskPDF
Oct 1, 2026

Which LLM to pick? Online Active Model Selection for Large Language Models

Large Language Models (LLMs) are increasingly applied to process streaming data, with practitioners relying on benchmarks to select the best model even though these signals only approximate real performance. While oracle annotations can provide reliable feedback, they are often costly and difficult to obtain at scale. To address this challenge, we propose ONLINE LLM PICKER, the first framework for active model selection for LLMs in online settings. Given an arbitrary stream of queries and a limited annotation budget, ONLINE LLM PICKER selects the most informative prompts for annotation to identify the best LLM among candidate models. Across multiple tasks including 10 datasets, for over 130 language models, we show that ONLINE LLM PICKER saves annotation cost by up to 71.67% while reliably identifying the best or near-best model for the stream. We also show that using the returned model for sequential generation on unannotated prompts across the stream reduces regret by up to a factor of 2.51x, indicating that ONLINE LLM PICKER can identify the best or near-best model well before processing all streaming prompts.

Alessandro Turrin, Patrik Okanovic, Torsten Hoefler +1

TU Delft · ETH Zurich

Model SelectionLLM EvaluationActive Learning
AskPDF
Jun 7, 2026

Harnessing Streaming Video in the Wild

Vision-Language Models (VLMs) are increasingly required to process unbounded video streams in applications such as video-call assistants, live commentary, and embodied robots. An ideal streaming system should support proactive interaction, long-horizon memory, and real-time processing, while resting on a VLM backbone capable of handling diverse in-the-wild streaming tasks. However, existing VLMs excel at offline video understanding but fall short in streaming capabilities and lack dedicated infrastructure for streaming deployment. We address this gap on three fronts. (i) For backbone capability, we construct Streaming-Train-248K, a streaming dataset paired with a novel training objective for adapting VLMs to streaming interaction and understanding. (ii) For real-world deployment, we introduce Streaming Harness, a plug-and-play system that endows any VLM with three core abilities: proactive interaction (per-second response decisions), long-term memory (12-hour context retention), and real-time processing (sub-second latency). (iii) To drive continued community progress on streaming capabilities, we design Streaming-Eval, a benchmark that reflects models' capabilities across diverse in-the-wild scenarios. Extensive experiments demonstrate consistent gains from our approach across all core capabilities required for streaming video understanding. We will open-source our data, code, and benchmark to advance the community's shift from offline video understanding to deployable streaming intelligence.

Dingyu Yao, Shuhuan Gu, Qingyi Si +8

Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China

VLM EvaluationStreaming Video UnderstandingVLM AdaptationStreaming Inference
AskPDF