From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Models

Junlong Tong; Zilong Wang; YuJie Ren; Peiran Yin; Hao Wu; Wei Zhang; Xiaoyu Shen

cs.CLMar 4, 2026

From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Models

Authors: Junlong Tong, Zilong Wang, YuJie Ren, Peiran Yin, Hao Wu, Wei Zhang, Xiaoyu Shen

Organizations: Shanghai Jiao Tong University · Institute of Digital Twin, Eastern Institute of Technology, Ningbo

Abstract

Standard Large Language Models (LLMs) are predominantly designed for static inference with pre-defined inputs, which limits their applicability in dynamic, real-time scenarios. To address this gap, the streaming LLM paradigm has emerged. However, existing definitions of streaming LLMs remain fragmented, conflating streaming generation, streaming inputs, and interactive streaming architectures, while a systematic taxonomy is still lacking. This paper provides a comprehensive overview and analysis of streaming LLMs. First, we establish a unified definition of streaming LLMs based on data flow and dynamic interaction to clarify existing ambiguities. Building on this definition, we propose a systematic taxonomy of current streaming LLMs and conduct an in-depth discussion on their underlying methodologies. Furthermore, we explore the applications of streaming LLMs in real-world scenarios and outline promising research directions to support ongoing advances in streaming intelligence. We maintain a continuously updated repository of relevant papers at https://github.com/EIT-NLP/Awesome-Streaming-LLMs.

Figures & tables

Figure 1: Illustration of three types of streaming large language models (LLMs). ( Left ) Output-streaming LLM performs streaming generation after static reading. ( Middle ) Sequential-streaming LLM performs streaming generation after streaming reading. ( Right ) Concurrent-streaming LLM performs streaming generation while streaming reading.

Figure 2: Overview of streaming LLM paradigms and their key challenges. The figure contrasts Output-streaming, Sequential-streaming, and Concurrent-streaming LLMs, highlighting their core goals and corresponding research components. Concurrent-streaming builds on the first two and adds extra challenges in real-time streaming architecture adaptation and interaction policy learning.

Figure 4: Illustration of structural conflicts when adapting batch-oriented LLMs (left) to concurrent streaming (right), where indicates the token generation direction, denotes attention dependencies, blocks represent the input, and blocks represent the output. (1) Attention contention : Ambiguous causal dependency between the newly inserted streaming input and historical outputs. (2) Position-ID conflict : The new streaming input and generated output compete for the identical position ID.

Table 1: Comparison of concurrent-streaming architecture adaptation methods from the perspectives of attention (Attn.) and position (Pos.). indicates the token generation direction, while denotes attention dependencies. blocks represent the input stream, and blocks represent the output stream.

p

indicates the corresponding position ID.

Figure 5: Illustration of interaction decision in concurrent streaming LLMs, where the model learns to dynamically schedule reading inputs and emitting outputs.

Streaming-In	Bound	Inc.	Cxt.	Example methods
Text	Memory	-	✓	StreamingDialogue Macháček et al. (2023)
Audio	Causal, Memory	✓	✓	WhisperStreaming Li et al. (2024b)
Video	Memory	-	✓	Timechat-online Yao et al. (2025)

Table 2: Summary of sequential streaming tasks. Incremental encoding ( Inc ) and context management ( Ctx ) are the key technical dimensions. The checkmark ( ✓ ) indicates the scope covered by existing research.

Task type	Level	Modality		Paradigm				Interaction policy			Example methods
Task type	Level	In	Out	R.	C.	I.	G.	Rule	SFT	RL	Example methods
Translation	$\mathcal{X}\to\mathcal{Y}$	T/S	T/S	✓	✓	✓	✓	✓	✓	✓	Seed LiveInterpret 2.0 Cheng et al. (2025b)
Detection	$\mathcal{X}\to\mathcal{Y}$	T/S/V	T	-	-	✓	-	-	✓	-	FineHarm Li et al. (2025h)
ASR	$\mathcal{X}\to\mathcal{Y}$	S	T	-	-	✓	✓	✓	✓	-	ReaLLM Seide et al. (2024) , Llama-omni Fang et al. (2024)
TTS	$\mathcal{X}\to\mathcal{Y}$	T	S	-	✓	-	✓	✓	✓	-	Cosyvoice Du et al. (2024) , DSM Zeghidour et al. (2025)
QA	$\mathcal{X}\to\mathcal{Y}$	T/S/V	T/S	-	✓	✓	-	-	✓	✓	Qwen3-omni Xu et al. (2025c) , VideoLLM-online Chen et al. (2024a)

Table 3: Summary of concurrent streaming tasks and representative methods. Tasks are categorized by processing depth ( Level ), where

\mathcal{X}\to\mathcal{Y}

denotes direct mapping (perception) and

\mathcal{X}\to\mathcal{Z}\to\mathcal{Y}

denotes intermediate processing with a latent state

\mathcal{Z}

(cognition). Modality: text ( T ), speech ( S ), vision ( V ). Streaming Paradigm: re-encoding ( R ), Concatenated ( C ), Interactive ( I ), Group ( G ). Interaction Policy: Rule-based ( Rule ), SFT-based ( SFT ), and RL-based ( RL ). The checkmark ( ✓ ) indicates the scope covered by existing research.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Typical Surveys	Primary Focus	Typical Technologies Covered	Differentiation in This Survey
Survey Category: Efficient LLMs
Li et al. (2024a)
Dantas et al. (2025)
Cheng et al. (2025a)
Wu et al. (2026b)	Compression/adaptation and memory bottlenecks.	1) Compression: quantization, pruning, distillation, low-rank; and 2) KV-cache management: selection / eviction, cache compression, offloading, sliding-window / hierarchical cache.	Prior surveys treat compression and KV-cache optimization as separate threads; we unify them under streaming interaction, highlighting online constraints and dynamic runtime budgeting.
Survey Category: Multimodal LLMs

Appendix

Table 1: Comparison between this survey and existing related surveys. We highlight the unique positioning of our work in the context of streaming interaction.

Streaming Generation
Mechanism			Modality-Out	Methods
Token	Block	Refinement
✓	-	-	T	GPT OpenAI (2023) , Gemini Team et al. (2023) , Qwen3 Yang et al. (2025a) , DeepSeek-V3 DeepSeek-AI et al. (2024) , InternVL Chen et al. (2024c) , ChatGLM GLM et al. (2024) , Gemma Gemma Team (2024)
✓	-	-	S	AudioLM Borsos et al. (2023a) , SpeechGPT Zhang et al. (2023a) , AudioPaLM Rubenstein et al. (2023) , FireRedTTS Guo et al. (2024a) , Moshi Défossez et al. (2024) , Llama-omni2 Fang et al. (2025) , Qwen3-Omni Xu et al. (2025c) , StyLLE Hao et al. () , Llmvox Shikhar et al. (2025) , SpeakStream Bai et al. (2025)
✓	-	-	V	DALLE Ramesh and others (2021) , VideoPoet Kondratyuk et al. (2024) , Chameleon Team (2024) , Emu3 Wang et al. (2024e) , Anole Chern et al. (2024) , Lumina-mGPT2.0 Xin et al. (2025) , Infinity Han et al. (2025)
-	✓	-	T	SAT Wang et al. (2018) , SoT Ning et al. (2023) , CtrlDiff Huang and Tang (2025) , PredSent Hwang et al. (2025) , Falcon Gao et al. (2025) , SSD-LM Han et al. (2023) , WeDLM Liu et al. (2025a) , Next-Block Tian et al. (2025) , Block Diffusion Arriola et al. (2025)

Appendix

Table 2: Summary of additional literature on output-streaming LLMs, complementing the discussion in Sec. 3 .

Incremental Encoding
Type		Modality-In	Methods
Fragmented Encoding	Atomic Encoding
✓	-	T	SimulMT Wang et al. (2024c) , Moshi Défossez et al. (2024) , Codec Ye et al. (2025) , dmel Bai et al. (2024) , Lightweight Audio Segmentation Frohmann et al. (2024) , Semantic VAD Shi et al. (2023)
✓	-	S	Whisper-Streaming Macháček et al. (2023) , SimulST Zhang and Feng (2023) , CTC Graves (2012) , Speechtokenizer Xin et al. (2024) , Moshi Défossez et al. (2024) , Codec Ye et al. (2025) , dmel Bai et al. (2024) , Lightweight Audio Segmentation Frohmann et al. (2024) , Semantic VAD Shi et al. (2023)
✓	-	V	S-ViT Zhao et al. (2023)
-	✓	T	SaT Frohmann et al. (2024) , SegFree Iranzo-Sánchez et al. (2024) , WtP Minixhofer et al. (2023) , subword regularization Kudo (2018) , SentencePiece Kudo and Richardson (2018) ,

Appendix

Table 3: Summary of additional literature on sequential-streaming LLMs, complementing the discussion in Sec. 4 .

Streaming Paradigm
Paradigm				Modality		Methods
R.	C.	I.	G.	In	Out
✓	-	-	-	T	T	Simul-LLM Agostinelli et al. (2024) , SiLLM Guo et al. (2024c) , TransLLaMA Koshkin et al. (2024b) , CAST Koshkin et al. (2024a) , RALCP Wang et al. (2024c)
✓	-	-	-	S	T	CAST Koshkin et al. (2024a) , TransLLaMA Koshkin et al. (2024b)
-	✓	-	-	T	S	LLMVoX Shikhar et al. (2025) , Mini-Omni Xie and Wu (2024)
-	✓	-	-	S	S	Mini-Omni Xie and Wu (2024)

Appendix

Table 4: Summary of additional literature on concurrent-streaming LLMs, complementing the discussion in Sec. 5 .

LLM Inference

Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.

See all 276 papers

ProactiveLLM: Learning Active Interaction for Streaming Large Language Models
May 30, 2026Junlong Tong, Yao Zhang, Anhao Zhao +3LLM Inference LLM Inference Acceleration
Eastern Institute of Technology, Ningbo · Shanghai Jiao Tong University · Hong Kong Polytechnic University +1
Which LLM to pick? Online Active Model Selection for Large Language Models
Oct 1, 2026Alessandro Turrin, Patrik Okanovic, Torsten Hoefler +1Model Selection LLM Evaluation
TU Delft · ETH Zurich
Harnessing Streaming Video in the Wild
Jun 7, 2026Dingyu Yao, Shuhuan Gu, Qingyi Si +8VLM Evaluation Streaming Video Understanding
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China

May 30, 2026J/K move · Enter open · S save

ProactiveLLM: Learning Active Interaction for Streaming Large Language Models

Junlong Tong, Yao Zhang, Anhao Zhao, Yingqi Fan+2

Eastern Institute of Technology, Ningbo · Shanghai Jiao Tong University · Hong Kong Polytechnic University+1

LLM Inference LLM Inference Acceleration

Open paper AskPDF arXiv

Standard Large Language Models (LLMs) follow a read-then-generate paradigm, causing unnecessary latency and computation. Streaming LLMs alleviate this issue by generating while receiving inputs, but still struggle to decide when to interact with the stream. Existing methods either hard-code interaction timing or rely on costly external alignment signals, such as timing labels, reasoning trajectories, or stronger teachers. In this paper, we propose ProactiveLLM, which achieves active interaction by leveraging the model's endogenous states to guide interaction decisions. The model first learns to perceive semantic sufficiency from partial inputs through two complementary training mechanisms: mask-based streaming modeling and synchronized privileged self-distillation (SPSD). The former applies monotonic random masking to the input during training, simulating progressively revealed streaming inputs and enabling the model to learn local semantic dependencies from partial-input views. The latter aligns the partial-context student view with a full-context teacher view generated by the same evolving model, allowing privileged full-context evidence to guide the student's understanding under incomplete observations. Together, these mechanisms induce endogenous sufficiency cues without requiring external teachers or annotations, providing a versatile foundation for the plug-and-play integration of diverse decision heads. Extensive evaluation across text and speech streaming tasks confirms that ProactiveLLM significantly reduces interaction latency while maintaining quality, validating its capacity for dynamic and active interaction. Code is publicly available at https://github.com/EIT-NLP/StreamingLLM/tree/main/ProactiveLLM.

Abstract

LLM Inference

Explore similar work

ProactiveLLM: Learning Active Interaction for Streaming Large Language Models

Which LLM to pick? Online Active Model Selection for Large Language Models

Harnessing Streaming Video in the Wild