StreamWise: Serving Multi-Modal Generation in Real-Time at Scale

Haoran Qiu; Gohar Irfan Chaudhry; Chaojie Zhang; Íñigo Goiri; Esha Choukse; Rodrigo Fonseca; Ricardo Bianchini

cs.DCMar 6, 2026

StreamWise: Serving Multi-Modal Generation in Real-Time at Scale

Authors: Haoran Qiu, Gohar Irfan Chaudhry, Chaojie Zhang, Íñigo Goiri, Esha Choukse, Rodrigo Fonseca, Ricardo Bianchini

Organizations: Microsoft Azure Research Redmond, USA · MIT CSAIL Cambridge, USA

Abstract

Advances in multi-modal generative models are enabling new applications, from storytelling to automated media synthesis. Most current workloads generate simple outputs (e.g., image generation from a prompt) in batch mode, often requiring several seconds even for basic results. Serving real-time multi-modal workflows at scale is costly and complex, requiring efficient coordination of diverse models (each with unique resource needs) across language, audio, image, and video, all under strict latency and resource constraints. We tackle these challenges through the lens of real-time podcast video generation, integrating LLMs, text-to-speech, and video-audio generation. To meet tight SLOs, we design an adaptive, modular serving system, StreamWise, that dynamically manages quality (e.g., resolution, sharpness), model/content parallelism, and resource-aware scheduling. We leverage heterogeneous hardware to maximize responsiveness and efficiency. For example, the system can lower video resolution and allocate more resources to early scenes. We quantify the trade-offs between latency, cost, and quality. The cheapest setup generates a 10-minute podcast video on A100 GPUs in 1.4 hours (8.4x slower than the real-time) for less than $25. StreamWise enables high-quality real-time streaming with a sub-second startup delay under $45.

Figures & tables

Figure 1. Workflow for video podcast generation.

Workflow	Example input	Generated output	Characteristic	Text	Audio	Image	Video
Podcast ( Zhu et al., 2025 )	Paper	Video podcast with characters discussing input	Section 2	Low	Low	Low	High
Short ( Barua et al., 2025 )	Movie	Extract key segments and generate highlight video	Heavy LLM	High	Low	–	Low
Movie ( Meng et al., 2025 )	High-level plot	Multi-scene movie with characters, scenes,…	Long output ( IMDb, 2025 )	Low	Low	Med	High
Animated story ( Shi et al., 2025 )	Storyboard	Fully-animated story with characters and scenes	Style LoRA ( Hu et al., 2021 )	Low	Low	Low	High
Lecture ( Zhang, 2025 )	Textbook	Video with professor and supporting visuals	Static content	Low	Low	Low	Med
Slide persona ( Sun et al., 2022 )	Slide deck	Embedded presenter persona narrating slides	Low resolution	Low	Low	Low	Med

Table 1. Multi-modal generation workflows including example inputs, multi-modal outputs to generate, and characteristics.

Class	Model	Architecture	Size	Release
LLM	Llama3.2 ( AI, 2024 )	Transformer	90B	09/2024
	Gemma3 ( DeepMind, 2025 )	Transformer	27B	03/2025
A2T	Whisper ( Radford et al., 2023 )	Transformer	1.5B	11/2023
TTS	Kokoro ( Hexgrad, 2025 )	Transformer	82M	01/2025
	XTTS ( Coqui.ai, 2024 )	Transformer	400M	09/2023
	OpenAudio ( Liao et al., 2024 )	Transformer	500M	11/2024

Table 2. Generative models for podcast video generation.

Figure 2. Video generation architecture using diffusion transformers and VAE with text, image, and audio inputs.

Figure 3. Latency to generate an 81-frame (5.1-second at 16 FPs) video at

640\times{}400

(16:10 aspect ratio) using 10 diffusion steps on A100 GPUs. It includes the sensitivity to each of these parameters. The red dashed line indicates real time.

Type	Model	Year	#GPUs	Reserved	Spot	TDP
CPU	Intel EMR	2024	-	$2.33	$0.83	350 W
GPU	V100 ( NVIDIA, 2017 )	2017	8	$10.79	$3.97	8 $\times{}$ 300 W
GPU	A100 ( NVIDIA, 2020 )	2020	8	$14.42	$8.52	8 $\times{}$ 400 W
GPU	H100 ( NVIDIA, 2022 )	2022	8	$43.16	$32.22	8 $\times{}$ 700 W
GPU	H200 ( NVIDIA, 2022 )	2024	8	$45.22	$33.76	8 $\times{}$ 700 W
GPU	GB200 ( NVIDIA, 2025a )	2025	4	$57.67	$43.04	4 $\times{}$ 1200 W

Table 3. Hardware, price per hour, and power.

Figure 4. Latency sensitivity to the NVIDIA GPU generation with 4 GPUs in a single server (limited to match GB200).

Figure 5. Sensitivity to the number of H200 GPUs across servers using USP with Ulysses and Ring attention.

Figure 6. Architecture overview for StreamWise showing an example request scheduling across four model instances.

Figure 7. Deadline-aware scheduling with disaggregated components and adaptive quality on A100, H100, and H200.

Model	#GPUs	TTFF	Time	#GPUs	TTFF	Time
	Low-cost			Cost-efficient
	8 $\times$ A100			32 $\times$ 8 $\times$ A100 + 8 $\times$ 8 $\times$ H200
StreamCast	-	1.2	3.5	-	1.2	3.5
Gemma	1	6.6	31.8	8+0	1.5	9.7
Flux	1	9.8	9.8	16+0	0.9	1.0
YOLO	0.5	0.1	0.6	0.5+0	0.1	0.1

Table 4. Provisioning and generation time (in seconds) for StreamCast generating a 10-minute podcast video. The video consists of 43 shots at 1280

\times

800, using 20 diffusion steps.

Figure 8. TTFF vs. cost generating a 10-minute high-quality video podcast using various hardware configurations. The cheapest and fastest setup lies in the bottom-left corner.

Figure 9. Contribution of each technique used by StreamWise at high quality, including the Naive baseline.

Figure 10. Penalty of disabling each technique in StreamWise when generating a 10-minute podcast at high quality.

Figure 11. Applicability of systems for efficient LLM serving. Dashed lines use Spot VMs to reduce cost.

Figure 12. Quality of the greedy heuristic against the optimal allocation. Optimal takes over 100× longer than greedy.

Figure 13. TTFF vs. cost Pareto frontiers with high, medium, and low output qualities. Includes our adaptive quality policy.

Figure 14. Energy efficiency for video podcast generation.

Figure 15. Sensitivity to the multi-modal workflow. Dashed lines denote Naive , while solid lines represent StreamWise.

Figure 16. Model-wise cost breakdown serving video generation requests using our cost-efficient adaptive quality policy.

Efficient Multimodal Inference

Efficient multimodal inference reduces the compute, memory, latency, and energy needed to process image, video, audio, text, and sensor inputs.

See all 227 papers

Aero Realtime: Fully Aligned Input-Output Streams for Low-Latency Streaming Multimodal Generation
Aug 9, 2026Kaichen Zhang, Wei Huang, Keming Wu +2Efficient Multimodal Inference Multimodal Large Language Models
1The University of Hong Kong · 2LMMs-Lab · 3Tsinghua University
Ripple: Real-Time Streaming Audio-Video Generation With Cross-Modal Recurrent Memory
Jul 29, 2026Yanbo Ding, Zhizhi Guo, Quanyue Song +4Audio-Video Generation Long Video Generation
Shenzhen Key Lab of Computer Vision and Pattern Recognition, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd., China · School of Artificial Intelligence, University of Chinese Academy of Sciences +2
OmniForcing: Unleashing Real-time Joint Audio-Visual Generation
Mar 12, 2026Yaofeng Su, Yuming Li, Zeyue Xue +7Audio-Video Generation Streaming Video Generation
JD Explore Academy · Fudan University · Peking University +1

Aug 9, 2026J/K move · Enter open · S save

Aero Realtime: Fully Aligned Input-Output Streams for Low-Latency Streaming Multimodal Generation

Kaichen Zhang, Wei Huang, Keming Wu, Bo Li+1

1The University of Hong Kong · 2LMMs-Lab · 3Tsinghua University

Efficient Multimodal Inference Multimodal Large Language Models Multimodal Generation

Open paper AskPDF arXiv

Existing streaming multimodal models process observations incrementally but still follow a turn-based prefill-then-decode pattern, making them non-duplex: new observations cannot naturally enter an active generation stream. Proactive alternatives use micro-turn polling or external response gates, which fragment continuous interaction, decouple response timing from language generation, and complicate KV-cache-friendly serving. We introduce Aero Realtime, a 4B streaming multimodal model with a duplex architecture for realtime generation. Aero Realtime aligns video, audio, and textual output on a shared temporal grid, where each approximately 80-ms audio slot predicts either a lexical token or a silence token. This allows input and output to advance together, enabling one autoregressive objective to learn both when to respond and what to generate. During inference, Aero Realtime appends only the newest multimodal slot, carries forward the previous output state, and reuses the KV cache for efficient incremental execution. We further provide a complete training and serving recipe, including realtime QA construction, slot-aligned supervision, hardware-aware distributed training, and resumable inference. On four NVIDIA A6000 workstation GPUs, Aero Realtime maintains 84-ms median and 173-ms P95 processing lag over 20 minutes of a continuously streamed video, remaining within 200~ms of the source timeline. These results demonstrate the feasibility of fully aligned input-output modeling for duplex, proactive, and hardware-aligned multimodal interaction.

Abstract

Efficient Multimodal Inference

Explore similar work

Aero Realtime: Fully Aligned Input-Output Streams for Low-Latency Streaming Multimodal Generation

Ripple: Real-Time Streaming Audio-Video Generation With Cross-Modal Recurrent Memory

OmniForcing: Unleashing Real-time Joint Audio-Visual Generation