A Low-Latency Interactive System for Real-Time Video Understanding Based on VLMs

Punan Dai; Jun Xu; Bingcong Lu; Zhengxue Cheng; Hongwei Hu; Ronghua Wu; Li Song

cs.MMSep 12, 2026

A Low-Latency Interactive System for Real-Time Video Understanding Based on VLMs

Authors: Punan Dai, Jun Xu, Bingcong Lu, Zhengxue Cheng, Hongwei Hu, Ronghua Wu, Li Song

Organizations: School of Information Science and Electronic Engineering, Shanghai Jiao Tong University, China · Ant Group, Shanghai, China

Abstract

Vision-language models are extending video understanding from offline clip analysis to continuous interactive streaming, but most research still emphasizes model capability rather than deployable low-latency interaction. This paper presents a unified edge-cloud system for real-time video VLM applications. Lightweight phone, smart glasses, PC, and pseudo-replay clients publish video and speech to a server runtime that provides shared ASR/TTS, session orchestration, backend adaptation, response delivery, and archive-backed measurement. The system integrates six representative video VLM backends with streaming or interaction-oriented capabilities and evaluates them across backend runtime, media transport, client-observed latency, and interaction behavior. With suitable backend selection and the WebRTC path, the tested system reaches approximately 0.9 to 1.0 s to first VLM text and 1.3 to 1.5 s to first non-silent TTS audio, while exposing backend adaptation costs and differences in real-time interaction behavior.

Streaming Video Understanding

Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.

See all 132 papers

Harnessing Streaming Video in the Wild
Jun 7, 2026Dingyu Yao, Shuhuan Gu, Qingyi Si +8VLM Evaluation Streaming Video Understanding
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
RIVER: A Real-Time Interaction Benchmark for Video LLMs
Mar 4, 2026Yansong Shi, Qingsong Zhao, Tianxiang Jiang +3Video-Language Model Evaluation Streaming Video Understanding
School of Information Science And Technology, University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · College of Computer Science and Artificial Intelligence, Fudan University +1
Don't Pause: Streaming Video-Language Synchrony for Online Video Understanding
Jun 5, 2026Zhenyu Yang, Kairui Zhang, Shengsheng Qian +2Streaming Video Understanding Language Model Decoding

Jun 7, 2026J/K move · Enter open · S save

Harnessing Streaming Video in the Wild

Dingyu Yao, Shuhuan Gu, Qingyi Si, Junhao Zhou+7

Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China

VLM Evaluation Streaming Video Understanding VLM Adaptation Streaming Inference

Open paper AskPDF arXiv

Vision-Language Models (VLMs) are increasingly required to process unbounded video streams in applications such as video-call assistants, live commentary, and embodied robots. An ideal streaming system should support proactive interaction, long-horizon memory, and real-time processing, while resting on a VLM backbone capable of handling diverse in-the-wild streaming tasks. However, existing VLMs excel at offline video understanding but fall short in streaming capabilities and lack dedicated infrastructure for streaming deployment. We address this gap on three fronts. (i) For backbone capability, we construct Streaming-Train-248K, a streaming dataset paired with a novel training objective for adapting VLMs to streaming interaction and understanding. (ii) For real-world deployment, we introduce Streaming Harness, a plug-and-play system that endows any VLM with three core abilities: proactive interaction (per-second response decisions), long-term memory (12-hour context retention), and real-time processing (sub-second latency). (iii) To drive continued community progress on streaming capabilities, we design Streaming-Eval, a benchmark that reflects models' capabilities across diverse in-the-wild scenarios. Extensive experiments demonstrate consistent gains from our approach across all core capabilities required for streaming video understanding. We will open-source our data, code, and benchmark to advance the community's shift from offline video understanding to deployable streaming intelligence.

Abstract

Streaming Video Understanding

Explore similar work

Harnessing Streaming Video in the Wild

RIVER: A Real-Time Interaction Benchmark for Video LLMs

Don't Pause: Streaming Video-Language Synchrony for Online Video Understanding