OmniStream: Mastering Perception, Reconstruction and Action in Continuous Streams

Yibin Yan; Jilan Xu; Shangzhe Di; Haoning Wu; Weidi Xie

cs.CVMar 12, 2026

OmniStream: Mastering Perception, Reconstruction and Action in Continuous Streams

Authors: Yibin Yan, Jilan Xu, Shangzhe Di, Haoning Wu, Weidi Xie

Organizations: School of Artificial Intelligence, SJTU · Shanghai Innovation Institute · VGG, Oxford

Abstract

Modern visual agents require representations that are general, causal, and physically structured to operate in real-time streaming environments. However, current vision foundation models remain fragmented, specializing narrowly in image semantic perception, offline temporal modeling, or spatial geometry. This paper introduces OmniStream, a unified streaming visual backbone that effectively perceives, reconstructs, and acts from diverse visual inputs. By incorporating causal spatiotemporal attention and 3D rotary positional embeddings (3D-RoPE), our model supports efficient, frame-by-frame online processing of video streams via a persistent KV-cache. We pre-train OmniStream using a synergistic multi-task framework coupling static and temporal representation learning, streaming geometric reconstruction, and vision-language alignment on 29 datasets. Extensive evaluations show that, even with a strictly frozen backbone, OmniStream achieves consistently competitive performance with specialized experts across image and video probing, streaming geometric reconstruction, complex video and spatial reasoning, as well as robotic manipulation (unseen at training). Rather than pursuing benchmark-specific dominance, our work demonstrates the viability of training a single, versatile vision backbone that generalizes across semantic, spatial, and temporal reasoning, i.e., a more meaningful step toward general-purpose visual understanding for interactive and embodied agents.

Figures & tables

Figure 1: Left: OmniStream supports a wide spectrum of tasks, including 2D/3D perception, vision-language understanding, and embodied robotic manipulation. Right: the frozen features of our single backbone achieve highly competitive or superior performance compared to leading domain-specific experts.

Figure 2: Overall framework of OmniStream. Equipped with 3D-RoPE and causal spatiotemporal attention, our unified backbone is trained via a multi-task framework that couples static and temporal representation learning, streaming geometric reconstruction, and vision-language alignment.

Task	Pre-training Dataset	Frames
Image SSL	DataComp-100M Gadre et al. (2023) , ImageNet-21K Ridnik et al. (2021)	$\sim$ 113M
Video SSL	Kinetics Carreira and Zisserman (2017) , SSv2 Goyal et al. (2017) , PE-Videos Bolya et al. (2025)	$\sim$ 20M
3D/4D Scenes	Co3Dv2 Reizenstein et al. (2021) , WildRGBD Xia et al. (2024) , BlendedMVS Yao et al. (2020) , MegaDepth Li and Snavely (2018) , MVS-Synth Huang et al. (2018) , PointOdyssey Zheng et al. (2023) , HyperSim Roberts et al. (2021) , TartanAir Wang et al. (2020) , DL3DV Ling et al. (2024) , ScanNet Dai et al. (2017) , ScanNet++ Yeshwanth et al. (2023) , ArkitScenes Baruch et al. (2021) , Virtual Kitti 2 Cabon et al. (2020) , MapFree Arnold et al. (2022) , Dynamic Replica Karaev et al. (2023) , Spring Mehl et al. (2023) , OmniWorld-Games Zhou et al. (2025) , Aria Synthetic Environments Pan et al. (2023) , Waymo-Open Sun et al. (2020)	$\sim$ 18M
Captioning	GRIT Peng et al. (2023) ; Byeon et al. (2022) , RefCOCO Series Kazemzadeh et al. (2014) , Blip3-OCR Xue et al. (2024) , SA1B-Caption Chen et al. (2024)	$\sim$ 50M
Total	-	$\sim$ 200M

Table 1: Overview of pre-training datasets. We leverage a diverse collection of images, videos, and 3D/4D data.

	Benchmark	Ours	DINOv3-L	V-JEPA2-L	CUT3R	LLaVA-Video	OpenVLA
Image	ImageNet ${}_{\text{cls}}$ $\uparrow$	84.7	86.7	-	-	-	-
	NYUv2 ${}_{\text{monodepth}}$ $\downarrow$	0.377	0.377	-	-	-	-
	ADE20K ${}_{\text{seg}}$ $\uparrow$	49.1	51.5	-	-	-	-
Video	SSv2 ${}_{\text{act}}\uparrow$	68.5	54.0	73.7	-	-	-
	K400 ${}_{\text{act}}\uparrow$	85.7	83.6	85.1	-	-	-
	DAVIS’17 ${}_{\text{vos}}\uparrow$	71.6	73.2	44.2	-	-	-

Table 2: Holistic evaluation. We compare OmniStream across 5 domains. Subscripts denote the specific task ( i.e. , cls: classification, monodepth: monocular depth estimation, seg: semantic segmentation, act: video action recognition, vos: video object segmentation, videodepth: video depth estimation, pose: camera pose estimation, vqa: visual question answering, mani: robotic manipulation). The evaluated metrics are ACC@1 for cls and act, RMSE for monodepth, mIOU for seg,

\mathcal{J}\&\mathcal{F}

for vos, absRel for videodepth, ATE for pose, Accuracy (Acc.) for vqa, Average Length (Avg. Len) for CALVIN, and Success Rate (SR %) for Simpler-Bridge. Here, “-” indicates that a specialized baseline is not natively applicable to or lacks the capability for the given task.

		Sintel		BONN		KITTI
Method	Param.	Abs Rel $\downarrow$	$\delta<1.25$ $\uparrow$	Abs Rel $\downarrow$	$\delta<1.25$ $\uparrow$	Abs Rel $\downarrow$	$\delta<1.25$ $\uparrow$
Span3R Wang and Agapito (2025)	600M	0.622	42.6	0.144	81.3	0.198	73.7
Cut3R Wang* et al. (2025)	600M	0.421	47.9	0.078	93.7	0.118	88.1
Point3R Wu et al. (2025b)	600M	0.481	44.8	0.066	95.8	0.093	93.5
Ours	400M	0.314	58.3	0.072	96.7	0.136	81.8

Table 3: Comparison on online video depth estimation.

		Sintel			TUM-dynamics			ScanNet
Method	Param.	ATE $\downarrow$	RPE ${}_{T}\downarrow$	RPE ${}_{R}\downarrow$	ATE $\downarrow$	RPE ${}_{T}\downarrow$	RPE ${}_{R}\downarrow$	ATE $\downarrow$	RPE ${}_{T}\downarrow$	RPE ${}_{R}\downarrow$
Span3R Wang and Agapito (2025)	600M	0.329	0.110	4.471	0.056	0.021	0.591	0.096	0.023	0.661
Cut3R Wang* et al. (2025)	600M	0.213	0.066	0.621	0.046	0.015	0.473	0.099	0.022	0.600
Point3R Wu et al. (2025b)	600M	0.442	0.154	1.897	0.058	0.031	0.758	0.097	0.035	2.791
Ours	400M	0.227	0.051	0.695	0.049	0.012	0.601	0.076	0.017	0.659

Table 4: Comparison on online video camera pose estimation.

Methods	Avg.	Obj. Count	Abs. Dist.	Obj. Size	Room Size	Rel. Dist	Rel. Dir.	Route Plan	Appr. Order
GPT-4o Hurst et al. (2024)	34.0	46.2	5.3	43.8	38.2	37.0	41.3	31.5	28.5
Gemini-1.5-Pro Team et al. (2024)	45.4	56.2	30.9	64.1	43.6	51.3	46.3	36.0	34.6
LLaVA-OneVision-7B Li et al. (2024a)	32.4	47.7	20.2	47.4	12.3	42.5	35.2	29.4	24.4
LLaVA-Video-7B Zhang et al. (2024b)	35.6	48.5	14.0	47.8	24.2	43.5	42.4	34.0	30.6
Qwen2.5-VL-7B Bai et al. (2025b)	32.7	34.5	19.4	47.6	40.8	32.8	24.5	32.5	29.4
SpaceR-7B Ouyang et al. (2025)	43.5	61.9	28.6	60.9	35.2	38.2	46.0	31.4	45.6

Table 5: Comparison on VSI-Bench.

Panel of Table 6 — Table 6: Comparison on CALVIN ABC-D. Entries marked with * are expert VLAs modified in VLM4VLA settings.

Streaming Video Understanding

Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.

See all 132 papers

StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering
May 25, 2026Ming Xie, Zizheng Huang, Xudong Tan +6Audio-Visual Understanding Streaming Video Understanding
Shanghai Innovation Institute · Fudan University · Nanjing University +2
Native Active Perception as Reasoning for Omni-Modal Understanding
Jun 17, 2026Zhenghao Xing, Ruiyang Xu, Yuxuan Wang +8Audio-Visual Understanding Active Perception
1The Chinese University of Hong Kong · †Work done during an internship at Qwen Team, Alibaba Group. · 2Shanghai Jiao Tong University +2
Foresight: planning future perception in streaming VLMs without retraining
Oct 2, 2026Ashok Prasad Neupane, Dipan Bartaula, Ankit Belbase +5Vision-Language Models Streaming Inference
Independent Researcher · NAAMII, Nepal · University College London, UK +2

May 25, 2026J/K move · Enter open · S save

StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering

Ming Xie, Zizheng Huang, Xudong Tan, Chao Wang+5

Shanghai Innovation Institute · Fudan University · Nanjing University+2

Audio-Visual Understanding Streaming Video Understanding Multimodal Memory Multimodal QA

Open paper AskPDF arXiv

While streaming omni-video understanding demands continuous perception and proactive, real-time interaction, this crucial area remains largely under-explored. Current omni-modal methods are inherently designed for offline settings, limiting their applicability in streaming scenarios due to two fundamental flaws. First, they lack robust mechanisms to manage continuously growing audio-visual context over long horizons and cannot autonomously initiate responses at opportune moments. Second, existing benchmarks are predominantly confined to offline, single-turn question answering, failing to capture continuous, multi-turn streaming interactions. To bridge these gaps, we propose StreamOV, a novel Streaming Omni-Video understanding framework for efficient online audio-visual reasoning with bounded memory and proactive response triggering. Specifically, StreamOV introduces a multimodal evidence-guided long-short term memory that condenses historical audio-visual context into compact informative evidence under a fixed budget. It further employs a hidden-state-driven trigger to decide when to respond, avoiding explicit silence-token generation and external routers. We also curate SOVBench, the first comprehensive benchmark for online, multi-turn omni-modal evaluation. Extensive experiments show that StreamOV achieves state-of-the-art performance across diverse streaming and omni-video benchmarks, demonstrating its effectiveness for both online and offline video understanding.

Abstract

Streaming Video Understanding

Explore similar work

StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering

Native Active Perception as Reasoning for Omni-Modal Understanding

Foresight: planning future perception in streaming VLMs without retraining