MOSS-VL Technical Report

Pengyu Wang; Chenkun Tan; Shaojun Zhou; Qirui Zhou; Yanxin Chen; Xingyang He; Huazheng Zeng; Jijun Cheng

cs.CVAug 15, 2026

MOSS-VL Technical Report

Authors: Pengyu Wang, Chenkun Tan, Shaojun Zhou, Qirui Zhou, Yanxin Chen, Xingyang He, Huazheng Zeng, Jijun Cheng,

Organizations: OpenMOSS Team

Abstract

We present MOSS-VL, an open vision-language model family that treats real-time interaction -- perceiving while it speaks -- as a first-class capability. It is co-designed across the stack: the language decoder attends to vision only through gated cross-attention, so the model can naturally see incoming frames while generating; a synthesized interaction corpus supervises when to speak, when to stay silent, and when to revise; and a staged curriculum concentrates all real-time-specific training in one light final stage over a strong offline foundation. Offline, MOSS-VL-Instruct is competitive at comparable scale and leads temporal-reasoning video sets. Across four streaming benchmarks, MOSS-VL-Realtime posts the best average on three (second on the fourth) among open-source streaming models, sweeping the three subsets that squarely test proactive behavior -- 66.0 vs. 37.5 for the best baseline on OmniMMI Proactive Alerting. With 11.3B parameters but visual tokens outside the decoded sequence, MOSS-VL widens its time-to-first-token advantage over same-backbone Qwen3-VL-8B from 2.8x to 5.1x as visual context grows. We release all five checkpoints, the training curriculum, and the real-time inference code at https://github.com/OpenMOSS/MOSS-VL.

Figures & tables

			Capability axes
Level	Regime	Defining behavior	Stream. input	Silence	Multi- resp.	Perceive while gen.
L1	offline	Watches the full video first, then answers questions about it (multi-turn allowed).	✗	–	✗	✗
L2	streaming	Video arrives continuously; the user may ask at any moment and the model answers immediately—but it is blind while replying.	✓	✗	✗	✗
L3	streaming	Adds waiting : if the answer is not yet determinable, the model stays silent until the key evidence appears, then answers.	✓	✓	✗	✗
L4	streaming	Adds persistent queries : one question stays resident and the model re-answers as the scene evolves (still blind during each individual reply).	✓	✓	✓	✗
L5	real-time	Adds perception during generation : the model keeps watching while it speaks, revising or interrupting its own reply the moment the evidence changes.	✓	✓	✓	✓

Table 1: Levels of video understanding, from offline to real-time. Each level lights up one additional capability axis; the dividing line between the streaming regime (L2–L4) and real-time (L5) is whether the model keeps perceiving while it generates—L2–L4 models are blind during a reply, an L5 model is not. MOSS-VL-Realtime achieves L5 behavior, demonstrated through live demos and the released real-time inference code, and is quantitatively validated at levels L2–L4 on four streaming benchmarks; a dedicated benchmark for L5 behavior remains an open problem.

Language decoder (initialized from Qwen3-8B)
Layers	48 = 36 self-attention + 12 gated cross-attention
Cross-attention placement	every 4th layer (indices $2,6,\ldots,46$ )
Hidden size / FFN size	4096 / 12288
Attention heads	32 query / 8 key–value (GQA), head dim. 128
Position encoding	XRoPE: interleaved 3-axis RoPE $(t,h,w)$ ,
	sections $(24,20,20)$ , base $5\times 10^{6}$

Table 2: MOSS-VL configuration, identical across all released checkpoints. Video-input ranges reflect the native training setup; the released processor defaults to 1 fps and 256 frames, and our evaluations keep 1 fps with the cap raised to 768 frames (§ 6 ).

Checkpoint	Initialized from	Training	Evaluated in
MOSS-VL-Base-0708	—	pre-training (§ 3 )	—
MOSS-VL-Instruct-0708	Base-0708	SFT (§ 4 )	offline ( Table 5 )
MOSS-VL-Realtime	Instruct-0708	Realtime-SFT (§ 4 )	streaming ( Table 6 )
MOSS-VL-Base-0408	—	pre-training	—
MOSS-VL-Instruct-0408	Base-0408	SFT	—

Table 3: Released MOSS-VL checkpoints. The 0708 run is the subject of this report: MOSS-VL-Base is the pre-trained model, MOSS-VL-Instruct is its instruction-tuned successor, and MOSS-VL-Realtime continues training from MOSS-VL-Instruct with Realtime-SFT. The 0408 checkpoints are an earlier, independently trained run of the same architecture, released alongside for research continuity. All five are available under the OpenMOSS-Team organization on HuggingFace.

Stage	Tokens	Samples	Max seq.	Trainable	Peak LR
Pre-training
1 Vision–language alignment	150.3B	219.4M	8K	Projection + cross-attn	$2{\times}10^{-4}$
2 Large-scale multimodal	203.0B	139.3M	64K	Full model	$5{\times}10^{-5}$
3 High-quality multimodal	459.0B	14.7M	128K	Full model	$1{\times}10^{-5}$
4 Annealing & long-context	450.1B	9.3M	256K	Full model	$1{\times}10^{-5}$
Post-training (§ 4 )

Table 4: The MOSS-VL training curriculum: four pre-training stages followed by SFT and Realtime-SFT (§ 4 ). The data of each stage is described in the corresponding subsection. Token counts are the tokens fed to the language model, with vision tokens counted after the

2{\times}2

token compression of the projection module (§ 2 ); sequence lengths are in tokens.

Benchmark	MOSS-VL (ours)	Qwen3-VL 8B	Qwen2.5-VL 7B	LLaVA-OV-2 8B	Gemma-4 12B-IT
Multimodal perception
MMBench-EN (v1.1) [ 22 ]	88.1	84.8	83.2	85.8	82.7
MMStar [ 6 ]	66.0	70.9	63.9	64.8	74.9
RealWorldQA	68.0	71.5	68.5	69.7	65.6
MME-RealWorld [ 55 ]	66.3	–	57.4	–	46.9
BLINK [ 12 ]	78.0	69.1	56.4	63.5	65.6

Table 5: Offline results: MOSS-VL-Instruct (0708 release) against open-source models of comparable scale, blocked by capability domain. Bold = best, underline = second best, “–” = not reported; LLaVA-OV-2 = LLaVA-OneVision-2-8B. Evaluation protocol and baseline provenance are spelled out in § 6.1 .

OVO-Bench [ 20 ]
Subset	MOSS-VL-Realtime	AURA	JoyAI-VL-Interaction	VideoChat3-4B	ViSpeak-7B
FAR	62.1	55.8	–	–	54.2
BT	72.6	60.4	–	–	57.5
RTVP	75.9	79.8	–	–	66.3
Avg	70.2	65.3	59.2	57.8	59.3
OmniMMI [ 44 ]

Table 6: Streaming benchmark results: MOSS-VL-Realtime against open-source streaming baselines at subset level. Bold = best, underline = second best; “–” = subset not reported in that model’s official report. Subset abbreviations, each benchmark’s Avg convention, and data provenance are spelled out in § 6.2 .

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Turn	English translation
(a) Conditional alert
User ( $t{=}0$ )	Watch in real time and judge whether the cat touches the carrot; output “Great!” whenever it does, and stay silent otherwise.
Model	“Great!”, emitted at each of the four contacts; every other frame yields the silence token.
(b) Real-time commentary
User ( $t{=}0$ )	Commentate on the video live.
Model	Portugal win a crucial free-kick chance; Ronaldo stands over the ball…

Appendix

Table 7: English translations (ours) of the two live-demo sessions in Figure 5 . The figure carries the Chinese excerpts; timestamps and the placement of each output on the stream are read off the figure’s timeline.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 736 papers

MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention
Jun 1, 2026Pengyu Wang, Chenkun Tan, Shaojun Zhou +18Streaming Video Understanding Efficient VLM Inference
Fudan University · Shanghai Innovation Institute
JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence
Jun 10, 2026Dingyu Yao, Junhao Zhou, Chenxu Yang +17Human-AI Interaction Vision-Language Agents
StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling
Jul 7, 2025Meng Wei, Chenyang Wan, Xiqian Yu +9Efficient VLM Inference Efficient VLA Models
1Shanghai AI Lab · 2The University of Hong Kong · 3Zhejiang University +1

Jun 1, 2026J/K move · Enter open · S save

MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention

Pengyu Wang, Chenkun Tan, Shaojun Zhou, Wei Huang+17

Fudan University · Shanghai Innovation Institute

Streaming Video Understanding Efficient VLM Inference Video-Language Models Cross-Modal Attention

Open paper AskPDF arXiv

Video understanding is shifting from the offline paradigm -- taking a fully recorded video as input and producing a single answer after it ends -- toward real-time interaction, in which the model perceives new frames while still replying, revises its answer as new evidence appears, and remains silent when there is nothing to say. We present MOSS-Video-Preview to validate this paradigm. Our central claim is that perception must not be blocked by generation; its natural realization is a two-channel architecture. We argue that a cross-attention backbone is better suited to real-time vision-language fusion than the prevailing decoder-only design: visual features enter through a side channel rather than joining the autoregressive sequence, so perception and generation run on separate, non-blocking pathways -- reducing the frequency of visual processing and exposing a clean channel-wise interface for independent compression. We complement this with a data synthesis pipeline that converts dense captions into real-time understanding QA whose answers are revised to match what the model has perceived so far, and we specialize an offline model on these data to elicit real-time behavior. Our model trails the strong Qwen2.5-VL-7B baseline overall -- a gap we attribute primarily to data and scale rather than the architecture -- yet attains competitive offline video and multimodal understanding, remains robust on the spatial and fine-grained temporal reasoning central to real-time use, and acquires behaviors that offline models lack: continuous perception, answer revision, and timely silence. On a single H200 with 256 frames per video, it achieves about a 5x speedup in time to first token and 2.7x higher decoding throughput, with negligible degradation in offline ability. Our study of paradigm, architecture, and data outlines a viable path toward real-time video understanding.

Abstract

Vision-Language Models

Explore similar work

MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention

JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence

StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling