We present MOSS-VL, an open vision-language model family that treats real-time interaction -- perceiving while it speaks -- as a first-class capability. It is co-designed across the stack: the language decoder attends to vision only through gated cross-attention, so the model can naturally see incoming frames while generating; a synthesized interaction corpus supervises when to speak, when to stay silent, and when to revise; and a staged curriculum concentrates all real-time-specific training in one light final stage over a strong offline foundation. Offline, MOSS-VL-Instruct is competitive at comparable scale and leads temporal-reasoning video sets. Across four streaming benchmarks, MOSS-VL-Realtime posts the best average on three (second on the fourth) among open-source streaming models, sweeping the three subsets that squarely test proactive behavior -- 66.0 vs. 37.5 for the best baseline on OmniMMI Proactive Alerting. With 11.3B parameters but visual tokens outside the decoded sequence, MOSS-VL widens its time-to-first-token advantage over same-backbone Qwen3-VL-8B from 2.8x to 5.1x as visual context grows. We release all five checkpoints, the training curriculum, and the real-time inference code at https://github.com/OpenMOSS/MOSS-VL.
Figures & tables
Capability axes
Level
Regime
Defining behavior
Stream. input
Silence
Multi- resp.
Perceive while gen.
L1
offline
Watches the full video first, then answers questions about it (multi-turn allowed).
✗
–
✗
✗
L2
streaming
Video arrives continuously; the user may ask at any moment and the model answers immediately—but it is blind while replying.
✓
✗
✗
✗
L3
streaming
Adds waiting : if the answer is not yet determinable, the model stays silent until the key evidence appears, then answers.
✓
✓
✗
✗
L4
streaming
Adds persistent queries : one question stays resident and the model re-answers as the scene evolves (still blind during each individual reply).
✓
✓
✓
✗
L5
real-time
Adds perception during generation : the model keeps watching while it speaks, revising or interrupting its own reply the moment the evidence changes.
✓
✓
✓
✓
Table 1: Levels of video understanding, from offline to real-time. Each level lights up one additional capability axis; the dividing line between the streaming regime (L2–L4) and real-time (L5) is whether the model keeps perceiving while it generates—L2–L4 models are blind during a reply, an L5 model is not. MOSS-VL-Realtime achieves L5 behavior, demonstrated through live demos and the released real-time inference code, and is quantitatively validated at levels L2–L4 on four streaming benchmarks; a dedicated benchmark for L5 behavior remains an open problem.
Language decoder (initialized from Qwen3-8B)
Layers
48 = 36 self-attention + 12 gated cross-attention
Cross-attention placement
every 4th layer (indices 2,6,…,46 )
Hidden size / FFN size
4096 / 12288
Attention heads
32 query / 8 key–value (GQA), head dim. 128
Position encoding
XRoPE: interleaved 3-axis RoPE (t,h,w) ,
sections (24,20,20) , base 5×106
Table 2: MOSS-VL configuration, identical across all released checkpoints. Video-input ranges reflect the native training setup; the released processor defaults to 1 fps and 256 frames, and our evaluations keep 1 fps with the cap raised to 768 frames (§ 6 ).
Checkpoint
Initialized from
Training
Evaluated in
MOSS-VL-Base-0708
—
pre-training (§ 3 )
—
MOSS-VL-Instruct-0708
Base-0708
SFT (§ 4 )
offline ( Table 5 )
MOSS-VL-Realtime
Instruct-0708
Realtime-SFT (§ 4 )
streaming ( Table 6 )
MOSS-VL-Base-0408
—
pre-training
—
MOSS-VL-Instruct-0408
Base-0408
SFT
—
Table 3: Released MOSS-VL checkpoints. The 0708 run is the subject of this report: MOSS-VL-Base is the pre-trained model, MOSS-VL-Instruct is its instruction-tuned successor, and MOSS-VL-Realtime continues training from MOSS-VL-Instruct with Realtime-SFT. The 0408 checkpoints are an earlier, independently trained run of the same architecture, released alongside for research continuity. All five are available under the OpenMOSS-Team organization on HuggingFace.
Stage
Tokens
Samples
Max seq.
Trainable
Peak LR
Pre-training
1 Vision–language alignment
150.3B
219.4M
8K
Projection + cross-attn
2×10−4
2 Large-scale multimodal
203.0B
139.3M
64K
Full model
5×10−5
3 High-quality multimodal
459.0B
14.7M
128K
Full model
1×10−5
4 Annealing & long-context
450.1B
9.3M
256K
Full model
1×10−5
Post-training (§ 4 )
Table 4: The MOSS-VL training curriculum: four pre-training stages followed by SFT and Realtime-SFT (§ 4 ). The data of each stage is described in the corresponding subsection. Token counts are the tokens fed to the language model, with vision tokens counted after the 2×2 token compression of the projection module (§ 2 ); sequence lengths are in tokens.
Benchmark
MOSS-VL (ours)
Qwen3-VL 8B
Qwen2.5-VL 7B
LLaVA-OV-2 8B
Gemma-4 12B-IT
Multimodal perception
MMBench-EN (v1.1) [ 22 ]
88.1
84.8
83.2
85.8
82.7
MMStar [ 6 ]
66.0
70.9
63.9
64.8
74.9
RealWorldQA
68.0
71.5
68.5
69.7
65.6
MME-RealWorld [ 55 ]
66.3
–
57.4
–
46.9
BLINK [ 12 ]
78.0
69.1
56.4
63.5
65.6
Table 5: Offline results: MOSS-VL-Instruct (0708 release) against open-source models of comparable scale, blocked by capability domain. Bold = best, underline = second best, “–” = not reported; LLaVA-OV-2 = LLaVA-OneVision-2-8B. Evaluation protocol and baseline provenance are spelled out in § 6.1 .
OVO-Bench [ 20 ]
Subset
MOSS-VL-Realtime
AURA
JoyAI-VL-Interaction
VideoChat3-4B
ViSpeak-7B
FAR
62.1
55.8
–
–
54.2
BT
72.6
60.4
–
–
57.5
RTVP
75.9
79.8
–
–
66.3
Avg
70.2
65.3
59.2
57.8
59.3
OmniMMI [ 44 ]
Table 6: Streaming benchmark results: MOSS-VL-Realtime against open-source streaming baselines at subset level. Bold = best, underline = second best; “–” = subset not reported in that model’s official report. Subset abbreviations, each benchmark’s Avg convention, and data provenance are spelled out in § 6.2 .
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Turn
English translation
(a) Conditional alert
User ( t=0 )
Watch in real time and judge whether the cat touches the carrot; output “Great!” whenever it does, and stay silent otherwise.
Model
“Great!”, emitted at each of the four contacts; every other frame yields the silence token.
(b) Real-time commentary
User ( t=0 )
Commentate on the video live.
Model
Portugal win a crucial free-kick chance; Ronaldo stands over the ball…
Appendix
Table 7: English translations (ours) of the two live-demo sessions in Figure 5 . The figure carries the Chinese excerpts; timestamps and the placement of each output on the stream are read off the figure’s timeline.
Video understanding is shifting from the offline paradigm -- taking a fully recorded video as input and producing a single answer after it ends -- toward real-time interaction, in which the model perceives new frames while still replying, revises its answer as new evidence appears, and remains silent when there is nothing to say. We present MOSS-Video-Preview to validate this paradigm. Our central claim is that perception must not be blocked by generation; its natural realization is a two-channel architecture. We argue that a cross-attention backbone is better suited to real-time vision-language fusion than the prevailing decoder-only design: visual features enter through a side channel rather than joining the autoregressive sequence, so perception and generation run on separate, non-blocking pathways -- reducing the frequency of visual processing and exposing a clean channel-wise interface for independent compression. We complement this with a data synthesis pipeline that converts dense captions into real-time understanding QA whose answers are revised to match what the model has perceived so far, and we specialize an offline model on these data to elicit real-time behavior. Our model trails the strong Qwen2.5-VL-7B baseline overall -- a gap we attribute primarily to data and scale rather than the architecture -- yet attains competitive offline video and multimodal understanding, remains robust on the spatial and fine-grained temporal reasoning central to real-time use, and acquires behaviors that offline models lack: continuous perception, answer revision, and timely silence. On a single H200 with 256 frames per video, it achieves about a 5x speedup in time to first token and 2.7x higher decoding throughput, with negligible degradation in offline ability. Our study of paradigm, architecture, and data outlines a viable path toward real-time video understanding.
Many moments in the real world do not wait for a user to ask. A fire starts on a security monitor, an expression flickers across a video call, or a product a viewer wants flashes by in a livestream. Yet today's large models remain mostly turn-based by design: they answer only when addressed, and even video-call apps that appear interactive still operate as question-answer systems, reacting only when polled or prompted. We argue for a different paradigm: a model that is present in the world like a person. It continuously watches what is happening now, decides on its own whether to speak or stay silent, interacts in real time, and delegates to a background model when the problem is hard. To advance interaction models and their adoption across domains, we make two fully open-sourced contributions. First, we release JoyAI-VL-Interaction, an 8B-scale, vision-first VL-interaction model. The model makes the response decision internally, choosing each second to stay silent, respond, or delegate to a background model, and it excels at vision-triggered responsiveness and time awareness. We pair it with a transferable training recipe, from which capabilities we never trained for emerge, such as guiding a shopper through changing app screens or improvising a lecture from a slide deck. Second, we release a complete, deployable system built around that model. The system streams any ongoing video into the model, making it genuinely present in the world. All other components are pluggable, including ASR/TTS modules, memory, visualization UI, and a background brain that can connect to any API or agent. Across six real-world scenarios, human raters prefer JoyAI-VL-Interaction over the in-app video-call assistants of Doubao and Gemini by a wide margin. To our knowledge, this is the first open, vision-driven interaction model released together with its training recipe, data, and complete deployable system.
Vision-and-Language Navigation (VLN) in real-world settings requires agents to process continuous visual streams and generate actions with low latency grounded in language instructions. While Video-based Large Language Models (Video-LLMs) have driven recent progress, current VLN methods based on Video-LLM often face trade-offs among fine-grained visual understanding, long-term context modeling and computational efficiency. We introduce StreamVLN, a streaming VLN framework that employs a hybrid slow-fast context modeling strategy to support multi-modal reasoning over interleaved vision, language and action inputs. The fast-streaming dialogue context facilitates responsive action generation through a sliding-window of multi-turn dialogues, while the slow-updating memory context compresses historical visual states using a 3D-aware token pruning strategy. With this slow-fast design, StreamVLN achieves real-time dialogues through KV cache reuse, supporting long video streams with bounded context size and inference cost. Experiments on VLN-CE benchmarks show state-of-the-art performance with low latency, ensuring robustness and efficiency in real-world deployment. The project page is: https://streamvln.github.io/.
Meng Wei, Chenyang Wan, Xiqian Yu +9
1Shanghai AI Lab · 2The University of Hong Kong · 3Zhejiang University +1