Organizations: The Hong Kong University of Science and Technology (Guangzhou) (hkust-gz.edu.cn) · Tsinghua University (tsinghua.edu.cn) · Tencent (tencent.com) · The Hong Kong University of Science and Technology (ust.hk)
Recognizing individual events does not guarantee correct execution of compositional instructions in streaming video. For example, given the request "Notify me when the pot boils after someone turns on the stove and leaves the kitchen", existing models may recognize the individual events yet notify too early or miss the intended moment, failing to track how their temporal conditions are satisfied over time. To evaluate this gap, we introduce CoTriggerBench, which composes annotated video events into temporally constrained instructions and verifies notification timing and silence through executable event plans and source-event replay. To address this challenge, we propose a real-time VLM agent, TEMPI. TEMPI decomposes instructions into atomic event queries and temporal constraints, and uses Scenario Memory to continuously maintain task progress as event evidence arrives, allowing earlier observations to support subsequent decisions beyond the detector's visual window. Even when no notification is required, relevant events continue to update the scenario state. On CoTriggerBench, TEMPI achieves 44.4% instruction-level success, outperforming the strongest closed-source and fine-tuned open-source VLM baselines by 15.6 (Qwen3.8-Omni-Flash-Realtime) and 18.1 (MOSS-VL-Realtime) percentage points, respectively. To support efficient on-device streaming inference, we further optimize the streaming inference pipeline by overlapping partial visual encoding and LLM prefix prefill with frame arrival. On an 8 GB RTX 5060, our optimized detector implementation achieves a 4.90× speedup.
Figures & tables
Figure 1: Atomic and compositional triggering. Figure 1(a) shows that existing models succeed with atomic queries and explicit execution but fail on the compositional instruction. Figure 1(b) reports request-level accuracy rates in atomic and composite triggering events. Most models achieve much higher trigger accuracy in atomic events than compositional events. † denotes closed models.
Figure 2: CoTriggerBench construction pipeline. The full construction pipeline comprises event normalization, task composition, query generation, verification, and dataset formation.
Figure 3: CoTriggerBench task types. This figure shows event timelines and notification outcomes for the five task types.
Figure 4: TEMPI architecture and streaming execution. A skill-guided TEMPI planner decomposes the instruction into atomic queries and compositional constraints to form the scenario plan. The detector supplies event/silence evidence from recent frames, and the TEMPI agent writes scenario state updates into scenario memory, activates dependent steps, and issues a notification once all constraints are satisfied. External backends can supply the same event interface.
Model
Params
Compositional Triggering
Atomic Event Detection
Strict (%) ↑
TS (%) ↑
AS (%) ↑
MAE (s) ↓
F1 (%) ↑
P (%) ↑
R (%) ↑
FP/min ↓
Zero-shot
JoyAI-VL-Interaction [ 29 ]
8B
19.2
21.6
8.5
6.84
21.3
15.8
32.8
5.62
MOSS-VL-Realtime [ 22 ]
11B
18.3
16.4
26.8
5.10
33.3
25.1
49.7
4.76
StreamingVLM [ 26 ]
7B
15.6
3.8
68.3
6.39
3.0
27.6
1.6
0.14
LiveCC-7B-Instruct [ 4 ]
7B
4.9
4.1
8.5
11.38
10.2
10.5
9.9
2.69
Table 2: Compositional triggering and atomic event detection. Strict, TS, and AS denote overall instruction success, positive-task trigger success, and adversarial silence. These rates, precision (P), recall (R), and their harmonic mean (F1) are reported as percentages. MAE is the mean absolute distance from the first notification to the unextended target interval, in seconds; FP/min counts false event runs per video minute. Params are in billions (B). For Em-Garde, 2B+7B denotes its embedding and proposal/response models, respectively. For TEMPI, 2B+4B denotes our detector and planner, respectively. The best result of each metric is shown in bold.
Backend
TS (%) ↑
AS (%) ↑
Strict (%) ↑
MAE (s) ↓
D
T
D
T
D
T
D
T
Zero-shot
StreamingVLM
3.8
0.8
68.3
96.3
15.6
18.3
6.39
4.51
LiveCC
4.1
9.3
8.5
47.6
4.9
16.3
11.38
6.99
Gemini Live
10.9
4.1
63.4
87.8
20.5
19.4
4.19
5.21
Qwen3.5-Omni
19.4
26.8
4.9
28.0
16.7
27.0
7.40
4.67
Table 3: Effect of TEMPI across perception backends. D/T: Direct/TEMPI execution. D supplies the full instruction directly to the backend; T uses it as an atomic-event detector in our framework. TEMPI-D denotes our 2B detector. TS, AS, and Strict are percentages; MAE is in seconds. Better values within each pair are bold.
Backend
FPS
Direct (%)
TEMPI (%)
Atomic events (%)
TS ↑
AS ↑
Strict ↑
TS ↑
AS ↑
Strict ↑
P ↑
R ↑
F1 ↑
JoyAI
1
23.5
37.8
26.1
30.3
78.0
39.1
57.9
44.5
50.3
JoyAI
5
21.6
67.1
29.9
24.0
76.8
33.7
53.0
37.4
43.9
MOSS
1
20.5
52.4
26.3
26.8
78.0
36.2
69.2
42.1
52.3
MOSS
5
7.4
78.0
20.3
9.6
91.5
24.6
66.9
18.7
29.2
Table 4: Effect of input frame rate. Better scores within each 1/5 fps pair are bold. Increasing the frame rate does not guarantee better performance; native settings yield higher TEMPI strict success and atomic-event F1 for both backends.
Planner
Planning quality (%) ↑
Latency (ms) ↓
Qty.
Sem.
Match
Type
P50
P95
Qwen3.5-2B
97.3
91.2
85.7
88.6
86.7
107.5
Qwen3.5-4B
99.3
92.7
87.7
90.9
129.0
160.6
Qwen3.5-9B
99.3
93.5
89.1
94.4
179.2
226.6
Qwen3.6-35B-A3B
98.7
93.4
88.6
98.2
159.6
244.2
Qwen3.8-27B
99.1
94.7
90.4
98.9
709.8
878.2
Table 5: Scenario Planning Agent size. Planning quality is reported as percentages. Qty.: correct subquery count rate. Sem.: position-wise semantic equivalence judged by Qwen3.8-Flash-Next-FP8 [ 20 ] . Match: correct count and equivalent queries. Type: task-type accuracy. Latency: complete-plan request time on one H800.
Model
Params
P50 ↓
P90 ↓
P99 ↓
TEMPI-D
2B
61.0
62.7
67.9
Intern-Decision-2B [ 35 ]
2B
75.2
82.0
85.7
OmniJev-2B [ 27 ]
2B
165.0
170.6
174.4
JoyAI-VL-Interaction [ 29 ]
8B
240.3
469.6
573.1
MOSS-VL-Realtime [ 22 ]
11B
91.6
95.5
219.9
StreamingVLM [ 26 ]
7B
93.6
98.5
105.8
Table 6: Detector GPU execution latency (ms). Measurements are obtained on one H800 with a batch size of 1. TEMPI-D denotes our 2B detector.
Figure 5: Local efficiency and early execution on an RTX 5060 (8 GB). Figure 5(a) shows how early ViT and prefix prefill overlap frame arrival; cached KV feeds suffix prefill after frame 4. Figure 5(b) reports decision latency P50 for one/three queries and independently evaluated strict success. † : VRAM measured per precision with three queries.
Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.
Streaming Video Large Language Models (VLMs) enable continuous video understanding, yet existing benchmarks focus on low-dynamic scenarios. Under bounded context budgets, models must balance temporal history, spatial resolution, and temporal granularity; sparse sampling at 1--2 FPS misses fast events. We introduce FastBench to evaluate high-dynamic perception in real-world video streams. Its trajectory-grounded pipeline combines QA generation from high-FPS clips, filtering of questions answerable at 2 FPS, answer verification using SAM3 and CoTracker3 trajectories, and three rounds of human inspection. FastBench contains 306 QA pairs across eight domains, six capabilities, and forward, instant, and backward temporal scopes, with human-annotated evidence intervals. We also present ProactiveFrame, a training-free baseline that adjusts incoming frame rates through text tokens. A dual-tier sliding window retains recent high-FPS observations while downsampling older ones into sparse history. Experiments reveal substantial limitations: the strongest model, Gemini-3.5-Flash, scores only 50.7%. Denser sampling improves Qwen3-VL-8B from 32.9% at 2 FPS to 44.6% at 24 FPS, but gains saturate as history is compressed. ProactiveFrame outperforms sparse uniform sampling by 5.4 and 1.5 percentage points, yet remains well below oracle-guided focusing, showing that current VLMs struggle to determine from the stream alone when finer temporal perception is needed. FastBench provides a testbed for high-dynamic streaming video understanding. Code and data: https://github.com/Ashone3/FastBench.
Streaming video models should respond at the moment an event unfolds, not after the moment has passed. Yet existing online VideoQA benchmarks remain largely retrospective. They pause the video at fixed timestamps, pose questions about current or past events, and score models only at those moments. This protocol leaves streaming outputs untested. To close this gap, we introduce SPOT-Bench, featuring multi-turn proactive queries that evaluate general streaming perception required by an always-on, real-time assistant. SPOT-Bench comes with Timeliness-F1, a consolidated metric that measures streaming outputs by their temporal precision and balanced coverage across the entire video. Our benchmark reveals that during streaming inference: (i) MLLMs detect events reliably in a stream but spam predictions unprompted; (ii) post-training MLLMs for silence reduces spamming but induces unresponsiveness; (iii) half of the streaming video expects no response, which we term dead-time - compute spent here does not affect response latency. These findings motivate Ctrl-SPOT, a training-free streaming controller for MLLMs, that retains their event perception while controlling their streaming outputs for balanced coverage. This establishes a strong baseline on SPOT-Bench.
Large Vision Language Models (LVLMs) exhibit strong Chain-of-Thought (CoT) capabilities, yet most existing paradigms assume full-video availability before inference, a batch-style process misaligned with real-world video streams where information arrives sequentially. Motivated by the streaming nature of video data, we investigate two streaming reasoning paradigms for LVLMs. The first, an interleaved paradigm, alternates between receiving frames and producing partial reasoning but remains constrained by strictly ordered cache updates. To better match streaming inputs, we propose Think-as-You-See (TaYS), a unified framework enabling true concurrent reasoning. TaYS integrates parallelized CoT generation, stream-constrained training, and stream-parallel inference. It further employs temporally aligned reasoning units, streaming attention masks and positional encodings, and a dual KV-cache that decouples visual encoding from textual reasoning. We evaluate all paradigms on the Qwen2.5-VL family across representative video CoT tasks, including event dynamics analysis, causal reasoning, and thematic understanding. Experiments show that TaYS consistently outperforms both batch and interleaved baselines, improving reasoning performance while substantially reducing time-to-first-token (TTFT) and overall reasoning delay. These results demonstrate the effectiveness of data-aligned streaming reasoning in enabling efficient and responsive video understanding for LVLMs. We release our code at https://github.com/EIT-NLP/StreamingLLM/tree/main/TaYS
Jialiang Zhang, Junlong Tong, Junyan Lin +4
Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ocean University of China · Shanghai Jiao Tong University +3