Organizations: The Hong Kong University of Science and Technology (Guangzhou) (hkust-gz.edu.cn) · Tsinghua University (tsinghua.edu.cn) · Tencent (tencent.com) · The Hong Kong University of Science and Technology (ust.hk)
Recognizing individual events does not guarantee correct execution of compositional instructions in streaming video. For example, given the request "Notify me when the pot boils after someone turns on the stove and leaves the kitchen", existing models may recognize the individual events yet notify too early or miss the intended moment, failing to track how their temporal conditions are satisfied over time. To evaluate this gap, we introduce CoTriggerBench, which composes annotated video events into temporally constrained instructions and verifies notification timing and silence through executable event plans and source-event replay. To address this challenge, we propose a real-time VLM agent, TEMPI. TEMPI decomposes instructions into atomic event queries and temporal constraints, and uses Scenario Memory to continuously maintain task progress as event evidence arrives, allowing earlier observations to support subsequent decisions beyond the detector's visual window. Even when no notification is required, relevant events continue to update the scenario state. On CoTriggerBench, TEMPI achieves 44.4% instruction-level success, outperforming the strongest closed-source and fine-tuned open-source VLM baselines by 15.6 (Qwen3.8-Omni-Flash-Realtime) and 18.1 (MOSS-VL-Realtime) percentage points, respectively. To support efficient on-device streaming inference, we further optimize the streaming inference pipeline by overlapping partial visual encoding and LLM prefix prefill with frame arrival. On an 8 GB RTX 5060, our optimized detector implementation achieves a 4.90× speedup.
Figures & tables
Figure 1: Atomic and compositional triggering. Figure 1(a) shows that existing models succeed with atomic queries and explicit execution but fail on the compositional instruction. Figure 1(b) reports request-level accuracy rates in atomic and composite triggering events. Most models achieve much higher trigger accuracy in atomic events than compositional events. † denotes closed models.
Figure 2: CoTriggerBench construction pipeline. The full construction pipeline comprises event normalization, task composition, query generation, verification, and dataset formation.
Figure 3: CoTriggerBench task types. This figure shows event timelines and notification outcomes for the five task types.
Figure 4: TEMPI architecture and streaming execution. A skill-guided TEMPI planner decomposes the instruction into atomic queries and compositional constraints to form the scenario plan. The detector supplies event/silence evidence from recent frames, and the TEMPI agent writes scenario state updates into scenario memory, activates dependent steps, and issues a notification once all constraints are satisfied. External backends can supply the same event interface.
Model
Params
Compositional Triggering
Atomic Event Detection
Strict (%) ↑
TS (%) ↑
AS (%) ↑
MAE (s) ↓
F1 (%) ↑
P (%) ↑
R (%) ↑
FP/min ↓
Zero-shot
JoyAI-VL-Interaction [ 29 ]
8B
19.2
21.6
8.5
6.84
21.3
15.8
32.8
5.62
MOSS-VL-Realtime [ 22 ]
11B
18.3
16.4
26.8
5.10
33.3
25.1
49.7
4.76
StreamingVLM [ 26 ]
7B
15.6
3.8
68.3
6.39
3.0
27.6
1.6
0.14
LiveCC-7B-Instruct [ 4 ]
7B
4.9
4.1
8.5
11.38
10.2
10.5
9.9
2.69
Table 2: Compositional triggering and atomic event detection. Strict, TS, and AS denote overall instruction success, positive-task trigger success, and adversarial silence. These rates, precision (P), recall (R), and their harmonic mean (F1) are reported as percentages. MAE is the mean absolute distance from the first notification to the unextended target interval, in seconds; FP/min counts false event runs per video minute. Params are in billions (B). For Em-Garde, 2B+7B denotes its embedding and proposal/response models, respectively. For TEMPI, 2B+4B denotes our detector and planner, respectively. The best result of each metric is shown in bold.
Backend
TS (%) ↑
AS (%) ↑
Strict (%) ↑
MAE (s) ↓
D
T
D
T
D
T
D
T
Zero-shot
StreamingVLM
3.8
0.8
68.3
96.3
15.6
18.3
6.39
4.51
LiveCC
4.1
9.3
8.5
47.6
4.9
16.3
11.38
6.99
Gemini Live
10.9
4.1
63.4
87.8
20.5
19.4
4.19
5.21
Qwen3.5-Omni
19.4
26.8
4.9
28.0
16.7
27.0
7.40
4.67
Table 3: Effect of TEMPI across perception backends. D/T: Direct/TEMPI execution. D supplies the full instruction directly to the backend; T uses it as an atomic-event detector in our framework. TEMPI-D denotes our 2B detector. TS, AS, and Strict are percentages; MAE is in seconds. Better values within each pair are bold.
Backend
FPS
Direct (%)
TEMPI (%)
Atomic events (%)
TS ↑
AS ↑
Strict ↑
TS ↑
AS ↑
Strict ↑
P ↑
R ↑
F1 ↑
JoyAI
1
23.5
37.8
26.1
30.3
78.0
39.1
57.9
44.5
50.3
JoyAI
5
21.6
67.1
29.9
24.0
76.8
33.7
53.0
37.4
43.9
MOSS
1
20.5
52.4
26.3
26.8
78.0
36.2
69.2
42.1
52.3
MOSS
5
7.4
78.0
20.3
9.6
91.5
24.6
66.9
18.7
29.2
Table 4: Effect of input frame rate. Better scores within each 1/5 fps pair are bold. Increasing the frame rate does not guarantee better performance; native settings yield higher TEMPI strict success and atomic-event F1 for both backends.
Planner
Planning quality (%) ↑
Latency (ms) ↓
Qty.
Sem.
Match
Type
P50
P95
Qwen3.5-2B
97.3
91.2
85.7
88.6
86.7
107.5
Qwen3.5-4B
99.3
92.7
87.7
90.9
129.0
160.6
Qwen3.5-9B
99.3
93.5
89.1
94.4
179.2
226.6
Qwen3.6-35B-A3B
98.7
93.4
88.6
98.2
159.6
244.2
Qwen3.8-27B
99.1
94.7
90.4
98.9
709.8
878.2
Table 5: Scenario Planning Agent size. Planning quality is reported as percentages. Qty.: correct subquery count rate. Sem.: position-wise semantic equivalence judged by Qwen3.8-Flash-Next-FP8 [ 20 ] . Match: correct count and equivalent queries. Type: task-type accuracy. Latency: complete-plan request time on one H800.
Model
Params
P50 ↓
P90 ↓
P99 ↓
TEMPI-D
2B
61.0
62.7
67.9
Intern-Decision-2B [ 35 ]
2B
75.2
82.0
85.7
OmniJev-2B [ 27 ]
2B
165.0
170.6
174.4
JoyAI-VL-Interaction [ 29 ]
8B
240.3
469.6
573.1
MOSS-VL-Realtime [ 22 ]
11B
91.6
95.5
219.9
StreamingVLM [ 26 ]
7B
93.6
98.5
105.8
Table 6: Detector GPU execution latency (ms). Measurements are obtained on one H800 with a batch size of 1. TEMPI-D denotes our 2B detector.
Figure 5: Local efficiency and early execution on an RTX 5060 (8 GB). Figure 5(a) shows how early ViT and prefix prefill overlap frame arrival; cached KV feeds suffix prefill after frame 4. Figure 5(b) reports decision latency P50 for one/three queries and independently evaluated strict success. † : VRAM measured per precision with three queries.
Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.