An assistant watching a worker can prepare for an emerging need before the worker is ready to receive help.As the need evolves, the assistant must maintain preparation until a response is due, or withdraw it if the need disappears. Video anticipation and interactive response benchmarks leave this interval between early evidence and response largely unmeasured. We introduce FuGIBench (Future-Grounded Interaction), a streaming-video benchmark in which a model chooses among waiting, reversible preparation, and response given a causal video prefix and a task contract. The benchmark contains 19,893 synthetic tool-assistance videos across ten tool tasks, in handover, ordinary-activity, and cancelled-need tracks. One primary score credits sustained preparation, incorporates response-interval alignment, and penalizes activity when the reference prescribes waiting. On a frozen set of 1,000 episodes, a reference system with a longer preparation lead has poorer response alignment, while similar handover leads accompany different levels of activity during ordinary work in matched scenarios. Together with cancellation analyses, these results show why evaluating preparation requires tracking decisions as needs develop, become actionable, or disappear.
Figures & tables
Figure 1: Prepare while a need develops; withdraw when it disappears. Frames from two tool-assistance videos are linked to their reference stages. (a) Preparation precedes the handover boundary τ . (b) Cancellation returns the reference to Wait without a response. Each state strip spans its own video duration; frame timestamps are in seconds.
Source family
Videos
Tool handover with initial waiting
13,194
Tool ordinary activity
3,502
Tool cancelled need
3,197
Total
19,893
Table 1: FuGIBench composition. Tool-assistance videos in the export inspected on November 6, 2026.
Figure 2: Three episode types cover preparation, inactivity, and withdrawal. The 1,000 evaluation episodes are sampled from the 19,893-video tool collection, with stages rescaled and boundaries remapped. Sequences show stage order, not measured durations.
Figure 3: Three intervals determine the target-episode score. Preparation uses the uninterrupted Prepare interval held to τ . Active time in reference Wait lowers silence. Recognition combines event/tool correctness with response-interval IoU: the illustrated overlap covers two-thirds of the union. Dots mark model emissions; each state holds until the next emission or stream end. Tool identity is read at the first active observation before the target. This is a scoring illustration, not a measured result.
Figure 4: Preparation and silence operating points. Markers show mean P over 600 targets and mean S over all 1,000 episodes for the trained reference, Qwen2.5-Omni ( ∙ ), Gemma-3n ( ∙ ), and two constant policies. Contours illustrate the harmonic-mean geometry for individual target episodes. Evaluating H at a plotted aggregate point does not recover mean behavioral reward, which is computed per episode.
Figure 5: Preparation and activation in matched scenarios. Mean preparation lead on handovers and mean active time on ordinary activity are shown separately for the 67 pairs with matched recorded workplace attributes. Both references receive video. Full descriptive results are supplementary.
Model
Mode
Overall ↑
R↑
P↑
S↑
Lead ↑
Cov. ↑
[0,1]
[0,1]
[0,1]
[0,1]
seconds
fraction
Evaluated without benchmark-specific training (zero-shot / prompt-adapted)
Qwen3-VL-8B-Instruct [ 4 ]
polled
0.298
0.213
0.004
0.916
0.04
0.06
Qwen2.5-Omni-7B [ 31 ]
polled
0.301
0.205
0.098
0.895
1.19
0.12
LLaVA-OneVision-2-8B-Instruct [ 2 ]
polled
0.300
0.200
0.005
0.995
0.05
0.01
JoyAI-VL-Interaction [ 33 ]
native
0.300
0.200
0.000
0.999
0.00
0.00
Table 2: Tool-assistance results on 1,000 episodes (600 handovers, 200 ordinary activities, 200 cancellations). R , P , and S are response-aligned recognition, preparation, and silence. On targets, R combines event-class and tool correctness with response-interval overlap. The tool is named at the first active observation before the target. Lead and coverage use all targets. All learned systems receive video. Trained references use a fixed tool-vocabulary mapping. Adapted denotes an output adapter without ground truth; reference configurations are in Section 6 .
Figure 6: Three measured behaviors explain the evaluation components. Selected diagnostic episodes from the trained reference; model states are held between observations through stream end. (a) Sustained preparation earns lead, while activation during reference Wait incurs a silence cost. (b) Early Response interrupts preparation, giving zero lead despite nonzero response overlap. (c) After cancellation, the model returns to Wait and remains there. Dashed lines mark target time τ or cancellation; all panels share the same time scale.
Reference
Trained parameters
Align
Joint
Native video + LoRA
Decoder LoRA
0
547
Expanded video
Bridge + decoder LoRA
547
547
Trained reference
Bridge + decoder LoRA
547
2,688
Table 3: Trained video references. All retain JoyAI’s native visual path. Expanded adds V-JEPA 2 recent/history features. Align and Joint are update counts; the two short references share their joint sampling schedule.
Training pool
Videos
Handover / ordinary / cancellation
9,873 / 2,469 / 1,660
Total
14,002
Positive source for timing policies
9,860
Joint schedule
Videos / prefixes / QA
Short references
14,002 / 14,002 / 3,488
Trained reference (partial)
4,224 / 68,832 / 17,184
Table 4: Training data and joint exposure. Video counts refer to the tool-video collection. QA examples are teacher-generated retention examples.
System
History / trigger
State parsing and no-update
Qwen, LLaVA, Gemma
Observed conversation; polled
Parse task JSON.
MiniCPM-o
Streaming KV; polled
Extract task JSON from marked output.
JoyAI native
Native streaming; emission
Parse response JSON; silence retains state.
MMDuet2
DTD/KV; two-frame turns
Parse task output; NO REPLY retains state.
VideoChat3
Streaming session; up to 16 rounds
Parse response JSON; silence or standby retains state.
LiveCC
Incremental KV; initial six frames, then pairs
Extract task JSON; silence or an unparseable state gives no-update.
Table 5: Model interfaces. Polled systems produce a decision on each observation; native systems use their own emission mechanisms. No-update retains the previous state, with an unknown initial state.
Policy
Overall ↑
R↑
P↑
S↑
Always Wait
0.300
0.200
0.000
1.000
Always Prepare
0.000
0.000
0.882
0.000
Always Response
0.012
0.023
0.000
0.000
Immediate Prepare → timed Response
0.019
0.038
0.324
0.000
Wait → timed Response
0.174
0.061
0.000
0.884
Training-median clock
0.147
0.038
0.107
0.669
Table 6: Deterministic policies on all 1,000 episodes. Active policies use the training-majority tool. Timed responses start at the training-median onset; the clock also waits for the training-median preparation onset. All follow the same observation schedule and held-state scoring as the video reference.
System
Lead
Cov.
A
Early
Rec.
Post
↑
↑
↑
↓
↑
↓
Trained reference
2.08
0.56
0.701
0.27
0.86
0.64
Expanded video
2.08
0.56
0.695
0.16
0.89
1.33
JoyAI + task head
5.36
0.93
0.536
0.07
0.08
5.10
Native video + LoRA
3.67
0.68
0.083
0.00
0.01
5.59
Qwen3-VL-8B
0.04
0.06
0.239
0.50
0.00
0.70
Table 7: Preparation, response, and withdrawal. Lead (s), coverage (Cov.), response alignment ( A ), and premature-response rate (Early) use all 600 targets, including zeros. Trajectory-derived cancellation-class recall (Rec.) and mean active seconds after cancellation (Post) use all 200 cancellations.
System
Ever
Held
Return
Stable
Median delay (s)
Qwen3-VL-8B
77/200
14/200
12/14
8/14
1.56
Qwen2.5-Omni-7B
39/200
11/200
1/11
0/11
1.96
LLaVA-OneVision-2-8B-Instruct
1/200
1/200
0/1
0/1
–
JoyAI-VL-Interaction
0/200
0/200
–
–
–
MMDuet2
0/200
0/200
–
–
–
LiveCC-7B-Instruct
0/200
0/200
–
–
–
Table 8: Conditional withdrawal for the full model panel. Ever, Held, Return, Stable, and Delay follow the definitions in the text. Each fraction retains its denominator. A dash indicates no eligible preparation at cancellation, or no observed return for the delay.
System
State lead (s)
Correct-tool lead (s)
Correct-tool coverage
Qwen3-VL-8B
0.04
0.03
0.035
Qwen2.5-Omni-7B
1.19
0.21
0.023
LLaVA-OneVision-2-8B-Instruct
0.05
0.04
0.003
JoyAI-VL-Interaction
0.00
0.00
0.000
MMDuet2
0.00
0.00
0.000
LiveCC-7B-Instruct
0.00
0.00
0.000
Table 9: Tool identity during preparation for the full panel. Means and coverage include all 600 targets. Correct-tool lead checks every observation in the trailing run; it does not multiply the state lead by the identity on the first active observation.
System
Lead ↑
Ordinary ↓
Diff.
Active ↓
Expanded video
2.27
0.15
2.12
8/67
Native video + LoRA
2.54
1.87
0.68
37/67
Qwen3-VL-8B-Instruct
0.08
0.63
-0.55
17/67
Gemma-3n-E4B-it
2.24
9.42
-7.18
61/67
Table 10: Workplace-matched pairs. Each of 67 pairs shares the scenario title, tool, workplace description, and clothing, lighting, and wall bin. Lead is the mean continuous Prepare -only lead on positives; Ordinary is the mean active time on ordinary videos, in seconds. Durations differ. Diff. is the descriptive mean per-pair difference, not a normalized score or causal effect; Active counts ordinary videos with any activation. Both references receive video.
Model trajectory
Lead (s)
A
Prepare [1,4) ; Response [4,8)
3
1.00
Prepare [1,3) ; Response [3,8)
0
0.80
Response [1,8)
0
0.57
Prepare [1,4) ; no response
3
0.00
Table 11: Preparation and response measure distinct behavior. The reference response is [4,8) s. Rows show illustrative held states, with no response outside the stated interval. In R , response IoU A is multiplied by event-class and first pre-target active tool correctness.
Streaming video understanding requires models to process continuous multimodal input while maintaining temporal context. Existing evaluations are predominantly reactive: they query a model at a selected timestamp and therefore do not assess when it should respond. Proactive interaction instead requires monitoring a standing request, responding within an appropriate interval after the target event, and otherwise remaining silent. We introduce LiveProBench, which evaluates models at one-second stream intervals without an explicit response cue. Its six subtasks vary trigger ambiguity and timing tolerance. Event Sensitivity geometrically combines response and silence rates on the same recording; four window-based subtasks distinguish early, in-window, and missed responses; and Duplicate Counting penalizes omissions and repetitions. Premature responses outnumber missed responses for half of the evaluated models, revealing a substantial gap in the temporal decision-making required for human-like interaction.
Kaixuan Du, Xin Wan, Hang Zhang +4
School of Automation Science and Electrical Engineering, Beihang University · Qwen Business Unit of Alibaba
Streaming video models should respond at the moment an event unfolds, not after the moment has passed. Yet existing online VideoQA benchmarks remain largely retrospective. They pause the video at fixed timestamps, pose questions about current or past events, and score models only at those moments. This protocol leaves streaming outputs untested. To close this gap, we introduce SPOT-Bench, featuring multi-turn proactive queries that evaluate general streaming perception required by an always-on, real-time assistant. SPOT-Bench comes with Timeliness-F1, a consolidated metric that measures streaming outputs by their temporal precision and balanced coverage across the entire video. Our benchmark reveals that during streaming inference: (i) MLLMs detect events reliably in a stream but spam predictions unprompted; (ii) post-training MLLMs for silence reduces spamming but induces unresponsiveness; (iii) half of the streaming video expects no response, which we term dead-time - compute spent here does not affect response latency. These findings motivate Ctrl-SPOT, a training-free streaming controller for MLLMs, that retains their event perception while controlling their streaming outputs for balanced coverage. This establishes a strong baseline on SPOT-Bench.
A video assistant must speak when its instruction warrants a response and stay silent otherwise. We introduce a benchmark that evaluates this decision for the complete system of model, memory, and response controller. InteractionBench covers query responses, event triggers, and ongoing updates in 1,060 interactions over 812 videos, with 69 negative streams and 53 suites that pair counted events with look-alike near misses. It scores content accuracy, timing accuracy, and silence compliance on the video clock. Timely speech costs silence across systems. Polled Qwen3-VL-8B reaches 77.8 timing accuracy but 10.9 silence compliance. A native real-time interaction system reaches 29.2 silence compliance at 66.8 timing accuracy, yet emits on 89.9% of negative streams. No open-weight system clears a third of the near-miss suites. Fewer replies help only when chosen, as random deletion merely trades timing for silence. Offline scores miss these failures and mispredict online behavior. Adding restraint is costly, as the native system's controller adds little by itself and agentic systems add it only at about 30 s per poll.Project page: https://www.enxinsong.com/projects/interactionbench/ Code: https://github.com/Espere-1119-Song/InteractionBench Data: https://huggingface.co/datasets/InteractionBench/InteractionBench
Enxin Song, Suhao Yu, Yifei Xu +7
University of Pennsylvania · Stanford University · University of Illinois at Urbana-Champaign +3