BLIP-Live: Balancing Timeliness & Firing Precision in Proactive Streaming Video Agents
Organizations: Salesforce AI Research
Abstract
Streaming video understanding is becoming increasingly important for real-world interactive systems. Such systems must be proactive for natural and useful human interaction: they should respond as soon as sufficient evidence becomes available while avoiding premature, redundant, or incorrect responses. We present BLIP-Live, a framework unifying data, metrics, harness, and training around a shared formulation of proactivity as effective causal response decisions balancing the two competing objectives of responding vs silence. First, we construct BLIP-Live-780K dataset containing human annotations of proactive response content and timing, which defines this desired behavior. We then propose PAUC-F, a metric to effectively measure our desired proactive behavior. PAUC-F combines existing response timeliness metrics with firing precision to reward early useful responses without rewarding excessive firing. With our desired behavior defined and measurable, we finally investigate model-agnostic inference harnesses that draw out proactive streaming capabilities of offline vision language models (VLMs), together with training pipelines that learn to further improve their proactive responsiveness. Across both settings, we find that combinations of sliding visual windows, online interaction memory, and structured response outputs consistently yield strongest proactive performance. Our experiments result in two models, with our BLIP-Live-Qwen model achieving state-of-the-art performance across four proactive streaming benchmarks, with gains up to 44% over baselines and outperforming multiple closed API models. Our code and data will be released publicly.
Figures & tables
| Dataset | Videos | Questions | Responses | Human-verified | |
|---|---|---|---|---|---|
| Content | Intervals | ||||
| MMDuet2 ( Wang et al., 2026 ) | 52,771 | 105,800 | 350,800 | No | No |
| VideoChat3-OL ( Li et al., 2026 ) | 191,000 | 617,183 | 617,183 | No | No |
| ESTP-IT ( Yu et al., 2025 ) | 1,166 | 80,000 | 80,000 | No | No |
| PARSE-Ego4D ( Abreu et al., 2025 ) | 5,515 | 18,360 | 18,360 | Yes | No |
| BLIP-Live-780K (Ours) | 177,789 | 177,789 | 784,944 | Yes | Yes |
| Policy | PAUC | Precision | PAUC-F |
|---|---|---|---|
| One perfect response; otherwise silent | 0.2575 | 1.0000 | 0.4095 |
| Same response + 1 out-of-window fire | 0.2575 | 0.5000 | 0.3399 |
| Same response + 5 out-of-window fires | 0.2575 | 0.1667 | 0.2024 |
| Same response + 10 out-of-window fires | 0.2575 | 0.0909 | 0.1344 |
| Same response + 100 out-of-window fires | 0.2575 | 0.0099 | 0.0191 |
| Model | Harness | [WEB] | [EGO] | [TV] | [VAD] | [Mean] | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.0 | 0.5 | 1.0 | 0.0 | 0.5 | 1.0 | 0.0 | 0.5 | 1.0 | 0.0 | 0.5 | 1.0 | 0.0 | 0.5 | 1.0 | ||
| Proprietary Models | ||||||||||||||||
| GPT-4.1 | ours | 57.92 | 60.19 | 62.31 | 33.15 | 34.33 | 35.36 | 31.30 | 32.56 | 33.58 | 28.00 | 28.80 | 29.49 | 37.59 | 38.97 | 40.19 |
| GPT-4.1-mini | ours | 54.36 | 57.19 | 59.81 | 21.63 | 22.07 | 22.44 | 24.07 | 24.82 | 25.42 | 19.92 | 20.36 | 20.74 | 30.00 | 31.11 | 32.10 |
| GPT-5.6-Sol | ours | 57.62 | 60.35 | 62.87 | 31.01 | 32.03 | 32.92 | 38.75 | 40.94 | 42.74 | 33.59 | 34.81 | 35.90 | 40.24 | 42.03 | 43.61 |
| Gemini-2.5-Pro | ours | 55.13 | 57.67 | 60.01 | 32.70 | 33.97 | 35.06 | 36.14 | 37.98 | 39.48 | 23.84 | 24.53 | 25.11 | 36.95 | 38.54 | 39.92 |
| Model | Harness | OR | AP | TRU | OL | OSC | EOL | EOSC | AR | All |
|---|---|---|---|---|---|---|---|---|---|---|
| Proprietary Models | ||||||||||
| GPT-4.1 | ours | 20.12 | 26.88 | 23.57 | 23.60 | 17.89 | 19.71 | 19.51 | 24.01 | 21.91 |
| GPT-4.1-mini | ours | 21.80 | 27.41 | 27.04 | 31.39 | 19.48 | 26.51 | 18.17 | 21.81 | 24.20 |
| Gemini-2.5-Pro | ours | 25.36 | 27.14 | 34.53 | 30.14 | 21.01 | 19.36 | 18.94 | 22.89 | 24.92 |
| Gemini-2.5-Flash | ours | 9.44 | 15.67 | 19.02 | 9.40 | 9.22 | 8.04 | 7.62 | 12.98 | 11.42 |
| Open-Weight Models (No Streaming Video Training) | ||||||||||
| Dataset / Method | WEB | EGO | TV | VAD | Mean |
|---|---|---|---|---|---|
| (a) Training-data ablation: fixed backbone, recipe, and harness | |||||
| Base model (no finetuning) | 54.24 | 39.29 | 34.47 | 43.75 | 42.94 |
| MMDuet2 data | 27.34 | 24.39 | 16.49 | 7.46 | 18.92 |
| VideoChat3 data | 55.15 | 39.71 | 40.53 | 45.72 | 45.28 |
| BLIP-Live-780K data | 57.48 | 46.05 | 43.11 | 49.67 | 49.08 |
| (b) Training-method comparison: fixed backbone, original data and recipes | |||||
| Split | Hold | PAUC-F | Fires | Precision (%) | PAUC (%) | Repeated (%) |
|---|---|---|---|---|---|---|
| WEB | With | 56.46 | 978 | 75.1 | 45.26 | 15 |
| WEB | Without | 32.90 | 2,840 | 25.8 | 45.39 | 65 |
| EGO | With | 43.80 | 1,955 | 42.2 | 45.52 | 40 |
| EGO | Without | 24.40 | 4,950 | 16.7 | 45.28 | 75 |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Harness | [WEB] | [EGO] | [TV] | [VAD] | [Mean] | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.0 | 0.5 | 1.0 | 0.0 | 0.5 | 1.0 | 0.0 | 0.5 | 1.0 | 0.0 | 0.5 | 1.0 | 0.0 | 0.5 | 1.0 | ||
| Proprietary Models | ||||||||||||||||
| GPT-4.1 | ours | 56.01 | 60.42 | 64.83 | 48.62 | 54.06 | 59.51 | 56.44 | 65.57 | 74.70 | 50.00 | 55.49 | 60.98 | 52.77 | 58.89 | 65.01 |
| GPT-4.1-mini | ours | 47.22 | 51.67 | 56.12 | 54.44 | 60.49 | 66.54 | 55.31 | 64.25 | 73.20 | 45.27 | 50.20 | 55.14 | 50.56 | 56.65 | 62.75 |
| GPT-5.6-Sol | ours | 52.54 | 57.26 | 61.97 | 49.44 | 55.04 | 60.63 | 52.15 | 60.90 | 69.65 | 43.28 | 47.58 | 51.87 | 49.35 | 55.19 | 61.03 |
| Gemini-2.5-Pro | ours | 51.93 | 56.63 | 61.33 | 46.32 | 51.80 | 57.29 | 54.05 | 63.20 | 72.35 | 43.79 | 48.65 | 53.50 | 49.02 | 55.07 | 61.12 |
| Dataset | [WEB] | [EGO] | [TV] | [VAD] | [Mean] | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.0 | 0.5 | 1.0 | 0.0 | 0.5 | 1.0 | 0.0 | 0.5 | 1.0 | 0.0 | 0.5 | 1.0 | 0.0 | 0.5 | 1.0 | |
| Base Model | 51.85 | 54.24 | 56.54 | 37.62 | 39.29 | 40.82 | 33.42 | 34.47 | 35.46 | 41.90 | 43.75 | 45.51 | 41.20 | 42.94 | 44.58 |
| MMDuet2 | 26.72 | 27.34 | 27.89 | 23.64 | 24.39 | 25.21 | 16.31 | 16.49 | 16.72 | 7.42 | 7.46 | 7.82 | 18.52 | 18.92 | 19.41 |
| VideoChat3 | 52.30 | 55.15 | 56.92 | 38.24 | 39.71 | 41.52 | 39.52 | 40.53 | 43.82 | 43.24 | 45.72 | 47.45 | 43.33 | 45.28 | 47.43 |
| BLIP-Live-780K | 53.13 | 57.48 | 61.41 | 43.77 | 46.05 | 48.13 | 40.71 | 43.11 | 45.25 | 46.56 | 49.67 | 52.55 | 46.04 | 49.08 | 51.84 |
| Method | [WEB] | [EGO] | [TV] | [VAD] | [Mean] | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.0 | 0.5 | 1.0 | 0.0 | 0.5 | 1.0 | 0.0 | 0.5 | 1.0 | 0.0 | 0.5 | 1.0 | 0.0 | 0.5 | 1.0 | |
| MMDuet2 | 31.82 | 32.76 | 33.54 | 28.95 | 29.57 | 30.70 | 21.36 | 22.20 | 22.54 | 12.88 | 13.03 | 13.42 | 23.75 | 24.39 | 25.05 |
| VideoChat3 | 34.67 | 35.86 | 36.88 | 30.12 | 30.36 | 31.34 | 37.97 | 40.75 | 42.84 | 24.28 | 24.67 | 24.88 | 31.76 | 32.91 | 33.99 |
| BLIP-Live-Qwen | 53.13 | 57.48 | 61.41 | 43.77 | 46.05 | 48.13 | 40.71 | 43.11 | 45.25 | 46.56 | 49.67 | 52.55 | 46.04 | 49.08 | 51.84 |
| Model | Explicit Proactive Task | Implicit Proactive Task | Contextual Q | Overall | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| OR | AP | TRU | OL | OSC | EOL | EOSC | AR | All | OFR | IFR | NAR | TU | All | ORC | TRC | All | ||
| Offline MLLMs Response-in-Last | ||||||||||||||||||
| LLaVA-OneVision | 7.2 | 11.5 | 4.9 | 10.0 | 4.9 | 6.9 | 5.6 | 3.2 | 6.8 | 3.8 | 6.3 | 11.6 | 29.8 | 12.9 | 10.8 | 5.7 | 8.2 | 8.7 |
| Qwen2-VL | 11.7 | 8.1 | 14.9 | 10.5 | 1.7 | 8.9 | 10.6 | 6.0 | 9.0 | 10.2 | 4.4 | 26.5 | 49.5 | 22.6 | 13.3 | 9.4 | 11.3 | 13.3 |
| MiniCPM-V | 12.3 | 12.6 | 10.7 | 13.7 | 8.6 | 7.5 | 11.9 | 5.5 | 10.4 | 11.8 | 9.2 | 36.0 | 55.3 | 28.1 | 32.6 | 25.4 | 29.0 | 18.1 |
| LLaVA-NeXT-Video | 8.3 | 9.4 | 7.4 | 10.2 | 7.8 | 7.4 | 10.3 | 5.6 | 8.3 | 6.4 | 6.7 | 21.1 | 45.9 | 20.0 | 10.1 | 9.8 | 9.9 | 11.9 |
| Method | WEB | EGO |
|---|---|---|
| Ours (Default) | 43.4 | 36.0 |
| StreamBridge Harness | 32.5 | 6.2 |
| Streaming Harness | 28.5 | 12.7 |
| General Harness Evolution | 40.2 | 31.4 |
| Split | Videos | Questions | Responses |
|---|---|---|---|
| Train | 176,021 | 176,021 | 777,278 |
| Validation | 1,768 | 1,768 | 7,666 |
| Total | 177,789 | 177,789 | 784,944 |
| Rater | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Split | Choice | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | Total | % |
| WEB | v1 | 0 | 0 | 1 | 0 | 0 | 4 | 3 | 0 | 2 | 2 | 12 | 6.0 |
| v2 | 20 | 20 | 19 | 20 | 16 | 17 | 18 | 20 | 18 | 18 | 188 | 94.0 | |
| EGO | v1 | 0 | 1 | 0 | 0 | 2 | 2 | 0 | 0 | 1 | 0 | 6 | 3.0 |
| v2 | 20 | 19 | 20 | 20 | 18 | 18 | 20 | 20 | 19 | 20 | 194 | 97.0 | |
| Component | Configuration |
|---|---|
| Decision rate | 1 Hz |
| Frames | Three most recent, , truncated at the video start |
| Timestamps | Absolute MM:SS immediately before each frame |
| Question | Standing query; a later query enters only at its reveal time |
| Memory | Last accepted response and its emission time. Acceptance replaces it; silence leaves it unchanged |
| Decode | Greedy, at most 192 new tokens |
| Task | Prompt Description |
|---|---|
| Harness tuning | Tune a frozen VLM harness so validation PAUC-F rises. Each round provides the current harness, one hypothesis, the editable surface, and prior scores and diagnostics. Propose the budgeted edits on that surface, and state the expected effect of each. Edits may only change the window, memory, prompt, or output protocol. Keep an edit only when validation PAUC-F improves. |
| Harness calibration | Calibrate the prompt and output schema of a frozen harness so validation PAUC-F rises. Each round provides the fixed harness, the previous candidate, and its score and diagnostics. Propose one edit, and state its expected effect. Only the prompt and output schema may change. Keep an edit only when validation PAUC-F improves. |
| Inference (Final) | Streaming video assistant. Each second, the input is the last accepted answer and its time (empty if none) plus the latest frames, each marked with absolute MM:SS . The newest stamp is now. Speak only when those frames show a useful change; otherwise return an empty list and leave the last answer as it stands. A spoken turn is one short factual sentence tagged with the current time in seconds. |
| Step | Rule |
|---|---|
| Timeline | 1 FPS frames with absolute MM:SS timestamps. Consecutive frames that share a caption are one event |
| States | First frame that supports the caption is Response ; later frames of that event are Hold ; every other frame is Silence |
| Chunks | At most 32 frames. A cut that would split a response from its hold is moved so they stay together. Original indices and timestamps are kept |
| Context | System instruction, standing query, and the last response that precedes the visual window |
| Extra chunks | Response-bearing chunks are kept, together with trailing and post-response chunks so sustained silence is seen |
| Response tokens | Every Response position, including its answer text |
| Setting | Value |
|---|---|
| Optimizer | AdamW |
| Learning rate | |
| Schedule | Cosine decay; 5% warmup |
| Training duration | One epoch |
| Per-device batch size | 1 |
| Gradient accumulation | 1 |
| Task | Prompt Description |
| Training and inference | Streaming assistant over frames labeled with absolute MM:SS times. Emit </Silence> when the frame is irrelevant, </Hold> when the event continues but no new answer is due, and </Response> plus one complete answer when the frames suffice. A prior answer may be attached; repeat it only when the new frames add evidence. Stay within what the frames show. |