cs.CVOct 12, 2026
SaveProactive Physical AI through Continuous Observation and Timely Intervention
Organizations: Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology · Guanxing Technology
Abstract
Physical AI is commonly formulated around an explicitly specified task: given a goal or instruction, a system perceives, reasons, and acts. In many real-world settings, however, important tasks emerge from continuously changing world states before a person explicitly formulates them. We study this complementary problem as Proactive Physical AI, where an AI system continuously observes the physical world, identifies situations that warrant intervention, and initiates appropriate actions at the right time. We develop ProAgent, an end-to-end system built around two components: a Jev-style proactive timing model (dubbed StreamingJev) that determines whether and when intervention is needed, and a Response Model that generates responses for people or invokes available device capabilities for physical execution. To support learning at scale, we develop an automated data pipeline that converts videos into supervision for proactive timing. Experiments demonstrate strong proactive performance under both explicit objectives and standing roles, combining more accurate intervention timing with fewer false triggers than the source-backbone baseline. Real-robot evaluations further demonstrate ProAgent's ability to identify emerging needs and translate them into appropriate physical actions.
Figures & tables
Figure 1: ProAgent architecture. Both models consume the causal world stream asynchronously; the Response Model can accumulate context while the Timing Model (StreamingJev) remains silent. A response decision enables output generation, not the ingestion of observations. Human-facing text is passed to external TTS. Physical outputs use typed function calls, while runtime validation and dispatch connect them to device providers. Feedback returns to the continuing observation loop.
Figure 2: Event-driven supervision with causal model inputs. Offline annotators use full videos to identify event context and intervention windows, while model inputs at time contain only observations up to . Event descriptions ground the construction of explicit-condition and role- or anomaly-conditioned timing supervision; the Timing Model learns causal decision sequences, with unknown intervals excluded from supervision.
Figure 3: Three complementary efficiency mechanisms. Incremental Caching reuses historical post-vision-encoder features and KV states subject to prefix and session validity. Dynamic Token Dropping (DTD) prunes redundant current-frame visual tokens using same-grid cross-frame similarity. Prefix-constrained Decoding terminates once all legal continuations imply the same binary decision and emits its canonical label. Token counts and the prefix tree are schematic; token counts do not imply a fixed retention ratio.
Figure 4: Recorded embodied interaction trajectories. Four frames per row show changes in the scene, robot interaction, and subsequent observations. (a) A person sits down holding their lower leg; the robot approaches and stays nearby. (b) A person slumps back onto a sofa; the robot approaches and remains nearby. (c) A visitor encounters the robot, moves with it toward a doorway, and enters. (d) A person preparing to leave returns to retrieve a phone before resuming departure. External-camera recordings document visible behavior; frame labels describe the observed actions and states. Timestamps are video positions, not model-trigger times. Quantitative performance is measured in the separate 60-case evaluation (Table 3 ).
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark / task | Input and constraint | Required decision | Metric | Full |
|---|---|---|---|---|
| OVO / SSR | Video clip and specified procedural step | Whether that step is currently being performed | Accuracy | 629 |
| OVO / CRR | Video clip and question | Whether the available evidence is sufficient to answer | Accuracy | 240 |
| StreamingBench / PO | Sequential video observations and response condition | When to trigger the first response | First-trigger accuracy | 250 |
Table B.1: Tasks used for timing evaluation. Counts are scoring units in the complete benchmark releases.
| OVO-Bench | StreamingBench | ||
| Timing Model | SSR | CRR | PO |
| Closed-source general-purpose models | |||
| GPT-6 | 73.93 | 66.25 | 73.20 |
| Gemini 3.7 Flash | 78.22 | 84.17 | 80.00 |
| Claude Fable 5 | 82.67 | 74.58 | 71.60 |
| Open-weight general-purpose models | |||
Table B.2: Intervention timing under benchmark-provided constraints. OVO-Bench measures response readiness at given checkpoints; StreamingBench measures first-response timing over sequential observations. Scores are percentages; higher is better. denotes semantic output mapping. marks a result reported in the original Dispider paper ( Qian et al., 2025 ) .
| Task | Total | Base correct | Ours correct | (pp) |
|---|---|---|---|---|
| OVO SSR | 629 | 328 | 475 | +23.37 |
| OVO CRR | 240 | 105 | 200 | +39.58 |
| StreamingBench PO | 250 | 29 | 143 | +45.60 |
Table B.3: Correct counts for StreamingJev versus Base. Both models are evaluated on the same probes or questions for each task. Differences are computed from the counts before rounding.
| Benchmark | Evaluated units | Capability and metric |
|---|---|---|
| OVO-Bench | 1,468 multiple-choice probes; 698 counting probes | Visual perception, backward tracing, and repetition counting; task accuracy and within-category macro accuracy. |
| Omni-MMI | 800 records; 1,690 answers | Action prediction (AP), state grounding (SG), and multi-turn dependency (MD); record accuracy, requiring all answers correct for SG/MD. |
| StreamBench v0.3 | 307 records; 1,838 questions | Search, memory, and conversational answering; semantic accuracy per task and a four-task macro. |
Table C.1: Response-evaluation tasks and full-release coverage.
| OVO-Bench | Omni-MMI | StreamBench v0.3 | ||||
| Response Model | Perception | Backward | AP | SG | MD | Core-task macro |
| Open-weight general-purpose models | ||||||
| Qwen3-VL-8B-Instruct | 61.13 | 41.95 | 35.50 | 11.33 | 13.67 | 49.91 |
| MiniCPM-V-4.6 | 62.30 | 45.27 | 38.00 | 9.00 | 9.33 | 60.38 |
| LLaVA-OneVision-7B | 64.33 | 44.11 | 23.50 | 5.00 | 5.67 | 52.27 |
| Qwen3.5-4B (source backbone) S | 66.98 | 53.81 | 35.00 | 13.00 | 11.67 | 50.54 |
Table C.2: Response capability at benchmark-supplied answering opportunities (%). OVO columns are task macros; Omni SG/MD require all-step correctness; StreamBench reports the selected core-task macro. marks the selected-clip configuration described below; dashes indicate unavailable scores.
| Perception | Backward tracing | |||||||||
| Model | OCR | ACR | ATR | STU | FPD | OJR | EPM | ASI | HLD | REC |
| Probes | 149 | 109 | 116 | 178 | 101 | 184 | 297 | 148 | 186 | 698 |
| MiniCPM-V-4.6 | 77.18 | 55.05 | 65.52 | 46.63 | 71.29 | 58.15 | 56.23 | 52.70 | 26.88 | 28.22 |
| LLaVA-OneVision-7B | 69.13 | 57.80 | 72.41 | 53.37 | 71.29 | 61.96 | 54.88 | 55.41 | 22.04 | 25.93 |
| LiveStar-8B | 60.40 | 51.38 | 63.79 | 43.82 | 71.29 | 59.24 | 54.55 | 61.49 | 15.05 | 0.00 |
| Dispider | 57.05 | 49.54 | 62.07 | 44.38 | 61.39 | 51.63 | 48.48 | 55.41 | 4.30 | 18.19 |
Table C.3: OVO content-task breakdown (%). Counts refer to full-release probes. REC is separate from the perception and backward macros.
| Model | OS | LM | SM | CI | KG | SF | Core macro |
|---|---|---|---|---|---|---|---|
| Questions | 299 | 303 | 339 | 297 | 298 | 302 | |
| Qwen3-VL-8B | 35.79 | 43.56 | 54.28 | 65.99 | 72.82 | 73.51 | 49.91 |
| Qwen3.5-4B | 37.12 | 43.23 | 53.10 | 68.69 | 75.84 | 71.85 | 50.54 |
| MiniCPM-V-4.6 | 53.18 | 54.13 | 60.47 | 73.74 | 73.15 | 72.19 | 60.38 |
| LLaVA-OneVision-7B | 35.45 | 47.85 | 53.39 | 72.39 | 83.56 | 73.84 | 52.27 |
| LiveStar-8B | 32.11 | 36.30 | 48.97 | 62.63 | 44.30 | 67.55 | 45.00 |
Table C.4: StreamBench v0.3 semantic accuracy under the causal visual-plus-ASR protocol (%). All models cover the same 307 records and 1,838 questions. Core macro averages OS, LM, SM, and CI before rounding.
| Original question (Explicit) | Standing responsibilities (Role) |
|---|---|
| What are the different steps involved in preparing avocado in this process? | You are a cooking assistant supporting everyday food preparation. Attend to progress, ingredient and utensil use, and safety. Offer brief guidance, useful updates, or reminders when helpful, without repetitive interruptions. |
| What is happening with the white car? | You are a driving assistant helping a driver understand surrounding traffic. Attend to road conditions, vehicle movements, and safe passage. Provide concise, timely information without continuously narrating normal driving. |
Table D.1: Examples of the Explicit-to-Role adaptation. Questions and role prompts are reproduced from the constructed subset; videos, answers, and reference time intervals are unchanged.
| Measure | Definition |
|---|---|
| Response latency | Time from the start of the processing cycle that produces a response to its first visible text, including input preparation and scheduling. For the dual-model system, the cycle starts with the triggering Timing request and ends with the Worker’s first visible text. |
| Decision frequency | Number of completed intervention decisions, including silence decisions, divided by effective playback duration. For the dual-model system, decisions are counted at the Timing Model. Completions after video end are recorded separately. |
| Completion latency | Time from the same processing-cycle start to the last response text; distinct from first-text latency and reference-time error. |
Table D.2: Runtime measures accompanying the paired evaluation. Response latency and completed decision frequency describe different parts of the streaming workload.
| System | Explicit (Hz) | Role (Hz) |
|---|---|---|
| General-purpose: open weights | ||
| Qwen3.5-4B | 0.4687 | 0.4780 |
| Qwen3-VL-8B-Instruct | 0.6215 | 0.6420 |
| MiniCPM-V-4.6 | 0.8196 | 0.8297 |
| General-purpose: closed source | ||
| GPT-6 | 0.0692 | 0.0610 |
Table D.3: Completed decision frequency in the paired evaluation. Decision Hz counts completed intervention decisions per second of effective playback, including silence decisions. For the dual-model systems, decisions are counted at the Timing Model.
| Category | Cases | Intervention function |
|---|---|---|
| Guidance | 20 | Timely information, instruction, and assistance during an ongoing activity. |
| Warning | 20 | Correction or warning in response to emerging errors and risks. |
| Interaction | 20 | Recognition of human needs and sustained proactive engagement. |
| Total | 60 |
Table E.1: Functional coverage of the embodied evaluation.
Figure E.1: Interpreting the cases through the ProPAI formulation. The columns relate each scenario to a possible assistance need, whether and when intervention may be appropriate, who or what could execute a response, and how subsequent observations should inform the next decision. The phone case highlights the person as a physical-task executor: a reminder can support object retrieval without requiring robot manipulation. The final column distinguishes an observed state change from evidence needed to establish completion. The intervention logic provides a conceptual interpretation of the observed actions and states in Figure 4 .
| Source | Training sequences | Share (%) |
|---|---|---|
| COIN | 58,811 | 26.22 |
| HoloAssist | 41,920 | 18.69 |
| Ego-Exo4D | 38,037 | 16.96 |
| Anomaly videos | 33,674 | 15.02 |
| HACS | 22,322 | 9.95 |
| YouCook2 | 17,673 | 7.88 |
Table G.1: Source distribution of the Timing Model training corpus. Counts refer to multi-turn decision sequences; percentages use the full training corpus as the denominator.
| Setting | Value |
|---|---|
| Backbone | Qwen3.5-4B |
| Trainable modules | Language model and native output head |
| Frozen modules | Visual encoder and visual merger |
| Epochs | 3 |
| Optimizer | AdamW |
| Peak learning rate |
Table G.2: Training configuration of StreamingJev.
| Representation | Fields | Role |
|---|---|---|
| Provider descriptor | schema_version , provider_id , capabilities | Identify the representation and device provider, and enumerate its operations. Optional metadata describes the device. |
| Capability descriptor | name , version , description , input_schema , output_schema | Specify a namespaced operation, its meaning, accepted arguments, and returned values. |
| Invocation | schema_version , request_id , provider_id , capability , arguments | Select a provider and versioned capability; bind the operation’s parameters. |
| Return | schema_version , request_id , success , result or error | Associate feedback with the request. Errors contain code , message , and optional details . |
Table G.3: Fields connecting capability discovery, invocation, and execution feedback.