Physical AI is commonly formulated around an explicitly specified task: given a goal or instruction, a system perceives, reasons, and acts. In many real-world settings, however, important tasks emerge from continuously changing world states before a person explicitly formulates them. We study this complementary problem as Proactive Physical AI, where an AI system continuously observes the physical world, identifies situations that warrant intervention, and initiates appropriate actions at the right time. We develop ProAgent, an end-to-end system built around two components: a Jev-style proactive timing model (dubbed StreamingJev) that determines whether and when intervention is needed, and a Response Model that generates responses for people or invokes available device capabilities for physical execution. To support learning at scale, we develop an automated data pipeline that converts videos into supervision for proactive timing. Experiments demonstrate strong proactive performance under both explicit objectives and standing roles, combining more accurate intervention timing with fewer false triggers than the source-backbone baseline. Real-robot evaluations further demonstrate ProAgent's ability to identify emerging needs and translate them into appropriate physical actions.
Figures & tables
Figure 1: ProAgent architecture. Both models consume the causal world stream asynchronously; the Response Model can accumulate context while the Timing Model (StreamingJev) remains silent. A response decision enables output generation, not the ingestion of observations. Human-facing text is passed to external TTS. Physical outputs use typed function calls, while runtime validation and dispatch connect them to device providers. Feedback returns to the continuing observation loop.
Figure 2: Event-driven supervision with causal model inputs. Offline annotators use full videos to identify event context and intervention windows, while model inputs at time t contain only observations up to t . Event descriptions ground the construction of explicit-condition and role- or anomaly-conditioned timing supervision; the Timing Model learns causal decision sequences, with unknown intervals excluded from supervision.
Figure 3: Three complementary efficiency mechanisms. Incremental Caching reuses historical post-vision-encoder features and KV states subject to prefix and session validity. Dynamic Token Dropping (DTD) prunes redundant current-frame visual tokens using same-grid cross-frame similarity. Prefix-constrained Decoding terminates once all legal continuations imply the same binary decision and emits its canonical label. Token counts and the prefix tree are schematic; token counts do not imply a fixed retention ratio.
Figure 4: Recorded embodied interaction trajectories. Four frames per row show changes in the scene, robot interaction, and subsequent observations. (a) A person sits down holding their lower leg; the robot approaches and stays nearby. (b) A person slumps back onto a sofa; the robot approaches and remains nearby. (c) A visitor encounters the robot, moves with it toward a doorway, and enters. (d) A person preparing to leave returns to retrieve a phone before resuming departure. External-camera recordings document visible behavior; frame labels describe the observed actions and states. Timestamps are video positions, not model-trigger times. Quantitative performance is measured in the separate 60-case evaluation (Table 3 ).
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark / task
Input and constraint
Required decision
Metric
Full N
OVO / SSR
Video clip and specified procedural step
Whether that step is currently being performed
Accuracy
629
OVO / CRR
Video clip and question
Whether the available evidence is sufficient to answer
Accuracy
240
StreamingBench / PO
Sequential video observations and response condition
When to trigger the first response
First-trigger accuracy
250
Appendix
Table B.1: Tasks used for timing evaluation. Counts are scoring units in the complete benchmark releases.
OVO-Bench
StreamingBench
Timing Model
SSR
CRR
PO
Closed-source general-purpose models
GPT-6
73.93
66.25
73.20
Gemini 3.7 Flash
78.22
84.17
80.00
Claude Fable 5
82.67
74.58
71.60
Open-weight general-purpose models
Appendix
Table B.2: Intervention timing under benchmark-provided constraints. OVO-Bench measures response readiness at given checkpoints; StreamingBench measures first-response timing over sequential observations. Scores are percentages; higher is better. † denotes semantic output mapping. ∗ marks a result reported in the original Dispider paper ( Qian et al., 2025 ) .
Task
Total
Base correct
Ours correct
Δ (pp)
OVO SSR
629
328
475
+23.37
OVO CRR
240
105
200
+39.58
StreamingBench PO
250
29
143
+45.60
Appendix
Table B.3: Correct counts for StreamingJev versus Base. Both models are evaluated on the same probes or questions for each task. Differences are computed from the counts before rounding.
Benchmark
Evaluated units
Capability and metric
OVO-Bench
1,468 multiple-choice probes; 698 counting probes
Visual perception, backward tracing, and repetition counting; task accuracy and within-category macro accuracy.
Omni-MMI
800 records; 1,690 answers
Action prediction (AP), state grounding (SG), and multi-turn dependency (MD); record accuracy, requiring all answers correct for SG/MD.
StreamBench v0.3
307 records; 1,838 questions
Search, memory, and conversational answering; semantic accuracy per task and a four-task macro.
Appendix
Table C.1: Response-evaluation tasks and full-release coverage.
OVO-Bench
Omni-MMI
StreamBench v0.3
Response Model
Perception
Backward
AP
SG
MD
Core-task macro
Open-weight general-purpose models
Qwen3-VL-8B-Instruct
61.13
41.95
35.50
11.33
13.67
49.91
MiniCPM-V-4.6
62.30
45.27
38.00
9.00
9.33
60.38
LLaVA-OneVision-7B
64.33
44.11
23.50
5.00
5.67
52.27
Qwen3.5-4B (source backbone) S
66.98
53.81
35.00
13.00
11.67
50.54
Appendix
Table C.2: Response capability at benchmark-supplied answering opportunities (%). OVO columns are task macros; Omni SG/MD require all-step correctness; StreamBench reports the selected core-task macro. S marks the selected-clip configuration described below; dashes indicate unavailable scores.
Perception
Backward tracing
Model
OCR
ACR
ATR
STU
FPD
OJR
EPM
ASI
HLD
REC
Probes
149
109
116
178
101
184
297
148
186
698
MiniCPM-V-4.6
77.18
55.05
65.52
46.63
71.29
58.15
56.23
52.70
26.88
28.22
LLaVA-OneVision-7B
69.13
57.80
72.41
53.37
71.29
61.96
54.88
55.41
22.04
25.93
LiveStar-8B
60.40
51.38
63.79
43.82
71.29
59.24
54.55
61.49
15.05
0.00
Dispider
57.05
49.54
62.07
44.38
61.39
51.63
48.48
55.41
4.30
18.19
Appendix
Table C.3: OVO content-task breakdown (%). Counts refer to full-release probes. REC is separate from the perception and backward macros.
Model
OS
LM
SM
CI
KG
SF
Core macro
Questions
299
303
339
297
298
302
Qwen3-VL-8B
35.79
43.56
54.28
65.99
72.82
73.51
49.91
Qwen3.5-4B
37.12
43.23
53.10
68.69
75.84
71.85
50.54
MiniCPM-V-4.6
53.18
54.13
60.47
73.74
73.15
72.19
60.38
LLaVA-OneVision-7B
35.45
47.85
53.39
72.39
83.56
73.84
52.27
LiveStar-8B
32.11
36.30
48.97
62.63
44.30
67.55
45.00
Appendix
Table C.4: StreamBench v0.3 semantic accuracy under the causal visual-plus-ASR protocol (%). All models cover the same 307 records and 1,838 questions. Core macro averages OS, LM, SM, and CI before rounding.
Original question (Explicit)
Standing responsibilities (Role)
What are the different steps involved in preparing avocado in this process?
You are a cooking assistant supporting everyday food preparation. Attend to progress, ingredient and utensil use, and safety. Offer brief guidance, useful updates, or reminders when helpful, without repetitive interruptions.
What is happening with the white car?
You are a driving assistant helping a driver understand surrounding traffic. Attend to road conditions, vehicle movements, and safe passage. Provide concise, timely information without continuously narrating normal driving.
Appendix
Table D.1: Examples of the Explicit-to-Role adaptation. Questions and role prompts are reproduced from the constructed subset; videos, answers, and reference time intervals are unchanged.
Measure
Definition
Response latency
Time from the start of the processing cycle that produces a response to its first visible text, including input preparation and scheduling. For the dual-model system, the cycle starts with the triggering Timing request and ends with the Worker’s first visible text.
Decision frequency
Number of completed intervention decisions, including silence decisions, divided by effective playback duration. For the dual-model system, decisions are counted at the Timing Model. Completions after video end are recorded separately.
Completion latency
Time from the same processing-cycle start to the last response text; distinct from first-text latency and reference-time error.
Appendix
Table D.2: Runtime measures accompanying the paired evaluation. Response latency and completed decision frequency describe different parts of the streaming workload.
System
Explicit (Hz)
Role (Hz)
General-purpose: open weights
Qwen3.5-4B
0.4687
0.4780
Qwen3-VL-8B-Instruct
0.6215
0.6420
MiniCPM-V-4.6
0.8196
0.8297
General-purpose: closed source
GPT-6
0.0692
0.0610
Appendix
Table D.3: Completed decision frequency in the paired evaluation. Decision Hz counts completed intervention decisions per second of effective playback, including silence decisions. For the dual-model systems, decisions are counted at the Timing Model.
Category
Cases
Intervention function
Guidance
20
Timely information, instruction, and assistance during an ongoing activity.
Warning
20
Correction or warning in response to emerging errors and risks.
Interaction
20
Recognition of human needs and sustained proactive engagement.
Total
60
Appendix
Table E.1: Functional coverage of the embodied evaluation.
Figure E.1: Interpreting the cases through the ProPAI formulation. The columns relate each scenario to a possible assistance need, whether and when intervention may be appropriate, who or what could execute a response, and how subsequent observations should inform the next decision. The phone case highlights the person as a physical-task executor: a reminder can support object retrieval without requiring robot manipulation. The final column distinguishes an observed state change from evidence needed to establish completion. The intervention logic provides a conceptual interpretation of the observed actions and states in Figure 4 .
Source
Training sequences
Share (%)
COIN
58,811
26.22
HoloAssist
41,920
18.69
Ego-Exo4D
38,037
16.96
Anomaly videos
33,674
15.02
HACS
22,322
9.95
YouCook2
17,673
7.88
Appendix
Table G.1: Source distribution of the Timing Model training corpus. Counts refer to multi-turn decision sequences; percentages use the full training corpus as the denominator.
Setting
Value
Backbone
Qwen3.5-4B
Trainable modules
Language model and native output head
Frozen modules
Visual encoder and visual merger
Epochs
3
Optimizer
AdamW
Peak learning rate
2.5×10−6
Appendix
Table G.2: Training configuration of StreamingJev.
Representation
Fields
Role
Provider descriptor
schema_version , provider_id , capabilities
Identify the representation and device provider, and enumerate its operations. Optional metadata describes the device.
Capability descriptor
name , version , description , input_schema , output_schema
Specify a namespaced operation, its meaning, accepted arguments, and returned values.
Physical AI aims to build embodied agents that perceive the world, understand and reason about it, and decide how to act. Yet the field has focused primarily on the last component: the action model that maps observations to low-level controls. The prevailing training recipe can erode the perceptual and reasoning capabilities needed for robust behavior, leaving even strong action models vulnerable to scene perturbations and long-horizon tasks. We introduce HarnessPAI, a model- and embodiment-agnostic Harness framework for Physical AI that treats code as the executable and evolvable interface that organizes the underlying action primitive. The framework separates two timescales: within a rollout, it executes open-loop at the program level, with a fixed program guiding and checking execution; across rollouts, it evolves closed-loop, using execution feedback to revise the program and distill failures into reusable skills. Across desktop robot arms, household robots, a robot vacuum, and a legged walking agent, HarnessPAI demonstrates consistent improvements against baselines. Beyond the harness, the converged program also acts as a low-cost, reliable source of expert data: it generates training data for fine-tuning VLA and WAM models on tasks where it is hard for these models to obtain successful trajectories on their own. Our results suggest that the frontier of Physical AI depends not only on stronger action models, but also on executable harnesses that integrate perception, task understanding and reasoning, and action execution into a unified, verifiable, and feedback-driven system. Website: https://darwin-agent.github.io/HarnessPAI
Physical AI is experiencing rapid growth with frontier foundation models increasing its capabilities across general environments. Physical AI tasks are characterized by inference properties that are markedly different from digital AI. They consist of multiple rounds of inference and action execution, generating a chunk of actions in each inference round, and asynchronously interleaving inference and execution. This makes existing digital AI serving systems unsuited for physical AI; a shortcoming that is critical for enabling their wide adoption, considering their size and the scale of the robot fleets they have to serve. To fill this gap, we design Kairos, the first multi-robot serving system that makes the generate-execute loop a first-class citizen, with active involvement in the execution phase. Across a wide range of physical AI models and robots, Kairos reduces the average end-to-end task latency by 31.8--66.5% over state-of-the-art digital AI serving practices, with gains scaling with the robot fleet size.
We introduce Kairos, a regret-aware native world-action model stack for Physical AI. Kairos is motivated by the view that a physical world model should not aim to fully simulate all future pixels, but should learn and maintain the information most relevant to embodiment control: object state, spatial relations, contact conditions, task progress, action consequences, failure boundaries, and deployment uncertainty. Kairos establishes three model-side prerequisites toward this goal. First, it learns control-relevant information through a Cross-Embodiment Data Curriculum, which organizes open-world videos, human behavioral data, and robot interactions into an intervention-strength progression from passive physical observation to intentional behavior and embodied action grounding. Second, it maintains control-sufficient states through a unified understanding, generation, and prediction architecture equipped with Hybrid Linear Temporal Attention, where local, mid-range, and global temporal pathways support multi-timescale state maintenance under efficient inference. Third, it deploys these states through a Deployment-Aware System Co-Design, treating latency, memory footprint, and hardware compatibility as first-order constraints for future observation, action, and feedback loops. Experiments on embodied world-model benchmarks, world-action benchmarks, long-horizon generation, and inference-efficiency evaluation show that Kairos achieves superior performance while offering a favorable efficiency to capability trade-off.