An always-on robot faces an endless stream that never resets: instructions arrive and lapse, the scene changes, and its own past actions reshape what it must reason about. Today's action models are built for the opposite: a fixed instruction, no mid-task intervention, single-step reasoning. In an open-ended world a robot must watch a live stream for far-future cues, recall its own far-past actions, and act on them under dual-arm concurrency. We present ARMS (Always-on Robot in Multi-modal Streams), a deliberately simple streaming policy: a single pretrained π0.5 backbone augmented by three lightweight modules that turn live perception, embodied states, and the robot's own past actions into context the backbone reads before it acts. The modules update this context asynchronously, so watching and recalling never block acting and the two arms act at once. Rather than inventing new mechanisms, ARMS integrates these learned context providers with an agent-causal self-history that logs which arm did what, and when. To supervise them without extra annotation, we build ARMS Dataset, whose staged construction script itself labels every module from real dual-arm teleoperation. Trained on it, ARMS reaches 45% on the combined task against 28% for the strongest of our four main baselines, and ablations confirm the memory module, the embodied-state head, and asynchronous concurrency are each necessary.
Figures & tables
Figure 1: Timeline of a Streaming Action Task. On the timeline of an always-on stream, different tasks may overlap: a foreground task, a proactive task, and a retrospective query. Handling them requires the model to watch, recall, and act at the right moment with the right embodiment.
Figure 2: The ARMS architecture. A high-frequency perception path runs a gated visual trigger (does the guard’s target object, e.g. the banana held in memory, appear?) and only on a positive trigger wakes the low-frequency reasoning-and-action path: a single π0.5 backbone (Transformer + action expert) that reads the injected context (visual tokens, compressed memory, the language instruction, and the embodied-state head’s per-arm primitives and streams) and autoregresses the dual-arm action token. The three modules share one backbone and one context interface rather than acting as separate planners.
Figure 3: Qualitative results on the real robot, a single never-reset concurrent episode. Foreground row: clearing cups into the box. Below (columns initial/cue/act/outcome ): two proactive guards fire as a banana then a carrot appear and a free arm stows each; the retrospective “put the second one you stowed in the box” resolves to the carrot and boxes it under inter-arm occlusion.