cs.ROMar 13, 2026
SaveRoboStream: Weaving Spatio-Temporal Reasoning with Memory in Vision-Language Models for Robotics
Organizations: Shenzhen International Graduate School, Tsinghua University · YuanxingGuangnian Robotics · China University of Mining and Technology · Shenzhen Technology University ·
Abstract
Enabling reliable long-horizon robotic manipulation is a crucial step toward open-world embodied intelligence. However, VLM-based planners treat each step as an isolated observation-to-action mapping, forcing them to reinfer scene geometry from raw pixels at every decision point while remaining unaware of how prior actions have reshaped the environment. Despite strong short-horizon performance, these systems lack the spatio-temporal reasoning required for persistent geometric anchoring and memory of action-triggered state transitions. Without persistent state tracking, perceptual errors accumulate across the execution horizon, temporarily occluded objects are catastrophically forgotten, and these compounding failures lead to precondition violations that cascade through subsequent steps. In contrast, humans maintain a persistent mental model that continuously tracks spatial relations and action consequences across interactions rather than reconstructing them at each instant. Inspired by this human capacity for causal spatio-temporal reasoning with persistent memory, we propose RoboStream, a training-free framework that achieves geometric anchoring through Spatio-Temporal Fusion Tokens (STF-Tokens), which bind visual evidence to 3D geometric attributes for persistent object grounding, and maintains causal continuity via a Causal Spatio-Temporal Graph (CSTG) that records action-triggered state transitions across steps. This design enables the planner to trace causal chains and preserve object permanence under occlusion without additional training or fine-tuning. RoboStream achieves 90.5% on long-horizon RLBench and 44.4% on challenging real-world block-building tasks, where both SoFar and VoxPoser score 11.1%, demonstrating that spatio-temporal reasoning and causal memory are critical missing components for reliable long-horizon manipulation.
Figures & tables
Figure 1: Qualitative comparison between existing methods [ 47 ] and RoboStream. This long-horizon task involves object occlusion and state restoration. Top (Existing Methods): Diffuse attention heatmaps and absent memory logs reveal two cascading failures: Ungrounded Spatial Perception misplaces the blue cube at Step 2, while Untracked Causal History leaves the planner unaware of the occluded cube at Step 4, resulting in an incorrect final state. Bottom (RoboStream): Concentrated attention heatmaps and persistent memory logs show that Spatio-Temporal Grounding enables accurate object placement at Step 2, while Causal Memory Tracking maintains awareness of the occluded cube at Step 4, enabling successful task completion.
Figure 2: Overview of the RoboStream Framework. The system processes multi-modal RGB-D inputs through three hierarchical stages: (1) Object-Centric Perception and Spatio-Temporal Token Fusion, where raw visual data is distilled into STF-Tokens by grounding visual evidence within 3D geometric primitives (centroids and Gaussian shapes); (2) 4D Causal Spatio-Temporal Graph Construction, which maintains persistent memory encoding object identities and state transitions across time steps; and (3) VLM-based Reasoning and Planning, where the VLM leverages both the CSTG and visual inputs to perform high-level task planning and precise robotic manipulation.
Figure 3: Qualitative results in real-world and simulated manipulation tasks. We evaluate on three challenging tasks: (a) Block Building, demanding precise bottom-to-top assembly toward a target configuration; (b) Block Disassembly, requiring structured top-to-bottom deconstruction and rearrangement; and (c) Block Hide and Restore, requiring causal memory maintenance under full occlusion. RoboStream consistently outperforms SoFar [ 47 ] and VoxPoser [ 26 ] , particularly in spatio-temporal perception and causal memory over extended horizons.
Figure 4: Performance comparison on zero-shot real-world manipulation tasks. We design 21 diverse tasks covering block building (Task A), block disassembly (Task B), and block hide and restore (Task C).
| Set | Policy | Pick Coke | Move Near |
| VA | RT-1-X [ 3 ] | 49.0 | 32.3 |
| RT-2-X [ 81 ] | 82.3 | 79.2 | |
| Octo-B [ 60 ] | 0.6 | 3.1 | |
| OpenVLA [ 34 ] | 54.5 | 47.7 | |
| SoFar [ 47 ] | 90.7 | 74.0 | |
| RoboStream-8B | 94.2 | 79.8 |
Table 1: Quantitative success rate comparison on the SIMPLER benchmark . We evaluate: (a) the Google Robot setup under Variant Aggregation (VA) and Visual Matching (VM) protocols, and (b) the WidowX + Bridge setup. For the WidowX tasks, we use the following abbreviations: Spoon (Put Spoon on Towel), Carrot (Put Carrot on Plate), Stack (Stack Green Block on Yellow Block), and Egg (Put Eggplant in Basket). “Zero” in the Data column indicates zero-shot evaluation.
| Tasks | SoFar [ 47 ] | VoxPoser [ 26 ] | RoboStream-8B | RoboStream-32B | RoboStream-235B |
| Bridge Between Towers | 52.0 | 8.0 | 76.0 | 88.0 | 88.0 |
| Cover Top Block with Box | 4.0 | 4.0 | 40.0 | 84.0 | 92.0 |
| Cover Bottom Block with Box | 0.0 | 0.0 | 16.0 | 68.0 | 96.0 |
| Place Blocks in Two Containers | 100.0 | 96.0 | 100.0 | 100.0 | 100.0 |
| Place Blocks in Two Cont. (Hard) | 4.0 | 4.0 | 24.0 | 76.0 | 88.0 |
| Stack Five Colors | 4.0 | 4.0 | 60.0 | 72.0 | 80.0 |
Table 2: Simulation evaluation of long-horizon manipulation on RLBench. All RoboStream variants are evaluated in a zero-shot manner without in-domain fine-tuning.
| STF- Tokens | CSTG | Bridge Between Towers | Cover Top Block | Cover Bottom Block | Place in Containers | Place in Containers (Hard) | Stack Five Colors | Stack Three Colors | Unstack then Stack | Average |
|---|---|---|---|---|---|---|---|---|---|---|
| ✗ | ✗ | 0.0 | 0.0 | 0.0 | 84.0 | 12.0 | 0.0 | 0.0 | 0.0 | 12.0 |
| ✓ | ✗ | 0.0 | 0.0 | 0.0 | 92.0 | 24.0 | 0.0 | 0.0 | 0.0 | 14.5 |
| ✗ | ✓ | 72.0 | 84.0 | 88.0 | 100.0 | 72.0 | 60.0 | 96.0 | 64.0 | 79.5 |
| ✓ | ✓ | 88.0 | 92.0 | 96.0 | 100.0 | 88.0 | 80.0 | 96.0 | 84.0 | 90.5 |
Table 5: Ablation study on RoboStream-235B. Success rates (%) across 8 long-horizon tasks, isolating contributions of CSTG and STF-Tokens.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Real-world experimental setup. (a) The Franka Research 3 robot arm used in our experiments. (b) The physical workspace equipped with the robot arm and an Intel RealSense D435i camera; the insets show the corresponding RGB view and depth map.
Figure 6: Assets for real-world experiments. A collection of 17 manipulation objects including geometric primitives, a triangular prism, and functional objects used across all task difficulty levels.
Figure 7: Long-horizon block manipulation (Easy). Execution of basic assembly and disassembly involving 2–3 objects, demonstrating fundamental spatial grounding and pick-and-place precision.
Figure 8: Long-horizon block manipulation (Medium). Intermediate execution requiring a strict sequence of 3–4 steps to manipulate 4 objects, verifying sequential planning and state tracking.
Figure 9: Long-horizon block manipulation (Hard). Extreme long-horizon execution involving 7 objects and 6–7 steps. The sequence highlights the model’s ability to maintain physical stability during complex “bottom-to-top” building and “top-to-bottom” disassembly without collapsing the structure.
Figure 10: Long-horizon Block Hide and Restore task (Occlusion). A stress-test for causal memory and object permanence. The robot hides the purple cube beneath a brown cup, executes intermediate distractor steps, and successfully retrieves the hidden object to restore the original configuration.
Figure 11: Visualization of short-horizon evaluation tasks. Execution examples from the SIMPLER environment (top) and RLBench (bottom). These tasks are designed to demonstrate the model’s zero-shot cross-embodiment generalization and its capability for high-precision spatial grounding in contact-rich scenarios.
Figure 12: Goal-Image Guided long-horizon Tasks. Execution sequences for five long-horizon tasks in RLBench where the target state is provided via a visual goal image. These tasks systematically evaluate the model’s ability to maintain spatial coherence and track multiple objects across extended dynamic trajectories.
Figure 13: Text-Guided long-horizon Tasks. Execution sequences in RLBench driven by natural language instructions. Tasks involving severe occlusion explicitly evaluate the CSTG’s capacity for causal memory and object permanence.
Table 6: Detailed results of real-world block building and block disassembly tasks.
| Task | Difficulty | Success Rate | ||||
| SoFar | VoxPoser | RoboStream-8B | RoboStream-32B | RoboStream-235B | ||
| Long-Horizon Memorize and Restore Task Initial State + Instruction | ||||||
| Hide the purple cube with the brown cup, then restore. | Hard | 0/3 | 0/3 | 2/3 | 2/3 | 3/3 |
| Hide the purple cube with the brown cup, then restore. | Hard | 0/3 | 0/3 | 1/3 | 2/3 | 3/3 |
| Hide the purple cube with the brown cup, place the yellow cube to the right of the cup, then restore. | Hard | 0/3 | 0/3 | 0/3 | 1/3 | 2/3 |
| Hide task Avg (%) | 0.0 | 0.0 | 33.3 | 55.6 | 88.9 | |
Table 7: Detailed results of real-world block hide and restore tasks.
Figure 14: Failure case distribution analysis of RoboStream from real-world experiments.
| Scale | STF- Tokens | CSTG | Bridge Between | Cover Bottom | Cover Top | Place in Cont. | Place in Cont. (H) | Stack Five | Stack Three | Unstack & Stack | Average |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 8B | ✗ | ✗ | 0.0 | 0.0 | 0.0 | 88.0 | 0.0 | 0.0 | 0.0 | 0.0 | 11.0 |
| ✓ | ✗ | 0.0 | 0.0 | 0.0 | 100.0 | 0.0 | 0.0 | 0.0 | 0.0 | 12.5 | |
| ✗ | ✓ | 68.0 | 8.0 | 24.0 | 100.0 | 12.0 | 40.0 | 88.0 | 24.0 | 45.5 | |
| ✓ | ✓ | 76.0 | 16.0 | 40.0 | 100.0 | 24.0 | 60.0 | 96.0 | 52.0 | 58.0 | |
| 32B | ✗ | ✗ | 0.0 | 0.0 | 0.0 | 48.0 | 0.0 | 0.0 | 0.0 | 0.0 | 6.0 |
| ✓ | ✗ | 0.0 | 0.0 | 0.0 | 56.0 | 12.0 | 0.0 | 0.0 | 0.0 | 8.5 |
Table 8: Comprehensive Ablation Results in Simulation. Success rates (%) across 8 long-horizon RLBench tasks. We compare the full RoboStream pipeline against variants removing Spatio-Temporal Fusion Tokens (w/o STF-Tokens), CSTG (w/o CSTG), or both (w/o Both) across 8B, 32B, and 235B scales. Data is averaged over 25 independent episodes per task. The 235B results correspond to the macro ablation study presented in the main text.
| Module | Step 1 | Step 2 | Step 3 | Step 4 | Step 5 | Step 6 | Step 7 | Avg |
| Perception | 0.24 | 0.22 | 0.30 | 0.25 | 0.25 | 0.22 | 0.24 | 0.25 |
| CSTG Construct | 0.14 | 0.15 | 0.17 | 0.17 | 0.17 | 0.16 | 0.17 | 0.16 |
| VLM Inference | 6.56 | 6.92 | 7.44 | 7.31 | 7.44 | 7.42 | 7.38 | 7.21 |
| Total Latency | 6.94 | 7.29 | 7.91 | 7.73 | 7.86 | 7.80 | 7.79 | 7.62 |
Table 9: Per-step latency breakdown on a hard 7-step disassembly case. Processing time (s) is reported for RoboStream-8B on one NVIDIA A100 GPU with a sliding window length of 3. Highlighted entries indicate the largest and second-largest values in each row.
| Backbone | GPU | Percep. | Mem. | VLM | Total |
| 8B | 1 A100 | 0.41 | 0.30 | 3.15 | 3.86 |
| 32B | 2 A100 | 0.42 | 0.34 | 7.85 | 8.61 |
| 235B | 8 A100 | 0.44 | 0.35 | 4.36 | 5.15 |
Table 10: Average per-step latency over all real-world cases. Latency is reported in seconds per planning step for different RoboStream backbone scales and NVIDIA A100 GPU configurations. Highlighted entries indicate the largest and second-largest values in each latency column.
| Task (RLBench) | SoFar | VoxPoser | RoboStream-8B |
| Close Drawer | 72.0 | 100.0 | 92.0 |
| Lamp Off | 0.0 | 100.0 | 76.0 |
| Meat Off Grill | 4.0 | 0.0 | 96.0 |
| Open Wine Bottle | 0.0 | 100.0 | 48.0 |
| Pick and Lift | 0.0 | 0.0 | 80.0 |
| Pick Up Cup | 8.0 | 12.0 | 68.0 |
Table 11: Short-horizon task evaluation results on RLBench.
| Method | Train-Free | Place | Place-H | Cover-B | Unstack | Avg. |
| SAM2Act [ 16 ] | ✗ | 100.0 | 20.0 | 0.0 | 0.0 | 30.0 |
| MemoryVLA [ 53 ] | ✗ | 96.0 | 8.0 | 0.0 | 0.0 | 26.0 |
| RoboStream-8B | ✓ | 100.0 | 24.0 | 16.0 | 52.0 | 48.0 |
| RoboStream-32B | ✓ | 100.0 | 76.0 | 68.0 | 76.0 | 80.0 |
| RoboStream-235B | ✓ | 100.0 | 88.0 | 96.0 | 84.0 | 92.0 |
Table 12: Memory baseline comparison on four RLBench long-horizon cases. Success rates (%) are reported for tasks requiring persistent object-state tracking. Place-H denotes the hard placement variant, Cover-B denotes covering the bottom block, and Unstack denotes unstack-then-stack manipulation.
Figure 15: System prompt of parsing and vqa reasoning.
Figure 16: System prompt of general manipulation task.
Figure 17: Visualization example of 6-DoF SpatialBench data samples.
Figure 18: Visualization example of Open6dor V2 data samples.
More in this topic
Long-Horizon Robotic Manipulation
Long-horizon robotic manipulation studies how robots plan, remember, and recover across extended sequences of physical tasks.
See all 413 papers