StreamPI: Streaming Multimodal Temporal Modeling for Vision-Language-Action Models

Zhe Liu; Jinghua Hou; Yuxiang Lu; Zhenya Yang; Xianzhe Fan; Junwei Luo; Junyi Li; Ruihua Han

cs.CVAug 26, 2026

StreamPI: Streaming Multimodal Temporal Modeling for Vision-Language-Action Models

Authors: Zhe Liu, Jinghua Hou, Yuxiang Lu, Zhenya Yang, Xianzhe Fan, Junwei Luo, Junyi Li, Ruihua Han,

Organizations: The University of Hong Kong · ACE Robotics

Abstract

Vision-Language-Action (VLA) models have demonstrated effectiveness in robot manipulation, yet state-of-the-art models such as pi0.5 operate under a single-frame paradigm, limiting their ability to retain past observations and develop precise spatial perception. In this paper, we propose StreamPI, a streaming multimodal temporal modeling framework that equips single-frame VLA with temporal reasoning capability without introducing any additional parameters. One core design is instruction-anchored temporal modeling. It treats each (visual observation, language instruction) pair as an atomic temporal unit: bidirectional attention within each pair enables cross-modal fusion, while causal attention across pairs preserves autoregressive streaming inference. This ensures the language instruction serves as a persistent semantic anchor throughout task execution. To bridge the gap between synchronous training and asynchronous real-robot deployment, we introduce a andom-interval streaming training strategy: a proper inter-frame interval (e.g., every 3 frames) enables faster and smoother action execution. Beyond this, randomizing the interval further improves robustness to frame-timing perturbations, supporting asynchronous deployment in practice. Furthermore, by leveraging the length extrapolation capability of the LLM backbone, StreamPI seamlessly inherits pretrained single-frame weights and supports flexible single-frame and multi-frame inference. Experiments on real-robot tasks spanning memory-dependent and precise perception scenarios, as well as the simulation benchmark LIBERO, demonstrate that StreamPI outperforms pi0.5 across diverse tasks.

Figures & tables

Figure 1: Comparison of three paradigms for VLA-based robot manipulation. (a) Single-frame VLA: Only the current observation is processed, lacking historical context and limiting temporal memory and spatial perception. (b) Window-based VLA: A window of $K$ frames is processed simultaneously, enriching temporal context at the cost of high computational overhead. (c) Streaming-based VLA (Ours): The model queries cached Key&Value representations from previous timesteps via a lightweight KV cache, achieving temporal memory and precise spatial perception.

Figure 2: The pipeline of StreamPI. To fully unleash the potential of multi-modal interaction, we use bidirectional attention for image-text pairs and causal attention for inter-frames with block-wise causal attention mask. We additional use the random interval sampling to improve the temporal robustness in the real-world deployment.

Figure 3: Visualization of real-world tasks and comparison of real-world performance. Left : Visualization of memory-dependent and precise perception-dependent real-world tasks. Right : Performance comparison of real-world tasks between ${\pi_{0.5}}$ and StreamPI.

Method	Libero-Spatial	Libero-Object	Libero-Goal	Libero-Long	Avg. Success Rate
Diffusion Policy Chi et al. (2025)	78.3	92.5	68.3	50.5	72.4
Octo Ghosh et al. (2024)	78.9	85.7	84.6	51.1	75.1
SpatialVLA Qu et al. (2025)	88.2	89.9	78.6	55.5	71.7
TraceVLA Zheng et al. (2024)	84.6	85.2	75.1	54.1	74.8
OpenVLA Kim et al. (2024)	84.7	88.4	79.2	53.7	75.9
CoT-VLA Zhao et al. (2025)	87.5	91.6	87.6	69.0	81.1

Table 1: Performance comparison on LIBERO ( Liu et al., 2023 ) . Success rates (%) are reported across four suites.

Method	1	2	3	4	5	Avg. Seq. Len
MemoryVLA Shi et al. (2025)	94.8	87.4	81.4	75.9	69.4	4.090
$\pi_{0.5}$ Intelligence et al. (2025)	94.2	88.7	85.7	83.2	79.5	4.313
StreamPI ( $T=5$ )	96.9	93.6	90.7	88.5	85.0	4.547

Table 2: Performance comparison on CALVIN ( Mees et al., 2022 ) . Success rates (%) at each sequence position and average sequence length are reported.

$T$ (Streaming Frames)	Intra-Pair	Inter-Pair	Spatial	Object	Goal	Long	Avg. Success Rate
$1$	Causal	—	97.4	97.8	96.8	91.0	95.8
$1$	Bidirectional	—	98.4	98.2	97.4	92.0	96.5
$3$	Causal	Causal	98.6	96.0	95.4	91.0	95.3
$3$	Bidirectional	Causal	98.8	98.6	98.6	93.8	97.5
$5$	Causal	Causal	98.0	97.6	96.8	90.6	95.5
$5$	Bidirectional	Causal	98.8	99.8	99.6	95.0	98.3

Table 3: Ablation on attention direction. Intra-pair denotes attention between image and text tokens within each temporal unit. Inter-pair denotes attention across temporal units and is always causal. Results are average success rates (%) on the LIBERO benchmark.

Training $\delta$	Training $T$	Test $T$	Spatial	Object	Goal	Long	Avg. Success Rate
$\pi_{0.5}$ ( $\delta{=}1$ )	1	1	98.4	98.2	97.4	92.0	96.5
Fixed ( $\delta{=}1$ )	3	3	98.0	98.4	96.8	92.2	96.4
Fixed ( $\delta{=}1$ )	5	5	98.4	97.4	98.6	93.4	97.0
Random $\delta$	3	3	98.8	98.6	98.6	93.8	97.5
Random $\delta$	5	5	98.8	99.8	99.6	95.0	98.3

Table 4: Effect of random-interval streaming training. Models are trained with fixed (

\delta{=}1

) or random interval (

\delta\sim\mathcal{U}[3,7]

) and evaluated at

\delta\in\{1,3,5\}

Training $T$	Test $T$	Spatial	Object	Goal	Long	Avg. Success Rate
$\pi_{0.5}$ (baseline)	1	98.4	98.2	97.4	92.0	96.5
$T{=}3$	1	98.0	98.8	97.8	93.4	97.0
$T{=}3$	3	98.8	98.6	98.6	93.8	97.5
$T{=}5$	1	97.4	98.8	98.4	93.6	97.1
	3	99.0	98.8	98.0	93.8	97.4
	5	98.8	99.8	99.6	95.0	98.3

Table 5: Cross-stream generalization. A model trained with

T{=}5

is evaluated at

T\in\{1,3,5\}

. Results are average success rates (%) on LIBERO-Long.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Streaming Frames	Inference Time (ms)
1	$94.4\pm 3.4$
3	$97.9\pm 5.1$
5	$103.6\pm 6.3$
8	$110.9\pm 10.2$
10	$117.9\pm 16.5$

Appendix

Table 6: The inference time of different streaming frames in streaming inference.

Model	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	S.R. (%)
$\pi_{0.5}$	1	0	1	0	0	1	1	1	1	0	1	0	0	0	0	46.7%
StreamPI	1	1	1	0	1	1	1	1	1	1	1	0	1	1	0	80.0%

Appendix

Table 7: Performance comparison of each trial on “Shell Game” task. S.R. denotes success rate.

Model	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	S.R. (%)
$\pi_{0.5}$	1	0	1	0	0	0	0	0	0	1	0	0	0	0	0	1	0	0	0	0	0	1	0	0	1	0	1	0	1	0	26.7
StreamPI	0	1	1	0	1	0	1	0	0	1	0	1	1	0	1	0	1	1	1	1	1	0	1	1	1	1	0	0	1	1	63.3

Appendix

Table 8: Performance comparison of each trial on “Pick up the Rolling Bottle” task. S.R. denotes success rate.

	M-F										M-M										F-M										S.R. (%)
Model	1	2	3	4	5	6	7	8	9	10	1	2	3	4	5	6	7	8	9	10	1	2	3	4	5	6	7	8	9	10	S.R. (%)
$\pi_{0.5}$	0	0	0	0	0	0	0	0	0	0	1	0	1	1	1	1	0	1	0	1	0	1	0	1	0	0	1	1	1	0	40.0
StreamPI	1	1	0	1	0	0	1	0	0	1	1	1	1	0	1	0	0	1	1	1	1	1	0	1	1	1	1	0	1	0	66.7

Appendix

Table 9: Performance comparison of each trial on “Pen Insertion into Narrow Bottle” task. S.R. denotes success rate.

Model	Center					Left-Far					Right-Far					Right-Near					Left-Near					S.R. (%)
$\pi_{0.5}$	1	0	1	1	1	0	0	0	1	0	0	0	0	0	0	1	1	1	1	1	1	1	1	1	1	60.0
StreamPI	1	1	1	1	1	1	1	1	0	1	1	1	0	1	1	1	1	1	1	1	1	1	1	1	1	92.0

Appendix

Table 10: Performance comparison of each trial on “Cup Insertion into Cup Sleeve” task. S.R. denotes success rate.

1	2	3	4	5	6	7	8	9	10	11	12	13	14	15
L $\rightarrow$ M	L $\rightarrow$ R	L $\rightarrow$ M $\rightarrow$ R	L $\rightarrow$ M $\rightarrow$ L	L $\rightarrow$ R $\rightarrow$ M	M $\rightarrow$ L	M $\rightarrow$ L $\rightarrow$ M	M $\rightarrow$ R	M $\rightarrow$ R $\rightarrow$ M	M $\rightarrow$ L $\rightarrow$ R	R $\rightarrow$ M	R $\rightarrow$ M $\rightarrow$ R	R $\rightarrow$ L	R $\rightarrow$ L $\rightarrow$ M	R $\rightarrow$ L $\rightarrow$ R

Appendix

Table 11: Each pattern of the Shell Game. ‘L’, ‘M’, and ‘R’ is the left, middle, and right position, respectively. A

\rightarrow

B denotes that move the cup that contains the target object from A to B.

Figure 5: Comparison of “Cup Insertion into Cup Sleeve”.

Figure 6: Comparison of “Pen Insertion into Narrow Bottle”.

Figure 7: Comparison of “Pick up the rolling bottle”.

Figure 8: The continuous qualitative results of “Shell Game”. For convenience, we use a red circle to indicate which cup contains the object.

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,154 papers

StreamingVLA: Streaming Vision-Language-Action Model with Action Flow Matching and Adaptive Early Observation
Mar 30, 2026Yiran Shi, Dongqi Guo, Tianchen Zhao +8Vision-Language-Action Models Efficient VLA Model Inference
Tsinghua University · Lenovo Group Ltd.
FlashVLA: Streaming Action Decoding for Fast and Asynchronous VLA Inference
Aug 27, 2026Zekai Li, Jiaming Tang, Zhijian LiuEfficient VLA Model Inference VLMs for Robotics
UC San Diego · MIT
ST- $π$ : Structured SpatioTemporal VLA for Robotic Manipulation
Apr 20, 2026Chuanhao Ma, Hanyu Zhou, Shihan Peng +3Language-Conditioned Robot Manipulation Robotics
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology · School of Computing, National University of Singapore · School of Computing, Macquarie University

Mar 30, 2026J/K move · Enter open · S save

StreamingVLA: Streaming Vision-Language-Action Model with Action Flow Matching and Adaptive Early Observation

Yiran Shi, Dongqi Guo, Tianchen Zhao, Feng Gao+7

Tsinghua University · Lenovo Group Ltd.

Vision-Language-Action Models Efficient VLA Model Inference

Open paper AskPDF arXiv

Vision-language-action (VLA) models have demonstrated exceptional performance in natural language-driven perception and control. However, the high computational cost of VLA models poses significant efficiency challenges, particularly for resource-constrained edge platforms in real-world deployments. However, since different stages of VLA (observation, action generation and execution) must proceed sequentially, and wait for the completion of the preceding stage, the system suffers from frequent halting and high latency. To address this, We conduct a systematic analysis to identify the challenges for fast and fluent generation, and propose enabling VLAs with the ability to asynchronously parallelize across VLA stages in a "streaming" manner. First, we eliminate the reliance on action chunking and adopt action flow matching, which learns the trajectory of action flows rather than denoising chunk-wise actions. It overlaps the latency of action generation and execution. Second, we design an action saliency-aware adaptive observation mechanism, thereby overlapping the latency of execution and observation. Without sacrificing performance, StreamingVLA achieves substantial speedup and improves the fluency of execution. It achieves a 2.4

\times

latency speedup and reduces execution halting by 6.5

\times

Abstract

Vision-Language-Action Models

Explore similar work

StreamingVLA: Streaming Vision-Language-Action Model with Action Flow Matching and Adaptive Early Observation

FlashVLA: Streaming Action Decoding for Fast and Asynchronous VLA Inference

ST-πππ: Structured SpatioTemporal VLA for Robotic Manipulation

ST- $π$ : Structured SpatioTemporal VLA for Robotic Manipulation