ST-VLA: Enabling 4D-Aware Spatiotemporal Understanding for General Robot Manipulation

You Wu; Zixuan Chen; Cunxu Ou; Wenxuan Wang; Wenbo Huang; Lin Cao; Yangtao Chen; Weichao Qiu

cs.ROMar 14, 2026

ST-VLA: Enabling 4D-Aware Spatiotemporal Understanding for General Robot Manipulation

Authors: You Wu, Zixuan Chen, Cunxu Ou, Wenxuan Wang, Wenbo Huang, Lin Cao, Yangtao Chen, Weichao Qiu, +4 more

Organizations: School of Computer Science, Nanjing University · School of Intelligence Science and Technology, Nanjing University · School of Electronic Science and Engineering, Nanjing University · School of Instrument Science and Engineering, Southeast University · Noah’s Ark Lab, Huawei.

Abstract

Robotic manipulation in open-world environments requires reasoning across semantics, geometry, and long-horizon action dynamics. Existing hierarchical Vision-Language-Action (VLA) frameworks typically use 2D representations to connect high-level reasoning with low-level control, but lack depth awareness and temporal consistency, limiting robustness in complex 3D scenes. We propose ST-VLA, a hierarchical VLA framework using a unified 3D-4D representation to bridge perception and action. ST-VLA converts 2D guidance into 3D trajectories and generates smooth spatial masks that capture 4D spatio-temporal context, providing a stable interface between semantic reasoning and continuous control. To enable effective learning of such representations, we introduce ST-Human, a large-scale human manipulation dataset with 14 tasks and 300k episodes, annotated with 2D, 3D, and 4D supervision via a semi-automated pipeline. Using ST-Human, we train ST-VLM, a spatio-temporal vision-language model that generates spatially grounded and temporally coherent 3D representations to guide policy execution. The smooth spatial masks focus on task-relevant geometry and stabilize latent representations, enabling online replanning and long-horizon reasoning. Experiments on RLBench and real-world manipulation tasks show that \method significantly outperforms state-of-the-art baselines, improving zero-shot success rates by 44.6% and 30.3%. These results demonstrate that offloading spatio-temporal reasoning to VLMs with unified 3D-4D representations substantially improves robustness and generalization for open-world robotic manipulation. Project website: https://oucx117.github.io/ST-VLA/.

Figures & tables

Figure 1: ST-VLM bridges the semantic-physical gap via unified 3D-4D spatio-temporal representations. (Left) Existing 2D-based VLMs face geometric ambiguity and temporal inconsistency due to the semantic-physical mismatch. (Right) Our ST-VLA utilizes unified 3D-4D representations with explicit trajectories and smooth spatial masks, ensuring robust long-horizon manipulation.

Figure 2: Overview of the ST-Human Dataset Construction and Unified 2D-3D-4D Task Generation.

Figure 3: The ST-VLA Pipeline. Given a global instruction and an RGB-D observation, the high-level ST-VLM generates sub-instructions and 2D trajectories. These are lifted to 3D and fused with SAM2 masks to form a unified 3D-4D representation, which conditions the low-level 3D policy for continuous action execution. Guidance is refreshed every $H$ steps for replanning and robustness to disturbances.

Closed-source models
Method	2D Tasks			3D Tasks					4D Tasks
Method	RoboRefit	Where2Place	ST-Human-Pointing	CVBench	SAT	CRPE	ST-Human-Spatial	ST-Human-Depth	ST-Human-Planning
GPT-5.2	16.65%	43.00%	20.50%	79.62%	66.00%	78.89%	68.00%	4.00%	88.00%
Gemini-3-flash	57.75%	87.00%	75.50%	83.94%	64.67%	80.21%	84.00%	6.00%	84.00%
Open-source models
PEEK-3B	10.05%	14.00%	3.00%	55.49%	52.67%	64.55%	50.00%	0.00%	88.00%

Table 1: VLM Performance Comparison on Selected 2D, 3D, and 4D Benchmarks. Best results are bolded.

	Method	Close Jar		Light Bulb In		Put Groceries		Average
	Method	Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen
Single-task	3DDA (Baseline)	71.9 $\pm 3.0$	44.4 $\pm 9.6$	20.8 $\pm 3.6$	25.9 $\pm 6.4$	33.3 $\pm 3.0$	00.0 $\pm 0.0$	42.0	23.4
	ST-VLA w/ 3DDA (Frozen)	98.2 $\pm 3.0$	66.7 $\pm 0.0$	37.5 $\pm 0.0$	25.9 $\pm 17.0$	42.1 $\pm 5.3$	5.6 $\pm 9.6$	59.3	32.7
	ST-VLA w/ 3DDA (FT)	100.0 $\pm 0.0$	100.0 $\pm 0.0$	45.8 $\pm 3.6$	81.5 $\pm 6.4$	54.4 $\pm 8.0$	16.7 $\pm 16.7$	66.7	66
	3DFA (Baseline)	61.4 $\pm 3.0$	38.9 $\pm 9.6$	22.9 $\pm 3.6$	11.1 $\pm 0.0$	5.3 $\pm 5.3$	00.0 $\pm 0.0$	29.9	16.7
	ST-VLA w/ 3DFA (Frozen)	68.4 $\pm 5.3$	44.4 $\pm 9.6$	41.7 $\pm 3.6$	11.1 $\pm 0.0$	12.3 $\pm 3.0$	00.0 $\pm 0.0$	40.8	18.5

Table 2: Results on Simulated Robot Manipulation Tasks. Background colors highlight ST-VLA variants.

Category	Success Rate (%)
Seen Tasks	97.4 $\pm 4.4$
LH 2-step (Unseen)	93.3 $\pm 11.5$
LH 3-step (Unseen)	100.0 $\pm 0.0$
Overall	97.3 $\pm 2.3$

Table 3: Success rates on long-horizon push-button tasks across three seeds, evaluated ST-VLA(3DFA) on seen buttons and unseen multi-step sequences.

Figure 4: Real-world experimental evaluation of ST-VLA . Left: Franka Emika Panda hardware setup. Right: Qualitative results across three dimensions: (1) zero-shot generalization to unseen object categories and geometries; (2) distractor robustness under task-irrelevant visual clutter; and (3) long-horizon chaining via sequential execution of multiple placement tasks.

Figure 5: Real-world zero-shot generalization results.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Overview of the Annotation Software Workflow. Top: The Video Selection interface for temporal segmentation, displaying frame sequences and action management lists. Bottom: The Data Labeling interface featuring a central 3D point cloud editor (with manipulation gizmo), strictly synchronized with a 2D RGB preview window (bottom-left) to ensure spatial-visual alignment.

Figure 7: Qualitative Examples of Dataset Tasks. Top: ’Building Blocks’ task demonstrating multi-object spatial arrangement. Bottom: ’Scene Cleaning’ task showing interaction with irregular trash objects. Both examples highlight the diversity in background textures and lighting conditions.

Task Name	Total Vars	Seen	Unseen	Excluded Examples (Unseen)
Close Jar	20	15	5	Specific Colors (e.g., Azure, Violet)
Light Bulb In	20	15	5	Specific Colors (e.g., Black, White)
Put Groceries	9	7	2	Novel Objects (e.g., Crackers, Soup)
Push Buttons	78	78	0	None (Fine-grained Color Training)

Appendix

Table 4: Dataset Statistics and Variation Splits. Seen indicates variations used for training (100 episodes/task); Unseen indicates variations held out for generalization testing.

Figure 8: Qualitative Visualization of SAM2 Segmentation. Top: ’Close Jar’ task, showing precise masks for colored jars and loose lids. Middle: ’Light Bulb In’ task, isolating distinct components. Bottom: ’Put Groceries In Cupboard’ task, demonstrating robust segmentation in clutter.

Figure 9: Qualitative Comparison of Trajectory Generation. We visualize the 2D trajectories generated by our model (top row) versus ground truth human demonstrations (bottom row) across three tasks: Close Jar (Left), Light Bulb In (Middle), and Put Groceries In Cupboard (Right). The model successfully captures the critical keypoints and temporal geometry of the manipulation paths.

Task name	Language Template	Variations	Variation Type
pick_place	“Put the $<$ color $>$ block in the plate.”	4	Color, Object Type
	(or “Put the $<$ object $>$ in the plate.”)
stack_cube	“Stack the $<$ color1 $>$ block on the $<$ color2 $>$ block.”	6	Color Pair

Appendix

Table 6: Properties of atomic tasks in real-world experiments.

Training Optimization
Parameter	Value
Model	Qwen/Qwen3-VL-4B-Instruct
Training Type	LoRA
Batch Size (Per Device)	6
Gradient Accumulation Steps	2
Total Batch Size	$6\times 2\times 8=96$

Appendix

Table 7: Hyperparameters for Qwen3-VL SFT Training

Training Optimization
Parameter	Value
Batch Size	64
Learning Rate	$1\times 10^{-4}$
Backbone Learning Rate	$1\times 10^{-6}$
Weight Decay	$1\times 10^{-10}$
LR Scheduler	Constant

Appendix

Table 8: Hyperparameters for Peract Training and Evaluation

Closed-source models
Method	RoboRefit	Where2Place	VABench-P	Part-Afford	ST-Human-Pointing	VABench-V $\downarrow$	RLBench $\downarrow$	ST-Human-Traj $\downarrow$
GPT-5.2	16.65%	43.00%	60.33%	23.50%	20.50%	174.00	66.34	130.08
Gemini-3-flash	57.75%	87.00%	61.00%	86.00%	75.50%	122.00	72.82	42.36
Open-source models
Embodied-R1-3B	85.05%	66.83%	73.50%	44.74%	61.00%	72.10	65.89	78.85
PEEK-3B	10.05%	14.00%	17.33%	8.95%	3.00%	188.89	81.59	193.04

Appendix

Table 9: Evaluation on 2D Tasks. Metric types include: Box-Hit (RoboRefit, VAB-P), Mask-Hit (W2Pl, Afford), and Success Rate (ST-Human-Point). For trajectory tasks, we report RMSE (

\downarrow

Closed-source models
Method	CVBench	CRPE	SAT		Blink	ST-Human-Spatial	ST-Human-Depth
Method	CVBench	CRPE	Val	Test	Blink	ST-Human-Spatial	Ratio Acc $\uparrow$	MAD(cm) $\downarrow$
GPT-5.2	79.62%	78.89%	66.56%	66.00%	49.08%	68.00%	4.00%	12.22
Gemini-3-flash	83.94%	80.21%	62.63%	64.67%	56.23%	84.00%	6.00%	13.82
Open-source models
Embodied-R1-3B	81.69%	74.78%	74.66%	70.00%	62.26%	56.00%	2.70%	54.00

Appendix

Table 10: Evaluation on 3D Tasks. Spatial tasks use multiple-choice accuracy. Depth estimation reports Ratio Accuracy (threshold 20%) and MAD.

Closed-source models
Method	Next Act	Prev Act	Progress Estimation			Plan Error $\downarrow$
Method	BERT $\uparrow$	BERT $\uparrow$	Step Acc $\uparrow$	Step MAE $\downarrow$	Status Acc $\uparrow$	RMSE	MAE
GPT-5.2	0.89	0.88	12.00%	4.62	88.0%	199.35	186.77
Gemini-3-flash	0.83	0.82	16.00%	4.10	84.0%	283.47	245.50
Open-source models
Embodied-R1-3B	0.79	0.78	14.00%	4.48	78.0%	129.94	76.86

Appendix

Table 11: Evaluation on 4D Tasks (ST-Human-Planning). Evaluating instruction generation (BERTScore), progress estimation (Step/Status metrics), and future planning error.

Explore similar work

ST- $π$ : Structured SpatioTemporal VLA for Robotic Manipulation
Apr 20, 2026Chuanhao Ma, Hanyu Zhou, Shihan Peng +3Language-Conditioned Robot Manipulation Robotics
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology · School of Computing, National University of Singapore · School of Computing, Macquarie University
StemVLA:An Open-Source Vision-Language-Action Model with Future 3D Spatial Geometry Knowledge and 4D Historical Representation
Feb 27, 2026Jiasong Xiao, Yutao She, Kai Li +2Long-Horizon Robotic Manipulation World Models for Robotics
Image Processing Ricoh Software Research Center (Beijing) Co., Ltd. Beijing, China · Guanghua School of Management Peking University Beijing, China · Image Processing Beijing YZH Engineering Technology Co., Ltd Beijing, China +2
ConsisVLA-4D: Advancing Spatiotemporal Consistency in Efficient 3D-Perception and 4D-Reasoning for Robotic Manipulation
May 6, 2026Wei Li, Jizhihui Liu, Li Yixing +3Multi-View Consistency Spatiotemporal Reasoning
Harbin Institute of Technology, Shenzhen · State Key Laboratory of Mobile Network and Mobile Multimedia Technology, ZTE, China · Shenzhen Loop Area Institute

Apr 20, 2026J/K move · Enter open · S save

ST- $π$ : Structured SpatioTemporal VLA for Robotic Manipulation

Chuanhao Ma, Hanyu Zhou, Shihan Peng, Yan Li+2

School of Artificial Intelligence and Automation, Huazhong University of Science and Technology · School of Computing, National University of Singapore · School of Computing, Macquarie University

Language-Conditioned Robot Manipulation Robotics Robotic Manipulation Vision-Language-Action Models

Open paper AskPDF arXiv

Vision-language-action (VLA) models have achieved great success on general robotic tasks, but still face challenges in fine-grained spatiotemporal manipulation. Typically, existing methods mainly embed spatiotemporal knowledge into visual and action representations, and directly perform a cross-modal mapping for step-level action prediction. However, such spatiotemporal reasoning remains largely implicit, making it difficult to handle multiple sequential behaviors with explicit spatiotemporal boundaries. In this work, we propose ST-

π

, a structured spatiotemporal VLA model for robotic manipulation. Our model is guided by two key designs: 1) Spatiotemporal VLM. We encode 4D observations and task instructions into latent spaces, and feed them into the LLM to generate a sequence of causally ordered chunk-level action prompts consisting of sub-tasks, spatial grounding and temporal grounding. 2) Spatiotemporal action expert. Conditioned on chunk-level action prompts, we design a structured dual-generator guidance to jointly model spatial dependencies and temporal causality, thus predicting step-level action parameters. Within this structured framework, the VLM explicitly plans global spatiotemporal behavior, and the action expert further refines local spatiotemporal control. In addition, we propose a real-world robotic dataset with structured spatiotemporal annotations for fine-tuning. Extensive experiments have been conducted to demonstrate the effectiveness of our model. Our code link: https://github.com/chuanhaoma/ST-pi.

Abstract

Explore similar work

ST-πππ: Structured SpatioTemporal VLA for Robotic Manipulation

StemVLA:An Open-Source Vision-Language-Action Model with Future 3D Spatial Geometry Knowledge and 4D Historical Representation

ConsisVLA-4D: Advancing Spatiotemporal Consistency in Efficient 3D-Perception and 4D-Reasoning for Robotic Manipulation

ST- $π$ : Structured SpatioTemporal VLA for Robotic Manipulation