ST-VLA: Enabling 4D-Aware Spatiotemporal Understanding for General Robot Manipulation
Authors: You Wu, Zixuan Chen, Cunxu Ou, Wenxuan Wang, Wenbo Huang, Lin Cao, Yangtao Chen, Weichao Qiu, +4 more
Organizations: School of Computer Science, Nanjing University · School of Intelligence Science and Technology, Nanjing University · School of Electronic Science and Engineering, Nanjing University · School of Instrument Science and Engineering, Southeast University · Noah’s Ark Lab, Huawei.
Robotic manipulation in open-world environments requires reasoning across semantics, geometry, and long-horizon action dynamics. Existing hierarchical Vision-Language-Action (VLA) frameworks typically use 2D representations to connect high-level reasoning with low-level control, but lack depth awareness and temporal consistency, limiting robustness in complex 3D scenes. We propose ST-VLA, a hierarchical VLA framework using a unified 3D-4D representation to bridge perception and action. ST-VLA converts 2D guidance into 3D trajectories and generates smooth spatial masks that capture 4D spatio-temporal context, providing a stable interface between semantic reasoning and continuous control. To enable effective learning of such representations, we introduce ST-Human, a large-scale human manipulation dataset with 14 tasks and 300k episodes, annotated with 2D, 3D, and 4D supervision via a semi-automated pipeline. Using ST-Human, we train ST-VLM, a spatio-temporal vision-language model that generates spatially grounded and temporally coherent 3D representations to guide policy execution. The smooth spatial masks focus on task-relevant geometry and stabilize latent representations, enabling online replanning and long-horizon reasoning. Experiments on RLBench and real-world manipulation tasks show that \method significantly outperforms state-of-the-art baselines, improving zero-shot success rates by 44.6% and 30.3%. These results demonstrate that offloading spatio-temporal reasoning to VLMs with unified 3D-4D representations substantially improves robustness and generalization for open-world robotic manipulation. Project website: https://oucx117.github.io/ST-VLA/.
Figures & tables
Figure 1: ST-VLM bridges the semantic-physical gap via unified 3D-4D spatio-temporal representations. (Left) Existing 2D-based VLMs face geometric ambiguity and temporal inconsistency due to the semantic-physical mismatch. (Right) Our ST-VLA utilizes unified 3D-4D representations with explicit trajectories and smooth spatial masks, ensuring robust long-horizon manipulation.
Figure 2: Overview of the ST-Human Dataset Construction and Unified 2D-3D-4D Task Generation.
Figure 3: The ST-VLA Pipeline. Given a global instruction and an RGB-D observation, the high-level ST-VLM generates sub-instructions and 2D trajectories. These are lifted to 3D and fused with SAM2 masks to form a unified 3D-4D representation, which conditions the low-level 3D policy for continuous action execution. Guidance is refreshed every H steps for replanning and robustness to disturbances.
Method
2D Tasks
3D Tasks
4D Tasks
RoboRefit
Where2Place
ST-Human-Pointing
CVBench
SAT
CRPE
ST-Human-Spatial
ST-Human-Depth
ST-Human-Planning
Closed-source models
GPT-5.2
16.65%
43.00%
20.50%
79.62%
66.00%
78.89%
68.00%
4.00%
88.00%
Gemini-3-flash
57.75%
87.00%
75.50%
83.94%
64.67%
80.21%
84.00%
6.00%
84.00%
Open-source models
PEEK-3B
10.05%
14.00%
3.00%
55.49%
52.67%
64.55%
50.00%
0.00%
88.00%
Table 1: VLM Performance Comparison on Selected 2D, 3D, and 4D Benchmarks. Best results are bolded.
Table 3: Success rates on long-horizon push-button tasks across three seeds, evaluated ST-VLA(3DFA) on seen buttons and unseen multi-step sequences.
Figure 4: Real-world experimental evaluation of ST-VLA . Left: Franka Emika Panda hardware setup. Right: Qualitative results across three dimensions: (1) zero-shot generalization to unseen object categories and geometries; (2) distractor robustness under task-irrelevant visual clutter; and (3) long-horizon chaining via sequential execution of multiple placement tasks.
Figure 6: Overview of the Annotation Software Workflow. Top: The Video Selection interface for temporal segmentation, displaying frame sequences and action management lists. Bottom: The Data Labeling interface featuring a central 3D point cloud editor (with manipulation gizmo), strictly synchronized with a 2D RGB preview window (bottom-left) to ensure spatial-visual alignment.
Figure 7: Qualitative Examples of Dataset Tasks. Top: ’Building Blocks’ task demonstrating multi-object spatial arrangement. Bottom: ’Scene Cleaning’ task showing interaction with irregular trash objects. Both examples highlight the diversity in background textures and lighting conditions.
Task Name
Total Vars
Seen
Unseen
Excluded Examples (Unseen)
Close Jar
20
15
5
Specific Colors (e.g., Azure, Violet)
Light Bulb In
20
15
5
Specific Colors (e.g., Black, White)
Put Groceries
9
7
2
Novel Objects (e.g., Crackers, Soup)
Push Buttons
78
78
0
None (Fine-grained Color Training)
Appendix
Table 4: Dataset Statistics and Variation Splits. Seen indicates variations used for training (100 episodes/task); Unseen indicates variations held out for generalization testing.
Figure 8: Qualitative Visualization of SAM2 Segmentation. Top: ’Close Jar’ task, showing precise masks for colored jars and loose lids. Middle: ’Light Bulb In’ task, isolating distinct components. Bottom: ’Put Groceries In Cupboard’ task, demonstrating robust segmentation in clutter.
Figure 9: Qualitative Comparison of Trajectory Generation. We visualize the 2D trajectories generated by our model (top row) versus ground truth human demonstrations (bottom row) across three tasks: Close Jar (Left), Light Bulb In (Middle), and Put Groceries In Cupboard (Right). The model successfully captures the critical keypoints and temporal geometry of the manipulation paths.
Task name
Language Template
Variations
Variation Type
pick_place
“Put the < color > block in the plate.”
4
Color, Object Type
(or “Put the < object > in the plate.”)
stack_cube
“Stack the < color1 > block on the < color2 > block.”
6
Color Pair
Appendix
Table 6: Properties of atomic tasks in real-world experiments.
Parameter
Value
Training Optimization
Model
Qwen/Qwen3-VL-4B-Instruct
Training Type
LoRA
Batch Size (Per Device)
6
Gradient Accumulation Steps
2
Total Batch Size
6×2×8=96
Appendix
Table 7: Hyperparameters for Qwen3-VL SFT Training
Parameter
Value
Training Optimization
Batch Size
64
Learning Rate
1×10−4
Backbone Learning Rate
1×10−6
Weight Decay
1×10−10
LR Scheduler
Constant
Appendix
Table 8: Hyperparameters for Peract Training and Evaluation
Method
RoboRefit
Where2Place
VABench-P
Part-Afford
ST-Human-Pointing
VABench-V ↓
RLBench ↓
ST-Human-Traj ↓
Closed-source models
GPT-5.2
16.65%
43.00%
60.33%
23.50%
20.50%
174.00
66.34
130.08
Gemini-3-flash
57.75%
87.00%
61.00%
86.00%
75.50%
122.00
72.82
42.36
Open-source models
Embodied-R1-3B
85.05%
66.83%
73.50%
44.74%
61.00%
72.10
65.89
78.85
PEEK-3B
10.05%
14.00%
17.33%
8.95%
3.00%
188.89
81.59
193.04
Appendix
Table 9: Evaluation on 2D Tasks. Metric types include: Box-Hit (RoboRefit, VAB-P), Mask-Hit (W2Pl, Afford), and Success Rate (ST-Human-Point). For trajectory tasks, we report RMSE ( ↓ ).
Method
CVBench
CRPE
SAT
Blink
ST-Human-Spatial
ST-Human-Depth
Val
Test
Ratio Acc ↑
MAD(cm) ↓
Closed-source models
GPT-5.2
79.62%
78.89%
66.56%
66.00%
49.08%
68.00%
4.00%
12.22
Gemini-3-flash
83.94%
80.21%
62.63%
64.67%
56.23%
84.00%
6.00%
13.82
Open-source models
Embodied-R1-3B
81.69%
74.78%
74.66%
70.00%
62.26%
56.00%
2.70%
54.00
Appendix
Table 10: Evaluation on 3D Tasks. Spatial tasks use multiple-choice accuracy. Depth estimation reports Ratio Accuracy (threshold 20%) and MAD.
Method
Next Act
Prev Act
Progress Estimation
Plan Error ↓
BERT ↑
BERT ↑
Step Acc ↑
Step MAE ↓
Status Acc ↑
RMSE
MAE
Closed-source models
GPT-5.2
0.89
0.88
12.00%
4.62
88.0%
199.35
186.77
Gemini-3-flash
0.83
0.82
16.00%
4.10
84.0%
283.47
245.50
Open-source models
Embodied-R1-3B
0.79
0.78
14.00%
4.48
78.0%
129.94
76.86
Appendix
Table 11: Evaluation on 4D Tasks (ST-Human-Planning). Evaluating instruction generation (BERTScore), progress estimation (Step/Status metrics), and future planning error.
Vision-language-action (VLA) models have achieved great success on general robotic tasks, but still face challenges in fine-grained spatiotemporal manipulation. Typically, existing methods mainly embed spatiotemporal knowledge into visual and action representations, and directly perform a cross-modal mapping for step-level action prediction. However, such spatiotemporal reasoning remains largely implicit, making it difficult to handle multiple sequential behaviors with explicit spatiotemporal boundaries. In this work, we propose ST-π, a structured spatiotemporal VLA model for robotic manipulation. Our model is guided by two key designs: 1) Spatiotemporal VLM. We encode 4D observations and task instructions into latent spaces, and feed them into the LLM to generate a sequence of causally ordered chunk-level action prompts consisting of sub-tasks, spatial grounding and temporal grounding. 2) Spatiotemporal action expert. Conditioned on chunk-level action prompts, we design a structured dual-generator guidance to jointly model spatial dependencies and temporal causality, thus predicting step-level action parameters. Within this structured framework, the VLM explicitly plans global spatiotemporal behavior, and the action expert further refines local spatiotemporal control. In addition, we propose a real-world robotic dataset with structured spatiotemporal annotations for fine-tuning. Extensive experiments have been conducted to demonstrate the effectiveness of our model. Our code link: https://github.com/chuanhaoma/ST-pi.
Chuanhao Ma, Hanyu Zhou, Shihan Peng +3
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology · School of Computing, National University of Singapore · School of Computing, Macquarie University
Vision-language-action (VLA) models integrate visual observations and language instructions to predict robot actions, demonstrating promising generalization in manipulation tasks. However, most existing approaches primarily rely on direct mappings from 2D visual inputs to action sequences, without explicitly modeling the underlying 3D spatial structure or temporal world dynamics. Such representations may limit spatial reasoning and long-horizon decision-making in dynamic environments. To address this limitation, we propose StemVLA, a novel framework that explicitly incorporates both future-oriented 3D spatial knowledge and historical 4D spatiotemporal representations into action prediction. First, instead of relying solely on observed images, StemVLA forecasts structured 3D future spatial-geometric world knowledge, enabling the model to anticipate upcoming scene geometry and object configurations. Second, to capture temporal consistency and motion dynamics, we feed historical image frames into a pretrained video-geometry transformer backbone to extract implicit 3D world representations, and further aggregate them across time using a temporal attention module, termed VideoFormer [20], forming a unified 4D historical spatiotemporal representation. By jointly modeling 2D observations, predicted 3D future structure, and aggregated 4D temporal dynamics, StemVLA enables more comprehensive world understanding for robot manipulation. Extensive experiments in simulation demonstrate that Stem-VLA achieves an average accuracy of 92.0% across the LIBERO subsets, and 86.0% on the long-horizon LIBERO-Long subset.
Jiasong Xiao, Yutao She, Kai Li +2
Image Processing Ricoh Software Research Center (Beijing) Co., Ltd. Beijing, China · Guanghua School of Management Peking University Beijing, China · Image Processing Beijing YZH Engineering Technology Co., Ltd Beijing, China +2
Current Vision-Language-Action (VLA) models primarily focus on mapping 2D observations to actions, but exhibit notable limitations in spatiotemporal perception and reasoning: 1) spatial representations often rely on additional sensors, introducing substantial computational overhead; 2) visual reasoning is typically limited to future-frame prediction, lacking alignment with the instruction-grounded scene and thus compromising spatiotemporal consistency. To address these challenges, we propose ConsisVLA-4D, a unified and efficient framework that enhances spatiotemporal consistency in 3D perception and 4D reasoning. Specifically, we design: 1) CV-Aligner, which ensures cross-view object semantic consistency by filtering instruction-relevant regions and aligning object identities across multiple viewpoints; 2) CO-Fuser, which guarantees cross-object spatial geometric consistency by eliminating spatial relation ambiguities between objects across views using compact latent representations. Building upon these, we introduce 3) CS-Thinker to achieve cross-scene spatiotemporal consistency as actions unfold. It learns implicit knowledge of local dynamics from object-semantic tokens of CV-Aligner and global depth from geometric tokens of CO-Fuser, thereby enhancing efficient visual reasoning under scene variations. Extensive experiments demonstrate that, benefiting from its efficient spatiotemporal consistency design, ConsisVLA-4D achieves 21.6% and 41.5% performance improvements, along with 2.3-fold and 2.4-fold inference speedups compared to OpenVLA on the LIBERO benchmark and real-world platforms, respectively.ConsisVLA-4D is open-sourced and publicly available at
Wei Li, Jizhihui Liu, Li Yixing +3
Harbin Institute of Technology, Shenzhen · State Key Laboratory of Mobile Network and Mobile Multimedia Technology, ZTE, China · Shenzhen Loop Area Institute