SG-VLA: Learning Spatially-Grounded Vision-Language-Action Models for Mobile Manipulation

Ruisen Tu; Arth Shukla; Sohyun Yoo; Xuanlin Li; Junxi Li; Jianwen Xie; Hao Su; Zhuowen Tu

cs.ROMar 24, 2026

SG-VLA: Learning Spatially-Grounded Vision-Language-Action Models for Mobile Manipulation

Authors: Ruisen Tu, Arth Shukla, Sohyun Yoo, Xuanlin Li, Junxi Li, Jianwen Xie, Hao Su, Zhuowen Tu

Organizations: UC San Diego · Lambda, Inc.

Abstract

Vision-Language-Action (VLA) models show promise for robotic control, yet performance in complex household environments remains sub-optimal. Mobile manipulation requires reasoning about global scene layout, fine-grained geometry, and high-dimensional continuous actions, making standard imitation learning insufficient. We introduce a framework for learning spatially-grounded VLA models that strengthens perception and representation through auxiliary task co-training and multi-modal input enhancement. Our method addresses the challenge of controlling a 13-dimensional action space involving coordinated base motion, arm articulation, and gripper actuation. To enrich spatial understanding, the model incorporates multi-view RGB observations, depth cues, and short temporal history, providing perspectives of both global scene structure and local manipulation context. To improve representation quality, we co-train auxiliary decoders that reconstruct interpretable intermediate signals - including global robot position, joint configurations, grasp affordances, target-object relative pose, and segmentation masks - from shared visual-language features. These objectives provide dense supervision that encourages the backbone to develop spatially grounded, manipulation-aware latent representations. Through extensive evaluation on home rearrangement tasks, our approach achieves consistent improvements across picking, placing, opening, and closing operations, substantially outperforming direct imitation learning. Our findings suggest that spatial grounding through auxiliary and multi-modal learning provides a strong direction for scaling VLA models toward general-purpose domestic robots.

Figures & tables

Panel of Figure 1 — Figure 1: Multi-view segmentation data used for auxiliary task training. Each group shows (left) the original RGB observation, (middle) segmentation masks for all objects in the scene with different colors representing different object instances, and (right) the processed binary mask highlighting only the target object of interest. (a) Head camera view provides a global perspective of the manipulation scene. (b) Hand camera view offers a close-up perspective focused on the manipulation area. The target object masks are derived from the full segmentation by identifying the target object ID and creating binary masks to focus the model’s attention on the relevant manipulation target.

Figure 2: SG-VLA Architecture Overview. The model processes multi-modal inputs, including RGB images and normalized depth maps from head and hand cameras, alongside a natural language instruction. During training time, the latent representation from LLM is then passed to the decoders for auxiliary task predictions. Finally, the model predicts a 13-dimensional action vector, generated either directly by the LLM backbone or through a specialized Flow Matching action expert depending on task type. The action vector consists of the following: $\Delta X$ , a 3D vector representing the base’s pose (position+orientation) ; $\Delta z$ , a 1D scalar representing the change in the torso’s height ; $\Delta q$ , a 7D vector representing the change in the arm’s joint angles ; $\Delta G$ , a 2D vector representing the change in the gripper’s state (one for each finger).

Figure 3: Multi-stage Training Scheme. Stage 1: Decoder adaptation phase where auxiliary decoders are trained while gradient flow to the VLM backbone is blocked, allowing decoders to learn from fixed VLM representations. Stage 2: Joint refinement phase with full gradient flow enabled, co-training all auxiliary decoders with the VLM backbone. Stage 3: Action head training phase where the VLM backbone is frozen to train the flow matching action head in isolation.

Decoder	Type	Key Components
Global Pose	MLP	Proj (896 $\to$ 512), 3-layer MLP, Avg Pooling
Grasp Succ.	MLP	Proj (896 $\to$ 512), 3-layer MLP, Avg Pooling
Object Pose	MLP	Proj (896 $\to$ 512), 3-layer MLP, Quat Norm
Joint Pose	Transf.	12 Mask Tokens, 2-layer Transformer, Sine-Cos Pos
Mask	CNN	4-stage Transpose Conv, BatchNorm, GELU

Table 1: Detailed decoder specifications for the auxiliary training objectives used in SG-VLA.

Figure 4: Sample execution trajectories for six household manipulation tasks in ManiSkill-HAB evaluation . Each row shows a temporal sequence for a task performed by SG-VLA. The large frames show the global environment, while the vertical insets display head depth, head RGB, hand depth, and hand RGB observations from top to bottom.

	Pick	Place	Open		Close
Method	All Obj. $\uparrow$	All Obj. $\uparrow$	Fridge $\uparrow$	Drawer $\uparrow$	Fridge $\uparrow$	Drawer $\uparrow$	Avg. $\uparrow$
OpenVLA [ 16 ]	0.00	0.19	0.02	0.00	0.00	0.04	0.04
+ Multiview	0.06	0.35	0.14	0.38	0.00	0.53	0.24
+ Multiview + Depth	0.12	0.41	0.43	0.30	0.00	0.67	0.32
Base VLM + Multiview	0.06	0.53	0.60	0.30	0.63	0.93	0.52
+ Multiview + Depth	0.16	0.56	0.67	0.36	0.83	1.00	0.60

Table 2: Performance comparison between models taking different modalities of inputs. The Base VLM [ 2 ] uses DINOv2 + SigLIP as dual visual encoder and Qwen2.5-0.5B [ 32 ] as LLM backbone. Reported numbers are success rates for each task. The best results are bolded .

		Pick	Place	Open		Close
Progressive	Method	All Obj. $\uparrow$	All Obj. $\uparrow$	Fridge $\uparrow$	Drawer $\uparrow$	Fridge $\uparrow$	Drawer $\uparrow$	Avg.S.R. $\uparrow$
No	SG-VLA	0.16	0.56	0.67	0.36	0.83	1.00	0.60
	+ all	0.03	0.50	0.60	0.23	0.67	1.00	0.51
Yes	+ is_grasped	0.30	0.53	0.83	0.57	0.80	0.93	0.66
	+ qpos	0.23	0.67	0.87	0.70	0.90	0.90	0.71
	+ global pos	0.07	0.27	0.90	0.70	0.97	1.00	0.65

Table 3: Ablation study on co-training with each auxiliary task. SG-VLA represents the best model (Base VLM + Multiview + Depth) in Table 2 . All the models are train on SetTable subset. “+all” here includes “is_grasped + qpos + global pos.” The best results are bolded and the second-best results are underlined .

	SetTable		PrepareGrocery		TidyHouse
Method	Pick $\uparrow$	Place $\uparrow$	Pick $\uparrow$	Place $\uparrow$	Pick $\uparrow$	Place $\uparrow$	Avg. S.R. $\uparrow$
SG-VLA	0.16	0.56	0.10	0.33	0.07	0.40	0.27
+ seg + obj_pos	0.26	0.78	0.13	0.60	0.33	0.73	0.47

Table 4: Performance comparison between models trained with and without Segmentation and Object Position reconstruction task. The best results are bolded .

	Pick	Place	Open		Close
Method	All Obj. $\uparrow$	All Obj. $\uparrow$	Fridge $\uparrow$	Drawer $\uparrow$	Fridge $\uparrow$	Drawer $\uparrow$	Avg. $\uparrow$
SG-VLA	0.13	0.70	0.87	0.77	0.90	1.00	0.73
SG-VLA + action head	0.27	0.80	0.76	0.60	0.76	0.97	0.69

Table 5: Performance comparison between SG-VLA trained with and without flow matching action head. The best results are bolded .

Explore similar work

AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding
Jun 4, 2026Qize Yu, Jiadi You, Yuran Wang +10Language-Conditioned Robot Manipulation Robotic Manipulation
Peking University · Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong +1
Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents
Jul 9, 2026Yixian Zhang, Huanming Zhang, Feng Gao +14Long-Horizon Robotic Manipulation Language-Conditioned Robot Manipulation
Tsinghua University · Striding AI · Purdue University +4
$M^2$ -VLA: Boosting Vision-Language Models for Generalizable Manipulation via Layer Mixture and Meta-Skills
Apr 27, 2026Siyao Xiao, Yuhong Zhang, Zhifang Liu +9Language-Conditioned Robot Manipulation VLM Adaptation
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China. · PengCheng Laboratory, Shenzhen, China. · College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, China. +1

Jun 4, 2026J/K move · Enter open · S save

AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding

Qize Yu, Jiadi You, Yuran Wang, Jiaqi Liang+9

Peking University · Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong+1

Language-Conditioned Robot Manipulation Robotic Manipulation Vision-Language-Action Models Robot Learning

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models leverage the rich world knowledge of pretrained vision-language models (VLMs) to enable instruction-following robotic manipulation. However, the structural mismatch between VLM semantic spaces and embodied control policies often hinders the learning of precise perception--action mappings. To address this challenge, we propose AffordanceVLA, a unified framework that introduces structured affordance forecasting as a task-oriented intermediate representation to establish a more precise and robust perception--action mapping. Specifically, we progressively model manipulation priors through three complementary components: 1) Which2Act for object-centric grounding via visual latent prediction to suppress distractions; 2) Where2Act for 2D interaction localization via affordance map estimation; and 3) How2Act for 3D geometric reasoning to guide manipulation policies. These affordance cues provide spatially grounded, semantically conditioned, and action-coupled intermediate representations, thereby naturally bridging vision, language and action. We integrate these modules into a Mixture-of-Transformer (MoT) architecture with specialized experts and train the model using a three-stage training strategy with a progressive data curriculum. To overcome the scarcity of dense affordance labels in robotic datasets, we also develop a robust automated data augmentation pipeline. Extensive experiments on simulation and real-world demonstrate that AffordanceVLA achieves strong performance across diverse manipulation scenarios.

Abstract

Explore similar work

AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding

Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents

M2M^2M2-VLA: Boosting Vision-Language Models for Generalizable Manipulation via Layer Mixture and Meta-Skills

$M^2$ -VLA: Boosting Vision-Language Models for Generalizable Manipulation via Layer Mixture and Meta-Skills