Robotic Manipulation is Vision-to-Geometry Mapping: Vision-Geometry Backbones over Language and Video Models

Zijian Song; Qichang Li; Jiawei Zhou; Zhenlong Yuan; Tianshui Chen; Liang Lin; Guangrun Wang

cs.ROApr 14, 2026

Robotic Manipulation is Vision-to-Geometry Mapping: Vision-Geometry Backbones over Language and Video Models

Authors: Zijian Song, Qichang Li, Jiawei Zhou, Zhenlong Yuan, Tianshui Chen, Liang Lin, Guangrun Wang

Organizations: Sun Yat-sen University · LongCat, Meituan · X-Era AI Lab · Guangdong University of Technology ·

Abstract

At its core, robotic manipulation is a problem of vision-to-geometry mapping ( $f(v) \rightarrow G$ ). Physical actions are fundamentally defined by geometric properties like 3D positions and spatial relationships. Consequently, we argue that the foundation for generalizable robotic control should be a vision-geometry backbone, rather than the widely adopted vision-language or video models. Conventional VLA and video-predictive models rely on backbones pretrained on large-scale 2D image-text or temporal pixel data. While effective, their representations are largely shaped by semantic concepts or 2D priors, which do not intrinsically align with the precise 3D geometric nature required for physical manipulation. Driven by this insight, we propose the Vision-Geometry-Action (VGA) model, which directly conditions action generation on pretrained 3D representations. Specifically, VGA replaces conventional language or video backbones with a pretrained 3D world model, establishing a seamless vision-to-geometry mapping that translates visual inputs directly into physical actions. To further enhance geometric consistency, we introduce Progressive Volumetric Modulation and jointly train action and 3D property prediction to preserve geometric representations. Extensive experiments validate the effectiveness of our approach. Across simulation benchmarks, VGA outperforms leading VLA, 3D-VLA, and WAM baselines, including $π_{0.5}$ , OpenVLA-OFT, GeoVLA, and Motus. In real-world deployments, VGA surpasses $π_{0.5}$ under unseen viewpoints and accurately follows language instructions for target grasping. These results highlight that operating on native 3D representations, rather than relying primarily on language or video priors, offers a promising direction toward generalizable physical intelligence. Project page: https://hcplab-sysu.github.io/VisionGeometryActionModel.

Figures & tables

Figure 1. Robotic manipulation as vision-to-geometry mapping ( $f(v)\rightarrow G$ ). Physical actions like reaching, grasping, and orienting are inherently driven by geometric properties, such as 3D position, rotation, and spatial relationships. Therefore, we argue that a vision-geometry backbone provides a stronger foundation for generalizable robotic control than prevalent vision-language or video models.

Figure 2. Overview of our VGA model. (a) The left column compares our VGA framework with representative robot learning paradigms. VGA differs from them by leveraging a pretrained 3D world model as the backbone, providing native 3D representations aligned with physical actions. (b) The right column illustrates the workflow of the VGA model. Multimodal inputs are tokenized into a unified sequence and processed by a pretrained VGGT transformer with alternating attention. The resulting latent features are then mapped by task-specific heads to produce multimodal outputs, each with corresponding supervision.

Vision-Language-Action Baselines
Method	Spatial	Object	Goal	Long	Avg
Octo ( 2024 )	78.9%	85.7%	84.6%	51.1%	75.1%
OpenVLA ( 2024 )	84.7%	88.4%	79.2%	53.7%	76.5%
GR00T-N1 ( 2025 )	94.4%	97.6%	93.0%	90.6%	93.9%
UniVLA ( 2025b )	96.5%	96.8%	95.6%	92.0%	95.2%
$\pi_{0}$ ( 2024 )	90.0%	86.0%	95.0%	73.0%	86.0%

Table 1. The success rate comparison on the LIBERO benchmark. The best are highlighted by bold . The results demonstrate VGA’s strong manipulation precision.

	adjust bottle	pick dual bottles	turn switch	grab roller	place object basket
$\pi_{0}$	99%/95%	59%/37%	41%/42%	98%/94%	67%/70%
X-VLA	100 %/ 99 %	47%/36%	40%/ 61 %	100 %/ 100 %	44%/39%
VGA	100 %/ 99 %	70 %/ 56 %	51 %/56%	100 %/ 100 %	81 %/ 77 %

Table 2. The success rate comparison on the Robotwin2.0. Each entry reports Easy/Hard.

	VGA	OpenVLA-oft	$\pi_{0}$	WorldVLA	UniVLA	RIPT-VLA
light	92.1 %	85.8%	79.6%	29.4%	59.1%	87.9%
language	81.5 %	81.5 %	61.0%	44.2%	71.8%	80.1%
camera	86.0 %	59.7%	15.8%	0.3%	4.3%	58.3%

Table 3. The success rate comparison on the LIBERO-Plus. VGA (col 1) demonstrates robust generalization to unseen viewpoints, lighting shifts, and implicit-goal instructions.

Figure 3. Simulation rollouts and depth predictions on the LIBERO benchmark. The results show that VGA achieves precise physical manipulation with precise depth predictions.

Method	PVM	Training Objective	Initialization	Training Strategy	Spatial	Object	Goal	Long	Avg
VGA w/o PVM	-	Action + 3D	VGGT-Pretrained	LoRA	96.2	99.0	97.4	90.0	95.7
VGA w/o joint-training	+	Action	VGGT-Pretrained	LoRA	97.8	99.2	97.8	94.0	97.2
VGA zeroinit, full-parameter	+	Action + 3D	Random	Full	86.6	90.4	88.4	81.0	86.6
VGA zeroinit, lora	+	Action + 3D	Random	LoRA	8.4	10.8	6.8	0.0	6.4
VGA full-parameter	+	Action + 3D	VGGT-Pretrained	Full	92.6	92.0	88.2	75.4	87.1
VGA	+	Action + 3D	VGGT-Pretrained	LoRA	99.0	99.6	98.6	95.0	98.1

Table 4. Ablation study across all LIBERO suites. We evaluate the effects of PVM, joint 3D supervision, VGGT pretraining, and training strategy. The results demonstrate the contribution of each design component.

Figure 4. Configuration of real world evaluation. The platform is equipped with one wrist camera and two fixed cameras. The fixed cameras are used for in-distribution and out-of-distribution evaluation, respectively.

Method	In-Distribution Evaluation				Out-of-Distribution Evaluation
Method	Pick Cube	Press Button	Stack Cube	Average	Pick Cube	Press Button	Stack Cube	Average
ACT	40%	40%	30%	37%	10%	10%	0%	7%
OpenVLA	30%	25%	0%	18%	5%	5%	0%	3%
$\pi_{0.5}$	85 %	85 %	60 %	77 %	50%	55%	50 %	52%
VGA (Ours)	80%	85 %	60 %	75%	70 %	65 %	40%	58 %

Table 5. Real world evaluation of fine-grained manipulation. We report success rates under ID and OOD camera viewpoints. VGA achieves the highest average success rate in the OOD setting, demonstrating strong zero-shot cross-view generalization.

Figure 5. Fine-grained and language-grounded real-world manipulation. (a–d) show VGA performing fine-grained manipulation under in-distribution and unseen camera views. (e–g) show VGA following language instructions to grasp the specified target among three visually similar objects under varying layouts. These results demonstrate VGA’s fine-grained control across ID and OOD views and accurate language following, highlighting its generalization and real-world deployment potential.

Method	pick carrot	pick eggplant	pick cucumber	avg
$\pi_{0.5}$	75%	75%	80%	76.7%
VGA	80%	80%	80%	80.0%

Table 6. Real-world evaluation of language-grounded manipulation. We report success rates for grasping the instructed vegetable in unseen layouts containing all three visually similar objects.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6. Language-conditioned grasping under varying layouts. This figure presents the results of real-world grasping with three visually similar objects arranged in different layouts. Each row corresponds to a different spatial configuration, and the robot is instructed to pick a target object. VGA consistently identifies and grasps the correct object regardless of its position, demonstrating robust language grounding and reliable real-world manipulation performance.

Figure 7. Impact of joint training and LoRA rank. Part (a) presents the ablation study on LoRA rank. The results plateau around rank 64. Part (b) compares the convergence speed between models trained with and without joint training, evaluated using checkpoints from 1K to 5K steps on LIBERO-Spatial. Joint training leads to faster convergence and improved early-stage performance, indicating better data efficiency.

Method	Latency $\downarrow$	Frequency $\uparrow$
RT-2 ( Zitkovich et al., 2023 )	~200 ms	~5 Hz
OpenVLA ( Kim et al., 2024 )	~160 ms	~6 Hz
$\pi_{0}$ ( Black et al., 2024 )	~100 ms	~10 Hz
UVA ( Li et al., 2025a )	~230 ms	~4 Hz
WorldVLA ( Cen et al., 2025 )	~330 ms	~3 Hz
TraceVLA ( Zheng et al., 2024 )	~160 ms	~6 Hz

Appendix

Table 7. Inference latency. VGA achieves low latency of approximately 100 ms (~10 Hz), demonstrating its strong efficiency for real-time robotic control.

Figure 8. Depth prediction comparison between VGGT and VGA. Red boxes highlight regions where VGA significantly differs from VGGT. VGA produces more accurate depth estimates, particularly for the robot gripper, demonstrating the benefit of joint training in improving spatial understanding and depth prediction accuracy.

Module	Parameters	Learnable
Vision Encoder	730.9M	0.0M (frozen)
Language Encoder	1543.3M	0.0M (frozen)
Proprio Encoder	1.1M	1.1M
Vision Projector	27.6M	27.6M
Language Projector	2.2M	2.2M
Transformer Backbone	987.0M	214.1M

Appendix

Table 8. VGA parameter count. This table shows the detailed breakdown of the total and trainable parameters across different modules in VGA. By leveraging the pretrained 3D prior of VGGT, VGA significantly reduces the number of trainable parameters, enabling an efficient and expressive design.

hyperparameter	value
# GPUs	4 x NVIDIA A100-SXM4-80GB GPU
learning rate (LR)	2e-4
total batch size	32
training strategy	LoRA tuning, LoRA rank = 64
input images	1 third-person camera image, 1 wrist-mounted camera image
input image size	224 x 224 px

Appendix

Table 9. VGA hyperparameters for both LIBERO and real-world experiments.

hyperparameter	value
# GPUs	1 x NVIDIA A100-SXM4-80GB GPU
learning rate (LR)	1e-5
total batch size	8
training strategy	full parameter tuning
input images	1 third-person camera image, 1 wrist-mounted camera image
input image size	224 x 224 px

Appendix

Table 10. ACT hyperparameters for real-world experiments.

hyperparameter	value
# GPUs	1 x NVIDIA A100-SXM4-80GB GPU
learning rate (LR)	5e-4
total batch size	8
training strategy	LoRA tuning, LoRA rank = 32
input images	1 third-person camera image, 1 wrist-mounted camera image
input image size	224 x 224 px

Appendix

Table 11. OpenVLA hyperparameters for real-world experiments.

hyperparameter	value
# GPUs	1 x NVIDIA A100-SXM4-80GB GPU
learning rate (LR)	5e-5 peak LR (2K steps linear warmup)
total batch size	64
training strategy	full parameter tuning
input images	1 third-person camera image, 1 wrist-mounted camera image
input image size	224 x 224 px

Appendix

Table 12.

\pi_{0.5}

hyperparameters for real-world experiments.

Figure 9. Representative tasks from the four LIBERO suites: (a) LIBERO-Spatial, which focuses on spatial relationships, (b) LIBERO-Object, which evaluates object generalization, (c) LIBERO-Goal, which tests compositional task understanding, and (d) LIBERO-Long, which requires long-horizon sequential reasoning.

Figure 10. Initial state variations for evaluation tasks. This figure illustrates the three evaluation tasks under four types of initial condition variations. Despite these diverse and challenging variations, our VGA consistently achieves high manipulation success rates across all settings.

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,685 papers

GEAR-VLA: Learning Geometry-Aware Action Representations for Generalizable Robotic Manipulation
Jun 7, 2026Yuan Zhang, Shiqi Zhang, Yedong Shen +11Generalization in Robotic Manipulation Cross-Embodiment Robot Learning
Anhui University · University of Science and Technology of China · iFLYTEK
VistaVLA: Geometry- and Semantic-Aware 3D Gaussian-Grounded VLA for Robotic Manipulation
Jul 14, 2026Mohan Liu, Zhihao Gu, Xuanyu Chen +5Language-Conditioned Robot Manipulation 3D Spatial Reasoning
EmPACT Lab, Nanyang Technological University, Singapore · Institute for Infocomm Research (I2R), A*STAR, Singapore
G $^3$ VLA: Geometric Inductive Bias for Vision-Language-Action Models
Jun 23, 2026Yue Peng, Yongzhe Zhao, Artur Habuda +5Multi-View Learning Spatial Grounding for VLA Models
New York University Shanghai · Technical University of Denmark · MBZUAI - Mohamed bin Zayed University of Artificial Intelligence

Jun 7, 2026J/K move · Enter open · S save

GEAR-VLA: Learning Geometry-Aware Action Representations for Generalizable Robotic Manipulation

Yuan Zhang, Shiqi Zhang, Yedong Shen, Shuai Dong+10

Anhui University · University of Science and Technology of China · iFLYTEK

Generalization in Robotic Manipulation Cross-Embodiment Robot Learning Geometric Representation Learning Robotic Manipulation Vision-Language-Action Models Robot Learning

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models achieve strong benchmark performance but still struggle in real-world deployment with unseen objects, background shifts, and different robot embodiments. We argue that this stems from the lack of a unified geometry-aware manipulation representation, leaving existing VLAs vulnerable to low-level trajectory supervision, misaligned 3D features, and embodiment differences. To address this, we propose GEAR-VLA, a VLA framework for learning unified geometry-aware action representations for generalizable robotic manipulation. GEAR-VLA adopts coarse-to-fine action learning, where multi-source embodied pretraining equips the VLM with embodied reasoning and discrete action understanding before latent action tokens connect action semantics to a gradient-decoupled DiT continuous action expert. It further performs semantic-aligned 3D integration by aligning a trainable 3D spatial backbone with the VLA representation while freezing the original VLM-aligned visual pathway. To share this representation across robots, GEAR-VLA uses embodiment canonicalization, where embodiment-aware states and embodiment-invariant actions confine robot differences to the low-level interface. Extensive simulation and real-world experiments demonstrate strong generalization: GEAR-VLA achieves state-of-the-art performance on LIBERO, zero-shot LIBERO-Plus, and RoboTwin 2.0, reaches 85.9% success on AgileX and 81.0% on the pretraining-unseen LDT-01 embodiment, and obtains 90.1% success on a 6,360-trial universal grasping benchmark with 212 unseen objects. Code and models will be released at https://github.com/babynabeauty/GEAR-VLA.

Abstract

Robotic Manipulation

Explore similar work

GEAR-VLA: Learning Geometry-Aware Action Representations for Generalizable Robotic Manipulation

VistaVLA: Geometry- and Semantic-Aware 3D Gaussian-Grounded VLA for Robotic Manipulation

G3^33VLA: Geometric Inductive Bias for Vision-Language-Action Models

G $^3$ VLA: Geometric Inductive Bias for Vision-Language-Action Models