3D-Mix for VLA: A Plug-and-Play Module for Integrating VGGT-based 3D Information into Vision-Language-Action Models

Bin Yu; Shijie Lian; Xiaopeng Lin; Zhaolong Shen; Yuliang Wei; Haishan Liu; Changti Wu; Hang Yuan

cs.ROMar 25, 2026

3D-Mix for VLA: A Plug-and-Play Module for Integrating VGGT-based 3D Information into Vision-Language-Action Models

Authors: Bin Yu, Shijie Lian, Xiaopeng Lin, Zhaolong Shen, Yuliang Wei, Haishan Liu, Changti Wu, Hang Yuan, +3 more

Organizations: HIT · ZGCA · HUST · HKUST(GZ) · BUAA · ECNU · ZGCI · DeepCybo

Abstract

Vision-Language-Action (VLA) models leverage Multimodal Large Language Models (MLLMs) for robotic control, but recent studies reveal that MLLMs exhibit limited spatial intelligence due to training predominantly on 2D data, resulting in inadequate 3D perception for manipulation tasks. While recent approaches incorporate specialized 3D vision models such as VGGT to enhance spatial understanding, they employ diverse integration mechanisms without systematic investigation, leaving the optimal fusion strategy unclear. We conduct a comprehensive pilot study comparing nine VGGT integration schemes on standardized benchmarks and find that semantic-conditioned gated fusion, which adaptively balances 2D semantic and 3D geometric features based on task context, achieved the strongest performance among all nine evaluated fusion schemes in our pilot study. We present 3D-Mix, a plug-and-play module that integrates into diverse VLA architectures (GR00T-style and $π$ -style) without modifying existing MLLM or action expert components. Experiments across six MLLM series (nine model variants, 2B--8B parameters) on SIMPLER and LIBERO show that 3D-Mix delivers consistent performance gains, averaging +7.0% on the out-of-domain (OOD) SIMPLER benchmark across all nine GR00T-style variants, establishing a principled approach for enhancing spatial intelligence in VLA systems.

Figures & tables

Figure 1: Overview of nine VGGT fusion schemes evaluated in our pilot study. Each scheme integrates VGGT-derived 3D geometric features into the VLA pipeline at a different location or via a different mechanism, ranging from early token-level injection to training-time alignment and cross-attention fusion.

Method	SIMPLER Tasks				SIMPLER Avg	LIBERO Avg
Method	Stack Green	Put Carrot	Put Spoon	Put Eggplant	SIMPLER Avg	LIBERO Avg
Base (Qwen3-VL-4B-Instruct)	22.92	57.29	62.50	88.54	57.81	96.5
+ AE Fusion	0.00	0.00	0.00	12.50	3.13	97.40
+ 3D-Tokens	26.04	53.12	76.04	67.79	56.25	97.64
+ Early Fusion	5.21	40.62	48.96	83.33	44.53	86.45
+ Visual Fusion	0.00	3.12	1.04	14.58	4.69	73.40

Table 1: Comparison of nine VGGT fusion schemes on the SIMPLER (out-of-domain, real-to-sim) and LIBERO (in-domain, multi-task) benchmarks.

Figure 2: Integration of 3D-Mix into GR00T-style and $\pi$ -style VLA Architectures

Method	SIMPLER Tasks				SIMPLER Avg	LIBERO Avg
Method	Stack Green	Put Carrot	Put Spoon	Put Eggplant	SIMPLER Avg	LIBERO Avg
Qwen3-VL-4B-Instruct	22.92	57.29	62.50	88.54	57.81	96.50
+ 3D-Mix	30.21	61.46	86.46	94.79	68.23 $\uparrow$ 10.42	98.05 $\uparrow$ 1.55
Qwen3-VL-8B-Instruct	31.50	41.70	83.30	77.10	58.40	97.30
+ 3D-Mix	33.30	53.12	95.83	70.83	63.27 $\uparrow$ 4.87	98.15 $\uparrow$ 0.85
Qwen2.5-VL-3B-Instruct	8.33	36.46	58.33	77.09	45.05	97.30

Table 2: Performance comparison of various MLLM backbones with and without 3D-Mix on GR00T-style VLA architecture across SIMPLER and LIBERO benchmarks.

Method	SIMPLER Tasks				SIMPLER Avg	LIBERO Avg
Method	Stack Green	Put Carrot	Put Spoon	Put Eggplant	SIMPLER Avg	LIBERO Avg
Qwen3-VL-4B-Instruct	2.08	42.71	43.75	54.17	35.68	96.50
+ 3D-Mix	3.12	41.67	66.67	58.33	42.45 $\uparrow$ 6.77	97.20 $\uparrow$ 0.70
Qwen3-VL-2B-Instruct	4.17	32.29	36.46	82.29	38.80	96.60
+ 3D-Mix	10.42	36.46	44.44	85.42	44.18 $\uparrow$ 5.38	97.55 $\uparrow$ 0.95
Qwen2.5-VL-3B-Instruct	8.33	47.92	53.12	61.46	42.71	95.10

Table 3: Performance comparison of various MLLM backbones with and without 3D-Mix on

\pi

-style VLA architecture across SIMPLER and LIBERO benchmarks.

Figure 3: Ablation study results. Each subfigure corresponds to one ablation experiment: (a) VGGT Frozen vs. Trainable. (b) 3D Information Sensitivity. (c) Sparse Layer Fusion Depth ( $\pi$ -style).

Explore similar work

3DVLA: Enhancing Vision-Language-Action Models via 3D Spatial and Instance Understanding
May 28, 2026Zhongyu Xia, Yousen Tang, Bingqing Wei +13D Spatial Reasoning Robotic Manipulation
Wangxuan Institute of Computer Technology, Peking University · This work was done as an intern at PKU.
VistaVLA: Geometry- and Semantic-Aware 3D Gaussian-Grounded VLA for Robotic Manipulation
Jul 14, 2026Mohan Liu, Zhihao Gu, Xuanyu Chen +5Language-Conditioned Robot Manipulation 3D Spatial Reasoning
EmPACT Lab, Nanyang Technological University, Singapore · Institute for Infocomm Research (I2R), A*STAR, Singapore
Mind-VLA: Instruction-Aware Spatial Representation Alignment for Vision-Language-Action Models
Aug 5, 2026Xingyu Ding, Yuzhong Zhao, Yang Wu +4Robotic Manipulation Vision-Language-Action Models
Nanjing University, Nanjing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

May 28, 2026J/K move · Enter open · S save

3DVLA: Enhancing Vision-Language-Action Models via 3D Spatial and Instance Understanding

Zhongyu Xia, Yousen Tang, Bingqing Wei, Yongtao Wang

Wangxuan Institute of Computer Technology, Peking University · This work was done as an intern at PKU.

3D Spatial Reasoning Robotic Manipulation Vision-Language-Action Models 3D Scene Understanding

Open paper AskPDF arXiv

Vision-Language-Action models have achieved remarkable progress in robotic manipulation, yet they suffer from a critical limitation: a lack of 3D scene understanding. This deficiency manifests as three intertwined challenges: weak extraction of 3D spatial positions without enforcing multi-view consistency, inadequate 3D instance understanding, and fragile reasoning under occlusion. Although mature 3D perception methods exist, their direct integration into VLA pipelines is hindered by architectural incompatibility and by heavy reliance on costly instance-level annotations. To address the above challenges, we propose 3DVLA, a plug-and-play framework that injects robust 3D reasoning into pretrained VLAs without requiring extra manual labels or discarding VLM priors. Specifically, 3DVLA tackles the three challenges through: (1) pervasive 3D feature encoding with explicit multi-view consistency constraints across all modalities and a Spatially-Conditioned Geometry Aggregation method, (2) an instance estimation module with high-level instance tokens for 3D instance awareness, and (3) a masked self-supervised 3D encoding branch that retains its predictor for visual token completion to handle occlusions. We integrate 3DVLA with multiple VLA baselines and evaluate on LIBERO-Plus and RoboTwin 2.0. Results show consistent and significant gains in manipulation performance, validating both the effectiveness and plug-and-play compatibility of our approach.

Abstract

Explore similar work

3DVLA: Enhancing Vision-Language-Action Models via 3D Spatial and Instance Understanding

VistaVLA: Geometry- and Semantic-Aware 3D Gaussian-Grounded VLA for Robotic Manipulation

Mind-VLA: Instruction-Aware Spatial Representation Alignment for Vision-Language-Action Models