ROCKET: Residual-Oriented Multi-Layer Alignment for Spatially-Aware Vision-Language-Action Models

Guoheng Sun; Tingting Du; Kaixi Feng; Chenxiang Luo; Xingguo Ding; Zheyu Shen; Ziyao Wang; Yexiao He

cs.CVFeb 20, 2026

ROCKET: Residual-Oriented Multi-Layer Alignment for Spatially-Aware Vision-Language-Action Models

Authors: Guoheng Sun, Tingting Du, Kaixi Feng, Chenxiang Luo, Xingguo Ding, Zheyu Shen, Ziyao Wang, Yexiao He, +1 more

Organizations: University of Maryland, College Park · University of Wisconsin, Madison · City University of Hong Kong · St. Paul’s School

Abstract

Vision-Language-Action (VLA) models enable instruction-following robotic manipulation, but they are typically pretrained on 2D data and lack 3D spatial understanding. An effective approach is representation alignment, where a strong vision foundation model is used to guide a 2D VLA model. However, existing methods usually apply supervision at only a single layer, failing to fully exploit the rich information distributed across depth; meanwhile, naïve multi-layer alignment can cause gradient interference. We introduce ROCKET, a residual-oriented multi-layer representation alignment framework that formulates multi-layer alignment as aligning one residual stream to another. Concretely, ROCKET employs a shared projector to align multiple layers of the VLA backbone with multiple layers of a powerful 3D vision foundation model via a layer-invariant mapping, which reduces gradient conflicts. We provide both theoretical justification and empirical analyses showing that a shared projector is sufficient and outperforms prior designs, and further propose a Matryoshka-style sparse activation scheme for the shared projector to balance multiple alignment losses. Our experiments show that, combined with a training-free layer selection strategy, ROCKET requires only about 4% of the compute budget while achieving 98.5% state-of-the-art success rate on LIBERO. We further demonstrate the superior performance of ROCKET across LIBERO-Plus and RoboTwin, as well as multiple VLA models. The code and model weights can be found at https://github.com/CASE-Lab-UMD/ROCKET-VLA.

Figures & tables

Method	Student	Teacher
SF	OpenVLA – 24 / 32	VGGT – 24 / 24
REPA	DiT – 8 / 24	DINOv2 – 24 / 24
3DRS	LLaVA – 32 / 32	VGGT – 24 / 24

Table 1: Comparison of representation alignment methods for spatial supervision. OpenVLA – 24 / 32 : with a 32-layer backbone, using the 24th-layer features for alignment.

Figure 1: Cosine similarity between gradients induced by different alignment losses in the VLA’s shallow layers. Red indicates gradient coherence, while Blue indicates gradient interference. Detailed results are provided in Fig. 13 and Fig. 14 .

Figure 2: Overview of ROCKET . On the left , we select a sample from LIBERO to showcase ROCKET ’s performance. On the right , we use the outputs from VGGT layers $\{1,3,9,15,21\}$ to directly predict the depth map, demonstrating that different layers contain rich 3D information. Some of the illustrations were generated by GPT-4o ( Hurst et al., 2024 ) .

Figure 5: Success rate (%) on LIBERO-Plus. Results are reported under the seven perturbations defined by LIBERO-Plus. Each column shows the average success rate over four task groups: Spatial, Object, Goal, and Long. Detailed results are provided in Appendix G .

Method	Spatial	Object	Goal	Long	Avg.
Baseline	96.4	98.2	95.0	82.2	93.0
Spatial Forcing	97.8	97.8	94.4	85.8	94.0
ROCKET	96.4	98.8	96.6	89.2	95.3

Table 3: Success rate (%) of the PI0.5 with different representation alignment strategies on the LIBERO.

Figure 6: Success rate (%) on RoboTwin 2.0 (ALOHA) under Easy and Hard settings across five bimanual tasks, comparing ROCKET with DP ( Chi et al., 2025 ) , RDT ( Liu et al., 2024 ) , and PI0 ( Black et al., 2024 ) .

# Layers	Spatial	Object	Goal	Long	Avg.
4	98.0	98.8	93.0	95.2	96.3
8	98.2	99.8	96.4	95.0	97.4
16	97.8	99.6	97.6	91.6	96.7
24	93.2	95.6	97.4	90.0	94.1

Table 4: The impact of the number of aligned layers on final performance, evaluated on LIBERO.

Strategy	Spatial	Object	Goal	Long	Avg.
Baseline	97.2	98.4	95.6	88.6	95.0
Uniform-8	98.2	99.8	96.4	95.0	97.4
E2M-Last1	96.2	98.0	97.2	93.6	96.3
Shallow	97.8	99.8	95.4	85.6	94.7
Middle	89.4	99.8	94.2	91.2	93.7
Deep	92.0	99.8	95.6	93.6	95.3

Table 5: Comparison of layer selection strategies on LIBERO.

Ratio	Spatial	Object	Goal	Long	Avg.
1%	75.0	88.8	65.0	65.6	73.6
5%	94.2	95.0	75.2	69.0	83.4
10%	94.0	91.8	78.6	72.6	84.3
100%	96.2	98.0	97.2	93.6	96.3

Table 6: Limited-data training on LIBERO, SR(%).

Method	Avg.	Cost	Details
ThinkAct	84.4	$6.0\times$	$1\times 128\times 75$ k
UniVLA	95.2	$0.96\times$	$1\times 192\times 8$ k
OpenVLA-OFT	97.1	$25.6\times$	$4\times 64\times 150$ k
GeoVLA	97.7	$3.2\times$	$1\times 256\times 20$ k
3D-CAVLA	98.1	$3.0\times$	$4\times 8\times 150$ k
Spatial Forcing	98.5	$24.0\times$	$4\times 64\times 150$ k

Table 7: Training cost comparison of same-size models on LIBERO. Cost is estimated as

\#\text{Model}\times\text{BatchSize}\times\text{Steps}

Method	Spatial	Object	Goal	Long	Avg.
Baseline	96.9	98.7	97.6	94.8	96.4
+ Multi-layer	93.6	99.2	42.2	85.0	80.0
+ Shared	99.0	99.8	97.0	96.8	98.2
+ Matryoshka	98.2	99.8	98.8	97.0	98.5

Table 8: Ablation study of ROCKET on LIBERO.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Steps	Spatial	Object	Goal	Long	Avg.
Baseline	10k	74.6	89.4	78.6	37.2	70.0
	20k	93.0	99.6	97.6	94.4	96.2
	30k	94.6	99.2	98.2	93.6	96.4
	50k	94.0	97.2	98.0	92.4	95.4
Spatial Forcing	10k	83.4	99.8	52.6	31.6	66.9
	20k	94.2	99.8	94.8	90.4	94.8

Appendix

Table 9: Success rates (%) on the LIBERO benchmarks. We compare the performance trajectories of multiple methods over the full training course; bold indicates the best final result, and underlining indicates the best result achieved during training.

Figure 7: Pairwise cosine similarity of flattened MLP parameters for independently initialized projectors. The similarities are nearly zero, suggesting that naive multi-layer alignment with multiple independent projectors learns distinct mapping trajectories, which can induce gradient conflicts and harm final performance.

Figure 8: Layer-wise CKA similarity between OpenVLA-7B and VGGT. We compute linear CKA between OpenVLA-7B vision-token representations and VGGT visual features across layers. Higher values indicate more similar representations. Shallow VGGT layers exhibit higher similarity to OpenVLA-7B, and similarity generally decreases with depth.

Figure 9: Per-task success rate (%) on LIBERO-Plus Spatial tasks under the seven perturbations.

Figure 10: Per-task success rate (%) on LIBERO-Plus Object tasks under the seven perturbations.

Figure 11: Per-task success rate (%) on LIBERO-Plus Goal tasks under the seven perturbations.

Figure 12: Per-task success rate (%) on LIBERO-Plus Long tasks under the seven perturbations.

Figure 13: Gradient cosine similarity in the early stage of training.

Figure 14: Gradient cosine similarity in the mid-to-late stage of training.

Figure 15: Similarity-based layer selection strategies. We evaluate two criteria for selecting alignment layers: Sim-1 (left) measures 10-layer window similarity by computing cosine similarity between layer

i

’s input and layer

i{+}9

’s output; Sim-2 (right) measures single-layer similarity between each layer’s input and output. For each criterion, we select 10 layers with either highest similarity ( Top , green squares) or lowest similarity ( Last , orange triangles).

Uniform Sampling
Strategy	# Layers	VLA Layers	VGGT Layers
Uniform-4	4	8, 16, 24, 32	5, 11, 17, 23
Uniform-8	8	4, 8, 12, 16, 20, 24, 28, 32	2, 5, 8, 11, 14, 17, 20, 23
Uniform-16	16	4, 5, 6, 7, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31	1, 2, 4, 5, 7, 8, 10, 11, 13, 14, 16, 17, 19, 20, 22, 23
Uniform-24	24	2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15, 16, 18, 19, 20, 22, 23, 24, 26, 27, 28, 30, 31, 32	0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23
Depth-based Sampling

Appendix

Table 10: Detailed layer selection for multi-layer alignment. We show the aligned layer indices for both the student VLA visual backbone and the teacher VGGT model. For OpenVLA, we count the projector between vision tower and LLM as a layer so there are total 33 indices in our default setting. The selected layers are sorted in ascending order for both VLA and VGGT. ∗ For OpenVLA (32 layers, or 33 including the projector), we select layers uniformly using indices 2–18 plus the last layer. † For PI0 and PI0.5 (18 layers), since layer 18 coincides with the last layer, we instead select indices 2–10 plus the last layer.

Explore similar work

Mind-VLA: Instruction-Aware Spatial Representation Alignment for Vision-Language-Action Models
Aug 5, 2026Xingyu Ding, Yuzhong Zhao, Yang Wu +4Robotic Manipulation Vision-Language-Action Models
Nanjing University, Nanjing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Models
Mar 16, 2026Yulin Luo, Hao Chen, Zhuangzhe Wu +10Vision-Language-Action Models Vision-Language Models
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · The Chinese University of Hong Kong · Simplexity Robotics
TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAM
Jul 29, 2026Hengyi Xie, Chenfei Yao, Xianjin Wu +4Efficient VLA Model Inference Efficient VLA Models
Huazhong University of Science and Technology

Aug 5, 2026J/K move · Enter open · S save

Mind-VLA: Instruction-Aware Spatial Representation Alignment for Vision-Language-Action Models

Xingyu Ding, Yuzhong Zhao, Yang Wu, Chaoyang Zhao+3

Nanjing University, Nanjing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

Robotic Manipulation Vision-Language-Action Models 3D Visual Grounding

Open paper AskPDF arXiv

Recent Vision-Language-Action (VLA) methods improve generalization by aligning their representations with 3D scene geometry. However, these methods are fundamentally instruction-agnostic: the representations align the entire scene uniformly, neglecting the 3D geometry of the specific target object designated by the language instruction. This causes failures on fine-grained manipulation and target occlusion tasks, where success depends on accurate 3D understanding of the target object rather than the entire scene. To address this, we present Mind-VLA, an instruction-aware spatial representation alignment method for VLA models. Specifically, Mind-VLA first obtains the target object specified by the language instruction, then prepares its target-object tri-view and extracts the corresponding VAE and VGGT features. Finally, the latent representation of the VLA model is aligned with these features to enable instruction-aware 3D understanding. Mind-VLA reaches 93.9% on LIBERO and 4.47 on CALVIN with a compact 345M-parameter backbone. On real-robot tasks with target occlusion, Mind-VLA reaches 54% average success, outperforming the best-performing instruction-agnostic method in real-robot comparison by 32 percentage points. Code will be publicly available.

Abstract

Explore similar work

Mind-VLA: Instruction-Aware Spatial Representation Alignment for Vision-Language-Action Models

Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Models

TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAM