ROCKET: Residual-Oriented Multi-Layer Alignment for Spatially-Aware Vision-Language-Action Models

Guoheng Sun; Tingting Du; Kaixi Feng; Chenxiang Luo; Xingguo Ding; Zheyu Shen; Ziyao Wang; Yexiao He

cs.CVFeb 20, 2026

ROCKET: Residual-Oriented Multi-Layer Alignment for Spatially-Aware Vision-Language-Action Models

Authors: Guoheng Sun, Tingting Du, Kaixi Feng, Chenxiang Luo, Xingguo Ding, Zheyu Shen, Ziyao Wang, Yexiao He, +1 more

Organizations: University of Maryland, College Park · University of Wisconsin, Madison · City University of Hong Kong · St. Paul’s School

Abstract

Vision-Language-Action (VLA) models enable instruction-following robotic manipulation, but they are typically pretrained on 2D data and lack 3D spatial understanding. An effective approach is representation alignment, where a strong vision foundation model is used to guide a 2D VLA model. However, existing methods usually apply supervision at only a single layer, failing to fully exploit the rich information distributed across depth; meanwhile, naïve multi-layer alignment can cause gradient interference. We introduce ROCKET, a residual-oriented multi-layer representation alignment framework that formulates multi-layer alignment as aligning one residual stream to another. Concretely, ROCKET employs a shared projector to align multiple layers of the VLA backbone with multiple layers of a powerful 3D vision foundation model via a layer-invariant mapping, which reduces gradient conflicts. We provide both theoretical justification and empirical analyses showing that a shared projector is sufficient and outperforms prior designs, and further propose a Matryoshka-style sparse activation scheme for the shared projector to balance multiple alignment losses. Our experiments show that, combined with a training-free layer selection strategy, ROCKET requires only about 4% of the compute budget while achieving 98.5% state-of-the-art success rate on LIBERO. We further demonstrate the superior performance of ROCKET across LIBERO-Plus and RoboTwin, as well as multiple VLA models. The code and model weights can be found at https://github.com/CASE-Lab-UMD/ROCKET-VLA.

Figures & tables

Method	Student	Teacher
SF	OpenVLA – 24 / 32	VGGT – 24 / 24
REPA	DiT – 8 / 24	DINOv2 – 24 / 24
3DRS	LLaVA – 32 / 32	VGGT – 24 / 24

Table 1: Comparison of representation alignment methods for spatial supervision. OpenVLA – 24 / 32 : with a 32-layer backbone, using the 24th-layer features for alignment.

Figure 1: Cosine similarity between gradients induced by different alignment losses in the VLA’s shallow layers. Red indicates gradient coherence, while Blue indicates gradient interference. Detailed results are provided in Fig. 13 and Fig. 14 .

Figure 2: Overview of ROCKET . On the left , we select a sample from LIBERO to showcase ROCKET ’s performance. On the right , we use the outputs from VGGT layers $\{1,3,9,15,21\}$ to directly predict the depth map, demonstrating that different layers contain rich 3D information. Some of the illustrations were generated by GPT-4o ( Hurst et al., 2024 ) .

Figure 5: Success rate (%) on LIBERO-Plus. Results are reported under the seven perturbations defined by LIBERO-Plus. Each column shows the average success rate over four task groups: Spatial, Object, Goal, and Long. Detailed results are provided in Appendix G .

Method	Spatial	Object	Goal	Long	Avg.
Baseline	96.4	98.2	95.0	82.2	93.0
Spatial Forcing	97.8	97.8	94.4	85.8	94.0
ROCKET	96.4	98.8	96.6	89.2	95.3

Table 3: Success rate (%) of the PI0.5 with different representation alignment strategies on the LIBERO.

Figure 6: Success rate (%) on RoboTwin 2.0 (ALOHA) under Easy and Hard settings across five bimanual tasks, comparing ROCKET with DP ( Chi et al., 2025 ) , RDT ( Liu et al., 2024 ) , and PI0 ( Black et al., 2024 ) .

# Layers	Spatial	Object	Goal	Long	Avg.
4	98.0	98.8	93.0	95.2	96.3
8	98.2	99.8	96.4	95.0	97.4
16	97.8	99.6	97.6	91.6	96.7
24	93.2	95.6	97.4	90.0	94.1

Table 4: The impact of the number of aligned layers on final performance, evaluated on LIBERO.

Strategy	Spatial	Object	Goal	Long	Avg.
Baseline	97.2	98.4	95.6	88.6	95.0
Uniform-8	98.2	99.8	96.4	95.0	97.4
E2M-Last1	96.2	98.0	97.2	93.6	96.3
Shallow	97.8	99.8	95.4	85.6	94.7
Middle	89.4	99.8	94.2	91.2	93.7
Deep	92.0	99.8	95.6	93.6	95.3

Table 5: Comparison of layer selection strategies on LIBERO.

Ratio	Spatial	Object	Goal	Long	Avg.
1%	75.0	88.8	65.0	65.6	73.6
5%	94.2	95.0	75.2	69.0	83.4
10%	94.0	91.8	78.6	72.6	84.3
100%	96.2	98.0	97.2	93.6	96.3

Table 6: Limited-data training on LIBERO, SR(%).

Method	Avg.	Cost	Details
ThinkAct	84.4	$6.0\times$	$1\times 128\times 75$ k
UniVLA	95.2	$0.96\times$	$1\times 192\times 8$ k
OpenVLA-OFT	97.1	$25.6\times$	$4\times 64\times 150$ k
GeoVLA	97.7	$3.2\times$	$1\times 256\times 20$ k
3D-CAVLA	98.1	$3.0\times$	$4\times 8\times 150$ k
Spatial Forcing	98.5	$24.0\times$	$4\times 64\times 150$ k

Table 7: Training cost comparison of same-size models on LIBERO. Cost is estimated as

\#\text{Model}\times\text{BatchSize}\times\text{Steps}

Method	Spatial	Object	Goal	Long	Avg.
Baseline	96.9	98.7	97.6	94.8	96.4
+ Multi-layer	93.6	99.2	42.2	85.0	80.0
+ Shared	99.0	99.8	97.0	96.8	98.2
+ Matryoshka	98.2	99.8	98.8	97.0	98.5

Table 8: Ablation study of ROCKET on LIBERO.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Steps	Spatial	Object	Goal	Long	Avg.
Baseline	10k	74.6	89.4	78.6	37.2	70.0
	20k	93.0	99.6	97.6	94.4	96.2
	30k	94.6	99.2	98.2	93.6	96.4
	50k	94.0	97.2	98.0	92.4	95.4
Spatial Forcing	10k	83.4	99.8	52.6	31.6	66.9
	20k	94.2	99.8	94.8	90.4	94.8

Appendix

Table 9: Success rates (%) on the LIBERO benchmarks. We compare the performance trajectories of multiple methods over the full training course; bold indicates the best final result, and underlining indicates the best result achieved during training.

Figure 7: Pairwise cosine similarity of flattened MLP parameters for independently initialized projectors. The similarities are nearly zero, suggesting that naive multi-layer alignment with multiple independent projectors learns distinct mapping trajectories, which can induce gradient conflicts and harm final performance.

Figure 8: Layer-wise CKA similarity between OpenVLA-7B and VGGT. We compute linear CKA between OpenVLA-7B vision-token representations and VGGT visual features across layers. Higher values indicate more similar representations. Shallow VGGT layers exhibit higher similarity to OpenVLA-7B, and similarity generally decreases with depth.

Figure 9: Per-task success rate (%) on LIBERO-Plus Spatial tasks under the seven perturbations.

Figure 10: Per-task success rate (%) on LIBERO-Plus Object tasks under the seven perturbations.

Figure 11: Per-task success rate (%) on LIBERO-Plus Goal tasks under the seven perturbations.

Figure 12: Per-task success rate (%) on LIBERO-Plus Long tasks under the seven perturbations.

Figure 13: Gradient cosine similarity in the early stage of training.

Figure 14: Gradient cosine similarity in the mid-to-late stage of training.

Figure 15: Similarity-based layer selection strategies. We evaluate two criteria for selecting alignment layers: Sim-1 (left) measures 10-layer window similarity by computing cosine similarity between layer

i

’s input and layer

i{+}9

’s output; Sim-2 (right) measures single-layer similarity between each layer’s input and output. For each criterion, we select 10 layers with either highest similarity ( Top , green squares) or lowest similarity ( Last , orange triangles).

Uniform Sampling
Strategy	# Layers	VLA Layers	VGGT Layers
Uniform-4	4	8, 16, 24, 32	5, 11, 17, 23
Uniform-8	8	4, 8, 12, 16, 20, 24, 28, 32	2, 5, 8, 11, 14, 17, 20, 23
Uniform-16	16	4, 5, 6, 7, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31	1, 2, 4, 5, 7, 8, 10, 11, 13, 14, 16, 17, 19, 20, 22, 23
Uniform-24	24	2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15, 16, 18, 19, 20, 22, 23, 24, 26, 27, 28, 30, 31, 32	0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23
Depth-based Sampling

Appendix

Table 10: Detailed layer selection for multi-layer alignment. We show the aligned layer indices for both the student VLA visual backbone and the teacher VGGT model. For OpenVLA, we count the projector between vision tower and LLM as a layer so there are total 33 indices in our default setting. The selected layers are sorted in ascending order for both VLA and VGGT. ∗ For OpenVLA (32 layers, or 33 including the projector), we select layers uniformly using indices 2–18 plus the last layer. † For PI0 and PI0.5 (18 layers), since layer 18 coincides with the last layer, we instead select indices 2–10 plus the last layer.

Explore similar work

Aug 5, 2026

Mind-VLA: Instruction-Aware Spatial Representation Alignment for Vision-Language-Action Models

Recent Vision-Language-Action (VLA) methods improve generalization by aligning their representations with 3D scene geometry. However, these methods are fundamentally instruction-agnostic: the representations align the entire scene uniformly, neglecting the 3D geometry of the specific target object designated by the language instruction. This causes failures on fine-grained manipulation and target occlusion tasks, where success depends on accurate 3D understanding of the target object rather than the entire scene. To address this, we present Mind-VLA, an instruction-aware spatial representation alignment method for VLA models. Specifically, Mind-VLA first obtains the target object specified by the language instruction, then prepares its target-object tri-view and extracts the corresponding VAE and VGGT features. Finally, the latent representation of the VLA model is aligned with these features to enable instruction-aware 3D understanding. Mind-VLA reaches 93.9% on LIBERO and 4.47 on CALVIN with a compact 345M-parameter backbone. On real-robot tasks with target occlusion, Mind-VLA reaches 54% average success, outperforming the best-performing instruction-agnostic method in real-robot comparison by 32 percentage points. Code will be publicly available.

Xingyu Ding, Yuzhong Zhao, Yang Wu +4

Nanjing University, Nanjing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

Robotic Manipulation Vision-Language-Action Models 3D Visual Grounding

Ask PDF

Mar 16, 2026

Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Models

Vision-Language-Action (VLA) models have recently emerged as a promising paradigm for robotic manipulation, in which reliable action prediction critically depends on accurately interpreting and integrating visual observations conditioned on language instructions. Although recent works have sought to enhance the visual capabilities of VLA models, most approaches treat the LLM backbone as a black box, providing limited insight into how visual information is grounded into action generation. Therefore, we perform a systematic analysis of multiple VLA models across different action-generation paradigms and observe that sensitivity to visual tokens progressively decreases in deeper layers during action generation. Motivated by this observation, we propose DeepVision-VLA, built on a Vision-Language Mixture-of-Transformers (VL-MoT) framework. This framework enables shared attention between the vision foundation model and the VLA backbone, injecting multi-level visual features from the vision expert into deeper layers of the VLA backbone to enhance visual representations for precise and complex manipulation. In addition, we introduce Action-Guided Visual Pruning (AGVP), which leverages shallow-layer attention to prune irrelevant visual tokens while preserving task-relevant ones, reinforcing critical visual cues for manipulation with minimal computational overhead. DeepVision-VLA outperforms prior state-of-the-art methods by 9.0% and 7.5% on simulated and real-world tasks, respectively, providing new insights for the design of visually enhanced VLA models.

Yulin Luo, Hao Chen, Zhuangzhe Wu +10

State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · The Chinese University of Hong Kong · Simplexity Robotics

Vision-Language-Action Models Vision-Language Models Robotic Manipulation Visual Token Pruning

Ask PDF

Jul 29, 2026

TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAM

Vision-language-action (VLA) models commonly adopt an LLM-centric

V \to L \to A

pathway, processing visual observations and language instructions through a large language model before predicting robot actions. Although effective, this design incurs substantial computation and memory overhead. In this work, we introduce TurboVLA, a compact VLA architecture built on a direct

V + L \to A

mapping. Instead of using a large language model as the central interface between perception and action, TurboVLA independently encodes visual observations and language instructions, directly exchanges information between them through lightweight bidirectional vision-language interaction, and predicts continuous action chunks with a compact decoder. This simple design directly constructs task-conditioned representations while avoiding the overhead of an LLM-centered execution pathway. On LIBERO, TurboVLA achieves 97.6% average success with only 0.2B parameters, 31.2 ms inference latency, and 0.9 GiB inference VRAM on a consumer-grade RTX 4090. Notably, a 0.4B TurboVLA achieves 88.06% success on RoboTwin 2.0, even matching or outperforming substantially larger VLA policies. These results demonstrate that the simple

V + L \to A

design of TurboVLA can achieve high performance without requiring an LLM-centric execution pathway, offering a new perspective on how vision, language, and action can be connected for efficient robotic manipulation. Code is available at https://github.com/H-EmbodVis/TurboVLA.

Hengyi Xie, Chenfei Yao, Xianjin Wu +4

Huazhong University of Science and Technology

Efficient VLA Model Inference Efficient VLA Models Robotic Manipulation Vision-Language-Action Models

Ask PDF