Bring Back Language to Actions: Restoring VLM capabilities in VLA Models via Parameter Merging
Authors: Mohamed Salim Aissi, Olivier Sigaud, Laure Soulier, Nicolas Thome
Organizations: Sorbonne Université, CNRS, ISIR, F-75005 Paris, France · Institut universitaire de France (IUF) · International Laboratory on Learning Systems (ILLS)
Vision-Language-Action models (VLAs) are obtained by fine-tuning pre-trained Vision-Language Models (VLMs) on robot demonstration data, enabling action prediction from visual observations and language instructions. However, this fine-tuning induces catastrophic forgetting, substantially degrading the semantic and visual grounding capabilities of the original VLM. We address this by merging the parameters of a VLM and its VLA derivative without retraining either model. Concretely, we extract the task vector (the difference between VLA and VLM weights) and decompose it via Singular Value Decomposition. For each component, we learn a scalar mixing coefficient controlling how much of the VLA adaptation is retained, optimised to jointly preserve action prediction and vision-language capabilities. Experiments show that our merged model performs well on both axes simultaneously, consistently outperforming established merging baselines.
Figures & tables
Figure 1: (Left) Given the same visual scene, a VLM responds accurately to descriptive language prompts but produces no action output. Fine-tuning into a VLA enables action prediction from a task goal, but at the cost of language grounding — the VLA generates incoherent text when prompted with the same descriptive query. Our merged model recovers both capabilities, producing coherent language responses while retaining the ability to predict actions. (Right) The VLM achieves strong language performance but cannot act, while VLAs excel at manipulation but lose language capability. Our merged models are the only ones that achieve the best of both worlds, reaching the upper-right region of the trade-off space.
Figure 2: Overview of our VLM–VLA merging method. (a) The task vector Δ(l) is extracted by subtracting the VLM weights from the VLA weights for each layer l . (b) Δ(l) is decomposed via Singular Value Decomposition, revealing the orthogonal directions of adaptation introduced during VLA fine-tuning. (c) A scalar mixing coefficient αk(l) is assigned to each singular component, controlling how much of each direction is retained in the merged model. The coefficients α are optimised by minimising a composite objective that jointly penalises degradation of language generation quality and action prediction accuracy with respect to both source models.
VLM Capabilities
Embodied Reasoning
Robot Manipulation
Model
Type
Captioning
VQAv2
EQA
Libero
RougeL
Meteor
BERTScore
Acc
Obj State
Spatial
Attri Recog
Spatial
Goal
Object
PaliGemma
VLM
33.6
23.6
88.9
53.8
44.4
21.7
32.4
0.0
0.0
0.0
π0Fast
VLA
1.2
0.5
68.2
9.9
15.5
5.7
3.8
70.0
82.0
88.0
π0.5
9.4
1.5
76.5
8.2
8.8
0
0
88.0
90.0
96.0
Model Soup α=0.3
Merged
14.6
11.4
82.7
32.4
22.2
15.1
22.7
0.0
0.0
0.0
Table 1: Vision-language, embodied reasoning, and robot manipulation performance of all methods. The pure VLM cannot act, and both VLAs lose language and reasoning ability despite strong action performance. Existing merging baselines largely collapse on manipulation, while our merged models recover strong performance on all three axes simultaneously, without retraining either source model.
Figure 3: Qualitative captioning examples. Given the same visual observation, the VLM produces coherent scene descriptions, the VLA ( π0.5 ) generates fragmented or incomplete text due to catastrophic forgetting, and Ours π0.5 merged model recovers meaningful captions while retaining action capabilities.
Figure 4: Granularity of mixing coefficients. We compare two levels of granularity for α — per-layer and per-component (ours) — and report the evolution of Ltext , Lact , DKL(πα∥πVLM) and DKL(πα∥πVLA) over training. Per-component α converges to substantially lower values on every metric, confirming that fine-grained SVD-based control is essential to simultaneously preserve language and action capabilities.
Objective
ROUGE-L ↑
SR ↑
w/o Ltext
21.9
95.6
w/o Lact
34.6
0
w/o DKL(πα∥πVLM)
7.9
94.1
w/o DKL(πα∥πVLA)
31.2
71.5
Full objective (ours)
29.5
93.3
Table 2: Contribution of each loss component. We ablate each term of the training objective and evaluate using ROUGE-L for language quality and Success Rate SR for action predictions. Best results in bold .
Figure 5: Effect of SVD rank. (Left) Training losses for varying rank r∈{4,16,64,256,1024,full} : Ltext is largely insensitive to rank, while Lact improves markedly with higher rank, indicating that the VLA adaptation is diffuse across many singular components. (Right) Downstream evaluation confirms this asymmetry: ROUGE-L saturates from r=16 , while LIBERO success rate requires r≥1024 to exceed 90% — language recovery needs far fewer singular directions than action preservation.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Ours π0Fast
Ours π0.5
Optimiser
Adam
Adam
Initial learning rate
0.01
0.01
Learning rate schedule
cosine decay
cosine decay
Total training steps
20,000
20,000
λ1 ( Ltext )
1
1
λ2 ( Lact )
1
5
Appendix
Table 3: Hyperparameters used to optimise α for each VLA variant. λ1,λ2 weight Ltext and Lact respectively; λ3,λ4 weight the KL regularisation terms toward the VLM and VLA respectively.
Model
Type
Open Fridge
Setup Cup
Navigate Kitchen
PaliGemma
VLM
0
0
0
π0Fast
VLA
78
68
89
π0.5
71
58
25
Model Soup α=0.3
Merged
0
0
0
Model Soup α=0.5
0
0
0
Model Soup α=0.7
0
0
0
Appendix
Table 4: Task success rate (%) on three RoboCasa kitchen tasks. All fixed-coefficient merging baselines collapse to zero, while our merged models retain meaningful performance on both VLA variants.
Figure 6: Per-layer similarity to source models. (Top) Ours π0Fast . (Bottom) Ours π0.5 . For each of the 602 weight matrices, we compute the cosine similarity between the VLM and VLA weights (blue), between the merged model and the VLM (orange), and between the merged model and the VLA (green). In both cases, the merged model is consistently closer to the VLA than to the VLM across all layers, while remaining well above the VLM–VLA baseline throughout.
Figure 7: Mean α vs. rank index. (Top) Ours π0.5 . (Bottom) Ours π0Fast . For each rank index k , we average αk(l) across all 602 layers. Both VLA variants show the same overall pattern: mean α is near 0 for the most dominant directions and rises monotonically toward the tail of the spectrum, confirming that this asymmetry is not specific to a single action decoding strategy.
Figure 8: Qualitative captioning examples. Given the same visual observation, the VLM produces coherent scene descriptions, the VLA ( π0.5 ) generates fragmented or incomplete text due to catastrophic forgetting, and Ours π0.5 merged model recovers meaningful captions while retaining action capabilities.
Vision-language-action (VLA) models can learn a wide range of robotic skills, but combining separately trained skills remains challenging. The original training data may be unavailable for joint retraining, while directly merging model weights can cause interference. We propose NestedVLA, which consolidates independently fine-tuned VLA policies without access to their original training data. Instead of merging all updates into one checkpoint, a hypernetwork embedded in the VLA generates action-expert weight updates for a selected task. Each task's compressed vision-language model update is stored separately, and both updates are added to a frozen base VLA to instantiate the corresponding policy. We evaluate NestedVLA by consolidating three policies fine-tuned with reinforcement learning on MetaWorld, ManiSkill, and CALVIN. It achieves 66.7% average success, improving on the strongest model-merging baseline by 41.1 percentage points. It performs within 1 percentage point of per-task fine-tuning on ManiSkill and substantially narrows the gap on CALVIN. In this three-policy setting, total storage is 1.5x that of a single full checkpoint. Code is available at https://github.com/WanyuGroup/NeurIPS2026-NestedVLA.
Guodong Du, Wanyu Lin
Department of Computing, The Hong Kong Polytechnic University
Vision-Language-Action (VLA) models enable instruction-driven robotic manipulation, but they inherit oversized language backbones from pretrained VLMs whose capacity far exceeds what is needed for short robotic instructions. This raises a basic question: how much of a VLA model is actually necessary for closed-loop control? In this work, we study architectural redundancy in VLA models by using transformer block removal as a controlled intervention. We introduce Drop-Then-Recovery (DTR), an analysis protocol that removes selected blocks from a pretrained VLA model and then fine-tunes the resulting model to measure whether the removed capacity was necessary for downstream control. To make this intervention reliable, we propose GateProbe, a one-shot virtual-gate sensitivity metric that ranks blocks by their contribution to the downstream action loss. Across multiple VLA architectures, manipulation benchmarks and even real-robot industrial scenarios, we find a strong asymmetry in post-removal recoverability: \ul{language backbones are highly redundant for standard robotic manipulation tasks, whereas vision and action pathways are substantially less tolerant to removal}. On LIBERO, removing half of the LLM blocks even improves OpenVLA-OFT from 95.0% to 98.3% under the same downstream fine-tuning budget, and retaining only two language blocks still recovers baseline-level performance. These results suggest that current VLA benchmarks may exert limited pressure on deep language grounding and compositional instruction understanding, and that future VLA architectures should allocate capacity more deliberately across language, vision, and action components. The code is available at https://github.com/s1ghhh/VLADrop.
Guoheng Sun, Kaixi Feng, Shwai He +8
University of Maryland, College Park · 2Cisco Research
Vision-Language-Action (VLA) models widely adopt pretrained Vision-Language Models (VLMs) as policy backbones, yet it remains unclear what kind of pretrained VLM representation is useful as a VLA initialization. In this paper, we study VLA initialization as a controlled representation-design problem along three axes: capability-level embodied VQA supervision, parameter-update strategy, and robot-data pretraining. Our experiments show that the original pretrained VLM representation is a key source of action performance. However, embodied VQA adaptation does not yield uniform gains: its benefit depends on downstream bottlenecks, and gains from different capability domains are not simply additive. For update strategy, LoRA provides a more reliable initialization than Full Finetune, indicating that overly reshaping the pretrained representation can weaken VLA initialization. Robot-data pretraining further improves VLA initialization, with the strongest variant obtained by staged LoRA-based training. Together, these findings suggest that effective VLM-to-VLA adaptation should inject action-relevant embodied and robot-trajectory signals while preserving the pretrained VLM representation that remains useful for action learning.