Bring Back Language to Actions: Restoring VLM capabilities in VLA Models via Parameter Merging
Organizations: Sorbonne Université, CNRS, ISIR, F-75005 Paris, France · Institut universitaire de France (IUF) · International Laboratory on Learning Systems (ILLS)
Abstract
Vision-Language-Action models (VLAs) are obtained by fine-tuning pre-trained Vision-Language Models (VLMs) on robot demonstration data, enabling action prediction from visual observations and language instructions. However, this fine-tuning induces catastrophic forgetting, substantially degrading the semantic and visual grounding capabilities of the original VLM. We address this by merging the parameters of a VLM and its VLA derivative without retraining either model. Concretely, we extract the task vector (the difference between VLA and VLM weights) and decompose it via Singular Value Decomposition. For each component, we learn a scalar mixing coefficient controlling how much of the VLA adaptation is retained, optimised to jointly preserve action prediction and vision-language capabilities. Experiments show that our merged model performs well on both axes simultaneously, consistently outperforming established merging baselines.
Figures & tables
| VLM Capabilities | Embodied Reasoning | Robot Manipulation | |||||||||
| Model | Type | Captioning | VQAv2 | EQA | Libero | ||||||
| RougeL | Meteor | BERTScore | Acc | Obj State | Spatial | Attri Recog | Spatial | Goal | Object | ||
| PaliGemma | VLM | 33.6 | 23.6 | 88.9 | 53.8 | 44.4 | 21.7 | 32.4 | 0.0 | 0.0 | 0.0 |
| VLA | 1.2 | 0.5 | 68.2 | 9.9 | 15.5 | 5.7 | 3.8 | 70.0 | 82.0 | 88.0 | |
| 9.4 | 1.5 | 76.5 | 8.2 | 8.8 | 0 | 0 | 88.0 | 90.0 | 96.0 | ||
| Model Soup α=0.3 | Merged | 14.6 | 11.4 | 82.7 | 32.4 | 22.2 | 15.1 | 22.7 | 0.0 | 0.0 | 0.0 |
| Objective | ROUGE-L | SR |
|---|---|---|
| w/o | 21.9 | 95.6 |
| w/o | 34.6 | 0 |
| w/o | 7.9 | 94.1 |
| w/o | 31.2 | 71.5 |
| Full objective (ours) | 29.5 | 93.3 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Ours | Ours |
|---|---|---|
| Optimiser | Adam | Adam |
| Initial learning rate | ||
| Learning rate schedule | cosine decay | cosine decay |
| Total training steps | 20,000 | 20,000 |
| ( ) | 1 | 1 |
| ( ) | 1 | 5 |
| Model | Type | Open Fridge | Setup Cup | Navigate Kitchen |
|---|---|---|---|---|
| PaliGemma | VLM | 0 | 0 | 0 |
| VLA | 78 | 68 | 89 | |
| 71 | 58 | 25 | ||
| Model Soup α=0.3 | Merged | 0 | 0 | 0 |
| Model Soup α=0.5 | 0 | 0 | 0 | |
| Model Soup α=0.7 | 0 | 0 | 0 |
Vision-Language-Action Models
Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.
See all 1,239 papers