NestedVLA: Learning to Consolidate and Generate Skills for Vision-Language-Action Models

Guodong Du; Wanyu Lin

cs.ROOct 16, 2026

NestedVLA: Learning to Consolidate and Generate Skills for Vision-Language-Action Models

Authors: Guodong Du, Wanyu Lin

Organizations: Department of Computing, The Hong Kong Polytechnic University

Abstract

Vision-language-action (VLA) models can learn a wide range of robotic skills, but combining separately trained skills remains challenging. The original training data may be unavailable for joint retraining, while directly merging model weights can cause interference. We propose NestedVLA, which consolidates independently fine-tuned VLA policies without access to their original training data. Instead of merging all updates into one checkpoint, a hypernetwork embedded in the VLA generates action-expert weight updates for a selected task. Each task's compressed vision-language model update is stored separately, and both updates are added to a frozen base VLA to instantiate the corresponding policy. We evaluate NestedVLA by consolidating three policies fine-tuned with reinforcement learning on MetaWorld, ManiSkill, and CALVIN. It achieves 66.7% average success, improving on the strongest model-merging baseline by 41.1 percentage points. It performs within 1 percentage point of per-task fine-tuning on ManiSkill and substantially narrows the gap on CALVIN. In this three-policy setting, total storage is 1.5x that of a single full checkpoint. Code is available at https://github.com/WanyuGroup/NeurIPS2026-NestedVLA.

Figures & tables

Figure 1: Motivation and overview of NestedVLA. (a) Model merging combines independently fine-tuned VLA weights into a shared model, where incompatible updates can impair the source tasks. (b) NestedVLA retains a compressed task-specific VLM update and uses an embedded task-conditioned hypernetwork to generate the action-expert update. Both updates are applied to a shared base to materialize the selected VLA.

Figure 2: NestedVLA pipeline. (1) Skill acquisition : each task is fine-tuned independently from the same base $\Theta_{0}$ and the resulting task vector $\tau_{m}$ is compressed into a sparse skill $z_{m}=\alpha\cdot\tau_{m}\odot M_{m}$ via drop-and-rescale. (2) Skill consolidation : a single nested hypernetwork $\mathcal{G}_{\phi}$ is trained against the action-expert portions of $\{z_{m}\}$ with a multi-task masked MSE loss. (3) Skill generation : at inference, querying $\mathcal{G}_{\phi}$ with task id $m$ generates the action-expert update; the corresponding compressed VLM update is retrieved and both compose with the frozen base.

Figure 3: Architecture of the nested hypernetwork $\mathcal{G}_{\phi}$ . Patches are addressed by $(\texttt{ptype},\texttt{layer},\texttt{patch\_h},\texttt{patch\_w})$ and mapped to four per-task patch embeddings ( patch path ). The task id $m$ is mapped to a task embedding and processed by a lightweight task encoder (2D convolutions plus average pooling) into a single task feature $f_{\text{task}}$ . $f_{\text{task}}$ is broadcast across all patches and concatenated with the patch embeddings, then passed through a shared FC trunk augmented with a per-task low-rank LoRA path ( LoRA path ). The trunk output is decoded by a multi-head output module that allocates one fully-connected projection per task ( multi-head path ) into a flat patch prediction. The precomputed action-expert mask $M_{m}^{A}$ and global rescale $\alpha$ produce $\hat{\tau}_{m}^{A}=\alpha\cdot\mathcal{G}_{\phi}(m)\odot M_{m}^{A}$ ; the compressed VLM update is retained separately.

Reference upper bounds
Method	Params	MetaWorld	ManiSkill	CALVIN	Avg.
Per-task FlowSDE (no merge)	3.0 $\times$	72.3	88.1	81.3	80.5
Static merging
Averaging	1.0 $\times$	3.1	41.3	0.0	14.8
TIES [ 62 ]	1.0 $\times$	2.0	37.2	0.0	13.1
WUDI [ 5 ]	1.0 $\times$	2.7	32.8	0.0	11.8

Table 1: Main result. Multi-task hypernet on three RL benchmarks. We report success_once (%) for MetaWorld and ManiSkill, and success_at_end (%) for CALVIN, averaged over

3

evaluation seeds (typical std.

\pm 2.7\%

). Params is the total checkpoint storage relative to a single fine-tuned model (

1.0\times\approx 3.61

B for

\pi_{0.5}

). NestedVLA outperforms every static and dynamic merging baseline by

41

pp on average at the lowest storage cost among methods that condition on task identity. Best per column in bold .

Variant	#params	MW $\downarrow$	MA $\downarrow$	CA $\downarrow$
w/o Multi-head path	161 M	1.33	1.71	1.36
w/o patch path	430 M	1.02	1.14	1.05
w/o LoRA path	417 M	0.91	0.92	0.89
narrow ( $h{=}1024$ )	221 M	1.43	1.56	1.55
Default	430 M	1.08	1.17	1.14

Table 2: Architecture ablations of

\mathcal{G}_{\phi}

2{,}000

steps.

\mathcal{G}_{\phi}

targets the action expert only; #params refers to the hypernetwork itself, not to the underlying

\pi_{0.5}

backbone. We ablate each of the three trainable paths – the per-task output projection ( Multi-head path ), the per-task patch embeddings ( patch path ), and the shared-FC LoRA ( LoRA path ) – and report the final in-mask standardized loss per task (lower is better; averaged over

3

seeds, std.

\pm 0.05

Task	NestedVLA	Per-task FlowSDE	Ideal ( $K{=}10\%$ )	Ideal ( $K{=}20\%$ )
MetaWorld	37.9	72.3	48.4	60.2
ManiSkill	87.2	88.1	87.5	87.5
CALVIN	75.0	81.3	78.1	85.4

Table 3: Per-task ceiling analysis. Ideal : the upper bound at sparsity

K

, computed by adding

\tau_{m}\odot M_{m}

directly to

\Theta_{0}

without learning

\mathcal{G}_{\phi}

. The MetaWorld gap is bounded by the ideal itself, indicating a sparsity-budget effect rather than a hypernetwork-capacity problem.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone
Component	Value
Base model	$\pi_{0.5}$ ( $\sim$ $3.61$ B params)
VLM half	$\sim$ $3.21$ B params (frozen, per-task stored)
Action expert	$\sim$ $0.40$ B params (target of $\mathcal{G}_{\phi}$ )
Trainable tensors	$162$ (18 blocks $\times$ 9 weight types)
Patch tokenization

Appendix

Table 4: Hypernetwork architecture and training hyperparameters.

Method	Hyperparameter(s) we use	Storage atom (per task)
Static merging
Averaging	none	—
TIES	trim ratio $k{=}0.8$ , scaling $0.5$	—
WUDI (v1)	$300$ optimization iterations	—
TSV-Merging	spectral retention $\sim\!95\%$	shared SVD factors
Dynamic merging

Appendix

Table 5: Hyperparameters used for every baseline in Table 1 . “Storage atom” summarises the per-task auxiliary state introduced beyond the merged base.

Method	Static	Shared	Mask	Experts / Skills	$\mathcal{G}_{\phi}$	Total
Per-task FlowSDE	–	–	–	$3.00$	–	$3.00\times$
Static merging
Averaging	$1.00$	–	–	–	–	$1.00\times$
TIES	$1.00$	–	–	–	–	$1.00\times$
WUDI	$1.00$	–	–	–	–	$1.00\times$
TSV-Merging	$1.00$	–	–	–	–	$1.00\times$

Appendix

Table 6: Bit-level storage decomposition for every method in Table 1 , in units of one fine-tuned

\pi_{0.5}

checkpoint (

P_{\text{base}}{=}3.61

B parameters);

M{=}3

tasks. Columns: (i) shared backbone or shared merged delta; (ii) shared low-rank or mean-delta residual; (iii) per-task masks (summed over

M

); (iv) per-task experts, scalars, or low-rank residuals (summed over

M

); (v) hypernetwork. “–” indicates the component is not used. Numbers may not sum exactly to the headline due to rounding.

Operator	Storage	Rec. err. (VLM / action)	Penalty (avg)
1-bit quantization	$\sim$ $6\%$	$0.60$ – $0.96$	$-71$ to $-81$ pp
Drop-and-rescale + SVD residual	$\sim$ $15\%$	$0.295$ / $0.187$	$-3$ to $-5$ pp
Drop-and-rescale (ours)	$10\%$	$0.300$ / $0.190$	$-3$ to $-5$ pp

Appendix

Table 7: Compression operator comparison on VLA fine-tuning deltas, per-task fidelity evaluation. Storage : effective storage relative to the dense

\tau_{m}

. Rec. err. : relative Frobenius error

\|\hat{\tau}_{m}-\tau_{m}\|_{F}/\|\tau_{m}\|_{F}

on VLM and action halves. Penalty : drop in success_once (MW/MA) or success_at_end (CA) vs. FlowSDE. Drop-and-rescale gives the best trade-off with no tuning.

Task	$K{=}0.05$	$K{=}0.10$	$K{=}0.15$	$K{=}0.20$	FlowSDE
MetaWorld	21.1	48.4	57.4	60.2	72.3
ManiSkill	86.3	87.5	91.6	87.5	88.1
CALVIN	63.5	78.1	78.1	85.4	81.3

Appendix

Table 8: Keep-ratio sweep at fixed rescale

\alpha=1.0

, per-task fidelity evaluation. Numbers are success_once (%) for MetaWorld and ManiSkill; success_at_end (%) for CALVIN. The optimal

K

is task-dependent: ManiSkill peaks at

K{=}0.15

, CALVIN at

K{=}0.20

; MetaWorld is monotonically improving but never reaches its FlowSDE reference. Per-task FlowSDE references: MW

72.3

, MA

88.1

, CA

81.3

Group	MA–MW	CA–MW	CA–MA	Sign agreement	IF
Action expert	0.062	0.069	0.106	0.518–0.533	0.379
VLM body	0.011	0.081	0.015	0.504–0.511	0.399
Action interfaces	0.114	0.082	0.073	0.533–0.546	0.373

Appendix

Table 9: Weight-space diagnostics relative to the pretrained reference. Pair columns are concatenated-delta cosine similarities; sign agreement spans three pairs. IF denotes interference.

VLM merge method	MetaWorld	ManiSkill	CALVIN
FT reference (no merge)	$66.8$	$90.0$	$86.5$
TIES VLM ( $k{=}0.8$ , $s{=}0.5$ )	$1.9$ ( $-64.9$ )	$64.7$ ( $-25.3$ )	$0.0$ ( $-86.5$ )
WUDI VLM (300 iter)	$4.7$ ( $-62.1$ )	$80.3$ ( $-9.7$ )	$0.0$ ( $-86.5$ )

Appendix

Table 10: VLM static merging sanity test. Only the VLM half is consolidated across the three tasks; the action expert is the GT per-task FlowSDE expert. Penalty is the drop in success_once (MetaWorld and ManiSkill) or success_at_end (CALVIN) relative to the same-batch FlowSDE reference. Both methods incur penalties exceeding

25

pp on at least two of three benchmarks, with CALVIN collapsing to

0\%

chain completion.

Variant	space	$p$	$w_{\text{out}}$	MW	MA	CA	Avg.
default	std	0	0.05	29.7	90.9	66.7	62.4
w/o out-penalty ( $w_{\text{out}}{=}0$ )	std	0	0	37.9	88.8	67.7	64.8
abs space	abs	0	0.05	23.0	87.8	67.7	59.5
mag-weighting ( $p{=}1$ )	std	1	0.05	27.7	86.9	56.2	57.0
abs $+$ mag $p{=}1$	abs	1	0.05	31.2	88.1	70.8	63.4

Appendix

Table 11: Loss-side ablations on top of the architecture in Table 2 . We vary the loss space ( std : per-tensor standardized; abs : raw absolute), the in-mask magnitude exponent

p

(each loss term is multiplied by

|\tau_{m}|^{p}

), and the out-of-mask penalty weight

w_{\text{out}}

. All runs share the same

6{,}000

-step schedule. Single-seed evaluation; the headline NestedVLA number in Table 1 is the

3

-seed average of the same configuration.

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,190 papers

$M^2$ -VLA: Boosting Vision-Language Models for Generalizable Manipulation via Layer Mixture and Meta-Skills
Apr 27, 2026Siyao Xiao, Yuhong Zhang, Zhifang Liu +9Language-Conditioned Robot Manipulation VLM Adaptation
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China. · PengCheng Laboratory, Shenzhen, China. · College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, China. +1
Decoupling the Declarative from the Procedural in Vision-Language-Action Models
Jun 19, 2026Nikolaos Tsagkas, Andreas Sochopoulos, Chris Xiaoxuan Lu +2Robotic Manipulation Transfer Learning
University of Edinburgh · UCL · Samsung AI Center - Cambridge
Continually Evolving Skill Knowledge in Vision Language Action Model
Nov 22, 2025Yuxuan Wu, Guangming Wang, Zhiheng Yang +4Continual Learning for VLA Models Continual Robot Learning
Shanghai Jiao Tong University · Shanghai Innovation Institute · University of Cambridge +4

Apr 27, 2026J/K move · Enter open · S save

$M^2$ -VLA: Boosting Vision-Language Models for Generalizable Manipulation via Layer Mixture and Meta-Skills

Siyao Xiao, Yuhong Zhang, Zhifang Liu, Zihan Gao+8

Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China. · PengCheng Laboratory, Shenzhen, China. · College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, China.+1

Language-Conditioned Robot Manipulation VLM Adaptation Generalization in Robotic Manipulation Robotic Manipulation Vision-Language-Action Models Robot Learning

Open paper AskPDF arXiv

Current Vision-Language-Action (VLA) models predominantly rely on end-to-end fine-tuning. While effective, this paradigm compromises the inherent generalization capabilities of Vision-Language Models (VLMs) and incurs catastrophic forgetting. To address these limitations, we propose

M^2

-VLA, which demonstrates that a generalized VLM is able to serve as a powerful backbone for robotic manipulation directly. However, it remains a key challenge to bridge the gap between the high-level semantic understanding of VLMs and the precise requirements of robotic control. To overcome this, we introduce the Mixture of Layers (MoL) strategy that selectively extracts task-critical information from dense semantic features. Furthermore, to facilitate efficient trajectory learning under constrained model capacity, we propose a Meta Skill Module (MSM) that integrates strong inductive biases. Extensive experiments in both simulated and real-world environments demonstrate the effectiveness of our approach. Furthermore, generalization and ablation studies validate the architecture's zero-shot capabilities and confirm the contribution of each key component. Our code and pre-trained models will be made publicly available.

Abstract

Vision-Language-Action Models

Explore similar work

M2M^2M2-VLA: Boosting Vision-Language Models for Generalizable Manipulation via Layer Mixture and Meta-Skills

Decoupling the Declarative from the Procedural in Vision-Language-Action Models

Continually Evolving Skill Knowledge in Vision Language Action Model

$M^2$ -VLA: Boosting Vision-Language Models for Generalizable Manipulation via Layer Mixture and Meta-Skills