Train Together or Merge Later? Unifying VLA Experts via a Shared Action Interface

Zhizhen Zhang; Yuxia Fu; Zijian Wang; Helen Huang; Yadan Luo

cs.CVSep 27, 2026

Train Together or Merge Later? Unifying VLA Experts via a Shared Action Interface

Authors: Zhizhen Zhang, Yuxia Fu, Zijian Wang, Helen Huang, Yadan Luo

Organizations: UQMM Lab, The University of Queensland

Abstract

Co-training offers a straightforward way to build a multi-task vision-language-action (VLA) policy, but can fall short of the performance achieved by training each task independently. The challenge is to retain these task-specific gains in a multi-task policy without joint post-training. Combining independently trained experts through model merging is a natural approach, yet strong individual experts do not necessarily yield a strong merged policy. We identify one source of this incompatibility: task-specific changes to the action interface, comprising action normalization and the action encoder and decoder. We propose PolicyWeave, combining merge-compatible post-training with context-guided sparse merging. During post-training, all experts retain the common base policy's action interface, while task adaptation is restricted to LoRA updates in the hidden layers of the action model. This makes the experts more compatible with existing model merging methods. However, merging all experts can still introduce interference from unrelated tasks at deployment. PolicyWeave scores each expert's LoRA updates using the initial visual-language context, determines the expert set through leave-one-layer-out ranking stability, and forms a sparse weighted merge of the selected updates that remains fixed for current task. We evaluate PolicyWeave with GR00T N1.5 on 18 RoboCasa365 tasks, using only 10% of the target-task demonstrations for supervised fine-tuning (SFT). Preserving the shared action interface raises the average success rate across four static merging methods from 17.0% to 52.8%. PolicyWeave achieves 64.7% success with these SFT experts and 74.1% after task-specific reinforcement learning (RL), compared with 60.7% for joint RL. Further evaluations on LIBERO-10 and an AgileX Piper arm support the deployment of independently learned skills in long-horizon and real-world manipulation.

Figures & tables

Figure 1: (a) Co-training, task-specific adaptation, and PolicyWeave . (b) Mean success (%) on 18 RoboCasa365 tasks with SFT and SFT then RL, comparing co-training with model merging.

Figure 2: A shared action interface supports effective merging of experts trained on 18 RoboCasa365 tasks. (a) Mean SFT success across these tasks for four static merging methods; Avg. denotes their unweighted mean. (b) Different action interfaces, the same weight-averaging (WA) rule. Paired OpenDrawer rollouts show successful opening only with the shared interface.

	Baselines		Merging methods
Task	Indiv.	Co- training	WA	TIES	ISO- CTS	TSV	SSR- Merge	Policy Weave
Close blender lid	22 $\rightarrow$ 32	8 $\rightarrow$ 24	8 $\rightarrow$ 8	8 $\rightarrow$ 18	10 $\rightarrow$ 0	4 $\rightarrow$ 10	14 $\rightarrow$ 10	20 $\rightarrow$ 30
Close fridge	68 $\rightarrow$ 74	64 $\rightarrow$ 70	74 $\rightarrow$ 74	66 $\rightarrow$ 78	62 $\rightarrow$ 68	78 $\rightarrow$ 68	70 $\rightarrow$ 82	78 $\rightarrow$ 76
Close toaster oven	80 $\rightarrow$ 80	68 $\rightarrow$ 76	46 $\rightarrow$ 56	54 $\rightarrow$ 40	68 $\rightarrow$ 76	68 $\rightarrow$ 68	46 $\rightarrow$ 44	78 $\rightarrow$ 76
Set up coffee mug	50 $\rightarrow$ 76	50 $\rightarrow$ 40	26 $\rightarrow$ 26	28 $\rightarrow$ 32	40 $\rightarrow$ 54	32 $\rightarrow$ 50	30 $\rightarrow$ 52	56 $\rightarrow$ 70
Navigate kitchen	22 $\rightarrow$ 68	28 $\rightarrow$ 28	2 $\rightarrow$ 6	6 $\rightarrow$ 10	8 $\rightarrow$ 14	14 $\rightarrow$ 22	6 $\rightarrow$ 6	20 $\rightarrow$ 62

Table 1: RoboCasa365 success (%), averaged over 50 episodes per task. Each pair reports SFT

\rightarrow

SFT then RL . All merging methods use the same shared-interface expert bank. Bold marks the best merging result for each task and stage.

Figure 3: RoboCasa365 Atomic-Seen leaderboard references.

Figure 4: Long-horizon performance on LIBERO-10. Left: published results for OpenVLA ( Kim et al., 2025 ) , SpatialVLA ( Qu et al., 2025 ) , CoT-VLA ( Zhao et al., 2025a ) , and co-trained GR00T N1.5 baseline. Right: success rates across merging methods using the same expert bank.

Method	Indiv. $\rightarrow$ Merged	Ret. $\uparrow$
MergeVLA	$89.6\rightarrow 70.8$	79.0
PolicyWeave	$80.8\rightarrow 75.4$	93.3

Table 2: LIBERO-10 merging retention.

Figure 5: Action normalization and closed-loop base drift on RoboCasa365. (a) Differences between task-specific and shared normalization of the same raw actions. (b) The largest differences occur in base-motion channels; rows follow (a). (c) Actual base displacement from a matched initial scene on PickPlaceCounterToStove. The views show step 160.

Setting	Success (%)
Individual	64.1
Encoder WA	17.1 ( $-47.0\%$ )
Decoder WA	60.4 ( $-3.7\%$ )
Base Normalization	12.3 ( $-51.8\%$ )

Table 3: Sensitivity of individual experts to action-interface changes.

Method	Top-1 Acc.	Mean $k$	Success (all)	Subset success
Trainable MLP	89.4	1	69.4	–
PolicyWeave (Top-1)	93.6	1	73.2	36.4
PolicyWeave (Top-2)	93.6	2	66.2	57.6
PolicyWeave	93.6	1.076	75.4	63.6
PolicyWeave w/o shared interface	90.2	1.112	70.0	0.0

Table 4: LIBERO-10 selection and merging (500 episodes). Accuracy and success are in %;

\bar{k}

is the mean number of selected experts. The three shared-interface PolicyWeave variants use the same context-response scorer. Subset success uses cases where adaptive selection merges multiple experts, defined separately for each expert bank.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Aspect	CORAL	MergeVLA	PolicyWeave
Expert construction	Isolated task LoRAs on a frozen base	Experts trained with a merge-oriented design	Independent task LoRAs with a shared action interface
Action model	Original action head retained and adapted by LoRA	Cross-attention-only expert; deeper task heads retained	Original architecture retained; action encoder and decoder fixed
Task-unknown selection	Externally specified instruction-to-expert function	Model-response-based task selection	Expert-update responses with stability-adaptive selection
Deployment	One selected LoRA folded into the base	Merged shared blocks with a selected mask and task head	Static policy from the shared-interface bank, or context-guided sparse merging
Static merging	Not part of the proposed method	Coupled with task-specific masks and heads	Existing methods produce one policy without task-specific selection

Appendix

Table 6: Comparison with CORAL and MergeVLA. “Task unknown” means that an expert index is not provided at inference time.

Benchmark	Data per task	Steps	Batch	LR	Weight decay
RoboCasa365	50 demos (10%)	10,000	128	$3\!\times\!10^{-5}$	$10^{-5}$
LIBERO-10	all (37.9 avg.)	10,000	128	$3\!\times\!10^{-5}$	$10^{-5}$

Appendix

Table 7: SFT configuration for each task-specific expert.

Training	RL steps	Envs	Env. steps (M)	Batch	Actor LR	Value LR
Individual	33.9 avg. (610 total)	16	23.74	8/304	$10^{-5}$	$10^{-4}$
Co-training	600 total	18	26.27	8/304	$10^{-5}$	$10^{-4}$

Appendix

Table 8: RoboCasa365 SFT then RL configurations at the selected checkpoints. M denotes million; batch sizes are micro/global.

RL step	0	100	200	300	400	500	600
Success	58.1	57.2	59.8	59.6	60.1	59.8	60.7

Appendix

Table 9: RoboCasa365 co-training RL checkpoint-selection curve. Values are mean validation success rates (%) over the 18 tasks.

Action interface	Individual	WA	TIES	ISO-CTS	TSV-Merge	Retention range
Task-specific	64.1	16.1	13.9	18.9	18.9	21.7–29.5
PolicyWeave	65.6	49.6	52.9	54.1	54.7	75.6–83.4

Appendix

Table 10: Static merging under task-specific and shared action interfaces on RoboCasa365. Retention is the success rate of each merging method divided by Individual success. All entries are percentages.

Metric	WA	TIES	ISO-CTS	TSV-Merge
Final success	0.0	0.0	0.0	0.0
Partial success	10.6	9.8	8.3	13.2

Appendix

Table 11: LIBERO-10 aggregate success rates without a shared action interface. Partial success measures the maximum fraction of goal predicates satisfied simultaneously during an episode, averaged across episodes. All entries are percentages.

	Baselines		Merging methods
Task	Indiv.	GR00T [-1pt]N1.5	WA	TIES	ISO- [-1pt]CTS	TSV	Policy [-1pt] Weave
soup + sauce $\rightarrow$ basket	88 [-1pt] 91	72 [-1pt] 80	24 [-1pt] 62	34 [-1pt] 67	28 [-1pt] 63	48 [-1pt] 73	84 [-1pt] 89
cream cheese + butter $\rightarrow$ basket	86 [-1pt] 92	84 [-1pt] 92	10 [-1pt] 51	12 [-1pt] 52	24 [-1pt] 58	34 [-1pt] 65	70 [-1pt] 84
turn on stove + moka pot	94 [-1pt] 96	78 [-1pt] 88	0 [-1pt] 50	0 [-1pt] 50	0 [-1pt] 50	0 [-1pt] 50	90 [-1pt] 94
black bowl $\rightarrow$ bottom drawer + close	90 [-1pt] 95	90 [-1pt] 94	0 [-1pt] 50	2 [-1pt] 51	2 [-1pt] 51	18 [-1pt] 58	74 [-1pt] 87
two mugs $\rightarrow$ two plates	66 [-1pt] 79	82 [-1pt] 89	6 [-1pt] 49	2 [-1pt] 46	10 [-1pt] 50	58 [-1pt] 76	66 [-1pt] 78

Appendix

Table 12: LIBERO-10 success (%), averaged over 50 episodes per task. Each cell reports final success above partial success , the maximum fraction of goal predicates satisfied simultaneously during an episode. Bold marks the best merging result for each task and metric.

Task index	FT (%)	Merged (%)
0	86	76
1	98	98
2	100	94
3	100	74
4	84	78
5	90	82

Appendix

Table 13: MergeVLA reproduction on LIBERO-10. FT evaluates the task-specific fine-tuned policies and Merged evaluates the resulting merged policy.

Expert bank	Episodes	$k=1$	$k=2$	$k=3$	$k=4$	$k>1$ (%)	Mean $k$
RoboCasa365 SFT	900	887	13	0	0	1.44	1.014
RoboCasa365 SFT then RL	900	889	11	0	0	1.22	1.012
LIBERO-10 SFT	500	467	29	3	1	6.60	1.076

Appendix

Table 14: Expert selection sizes under PolicyWeave . Columns

k=1

through

k=4

give episode counts;

k>1

gives the percentage of episodes selecting multiple experts. No episode selects more than four experts.

Panel of Figure 7 — Figure 7: Row-normalized Top-1 routing confusion matrices (%) for the SFT banks: (a) RoboCasa365 (900 episodes); (b) LIBERO-10 (500 episodes). Rows indicate the requested task; columns indicate the selected expert.

Top-1 routing accuracy
Condition	Trainable MLP	Sem-NN	PolicyWeave
Native	94.33	80.56	99.44
Lexical/syntactic	79.67	83.78	91.33
Goal-state	67.78	82.56	81.78
Closed-loop success
Native	63.1	62.0	64.7

Appendix

Table 15: RoboCasa365 SFT selection baselines (%). Routing accuracy uses 900 recorded contexts per condition; closed-loop success uses 50 episodes per task. Sem-NN denotes task-name-based semantic retrieval.

Timestep	0	16	32	48
Top-1 accuracy (%)	99.44	99.56	99.89	99.89

Appendix

Table 16: PolicyWeave expert identification from early-trajectory RoboCasa365 observations (900 contexts per timestep).

	Normalized response			Raw response
Scoring layers	$\mathrm{K}$	$\mathrm{V}$	$\mathrm{K+V}$	$\mathrm{K}$	$\mathrm{V}$	$\mathrm{K+V}$
Cross 0	83.33	83.89	89.00	76.89	77.89	85.89
Cross 1	95.78	97.56	98.00	96.33	98.00	98.67
Cross 2	93.11	95.56	94.78	93.11	96.33	95.67
Cross 3	99.56	98.33	99.22	99.56	96.89	99.56
Cross 4	96.67	99.11	99.22	96.11	97.22	98.11

Appendix

Table 17: Top-1 routing accuracy (%) for all 66 scorer configurations on native instructions.

Experts	Top-1 (%) $\uparrow$	Expert recall (%) $\uparrow$	Mean $k$ $\downarrow$	Storage ( $10^{6}$ bytes)	Route p50 / p95 (ms)
1	100.00	100.00	1.000	19.17	80.67 / 146.15
2	98.00	98.00	1.020	38.35	80.22 / 143.20
4	98.00	98.00	1.020	76.70	79.89 / 140.64
8	98.00	98.00	1.020	153.39	84.73 / 128.61
12	98.00	98.00	1.020	230.09	83.45 / 142.28
18	98.00	98.00	1.020	345.14	90.61 / 139.86

Appendix

Table 18: PolicyWeave statistics as the available expert bank grows. Each bank size is evaluated on 250 native-instruction observations. Routing time reports p50 / p95 latency for a batch of five with the model and expert bank resident on the GPU.

Setting	Shared GR00T adaptation	PolicyWeave experts	GR00T co-training	$\pi_{0.5}$ co-training
Demonstrations	1 per task	50 per expert	51 per task	51 per task
Maximum steps	3,000	10,000	40,000	40,000
Batch size	128	256	256	256
Gradient accumulation	8	1	1	1
Peak learning rate	$3\times 10^{-5}$	$10^{-4}$	$10^{-4}$	$10^{-4}$
LoRA rank / scale	–	8 / 16	8 / 16	8 / 16

Appendix

Table 19: Piper training configurations. Batch sizes are effective totals; the expert configuration applies independently to each of the four tasks.

Figure 8: Successful PolicyWeave rollouts on the Piper arm. Each row follows one video from left to right; timestamps indicate elapsed video time. Insets show the wrist-camera view.

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,194 papers

NestedVLA: Learning to Consolidate and Generate Skills for Vision-Language-Action Models
Oct 16, 2026Guodong Du, Wanyu LinVision-Language-Action Models Continual Learning for VLA Models
Department of Computing, The Hong Kong Polytechnic University
WeaveLA: Event Driven Cross-Subtask Latent Memory Weaving for Repetitive Robot Manipulation
Jun 16, 2026Shoujing Zhu, Zhenyang Liu, Fungmiu Wang +6Long-Horizon Robotic Manipulation Vision-Language-Action Models
Fudan University · Shanghai Innovation Institute · School of Data Science, The Chinese University of Hong Kong, Shenzhen +1
APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
Jun 10, 2026Kechun Xu, Zhenjie Zhu, Anzhe Chen +2Visuomotor Policy Learning Vision-Language-Action Models
Zhejiang University · Zhejiang Humanoid Robot Innovation Center

Oct 16, 2026J/K move · Enter open · S save

NestedVLA: Learning to Consolidate and Generate Skills for Vision-Language-Action Models

Guodong Du, Wanyu Lin

Department of Computing, The Hong Kong Polytechnic University

Vision-Language-Action Models Continual Learning for VLA Models Efficient VLA Models

Open paper AskPDF arXiv

Vision-language-action (VLA) models can learn a wide range of robotic skills, but combining separately trained skills remains challenging. The original training data may be unavailable for joint retraining, while directly merging model weights can cause interference. We propose NestedVLA, which consolidates independently fine-tuned VLA policies without access to their original training data. Instead of merging all updates into one checkpoint, a hypernetwork embedded in the VLA generates action-expert weight updates for a selected task. Each task's compressed vision-language model update is stored separately, and both updates are added to a frozen base VLA to instantiate the corresponding policy. We evaluate NestedVLA by consolidating three policies fine-tuned with reinforcement learning on MetaWorld, ManiSkill, and CALVIN. It achieves 66.7% average success, improving on the strongest model-merging baseline by 41.1 percentage points. It performs within 1 percentage point of per-task fine-tuning on ManiSkill and substantially narrows the gap on CALVIN. In this three-policy setting, total storage is 1.5x that of a single full checkpoint. Code is available at https://github.com/WanyuGroup/NeurIPS2026-NestedVLA.

Abstract

Vision-Language-Action Models

Explore similar work

NestedVLA: Learning to Consolidate and Generate Skills for Vision-Language-Action Models

WeaveLA: Event Driven Cross-Subtask Latent Memory Weaving for Repetitive Robot Manipulation

APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies