Co-training offers a straightforward way to build a multi-task vision-language-action (VLA) policy, but can fall short of the performance achieved by training each task independently. The challenge is to retain these task-specific gains in a multi-task policy without joint post-training. Combining independently trained experts through model merging is a natural approach, yet strong individual experts do not necessarily yield a strong merged policy. We identify one source of this incompatibility: task-specific changes to the action interface, comprising action normalization and the action encoder and decoder. We propose PolicyWeave, combining merge-compatible post-training with context-guided sparse merging. During post-training, all experts retain the common base policy's action interface, while task adaptation is restricted to LoRA updates in the hidden layers of the action model. This makes the experts more compatible with existing model merging methods. However, merging all experts can still introduce interference from unrelated tasks at deployment. PolicyWeave scores each expert's LoRA updates using the initial visual-language context, determines the expert set through leave-one-layer-out ranking stability, and forms a sparse weighted merge of the selected updates that remains fixed for current task. We evaluate PolicyWeave with GR00T N1.5 on 18 RoboCasa365 tasks, using only 10% of the target-task demonstrations for supervised fine-tuning (SFT). Preserving the shared action interface raises the average success rate across four static merging methods from 17.0% to 52.8%. PolicyWeave achieves 64.7% success with these SFT experts and 74.1% after task-specific reinforcement learning (RL), compared with 60.7% for joint RL. Further evaluations on LIBERO-10 and an AgileX Piper arm support the deployment of independently learned skills in long-horizon and real-world manipulation.
Figures & tables
Figure 1: (a) Co-training, task-specific adaptation, and PolicyWeave . (b) Mean success (%) on 18 RoboCasa365 tasks with SFT and SFT then RL, comparing co-training with model merging.
Figure 2: A shared action interface supports effective merging of experts trained on 18 RoboCasa365 tasks. (a) Mean SFT success across these tasks for four static merging methods; Avg. denotes their unweighted mean. (b) Different action interfaces, the same weight-averaging (WA) rule. Paired OpenDrawer rollouts show successful opening only with the shared interface.
Baselines
Merging methods
Task
Indiv.
Co- training
WA
TIES
ISO- CTS
TSV
SSR- Merge
Policy Weave
Close blender lid
22 → 32
8 → 24
8 → 8
8 → 18
10 → 0
4 → 10
14 → 10
20 → 30
Close fridge
68 → 74
64 → 70
74 → 74
66 → 78
62 → 68
78 → 68
70 → 82
78 → 76
Close toaster oven
80 → 80
68 → 76
46 → 56
54 → 40
68 → 76
68 → 68
46 → 44
78 → 76
Set up coffee mug
50 → 76
50 → 40
26 → 26
28 → 32
40 → 54
32 → 50
30 → 52
56 → 70
Navigate kitchen
22 → 68
28 → 28
2 → 6
6 → 10
8 → 14
14 → 22
6 → 6
20 → 62
Table 1: RoboCasa365 success (%), averaged over 50 episodes per task. Each pair reports SFT → SFT then RL . All merging methods use the same shared-interface expert bank. Bold marks the best merging result for each task and stage.
Figure 4: Long-horizon performance on LIBERO-10. Left: published results for OpenVLA ( Kim et al., 2025 ) , SpatialVLA ( Qu et al., 2025 ) , CoT-VLA ( Zhao et al., 2025a ) , and co-trained GR00T N1.5 baseline. Right: success rates across merging methods using the same expert bank.
Method
Indiv. → Merged
Ret. ↑
MergeVLA
89.6→70.8
79.0
PolicyWeave
80.8→75.4
93.3
Table 2: LIBERO-10 merging retention.
Figure 5: Action normalization and closed-loop base drift on RoboCasa365. (a) Differences between task-specific and shared normalization of the same raw actions. (b) The largest differences occur in base-motion channels; rows follow (a). (c) Actual base displacement from a matched initial scene on PickPlaceCounterToStove. The views show step 160.
Setting
Success (%)
Individual
64.1
Encoder WA
17.1 ( −47.0% )
Decoder WA
60.4 ( −3.7% )
Base Normalization
12.3 ( −51.8% )
Table 3: Sensitivity of individual experts to action-interface changes.
Method
Top-1 Acc.
Mean k
Success (all)
Subset success
Trainable MLP
89.4
1
69.4
–
PolicyWeave (Top-1)
93.6
1
73.2
36.4
PolicyWeave (Top-2)
93.6
2
66.2
57.6
PolicyWeave
93.6
1.076
75.4
63.6
PolicyWeave w/o shared interface
90.2
1.112
70.0
0.0
Table 4: LIBERO-10 selection and merging (500 episodes). Accuracy and success are in %; kˉ is the mean number of selected experts. The three shared-interface PolicyWeave variants use the same context-response scorer. Subset success uses cases where adaptive selection merges multiple experts, defined separately for each expert bank.
Figure 10
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Aspect
CORAL
MergeVLA
PolicyWeave
Expert construction
Isolated task LoRAs on a frozen base
Experts trained with a merge-oriented design
Independent task LoRAs with a shared action interface
Original architecture retained; action encoder and decoder fixed
Task-unknown selection
Externally specified instruction-to-expert function
Model-response-based task selection
Expert-update responses with stability-adaptive selection
Deployment
One selected LoRA folded into the base
Merged shared blocks with a selected mask and task head
Static policy from the shared-interface bank, or context-guided sparse merging
Static merging
Not part of the proposed method
Coupled with task-specific masks and heads
Existing methods produce one policy without task-specific selection
Appendix
Table 6: Comparison with CORAL and MergeVLA. “Task unknown” means that an expert index is not provided at inference time.
Benchmark
Data per task
Steps
Batch
LR
Weight decay
RoboCasa365
50 demos (10%)
10,000
128
3×10−5
10−5
LIBERO-10
all (37.9 avg.)
10,000
128
3×10−5
10−5
Appendix
Table 7: SFT configuration for each task-specific expert.
Training
RL steps
Envs
Env. steps (M)
Batch
Actor LR
Value LR
Individual
33.9 avg. (610 total)
16
23.74
8/304
10−5
10−4
Co-training
600 total
18
26.27
8/304
10−5
10−4
Appendix
Table 8: RoboCasa365 SFT then RL configurations at the selected checkpoints. M denotes million; batch sizes are micro/global.
RL step
0
100
200
300
400
500
600
Success
58.1
57.2
59.8
59.6
60.1
59.8
60.7
Appendix
Table 9: RoboCasa365 co-training RL checkpoint-selection curve. Values are mean validation success rates (%) over the 18 tasks.
Action interface
Individual
WA
TIES
ISO-CTS
TSV-Merge
Retention range
Task-specific
64.1
16.1
13.9
18.9
18.9
21.7–29.5
PolicyWeave
65.6
49.6
52.9
54.1
54.7
75.6–83.4
Appendix
Table 10: Static merging under task-specific and shared action interfaces on RoboCasa365. Retention is the success rate of each merging method divided by Individual success. All entries are percentages.
Metric
WA
TIES
ISO-CTS
TSV-Merge
Final success
0.0
0.0
0.0
0.0
Partial success
10.6
9.8
8.3
13.2
Appendix
Table 11: LIBERO-10 aggregate success rates without a shared action interface. Partial success measures the maximum fraction of goal predicates satisfied simultaneously during an episode, averaged across episodes. All entries are percentages.
Baselines
Merging methods
Task
Indiv.
GR00T [-1pt]N1.5
WA
TIES
ISO- [-1pt]CTS
TSV
Policy [-1pt] Weave
soup + sauce → basket
88 [-1pt] 91
72 [-1pt] 80
24 [-1pt] 62
34 [-1pt] 67
28 [-1pt] 63
48 [-1pt] 73
84 [-1pt] 89
cream cheese + butter → basket
86 [-1pt] 92
84 [-1pt] 92
10 [-1pt] 51
12 [-1pt] 52
24 [-1pt] 58
34 [-1pt] 65
70 [-1pt] 84
turn on stove + moka pot
94 [-1pt] 96
78 [-1pt] 88
0 [-1pt] 50
0 [-1pt] 50
0 [-1pt] 50
0 [-1pt] 50
90 [-1pt] 94
black bowl → bottom drawer + close
90 [-1pt] 95
90 [-1pt] 94
0 [-1pt] 50
2 [-1pt] 51
2 [-1pt] 51
18 [-1pt] 58
74 [-1pt] 87
two mugs → two plates
66 [-1pt] 79
82 [-1pt] 89
6 [-1pt] 49
2 [-1pt] 46
10 [-1pt] 50
58 [-1pt] 76
66 [-1pt] 78
Appendix
Table 12: LIBERO-10 success (%), averaged over 50 episodes per task. Each cell reports final success above partial success , the maximum fraction of goal predicates satisfied simultaneously during an episode. Bold marks the best merging result for each task and metric.
Task index
FT (%)
Merged (%)
0
86
76
1
98
98
2
100
94
3
100
74
4
84
78
5
90
82
Appendix
Table 13: MergeVLA reproduction on LIBERO-10. FT evaluates the task-specific fine-tuned policies and Merged evaluates the resulting merged policy.
Expert bank
Episodes
k=1
k=2
k=3
k=4
k>1 (%)
Mean k
RoboCasa365 SFT
900
887
13
0
0
1.44
1.014
RoboCasa365 SFT then RL
900
889
11
0
0
1.22
1.012
LIBERO-10 SFT
500
467
29
3
1
6.60
1.076
Appendix
Table 14: Expert selection sizes under PolicyWeave . Columns k=1 through k=4 give episode counts; k>1 gives the percentage of episodes selecting multiple experts. No episode selects more than four experts.
Figure 7: Row-normalized Top-1 routing confusion matrices (%) for the SFT banks: (a) RoboCasa365 (900 episodes); (b) LIBERO-10 (500 episodes). Rows indicate the requested task; columns indicate the selected expert.
Condition
Trainable MLP
Sem-NN
PolicyWeave
Top-1 routing accuracy
Native
94.33
80.56
99.44
Lexical/syntactic
79.67
83.78
91.33
Goal-state
67.78
82.56
81.78
Closed-loop success
Native
63.1
62.0
64.7
Appendix
Table 15: RoboCasa365 SFT selection baselines (%). Routing accuracy uses 900 recorded contexts per condition; closed-loop success uses 50 episodes per task. Sem-NN denotes task-name-based semantic retrieval.
Timestep
0
16
32
48
Top-1 accuracy (%)
99.44
99.56
99.89
99.89
Appendix
Table 16: PolicyWeave expert identification from early-trajectory RoboCasa365 observations (900 contexts per timestep).
Normalized response
Raw response
Scoring layers
K
V
K+V
K
V
K+V
Cross 0
83.33
83.89
89.00
76.89
77.89
85.89
Cross 1
95.78
97.56
98.00
96.33
98.00
98.67
Cross 2
93.11
95.56
94.78
93.11
96.33
95.67
Cross 3
99.56
98.33
99.22
99.56
96.89
99.56
Cross 4
96.67
99.11
99.22
96.11
97.22
98.11
Appendix
Table 17: Top-1 routing accuracy (%) for all 66 scorer configurations on native instructions.
Experts
Top-1 (%) ↑
Expert recall (%) ↑
Mean k↓
Storage ( 106 bytes)
Route p50 / p95 (ms)
1
100.00
100.00
1.000
19.17
80.67 / 146.15
2
98.00
98.00
1.020
38.35
80.22 / 143.20
4
98.00
98.00
1.020
76.70
79.89 / 140.64
8
98.00
98.00
1.020
153.39
84.73 / 128.61
12
98.00
98.00
1.020
230.09
83.45 / 142.28
18
98.00
98.00
1.020
345.14
90.61 / 139.86
Appendix
Table 18: PolicyWeave statistics as the available expert bank grows. Each bank size is evaluated on 250 native-instruction observations. Routing time reports p50 / p95 latency for a batch of five with the model and expert bank resident on the GPU.
Setting
Shared GR00T adaptation
PolicyWeave experts
GR00T co-training
π0.5 co-training
Demonstrations
1 per task
50 per expert
51 per task
51 per task
Maximum steps
3,000
10,000
40,000
40,000
Batch size
128
256
256
256
Gradient accumulation
8
1
1
1
Peak learning rate
3×10−5
10−4
10−4
10−4
LoRA rank / scale
–
8 / 16
8 / 16
8 / 16
Appendix
Table 19: Piper training configurations. Batch sizes are effective totals; the expert configuration applies independently to each of the four tasks.
Figure 8: Successful PolicyWeave rollouts on the Piper arm. Each row follows one video from left to right; timestamps indicate elapsed video time. Insets show the wrist-camera view.
Vision-language-action (VLA) models can learn a wide range of robotic skills, but combining separately trained skills remains challenging. The original training data may be unavailable for joint retraining, while directly merging model weights can cause interference. We propose NestedVLA, which consolidates independently fine-tuned VLA policies without access to their original training data. Instead of merging all updates into one checkpoint, a hypernetwork embedded in the VLA generates action-expert weight updates for a selected task. Each task's compressed vision-language model update is stored separately, and both updates are added to a frozen base VLA to instantiate the corresponding policy. We evaluate NestedVLA by consolidating three policies fine-tuned with reinforcement learning on MetaWorld, ManiSkill, and CALVIN. It achieves 66.7% average success, improving on the strongest model-merging baseline by 41.1 percentage points. It performs within 1 percentage point of per-task fine-tuning on ManiSkill and substantially narrows the gap on CALVIN. In this three-policy setting, total storage is 1.5x that of a single full checkpoint. Code is available at https://github.com/WanyuGroup/NeurIPS2026-NestedVLA.
Guodong Du, Wanyu Lin
Department of Computing, The Hong Kong Polytechnic University
Vision-Language-Action (VLA) policies have achieved remarkable single-step manipulation, yet they remain brittle precisely where each stage depends on what was just completed. The core issue is structural: short-window VLAs lack an explicit channel for rouxting information across sub-task boundaries, and existing memory-augmented variants either write at every frame, retrieve from demonstration-time stages, or fire at sub-goal events without performing an explicit sub-task-to-sub-task hand-off into the action expert. We identify the sub-goal completion event as the natural temporal unit for cross-subtask memory hand-off, and present WeaveLA (Weave Latent memory for Vision-Language-Action policies), a cross-subtask memory interface that, on top of a frozen VLA backbone, compresses each completed segment into latent tokens via query-driven attention pooling and routes them directly into the action-generation path of the next sub-task. This event-triggered, action-side design preserves the base policy's short-window interface while adding a lightweight cross-subtask channel. Through stratified evaluation on RoboMME with a π0.5 backbone, WeaveLA's gains land exactly where the channel is needed: on the hardest repetition slice (SwingXtimes, N=3), success rises from 0% to 47.8%, while single-execution episodes remain unchanged. Per-episode paired analysis confirms the gains are confined to tasks whose causal structure requires cross-subtask information.
Shoujing Zhu, Zhenyang Liu, Fungmiu Wang +6
Fudan University · Shanghai Innovation Institute · School of Data Science, The Chinese University of Hong Kong, Shenzhen +1
Vision-Language-Action (VLA) models that couple pretrained Vision-Language Models (VLMs) with continuous action experts have achieved strong manipulation performance, yet generalization to out-of-distribution (OOD) language instructions remains poor. A known challenge is the structural imbalance in VLA data, where language is far less diverse than visual and action content, making policies prone to visual shortcuts. While discrete-action methods mitigate this through vision-language co-training, continuous action experts lack such protection: they start from random initialization and learn entirely from imbalanced data, producing noisy gradients that corrupt the VLM and fail to exploit its language capability. We address this from a Bayesian perspective, factorizing the policy into a language-agnostic Vision-Action (VA) prior and a language-conditioned VLA likelihood, and propose APT, a two-stage training method emphasizing Action expert PreTraining. In Stage 1, the action expert is pretrained as a VA prior on vision-action pairs from a frozen VLM, bypassing the language imbalance. In Stage 2, language tokens are injected through a gated fusion mechanism that integrates VLM features while preserving the learned visuomotor prior. APT applies to mainstream VLA architectures, including the π and GR00T-style architectures. Comprehensive experiments validate that APT achieves consistent gains on unseen instructions and compositional tasks. Project Page: https://xukechun.github.io/papers/APT/
Kechun Xu, Zhenjie Zhu, Anzhe Chen +2
Zhejiang University · Zhejiang Humanoid Robot Innovation Center