Vision-Language-Action (VLA) models promise a single language-conditioned policy adaptable to many manipulation behaviors, yet deployment additionally requires skill retention and long-horizon composition: a policy must retain previously acquired skills and reuse them inside composite tasks whose behavior changes across phases. The standard recipe absorbs every new task into one shared update, which provides neither a parameter-level locus for reusable skills nor a mechanism for invoking each skill along the trajectory. We observe that this missing structure is latent rather than absent: per-skill low-rank updates of a pretrained VLA occupy a robustly distinguishable, low-dimensional subspace, and a single composite-task LoRA concentrates roughly half of its Frobenius energy in this subspace, against a near-zero matched random-rotation control. Building on this geometry, we introduce PRISM, which represents reusable skills as parameter-level primitives: each skill family is stored as a dedicated low-rank action-expert adapter in a primitive bank, primitives are augmented with skill-specific low-rank residuals and composed at inference by a per-chunk VLM-conditioned router that reads the prefix representation to track task progress, and the router is supervised by per-chunk labels from a deterministic, task-conditioned, action-derived oracle. During composite training, the vision encoder, the language LoRA, the skill residuals, the router, and the shared action projections and time MLP are updated; every primitive is held frozen, so the base per-skill action-expert adapters are preserved by construction. On RoboCasa365, PRISM achieves 56.43% atomic-skill success and reaches 20.00% on long-horizon composite tasks, while a standard π0.5-LoRA two-stage fine-tune reaches 17.86% atomic and 17.50% composite success. We additionally validate PRISM on an AgileX PIPER real-robot setup.
Figures & tables
Figure 1: Within-skill LoRAs concentrate; across-skill LoRAs spread across N . (a) Cumulative singular energy ρ≤k vs k/T : within-PP saturates at the first component, while cross-skill rises along the uniform null ρ≤k=k/T (dashed). (b) Top- 1 energy ρ1 : 0.948 vs 0.265 . (c) PR/T ( 1.0= maximal spread, 1/T= collapse): 0.37 vs 0.93 . (d) Effective rank /T : 0.42 vs 0.97 . Across all metrics, within-skill LoRAs are more concentrated than across-skill LoRAs.
Figure 2: Skill geometry, N=6 bank. Pairwise cosine cij between layer-wise-flattened per-skill update tensors ΔWi . Five manipulation skills form a coherent cluster ( 0.05 – 0.16 ); navigation is a clear outlier ( ≤0.043 ).
Figure 3: Per-skill update tensors span an effectively N -dim subspace. Normalized spectra overlap across N=6 and N=11 ; PR=5.59 at N=6 and PR=10.78 at N=11 .
Figure 4: Overview of the PRISM framework. (1) Skill primitive learning (§ 4 ): each atomic skill is fitted with its own rank- r action-expert LoRA (Bi,Ai) , yielding a primitive bank of disjoint slots, frozen after atomic training. (2) Router learning from composite demonstrations : a prefix-conditioned router rθ reads the VLM prefix ht and emits a per-chunk mixture α(t) over the bank, supervised by per-chunk labels from a deterministic action-derived oracle A . (3) Chunk-wise routed execution (inference): each LoRA-injected action-expert layer applies the routed mixture at the activation level (Eq. 5 ), producing an action chunk.
Method
Atomic SR
Δ Atomic
Composite SR
Δ Composite
π0.5 (zero-shot)
0.00
−17.86
0.00
−17.50
π0.5 -LoRA, atomic-only
48.04
+30.18
0.00
−17.50
π0.5 -LoRA, atomic + comp. †
17.86
0.00
17.50
0.00
π0.5 -LoRA + replay
49.82
+31.96
10.25
−7.25
PRISM, bank only (Phase 1)
50.71
+32.85
0.00
−17.50
PRISM (ours)
56.43
+38.57
20.00
+2.50
Table 1: Atomic and composite task success rate on RoboCasa365. Atomic SR is the macro-average over 28 atomic tasks ( 18 PP + 3 Lever + 7 Close); Composite SR is the macro-average over 4 composite tasks. “Atomic-only” rows are stage-1 checkpoints; “atomic + comp.” rows are the full two-stage fine-tune. PRISM bank only is the Phase-1 checkpoint without a fitted router. Δ columns report SR difference (pp) against the π0.5 -LoRA two-stage fine-tune ( † ).
Figure 5: Real-robot composite rollout under PRISM. A successful inference on the Cup → Table → Doll → Plate composite task: the robot first lifts the cup off the doll ( t0 – t4 ), then picks up the doll and places it on the plate ( t5 – t9 ). The router supplies a per-chunk mixture α(t) over the bank that switches from the Cup → Table primitive to the Doll → Plate primitive mid-trajectory.
Cup → Table
Doll → Plate
Composite
Method
SR
Δ
SR
Δ
SR
Δ
π0.5 (zero-shot)
0.0
−25.0
0.0
−35.0
0.0
−45.0
π0.5 -LoRA, atomic-only
55.0
+30.0
75.0
+40.0
0.0
−45.0
π0.5 -LoRA, atomic + comp. †
25.0
+0
35.0
+0
45.0
+0
π0.5 -LoRA + replay
50.0
+25.0
65.0
+30.0
50.0
+5.0
PRISM, bank only (Phase 1)
65.0
+40.0
80.0
+45.0
0.0
−45.0
Table 3: Real-robot success rate on AgileX PIPER. Two atomic skills (Cup → Table, Doll → Plate) and a Composite chaining them. Δ columns report absolute SR difference (pp) against the π0.5 -LoRA two-stage fine-tune ( † ). SR ↑ , Δ↑ .
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Episodes
Frames
Pick & Place ( 18 tasks)
PickPlaceCabinetToCounter
106
20,201
PickPlaceCounterToBlender
106
38,892
PickPlaceCounterToCabinet
108
24,225
PickPlaceCounterToDrawer
107
28,225
PickPlaceCounterToMicrowave
110
42,012
Appendix
Table 4: RoboCasa365 atomic dataset. Per-task episode and frame counts for the 28 atomic tasks used in PRISM’s training and evaluation.
Figure 6: Real-robot training data: fisheye observations. Representative frames from the wrist-mounted fisheye RGB stream used for teleoperated training, sampled left-to-right along each trajectory. Top: Cup → Table (Atomic 1) – the red cup is lifted off the doll and placed on the table. Middle: Doll → Plate (Atomic 2) – the doll is picked up and placed on the patterned plate. Bottom: the Composite chains the two atomic skills in sequence.
Vision-language-action (VLA) models can learn a wide range of robotic skills, but combining separately trained skills remains challenging. The original training data may be unavailable for joint retraining, while directly merging model weights can cause interference. We propose NestedVLA, which consolidates independently fine-tuned VLA policies without access to their original training data. Instead of merging all updates into one checkpoint, a hypernetwork embedded in the VLA generates action-expert weight updates for a selected task. Each task's compressed vision-language model update is stored separately, and both updates are added to a frozen base VLA to instantiate the corresponding policy. We evaluate NestedVLA by consolidating three policies fine-tuned with reinforcement learning on MetaWorld, ManiSkill, and CALVIN. It achieves 66.7% average success, improving on the strongest model-merging baseline by 41.1 percentage points. It performs within 1 percentage point of per-task fine-tuning on ManiSkill and substantially narrows the gap on CALVIN. In this three-policy setting, total storage is 1.5x that of a single full checkpoint. Code is available at https://github.com/WanyuGroup/NeurIPS2026-NestedVLA.
Guodong Du, Wanyu Lin
Department of Computing, The Hong Kong Polytechnic University
Vision-Language-Action (VLA) models have shown strong promise for general-purpose robotic manipulation, yet adapting them to new tasks and domains remains inefficient: existing methods often rely on parameter tuning, incurring substantial costs and risking catastrophic forgetting of previously learned tasks. To address this, we propose Optimus-R, a memory-centric VLA framework that formulates robotic adaptation as explicit query-skill memory tuning. Optimus-R introduces: (i) An Inline Memory Interface for skill extraction. It inserts learnable memory tokens into the VLA prefix stream, allowing the backbone to derive control-aware query and skill representations within the native action-conditioning pathway. (ii) A Query-Skill Memory Bank for skill learning. It externalizes skills into query prototypes for deciding what to retrieve and skill values for specifying how to act, supporting skill reuse and expansion with limited parameter updates. (iii) A lightweight Bridge-and-Adapt mechanism for skill updating. It aligns target-domain queries and skills with the existing memory space through a lightweight adapter and residual memory updates. Experiments on in-domain adaptation, cross-domain adaptation, and lifelong learning show that Optimus-R enables data-efficient skill learning while mitigating catastrophic forgetting.
Zaijing Li, Rui Shao, Bing Hu +3
Harbin Institute of Technology (Shenzhen) · Pengcheng Laboratory
Vision-Language-Action (VLA) models offer a promising paradigm for generalist robotic policies, yet their adaptation is hindered by data inefficiency and poor generalization. We argue that these bottlenecks stem from the prevailing Direct Instruction-to-Control Mapping, which forces models to memorize monolithic trajectories rather than reusable motion patterns, i.e., primitives. We propose PrimitiveVLA, a framework that shifts this paradigm toward a Primitive-Centric Disassemble & Assemble paradigm. Supported by a shared Multimodal Canonical Representation (MCR), PrimitiveVLA unifies two phases: (1) Fine-tuning-phase Disassembly, which uses an automated pipeline to disassemble demonstrations into reusable primitives; and (2) Inference-phase Assembly, which employs a VLM-based planner and an LLM-generated switch module for robust closed-loop execution. By disassembling tasks into reusable primitives, PrimitiveVLA enables VLA models to learn invariant motion patterns instead of task-specific trajectories. Extensive experiments show that our framework improves data efficiency and achieves superior zero-shot generalization across unseen and long-horizon tasks.
Yutai Li, Shaohui Peng, Jiaming Guo +8
State Key Lab of Processors, Institute of Computing Technology, CAS · Jiangsu Key Laboratory of AI for Industries, Institute of AI for Industries, CAS · University of Chinese Academy of Sciences +3