Adaptive Capacity Allocation for Vision Language Action Fine-tuning
Authors: Donghoon Kim, Minji Bae, Unghui Nam, Gyeonghun Kim, Suyun Lee, Kyuhong Shim, Byonghyo Shim
Organizations: Department of Electrical and Computer Engineering, Seoul National University, Seoul 08826, Republic of Korea · Department of Computer Science and Engineering, Sungkyunkwan University, Suwon 16419, Republic of Korea
Vision language action models (VLAs) are increasingly used for Physical AI, but deploying a pre-trained VLA model to unseen environments, embodiments, or tasks still requires adaptation. Parameter-efficient fine-tuning (PEFT), especially LoRA, is common for VLA policies, yet the exposed capacity knob, the rank, does not transfer uniformly: robotics transfer exhibits a higher and task-varying intrinsic rank than language fine-tuning. Small ranks suffice for LLMs (e.g., r∈{4,8}), while spectral analyses indicate VLAs may require much larger ranks (e.g., r≈128) or near-full rank, a mismatch that worsens in multi-task settings. We present LoRA-SP (Select-Prune), a rank-adaptive fine-tuning method that replaces fixed-rank updates with input- and layer-wise capacity. LoRA-SP uses an SVD-style parameterization with a small router whose nonnegative scores act as singular values over a shared vector bank. The active set is chosen by an energy target on the cumulative squared scores E(k)≥η, providing a direct link to approximation error via our spectral analysis. During training, η concentrates energy on a few directions and teaches the router to rely on fewer vectors while preserving accuracy. This yields compact adapters that reduce cross-task interference and improve generalization. On four real-robot manipulation tasks collected on an unseen AgileX PiPER arm, across two VLA backbones (π0 and SmolVLA), LoRA-SP matches or exceeds full fine-tuning with far fewer trainable parameters, and improves multi-task success by up to 31.6% over standard LoRA while remaining robust to rank choice.
Figures & tables
Fig. 1 : Rank–performance curves (accuracy/success relative to full fine-tuning; 1.0 = full FT). LLM (LLaMA-7B) reaches near-full-FT performance with very small ranks ( r∈{4,8} ), whereas VLA ( π0 -3.5B) improves steadily and only approaches parity around r≈128 , consistent with a higher intrinsic dimension in the VLA transfer setting.
Fig. 2 : VLA models are pre-trained on diverse manipulation tasks and robot embodiment (e.g., Franka Emika, WidowX, UR5e, Stretch) data. We compare transfers to a seen embodiment (Franka) versus an unseen embodiment (PiPER). Unseen-embodiment transfer changes both the robot’s kinematic specification (DoF, link lengths, joint limits) and the perception geometry (camera intrinsics/extrinsics, viewpoint) and workspace scale, which pushes the update to require a higher intrinsic rank ; by contrast, when the embodiment is seen, adaptation primarily compensates for perception/scale shifts and works with a lower rank .
Fig. 3 : Rank sensitivity in single- and multi-task LoRA fine-tuning on π0 model. (a) LoRA modules trained independently on each single task. While single-task modules also require higher ranks to reach full performance, their variance across tasks is lower than in the multi-task setting. Together, the results highlight the difficulty of choosing a single global rank that balances efficiency and accuracy across tasks, motivating rank-adaptive allocation. (b) Multi-task LoRA fine-tuning across four manipulation tasks. Success rate increases with rank but exhibits substantial variance across tasks, reflecting interference and heterogeneous capacity needs.
Fig. 4 : Spectral Rank Variation by Embodiment During π0 Fine-tuning. Number of singular values required to capture 99% of the total energy (normalized by the full rank) across different layers and modules. We compare π0 models fine-tuned on in-domain and out-of-domain data. The in-domain model is fine-tuned on the DROID dataset, which uses the robotic arm (Franka Panda) included in π0 ’s pretraining data. The out-of-domain model is fine-tuned on a dataset collected with the AgileX PiPER robotic arm, an embodiment absent from the pretraining data. The results show that the required rank varies by embodiment, and generalizing to a novel embodiment demands higher-rank to achieve comparable performance.
Fig. 5 : LoRA-SP (Select–Prune). (I) Overview: a wide vector bank (U,V) is trained together with a router on the backbone W0 . (II) Select: the router produces vector-level scores that act as singular values, forming an input- and layer-conditioned update ΔW=UΣ(x)V ; the histogram illustrates the spectral energy distribution across vectors. (III) Prune: only the smallest set of basis vectors whose cumulative energy exceeds the target η are kept, progressively reducing the active rank while maintaining accuracy.
Fig. 6 : Layer-wise distributions of active rank learned by LoRA-SP on validation data. Results are shown for (a) vision tower, (b) language model, and (c) action expert, for both SmolVLA (top) and π0 (bottom) backbones. Each boxplot shows min–LQ–median–UQ–max of the active rank across inputs. The vision tower consistently requires the highest ranks, the action expert shows wide variability, while the language model layers remain comparatively low and stable. This pattern highlights strong heterogeneity across modules and underscores the limitation of using a single global rank for VLA adaptation.
Fig. 7 : Real-world experimental setup for four tasks: (a) Open the Pot, (b) Pour the Block into the Basket, (c) Press the Button, and (d) Pick and Place the Grape in the Basket.
Robotics Institute, Carnegie Mellon University, Pittsburgh, USA · Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, USA