Full-Weight VLA Fine-Tuning Without an fp32 Master Copy Silently Erases Most Updates

Shubham Singh

cs.ROOct 22, 2026

Full-Weight VLA Fine-Tuning Without an fp32 Master Copy Silently Erases Most Updates

Authors: Shubham Singh

Abstract

When a vision-language-action (VLA) model is fine-tuned with its trainable weights in bfloat16 (bf16) and no fp32 master copy, AdamW writes each update straight into bf16, silently rounding away any update smaller than 1/512 of a weight. In 9 of 12 open VLA codebases a default or documented recipe trains this way, 4 on full weight matrices, including LeRobot's SmolVLA default and its $π_0$ / $π_{0.5}$ guides. In our LIBERO runs without the copy, a probe every 100 steps finds at least 79% of updates to bf16 weights erased. On the Hugging Face Hub, 99% of sampled full-weight SmolVLA fine-tunes store the action expert in bf16, and 84% of recent $π_{0.5}$ fine-tunes with a training configuration set the bf16 flag. On LIBERO-Object, with rules fixed in advance, an fp32 copy raises success over LeRobot's storage from 4.3% to 49.3% for SmolVLA and from 5.1% to 80.9% for $π_{0.5}$ at a ten-times lower learning rate, where both models are undertrained, and from 89.0% to 97.7% for $π_{0.5}$ at its default rate. For SmolVLA at its default rate it adds 10.2 points with open-loop 50-action chunks but only 4.2 when re-planning every 10 actions. In these five comparisons and two more against SmolVLA with every trainable weight in bf16, every run with the copy beats every run without it (p < 0.05); all gaps but the 8.7- and 4.2-point ones clear our 10-point threshold. The fix is to keep the trainable weights in fp32 during training, a one-line change for SmolVLA in LeRobot.

Figures & tables

Fig. 1: Without an fp32 copy, measured erasure always exceeds the share predicted from the weights. Lines:

E(u)

from released weights (solid: SmolVLA’s action expert in smolvla_base ; dashed: the weight matrices that

\pi_{0.5}

’s documented recipe stores in bf16, in pi05_libero_finetuned_v044 ). Markers: the share of those weights unchanged after an AdamW step, every 100 steps, at that step’s learning rate; for

\pi_{0.5}

, the bf16 weights in tensors that received a gradient, including the token embedding, which the dashed line leaves out. SmolVLA’s runs of LeRobot’s recipe, also above

E

, are left out for legibility.

Codebase: recipe	Model (size)	Default $\eta$	$E(\eta)$
LeRobot: default †	SmolVLA (0.45B)	$10^{-4}$	12%
LeRobot: as documented †	$\pi_{0.5}$ (4.1B)	$2.5{\times}10^{-5}$	83%
openpi: PyTorch †	$\pi_{0}$	$2.5{\times}10^{-5}$	78%
openpi: PyTorch †	$\pi_{0.5}$	$5{\times}10^{-5}$	65%
RLinf: RL (PPO)	$\pi_{0}$	$5{\times}10^{-6}$	94%
RLinf: RL	OpenVLA-OFT (7.5B)	$2{\times}10^{-5}$	71% ∗

TABLE I: Default VLA fine-tuning recipes that store trainable weights in bf16 with no fp32 copy.

E

: share of weights that an update the size of the learning rate cannot change; a lower bound on the share left unchanged per step (Section II ).

Fig. 2: The fp32 copy raises LIBERO-Object success in every comparison, most at the ten-times lower learning rate. Colors and line styles as in the legend (all-bf16: SmolVLA only). (a) Success over 500 tries per run: dots are seeds with Wilson 95% intervals (scoring noise only), bars the means of the seeds, and numbers each blue or green cluster’s mean minus the orange mean of its group, in points; SmolVLA executes 50-action chunks open loop except in the second group (retrained, re-planning every 10 actions). (b, c) Training loss, mean over seeds (exponential moving average).

VLA Model Adaptation

See all 41 papers

Jun 11, 2026

Output-Level Regularization Eliminates the Seed Lottery in Single-GPU VLA Fine-Tuning

Fine-tuning a vision-language-action model (VLA-JEPA) on a single GPU should be simple: load a pretrained checkpoint, run training, deploy. There is a hidden danger. Run the same fine-tuning code thirteen times -- same data, same architecture, different random seed -- and twelve runs produce a robot succeeding 91--94% of the time, while one run silently degrades to 65.2%: a 29 pp gap with no error message, no warning, and no way to predict which seed will fail. We call this the seed lottery. We trace the cause to output collapse: the action predictor quietly learns to produce nearly identical outputs regardless of what the robot sees. Existing weight-level methods (L2, EWC) are structurally blind to this collapse -- they penalize weight changes, but collapse occurs in directions weights can move freely without affecting outputs, a gap we formalize via the Jacobian null-space. Across 7 methods x up to 13 seeds x 3 LIBERO benchmarks, three output-level regularizers -- VICReg (n=12 seeds), Dropout (n=4), and a halved learning rate (n=5) -- each eliminate every catastrophic seed (0/21 combined collapses vs. 1/13 Baseline; F(12,11)=28.7, p<0.001), while weight-level methods (L2, EWC) preserve the lottery. The simplest fix is changing one number in your optimizer config.

Jeffrin Sam, Dzmitry Tsetserukou

Skolkovo Institute of Science and Technology (Skoltech), Moscow, Russia

VLM Adaptation Vision-Language-Action Models

Ask PDF

Oct 2, 2026

CHASE-VLA: Post-Training Quantization Framework for Vision-Language-Action Models with Chunk-Aware Scale Estimation

Vision-Language-Action (VLA) models map visual observations and language instructions to continuous robot actions, but a diffusion-based action expert (AE) poses a key challenge for low-bit post-training quantization (PTQ). The AE is repeatedly invoked across denoising steps and policy queries, where fixed calibration scales can be mismatched with activation ranges that vary with denoising progress and intended motion. We propose CHASE-VLA, a chunk-aware PTQ method that exploits a VLA-specific signal readily available from the policy: the generated action chunk, including its unexecuted future suffix. Rather than relying only on static scale matching for AE layers, CHASE-VLA combines the previously generated chunk as causal action context with denoising step group information to adapt AE activation scales. This enables W4A4 quantization of both MLP and attention projections in the repeated AE without modifying the pretrained policy. On LIBERO, CHASE-VLA achieves 97.3% average success rate on

π_{0.5}

when both MLP and attention projections in the AE are quantized to W4A4, restoring FP16-level performance. CHASE-VLA also reduces the weight storage of the quantized AE linear layers by 73.4% and their single-chunk memory traffic by 70.9% and 71.2% on

π_{0.5}

and GR00T N1.6, respectively, with a predictor overhead of at most 1.26% of the saved storage.

Jin Hyun, Jung Gyu Min, Gyuhyun Jung +1

KAIST, Daejeon, South Korea · POSTECH, Pohang, South Korea

Post-Training Quantization VLM Quantization Efficient VLA Model Inference Activation Quantization

Ask PDF

Aug 16, 2026

Bit-Flip Attacks on Vision-Language-Action Models: Action-Decoding Architecture Shapes the Vulnerability

Quantized Vision-Language-Action (VLA) models expose a weight-fault surface: Rowhammer-style faults can corrupt deployed INT8 bits. We present the first bit-flip attack on a VLA: a few gradient-selected flips reduce closed-loop success to

0\%

, while hundreds of random flips are harmless. Across four model variants spanning three action-head families, damaging bits concentrate in a few action-generating layers, but the empirical budget depends sharply on the head: direct regression and token policies fall in

1

5

flips, whereas the evaluated flow-matching policies require

{\sim}100

300

. Our fixed-direction manifold-escape loss cuts \pizero{}'s budget from

{\sim}1000

{\sim}100

flips, and a matched five-direction sweep shows that the attack is not specific to an all-positive direction. On a direct head, protecting

3.1\%

of weights preserves

60\%

success at

K{=}100

, and protecting

5.3\%

moves the open-loop break threshold from 3 to 100 flips. Finally, task-calibrated emulated

K{=}100

flips yield

0/20

real-robot successes, versus

14/20

clean and

16/20

global-random. Weight integrity is therefore a security boundary for embodied foundation models. Code is included as ancillary material.

Yudong Gao, Linghan Chen, Wenhan Wu +5

The Hong Kong University of Science and Technology · Adelaide University · Wuhan University +3

Adversarial Attacks Adversarial Attacks on VLA Models Vision-Language-Action Models Robustness of VLA Models

Ask PDF

Abstract

VLA Model Adaptation

Explore similar work

Output-Level Regularization Eliminates the Seed Lottery in Single-GPU VLA Fine-Tuning

CHASE-VLA: Post-Training Quantization Framework for Vision-Language-Action Models with Chunk-Aware Scale Estimation

Bit-Flip Attacks on Vision-Language-Action Models: Action-Decoding Architecture Shapes the Vulnerability