cs.ROSep 23, 2026

Uncertainty-Gated Exploration Noise Suppresses Task Collapse in Online RL Fine-Tuning of a Flow-Matching Vision-Language-Action Policy

Authors: Mehmet Turan Yardımcı, Yunus Emre Çoğurcu

Organizations: Karlsruhe Institute of Technology · Çukurova University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Online reinforcement learning fine-tuning of pretrained flow-matching vision-language-action (VLA) policies promises robots that keep learning after deployment, but continued updates often destroy competence on individual tasks while the aggregate still looks healthy. We study this failure mode, which we call task collapse, under a matched small-compute budget on LIBERO-10 with a 450M-parameter SmolVLA policy trained by PPO with stochastic (SDE) sampling. Three exploration-noise policies differ in one live variable: a fixed noise scale, a ReinFlow-style learned noise network, and an uncertainty-gated controller that redistributes exploration across task streams from task-agnostic novelty and competence signals, without task labels or episode boundaries. Under the pooled definition, fixed noise collapses tasks in two of three seeds and learned noise in every seed measured to iteration 200, while the controller collapses none in any of its three seeds. Measured parameter displacement shows the controller's action expert keeps changing, while its mean applied noise is close to the fixed scale in the available logs. The matched comparison supports the controller's effect on task preservation; the separate contributions of its adaptation across states and over time are not disentangled. A lower fixed scale slows the decline but does not stop it. No arm improves on the behavior-cloning baseline in this budget. Two properties of that regime are measured beside this result, not offered as its cause: following the reference recipe, training runs in bfloat16 with no fp32 master copy, under which 96.02% of the action expert's elements stay bit-identical across three consecutive iterations, and an fp32 master copy at the reference learning rate collapses both arms in a single-seed observation. We release tools measuring per-task collapse under four definitions, rescoring noise and instrument tares.

Figures & tables

ArmSeedLastPool SRWilson 95%PoolPointSust.Ever
K controller13372000.585[0.555, 0.614]0/70/70/70/7
K controller23372000.564[0.534, 0.594]0/70/70/70/7
K controller33372000.589[0.559, 0.618]0/71/70/71/7
F fixed 0.2513372000.316[0.289, 0.345]4/77/76/77/7
F fixed 0.2523372000.472[0.442, 0.503]1/74/72/74/7
F fixed 0.2533372000.510[0.480, 0.541]0/70/71/72/7
Table 1: Plateau pool over the last five canonical scans, each arm against its own instrument tare (campaign tare 136/210 == 0.648, local tare 121/210 == 0.576). Four collapse columns share one threshold, a task below half of its own tare, and differ where a task crossed and later recovered: Pool is the primary definition, Point the single-scan one the survival analysis uses, Sust. two consecutive scans, Ever any scan. All four are stated formally in Appendix C . Last is each run’s last measured scan; a run censored early has had fewer opportunities to collapse. Every campaign row is scanned every 20 iterations and pools iterations 120 to 200. The one exception is the local-instrument row in the last block, whose iteration-160 scan is a 14-episode fragment the 210-episode rule excludes, so it pools 100 to 200; that row is marked not comparable to the others. The fp32 rows are a pilot and belong with the analysis. Every row is a single seed and carries no arm-level verdict.
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

armlastpooledlast scansustainedeverfirst crossing
K controller, seed 23372000/70/70/70/7none
K controller, seed 33372000/71/70/71/7160
F fixed 0.25, seed 13372004/77/76/77/720
F fixed 0.25, seed 23372001/74/72/74/7120
F fixed 0.18, seed 13372001/72/71/73/760
F fixed 0.25, seed 33372000/70/71/72/780
Appendix
Table 2: The four collapse definitions side by side, generated from the same producer that writes the owner file. All four share one threshold, half of each task’s own tare on the same instrument, and differ only in what they apply it to: pooled over the last five complete scans, last scan over one, sustained over two consecutive, ever cumulative. Last is the final complete scan and first crossing the earliest iteration at which any task crossed the threshold on any scan, which is the first event of the ever column and not of the sustained one. A row can therefore read 0/7 sustained and still carry a first crossing, as the seed-3337 controller row does at 160: that task crossed once and did not cross on the following scan. The owner file carries a separate first-sustained column, which for that row reads none; it is not printed here. None appears where no crossing occurs at all. Counts are tasks out of seven. Every row reached iteration 200 and was scanned every 20 iterations, so the three columns that count opportunities rather than time are read across rows at one cadence; runs that did not reach it enter no count in this paper and stay in the released measurement files.
armprecisionlast scanpooledWilson 95%first crossing
K, seed 1337bfloat16 in place200614/1050 = 0.5848[0.5547, 0.6142]none
K, seed 1337fp32 master copy200297/1050 = 0.2829[0.2564, 0.3109]iteration 60
F 0.25, seed 1337bfloat16 in place200332/1050 = 0.3162[0.2888, 0.3449]iteration 20
F 0.25, seed 1337fp32 master copy20090/1050 = 0.0857[0.0703, 0.1042]iteration 20
Appendix
Table 3: Pooled canonical success over the last five canonical scans of each series, 1050 episodes per row, against the campaign tare of 0.6476. The run-to-run rescoring SD on this instrument is 0.0375 and the pooled Wilson half-width is about 0.030, so the separation between the two precisions of the same arm is about eight times the repeat spread in the controller family and six times it in the fixed family. The first-crossing column gives the earliest iteration at which any task fell below half of its own tare value. Each row is a single seed. Supplementary files t1_plateau_pool.md and t6_fp32_pilot.md .
groupelementsunchangedunchanged %tensors that moved
frozen side (VLM body)350165184350165184100.000/345
action expert body982458409433824396.02112/145
flow head1635152540.0010/10
VLM LoRA81920049666760.63124/128
value head43494410.005/5
total45130032044500014998.60 
Appendix
Table 4: Element-level change across three outer iterations of k_corrected_s1337 , iter_000033 against iter_000036 . This is an earlier seed-1337 controller run, not one of the released campaign arms; the Reproducibility Statement describes it, and its run configuration is in the archive. 96.02% of the action expert’s elements are bit-identical while 112 of its 145 tensors move in at least one element. A tensor counts as moved when any one of its elements changes, so the two counts do not contradict each other; whether an unchanged element received an update below bfloat16 resolution, or none, was not measured. The frozen side is bit-identical in 100% of its elements, which is the null control that shows the comparison works. This counts elements over three iterations and Table 5 counts tensors over forty; neither bounds the other. Every count is over distinct tensors. The checkpoint’s model_state_dict holds 1261 entries because the flow model is stored under two names, policy.model.* and flow_model.* : all 628 pairs are equal in both checkpoints and no pair moves differently, so an entry count is twice the distinct count except for the value head, which is stored once. Counted once, the policy holds 451300320 parameters, the published size of 450M. The group percentages are the same under either count; the total’s is 98.60 counted once and 98.65 over entries. The counts are from supplementary file bf16_swallowed_updates.md , an authored record of readings taken by comparing the two checkpoints’ model_state_dict element by element. The arithmetic beside them is measured by the shipped script test_fp32_master_swallow.py : a weight of magnitude 0.02 sits 1.221×10−41.221\times 10^{-4} from its nearest bfloat16 neighbour, and twenty consecutive updates of 5×10−65\times 10^{-6} move it by 0 in place while accumulating to 9.999×10−59.999\times 10^{-5} on an fp32 master copy.
armseedvariantwindowexpertLoRAcriticbit-ident. %
F1337fixed 0.25100–1400.00090.00480.017513.3
F1337fixed 0.25160–2000.00090.00460.018613.3
F1337fp32 master100–1400.00340.00570.01771.4
F1337fp32 master160–2000.00340.00560.02091.4
F1337fixed 0.15100–1400.00090.00440.018113.3
F1337fixed 0.15160–2000.00090.00460.015613.3
Appendix
Table 5: Relative movement of the three trainable groups over 40-iteration windows, and the fraction of their tensors that are bit-identical across the window. Movement is ∥Wt1−Wt0∥F/∥Wt0∥F\lVert W_{t_{1}}-W_{t_{0}}\rVert_{F}/\lVert W_{t_{0}}\rVert_{F} , with Wt0W_{t_{0}} and Wt1W_{t_{1}} a group’s weights at the first and the last outer iteration of the window; it is the quantity of Figure 3 a, which fixes t0=0t_{0}=0 . The last column counts tensors, not elements. The fixed arm is measured at its campaign value σ=0.25\sigma=0.25 on all three seeds, at the 0.150.15 ablation on both of its seeds and at the 0.180.18 ablation on its one; the rows marked fp32 master are the precision pilot of Figure 4 , one seed per arm. The controller rows marked controller dir. are the local seed-1337 run, the one scored on the local instrument in Table 1 ; its values differ from the campaign seed-1337 run in every window, so the two are separate runs and are not pooled. The supplementary file records 39 measured cells of 63; six of them are rows of a run held under more than one directory name and are equal to the last digit in every group, so they are merged and 33 distinct windows are printed. Absent windows are unmeasured, not zero. Rows are emitted by movement_table_tex.py , which parses parameter_movement_by_arm.md rather than retyping it.
benchmark familytasksinputbaselinesseedserror bar
OpenAI Gym4stateDPPO, FQL3mean ±\pm 1 SD
Franka Kitchen3stateDPPO, FQL3, 5 on twomean ±\pm 1 SD
Robomimic3pixelDPPO, FQL3mean ±\pm 1 SD
ours, LIBERO-107pixel and stateBC tare, three arms3 per armWilson and run-to-run SD
ours, their gym8 of 9state, pixelthree arms3 per armseed SD, eval SD
Appendix
Table 6: Evaluation profile of Zhang et al. (2025c) beside ours. Cells are read from the paper’s own text and tables; a string that names an environment is not counted as an experiment on it. Our first row is the campaign described in Section 4 ; our second is the re-run of Appendix F.1 , where 9 tasks were attempted and eight completed at three seeds in all three arms.
filelinewhat it sets
examples/embodiment/config/model/pi0.yaml6precision: null
examples/embodiment/config/model/pi0_5.yaml6precision: null, with the comment that it must not be changed
examples/embodiment/config/libero_10_ppo_openpi.yaml118precision: ${actor.model.precision}
examples/embodiment/config/libero_10_ppo_openpi.yaml152–154param_dtype, reduce_dtype and buffer_dtype all bound to the same variable
rlinf/config.py379–384use_fsdp_mixed_precision = not (all_none or all_fp32)
examples/embodiment/config/training_backend/fsdp.yaml22–31mixed_precision null, amp_autocast.enabled False, grad_scaler.enabled False
Appendix
Table 7: The precision lines of the pinned RLinf clone (commit 832db3f5 ), of openpi and of LeRobot SmolVLA, and our own regime read from the weights of k_corrected_s1337 , the run of Table 4 . Autocast and the gradient scaler are off in the reference recipe, so training is bfloat16 in place rather than mixed precision. Supplementary file reference_precision.md .
filewhat it holdshow it is made
arm_table.mdevery arm, its pooled value, both rulers, the four collapse columnscode/make_arm_table.py
t1_plateau_pool.mdthe plateau pool of every seriescode/make_figures_legacy.py
t5_collapse_definitions.mdthe collapse definitions read side by sidecode/collapse_three_ways.py
t6_fp32_pilot.mdthe fp32 master-copy pilotcode/make_t6_reading.py
parameter_movement_by_arm.mdrelative movement of each trainable group per arm and windowcode/measure_param_movement.py
bf16_swallowed_updates.mdthe element-level bfloat16 measurement and the fp32 arithmetic beside itauthored record, append-only journal of readings taken from checkpoint tensors
Appendix
Table 8: The measurement files in the supplementary archive, what each holds, and how each was made.
 DPPOWSRLthis paper
benchmark families544
policy classes1, diffusion (MLP, UNet, ViT-MLP); Gaussian and GMM as baselines1, soft actor-critic policy (MLP)2
real robotyes, Franka, 20 trialsyes, Franka peg insertion, 20 trialsno
named baselines116 in simulation, SERL on the robotnone; 3 arms against the policy’s own BC tare
seeds5 (Gym, D3IL), 3 (Robomimic, Kitchen, Furniture-Bench)not stated3 per arm
variability reportedmean over seeds; two figures state SD not shownno measure statedWilson per scoring and repeat-scoring SD
Appendix
Table 9: Evaluation profiles of two recent RL fine-tuning papers, DPPO ( Ren et al., 2024 ) and WSRL ( Zhou et al., 2025 ) , and this work. Counts are read from the full text of arXiv version 3 of each paper, from its experiment sections and appendices; a policy-class cell counts the classes the method itself fine-tunes, and the classes fine-tuned only as baselines are named after it. The first row counts benchmark families, the unit of Table 6 : a benchmark family is a suite such as OpenAI Gym or Robomimic, not one task inside it, and DPPO’s and WSRL’s counts are suites. Ours are LIBERO-10 and the benchmark families of Table 6 that hold a task of Appendix F.1 completed in all three arms, 4 in all. Our two policy classes are the pretrained vision-language-action policy of the main campaign and the small flow policies of that re-run.

Explore similar work

CardsList
Oct 11, 2025cs.LG

Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models

Vision-Language-Action (VLA) models such as OpenVLA, Octo, and π0π_0 have shown strong generalization by leveraging large-scale demonstrations, yet their performance is still fundamentally constrained by the quality and coverage of supervised data. Reinforcement learning (RL) provides a promising path for improving and fine-tuning VLAs through online interaction. However, conventional policy gradient methods are computationally infeasible in the context of flow-matching based models due to the intractability of the importance sampling process, which requires explicit computation of policy ratios. To overcome this limitation, we propose Flow Policy Optimization (FPO) algorithm, which reformulates importance sampling by leveraging per-sample changes in the conditional flow-matching objective. Furthermore, FPO achieves stable and scalable online reinforcement fine-tuning of the π0π_0 model by integrating structure-aware credit assignment to enhance gradient efficiency, clipped surrogate objectives to stabilize optimization, multi-step latent exploration to encourage diverse policy updates, and a Q-ensemble mechanism to provide robust value estimation. We evaluate FPO on the LIBERO benchmark and the ALOHA simulation task against supervised, preference-aligned, diffusion-based, autoregressive online RL, and π0π_0-FAST baselines, observing consistent improvements over the imitation prior and strong alternatives with stable learning under sparse rewards. In addition, ablation studies and analyses of the latent space dynamics further highlight the contributions of individual components within FPO, validating the effectiveness of the proposed computational modules and the stable convergence of the conditional flow-matching objective during online RL.

Mingyang Lyu, Yinqian Sun, Erliang Lin +4

Brain-inspired Cognitive AI Lab, Institute of Automation, Chinese Academy of Sciences, Beijing, China. · Beijing Institute of AI Safety and Governance, China. · University of Chinese Academy of Sciences (UCAS), Beijing, China. +3

Flow-Matching Vision-Language-ActionReinforcement Fine-TuningConditional Flow MatchingModel Fine-TuningLatent Dynamics
AskPDF
May 25, 2026cs.RO

EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models

The ability to efficiently and reliably learn new tasks has been a foundational challenge in robotics. Vision-Language-Action (VLA) models have demonstrated strong generalization across diverse manipulation tasks, yet pretrained policies consistently fall short of the reliability required for real-world deployment. Reinforcement learning (RL) fine-tuning offers a promising path to bridge this gap, but existing approaches either train from scratch without fully leveraging pretrained priors, or fine-tune VLAs without achieving the sample efficiency and success rates that practical deployment demands. We present EXPO-FT, a system for stable, sample-efficient RL finetuning of pretrained VLA policies that closes this gap. Our system solves a suite of challenging manipulation tasks, including routing string lights and inserting the plug to light it up, striking a pool ball into a pocket, and inserting a flower into a wine bottle, each requiring combinations of high precision, dynamic actions, and robustness to varied initial states. Our system achieves perfect task performance (30/30 successes) across all evaluated tasks within an average of 19.1 minutes of online robot data, outperforming both prior RL-from-scratch and VLA finetuning approaches. We release an open-source codebase with the aim of facilitating broader adoption of RL finetuning of VLA models in robotics.

Perry Dong, Kuo-Han Hung, Tian Gao +2

Stanford University

Reinforcement Fine-Tuning
AskPDF
Jun 3, 2026cs.RO

FlowPRO: Reward-Free Reinforced Fine-Tuning of Flow-Matching VLAs via Proximalized Preference Optimization

Post-training Vision-Language-Action (VLA) models into policies that can be reliably deployed on real robots remains a major bottleneck. SFT and DAgger exploit failure signals only indirectly, and reward-based RL is bottlenecked by the difficulty of real-world reward design and of training reliable critics. We present FlowPRO, a reward-free offline reinforced fine-tuning framework for flow-matching VLAs. Algorithmically, we propose RPRO (Robotic Flow-matching Proximalized Preference Optimization), a preference-optimization objective tailored to the flow-matching action head of VLA models. RPRO pairs a contrastive optimizer with an explicit proximal regularizer that anchors the absolute magnitude of the implicit reward, thereby eliminating the reward-hacking failure mode of plain Flow-DPO. On the data side, a teleoperated intervention-and-rollback paradigm produces naturally paired positive and negative trajectories (τw,τl)(τ^w, τ^l) on a real robot from a single operator action; a Smooth Interpolation procedure, combined with batch mixing, then converts these sparse corrections into dense per-state supervision while preserving the base policy's capabilities. On four long-horizon bimanual tasks, FlowPRO attains the highest success rate, outperforming four representative baselines, and ablations confirm the contribution of each loss component.

Yihao Wu, He Zhang, Junbo Tan +2

Tencent Robotics X · Futian Laboratory · Tsinghua University

Flow-Matching Vision-Language-ActionReinforcement Fine-Tuning
AskPDF