Uncertainty-Gated Exploration Noise Suppresses Task Collapse in Online RL Fine-Tuning of a Flow-Matching Vision-Language-Action Policy

Mehmet Turan Yardımcı; Yunus Emre Çoğurcu

cs.ROSep 23, 2026

Uncertainty-Gated Exploration Noise Suppresses Task Collapse in Online RL Fine-Tuning of a Flow-Matching Vision-Language-Action Policy

Authors: Mehmet Turan Yardımcı, Yunus Emre Çoğurcu

Organizations: Karlsruhe Institute of Technology · Çukurova University

Abstract

Online reinforcement learning fine-tuning of pretrained flow-matching vision-language-action (VLA) policies promises robots that keep learning after deployment, but continued updates often destroy competence on individual tasks while the aggregate still looks healthy. We study this failure mode, which we call task collapse, under a matched small-compute budget on LIBERO-10 with a 450M-parameter SmolVLA policy trained by PPO with stochastic (SDE) sampling. Three exploration-noise policies differ in one live variable: a fixed noise scale, a ReinFlow-style learned noise network, and an uncertainty-gated controller that redistributes exploration across task streams from task-agnostic novelty and competence signals, without task labels or episode boundaries. Under the pooled definition, fixed noise collapses tasks in two of three seeds and learned noise in every seed measured to iteration 200, while the controller collapses none in any of its three seeds. Measured parameter displacement shows the controller's action expert keeps changing, while its mean applied noise is close to the fixed scale in the available logs. The matched comparison supports the controller's effect on task preservation; the separate contributions of its adaptation across states and over time are not disentangled. A lower fixed scale slows the decline but does not stop it. No arm improves on the behavior-cloning baseline in this budget. Two properties of that regime are measured beside this result, not offered as its cause: following the reference recipe, training runs in bfloat16 with no fp32 master copy, under which 96.02% of the action expert's elements stay bit-identical across three consecutive iterations, and an fp32 master copy at the reference learning rate collapses both arms in a single-seed observation. We release tools measuring per-task collapse under four definitions, rescoring noise and instrument tares.

Figures & tables

Arm	Seed	Last	Pool SR	Wilson 95%	Pool	Point	Sust.	Ever
K controller	1337	200	0.585	[0.555, 0.614]	0/7	0/7	0/7	0/7
K controller	2337	200	0.564	[0.534, 0.594]	0/7	0/7	0/7	0/7
K controller	3337	200	0.589	[0.559, 0.618]	0/7	1/7	0/7	1/7
F fixed 0.25	1337	200	0.316	[0.289, 0.345]	4/7	7/7	6/7	7/7
F fixed 0.25	2337	200	0.472	[0.442, 0.503]	1/7	4/7	2/7	4/7
F fixed 0.25	3337	200	0.510	[0.480, 0.541]	0/7	0/7	1/7	2/7

Table 1: Plateau pool over the last five canonical scans, each arm against its own instrument tare (campaign tare 136/210

=

0.648, local tare 121/210

=

0.576). Four collapse columns share one threshold, a task below half of its own tare, and differ where a task crossed and later recovered: Pool is the primary definition, Point the single-scan one the survival analysis uses, Sust. two consecutive scans, Ever any scan. All four are stated formally in Appendix C . Last is each run’s last measured scan; a run censored early has had fewer opportunities to collapse. Every campaign row is scanned every 20 iterations and pools iterations 120 to 200. The one exception is the local-instrument row in the last block, whose iteration-160 scan is a 14-episode fragment the 210-episode rule excludes, so it pools 100 to 200; that row is marked not comparable to the others. The fp32 rows are a pilot and belong with the analysis. Every row is a single seed and carries no arm-level verdict.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

arm	last	pooled	last scan	sustained	ever	first crossing
K controller, seed 2337	200	0/7	0/7	0/7	0/7	none
K controller, seed 3337	200	0/7	1/7	0/7	1/7	160
F fixed 0.25, seed 1337	200	4/7	7/7	6/7	7/7	20
F fixed 0.25, seed 2337	200	1/7	4/7	2/7	4/7	120
F fixed 0.18, seed 1337	200	1/7	2/7	1/7	3/7	60
F fixed 0.25, seed 3337	200	0/7	0/7	1/7	2/7	80

Appendix

Table 2: The four collapse definitions side by side, generated from the same producer that writes the owner file. All four share one threshold, half of each task’s own tare on the same instrument, and differ only in what they apply it to: pooled over the last five complete scans, last scan over one, sustained over two consecutive, ever cumulative. Last is the final complete scan and first crossing the earliest iteration at which any task crossed the threshold on any scan, which is the first event of the ever column and not of the sustained one. A row can therefore read 0/7 sustained and still carry a first crossing, as the seed-3337 controller row does at 160: that task crossed once and did not cross on the following scan. The owner file carries a separate first-sustained column, which for that row reads none; it is not printed here. None appears where no crossing occurs at all. Counts are tasks out of seven. Every row reached iteration 200 and was scanned every 20 iterations, so the three columns that count opportunities rather than time are read across rows at one cadence; runs that did not reach it enter no count in this paper and stay in the released measurement files.

arm	precision	last scan	pooled	Wilson 95%	first crossing
K, seed 1337	bfloat16 in place	200	614/1050 = 0.5848	[0.5547, 0.6142]	none
K, seed 1337	fp32 master copy	200	297/1050 = 0.2829	[0.2564, 0.3109]	iteration 60
F 0.25, seed 1337	bfloat16 in place	200	332/1050 = 0.3162	[0.2888, 0.3449]	iteration 20
F 0.25, seed 1337	fp32 master copy	200	90/1050 = 0.0857	[0.0703, 0.1042]	iteration 20

Appendix

Table 3: Pooled canonical success over the last five canonical scans of each series, 1050 episodes per row, against the campaign tare of 0.6476. The run-to-run rescoring SD on this instrument is 0.0375 and the pooled Wilson half-width is about 0.030, so the separation between the two precisions of the same arm is about eight times the repeat spread in the controller family and six times it in the fixed family. The first-crossing column gives the earliest iteration at which any task fell below half of its own tare value. Each row is a single seed. Supplementary files t1_plateau_pool.md and t6_fp32_pilot.md .

group	elements	unchanged	unchanged %	tensors that moved
frozen side (VLM body)	350165184	350165184	100.00	0/345
action expert body	98245840	94338243	96.02	112/145
flow head	1635152	54	0.00	10/10
VLM LoRA	819200	496667	60.63	124/128
value head	434944	1	0.00	5/5
total	451300320	445000149	98.60

Appendix

Table 4: Element-level change across three outer iterations of k_corrected_s1337 , iter_000033 against iter_000036 . This is an earlier seed-1337 controller run, not one of the released campaign arms; the Reproducibility Statement describes it, and its run configuration is in the archive. 96.02% of the action expert’s elements are bit-identical while 112 of its 145 tensors move in at least one element. A tensor counts as moved when any one of its elements changes, so the two counts do not contradict each other; whether an unchanged element received an update below bfloat16 resolution, or none, was not measured. The frozen side is bit-identical in 100% of its elements, which is the null control that shows the comparison works. This counts elements over three iterations and Table 5 counts tensors over forty; neither bounds the other. Every count is over distinct tensors. The checkpoint’s model_state_dict holds 1261 entries because the flow model is stored under two names, policy.model.* and flow_model.* : all 628 pairs are equal in both checkpoints and no pair moves differently, so an entry count is twice the distinct count except for the value head, which is stored once. Counted once, the policy holds 451300320 parameters, the published size of 450M. The group percentages are the same under either count; the total’s is 98.60 counted once and 98.65 over entries. The counts are from supplementary file bf16_swallowed_updates.md , an authored record of readings taken by comparing the two checkpoints’ model_state_dict element by element. The arithmetic beside them is measured by the shipped script test_fp32_master_swallow.py : a weight of magnitude 0.02 sits

1.221\times 10^{-4}

from its nearest bfloat16 neighbour, and twenty consecutive updates of

5\times 10^{-6}

move it by 0 in place while accumulating to

9.999\times 10^{-5}

on an fp32 master copy.

arm	seed	variant	window	expert	LoRA	critic	bit-ident. %
F	1337	fixed 0.25	100–140	0.0009	0.0048	0.0175	13.3
F	1337	fixed 0.25	160–200	0.0009	0.0046	0.0186	13.3
F	1337	fp32 master	100–140	0.0034	0.0057	0.0177	1.4
F	1337	fp32 master	160–200	0.0034	0.0056	0.0209	1.4
F	1337	fixed 0.15	100–140	0.0009	0.0044	0.0181	13.3
F	1337	fixed 0.15	160–200	0.0009	0.0046	0.0156	13.3

Appendix

Table 5: Relative movement of the three trainable groups over 40-iteration windows, and the fraction of their tensors that are bit-identical across the window. Movement is

\lVert W_{t_{1}}-W_{t_{0}}\rVert_{F}/\lVert W_{t_{0}}\rVert_{F}

, with

W_{t_{0}}

and

W_{t_{1}}

a group’s weights at the first and the last outer iteration of the window; it is the quantity of Figure 3 a, which fixes

t_{0}=0

. The last column counts tensors, not elements. The fixed arm is measured at its campaign value

\sigma=0.25

on all three seeds, at the

0.15

ablation on both of its seeds and at the

0.18

ablation on its one; the rows marked fp32 master are the precision pilot of Figure 4 , one seed per arm. The controller rows marked controller dir. are the local seed-1337 run, the one scored on the local instrument in Table 1 ; its values differ from the campaign seed-1337 run in every window, so the two are separate runs and are not pooled. The supplementary file records 39 measured cells of 63; six of them are rows of a run held under more than one directory name and are equal to the last digit in every group, so they are merged and 33 distinct windows are printed. Absent windows are unmeasured, not zero. Rows are emitted by movement_table_tex.py , which parses parameter_movement_by_arm.md rather than retyping it.

benchmark family	tasks	input	baselines	seeds	error bar
OpenAI Gym	4	state	DPPO, FQL	3	mean $\pm$ 1 SD
Franka Kitchen	3	state	DPPO, FQL	3, 5 on two	mean $\pm$ 1 SD
Robomimic	3	pixel	DPPO, FQL	3	mean $\pm$ 1 SD
ours, LIBERO-10	7	pixel and state	BC tare, three arms	3 per arm	Wilson and run-to-run SD
ours, their gym	8 of 9	state, pixel	three arms	3 per arm	seed SD, eval SD

Appendix

Table 6: Evaluation profile of Zhang et al. (2025c) beside ours. Cells are read from the paper’s own text and tables; a string that names an environment is not counted as an experiment on it. Our first row is the campaign described in Section 4 ; our second is the re-run of Appendix F.1 , where 9 tasks were attempted and eight completed at three seeds in all three arms.

file	line	what it sets
examples/embodiment/config/model/pi0.yaml	6	precision: null
examples/embodiment/config/model/pi0_5.yaml	6	precision: null, with the comment that it must not be changed
examples/embodiment/config/libero_10_ppo_openpi.yaml	118	precision: ${actor.model.precision}
examples/embodiment/config/libero_10_ppo_openpi.yaml	152–154	param_dtype, reduce_dtype and buffer_dtype all bound to the same variable
rlinf/config.py	379–384	use_fsdp_mixed_precision = not (all_none or all_fp32)
examples/embodiment/config/training_backend/fsdp.yaml	22–31	mixed_precision null, amp_autocast.enabled False, grad_scaler.enabled False

Appendix

Table 7: The precision lines of the pinned RLinf clone (commit 832db3f5 ), of openpi and of LeRobot SmolVLA, and our own regime read from the weights of k_corrected_s1337 , the run of Table 4 . Autocast and the gradient scaler are off in the reference recipe, so training is bfloat16 in place rather than mixed precision. Supplementary file reference_precision.md .

file	what it holds	how it is made
arm_table.md	every arm, its pooled value, both rulers, the four collapse columns	code/make_arm_table.py
t1_plateau_pool.md	the plateau pool of every series	code/make_figures_legacy.py
t5_collapse_definitions.md	the collapse definitions read side by side	code/collapse_three_ways.py
t6_fp32_pilot.md	the fp32 master-copy pilot	code/make_t6_reading.py
parameter_movement_by_arm.md	relative movement of each trainable group per arm and window	code/measure_param_movement.py
bf16_swallowed_updates.md	the element-level bfloat16 measurement and the fp32 arithmetic beside it	authored record, append-only journal of readings taken from checkpoint tensors

Appendix

Table 8: The measurement files in the supplementary archive, what each holds, and how each was made.

	DPPO	WSRL	this paper
benchmark families	5	4	4
policy classes	1, diffusion (MLP, UNet, ViT-MLP); Gaussian and GMM as baselines	1, soft actor-critic policy (MLP)	2
real robot	yes, Franka, 20 trials	yes, Franka peg insertion, 20 trials	no
named baselines	11	6 in simulation, SERL on the robot	none; 3 arms against the policy’s own BC tare
seeds	5 (Gym, D3IL), 3 (Robomimic, Kitchen, Furniture-Bench)	not stated	3 per arm
variability reported	mean over seeds; two figures state SD not shown	no measure stated	Wilson per scoring and repeat-scoring SD

Appendix

Table 9: Evaluation profiles of two recent RL fine-tuning papers, DPPO ( Ren et al., 2024 ) and WSRL ( Zhou et al., 2025 ) , and this work. Counts are read from the full text of arXiv version 3 of each paper, from its experiment sections and appendices; a policy-class cell counts the classes the method itself fine-tunes, and the classes fine-tuned only as baselines are named after it. The first row counts benchmark families, the unit of Table 6 : a benchmark family is a suite such as OpenAI Gym or Robomimic, not one task inside it, and DPPO’s and WSRL’s counts are suites. Ours are LIBERO-10 and the benchmark families of Table 6 that hold a task of Appendix F.1 completed in all three arms, 4 in all. Our two policy classes are the pretrained vision-language-action policy of the main campaign and the small flow policies of that re-run.

Explore similar work

Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models
Oct 11, 2025Mingyang Lyu, Yinqian Sun, Erliang Lin +4Flow-Matching Vision-Language-Action Reinforcement Fine-Tuning
Brain-inspired Cognitive AI Lab, Institute of Automation, Chinese Academy of Sciences, Beijing, China. · Beijing Institute of AI Safety and Governance, China. · University of Chinese Academy of Sciences (UCAS), Beijing, China. +3
EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models
May 25, 2026Perry Dong, Kuo-Han Hung, Tian Gao +2Reinforcement Fine-Tuning
Stanford University
FlowPRO: Reward-Free Reinforced Fine-Tuning of Flow-Matching VLAs via Proximalized Preference Optimization
Jun 3, 2026Yihao Wu, He Zhang, Junbo Tan +2Flow-Matching Vision-Language-Action Reinforcement Fine-Tuning
Tencent Robotics X · Futian Laboratory · Tsinghua University

Oct 11, 2025, cs.LGJ/K move · Enter open · S save

Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models

Mingyang Lyu, Yinqian Sun, Erliang Lin, Huangrui Li+3

Brain-inspired Cognitive AI Lab, Institute of Automation, Chinese Academy of Sciences, Beijing, China. · Beijing Institute of AI Safety and Governance, China. · University of Chinese Academy of Sciences (UCAS), Beijing, China.+3

Flow-Matching Vision-Language-Action Reinforcement Fine-Tuning Conditional Flow Matching Model Fine-Tuning Latent Dynamics

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models such as OpenVLA, Octo, and

π_0

have shown strong generalization by leveraging large-scale demonstrations, yet their performance is still fundamentally constrained by the quality and coverage of supervised data. Reinforcement learning (RL) provides a promising path for improving and fine-tuning VLAs through online interaction. However, conventional policy gradient methods are computationally infeasible in the context of flow-matching based models due to the intractability of the importance sampling process, which requires explicit computation of policy ratios. To overcome this limitation, we propose Flow Policy Optimization (FPO) algorithm, which reformulates importance sampling by leveraging per-sample changes in the conditional flow-matching objective. Furthermore, FPO achieves stable and scalable online reinforcement fine-tuning of the

π_0

model by integrating structure-aware credit assignment to enhance gradient efficiency, clipped surrogate objectives to stabilize optimization, multi-step latent exploration to encourage diverse policy updates, and a Q-ensemble mechanism to provide robust value estimation. We evaluate FPO on the LIBERO benchmark and the ALOHA simulation task against supervised, preference-aligned, diffusion-based, autoregressive online RL, and

π_0

-FAST baselines, observing consistent improvements over the imitation prior and strong alternatives with stable learning under sparse rewards. In addition, ablation studies and analyses of the latent space dynamics further highlight the contributions of individual components within FPO, validating the effectiveness of the proposed computational modules and the stable convergence of the conditional flow-matching objective during online RL.

Abstract

Explore similar work

Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models

EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models

FlowPRO: Reward-Free Reinforced Fine-Tuning of Flow-Matching VLAs via Proximalized Preference Optimization