StructRL: Structured Action-Space Exploration for Flow-Based VLAs

Jiarui Yang; Bin Zhu; Jingjing Chen; Na Zou; Yanwei Fu; Jianggang Zhu; Yu-Gang Jiang

cs.ROAug 15, 2026

StructRL: Structured Action-Space Exploration for Flow-Based VLAs

Authors: Jiarui Yang, Bin Zhu, Jingjing Chen, Na Zou, Yanwei Fu, Jianggang Zhu, Yu-Gang Jiang

Organizations: Fudan University · Singapore Management University · Shanghai Artificial Intelligence Laboratory

Abstract

Flow-based Vision-Language-Action (VLA) models are now widely used for continuous robotic manipulation, and online reinforcement learning (RL) is emerging as a key technique for adapting them to new tasks. Existing RL methods typically inject stochasticity inside the denoising chain, often through isotropic or temporally independent noise. However, effective robot exploration calls for structured noise: temporally smooth and scaled differently across action groups. We show that simply switching the in-chain noise to a structured form does not suffice: noise added at an intermediate flow time can be weakened by the remaining denoising steps before execution, a phenomenon we call Structured Noise Dilution. We propose StructRL, which avoids dilution by relocating policy stochasticity to the action space via three coupled choices: (i) a deterministic ODE decoder, (ii) structured noise injected directly in the action space, and (iii) last-step replay, where policy-gradient updates avoid assigning likelihoods to intermediate denoising states. This keeps structured exploration tied to the executed action while providing a tractable training signal for the flow decoder. Across three flow-based VLA models on multiple simulated manipulation benchmarks and two real-world tasks, StructRL improves exploration efficiency and OOD performance over prior in-chain baselines, demonstrating the effectiveness of structured action-space exploration for adapting flow-based VLA with RL. Project page: https://flyfaerss.github.io/structrl/

Figures & tables

Figure 1: Structured exploration for flow-based VLA-RL. (a) Robot exploration benefits from smooth action chunks and group-aware noise scales. (b) In-chain noise can lose its intended structure during the remaining denoising steps. (c) StructRL keeps denoising deterministic and applies structured noise directly in the action space.

	Action-space			In-chain
	pos	rot	grip	pos	rot	grip
$\hat{\rho}_{\text{SFT}}$	$0.78$	$0.72$	$0.57$	$0.78$	$0.72$	$0.57$
$\rho_{\text{inj}}\!=\!0.95$	$0.70$	$0.64$	$0.63$	$0.79$	$0.72$	$0.58$
$\hat{\sigma}_{\text{SFT}}$	$0.10$	$0.08$	$0.21$	$0.10$	$0.08$	$0.21$
$R_{\text{approach}}$	$0.23$	$0.13$	$0.19$	$0.11$	$0.08$	$0.19$

Table 1: Structured noise dilution. Output statistics after injecting structured noise into a GR00T N1.5 SFT model on LIBERO-Long. We compare action-space injection with in-chain injection under two requested structures: strong temporal correlation (

\rho_{\text{inj}}\!=\!0.95

) and a position-heavy scale ratio

R_{\text{approach}}\!=\!(1.0,0.5,0.05)

, both using base noise level

\sigma\!=\!0.2

. Action-space injection better follows the requested structure, while in-chain injection stays close to the SFT baseline.

Figure 2: The StructRL framework. The reverse flow runs as a deterministic ODE to predict a clean actions

x_{0}^{\text{clean}}

. StructRL then samples structured noise

\Delta_{0}

directly in the action space, with AR(1) temporal correlation and action-group-aware scales. During RL training, the likelihood is evaluated on this action-space noise, and last-step replay passes policy-gradient updates back to the flow decoder.

Model	Spatial	Object	Goal	Long	Avg.	$\Delta$ Avg.
GR00T N1.5 – Few-shot SFT	41.4	58.6	48.2	61.9	52.5	—
$+\,\pi_{\text{RL}}$ (Flow-SDE $+$ PPO)	96.6	100	93.8	95.6	96.5	+44.0
$+\,$ Baseline	96.2	99.4	91.8	95.6	95.8	+43.3
$+\,$ StructRL	99.2	100	97.6	99.0	99.0	+46.3
$\pi_{0}$ – Few-shot SFT	65.3	64.4	49.8	51.2	57.6	—
$+\,\pi_{\text{RL}}$ (Flow-SDE $+$ PPO)	98.4	99.4	96.2	90.2	96.0	+38.4

Table 2: LIBERO benchmark results. Average success rates (%) across four task suites, each containing

10

tasks. The few-shot SFT models are directly taken from the

\pi_{\text{RL}}

checkpoints.

Model	IND	OOD
Model	IND	Vision	Semantic	Execution	Avg.
$\pi_{0}$ – SFT	38.4	32.6	8.4	13.2	18.1
$+\,\pi_{\text{RL}}$ (Flow-SDE $+$ PPO)	78.8	61.1	25.4	31.5	39.3
$+\,\pi$ -StepNFT	79.2	69.1	49.1	33.1	50.4
$+\,$ Baseline	84.1	79.4	63.3	59.1	67.3
$+\,$ StructRL	87.8	80.0	64.2	56.2	66.8

Table 3: ManiSkill benchmark results. Success rates (%) on in-distribution (IND) tasks and three OOD splits: Vision (5 tasks), Semantic (5 tasks), and Execution (2 tasks). Avg. reports the mean success rate across the OOD splits.

Panel of Figure 3 — Figure 3: Four ablations on LIBERO-Long with GR00T N1.5. (a) Moderate action-space noise gives stable online improvement. (b) Structured action-space noise explores more efficiently than Gaussian or in-chain variants. (c) Matched train–inference replay improves performance, with larger matched jumps further improving exploration efficiency. (d) Last-step replay mechanism provides an effective optimization signal for the full flow field across different denoising steps.

Panel of Figure 4 — Figure 4: Real-world tasks and online-RL comparisons. Both task-specific policies are initialized by SFT on 10 pre-collected demos and achieve 0% success rate (SR) before online RL.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

(a) Lag-1 autocorrelation $\hat{\rho}_{\text{out}}$
Injected structure	Action-space output			In-chain output
Injected structure	pos	rot	grip	pos	rot	grip
SFT baseline ( $\rho_{\text{SFT}}$ , no injection)	$0.784$	$0.718$	$0.566$	$0.784$	$0.718$	$0.566$
$\rho_{\text{inj}}\!=\!0.00$ (no banding)	$0.115$	$-0.033$	$0.225$	$0.785$	$0.718$	$0.564$
$\rho_{\text{inj}}\!=\!0.30$	$0.310$	$0.203$	$0.378$	$0.787$	$0.721$	$0.562$
$\rho_{\text{inj}}\!=\!0.70$	$0.556$	$0.493$	$0.560$	$0.788$	$0.725$	$0.577$

Appendix

Table A1: Full per-condition data at base noise level

\sigma\!=\!0.2

. GR00T N1.5 SFT model on LIBERO-Long. (a) Lag-1 autocorrelation

\hat{\rho}_{\text{out}}

under different injected

\rho_{\text{inj}}

. (b) Per-group standard deviation

\hat{\sigma}_{\text{out}}

under different magnitude profiles (target

\sigma_{\text{inj}}\!=\!0.2\cdot R

). Action-space injection tracks the requested structure, while in-chain injection stays close to the SFT baseline.

Model	Len1	Len2	Len3	Len4	Len5	Avg.
$\pi_{0.5}$ – SFT	0.927	0.843	0.767	0.688	0.613	3.838
$+\,\pi_{\text{RL}}$ (Flow-SDE $+$ PPO)	0.997	0.982	0.958	0.910	0.870	4.717
$+\,\pi_{\text{RL}}$ (Flow-Noise $+$ PPO)	0.996	0.976	0.939	0.896	0.845	4.652
$+\,$ Baseline	0.998	0.990	0.964	0.929	0.868	4.749
$+\,$ StructRL	1.000	0.993	0.968	0.934	0.880	4.775

Appendix

Table F1: Long-horizon chaining on CALVIN. Per-length success rate (fraction in

[0,1]

) and average sequence length on

\pi_{0.5}

. StructRL achieves the highest average and maintains the strongest performance at longer sequence lengths.

Parameters	LIBERO
	GR00T N1.5				$\pi_{0}$				$\pi_{0.5}$
	Spatial	Object	Goal	Long	Spatial	Object	Goal	Long	Spatial	Object	Goal	Long
Train epochs	200	200	200	200	200	200	400	400	400	400	400	400
Batch size	1024	1024	1024	1024	2048	2048	2048	2048	2048	2048	2048	2048
Update epochs	1	1	1	1	1	1	4	4	1	1	4	4
Actor lr	2e-5	2e-5	2e-5	2e-5	2e-5	2e-5	5e-6	5e-6	5e-6	5e-6	5e-6	5e-6

Appendix

Table G1: Training hyperparameters for LIBERO across the three VLA backbones and four task suites.

Parameters	CALVIN	ManiSkill
Parameters	$\pi_{0.5}$	$\pi_{0}$	$\pi_{0.5}$
Train epochs	300	400	400
Batch size	2048	5120	5120
Update epochs	4	4	5
Actor lr	5e-6	2e-5	2e-5
Value lr	1e-4	2e-5	2e-5

Appendix

Table G2: Training hyperparameters for CALVIN and ManiSkill multitask settings.

RL for Robotics

Reinforcement learning for robotics trains robots to act through interaction, from locomotion and navigation to dexterous manipulation.

See all 237 papers

Sep 28, 2026

StructRL: Online Structured Reinforcement Learning for Long-Horizon Vision-Language-Action Tasks

Vision-language-action (VLA) models perform well on shorter-horizon manipulation tasks but still struggle with long-horizon tasks that require multiple dependent manipulations from a single command. Online reinforcement learning (RL) can improve these policies through environment interaction, yet many existing methods provide reward only after the complete task succeeds. However, such terminal supervision is sparse and does not distinguish early failures from rollouts that make substantial partial progress. We propose StructRL, an online RL framework that constructs structured intermediate supervision from verifiable subtask completions. StructRL decomposes each task into verifiable subtasks, grants intermediate rewards only after the prerequisite subtasks have been completed, and scales each reward according to completion pace. Across RoboCasa365 and LIBERO-Long with GR00T-N1.5 and pi 0.5, StructRL consistently outperforms evaluated online RL baselines. These results show that verifiable, structured intermediate rewards improve long-horizon VLA post-training. Code is available at https://github.com/amazon-science/StructRL.

Ziyi Yin, Sangmin Woo, Kang Zhou +4

The Pennsylvania State University · Work done during an internship at Amazon · Amazon AWS AI

Reward Shaping Long-Horizon Robotic Manipulation Vision-Language-Action Models Reinforcement Learning with Verifiable Rewards

Ask PDF

Jul 14, 2026

ExToken: Structured Exploration for Efficient Vision-Language-Action Reinforcement Fine-tuning

Reinforcement Learning (RL) has demonstrated significant potential for improving Vision-Language-Action (VLA) models on complex manipulation tasks. However, its practical scalability remains severely limited by the substantial cost of environmental interactions. In this work, we first investigate the exploration stagnation bottleneck in current VLA-RL frameworks and reveal that trajectory diversity is fundamentally more important to sample efficiency than the sheer quantity of collected rollouts. Motivated by these insights, we introduce RL Exploration Token (ExToken), a simple yet general framework that condition VLA policies on discrete behavioral priors derived from offline demonstrations for structured exploration. By conditioning the policy on different tokens during rollout collection, ExToken encourages the agent to explore diverse behavioral modes, substantially improving state-action coverage and exploration efficiency. To bridge exploration during training with deterministic inference at deployment, ExToken further incorporates a state-conditioned token selector that adaptively predicts effective behavioral modes for unseen scenarios. Extensive experiments across simulated and real-world robotic manipulation tasks demonstrate that ExToken consistently accelerates convergence, improves task performance, and exhibits strong robustness under highly constrained interaction budgets.

Yilun Kong, Yunpeng Qing, Guozheng Ma +4

Nanyang Technological University · ACE Robotics · Zhejiang University +1

RL Exploration Robotic Manipulation Vision-Language-Action Models Sample-Efficient RL RL for VLMs

Ask PDF

Jun 3, 2026

FlowPRO: Reward-Free Reinforced Fine-Tuning of Flow-Matching VLAs via Proximalized Preference Optimization

Post-training Vision-Language-Action (VLA) models into policies that can be reliably deployed on real robots remains a major bottleneck. SFT and DAgger exploit failure signals only indirectly, and reward-based RL is bottlenecked by the difficulty of real-world reward design and of training reliable critics. We present FlowPRO, a reward-free offline reinforced fine-tuning framework for flow-matching VLAs. Algorithmically, we propose RPRO (Robotic Flow-matching Proximalized Preference Optimization), a preference-optimization objective tailored to the flow-matching action head of VLA models. RPRO pairs a contrastive optimizer with an explicit proximal regularizer that anchors the absolute magnitude of the implicit reward, thereby eliminating the reward-hacking failure mode of plain Flow-DPO. On the data side, a teleoperated intervention-and-rollback paradigm produces naturally paired positive and negative trajectories

(τ^w, τ^l)

on a real robot from a single operator action; a Smooth Interpolation procedure, combined with batch mixing, then converts these sparse corrections into dense per-state supervision while preserving the base policy's capabilities. On four long-horizon bimanual tasks, FlowPRO attains the highest success rate, outperforming four representative baselines, and ablations confirm the contribution of each loss component.

Yihao Wu, He Zhang, Junbo Tan +2

Tencent Robotics X · Futian Laboratory · Tsinghua University

Offline RL Vision-Language-Action Models VLMs for Robotics RL Fine-Tuning Preference-Based RL

Ask PDF

Abstract

RL for Robotics

Explore similar work

StructRL: Online Structured Reinforcement Learning for Long-Horizon Vision-Language-Action Tasks

ExToken: Structured Exploration for Efficient Vision-Language-Action Reinforcement Fine-tuning

FlowPRO: Reward-Free Reinforced Fine-Tuning of Flow-Matching VLAs via Proximalized Preference Optimization