StructRL: Structured Action-Space Exploration for Flow-Based VLAs

Jiarui Yang; Bin Zhu; Jingjing Chen; Na Zou; Yanwei Fu; Jianggang Zhu; Yu-Gang Jiang

cs.ROAug 15, 2026

StructRL: Structured Action-Space Exploration for Flow-Based VLAs

Authors: Jiarui Yang, Bin Zhu, Jingjing Chen, Na Zou, Yanwei Fu, Jianggang Zhu, Yu-Gang Jiang

Organizations: Fudan University · Singapore Management University · Shanghai Artificial Intelligence Laboratory

Abstract

Flow-based Vision-Language-Action (VLA) models are now widely used for continuous robotic manipulation, and online reinforcement learning (RL) is emerging as a key technique for adapting them to new tasks. Existing RL methods typically inject stochasticity inside the denoising chain, often through isotropic or temporally independent noise. However, effective robot exploration calls for structured noise: temporally smooth and scaled differently across action groups. We show that simply switching the in-chain noise to a structured form does not suffice: noise added at an intermediate flow time can be weakened by the remaining denoising steps before execution, a phenomenon we call Structured Noise Dilution. We propose StructRL, which avoids dilution by relocating policy stochasticity to the action space via three coupled choices: (i) a deterministic ODE decoder, (ii) structured noise injected directly in the action space, and (iii) last-step replay, where policy-gradient updates avoid assigning likelihoods to intermediate denoising states. This keeps structured exploration tied to the executed action while providing a tractable training signal for the flow decoder. Across three flow-based VLA models on multiple simulated manipulation benchmarks and two real-world tasks, StructRL improves exploration efficiency and OOD performance over prior in-chain baselines, demonstrating the effectiveness of structured action-space exploration for adapting flow-based VLA with RL. Project page: https://flyfaerss.github.io/structrl/

Figures & tables

Figure 1: Structured exploration for flow-based VLA-RL. (a) Robot exploration benefits from smooth action chunks and group-aware noise scales. (b) In-chain noise can lose its intended structure during the remaining denoising steps. (c) StructRL keeps denoising deterministic and applies structured noise directly in the action space.

	Action-space			In-chain
	pos	rot	grip	pos	rot	grip
$\hat{\rho}_{\text{SFT}}$	$0.78$	$0.72$	$0.57$	$0.78$	$0.72$	$0.57$
$\rho_{\text{inj}}\!=\!0.95$	$0.70$	$0.64$	$0.63$	$0.79$	$0.72$	$0.58$
$\hat{\sigma}_{\text{SFT}}$	$0.10$	$0.08$	$0.21$	$0.10$	$0.08$	$0.21$
$R_{\text{approach}}$	$0.23$	$0.13$	$0.19$	$0.11$	$0.08$	$0.19$

Table 1: Structured noise dilution. Output statistics after injecting structured noise into a GR00T N1.5 SFT model on LIBERO-Long. We compare action-space injection with in-chain injection under two requested structures: strong temporal correlation (

\rho_{\text{inj}}\!=\!0.95

) and a position-heavy scale ratio

R_{\text{approach}}\!=\!(1.0,0.5,0.05)

, both using base noise level

\sigma\!=\!0.2

. Action-space injection better follows the requested structure, while in-chain injection stays close to the SFT baseline.

Figure 2: The StructRL framework. The reverse flow runs as a deterministic ODE to predict a clean actions

x_{0}^{\text{clean}}

. StructRL then samples structured noise

\Delta_{0}

directly in the action space, with AR(1) temporal correlation and action-group-aware scales. During RL training, the likelihood is evaluated on this action-space noise, and last-step replay passes policy-gradient updates back to the flow decoder.

Model	Spatial	Object	Goal	Long	Avg.	$\Delta$ Avg.
GR00T N1.5 – Few-shot SFT	41.4	58.6	48.2	61.9	52.5	—
$+\,\pi_{\text{RL}}$ (Flow-SDE $+$ PPO)	96.6	100	93.8	95.6	96.5	+44.0
$+\,$ Baseline	96.2	99.4	91.8	95.6	95.8	+43.3
$+\,$ StructRL	99.2	100	97.6	99.0	99.0	+46.3
$\pi_{0}$ – Few-shot SFT	65.3	64.4	49.8	51.2	57.6	—
$+\,\pi_{\text{RL}}$ (Flow-SDE $+$ PPO)	98.4	99.4	96.2	90.2	96.0	+38.4

Table 2: LIBERO benchmark results. Average success rates (%) across four task suites, each containing

10

tasks. The few-shot SFT models are directly taken from the

\pi_{\text{RL}}

checkpoints.

Model	IND	OOD
Model	IND	Vision	Semantic	Execution	Avg.
$\pi_{0}$ – SFT	38.4	32.6	8.4	13.2	18.1
$+\,\pi_{\text{RL}}$ (Flow-SDE $+$ PPO)	78.8	61.1	25.4	31.5	39.3
$+\,\pi$ -StepNFT	79.2	69.1	49.1	33.1	50.4
$+\,$ Baseline	84.1	79.4	63.3	59.1	67.3
$+\,$ StructRL	87.8	80.0	64.2	56.2	66.8

Table 3: ManiSkill benchmark results. Success rates (%) on in-distribution (IND) tasks and three OOD splits: Vision (5 tasks), Semantic (5 tasks), and Execution (2 tasks). Avg. reports the mean success rate across the OOD splits.

Panel of Figure 3 — Figure 3: Four ablations on LIBERO-Long with GR00T N1.5. (a) Moderate action-space noise gives stable online improvement. (b) Structured action-space noise explores more efficiently than Gaussian or in-chain variants. (c) Matched train–inference replay improves performance, with larger matched jumps further improving exploration efficiency. (d) Last-step replay mechanism provides an effective optimization signal for the full flow field across different denoising steps.

Panel of Figure 4 — Figure 4: Real-world tasks and online-RL comparisons. Both task-specific policies are initialized by SFT on 10 pre-collected demos and achieve 0% success rate (SR) before online RL.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

(a) Lag-1 autocorrelation $\hat{\rho}_{\text{out}}$
Injected structure	Action-space output			In-chain output
Injected structure	pos	rot	grip	pos	rot	grip
SFT baseline ( $\rho_{\text{SFT}}$ , no injection)	$0.784$	$0.718$	$0.566$	$0.784$	$0.718$	$0.566$
$\rho_{\text{inj}}\!=\!0.00$ (no banding)	$0.115$	$-0.033$	$0.225$	$0.785$	$0.718$	$0.564$
$\rho_{\text{inj}}\!=\!0.30$	$0.310$	$0.203$	$0.378$	$0.787$	$0.721$	$0.562$
$\rho_{\text{inj}}\!=\!0.70$	$0.556$	$0.493$	$0.560$	$0.788$	$0.725$	$0.577$

Appendix

Table A1: Full per-condition data at base noise level

\sigma\!=\!0.2

. GR00T N1.5 SFT model on LIBERO-Long. (a) Lag-1 autocorrelation

\hat{\rho}_{\text{out}}

under different injected

\rho_{\text{inj}}

. (b) Per-group standard deviation

\hat{\sigma}_{\text{out}}

under different magnitude profiles (target

\sigma_{\text{inj}}\!=\!0.2\cdot R

). Action-space injection tracks the requested structure, while in-chain injection stays close to the SFT baseline.

Model	Len1	Len2	Len3	Len4	Len5	Avg.
$\pi_{0.5}$ – SFT	0.927	0.843	0.767	0.688	0.613	3.838
$+\,\pi_{\text{RL}}$ (Flow-SDE $+$ PPO)	0.997	0.982	0.958	0.910	0.870	4.717
$+\,\pi_{\text{RL}}$ (Flow-Noise $+$ PPO)	0.996	0.976	0.939	0.896	0.845	4.652
$+\,$ Baseline	0.998	0.990	0.964	0.929	0.868	4.749
$+\,$ StructRL	1.000	0.993	0.968	0.934	0.880	4.775

Appendix

Table F1: Long-horizon chaining on CALVIN. Per-length success rate (fraction in

[0,1]

) and average sequence length on

\pi_{0.5}

. StructRL achieves the highest average and maintains the strongest performance at longer sequence lengths.

Parameters	LIBERO
	GR00T N1.5				$\pi_{0}$				$\pi_{0.5}$
	Spatial	Object	Goal	Long	Spatial	Object	Goal	Long	Spatial	Object	Goal	Long
Train epochs	200	200	200	200	200	200	400	400	400	400	400	400
Batch size	1024	1024	1024	1024	2048	2048	2048	2048	2048	2048	2048	2048
Update epochs	1	1	1	1	1	1	4	4	1	1	4	4
Actor lr	2e-5	2e-5	2e-5	2e-5	2e-5	2e-5	5e-6	5e-6	5e-6	5e-6	5e-6	5e-6

Appendix

Table G1: Training hyperparameters for LIBERO across the three VLA backbones and four task suites.

Parameters	CALVIN	ManiSkill
Parameters	$\pi_{0.5}$	$\pi_{0}$	$\pi_{0.5}$
Train epochs	300	400	400
Batch size	2048	5120	5120
Update epochs	4	4	5
Actor lr	5e-6	2e-5	2e-5
Value lr	1e-4	2e-5	2e-5

Appendix

Table G2: Training hyperparameters for CALVIN and ManiSkill multitask settings.

RL for Robotics

Reinforcement learning for robotics trains robots to act through interaction, from locomotion and navigation to dexterous manipulation.

See all 237 papers

StructRL: Online Structured Reinforcement Learning for Long-Horizon Vision-Language-Action Tasks
Sep 28, 2026Ziyi Yin, Sangmin Woo, Kang Zhou +4Reward Shaping Long-Horizon Robotic Manipulation
The Pennsylvania State University · Work done during an internship at Amazon · Amazon AWS AI
ExToken: Structured Exploration for Efficient Vision-Language-Action Reinforcement Fine-tuning
Jul 14, 2026Yilun Kong, Yunpeng Qing, Guozheng Ma +4RL Exploration Robotic Manipulation
Nanyang Technological University · ACE Robotics · Zhejiang University +1
FlowPRO: Reward-Free Reinforced Fine-Tuning of Flow-Matching VLAs via Proximalized Preference Optimization
Jun 3, 2026Yihao Wu, He Zhang, Junbo Tan +2Offline RL Vision-Language-Action Models
Tencent Robotics X · Futian Laboratory · Tsinghua University

Sep 28, 2026J/K move · Enter open · S save

StructRL: Online Structured Reinforcement Learning for Long-Horizon Vision-Language-Action Tasks

Ziyi Yin, Sangmin Woo, Kang Zhou, Sungyeon Kim+3

The Pennsylvania State University · Work done during an internship at Amazon · Amazon AWS AI

Reward Shaping Long-Horizon Robotic Manipulation Vision-Language-Action Models Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Vision-language-action (VLA) models perform well on shorter-horizon manipulation tasks but still struggle with long-horizon tasks that require multiple dependent manipulations from a single command. Online reinforcement learning (RL) can improve these policies through environment interaction, yet many existing methods provide reward only after the complete task succeeds. However, such terminal supervision is sparse and does not distinguish early failures from rollouts that make substantial partial progress. We propose StructRL, an online RL framework that constructs structured intermediate supervision from verifiable subtask completions. StructRL decomposes each task into verifiable subtasks, grants intermediate rewards only after the prerequisite subtasks have been completed, and scales each reward according to completion pace. Across RoboCasa365 and LIBERO-Long with GR00T-N1.5 and pi 0.5, StructRL consistently outperforms evaluated online RL baselines. These results show that verifiable, structured intermediate rewards improve long-horizon VLA post-training. Code is available at https://github.com/amazon-science/StructRL.

Abstract

RL for Robotics

Explore similar work

StructRL: Online Structured Reinforcement Learning for Long-Horizon Vision-Language-Action Tasks

ExToken: Structured Exploration for Efficient Vision-Language-Action Reinforcement Fine-tuning

FlowPRO: Reward-Free Reinforced Fine-Tuning of Flow-Matching VLAs via Proximalized Preference Optimization