Sample-Efficient Reinforcement Learning for Policy Improvement via Flow-Reversed Action Anchors

Carl Qi; Weijun Dong; Xiaojie Wang; Peiyi Chen; Rahul Sajnani; Mingyuan Zhang; Amy Zhang; Jose Barreiros

cs.ROOct 16, 2026

Sample-Efficient Reinforcement Learning for Policy Improvement via Flow-Reversed Action Anchors

Authors: Carl Qi, Weijun Dong, Xiaojie Wang, Peiyi Chen, Rahul Sajnani, Mingyuan Zhang, Amy Zhang, Jose Barreiros,

Organizations: Amazon · UT Austin · UC San Diego · Brown University ·

Abstract

Generalist robot policies offer strong behavioral priors, but achieving deployment-level reliability and throughput remains difficult. Reinforcement learning (RL) can improve these policies, yet its effectiveness is constrained by the cost of online interaction. Existing methods adapt frozen flow-matching policies by treating noise space as a continuous action space. We show that this behaviorally unstructured space makes exploration inefficient, often preventing these methods from extracting meaningful learning signals on challenging tasks. We introduce AnchorQ, which enables coarse-to-fine policy improvement through a structured, refinable latent action representation. At each decision step, AnchorQ uses flow reversal to map reference behaviors to state-conditioned latent anchors, samples local particles around those anchors, and applies value-guided sampling to explore promising refinements. Our analysis shows that anchors reduce identification complexity at the cost of support bias; particle refinement trades this bias against action-class complexity; and critic accuracy determines the reliability of KL-regularized, value-guided selection. Across simulated and real-world manipulation tasks, AnchorQ outperforms state-of-the-art post-training baselines in sample efficiency, delivering a 37% improvement in real-world success rate and 97% higher simulated task throughput under the same interaction budget. Additional visualizations are available on our website https://anchorq-rl.com/.

Figures & tables

Figure 1: Overview of AnchorQ . (a) A small set of reference action anchors defines coarse behaviors. (b) Flow reversal maps these references into state-conditioned noise anchors. (c) AnchorQ samples local particles around every anchor and reweights them using the learned critic to form refined barycenters. (d) It selects the highest-value barycenter, decodes it through the frozen flow policy, and executes the resulting action. The critic is trained online using Bellman updates.

Method	LIBERO-15T		LIBERO Hard-10		RoboTwin-6T
$\pi_{0.5}$ zero-shot	$8.5\pm 0.4$		$1.6\pm 0.4$		$47.7\pm 1.6$
VLM+FRS zero-shot	$21.9\pm 0.6$		$2.0\pm 0.3$		$54.2\pm 0.2$
Online RL env. steps	$250$ k	$500$ k	$250$ k	$500$ k	$500$ k
DSRL	$37.2\pm 1.6$	$48.3\pm 3.8$	${7.8\pm 4.6}^{\dagger}$	${25.2\pm 3.4}^{\dagger}$	$47.2\pm 4.5$
Residual RL	$22.8\pm 0.3$	$26.7\pm 0.5$	$9.6\pm 2.1$	$14.6\pm 0.8$	$0.0\pm 0.0$
AnchorQ (Ours)	$\mathbf{86.0\pm 0.8}$	$\mathbf{95.5\pm 0.6}$	$\mathbf{52.1\pm 2.7}$	$\mathbf{62.1\pm 4.0}$	$\mathbf{70.5\pm 3.0}$

Table 1: Performance on manipulation tasks in simulation. Success rate (%) on the two held-out LIBERO-90 splits from Tang et al. (2026) and on RoboTwin ( Chen et al., 2025 ) . We report mean

\pm

standard error over four seeds. Results marked with † use DSRL+FRS ( Tang et al., 2026 ) . AnchorQ achieves the highest success rate across all three suites.

Figure 2: Success rate on manipulation tasks in simulation ; mean

\pm 1

standard error over four seeds. (Left) 15-task held-out LIBERO-90 split. (Center) 10-task hard split. (Right) 6-task bimanual suite in RoboTwin. AnchorQ consistently outperforms the baselines across all three task suites.

Figure 4: Anchor & particle-tilting analysis on LIBERO task 81. (a) Angle between realized end-effector displacement and anchor direction, grouped by anchor: AnchorQ shows a heavier right tail on every axis, indicating finer steering beyond the discrete anchor. (b) Executed noise projected onto the top two pooled principal components (top) and noise magnitude

\|z\|

(bottom): reverse-flow anchors span the shaded band, AnchorQ explores across it, DSRL concentrates near zero, and the

\pi_{0.5}

prior lies above.

Figure 5: Real world steering results. We evaluate on three manipulation tasks using a Franka robot with a DROID setup. Each RL method receives 100 online training episodes per task and is evaluated on 20 held out configurations. Error bars on the aggregate success rate show 95% Wilson score confidence intervals. AnchorQ achieves higher success rates than the frozen $\pi_{0.5}$ policy and DSRL on all three tasks.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Short Key	LIBERO-90	RoboTwin	Real world	Meaning
Environment and action space
$n_{\mathrm{exec}}$	pi0_replan_steps	$10$	$30$	$10$	substeps per outer decision
$K$	pi0_flow_steps	$10$	$10$	$10$	Euler steps in the flow ODE
$C$	num_anchors	$7$	$37$	$7$	reference candidates including defer
$(H,D)$	noise_shape	$(10,32)$	$(50,32)$	$(15,32)$	noise trajectory shape
Critic, replay, and exploration

Appendix

Table 3: Hyperparameters by benchmark. Settings for LIBERO-90 ( Liu et al., 2023a ) , RoboTwin ( Chen et al., 2025 ) , and the real-world experiments.

Figure 6: Overview of the experimental tasks. Representative tasks from the LIBERO simulation benchmark, the bimanual RoboTwin benchmark, and our real-world Franka setup.

Figure 7: Per-task performance on the LIBERO-90 15-task suite. Success rate over training for the particle-tilting ablations and steering baselines. Curves show the mean and shaded regions show

\pm 1

standard error across seeds.

Figure 8: Per-task performance on the LIBERO-90 hard-10 suite. Success rate over training for the adaptive-tilting, anchor-only, DSRL, and Residual RL variants. Curves show the mean and shaded regions show

\pm 1

standard error across seeds.

Figure 9: Per-task performance on the RoboTwin 6-task suite. Success rate over training for the particle-tilting ablations and steering baselines. Curves show the mean and shaded regions show

\pm 1

standard error across seeds.

Figure 10: Per-task ablations on four LIBERO-90 tasks. AnchorQ with adaptive tilting learns consistently across all four tasks. Fixed tilting, anchor-only learning, and the steering baselines show substantially greater variation across tasks. Curves show the mean and shaded regions show

\pm 1

standard error across seeds.

Figure 11: Anchor-only learning versus adaptive particle tilting on LIBERO-90 task $81$ . The task is to place the book into the front compartment of the caddy. Top: the no-tilting, anchor-only ablation fails to place the book. Bottom: AnchorQ completes the insertion. Colored overlays identify the nearest anchor and its reference direction; yellow denotes a base-policy action.

Method	Task 41	Task 81	Task 75	Task 73
$\pi_{0.5}$ zero-shot	$2.5\pm 2.5$	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$
VLM-FRS zero-shot	$3.0\pm 1.0$	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$
DSRL	$35.0\pm 23.6$	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$
Residual RL	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$
AnchorQ -AdaBeta-AdaSig	$90.0\pm 2.6$	$\mathbf{83.0\pm 7.5}$	$\mathbf{89.0\pm 1.9}$	$\mathbf{91.0\pm 3.0}$
AdaBeta, $\sigma{=}0.3$	$\mathbf{95.0\pm 1.9}$	$81.0\pm 3.4$	$85.0\pm 4.1$	$90.0\pm 2.6$

Appendix

Table 4: Per-task success rate after

500

k environment steps. Success rate (%), reported as mean

\pm

standard error across seeds. The best result on each task is bold.

Mechanism	Effect
online flow-reversal	small structured online action class
local particles	controlled relaxation of flow-reversed anchor support
exponential tilting	value-directed exploitation of that support
adaptive $\beta$	epistemically calibrated exploitation

Appendix

Table 15

Robot Policy Learning

Robot policy learning develops controllers that map sensory observations, task goals, and interaction history to actions for embodied robots.

See all 377 papers

Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning
May 3, 2026Sungyoung Lee, Dohyeong Kim, Eshan Balachandar +2Distributional RL Q-Learning
The University of Texas at Austin, Austin, TX, USA · Independent Researcher, Seoul, South Korea.
QPILOTS: Efficient Test-Time Q-Steering for Flow Policies
Jun 11, 2026Yifan Ruan, Chenyang Cao, Andreas Burger +7Test-Time RL Offline-To-Online RL
University of Toronto · Vector Institute · LG Electronics
Refinement-Based Flow Policy Optimization
Sep 14, 2026Bumgeun Park, Hyukjun Yang, Donghwan LeeRL Control Flow Matching
Korea Advanced Institute of Science and Technology (KAIST)

May 3, 2026J/K move · Enter open · S save

Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning

Sungyoung Lee, Dohyeong Kim, Eshan Balachandar, Zelal Su Mustafaoglu+1

The University of Texas at Austin, Austin, TX, USA · Independent Researcher, Seoul, South Korea.

Distributional RL Q-Learning Robotic RL Offline RL Policy Learning

Open paper AskPDF arXiv

We propose Flow-Anchored Noise-conditioned Q-Learning (FAN), a highly efficient and high-performing offline reinforcement learning (RL) algorithm. Recent work has shown that expressive flow policies and distributional critics improve offline RL performance, but at a high computational cost. Specifically, flow policies require iterative sampling to produce a single action, and distributional critics require computation over multiple samples (e.g., quantiles) to estimate value. To address these inefficiencies while maintaining high performance, we introduce FAN. Our method employs a behavior regularization technique that uses a single flow policy iteration and requires a single Gaussian noise sample for distributional critics. Our theoretical analysis of convergence and performance bounds demonstrates that these simplifications not only improve efficiency but also lead to superior task performance. Experiments on robotic manipulation and locomotion tasks demonstrate that FAN achieves state-of-the-art performance while significantly reducing both training and inference runtimes. We release our code at https://github.com/brianlsy98/FAN.

Abstract

Robot Policy Learning

Explore similar work

Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning

QPILOTS: Efficient Test-Time Q-Steering for Flow Policies

Refinement-Based Flow Policy Optimization