Generalist robot policies offer strong behavioral priors, but achieving deployment-level reliability and throughput remains difficult. Reinforcement learning (RL) can improve these policies, yet its effectiveness is constrained by the cost of online interaction. Existing methods adapt frozen flow-matching policies by treating noise space as a continuous action space. We show that this behaviorally unstructured space makes exploration inefficient, often preventing these methods from extracting meaningful learning signals on challenging tasks. We introduce AnchorQ, which enables coarse-to-fine policy improvement through a structured, refinable latent action representation. At each decision step, AnchorQ uses flow reversal to map reference behaviors to state-conditioned latent anchors, samples local particles around those anchors, and applies value-guided sampling to explore promising refinements. Our analysis shows that anchors reduce identification complexity at the cost of support bias; particle refinement trades this bias against action-class complexity; and critic accuracy determines the reliability of KL-regularized, value-guided selection. Across simulated and real-world manipulation tasks, AnchorQ outperforms state-of-the-art post-training baselines in sample efficiency, delivering a 37% improvement in real-world success rate and 97% higher simulated task throughput under the same interaction budget. Additional visualizations are available on our website https://anchorq-rl.com/.
Figures & tables
Figure 1: Overview of AnchorQ . (a) A small set of reference action anchors defines coarse behaviors. (b) Flow reversal maps these references into state-conditioned noise anchors. (c) AnchorQ samples local particles around every anchor and reweights them using the learned critic to form refined barycenters. (d) It selects the highest-value barycenter, decodes it through the frozen flow policy, and executes the resulting action. The critic is trained online using Bellman updates.
Method
LIBERO-15T
LIBERO Hard-10
RoboTwin-6T
π0.5 zero-shot
8.5±0.4
1.6±0.4
47.7±1.6
VLM+FRS zero-shot
21.9±0.6
2.0±0.3
54.2±0.2
Online RL env. steps
250 k
500 k
250 k
500 k
500 k
DSRL
37.2±1.6
48.3±3.8
7.8±4.6†
25.2±3.4†
47.2±4.5
Residual RL
22.8±0.3
26.7±0.5
9.6±2.1
14.6±0.8
0.0±0.0
AnchorQ (Ours)
86.0±0.8
95.5±0.6
52.1±2.7
62.1±4.0
70.5±3.0
Table 1: Performance on manipulation tasks in simulation. Success rate (%) on the two held-out LIBERO-90 splits from Tang et al. (2026) and on RoboTwin ( Chen et al., 2025 ) . We report mean ± standard error over four seeds. Results marked with † use DSRL+FRS ( Tang et al., 2026 ) . AnchorQ achieves the highest success rate across all three suites.
Figure 2: Success rate on manipulation tasks in simulation ; mean ±1 standard error over four seeds. (Left) 15-task held-out LIBERO-90 split. (Center) 10-task hard split. (Right) 6-task bimanual suite in RoboTwin. AnchorQ consistently outperforms the baselines across all three task suites.
Figure 4
Figure 4: Anchor & particle-tilting analysis on LIBERO task 81. (a) Angle between realized end-effector displacement and anchor direction, grouped by anchor: AnchorQ shows a heavier right tail on every axis, indicating finer steering beyond the discrete anchor. (b) Executed noise projected onto the top two pooled principal components (top) and noise magnitude ∥z∥ (bottom): reverse-flow anchors span the shaded band, AnchorQ explores across it, DSRL concentrates near zero, and the π0.5 prior lies above.
Figure 5: Real world steering results. We evaluate on three manipulation tasks using a Franka robot with a DROID setup. Each RL method receives 100 online training episodes per task and is evaluated on 20 held out configurations. Error bars on the aggregate success rate show 95% Wilson score confidence intervals. AnchorQ achieves higher success rates than the frozen π0.5 policy and DSRL on all three tasks.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Short Key
LIBERO-90
RoboTwin
Real world
Meaning
Environment and action space
nexec
pi0_replan_steps
10
30
10
substeps per outer decision
K
pi0_flow_steps
10
10
10
Euler steps in the flow ODE
C
num_anchors
7
37
7
reference candidates including defer
(H,D)
noise_shape
(10,32)
(50,32)
(15,32)
noise trajectory shape
Critic, replay, and exploration
Appendix
Table 3: Hyperparameters by benchmark. Settings for LIBERO-90 ( Liu et al., 2023a ) , RoboTwin ( Chen et al., 2025 ) , and the real-world experiments.
Figure 6: Overview of the experimental tasks. Representative tasks from the LIBERO simulation benchmark, the bimanual RoboTwin benchmark, and our real-world Franka setup.
Figure 7: Per-task performance on the LIBERO-90 15-task suite. Success rate over training for the particle-tilting ablations and steering baselines. Curves show the mean and shaded regions show ±1 standard error across seeds.
Figure 8: Per-task performance on the LIBERO-90 hard-10 suite. Success rate over training for the adaptive-tilting, anchor-only, DSRL, and Residual RL variants. Curves show the mean and shaded regions show ±1 standard error across seeds.
Figure 9: Per-task performance on the RoboTwin 6-task suite. Success rate over training for the particle-tilting ablations and steering baselines. Curves show the mean and shaded regions show ±1 standard error across seeds.
Figure 10: Per-task ablations on four LIBERO-90 tasks. AnchorQ with adaptive tilting learns consistently across all four tasks. Fixed tilting, anchor-only learning, and the steering baselines show substantially greater variation across tasks. Curves show the mean and shaded regions show ±1 standard error across seeds.
Figure 11: Anchor-only learning versus adaptive particle tilting on LIBERO-90 task 81 . The task is to place the book into the front compartment of the caddy. Top: the no-tilting, anchor-only ablation fails to place the book. Bottom: AnchorQ completes the insertion. Colored overlays identify the nearest anchor and its reference direction; yellow denotes a base-policy action.
Method
Task 41
Task 81
Task 75
Task 73
π0.5 zero-shot
2.5±2.5
0.0±0.0
0.0±0.0
0.0±0.0
VLM-FRS zero-shot
3.0±1.0
0.0±0.0
0.0±0.0
0.0±0.0
DSRL
35.0±23.6
0.0±0.0
0.0±0.0
0.0±0.0
Residual RL
0.0±0.0
0.0±0.0
0.0±0.0
0.0±0.0
AnchorQ -AdaBeta-AdaSig
90.0±2.6
83.0±7.5
89.0±1.9
91.0±3.0
AdaBeta, σ=0.3
95.0±1.9
81.0±3.4
85.0±4.1
90.0±2.6
Appendix
Table 4: Per-task success rate after 500 k environment steps. Success rate (%), reported as mean ± standard error across seeds. The best result on each task is bold.
Mechanism
Effect
online flow-reversal
small structured online action class
local particles
controlled relaxation of flow-reversed anchor support
We propose Flow-Anchored Noise-conditioned Q-Learning (FAN), a highly efficient and high-performing offline reinforcement learning (RL) algorithm. Recent work has shown that expressive flow policies and distributional critics improve offline RL performance, but at a high computational cost. Specifically, flow policies require iterative sampling to produce a single action, and distributional critics require computation over multiple samples (e.g., quantiles) to estimate value. To address these inefficiencies while maintaining high performance, we introduce FAN. Our method employs a behavior regularization technique that uses a single flow policy iteration and requires a single Gaussian noise sample for distributional critics. Our theoretical analysis of convergence and performance bounds demonstrates that these simplifications not only improve efficiency but also lead to superior task performance. Experiments on robotic manipulation and locomotion tasks demonstrate that FAN achieves state-of-the-art performance while significantly reducing both training and inference runtimes. We release our code at https://github.com/brianlsy98/FAN.
Sungyoung Lee, Dohyeong Kim, Eshan Balachandar +2
The University of Texas at Austin, Austin, TX, USA · Independent Researcher, Seoul, South Korea.
Flow-matching and diffusion policies are expressive action generators, but optimizing them with temporal-difference reinforcement learning (RL) remains difficult. Effective policy extraction requires exploiting the critic's action gradient, yet directly backpropagating this signal through a multi-step denoising process can be numerically unstable. Existing methods work around this either by discarding gradient information, distilling the policy into a simpler one-step actor, or repeatedly fine-tuning the denoising policy as the critic improves. We propose QPILOTS, a method that leaves the original policy unmodified and steers the denoising process at inference time. At each denoising step, instead of evaluating the critic on the noisy intermediate action where critic predictions are unreliable, we first project that intermediate state to an estimate of the final clean action and compute the critic gradient there. We introduce two variants: QPILOTS-U uses a fast single-point approximation, while QPILOTS-M draws differentiable posterior samples via a learned auxiliary network. On a standard offline-to-online RL benchmark, QPILOTS achieves the best aggregate performance, reaching an average success rate of 90% across 50 tasks. We also apply QPILOTS to steer a large, frozen, pretrained Vision-Language Action (VLA) foundation model, outperforming or matching prior inference-time approaches across six manipulation tasks in simulation.
Yifan Ruan, Chenyang Cao, Andreas Burger +7
University of Toronto · Vector Institute · LG Electronics
Flow-based policies offer an expressive representation for online reinforcement learning, but conventional flow matching requires samples drawn from the distribution to be modeled. This poses a challenge when the desired action distribution is defined only implicitly by a Q-function, since directly sampling actions from the resulting distribution is generally intractable. We propose Refinement-Based Flow Policy Optimization (RFPO), a novel framework for training a flow policy in online reinforcement learning by alternating between Q-guided sample refinement and self-target flow matching. RFPO first generates actions from Gaussian noise using the current flow policy and then uses a finite-step stochastic refinement procedure to move them toward an energy-based distribution induced by the Q-function. Each refined action is then paired with its corresponding initial noise sample and used as a fixed target for flow-matching training. By repeatedly refining its own outputs and learning from the resulting targets, RFPO incorporates Q-guidance into the policy without requiring direct samples from the target distribution, while retaining the capacity to represent multiple action modes. We further provide a theoretical analysis of the distributional dynamics induced by RFPO. Across six continuous-control tasks, RFPO matches or outperforms a standard Gaussian-policy baseline on almost every task. Experiments on six synthetic two-dimensional target distributions with diverse geometries demonstrate that RFPO captures complex multimodal structure without mode collapse.
Bumgeun Park, Hyukjun Yang, Donghwan Lee
Korea Advanced Institute of Science and Technology (KAIST)