Recent robot learning paradigms increasingly rely on large offline datasets of robotic interactions to train control policies. Expressive generative models enable rich and multimodal action representations, expanding the capability of this paradigm for complex robotic control. However, policy improvement with multi-step generative actors remains challenging. In offline reinforcement learning (RL), incorporating value-based objectives along generative trajectories often introduces substantial training complexity, including backpropagation through time (BPTT), auxiliary architectures, or distillation losses. We propose Value-Guided Flow Matching (VGFM), a scalable offline RL framework that enables dense value-guided shaping within a flow-based policy while avoiding BPTT and additional algorithmic overhead. VGFM parameterizes the policy as a conditional flow-matching model in action (x-prediction) space, ensuring that each intermediate flow step produces a valid robot action that can be directly evaluated by a standard offline RL critic. This design allows value guidance to be applied at randomly sampled flow times without differentiating through the entire generative trajectory, while preserving inference-time flexibility by varying the discretization of the underlying flow ODE without retraining. Evaluated on robotic locomotion and manipulation tasks in OGBench, VGFM achieves strong performance across a wide range of tasks under rigorous evaluation protocols. With minimal hyperparameter tuning, these results demonstrate that VGFM provides a simple, scalable, and effective approach for expressive policy learning in long-horizon, goal-oriented robotic control.
Figures & tables
Fig. 1 : Overall illustration of the proposed VGFM framework and benchmark tasks it is evaluated on. (a) Examples of robotic manipulation and locomotion environments used for evaluation, spanning maze navigation, goal-directed interaction, and multi-stage object manipulation. (b) Illustration of how VGFM applies value guidance (Q-loss) to each intermediate step of a flow-based policy. A 2D unimodal Gaussian is transported to a multimodal 2D action distribution over multiple steps, with Q-loss applied at each intermediate flow step.
Method
BPTT
Inference Scalability
Auxiliary Loss
DQL [ 7 ]
Yes
Yes
None
FQL [ 10 ]
No
No
Distillation Loss
SSCP [ 12 ]
No
Poor
Completion Loss
SORL [ 11 ]
Yes
Yes
Shortcut Loss
VGFM (Ours)
No
Yes
None
TABLE I : Comparison of generative policy methods along three practical dimensions.
Fig. 2 : Training curves of VGFM for all 8 OGBench task groups. Plots show normalized return versus gradient steps. Solid lines denote mean performance over 8 random training seeds; shaded regions indicate ± one standard deviation.
Gaussian Policies
Generative Policies
Task Category
BC
IQL
ReBRAC
IDQL
CAC
IFQL
FQL
SORL
VGFM
antmaze-large-singletask ( 5 tasks)
11±1
53±3
81 ±5
21±5
33±4
28±5
79 ±3
89±2
87±1
antmaze-giant-singletask ( 5 tasks)
0±0
4±1
26±8
0±0
0±0
3±2
9±6
9±6
23±9
humanoidmaze-medium-singletask ( 5 tasks)
2±1
33±2
22±8
1±0
53±8
60±14
58±5
64±4
73±8
humanoidmaze-large-singletask ( 5 tasks)
1±0
2±1
2±1
1±0
0±0
11±2
4±2
5±2
9±3
antsoccer-arena-singletask ( 5 tasks)
1±0
8±2
0±0
12±4
2±4
33±6
60±2
69±2
39±2
TABLE II : OGBench Offline RL results. For each task group, we highlight the highest-performing score with boldface and additionally bold any method whose performance is within 95% of the best. VGFM achieves the best or near-best performance on most of the OGBench robotic locomotion and manipulation tasks.
Fig. 3 : Task success vs. Euler steps T during policy inference. Multi-step flow integration improves control precision and long-horizon task performance in OGBench robotics environments, positioning VGFM as an expressive inference-time scalable method.
Fig. 4 : Inference cost vs. discretization level T . Despite improved performance at larger inference steps T , VGFM incurs near-constant wall-clock cost, comparable to single-step FQL (all evaluations done on an NVIDIA RTX A6000 GPU).
Flow-matching policies have emerged as an effective policy parameterization for robot learning. They iteratively generate actions from noise, enabling highly expressive modeling of complex and multimodal action distributions. However, prior works observed that scaling these policies with value-gradient reinforcement learning (RL) often leads to training instability. Existing methods attribute this instability to iterative generation and therefore avoid end-to-end value-gradient optimization by sacrificing iterative generation, high expressiveness, or value-gradient optimization. Contrary to prior belief, we show the instability does not stem from iterative generation itself, but from the vanilla sampling strategy originally designed for behavior cloning, which becomes brittle under value-gradient RL. Motivated by this insight, we propose VINE, an RL-oriented sampling method that enables stable end-to-end value-gradient optimization for flow-matching policies. Instead of following a single flow trajectory, VINE reconstructs a new interpolation state at every denoising step, creating a stable differentiable path for value-gradient propagation while remaining compatible with the original flow-matching denoising process. As a result, VINE preserves the expressiveness and iterative generation of flow-matching without sacrificing end-to-end value-gradient optimization. Despite performing end-to-end backpropagation through all ten denoising steps, VINE achieves stable policy improvement and consistently outperforms state-of-the-art RL methods on the OGBench offline RL benchmark and real-world robotic manipulation task. Videos are available on our website: https://agibottech.github.io/vine.
Rushuai Yang, Zhuo Han, Houlin Li +10
1AgiBot · 3Peking University · 2The Hong Kong University of Science and Technology
We propose Q-Guided Value-Gradient Matching (Q-VGM), an offline-to-online RL method for fine-tuning flow-matching vision-language-action policies with a learned critic. Directly applying critic gradients to flow policies requires backpropagation through the multi-step denoising process (BPTT), which is costly and unstable at VLA scale. Motivated by an optimal-control view of denoising, we derive a local approximation connecting clean-action value gradients to local velocity corrections at late denoising steps. Q-VGM uses this connection to construct critic-guided targets for fine-tuning the pretrained action expert through velocity matching. Gradients pass only through local velocity predictions, avoiding backpropagation through the denoising trajectory. An action-sensitive chunk critic is trained with IQL offline and TD learning online. On LIBERO, Q-VGM improves the few-shot-SFT four-suite average from 84.6% to 90.7% offline using 150 policy rollouts per suite, and reaches 98.4% after online training. Compared with on-policy RL, Q-VGM achieves 5.3x higher sample efficiency on average across four LIBERO suites, measured by the episode budget for 95% rollout success rate. On three real-world bimanual tasks, offline Q-VGM improves average success from 66.7% to 98.3%.
Ziqian Wang, Rui Zhang, Yitian Liu +3
Shanghai Jiao Tong University · University of Michigan, Ann Arbor
Generative policies based on expressive model classes, such as diffusion and flow matching, are well-suited to complex control problems with highly multimodal action distributions. Their expressivity, however, comes at a significant inference cost: generating each action typically requires simulating many steps of the generative process, compounding latency across sequential decision-making rollouts. We introduce flow map policies, a novel class of generative policies designed for fast action generation by learning to take arbitrary-size jumps including one-step jumps-across the generative dynamics of existing flow-based policies. We instantiate flow map policies for offline-to-online reinforcement learning (RL) and formulate online adaptation as a trust-region optimization problem that improves the critic's Q-value while remaining close to the offline policy. We theoretically derive FLOW MAP Q-GUIDANCE (FMQ), a principled closed-form learning target that is optimal for adapting offline flow map policies under a critic-guided trust-region constraint. We further introduce Q-GUIDED BEAM SEARCH (QGBS), a stochastic flow-map sampler that combines renoising with beam search to enable iterative inference-time refinement. Across 12 challenging robotic manipulation and locomotion tasks from OGBench and RoboMimic, FMQ achieves state-of-the-art performance in offline-to-online RL, outperforming the previous one-step policy MVP by a relative improvement of 21.3% on the average success rate.