cs.LGMay 22, 2026

Score-Based One-step MeanFlow Policy Optimization

Authors: Kyungyoon KimDonghyeon KiHee-Jun AhnByung-Jun Lee

Organizations: Korea University, Decision Making Lab · Gauss Labs Inc.

Abstract

Diffusion and flow matching have emerged as expressive policy classes in reinforcement learning, but their reliance on multi-step denoising imposes substantial computational overhead at inference time, which is particularly problematic in online RL. MeanFlow offers a promising alternative by learning an average velocity field that maps noise to data in a single network evaluation. However, MeanFlow typically requires samples from the target distribution to construct its target velocity field, which are unavailable in online RL. We propose Score-Based One-step MeanFlow Policy Optimization (SOM), an actor-critic algorithm that resolves this by constructing the target velocity field directly from the Q-function via score estimation and a probability flow ODE, thereby concentrating probability mass on high-value modes. In the fully online RL setting, SOM achieves state-of-the-art performance on locomotion tasks with a single generation step, while substantially reducing both training and inference time compared to prior diffusion- and flow-matching-based policies.

Explore similar work

CardsList
  1. Mean Flow Policy Optimization

    Apr 16, 2026Xiaoyi Dong, Xi Sheryl Zhang, Jian ChengFlow-Based PolicyGenerative Flow Networks

  2. Moment Matching Q-Learning

    May 27, 2026Yiyan, Liang, Sifei Liu +1Q-LearningFlow-Based Policy

  3. Aligning Flow Map Policies with Optimal Q-Guidance

    May 12, 2026Christos Ziakas, Alessandra Russo, Avishek Joey BoseFlow-Based PolicyNeural Guidance