Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning

Yongjae Shin; Jongseong Chae; Jongeui Park; Youngchul Sung

cs.LGFeb 20, 2026

Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning

Authors: Yongjae Shin, Jongseong Chae, Jongeui Park, Youngchul Sung

Organizations: KAIST

Abstract

Generative models have recently demonstrated remarkable success across diverse domains, motivating their adoption as expressive policies in reinforcement learning (RL). While they have shown strong performance in offline RL, particularly where the target distribution is well defined, their extension to online fine-tuning has largely been treated as a direct continuation of offline pre-training, leaving key challenges unaddressed. In this paper, we propose Flow Matching with Injected Noise for Offline-to-Online RL (FINO), a novel method that leverages flow matching-based policies to enhance sample efficiency for offline-to-online RL. FINO facilitates effective exploration by injecting noise into policy training, thereby encouraging a broader range of actions beyond those observed in the offline dataset. In addition to exploration-enhanced flow policy training, we combine an entropy-guided sampling mechanism to balance exploration and exploitation, allowing the policy to adapt its behavior throughout online fine-tuning. Experiments across diverse, challenging tasks demonstrate that FINO consistently achieves superior performance under limited online budgets.

Figures & tables

Figure 1: Comparison of FQL and FINO (ours) in terms of performance and exploration patterns on the environment antmaze-giant-navigate . The green circle and red star indicate the initial and goal states, respectively.

Figure 2: Toy example: blue contours represent the log-density of model samples; red circles denote the dataset.

Task	ReBRAC		Cal-QL		RLPD		IFQL		FQL		FINO
OGBench humanoidmaze-medium-navigate	21 $\pm$ 5 $\rightarrow$	3 $\pm$ 3	0 $\pm$ 0 $\rightarrow$	0 $\pm$ 0	0 $\pm$ 0 $\rightarrow$	1 $\pm$ 1	59 $\pm$ 7 $\rightarrow$	70 $\pm$ 5	53 $\pm$ 6 $\rightarrow$	61 $\pm$ 2	50 $\pm$ 7 $\rightarrow$	97 $\pm$ 1
OGBench humanoidmaze-large-navigate	2 $\pm$ 1 $\rightarrow$	1 $\pm$ 0	0 $\pm$ 0 $\rightarrow$	0 $\pm$ 0	0 $\pm$ 0 $\rightarrow$	0 $\pm$ 0	11 $\pm$ 3 $\rightarrow$	10 $\pm$ 2	5 $\pm$ 1 $\rightarrow$	10 $\pm$ 3	6 $\pm$ 2 $\rightarrow$	33 $\pm$ 7
OGBench antmaze-large-navigate	85 $\pm$ 3 $\rightarrow$	99 $\pm$ 0	12 $\pm$ 9 $\rightarrow$	12 $\pm$ 8	0 $\pm$ 0 $\rightarrow$	80 $\pm$ 7	32 $\pm$ 4 $\rightarrow$	72 $\pm$ 6	81 $\pm$ 2 $\rightarrow$	92 $\pm$ 1	81 $\pm$ 2 $\rightarrow$	99 $\pm$ 0
OGBench antmaze-giant-navigate	35 $\pm$ 7 $\rightarrow$	96 $\pm$ 4	2 $\pm$ 3 $\rightarrow$	0 $\pm$ 0	0 $\pm$ 0 $\rightarrow$	47 $\pm$ 9	1 $\pm$ 1 $\rightarrow$	0 $\pm$ 0	16 $\pm$ 6 $\rightarrow$	71 $\pm$ 5	14 $\pm$ 6 $\rightarrow$	79 $\pm$ 0
OGBench antsoccer-arena-navigate	0 $\pm$ 0 $\rightarrow$	0 $\pm$ 0	0 $\pm$ 0 $\rightarrow$	0 $\pm$ 0	0 $\pm$ 0 $\rightarrow$	2 $\pm$ 1	33 $\pm$ 4 $\rightarrow$	35 $\pm$ 5	61 $\pm$ 2 $\rightarrow$	74 $\pm$ 4	57 $\pm$ 3 $\rightarrow$	77 $\pm$ 5
OGBench cube-double-play	9 $\pm$ 2 $\rightarrow$	28 $\pm$ 2	2 $\pm$ 3 $\rightarrow$	0 $\pm$ 0	0 $\pm$ 0 $\rightarrow$	2 $\pm$ 3	14 $\pm$ 1 $\rightarrow$	40 $\pm$ 2	31 $\pm$ 3 $\rightarrow$	73 $\pm$ 2	34 $\pm$ 3 $\rightarrow$	79 $\pm$ 2

Table 1: Performance of FINO and baselines across OGBench and D4RL tasks. Results show scores after offline pre-training and after online fine-tuning, averaged over 10 seeds with mean and 95% confidence intervals. D4RL antmaze and adroit aggregate six and four tasks, respectively, while OGBench reports results over five tasks (task names abbreviated by omitting the singletask suffix). Full results are presented in Table 4 .

Figure 3: Aggregate performance across two benchmark domains. Each figure reports the averaged learning curves over the common environments within the respective domain. Full results are presented in Figures 9 and 10 .

Figure 4: Comparison between FINO and the direct action noise injection baseline. Each plot shows results aggregated over five tasks and averaged across 10 seeds, with shaded regions indicating 95% confidence intervals.

Figure 5: Comparison between FINO and the entropy-regulated noise scaling baseline. Full results are presented in Table 5 .

Figure 6: Comparison of performance and computational efficiency. The left figure shows the learning curve on the humanoidmaze-medium-navigate task, averaged over five tasks with 10 random seeds, with shaded regions denoting 95% confidence intervals. The middle and right figures report the training and inference time per step of each baseline. Full results are presented in Table 5 .

Figure 7: Comparison of performance with varying

N_{\text{sample}}

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Action samples of two one-step policies: blue contours indicate the log-density of the sampled actions, and red × marks denote the centers of the dataset.

Hyperparameter	Value
Noise constant $\eta$	0.05 $\cdot\|\mathcal{A}\|$
Candidate action samples $N_{\text{sample}}$	0.5 $\cdot\dim(\mathcal{A})$
Target entropy $\bar{\mathcal{H}}$	$-\dim(\mathcal{A})$
Entropy update steps $N_{\xi}$	50,000
Standard deviation of noise $\alpha_{t}$	$\eta\cdot\exp(5(t-1))$
Hyperparameter (from FQL)	Value

Appendix

Table 2: Hyperparameters

Task	ReBRAC $(\alpha_{1},\alpha_{2})$	Cal-QL $(\alpha,\beta)$	IFQL ( $N$ )	FQL ( $\alpha$ )
humanoidmaze-medium-navigate-v0	(0.01, 0.01)	(5, 0.8)	32	100
humanoidmaze-large-navigate-v0	(0.01, 0.01)	(5, 0.8)	32	30
antmaze-large-navigate-v0	(0.003, 0.01)	(5, 0.8)	32	30
antmaze-giant-navigate-v0	(0.003, 0.01)	(5, 0.8)	32	10
antsoccer-arena-navigate-v0	(0.01, 0.01)	(5, 0.2)	64	30
cube-double-play-v0	(0.1, 0)	(0.01, 1)	32	300

Appendix

Table 3: Task-specific hyperparameters for each baseline.

Environment

ReBRAC

Cal-QL

RLPD

IFQL

FQL

FINO

OGBench humanoidmaze-medium-navigate-task1

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

OGBench humanoidmaze-medium-navigate-task2

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

OGBench humanoidmaze-medium-navigate-task3

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

OGBench humanoidmaze-medium-navigate-task4

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

OGBench humanoidmaze-medium-navigate-task5

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

100

\pm

\rightarrow

\pm

\pm

\rightarrow

100

\pm

OGBench humanoidmaze-large-navigate-task1

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

Appendix

Table 4: Full results for main experiments (corresponding to Table 1 and Fig. 3 ). Scores show offline pre-training

\rightarrow

online fine-tuning, averaged over 10 seeds (mean

\pm

95% CI). For OGBench, the singletask suffix is omitted.

Environment	FINO		Direct Noise		w/o Noise		w/o Guidance
OGBench humanoidmaze-medium-navigate-task1	13 $\pm$ 3 $\rightarrow$	91 $\pm$ 3	14 $\pm$ 5 $\rightarrow$	0 $\pm$ 1	18 $\pm$ 5 $\rightarrow$	0 $\pm$ 0	16 $\pm$ 3 $\rightarrow$	50 $\pm$ 16
OGBench humanoidmaze-medium-navigate-task2	77 $\pm$ 25 $\rightarrow$	99 $\pm$ 1	88 $\pm$ 16 $\rightarrow$	96 $\pm$ 6	80 $\pm$ 19 $\rightarrow$	99 $\pm$ 2	71 $\pm$ 25 $\rightarrow$	84 $\pm$ 20
OGBench humanoidmaze-medium-navigate-task3	52 $\pm$ 24 $\rightarrow$	99 $\pm$ 1	54 $\pm$ 22 $\rightarrow$	60 $\pm$ 32	45 $\pm$ 24 $\rightarrow$	49 $\pm$ 32	48 $\pm$ 23 $\rightarrow$	79 $\pm$ 25
OGBench humanoidmaze-medium-navigate-task4	11 $\pm$ 4 $\rightarrow$	94 $\pm$ 3	7 $\pm$ 10 $\rightarrow$	31 $\pm$ 24	2 $\pm$ 4 $\rightarrow$	20 $\pm$ 26	0 $\pm$ 0 $\rightarrow$	74 $\pm$ 16
OGBench humanoidmaze-medium-navigate-task5	99 $\pm$ 1 $\rightarrow$	100 $\pm$ 1	99 $\pm$ 1 $\rightarrow$	100 $\pm$ 1	99 $\pm$ 1 $\rightarrow$	99 $\pm$ 1	98 $\pm$ 1 $\rightarrow$	99 $\pm$ 1
OGBench humanoidmaze-large-navigate-task1	5 $\pm$ 4 $\rightarrow$	5 $\pm$ 9	4 $\pm$ 4 $\rightarrow$	0 $\pm$ 0	3 $\pm$ 4 $\rightarrow$	8 $\pm$ 14	4 $\pm$ 4 $\rightarrow$	0 $\pm$ 1

Appendix

Table 5: Full results for ablation studies (Fig. 5 , Fig. 6 ). Scores show offline pre-training

\rightarrow

online fine-tuning, averaged over 10 seeds (mean

\pm

95% CI). For OGBench, the singletask suffix is omitted.

Figure 9: Full results on OGBench environments. Each row corresponds to one environment, with five single-task variants shown side by side. Shaded areas denote 95% confidence intervals over 10 seeds.

Figure 10: Full results on D4RL environments. For AntMaze tasks, the prefix “antmaze-” is omitted for clarity. Shaded areas denote 95% confidence intervals over 10 seeds.

Explore similar work

Oct 9, 2026

Reinforcement Learning via Fixed-point Flow Matching

In maximum entropy and behavior-constrained reinforcement learning (RL), the optimal policy is given by a Boltzmann distribution. Generating samples from this distribution can be challenging, as it is often complex and multimodal. While flow policies trained with flow matching provide a powerful framework for representing such distributions, they traditionally rely on access to training data, which is generally unavailable in online RL. To address this limitation, we propose fixed-point flow matching (FFM), a data-free procedure for training and fine-tuning flow policies in RL. FFM leverages the target flow identity to formulate the actor update as a fixed-point iteration, utilizing a flow-matching-style regression objective where the critic's action gradients serve as regression targets. We show that the optimal policy is a fixed point of this iteration. Through extensive experiments, we demonstrate that FFM effectively fine-tunes pretrained flow policies in the offline-to-online setting and enables efficient and accurate training of flow policies from scratch in online RL. These results establish fixed-point flow matching as a principled approach for learning expressive flow-based policies without requiring training data.

Denis Blessing, Onur Celik, Gerhard Neumann

Karlsruhe Institute of Technology (KIT) Autonomous Learning Robots Lab (ALR)

Reinforcement Learning Flow Matching Maximum Entropy RL Flow-Based Generative Modeling

Ask PDF

May 12, 2026

Discrete Flow Matching for Offline-to-Online Reinforcement Learning

Many reinforcement learning (RL) tasks have discrete action spaces, but most generative policy methods based on diffusion and flow matching are designed for continuous control. Meanwhile, generative policies usually rely heavily on offline datasets and offline-to online RL is itself challenging, as the policy must improve from new interaction without losing useful behavior learned from static data. To address those challenges, we introduce DRIFT, an online fine-tuning method that updates an offline pretrained continuous time Markov chain (CTMC) policy with an advantage-weighted discrete flow matching loss. To preserve useful pretrained knowledge, we add a path-space penalty that regularizes the full CTMC trajectory distribution, rather than only the final action distribution. For large discrete action spaces, we introduce a candidate-set approximation that updates the actor over a small subset of actions sampled from reference-policy rollouts and uniform exploration. Our theoretical analysis shows that focusing on a small set of high probability actions can reduce computation while keeping the approximation error small, making our method practical for large action spaces. We evaluate on macro-action MinAtar games where offline-pretrained DRIFT outperforms its online-only counterpart across all five games and achieves the highest mean return on three and on oracle free Jericho, DRIFT achieves a mean normalized score of 7.3% with a lightweight GRU encoder, outperforming the evaluated pretrained language-model baselines. Ablations demonstrate the importance of appropriately weighted path-space regularization and reference-guided candidate construction.

Fairoz Nower Khan, Nabuat Zaman Nahim, Peizhong Ju

Department of Computer Science, University of Kentucky

Flow Matching Reinforcement Learning RL Fine-Tuning Offline-To-Online RL Discrete Flow Matching

Ask PDF

Feb 5, 2026

Flow Matching for Offline Reinforcement Learning with Discrete Actions

Generative policies based on diffusion models and flow matching have shown strong promise for offline reinforcement learning (RL), but their applicability remains largely confined to continuous action spaces. To address a broader range of offline RL settings, we extend flow matching to a general framework that supports discrete action spaces with multiple objectives. Specifically, we replace continuous flows with continuous-time Markov chains, trained using a Q-weighted flow matching objective. We then extend our design to multi-agent settings, mitigating the exponential growth of joint action spaces via a factorized conditional path. We theoretically show that, under idealized conditions, optimizing this objective recovers the optimal policy. Extensive experiments further demonstrate that our method performs robustly across diverse settings and benchmarks, including high-dimensional control, multi-agent games, and dynamically changing preferences over multiple objectives, while outperforming traditional offline RL methods in practical multi-modal decision-making scenarios. Our discrete framework can also be applied to continuous-control problems through action quantization, providing a flexible trade-off between representational complexity and performance.

Fairoz Nower Khan, Nabuat Zaman Nahim, Ruiquan Huang +2

Department of Computer Science, University of Kentucky · Department of Computing and Information Sciences, Rochester Institute of Technology

Reinforcement Learning Offline RL Flow Matching Multi-Agent Reinforcement Learning Multi-Objective Reinforcement Learning

Ask PDF