Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning

Yongjae Shin; Jongseong Chae; Jongeui Park; Youngchul Sung

cs.LGFeb 20, 2026

Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning

Authors: Yongjae Shin, Jongseong Chae, Jongeui Park, Youngchul Sung

Organizations: KAIST

Abstract

Generative models have recently demonstrated remarkable success across diverse domains, motivating their adoption as expressive policies in reinforcement learning (RL). While they have shown strong performance in offline RL, particularly where the target distribution is well defined, their extension to online fine-tuning has largely been treated as a direct continuation of offline pre-training, leaving key challenges unaddressed. In this paper, we propose Flow Matching with Injected Noise for Offline-to-Online RL (FINO), a novel method that leverages flow matching-based policies to enhance sample efficiency for offline-to-online RL. FINO facilitates effective exploration by injecting noise into policy training, thereby encouraging a broader range of actions beyond those observed in the offline dataset. In addition to exploration-enhanced flow policy training, we combine an entropy-guided sampling mechanism to balance exploration and exploitation, allowing the policy to adapt its behavior throughout online fine-tuning. Experiments across diverse, challenging tasks demonstrate that FINO consistently achieves superior performance under limited online budgets.

Figures & tables

Figure 1: Comparison of FQL and FINO (ours) in terms of performance and exploration patterns on the environment antmaze-giant-navigate . The green circle and red star indicate the initial and goal states, respectively.

Figure 2: Toy example: blue contours represent the log-density of model samples; red circles denote the dataset.

Task	ReBRAC		Cal-QL		RLPD		IFQL		FQL		FINO
OGBench humanoidmaze-medium-navigate	21 $\pm$ 5 $\rightarrow$	3 $\pm$ 3	0 $\pm$ 0 $\rightarrow$	0 $\pm$ 0	0 $\pm$ 0 $\rightarrow$	1 $\pm$ 1	59 $\pm$ 7 $\rightarrow$	70 $\pm$ 5	53 $\pm$ 6 $\rightarrow$	61 $\pm$ 2	50 $\pm$ 7 $\rightarrow$	97 $\pm$ 1
OGBench humanoidmaze-large-navigate	2 $\pm$ 1 $\rightarrow$	1 $\pm$ 0	0 $\pm$ 0 $\rightarrow$	0 $\pm$ 0	0 $\pm$ 0 $\rightarrow$	0 $\pm$ 0	11 $\pm$ 3 $\rightarrow$	10 $\pm$ 2	5 $\pm$ 1 $\rightarrow$	10 $\pm$ 3	6 $\pm$ 2 $\rightarrow$	33 $\pm$ 7
OGBench antmaze-large-navigate	85 $\pm$ 3 $\rightarrow$	99 $\pm$ 0	12 $\pm$ 9 $\rightarrow$	12 $\pm$ 8	0 $\pm$ 0 $\rightarrow$	80 $\pm$ 7	32 $\pm$ 4 $\rightarrow$	72 $\pm$ 6	81 $\pm$ 2 $\rightarrow$	92 $\pm$ 1	81 $\pm$ 2 $\rightarrow$	99 $\pm$ 0
OGBench antmaze-giant-navigate	35 $\pm$ 7 $\rightarrow$	96 $\pm$ 4	2 $\pm$ 3 $\rightarrow$	0 $\pm$ 0	0 $\pm$ 0 $\rightarrow$	47 $\pm$ 9	1 $\pm$ 1 $\rightarrow$	0 $\pm$ 0	16 $\pm$ 6 $\rightarrow$	71 $\pm$ 5	14 $\pm$ 6 $\rightarrow$	79 $\pm$ 0
OGBench antsoccer-arena-navigate	0 $\pm$ 0 $\rightarrow$	0 $\pm$ 0	0 $\pm$ 0 $\rightarrow$	0 $\pm$ 0	0 $\pm$ 0 $\rightarrow$	2 $\pm$ 1	33 $\pm$ 4 $\rightarrow$	35 $\pm$ 5	61 $\pm$ 2 $\rightarrow$	74 $\pm$ 4	57 $\pm$ 3 $\rightarrow$	77 $\pm$ 5
OGBench cube-double-play	9 $\pm$ 2 $\rightarrow$	28 $\pm$ 2	2 $\pm$ 3 $\rightarrow$	0 $\pm$ 0	0 $\pm$ 0 $\rightarrow$	2 $\pm$ 3	14 $\pm$ 1 $\rightarrow$	40 $\pm$ 2	31 $\pm$ 3 $\rightarrow$	73 $\pm$ 2	34 $\pm$ 3 $\rightarrow$	79 $\pm$ 2

Table 1: Performance of FINO and baselines across OGBench and D4RL tasks. Results show scores after offline pre-training and after online fine-tuning, averaged over 10 seeds with mean and 95% confidence intervals. D4RL antmaze and adroit aggregate six and four tasks, respectively, while OGBench reports results over five tasks (task names abbreviated by omitting the singletask suffix). Full results are presented in Table 4 .

Figure 3: Aggregate performance across two benchmark domains. Each figure reports the averaged learning curves over the common environments within the respective domain. Full results are presented in Figures 9 and 10 .

Figure 4: Comparison between FINO and the direct action noise injection baseline. Each plot shows results aggregated over five tasks and averaged across 10 seeds, with shaded regions indicating 95% confidence intervals.

Figure 5: Comparison between FINO and the entropy-regulated noise scaling baseline. Full results are presented in Table 5 .

Figure 6: Comparison of performance and computational efficiency. The left figure shows the learning curve on the humanoidmaze-medium-navigate task, averaged over five tasks with 10 random seeds, with shaded regions denoting 95% confidence intervals. The middle and right figures report the training and inference time per step of each baseline. Full results are presented in Table 5 .

Figure 7: Comparison of performance with varying

N_{\text{sample}}

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Action samples of two one-step policies: blue contours indicate the log-density of the sampled actions, and red × marks denote the centers of the dataset.

Hyperparameter	Value
Noise constant $\eta$	0.05 $\cdot\|\mathcal{A}\|$
Candidate action samples $N_{\text{sample}}$	0.5 $\cdot\dim(\mathcal{A})$
Target entropy $\bar{\mathcal{H}}$	$-\dim(\mathcal{A})$
Entropy update steps $N_{\xi}$	50,000
Standard deviation of noise $\alpha_{t}$	$\eta\cdot\exp(5(t-1))$
Hyperparameter (from FQL)	Value

Appendix

Table 2: Hyperparameters

Task	ReBRAC $(\alpha_{1},\alpha_{2})$	Cal-QL $(\alpha,\beta)$	IFQL ( $N$ )	FQL ( $\alpha$ )
humanoidmaze-medium-navigate-v0	(0.01, 0.01)	(5, 0.8)	32	100
humanoidmaze-large-navigate-v0	(0.01, 0.01)	(5, 0.8)	32	30
antmaze-large-navigate-v0	(0.003, 0.01)	(5, 0.8)	32	30
antmaze-giant-navigate-v0	(0.003, 0.01)	(5, 0.8)	32	10
antsoccer-arena-navigate-v0	(0.01, 0.01)	(5, 0.2)	64	30
cube-double-play-v0	(0.1, 0)	(0.01, 1)	32	300

Appendix

Table 3: Task-specific hyperparameters for each baseline.

Environment

ReBRAC

Cal-QL

RLPD

IFQL

FQL

FINO

OGBench humanoidmaze-medium-navigate-task1

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

OGBench humanoidmaze-medium-navigate-task2

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

OGBench humanoidmaze-medium-navigate-task3

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

OGBench humanoidmaze-medium-navigate-task4

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

OGBench humanoidmaze-medium-navigate-task5

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

100

\pm

\rightarrow

\pm

\pm

\rightarrow

100

\pm

OGBench humanoidmaze-large-navigate-task1

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

\pm

\rightarrow

\pm

Appendix

Table 4: Full results for main experiments (corresponding to Table 1 and Fig. 3 ). Scores show offline pre-training

\rightarrow

online fine-tuning, averaged over 10 seeds (mean

\pm

95% CI). For OGBench, the singletask suffix is omitted.

Environment	FINO		Direct Noise		w/o Noise		w/o Guidance
OGBench humanoidmaze-medium-navigate-task1	13 $\pm$ 3 $\rightarrow$	91 $\pm$ 3	14 $\pm$ 5 $\rightarrow$	0 $\pm$ 1	18 $\pm$ 5 $\rightarrow$	0 $\pm$ 0	16 $\pm$ 3 $\rightarrow$	50 $\pm$ 16
OGBench humanoidmaze-medium-navigate-task2	77 $\pm$ 25 $\rightarrow$	99 $\pm$ 1	88 $\pm$ 16 $\rightarrow$	96 $\pm$ 6	80 $\pm$ 19 $\rightarrow$	99 $\pm$ 2	71 $\pm$ 25 $\rightarrow$	84 $\pm$ 20
OGBench humanoidmaze-medium-navigate-task3	52 $\pm$ 24 $\rightarrow$	99 $\pm$ 1	54 $\pm$ 22 $\rightarrow$	60 $\pm$ 32	45 $\pm$ 24 $\rightarrow$	49 $\pm$ 32	48 $\pm$ 23 $\rightarrow$	79 $\pm$ 25
OGBench humanoidmaze-medium-navigate-task4	11 $\pm$ 4 $\rightarrow$	94 $\pm$ 3	7 $\pm$ 10 $\rightarrow$	31 $\pm$ 24	2 $\pm$ 4 $\rightarrow$	20 $\pm$ 26	0 $\pm$ 0 $\rightarrow$	74 $\pm$ 16
OGBench humanoidmaze-medium-navigate-task5	99 $\pm$ 1 $\rightarrow$	100 $\pm$ 1	99 $\pm$ 1 $\rightarrow$	100 $\pm$ 1	99 $\pm$ 1 $\rightarrow$	99 $\pm$ 1	98 $\pm$ 1 $\rightarrow$	99 $\pm$ 1
OGBench humanoidmaze-large-navigate-task1	5 $\pm$ 4 $\rightarrow$	5 $\pm$ 9	4 $\pm$ 4 $\rightarrow$	0 $\pm$ 0	3 $\pm$ 4 $\rightarrow$	8 $\pm$ 14	4 $\pm$ 4 $\rightarrow$	0 $\pm$ 1

Appendix

Table 5: Full results for ablation studies (Fig. 5 , Fig. 6 ). Scores show offline pre-training

\rightarrow

online fine-tuning, averaged over 10 seeds (mean

\pm

95% CI). For OGBench, the singletask suffix is omitted.

Figure 9: Full results on OGBench environments. Each row corresponds to one environment, with five single-task variants shown side by side. Shaded areas denote 95% confidence intervals over 10 seeds.

Figure 10: Full results on D4RL environments. For AntMaze tasks, the prefix “antmaze-” is omitted for clarity. Shaded areas denote 95% confidence intervals over 10 seeds.

Explore similar work

Reinforcement Learning via Fixed-point Flow Matching
Oct 9, 2026Denis Blessing, Onur Celik, Gerhard NeumannReinforcement Learning Flow Matching
Karlsruhe Institute of Technology (KIT) Autonomous Learning Robots Lab (ALR)
Discrete Flow Matching for Offline-to-Online Reinforcement Learning
May 12, 2026Fairoz Nower Khan, Nabuat Zaman Nahim, Peizhong JuFlow Matching Reinforcement Learning
Department of Computer Science, University of Kentucky
Flow Matching for Offline Reinforcement Learning with Discrete Actions
Feb 5, 2026Fairoz Nower Khan, Nabuat Zaman Nahim, Ruiquan Huang +2Reinforcement Learning Offline RL
Department of Computer Science, University of Kentucky · Department of Computing and Information Sciences, Rochester Institute of Technology

Oct 9, 2026J/K move · Enter open · S save

Reinforcement Learning via Fixed-point Flow Matching

Denis Blessing, Onur Celik, Gerhard Neumann

Karlsruhe Institute of Technology (KIT) Autonomous Learning Robots Lab (ALR)

Reinforcement Learning Flow Matching Maximum Entropy RL Flow-Based Generative Modeling

Open paper AskPDF arXiv

In maximum entropy and behavior-constrained reinforcement learning (RL), the optimal policy is given by a Boltzmann distribution. Generating samples from this distribution can be challenging, as it is often complex and multimodal. While flow policies trained with flow matching provide a powerful framework for representing such distributions, they traditionally rely on access to training data, which is generally unavailable in online RL. To address this limitation, we propose fixed-point flow matching (FFM), a data-free procedure for training and fine-tuning flow policies in RL. FFM leverages the target flow identity to formulate the actor update as a fixed-point iteration, utilizing a flow-matching-style regression objective where the critic's action gradients serve as regression targets. We show that the optimal policy is a fixed point of this iteration. Through extensive experiments, we demonstrate that FFM effectively fine-tunes pretrained flow policies in the offline-to-online setting and enables efficient and accurate training of flow policies from scratch in online RL. These results establish fixed-point flow matching as a principled approach for learning expressive flow-based policies without requiring training data.

Abstract

Explore similar work

Reinforcement Learning via Fixed-point Flow Matching

Discrete Flow Matching for Offline-to-Online Reinforcement Learning

Flow Matching for Offline Reinforcement Learning with Discrete Actions