Variance-Averse $n$-Step Offline Reinforcement Learning for Sparse Long-Horizon Environments

Guhyeon Kang; Minhae Kwon

cs.LGOct 6, 2026

Variance-Averse $n$ -Step Offline Reinforcement Learning for Sparse Long-Horizon Environments

Authors: Guhyeon Kang, Minhae Kwon

Organizations: Department of Electrical and Computer Engineering Sungkyunkwan University, Republic of Korea

Abstract

Generative actors are transforming offline reinforcement learning (RL) by enabling expressive policy classes that model complex action distributions. However, this expressiveness also exposes a key challenge in heterogeneous datasets: generative policies can reproduce unreliable action modes whose return distributions exhibit high variance, occasionally yielding high returns by chance but lacking consistency. Consequently, maximizing the expected $Q$ -value alone is insufficient for identifying reliable actions. We propose VAN-Flow (Variance-Averse $n$ -step Flow), a framework that promotes reliable actions in generative offline RL. VAN-Flow combines (i) a categorical distributional critic, (ii) a variance-averse expectation operator that smoothly reweights atom probabilities to favor actions with both high returns and low dispersion, and (iii) a flow-matching generative actor guided via rejection sampling. Unlike CVaR or mean-variance objectives, the operator redistributes probability mass over the categorical return distribution without hard truncation or auxiliary penalty terms. Across more than 40 tasks from D4RL and OGBench, VAN-Flow consistently outperforms strong baselines, with the largest gains in long-horizon and high-variance regimes where reliable action selection becomes critical.

Figures & tables

Figure 1 : Success rate on antmaze-large under two dataset variance regimes: (a) navigate (low-variance) and (b) explore (high-variance). In (b), the shaded bar denotes the low-variance reference, and the downward arrow indicates the performance drop.

Figure 2 : Example of variance-averse return with varying

\delta

and return variance.

	Gaussian-based				Flow-based					Proposed
Task	IQL	HIQL	ReBRAC	ReBRAC- $n$	FQL	BFN	FQL- $n$	BFN- $n$	QC	VAN-Flow
Standard tasks
antmaze-large-navigate	53	91	81	82	79	71	91	81	49	95
antmaze-teleport-navigate	23	42	7	35	26	40	22	41	40	57
scene-play	0	38	11	40	57	85	18	57	84	93
puzzle-3x3-play	0	12	55	10	100	98	98	58	100	100

Table 1 : Performance comparison on OGBench. Each entry averages five tasks per environment. Best results are in bold.

	Pure Offline RL			$n$ -step					Dist.	Dist.+ $n$	Proposed
Task	IQL	ReBRAC	FQL	Retrace( $\lambda$ )	PQL	LEQ	QC	TD3BC+MS	PA-RL	D4PG	VAN-Flow
umaze	77.0 $\pm 5.5$	97.7 $\pm 1.5$	96.0 $\pm 2.0$	96.0 $\pm 3.0$	94.0 $\pm 3.0$	94.4 $\pm 6.3$	95.0 $\pm 1.4$	–	–	87.6 $\pm 4.1$	98.0 $\pm 2.8$
umaze-diverse	54.2 $\pm 5.5$	83.5 $\pm 7.0$	89.0 $\pm 2.0$	84.0 $\pm 10.0$	63.0 $\pm 6.0$	71.0 $\pm 12.3$	78.0 $\pm 8.5$	–	–	61.3 $\pm 9.4$	95.6 $\pm 2.6$
medium-play	65.7 $\pm 11.7$	89.5 $\pm 3.3$	78.0 $\pm 7.0$	79.0 $\pm 10.0$	81.0 $\pm 6.0$	58.8 $\pm 33.0$	72.0 $\pm 2.4$	70.0	88.0	89.1 $\pm 3.0$	90.7 $\pm 3.1$
medium-diverse	73.7 $\pm 5.4$	83.5 $\pm 8.2$	71.0 $\pm 13.0$	53.0 $\pm 17.0$	60.0 $\pm 33.0$	46.2 $\pm 23.2$	64.0 $\pm 2.8$	66.0	88.0	81.3 $\pm 4.1$	89.3 $\pm 5.0$
large-play	42.0 $\pm 4.5$	52.2 $\pm 29.0$	84.0 $\pm 7.0$	12.0 $\pm 26.0$	33.0 $\pm 10.0$	58.6 $\pm 9.1$	80.0 $\pm 2.8$	22.0	87.0	71.3 $\pm 4.1$	86.5 $\pm 5.7$

Table 2 : Offline performance on D4RL AntMaze tasks. Scores are normalized following the D4RL protocol. Entries marked “–” indicate results not reported in prior work. Best results are in bold.

Figure 3 : Effect of flow-based policies and categorical critics on humanoidmaze-giant-navigate .

Task	Metric	$\mathbb{E}[Z]$	CVaR	Entropic	Mean–Var	$\mathcal{E}(Z)$
humanoidmaze -large-navigate	$\mathrm{Var}[Z]\!\downarrow$	$1.00$	$0.95$	$0.97$	$0.94$	$\mathbf{0.93}$
humanoidmaze -large-navigate	$\mathrm{Success}\!\uparrow$	$77$	$83$	$79$	$80$	$\mathbf{87}$
antmaze -teleport-navigate	$\mathrm{Var}[Z]\!\downarrow$	$1.00$	$1.08$	$0.93$	$0.90$	$\mathbf{0.62}$
antmaze -teleport-navigate	$\mathrm{Success}\!\uparrow$	$38$	$54$	$44$	$46$	$\mathbf{71}$
antmaze-large-explore	$\mathrm{Var}[Z]\!\downarrow$	$1.00$	$0.87$	$0.72$	$0.77$	$\mathbf{0.66}$
antmaze-large-explore	$\mathrm{Success}\!\uparrow$	$84$	$2$	$86$	$85$	$\mathbf{93}$

Table 3 : Ablation comparing return aggregation rules.

Figure 4 : Effect of

Q

guidance on policy performance across multiple tasks. Bars report mean success rate and time efficiency.

Figure 5 : Sensitivity analysis with respect to key algorithmic parameters.

Method	Complexity	Step (ms)	Wall (h)
FQL	$\mathcal{O}(UKd^{2})$	1.1	0.32
BFN	$\mathcal{O}(UMKd^{2})$	2.1	0.60
QC	$\mathcal{O}(UMKd^{2})$	3.0	0.83
VAN-Flow	$\mathcal{O}(UMKd^{2})$	1.9	0.54

Table 4 : Runtime comparison on humanoidmaze-large-navigate .

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Description
$\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma)$	Markov decision process
$\pi_{\theta}(\cdot\mid s)$	Flow-based policy (actor) with parameters $\theta$
$v^{\pi}_{\theta}(\tau,s,x)$	Flow-matching velocity field (policy parameterization)
$Q_{\psi}(s,a)$	Scalar critic (used in standard 1-step formulations)
$Z_{\psi}(s,a)$	Categorical return distribution critic used in VAN-Flow
$Z(s,a)$	Return random variable (categorical distribution) for $(s,a)$

Appendix

Table 5 : Notation table.

Measurement	Value
Median realized gap $\|\mathcal{E}-\mathcal{E}^{\mathrm{sp}}\|$ (fraction of the value range $R$ )	$0.37\%$
States in which $\mathcal{E}$ and $\mathcal{E}^{\mathrm{sp}}$ select the same candidate	$96.9\%$
Kendall’s $\tau$ between the two candidate rankings	$0.99$

Appendix

Table 6 : Discretization gap between the implemented operator

\mathcal{E}

and its spectral form

\mathcal{E}^{\mathrm{sp}}

on trained critics.

Hyperparameter	Value
Optimization
Learning rate (critic) $\alpha_{Z}$	$3\times 10^{-4}$
Learning rate (actor) $\alpha_{\pi}$	$3\times 10^{-4}$
Optimizer	Adam [ 54 ]
Gradient steps	$10^{6}$ (OGBench), $5\times 10^{5}$ (D4RL)
Minibatch size	256

Appendix

Table 7 : Hyperparameters for VAN-Flow.

Variant	Dist	VE	Flow	antmaze-teleport (8d)	antmaze-giant (8d)	humanoidmaze-large (21d)
VAN-Flow (Ours)	✓	✓	✓	$\mathbf{71\!\pm\!7}$	$\mathbf{58\!\pm\!4}$	$\mathbf{87\!\pm\!6}$
w/o VE	✓	✗	✓	$38\!\pm\!7$	$36\!\pm\!6$	$77\!\pm\!4$
w/o Flow	✓	✓	✗	$42\!\pm\!7$	$38\!\pm\!13$	$0\!\pm\!0$
w/o VE & Flow	✓	✗	✗	$36\!\pm\!9$	$32\!\pm\!5$	$0\!\pm\!0$

Appendix

Table 8 : Component-wise ablation across action-dimension regimes. All variants use

n

-step returns. Scores denote success rate (%), averaged over five random seeds.

	VAN-DDPM		VAN-Flow
Steps	Perf.	Time (ms)	Perf.	Time (ms)
1	$0\raisebox{0.68887pt}{\scriptsize$ \pm 0 $}$	2.0	$0\raisebox{0.68887pt}{\scriptsize$ \pm 0 $}$	1.6
3	$0\raisebox{0.68887pt}{\scriptsize$ \pm 0 $}$	3.8	$\mathbf{96\raisebox{0.68887pt}{\scriptsize$ \pm 3 $}}$	3.3
5	$0\raisebox{0.68887pt}{\scriptsize$ \pm 0 $}$	5.7	$96\raisebox{0.68887pt}{\scriptsize$ \pm 2 $}$	5.0
10	$57\raisebox{0.68887pt}{\scriptsize$ \pm 17 $}$	8.4	$87\raisebox{0.68887pt}{\scriptsize$ \pm 6 $}$	7.2
20	$\mathbf{94\raisebox{0.68887pt}{\scriptsize$ \pm 4 $}}$	11.9	$76\raisebox{0.68887pt}{\scriptsize$ \pm 6 $}$	10.2

Appendix

Table 9 : Comparison between flow matching (VAN-Flow) and diffusion (VAN-DDPM) policies on humanoidmaze-large-navigate . Performance denotes success rate (%); Time denotes cumulative inference time per action (ms). Bold entries indicate the first step count achieving

>

90% success rate.

Figure 8 : Learning curve on the antmaze task in OGBench

Figure 9 : Learning curve on the humanoidmaze task in OGBench

Figure 10 : Learning curve on the puzzle task in OGBench

Figure 11 : Learning curve on the scene task in OGBench

Figure 12 : Learning curve on the antmaze task in D4RL

Env	Task	Gaussian-based				Flow-based					Proposed
Env	Task	IQL	ReBRAC	ReBRAC- $n$	HIQL	FQL	BFN	FQL- $n$	BFN- $n$	QC	VAN-Flow
antmaze-large-navigate	task1	48 $\pm 9$	91 $\pm 10$	64.0 $\pm 25.8$	93.0 $\pm 3.0$	80 $\pm 8$	93.0 $\pm 1.4$	94.2 $\pm 2.8$	83.1 $\pm 1.4$	49.2 $\pm 9.9$	94.6 $\pm 4.2$
	task2	42 $\pm 6$	88 $\pm 4$	78.3 $\pm 5.9$	78.0 $\pm 9.0$	57 $\pm 10$	90.2 $\pm 2.8$	80.0 $\pm 3.4$	88.0 $\pm 11.3$	43.4 $\pm 2.4$	89.4 $\pm 7.0$
	task3	72 $\pm 7$	51 $\pm 18$	92.8 $\pm 2.3$	96.0 $\pm 2.0$	93 $\pm 3$	89.1 $\pm 7.0$	97.5 $\pm 1.4$	74.5 $\pm 5.6$	71.8 $\pm 2.8$	98.0 $\pm 2.0$
	task4	51 $\pm 9$	84 $\pm 7$	90.0 $\pm 2.8$	94.0 $\pm 2.0$	80 $\pm 4$	9.0 $\pm 1.4$	89.1 $\pm 7.0$	73.1 $\pm 1.4$	22.0 $\pm 2.8$	95.4 $\pm 3.1$
	task5	54 $\pm 22$	90 $\pm 2$	90.8 $\pm 2.7$	94.0 $\pm 3.0$	83 $\pm 4$	72.0 $\pm 5.6$	94.7 $\pm 5.6$	85.1 $\pm 1.4$	61.6 $\pm 1.4$	98.0 $\pm 2.0$

Appendix

Table 10 : Task-wise performance comparison in OGBench (singletask setting).

Task	Gaussian-based			Diffusion Policies			Flow-based				Proposed
Task	BC	IQL	ReBRAC	IDQL	SRPO	CAC	FAWAC	FBRAC	IFQL	FQL	Ours
pen-human-v1	71	78	103	76	69	64	67	77	71	53	82.3 $\pm 3.5$
pen-cloned-v1	52	83	103	64	61	56	62	67	80	74	107.5 $\pm 14.9$
pen-expert-v1	110	128	152	140	134	103	118	119	139	142	146.6 $\pm 5.5$
door-human-v1	2	3	0	6	3	5	2	4	7	0	2.7 $\pm 1.3$
door-cloned-v1	0	3	0	0	0	1	0	0	2	2	3.0 $\pm 2.8$

Appendix

Table 11 : Performance comparison in D4RL Adroit

Task	IQL	ReBRAC	Diffusion-QL	Diffuser	VAN-Flow
halfcheetah-medium	47.4	$\mathbf{65.6\raisebox{0.68887pt}{\scriptsize$ \pm 1.0 $}}$	51.1 $\pm 0.5$	42.8 $\pm 0.3$	63.9 $\pm 2.1$
halfcheetah-medium-replay	44.2	51.0 $\pm 0.8$	47.8 $\pm 0.3$	37.7 $\pm 0.5$	$\mathbf{51.4\raisebox{0.68887pt}{\scriptsize$ \pm 0.3 $}}$
hopper-medium	66.3	$\mathbf{102.0\raisebox{0.68887pt}{\scriptsize$ \pm 1.0 $}}$	90.5 $\pm 4.6$	74.3 $\pm 1.4$	95.3 $\pm 2.5$
hopper-medium-replay	94.7	98.1 $\pm 5.3$	$\mathbf{101.3\raisebox{0.68887pt}{\scriptsize$ \pm 0.6 $}}$	93.6 $\pm 0.4$	100.3 $\pm 1.2$
walker2d-medium	78.3	82.5 $\pm 3.6$	87.0 $\pm 0.9$	79.6 $\pm 0.6$	$\mathbf{88.2\raisebox{0.68887pt}{\scriptsize$ \pm 1.8 $}}$
walker2d-medium-replay	73.9	77.3 $\pm 7.9$	$\mathbf{95.5\raisebox{0.68887pt}{\scriptsize$ \pm 1.5 $}}$	70.6 $\pm 1.6$	81.0 $\pm 4.1$

Appendix

Table 12 : Normalized returns on D4RL locomotion tasks. Best results are in bold, and results within one standard deviation of the best are also bolded.

Task	VAN-Flow	QC	Task	VAN-Flow	QC
antmaze-teleport-navigate	57 $\rightarrow$ 96	40 $\rightarrow$ 46	scene-play	93 $\rightarrow$ 100	84 $\rightarrow$ 99
antmaze-large-navigate	95 $\rightarrow$ 99	49 $\rightarrow$ 98	puzzle-3x3-play	100 $\rightarrow$ 100	100 $\rightarrow$ 100
Long-horizon tasks			Noisy tasks
humanoidmaze-large-navigate	81 $\rightarrow$ 98	16 $\rightarrow$ 16	antmaze-large-explore	84 $\rightarrow$ 98	58 $\rightarrow$ 96
humanoidmaze-giant-navigate	92 $\rightarrow$ 98	12 $\rightarrow$ 67	puzzle-3x3-noisy	97 $\rightarrow$ 100	100 $\rightarrow$ 100
antmaze-giant-navigate	68 $\rightarrow$ 98	2 $\rightarrow$ 69

Appendix

Table 13 : Offline-to-online fine-tuning performance across diverse long-horizon and noisy control tasks. Scores are reported before online fine-tuning in gray and after fine-tuning in bold. The offline scores are those reported in Tables 1 and 2 .

Task	$\boldsymbol{\mathbb{E}[Z]}$	CVaR	CPW	Norm	Wang	Entropic	Sharpe	Sortino	Mean–Var.	VE
teleport	$38\!\pm\!7$	$54\!\pm\!16$	$37\!\pm\!5$	$36\!\pm\!9$	$40\!\pm\!10$	$44\!\pm\!16$	$25\!\pm\!11$	$30\!\pm\!8$	$46\!\pm\!8$	$\mathbf{71\!\pm\!7}$
explore	$84\!\pm\!2$	$2\!\pm\!1$	$87\!\pm\!4$	$84\!\pm\!8$	$86\!\pm\!2$	$86\!\pm\!4$	$64\!\pm\!27$	$70\!\pm\!8$	$85\!\pm\!4$	$\mathbf{93\!\pm\!3}$

Appendix

Table 14 : Comparison with representative risk-sensitive objectives. Scores denote success rates averaged over five random seeds.

Figure 13 : Additional sensitivity results on the number of

n

Figure 14 : Additional sensitivity results on the number of rejection samples

Figure 15 : Additional sensitivity results on the number of flow steps.

Task	$\delta=0$	$1$	$2$ (default)	$3$	$4$	$5$	$10$
antmaze-giant-navigate	44.0 $\pm 11.8$	53.6 $\pm 6.9$	$\mathbf{58.0\raisebox{0.68887pt}{\scriptsize$ \pm 4.0 $}}$	30.0 $\pm 3.3$	29.5 $\pm 6.7$	31.6 $\pm 4.1$	30.4 $\pm 5.9$
antmaze-large-explore	82.0 $\pm 8.6$	89.5 $\pm 3.4$	$\mathbf{93.6\raisebox{0.68887pt}{\scriptsize$ \pm 3.0 $}}$	90.8 $\pm 4.5$	65.2 $\pm 33.0$	0.0 $\pm 0.0$	0.0 $\pm 0.0$
humanoidmaze-large-navigate	78.0 $\pm 3.8$	82.0 $\pm 7.7$	$\mathbf{87.4\raisebox{0.68887pt}{\scriptsize$ \pm 5.8 $}}$	83.2 $\pm 3.2$	81.6 $\pm 6.1$	82.6 $\pm 2.7$	84.8 $\pm 2.7$

Appendix

Table 15 : Success rate for different values of the aversion coefficient

\delta

(task1 of each environment, five seeds).

Task	$-100$	$-170$	$-200$ (rule)	$-230$	$-300$
antmaze-large-navigate	0.0 $\pm 0.0$	94.4 $\pm 3.7$	94.6 $\pm 4.2$	93.2 $\pm 4.7$	96.0 $\pm 1.4$
puzzle-3x3-noisy	0.0 $\pm 0.0$	81.1 $\pm 22.5$	98.6 $\pm 2.8$	99.6 $\pm 0.8$	94.2 $\pm 11.0$

Appendix

Table 16 : Success rate for different lower bounds

V_{\min}

of the categorical support (

V_{\max}=0

I=101

, five seeds).

Task	$I=51$	$I=101$ (default)	$I=201$	$I=501$
antmaze-large-navigate	91.6 $\pm 4.6$ (1.28h)	94.6 $\pm 4.2$ (1.26h)	92.4 $\pm 3.9$ (1.23h)	95.6 $\pm 2.3$ (1.31h)
humanoidmaze-giant-navigate	79.0 $\pm 9.5$ (2.28h)	85.4 $\pm 5.0$ (2.32h)	83.5 $\pm 6.5$ (2.24h)	79.5 $\pm 9.2$ (2.26h)
puzzle-3x3-noisy	98.4 $\pm 2.3$ (1.22h)	98.6 $\pm 2.8$ (1.19h)	98.8 $\pm 1.6$ (1.19h)	97.2 $\pm 3.0$ (1.22h)
scene-play	20.5 $\pm 18.6$ (1.55h)	92.4 $\pm 4.5$ (1.57h)	86.8 $\pm 5.9$ (1.58h)	86.8 $\pm 4.3$ (1.57h)

Appendix

Table 17 : Success rate and training wall-clock time (hours) for different numbers of atoms

I

(five seeds).

Explore similar work

May 3, 2026cs.LG

Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning

We propose Flow-Anchored Noise-conditioned Q-Learning (FAN), a highly efficient and high-performing offline reinforcement learning (RL) algorithm. Recent work has shown that expressive flow policies and distributional critics improve offline RL performance, but at a high computational cost. Specifically, flow policies require iterative sampling to produce a single action, and distributional critics require computation over multiple samples (e.g., quantiles) to estimate value. To address these inefficiencies while maintaining high performance, we introduce FAN. Our method employs a behavior regularization technique that uses a single flow policy iteration and requires a single Gaussian noise sample for distributional critics. Our theoretical analysis of convergence and performance bounds demonstrates that these simplifications not only improve efficiency but also lead to superior task performance. Experiments on robotic manipulation and locomotion tasks demonstrate that FAN achieves state-of-the-art performance while significantly reducing both training and inference runtimes. We release our code at https://github.com/brianlsy98/FAN.

Sungyoung Lee, Dohyeong Kim, Eshan Balachandar +2

The University of Texas at Austin, Austin, TX, USA · Independent Researcher, Seoul, South Korea.

Flow Policies Model-Based Reinforcement Learning Efficient Planning

Ask PDF

Jun 16, 2026cs.LG

Reversal Q-Learning

Iterative generative modeling techniques, such as flow matching, provide powerful tools to model complex behaviors for effective offline reinforcement learning (RL). In this work, we propose a new off-policy RL algorithm that trains a flow policy based on prior data. Our idea starts from the "expanded" Markov decision process (MDP) framework, which treats individual flow refinement steps as separate actions in an MDP. To enable off-policy RL within this framework, we apply two techniques: we generate virtual on-policy trajectories (by "reversing" flows) to make this framework compatible with prior data, and we apply a bias-and-variance reduction technique to mitigate the curse of horizon in off-policy RL. We call the resulting algorithm Reversal Q-learning (RQL). RQL has several advantages over previous flow-based RL methods: it does not suffer from backpropagation through time, makes better use of the learned value function, and directly trains the full, expressive flow policy. Through our experiments on 50 challenging simulated robotic tasks, we show that RQL leads to the best average offline RL performance compared to state-of-the-art flow-based offline RL algorithms.

Aditya Oberai, Seohong Park, Sergey Levine

University of California, Berkeley.

Flow Policies Model-Based Reinforcement Learning Q-Learning On-Policy

Ask PDF

May 12, 2026cs.LG

Discrete Flow Matching for Offline-to-Online Reinforcement Learning

Many reinforcement learning (RL) tasks have discrete action spaces, but most generative policy methods based on diffusion and flow matching are designed for continuous control. Meanwhile, generative policies usually rely heavily on offline datasets and offline-to-online RL is itself challenging, as the policy must improve from new interaction without losing useful behavior learned from static data. To address those challenges, we introduce DRIFT, an online fine-tuning method that updates an offline pretrained continuous-time Markov chain (CTMC) policy with an advantage-weighted discrete flow matching loss. To preserve useful pretrained knowledge, we add a path-space penalty that regularizes the full CTMC trajectory distribution, rather than only the final action distribution. For large discrete action spaces, we introduce a candidate-set approximation that updates the actor over a small subset of actions sampled from reference-policy rollouts and uniform exploration. Our theoretical analysis shows that the candidate-set error is controlled by missing target probability mass, and the induced CTMC generator error decreases as the candidate set covers more high-probability actions. Experiments on prevailing discrete action RL task show that our method provides stable offline-to-online improvement across all tasks, achieving the highest average score on Jericho with a simple GRU encoder while outperforming methods that use pretrained language models. Controlled experiments further confirm that the path-space penalty remains bounded during fine-tuning and that the CTMC generator adapts to shifted rewards faster than deterministic baselines. The candidate-set mechanism is supported by a stability analysis showing that the generator error decreases exponentially with candidate coverage.

Fairoz Nower Khan, Nabuat Zaman Nahim, Peizhong Ju

Department of Computer Science, University of Kentucky

Flow Policies Offline Reinforcement Learning Conditional Flow Matching Markov

Ask PDF