Variance-Averse $n$-Step Offline Reinforcement Learning for Sparse Long-Horizon Environments

Guhyeon Kang; Minhae Kwon

cs.LGOct 6, 2026

Variance-Averse $n$ -Step Offline Reinforcement Learning for Sparse Long-Horizon Environments

Authors: Guhyeon Kang, Minhae Kwon

Organizations: Department of Electrical and Computer Engineering Sungkyunkwan University, Republic of Korea

Abstract

Generative actors are transforming offline reinforcement learning (RL) by enabling expressive policy classes that model complex action distributions. However, this expressiveness also exposes a key challenge in heterogeneous datasets: generative policies can reproduce unreliable action modes whose return distributions exhibit high variance, occasionally yielding high returns by chance but lacking consistency. Consequently, maximizing the expected $Q$ -value alone is insufficient for identifying reliable actions. We propose VAN-Flow (Variance-Averse $n$ -step Flow), a framework that promotes reliable actions in generative offline RL. VAN-Flow combines (i) a categorical distributional critic, (ii) a variance-averse expectation operator that smoothly reweights atom probabilities to favor actions with both high returns and low dispersion, and (iii) a flow-matching generative actor guided via rejection sampling. Unlike CVaR or mean-variance objectives, the operator redistributes probability mass over the categorical return distribution without hard truncation or auxiliary penalty terms. Across more than 40 tasks from D4RL and OGBench, VAN-Flow consistently outperforms strong baselines, with the largest gains in long-horizon and high-variance regimes where reliable action selection becomes critical.

Figures & tables

Figure 1 : Success rate on antmaze-large under two dataset variance regimes: (a) navigate (low-variance) and (b) explore (high-variance). In (b), the shaded bar denotes the low-variance reference, and the downward arrow indicates the performance drop.

Figure 2 : Example of variance-averse return with varying

\delta

and return variance.

	Gaussian-based				Flow-based					Proposed
Task	IQL	HIQL	ReBRAC	ReBRAC- $n$	FQL	BFN	FQL- $n$	BFN- $n$	QC	VAN-Flow
Standard tasks
antmaze-large-navigate	53	91	81	82	79	71	91	81	49	95
antmaze-teleport-navigate	23	42	7	35	26	40	22	41	40	57
scene-play	0	38	11	40	57	85	18	57	84	93
puzzle-3x3-play	0	12	55	10	100	98	98	58	100	100

Table 1 : Performance comparison on OGBench. Each entry averages five tasks per environment. Best results are in bold.

	Pure Offline RL			$n$ -step					Dist.	Dist.+ $n$	Proposed
Task	IQL	ReBRAC	FQL	Retrace( $\lambda$ )	PQL	LEQ	QC	TD3BC+MS	PA-RL	D4PG	VAN-Flow
umaze	77.0 $\pm 5.5$	97.7 $\pm 1.5$	96.0 $\pm 2.0$	96.0 $\pm 3.0$	94.0 $\pm 3.0$	94.4 $\pm 6.3$	95.0 $\pm 1.4$	–	–	87.6 $\pm 4.1$	98.0 $\pm 2.8$
umaze-diverse	54.2 $\pm 5.5$	83.5 $\pm 7.0$	89.0 $\pm 2.0$	84.0 $\pm 10.0$	63.0 $\pm 6.0$	71.0 $\pm 12.3$	78.0 $\pm 8.5$	–	–	61.3 $\pm 9.4$	95.6 $\pm 2.6$
medium-play	65.7 $\pm 11.7$	89.5 $\pm 3.3$	78.0 $\pm 7.0$	79.0 $\pm 10.0$	81.0 $\pm 6.0$	58.8 $\pm 33.0$	72.0 $\pm 2.4$	70.0	88.0	89.1 $\pm 3.0$	90.7 $\pm 3.1$
medium-diverse	73.7 $\pm 5.4$	83.5 $\pm 8.2$	71.0 $\pm 13.0$	53.0 $\pm 17.0$	60.0 $\pm 33.0$	46.2 $\pm 23.2$	64.0 $\pm 2.8$	66.0	88.0	81.3 $\pm 4.1$	89.3 $\pm 5.0$
large-play	42.0 $\pm 4.5$	52.2 $\pm 29.0$	84.0 $\pm 7.0$	12.0 $\pm 26.0$	33.0 $\pm 10.0$	58.6 $\pm 9.1$	80.0 $\pm 2.8$	22.0	87.0	71.3 $\pm 4.1$	86.5 $\pm 5.7$

Table 2 : Offline performance on D4RL AntMaze tasks. Scores are normalized following the D4RL protocol. Entries marked “–” indicate results not reported in prior work. Best results are in bold.

Figure 3 : Effect of flow-based policies and categorical critics on humanoidmaze-giant-navigate .

Task	Metric	$\mathbb{E}[Z]$	CVaR	Entropic	Mean–Var	$\mathcal{E}(Z)$
humanoidmaze -large-navigate	$\mathrm{Var}[Z]\!\downarrow$	$1.00$	$0.95$	$0.97$	$0.94$	$\mathbf{0.93}$
humanoidmaze -large-navigate	$\mathrm{Success}\!\uparrow$	$77$	$83$	$79$	$80$	$\mathbf{87}$
antmaze -teleport-navigate	$\mathrm{Var}[Z]\!\downarrow$	$1.00$	$1.08$	$0.93$	$0.90$	$\mathbf{0.62}$
antmaze -teleport-navigate	$\mathrm{Success}\!\uparrow$	$38$	$54$	$44$	$46$	$\mathbf{71}$
antmaze-large-explore	$\mathrm{Var}[Z]\!\downarrow$	$1.00$	$0.87$	$0.72$	$0.77$	$\mathbf{0.66}$
antmaze-large-explore	$\mathrm{Success}\!\uparrow$	$84$	$2$	$86$	$85$	$\mathbf{93}$

Table 3 : Ablation comparing return aggregation rules.

Figure 4 : Effect of

Q

guidance on policy performance across multiple tasks. Bars report mean success rate and time efficiency.

Figure 5 : Sensitivity analysis with respect to key algorithmic parameters.

Method	Complexity	Step (ms)	Wall (h)
FQL	$\mathcal{O}(UKd^{2})$	1.1	0.32
BFN	$\mathcal{O}(UMKd^{2})$	2.1	0.60
QC	$\mathcal{O}(UMKd^{2})$	3.0	0.83
VAN-Flow	$\mathcal{O}(UMKd^{2})$	1.9	0.54

Table 4 : Runtime comparison on humanoidmaze-large-navigate .

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Description
$\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma)$	Markov decision process
$\pi_{\theta}(\cdot\mid s)$	Flow-based policy (actor) with parameters $\theta$
$v^{\pi}_{\theta}(\tau,s,x)$	Flow-matching velocity field (policy parameterization)
$Q_{\psi}(s,a)$	Scalar critic (used in standard 1-step formulations)
$Z_{\psi}(s,a)$	Categorical return distribution critic used in VAN-Flow
$Z(s,a)$	Return random variable (categorical distribution) for $(s,a)$

Appendix

Table 5 : Notation table.

Measurement	Value
Median realized gap $\|\mathcal{E}-\mathcal{E}^{\mathrm{sp}}\|$ (fraction of the value range $R$ )	$0.37\%$
States in which $\mathcal{E}$ and $\mathcal{E}^{\mathrm{sp}}$ select the same candidate	$96.9\%$
Kendall’s $\tau$ between the two candidate rankings	$0.99$

Appendix

Table 6 : Discretization gap between the implemented operator

\mathcal{E}

and its spectral form

\mathcal{E}^{\mathrm{sp}}

on trained critics.

Hyperparameter	Value
Optimization
Learning rate (critic) $\alpha_{Z}$	$3\times 10^{-4}$
Learning rate (actor) $\alpha_{\pi}$	$3\times 10^{-4}$
Optimizer	Adam [ 54 ]
Gradient steps	$10^{6}$ (OGBench), $5\times 10^{5}$ (D4RL)
Minibatch size	256

Appendix

Table 7 : Hyperparameters for VAN-Flow.

Variant	Dist	VE	Flow	antmaze-teleport (8d)	antmaze-giant (8d)	humanoidmaze-large (21d)
VAN-Flow (Ours)	✓	✓	✓	$\mathbf{71\!\pm\!7}$	$\mathbf{58\!\pm\!4}$	$\mathbf{87\!\pm\!6}$
w/o VE	✓	✗	✓	$38\!\pm\!7$	$36\!\pm\!6$	$77\!\pm\!4$
w/o Flow	✓	✓	✗	$42\!\pm\!7$	$38\!\pm\!13$	$0\!\pm\!0$
w/o VE & Flow	✓	✗	✗	$36\!\pm\!9$	$32\!\pm\!5$	$0\!\pm\!0$

Appendix

Table 8 : Component-wise ablation across action-dimension regimes. All variants use

n

-step returns. Scores denote success rate (%), averaged over five random seeds.

	VAN-DDPM		VAN-Flow
Steps	Perf.	Time (ms)	Perf.	Time (ms)
1	$0\raisebox{0.68887pt}{\scriptsize$ \pm 0 $}$	2.0	$0\raisebox{0.68887pt}{\scriptsize$ \pm 0 $}$	1.6
3	$0\raisebox{0.68887pt}{\scriptsize$ \pm 0 $}$	3.8	$\mathbf{96\raisebox{0.68887pt}{\scriptsize$ \pm 3 $}}$	3.3
5	$0\raisebox{0.68887pt}{\scriptsize$ \pm 0 $}$	5.7	$96\raisebox{0.68887pt}{\scriptsize$ \pm 2 $}$	5.0
10	$57\raisebox{0.68887pt}{\scriptsize$ \pm 17 $}$	8.4	$87\raisebox{0.68887pt}{\scriptsize$ \pm 6 $}$	7.2
20	$\mathbf{94\raisebox{0.68887pt}{\scriptsize$ \pm 4 $}}$	11.9	$76\raisebox{0.68887pt}{\scriptsize$ \pm 6 $}$	10.2

Appendix

Table 9 : Comparison between flow matching (VAN-Flow) and diffusion (VAN-DDPM) policies on humanoidmaze-large-navigate . Performance denotes success rate (%); Time denotes cumulative inference time per action (ms). Bold entries indicate the first step count achieving

>

90% success rate.

Figure 8 : Learning curve on the antmaze task in OGBench

Figure 9 : Learning curve on the humanoidmaze task in OGBench

Figure 10 : Learning curve on the puzzle task in OGBench

Figure 11 : Learning curve on the scene task in OGBench

Figure 12 : Learning curve on the antmaze task in D4RL

Env	Task	Gaussian-based				Flow-based					Proposed
Env	Task	IQL	ReBRAC	ReBRAC- $n$	HIQL	FQL	BFN	FQL- $n$	BFN- $n$	QC	VAN-Flow
antmaze-large-navigate	task1	48 $\pm 9$	91 $\pm 10$	64.0 $\pm 25.8$	93.0 $\pm 3.0$	80 $\pm 8$	93.0 $\pm 1.4$	94.2 $\pm 2.8$	83.1 $\pm 1.4$	49.2 $\pm 9.9$	94.6 $\pm 4.2$
	task2	42 $\pm 6$	88 $\pm 4$	78.3 $\pm 5.9$	78.0 $\pm 9.0$	57 $\pm 10$	90.2 $\pm 2.8$	80.0 $\pm 3.4$	88.0 $\pm 11.3$	43.4 $\pm 2.4$	89.4 $\pm 7.0$
	task3	72 $\pm 7$	51 $\pm 18$	92.8 $\pm 2.3$	96.0 $\pm 2.0$	93 $\pm 3$	89.1 $\pm 7.0$	97.5 $\pm 1.4$	74.5 $\pm 5.6$	71.8 $\pm 2.8$	98.0 $\pm 2.0$
	task4	51 $\pm 9$	84 $\pm 7$	90.0 $\pm 2.8$	94.0 $\pm 2.0$	80 $\pm 4$	9.0 $\pm 1.4$	89.1 $\pm 7.0$	73.1 $\pm 1.4$	22.0 $\pm 2.8$	95.4 $\pm 3.1$
	task5	54 $\pm 22$	90 $\pm 2$	90.8 $\pm 2.7$	94.0 $\pm 3.0$	83 $\pm 4$	72.0 $\pm 5.6$	94.7 $\pm 5.6$	85.1 $\pm 1.4$	61.6 $\pm 1.4$	98.0 $\pm 2.0$

Appendix

Table 10 : Task-wise performance comparison in OGBench (singletask setting).

Task	Gaussian-based			Diffusion Policies			Flow-based				Proposed
Task	BC	IQL	ReBRAC	IDQL	SRPO	CAC	FAWAC	FBRAC	IFQL	FQL	Ours
pen-human-v1	71	78	103	76	69	64	67	77	71	53	82.3 $\pm 3.5$
pen-cloned-v1	52	83	103	64	61	56	62	67	80	74	107.5 $\pm 14.9$
pen-expert-v1	110	128	152	140	134	103	118	119	139	142	146.6 $\pm 5.5$
door-human-v1	2	3	0	6	3	5	2	4	7	0	2.7 $\pm 1.3$
door-cloned-v1	0	3	0	0	0	1	0	0	2	2	3.0 $\pm 2.8$

Appendix

Table 11 : Performance comparison in D4RL Adroit

Task	IQL	ReBRAC	Diffusion-QL	Diffuser	VAN-Flow
halfcheetah-medium	47.4	$\mathbf{65.6\raisebox{0.68887pt}{\scriptsize$ \pm 1.0 $}}$	51.1 $\pm 0.5$	42.8 $\pm 0.3$	63.9 $\pm 2.1$
halfcheetah-medium-replay	44.2	51.0 $\pm 0.8$	47.8 $\pm 0.3$	37.7 $\pm 0.5$	$\mathbf{51.4\raisebox{0.68887pt}{\scriptsize$ \pm 0.3 $}}$
hopper-medium	66.3	$\mathbf{102.0\raisebox{0.68887pt}{\scriptsize$ \pm 1.0 $}}$	90.5 $\pm 4.6$	74.3 $\pm 1.4$	95.3 $\pm 2.5$
hopper-medium-replay	94.7	98.1 $\pm 5.3$	$\mathbf{101.3\raisebox{0.68887pt}{\scriptsize$ \pm 0.6 $}}$	93.6 $\pm 0.4$	100.3 $\pm 1.2$
walker2d-medium	78.3	82.5 $\pm 3.6$	87.0 $\pm 0.9$	79.6 $\pm 0.6$	$\mathbf{88.2\raisebox{0.68887pt}{\scriptsize$ \pm 1.8 $}}$
walker2d-medium-replay	73.9	77.3 $\pm 7.9$	$\mathbf{95.5\raisebox{0.68887pt}{\scriptsize$ \pm 1.5 $}}$	70.6 $\pm 1.6$	81.0 $\pm 4.1$

Appendix

Table 12 : Normalized returns on D4RL locomotion tasks. Best results are in bold, and results within one standard deviation of the best are also bolded.

Task	VAN-Flow	QC	Task	VAN-Flow	QC
antmaze-teleport-navigate	57 $\rightarrow$ 96	40 $\rightarrow$ 46	scene-play	93 $\rightarrow$ 100	84 $\rightarrow$ 99
antmaze-large-navigate	95 $\rightarrow$ 99	49 $\rightarrow$ 98	puzzle-3x3-play	100 $\rightarrow$ 100	100 $\rightarrow$ 100
Long-horizon tasks			Noisy tasks
humanoidmaze-large-navigate	81 $\rightarrow$ 98	16 $\rightarrow$ 16	antmaze-large-explore	84 $\rightarrow$ 98	58 $\rightarrow$ 96
humanoidmaze-giant-navigate	92 $\rightarrow$ 98	12 $\rightarrow$ 67	puzzle-3x3-noisy	97 $\rightarrow$ 100	100 $\rightarrow$ 100
antmaze-giant-navigate	68 $\rightarrow$ 98	2 $\rightarrow$ 69

Appendix

Table 13 : Offline-to-online fine-tuning performance across diverse long-horizon and noisy control tasks. Scores are reported before online fine-tuning in gray and after fine-tuning in bold. The offline scores are those reported in Tables 1 and 2 .

Task	$\boldsymbol{\mathbb{E}[Z]}$	CVaR	CPW	Norm	Wang	Entropic	Sharpe	Sortino	Mean–Var.	VE
teleport	$38\!\pm\!7$	$54\!\pm\!16$	$37\!\pm\!5$	$36\!\pm\!9$	$40\!\pm\!10$	$44\!\pm\!16$	$25\!\pm\!11$	$30\!\pm\!8$	$46\!\pm\!8$	$\mathbf{71\!\pm\!7}$
explore	$84\!\pm\!2$	$2\!\pm\!1$	$87\!\pm\!4$	$84\!\pm\!8$	$86\!\pm\!2$	$86\!\pm\!4$	$64\!\pm\!27$	$70\!\pm\!8$	$85\!\pm\!4$	$\mathbf{93\!\pm\!3}$

Appendix

Table 14 : Comparison with representative risk-sensitive objectives. Scores denote success rates averaged over five random seeds.

Figure 13 : Additional sensitivity results on the number of

n

Figure 14 : Additional sensitivity results on the number of rejection samples

Figure 15 : Additional sensitivity results on the number of flow steps.

Task	$\delta=0$	$1$	$2$ (default)	$3$	$4$	$5$	$10$
antmaze-giant-navigate	44.0 $\pm 11.8$	53.6 $\pm 6.9$	$\mathbf{58.0\raisebox{0.68887pt}{\scriptsize$ \pm 4.0 $}}$	30.0 $\pm 3.3$	29.5 $\pm 6.7$	31.6 $\pm 4.1$	30.4 $\pm 5.9$
antmaze-large-explore	82.0 $\pm 8.6$	89.5 $\pm 3.4$	$\mathbf{93.6\raisebox{0.68887pt}{\scriptsize$ \pm 3.0 $}}$	90.8 $\pm 4.5$	65.2 $\pm 33.0$	0.0 $\pm 0.0$	0.0 $\pm 0.0$
humanoidmaze-large-navigate	78.0 $\pm 3.8$	82.0 $\pm 7.7$	$\mathbf{87.4\raisebox{0.68887pt}{\scriptsize$ \pm 5.8 $}}$	83.2 $\pm 3.2$	81.6 $\pm 6.1$	82.6 $\pm 2.7$	84.8 $\pm 2.7$

Appendix

Table 15 : Success rate for different values of the aversion coefficient

\delta

(task1 of each environment, five seeds).

Task	$-100$	$-170$	$-200$ (rule)	$-230$	$-300$
antmaze-large-navigate	0.0 $\pm 0.0$	94.4 $\pm 3.7$	94.6 $\pm 4.2$	93.2 $\pm 4.7$	96.0 $\pm 1.4$
puzzle-3x3-noisy	0.0 $\pm 0.0$	81.1 $\pm 22.5$	98.6 $\pm 2.8$	99.6 $\pm 0.8$	94.2 $\pm 11.0$

Appendix

Table 16 : Success rate for different lower bounds

V_{\min}

of the categorical support (

V_{\max}=0

I=101

, five seeds).

Task	$I=51$	$I=101$ (default)	$I=201$	$I=501$
antmaze-large-navigate	91.6 $\pm 4.6$ (1.28h)	94.6 $\pm 4.2$ (1.26h)	92.4 $\pm 3.9$ (1.23h)	95.6 $\pm 2.3$ (1.31h)
humanoidmaze-giant-navigate	79.0 $\pm 9.5$ (2.28h)	85.4 $\pm 5.0$ (2.32h)	83.5 $\pm 6.5$ (2.24h)	79.5 $\pm 9.2$ (2.26h)
puzzle-3x3-noisy	98.4 $\pm 2.3$ (1.22h)	98.6 $\pm 2.8$ (1.19h)	98.8 $\pm 1.6$ (1.19h)	97.2 $\pm 3.0$ (1.22h)
scene-play	20.5 $\pm 18.6$ (1.55h)	92.4 $\pm 4.5$ (1.57h)	86.8 $\pm 5.9$ (1.58h)	86.8 $\pm 4.3$ (1.57h)

Appendix

Table 17 : Success rate and training wall-clock time (hours) for different numbers of atoms

I

(five seeds).

Explore similar work

Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning
May 3, 2026Sungyoung Lee, Dohyeong Kim, Eshan Balachandar +2Flow Policies Model-Based Reinforcement Learning
The University of Texas at Austin, Austin, TX, USA · Independent Researcher, Seoul, South Korea.
Reversal Q-Learning
Jun 16, 2026Aditya Oberai, Seohong Park, Sergey LevineFlow Policies Model-Based Reinforcement Learning
University of California, Berkeley.
Discrete Flow Matching for Offline-to-Online Reinforcement Learning
May 12, 2026Fairoz Nower Khan, Nabuat Zaman Nahim, Peizhong JuFlow Policies Offline Reinforcement Learning
Department of Computer Science, University of Kentucky

May 3, 2026, cs.LGJ/K move · Enter open · S save

Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning

Sungyoung Lee, Dohyeong Kim, Eshan Balachandar, Zelal Su Mustafaoglu+1

The University of Texas at Austin, Austin, TX, USA · Independent Researcher, Seoul, South Korea.

Flow Policies Model-Based Reinforcement Learning Efficient Planning

Open paper AskPDF arXiv

We propose Flow-Anchored Noise-conditioned Q-Learning (FAN), a highly efficient and high-performing offline reinforcement learning (RL) algorithm. Recent work has shown that expressive flow policies and distributional critics improve offline RL performance, but at a high computational cost. Specifically, flow policies require iterative sampling to produce a single action, and distributional critics require computation over multiple samples (e.g., quantiles) to estimate value. To address these inefficiencies while maintaining high performance, we introduce FAN. Our method employs a behavior regularization technique that uses a single flow policy iteration and requires a single Gaussian noise sample for distributional critics. Our theoretical analysis of convergence and performance bounds demonstrates that these simplifications not only improve efficiency but also lead to superior task performance. Experiments on robotic manipulation and locomotion tasks demonstrate that FAN achieves state-of-the-art performance while significantly reducing both training and inference runtimes. We release our code at https://github.com/brianlsy98/FAN.

Abstract

Explore similar work

Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning

Reversal Q-Learning

Discrete Flow Matching for Offline-to-Online Reinforcement Learning