Variational Consequence-Driven Offline Reinforcement Learning

Wen Jiang; Ke Jiang; Xiaoyang Tan

cs.LGOct 28, 2026

Variational Consequence-Driven Offline Reinforcement Learning

Authors: Wen Jiang, Ke Jiang, Xiaoyang Tan

Organizations: College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence · Graduate School of Information Science and Technology, The University of Osaka

Abstract

Offline reinforcement learning (RL) typically mitigates distribution shift by imposing divergence constraints on the action distributions of the learned and behavior policies. However, this paradigm cannot ensure the underlying dynamic consistency structure: even minor action deviations can lead to drastically different transitions, inadvertently driving the agent away from the offline dataset. To address this, we explore policy constraints through their induced state-consequence distributions rather than relying on pure action-distribution matching. Nevertheless, directly enforcing such consequence-aware constraints is computationally intractable without access to the true underlying dynamics model. To bypass this issue, we propose Structural Behavior Regularization (StBR), which replaces the intractable objective with a closed-form latent divergence. Structuring this latent space to reflect transition dynamics renders the latent divergence a theoretically bounded surrogate for the true consequence-distribution divergence. By regularizing the learned policy with this latent divergence, StBR adaptively tightens the behavior constraint in dynamics-sensitive regions, effectively preventing deviations outside the offline dataset. Empirically, we demonstrate that StBR achieves superior performance across diverse offline RL benchmarks.

Figures & tables

Figure 1: Conceptual illustration of divergence-based behavior regularization. Purely action distribution matching (left) neglects dynamic consistency, whereas ideal consequence-aware constraints (right) are computationally intractable without access to the true dynamics. Our StBR (middle) employs a Gaussian latent space as a tractable, closed-form surrogate of consequence-aware constraints.

Figure 2: The architecture of the proposed StBR framework.The blue flow represents the learning of latent representations (Eq. ( 8 )) through the proposed State Belief Gaussian Fitting. The red flow highlights the application of the surrogate latent divergence for behavior regularization in Eq. ( 12 ).

Task Name

CQL

TD3BC

IQL

PBRL

POR

SDC

SPOT

FQL

ENSP

StBR (Ours)

halfcheetah

medium

41.2±0.4

46.9±0.4

48.2±0.1

47.1±0.1

57.9±1.5

48.8±0.5

45.9±0.5

58.4 ±1.0

54.4±2.2

55.4±0.6

60.7 ±0.7

medium-replay

32.9±3.4

45.3±0.3

44.6±0.1

44.3±0.3

45.1±8.0

43.5±0.9

45.1±0.5

52.2±1.2

53.2 ±0.6

43.4±1.5

54.4 ±1.0

medium-expert

50.4±4.0

95.0±1.4

91.7±1.9

89.2±1.2

92.3±1.1

94.7±2.2

76.3±5.2

86.9±4.3

95.3 ±1.3

88.1±4.9

97.8 ±0.2

hopper

medium

46.1±1.8

61.9±6.4

59.0±0.7

63.6±2.9

75.3±31.2

78.6±7.2

64.7±3.5

86.0 ±8.7

77.1±10.3

80.4±14.7

100.9 ±1.0

medium-replay

22.4±5.9

86.3±7.3

65.4±14.0

95.4±4.9

100.6 ±1.0

98.9±2.1

94.8±6.5

100.2±1.0

79.5±4.5

91.0±12.9

100.8 ±1.9

medium-expert

62.4±13.9

96.9±15.1

101.6±2.0

101.6±7.4

110.8 ±0.8

90.0±12.1

99.9±8.5

99.3±7.1

103.5±6.7

65.5±12.5

111.6 ±0.9

Table 1: The mean and standard deviation of normalized scores after 1M steps of offline RL training, averaged over 5 seeds and the final 10 evaluations (the final 4 for AntMaze). The highest score in each task is bolded, and the second-best is underlined.

Figure 3: Hyperparameter sensitivity analysis. (a-b) When evaluating the performance at varying levels of constraint strength (

\lambda

) on two representative tasks, StBR takes shape of the "upper envelope" of all methods, showing that StBR is highly robust to parameter choices, making it significantly easier to achieve superior performance during practical deployment. (c-d) Sensitivity analysis of SBGF-related hyperparameters regarding

\beta

and latent space dimension.

Figure 4: Theoretical validation experiments. (a-b) Demonstrating that a larger $\beta$ should be chosen in high-dimensional tasks and highly stochastic environments to maintain superior performance. (c) Correlation analysis between dynamic sensitivity and different metrics: the proposed consequence-aware divergence and the action-distribution measure. Results validate that latent divergence (left, Spearman $r=0.54$ ) exhibits a significantly stronger correlation with dynamics sensitivity than the VAE-based density measure (right, $r=0.16$ ).

Figure 5: Percent performance difference of StBR variants compared to the original algorithm.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

	Hyperparameter	Value
StBR	Optimizer	Adam
	Activation function	ReLU
	Critic learning rate	$3\times 10^{-4}$
	Actor learning rate	$3\times 10^{-4}$
	Batch size	256
	Discount factor	0.99

Appendix

Table 2: Hyperparameters in StBR

Task Name		$\lambda$	$\beta$	$\lambda_{ENSP}$
halfcheetah	medium	0.03	0.001	0.05
	medium-replay	0.03	0.001	0.05
	medium-expert	0.05	0.001	0.2
hopper	medium	0.03	0.001	0.1
	medium-replay	0.05	0.001	0.05
	medium-expert	0.05	0.001	0.2

Appendix

Table 3: Detailed configurations of

\lambda

and

\beta

across different tasks.

Figure 6: The results of StBR and StBR+BC on the MuJoCo task. The shaded area indicates the minimum and maximum score across 5 seeds, with the solid line depicting their mean.

Figure 7: The results of StBR and StBR+BC on the AntMaze and Adroit task. The shaded area indicates the minimum and maximum score across 5 seeds, with the solid line depicting their mean.

Figure 8: The figure tracks the latent divergence across gradient steps for OOD actions and learned policy’s actions. Notably, the latent divergence of the constrained policy is significantly lower than that of the OOD actions, demonstrating that our latent divergence effectively separates in-distribution (ID) from OOD actions.

Limited Valuable Dataset	CQL	SDC	FQL	StBR
Walker2d-r0.5-e0.5	88.3±4.9	88.8±3.9	108.9±4.5	113.9 ±3.1
Walker2d-r0.6-e0.4	81.5±6.3	65.9±5.0	91.7±6.6	106.1 ±2.4
Walker2d-r0.7-e0.3	48.3±5.8	85.5±4.7	88.7±9.1	99.9 ±8.1
Walker2d-r0.8-e0.2	50.5±7.2	49.8±5.6	70.4±11.5	87.7 ±5.5
Walker2d-r0.9-e0.1	17.6±3.0	26.8±2.1	66.7±23.6	73.4 ±22.9

Appendix

Table 4: Normalized scores on mixed walker2d datasets with varying proportions of random (r) and expert (e) transitions. Results are averaged over four seeds. StBR demonstrates significant robustness against limited valuable data dilution.

Dataset-size(%)	100	50	25	10	5	1
StBR(Ours)	100.9±1.0	100.4±1.2	99.5±1.5	97.2±4.1	78.2±2.7	65.6±14.5
SPOT	86.0±8.7	83.2±14.0	83.6±6.2	72.9±4.5	44.1±15.0	38.8±19.0

Appendix

Table 5: Normalized scores with varying dataset-size. Results are averaged over 5 seeds.

Task Name	$\sigma_{oos}=0$	$\sigma_{oos}=0.001$	$\sigma_{oos}=0.01$	$\sigma_{oos}=0.1$
AntMaze-medium-play	94.2±1.9	97.1 ±2.4	92.6±3.1	44.4±7.4
AntMaze-medium-diverse	90.6±3.6	95.1 ±1.8	91.0±4.5	47.4±4.8
AntMaze-large-play	84.2±2.7	88.5 ±2.7	84.5±2.2	19.6±6.2
AntMaze-large-diverse	85.8±5.1	86.2 ±3.3	83.1±6.1	12.8±7.4
Total	354.8	354.8 $\rightarrow$ 366.9	354.8 $\rightarrow$ 351.2	354.8 $\rightarrow$ 124.2

Appendix

Table 6: The mean and standard deviation of averaged normalized scores after 1M steps of OOS state recovery training, averaged over 5 seeds and the final 4 evaluations. The highest scores in each task are bolded.

	Task Name	MOPO	RAMBO	StBR+MB	StBR(ours)
halfcheetah	medium	42.3±1.6	71.0 ±2.9	41.4±2.8	60.7 ±0.7
	medium-replay	53.1±2.0	67.0 ±1.5	55.8 ±3.1	54.4±1.0
	medium-expert	63.3±38.0	79.3±2.9	90.6 ±5.9	97.8 ±0.2
hopper	medium	28.0±12.4	91.2 ±16.3	80.3±7.3	100.9 ±1.0
	medium-replay	67.5±24.7	97.6 ±3.4	90.6±2.6	100.8 ±1.9
	medium-expert	23.7±6.0	89.5 ±11.1	77.8±6.5	111.6 ±0.9

Appendix

Table 7: The mean and standard deviation of averaged normalized scores after 3M steps of model-based offline RL training, averaged over 5 seeds and the final 10 evaluations. The top-2 highest scores in each task are bolded.

	BC	CQL	TD3BC	IQL	PBRL	POR	SDC	SPOT	FQL	StBR(ours)
Runtime (ms/step)	2.0	24.2	4.7	6.5	26.7	8.8	25.0	6.7	6.6	6.8
Wall time (minutes)	33	403	78	108	444	147	417	111	110	114
Inference time (ms)	-	-	476	-	-	909	-	-	492	476

Appendix

Table 8: The runtime of StBR on hopper-medium-v2 task using a GeForce RTX 2080Ti GPU and Intel Xeon Gold 5218 CPU.

AntMaze	IQL	Cal-QL	StBR (Ours)
umaze	89.6 $\rightarrow$ 96.2	76.8 $\rightarrow$ 99.8	99.2 $\rightarrow 99.6_{\text{±}0.4}$
umaze-diverse	65.6 $\rightarrow$ 87.2	32.0 $\rightarrow$ 98.5	95.6 $\rightarrow\textbf{99.4}_{\text{±}0.7}$
medium-play	76.4 $\rightarrow$ 94.0	71.8 $\rightarrow$ 98.8	93.6 $\rightarrow\textbf{99.0}_{\text{±}0.8}$
medium-diverse	72.8 $\rightarrow$ 92.0	62.0 $\rightarrow$ 98.3	89.8 $\rightarrow\textbf{99.1}_{\text{±}0.7}$
large-play	42.0 $\rightarrow$ 51.0	31.8 $\rightarrow$ 97.3	78.0 $\rightarrow 96.9_{\text{±}1.1}$
large-diverse	46.0 $\rightarrow$ 58.4	44.0 $\rightarrow$ 91.5	77.8 $\rightarrow\textbf{96.5}_{\text{±}1.2}$

Appendix

Table 9: Online fine-tuning results on AntMaze tasks, showing initial performance after offline RL and performance after 1M steps of online RL. All results are averaged over 5 seeds.

Figure 9: Online fine-tuning training curves on AntMaze tasks, showing initial performance after offline RL and performance after 1M steps of online RL. (5 seeds)

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 304 papers

Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation
Jul 21, 2026Li-Rong Zhou, Qin-Wen Luo, Sheng-Jun HuangReinforcement Learning Offline RL
Nanjing University of Aeronautics and Astronautics, Nanjing, China
Latent Policy Steering through One-Step Flow Policies
Mar 5, 2026Hokyun Im, Andrey Kolobov, Jianlong Fu +1RL for Robotics Latent Action Learning
Department of Artificial Intelligence, Yonsei University · Microsoft Research
Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Belief
May 30, 2026Hongqiang Lin, Pengfei Wang, Nenggan ZhengReinforcement Learning Bayesian RL
Zhejiang University, Hangzhou, China.

Jul 21, 2026J/K move · Enter open · S save

Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation

Li-Rong Zhou, Qin-Wen Luo, Sheng-Jun Huang

Nanjing University of Aeronautics and Astronautics, Nanjing, China

Reinforcement Learning Offline RL Uncertainty Estimation in RL Preference-Based RL

Open paper AskPDF arXiv

Offline reinforcement learning (RL) aims to learn an effective policy from a static dataset, but its performance is fundamentally limited by dataset coverage. Action preference queries leverage expert feedback without additional environment interaction, enabling policy improvement during offline training. However, existing methods still face two key challenges: selecting informative preference queries and effectively exploiting the collected feedback. Current approaches typically rely only on the distance between policy actions and dataset actions for query selection, while enforcing fixed constraints that keep the policy close to queried preferences. Such strategies often lead to unstable policy updates and integrate poorly with value regularization. To address these limitations, we propose Conservative Query and Adaptive Regularization under Uncertainty Estimation, a lightweight framework that jointly improves preference querying and preference exploitation. Specifically, we employ a Morse network to estimate the uncertainty of policy actions with respect to the offline dataset. Based on this uncertainty, we introduce a conservative query strategy that selectively queries actions near the dataset to preserve Bellman-update stability, together with an uncertainty-aware adaptive regularization scheme that dynamically adjusts data-level constraints during policy optimization. We integrate our framework with CQL and evaluate it extensively on the D4RL benchmark. Experimental results demonstrate superior or competitive performance across a wide range of tasks.

Abstract

Offline RL

Explore similar work

Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation

Latent Policy Steering through One-Step Flow Policies

Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Belief