OPRIDE: Offline Preference-based Reinforcement Learning via In-Dataset Exploration

Yiqin Yang; Hao Hu; Yihuan Mao; Jin Zhang; Chengjie Wu; Yuhua Jiang; Xu Yang; Runpeng Xie

cs.LGFeb 19, 2026

OPRIDE: Offline Preference-based Reinforcement Learning via In-Dataset Exploration

Authors: Yiqin Yang, Hao Hu, Yihuan Mao, Jin Zhang, Chengjie Wu, Yuhua Jiang, Xu Yang, Runpeng Xie,

Organizations: The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · Moonshot AI · Tsinghua University · Amazon ·

Abstract

Preference-based reinforcement learning (PbRL) can help avoid sophisticated reward designs and align better with human intentions, showing great promise in various real-world applications. However, obtaining human feedback for preferences can be expensive and time-consuming, which forms a strong barrier for PbRL. In this work, we address the problem of low query efficiency in offline PbRL, pinpointing two primary reasons: inefficient exploration and overoptimization of learned reward functions. In response to these challenges, we propose a novel algorithm, Offline PbRL via In-Dataset Exploration (OPRIDE), designed to enhance the query efficiency of offline PbRL. OPRIDE consists of two key features: a principled exploration strategy that maximizes the informativeness of the queries and a discount scheduling mechanism aimed at mitigating overoptimization of the learned reward functions. Through empirical evaluations, we demonstrate that OPRIDE significantly outperforms prior methods, achieving strong performance with notably fewer queries. Moreover, we provide theoretical guarantees of the algorithm's efficiency. Experimental results across various locomotion, manipulation, and navigation tasks underscore the efficacy and versatility of our approach.

Figures & tables

Figure 1: The procedure of OPRIDE consists of two phases. In the first offline phase, we select query based on exploration mechanism. The blue circles $\bullet$ and red triangles $\blacktriangle$ represent the value estimation $V_{\psi_{1}}$ and $V_{\psi_{2}}$ , respectively. In the second stage, we first learn an reward function based on the preference dataset and then annotate the reward-free dataset. Next, we adjust the discount factor to reduce the impact of noise in the reward learning.

\mathcal{D}_{\text{pref}}\leftarrow\mathcal{D}_{\text{pref}}\cup\{(\tau^{k,1},\tau^{k,2},o_{k})\}.

Algorithm 1 Offline Preference-Based Reinforcement Learning with In-Dataset Exploration

Domain	Task	OPRL	PT	PT+PDS	IDRL	OPRIDE
Metaworld	lever-pull	63.2 $\pm$ 10.4	49.2 $\pm$ 3.7	51.7 $\pm$ 0.1	33.1 $\pm$ 1.2	51.8 $\pm$ 1.6
	peg-insert-side	3.5 $\pm$ 1.8	16.8 $\pm$ 0.1	12.4 $\pm$ 1.4	67.4 $\pm$ 0.1	79.0 $\pm$ 0.2
	plate-slide	77.4 $\pm$ 1.6	4.9 $\pm$ 0.0	37.3 $\pm$ 2.3	79.6 $\pm$ 3.5	79.9 $\pm$ 4.6
	push	10.6 $\pm$ 1.5	16.7 $\pm$ 5.0	1.8 $\pm$ 0.4	30.7 $\pm$ 5.3	59.1 $\pm$ 5.4
	push-back	0.8 $\pm$ 0.0	1.1 $\pm$ 0.4	1.1 $\pm$ 0.1	14.0 $\pm$ 1.1	17.7 $\pm$ 2.0
	push-wall	7.4 $\pm$ 4.2	74.8 $\pm$ 14.4	3.4 $\pm$ 0.9	89.2 $\pm$ 3.2	102.2 $\pm$ 1.2

Table 1: Performance of offline RL algorithm on the reward-labeled dataset with different preference reward learning methods on the Meta-World tasks over five random seeds.

Domain	Task	OPRL	PT	PT+PDS	IDRL	OPRIDE
Antmaze	umaze	76.3 $\pm$ 3.7	77.5 $\pm$ 4.5	84.5 $\pm$ 8.5	85.5 $\pm$ 3.4	87.5 $\pm$ 5.6
	umaze-diverse	72.5 $\pm$ 3.4	68.0 $\pm$ 3.0	78.0 $\pm$ 6.0	69.1 $\pm$ 4.2	73.1 $\pm$ 2.4
	medium-play	0.0 $\pm$ 0.0	63.5 $\pm$ 0.5	72.5 $\pm$ 6.5	63.8 $\pm$ 4.1	62.2 $\pm$ 2.0
	medium-diverse	0.0 $\pm$ 0.0	63.5 $\pm$ 4.5	58.0 $\pm$ 4.0	65.7 $\pm$ 4.1	69.4 $\pm$ 5.2
	large-play	7.3 $\pm$ 0.9	6.5 $\pm$ 2.5	9.0 $\pm$ 8.0	18.7 $\pm$ 3.4	27.5 $\pm$ 12.5
	large-diverse	6.9 $\pm$ 2.4	23.5 $\pm$ 0.5	8.5 $\pm$ 2.5	14.3 $\pm$ 2.5	21.5 $\pm$ 1.5

Table 2: Performance of offline RL algorithm on the reward-labeled dataset with different preference reward learning methods on the Antmaze tasks over five random seeds.

Task	PT	PDS + Random Query	VDS + Random Query	VDS + Disagreement	OPRIDE (VDS+IDE)
bin-picking	31.9 $\pm$ 16.2	53.4 $\pm$ 19.0	71.9 $\pm$ 9.0	78.5 $\pm$ 17.8	93.3 $\pm$ 3.2
button-press-wall	58.8 $\pm$ 0.9	59.4 $\pm$ 0.9	77.2 $\pm$ 0.8	67.4 $\pm$ 5.4	77.7 $\pm$ 0.1
door-close	65.1 $\pm$ 10.1	62.4 $\pm$ 8.7	72.3 $\pm$ 0.1	88.3 $\pm$ 0.7	94.8 $\pm$ 1.1
faucet-close	57.8 $\pm$ 0.9	46.2 $\pm$ 0.2	59.4 $\pm$ 8.5	48.7 $\pm$ 0.6	73.1 $\pm$ 0.8
peg-insert-side	16.8 $\pm$ 0.1	12.4 $\pm$ 1.4	13.8 $\pm$ 4.4	9.7 $\pm$ 8.5	79.0 $\pm$ 0.2
reach	82.0 $\pm$ 0.8	84.3 $\pm$ 0.9	83.3 $\pm$ 0.1	86.6 $\pm$ 0.1	88.0 $\pm$ 0.5

Table 3: Ablation of the query selection module on the Meta-World tasks. We report the performance of offline RL algorithm on the reward-labeled dataset with various query selection and policy extration mechanism. IDE and VDS represent the In-Dataset Exploration module and the Variance-based Discount Scheduling module proposed in Section 3.1 , respectively.

Domain	Tasks	Zero	Random	Negative	OPRIDE
Metaworld	coffee-push	7.6 $\pm$ 4.3	5.8 $\pm$ 2.7	0.7 $\pm$ 0.1	59.4 $\pm$ 24.8
	disassemble	9.3 $\pm$ 0.4	16.8 $\pm$ 7.3	10.1 $\pm$ 0.2	26.6 $\pm$ 4.9
	hammer	38.1 $\pm$ 6.4	46.1 $\pm$ 2.4	22.6 $\pm$ 1.8	50.3 $\pm$ 3.2
	push	57.5 $\pm$ 1.5	34.4 $\pm$ 17.3	4.6 $\pm$ 2.3	59.1 $\pm$ 5.4
	push-wall	81.9 $\pm$ 3.8	80.1 $\pm$ 0.9	17.6 $\pm$ 1.9	102.2 $\pm$ 1.2
	soccer	33.3 $\pm$ 1.6	41.1 $\pm$ 8.8	44.0 $\pm$ 6.4	45.4 $\pm$ 3.9

Table 4: Comparison between the survival instinct and OPRIDE.

Figure 2: Performance of offline preference-based RL algorithms with various queries. OPRIDE achieves a better query efficiency across tasks and number of queries.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

\mathcal{D}_{\text{pref}}\leftarrow\mathcal{D}_{\text{pref}}\cup\{(\tau_{k}^{1},\tau_{k}^{2},o_{k})\}.

Appendix

Algorithm 2 Theoretical Analysis Version of OPRIDE

\mathcal{L}(q,q^{\prime},\pi;\mathcal{D})=\sum_{k=1}^{K}\sum_{t=0}^{T}\left(q_{t}(s^{k}_{t},a^{k}_{t})-(r(s^{k}_{t},a^{k}_{t})+\gamma q^{\prime}_{t+1}(s^{k}_{t+1},\pi_{t+1}))\right)^{2}.

(14)

Appendix

Algorithm 3 Bellman-consistent Pessimism (BCP)

\mathcal{L}(q,q^{\prime},\pi;\mathcal{D})=\sum_{k=1}^{K}\sum_{t=0}^{T}\left(q_{t}(s^{k}_{t},a^{k}_{h})-(r(s^{k}_{t},a^{k}_{t})+\gamma q^{\prime}(s^{k}_{t+1},\pi_{t+1}))\right)^{2}.

(22)

Appendix

Algorithm 4 Bellman-consistent Pessimism Evaluation

Task	OPRL	PT	PT+PDS	IDRL	OPRIDE
assembly-v2	10.1 $\pm$ 0.5	10.2 $\pm$ 0.7	12.8 $\pm$ 0.6	10.3 $\pm$ 1.9	14.2 $\pm$ 1.3
basketball-v2	11.7 $\pm$ 10.2	80.7 $\pm$ 0.1	78.7 $\pm$ 2.0	82.7 $\pm$ 2.5	61.4 $\pm$ 2.3
bin-picking-v2	82.0 $\pm$ 5.6	31.9 $\pm$ 16.2	53.4 $\pm$ 19.0	84.7 $\pm$ 2.9	93.3 $\pm$ 3.2
button-press-wall-v2	51.7 $\pm$ 1.6	58.8 $\pm$ 0.9	59.4 $\pm$ 0.9	69.0 $\pm$ 1.0	77.7 $\pm$ 0.1
box-close-v2	15.0 $\pm$ 0.7	17.7 $\pm$ 0.1	17.2 $\pm$ 0.3	16.9 $\pm$ 0.6	16.8 $\pm$ 0.4
coffee-push-v2	1.7 $\pm$ 1.7	1.3 $\pm$ 0.5	1.3 $\pm$ 0.5	42.0 $\pm$ 3.8	59.4 $\pm$ 24.8

Appendix

Table 5: Performance of offline RL algorithm on the reward-labeled dataset with different preference reward learning methods on the Meta-World tasks.

Domain	Tasks	OPRL	PT	PT+PDS	OPRIDE
Mujoco	hopper-medium	23.0 $\pm$ 0.1	36.9 $\pm$ 2.1	35.8 $\pm$ 1.8	38.5 $\pm$ 2.2
	hopper-medium-expert	57.7 $\pm$ 23.7	68.0 $\pm$ 2.6	69.1 $\pm$ 1.7	92.3 $\pm$ 15.8
	walker2d-medium	70.6 $\pm$ 1.1	71.7 $\pm$ 2.6	70.9 $\pm$ 1.8	72.7 $\pm$ 1.8
	walker2d-medium-expert	108.3 $\pm$ 3.8	109.4 $\pm$ 0.3	108.4 $\pm$ 0.5	110.3 $\pm$ 0.2
	halfcheetah-medium	41.9 $\pm$ 0.1	42.1 $\pm$ 0.1	41.5 $\pm$ 0.1	42.4 $\pm$ 0.1
	halfcheetah-medium-expert	81.8 $\pm$ 0.6	81.9 $\pm$ 0.1	82.4 $\pm$ 0.2	86.5 $\pm$ 1.5

Appendix

Table 6: Performance of offline RL algorithm on the reward-labeled dataset with different preference reward learning methods on the Mujoco tasks.

$\gamma_{\text{\rm small}}$	0.5	0.6	0.7	0.8	0.9	0.95
bin-picking	72.1 $\pm$ 23.9	87.8 $\pm$ 2.7	93.3 $\pm$ 3.2	84.6 $\pm$ 9.4	74.8 $\pm$ 33.5	70.9 $\pm$ 9.4
button-press-wall	77.6 $\pm$ 0.3	77.4 $\pm$ 0.3	77.7 $\pm$ 0.1	71.0 $\pm$ 0.7	69.2 $\pm$ 0.9	68.1 $\pm$ 9.7
door-close	88.4 $\pm$ 0.8	89.9 $\pm$ 0.7	94.8 $\pm$ 1.1	90.0 $\pm$ 2.1	91.1 $\pm$ 1.5	87.6 $\pm$ 0.7
faucet-close	58.1 $\pm$ 5.2	58.2 $\pm$ 12.5	73.1 $\pm$ 0.8	61.4 $\pm$ 2.7	55.1 $\pm$ 3.7	57.4 $\pm$ 12.1

Appendix

Table 7: Performance of offline RL algorithm on the reward-labeled dataset with various discount factor values

\gamma_{\text{\rm small}}

on the high variance data points.

Domain	Tasks	OPRIDE (Reward Difference)	OPRIDE (Value Function Difference)
Metaworld	bin-picking	78.5 $\pm$ 17.8	93.3 $\pm$ 3.2
	button-press-wall	67.4 $\pm$ 5.4	77.7 $\pm$ 0.1
	door-close	88.3 $\pm$ 0.7	94.8 $\pm$ 1.1
	faucet-close	48.7 $\pm$ 0.6	73.1 $\pm$ 0.8
	peg-insert-side	9.7 $\pm$ 8.5	79.0 $\pm$ 0.2
	reach	86.6 $\pm$ 0.1	88.0 $\pm$ 0.5

Appendix

Table 8: Ablation study on the metaworld tasks.

Domain	Tasks	OPRIDE (Penalise Reward)	OPRIDE (Penalise Discount Factor)
Metaworld	bin-picking	53.4 $\pm$ 19.0	93.3 $\pm$ 3.2
	button-press-wall	59.4 $\pm$ 0.9	77.7 $\pm$ 0.1
	door-close	62.4 $\pm$ 8.7	94.8 $\pm$ 1.1
	faucet-close	46.2 $\pm$ 0.2	73.1 $\pm$ 0.8
	peg-insert-side	12.4 $\pm$ 1.4	79.0 $\pm$ 0.2
	reach	84.3 $\pm$ 0.9	88.0 $\pm$ 0.5

Appendix

Table 9: Ablation studies about penalizing rewards and the discount factor.

$m\%$	$10\%$	$20\%$	$30\%$	$40\%$	$50\%$
bin-picking	71.4 $\pm$ 3.3	85.9 $\pm$ 3.9	93.3 $\pm$ 3.2	88.7 $\pm$ 2.6	73.2 $\pm$ 2.7
button-press-wall	67.2 $\pm$ 2.1	70.2 $\pm$ 0.6	77.7 $\pm$ 0.1	77.5 $\pm$ 0.3	77.5 $\pm$ 0.6
door-close	87.9 $\pm$ 1.2	89.6 $\pm$ 1.0	94.8 $\pm$ 1.1	89.6 $\pm$ 0.8	87.3 $\pm$ 0.9
faucet-close	56.8 $\pm$ 2.6	62.3 $\pm$ 1.6	73.1 $\pm$ 0.8	58.7 $\pm$ 1.9	57.6 $\pm$ 2.1

Appendix

Table 10: Ablation study for the hyperparameter

m

from 10% to 50%.

Tasks	Continuous annealing	Threshold-based
bin-picking	94.1 $\pm$ 3.7	93.3 $\pm$ 3.2
button-press-wall	77.4 $\pm$ 0.3	77.7 $\pm$ 0.1
door-close	94.3 $\pm$ 1.7	94.8 $\pm$ 1.1
faucet-close	71.6 $\pm$ 0.9	73.1 $\pm$ 0.8
peg-insert-side	80.5 $\pm$ 0.3	79.0 $\pm$ 0.2
reach	87.5 $\pm$ 0.6	88.0 $\pm$ 0.5

Appendix

Table 11: Ablation study for the discount factor mechanism on the Meta-World tasks.

$M$	2	5	10
bin-picking	1.1h	1.2h	1.3h
button-press-wall	1.1h	1.1h	1.2h
door-close	1.1h	1.2h	1.3h
faucet-close	0.9h	1.0h	1.2h
peg-insert-side	1.0h	1.1h	1.3h

Appendix

Table 12: Computational Cost for OPRIDE with ensemble

M

, where

h

is the hour. The report time is the sum of the reward function training time and the policy training time.

Query Number	1 (BC)	10 (0.1%)	13	20 (0.2%)	30 (0.3%)	50 (0.5%)	100 (1.0%)
Hammer-v2	13.2 $\pm$ 3.3	50.3 $\pm$ 3.2	55.4 $\pm$ 2.6	75.9 $\pm$ 2.2	79.1 $\pm$ 2.6	79.5 $\pm$ 2.7	79.9 $\pm$ 2.3
Coffee-push-v2	2.5 $\pm$ 1.1	56.7 $\pm$ 2.8	57.6 $\pm$ 2.5	55.3 $\pm$ 3.1	58.9 $\pm$ 3.7	59.4 $\pm$ 3.5	59.5 $\pm$ 3.4
Disassemble-v2	9.3 $\pm$ 1.7	26.6 $\pm$ 2.8	27.8 $\pm$ 3.1	30.6 $\pm$ 5.2	35.7 $\pm$ 4.8	37.2 $\pm$ 4.6	38.3 $\pm$ 5.1
Push-v2	1.2 $\pm$ 0.6	50.9 $\pm$ 1.8	63.7 $\pm$ 2.1	76.8 $\pm$ 2.4	82.7 $\pm$ 2.6	83.6 $\pm$ 2.3	84.0 $\pm$ 2.2

Appendix

Table 13: Performance of OPRIDE with various query numbers.

Tasks	PT	IPL	CPL	LiRE	OPRIDE
lever-pull	49.2 $\pm$ 3.7	50.2 $\pm$ 2.1	50.1 $\pm$ 2.5	51.2 $\pm$ 1.8	51.8 $\pm$ 1.6
peg-insert-side	16.8 $\pm$ 0.1	53.8 $\pm$ 0.4	54.7 $\pm$ 0.3	63.1 $\pm$ 0.2	79.0 $\pm$ 0.2
plate-slide	4.9 $\pm$ 0.0	60.9 $\pm$ 5.6	61.7 $\pm$ 4.8	68.3 $\pm$ 4.3	79.9 $\pm$ 4.6
push	16.7 $\pm$ 5.0	38.5 $\pm$ 4.2	39.2 $\pm$ 3.9	45.5 $\pm$ 3.7	59.1 $\pm$ 5.4
push-back	1.1 $\pm$ 0.4	9.8 $\pm$ 1.6	9.3 $\pm$ 1.8	13.2 $\pm$ 1.5	17.7 $\pm$ 2.0
push-wall	74.8 $\pm$ 14.4	85.8 $\pm$ 4.7	87.3 $\pm$ 3.8	90.2 $\pm$ 3.6	102.2 $\pm$ 1.2

Appendix

Table 14: Comparison with one-stage framework and sequential ranked list method.

Tasks	OPRL	PT	PT+PDS	IDRL	OPRIDE
Pong	9.6 $\pm$ 1.4	9.4 $\pm$ 1.0	8.5 $\pm$ 1.8	15.3 $\pm$ 1.1	17.8 $\pm$ 1.3
Breakout	125.9 $\pm$ 14.2	79.9 $\pm$ 13.9	86.3 $\pm$ 13.8	153.7 $\pm$ 14.5	256.7 $\pm$ 14.3
Q*bert	7924.6 $\pm$ 376.1	7482.9 $\pm$ 353.4	6844.2 $\pm$ 394.6	8294.1 $\pm$ 359.7	13535.2 $\pm$ 327.2
Seaquest	2784.1 $\pm$ 72.7	2538.3 $\pm$ 78.9	2459.6 $\pm$ 74.5	2941.2 $\pm$ 69.2	3478.4 $\pm$ 71.3
Asterix	164.9 $\pm$ 21.5	155.8 $\pm$ 24.6	146.3 $\pm$ 27.4	357.4 $\pm$ 30.1	426.9 $\pm$ 28.7

Appendix

Table 15: Experiments on pixel-based Atari tasks with human-in-the-loop.

Hyperparameter	Value
Optimizer	Adam
Critic learning rate	3e-4
Actor learning rate	3e-4
Mini-batch size	256
Discount factor	0.99
Target update rate	5e-3

Appendix

Table 16: Hyper-parameters sheet of Algorithms.

Preference-Based RL

Preference-based reinforcement learning learns policies from comparisons or rankings of behaviors instead of requiring hand-designed numerical rewards.

See all 127 papers

From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
May 31, 2026Jun-Jie Yang, Chia-Heng Hsu, Kui-Yuan Chen +1Pairwise Preference Learning Representation Learning
Department of Computer Science, National Yang Ming Chiao Tung University, Hsinchu, Taiwan
PrefO2O: Zero-Shot Preference-Based Reinforcement Learning via Offline-to-Online Latent Preference Transfer
Oct 22, 2026Ruiqi Wang, Dezhong Zhao, Xuesu Xiao +3Preference-Based RL Zero-Shot RL
University of Southern California, Los Angeles, CA, USA. · Beijing University of Chemical Technology, Beijing, China. · George Mason University, Fairfax, VA, USA. +1
Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation
Jul 21, 2026Li-Rong Zhou, Qin-Wen Luo, Sheng-Jun HuangReinforcement Learning Offline RL
Nanjing University of Aeronautics and Astronautics, Nanjing, China

May 31, 2026J/K move · Enter open · S save

From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning

Jun-Jie Yang, Chia-Heng Hsu, Kui-Yuan Chen, Ping-Chun Hsieh

Department of Computer Science, National Yang Ming Chiao Tung University, Hsinchu, Taiwan

Pairwise Preference Learning Representation Learning Pairwise Preference Evaluation Offline RL Successor Representations Preference-Based RL

Open paper AskPDF arXiv

Preference-based reinforcement learning (PbRL) avoids explicit reward engineering by learning from pairwise human preference feedback. Existing offline PbRL methods typically follow a two-stage pipeline, first learning a reward or preference model from labeled preferences and then performing offline RL on unlabeled data. We revisit offline PbRL through the lens of reward-free representation learning (RFRL) from the zero-shot RL literature, and propose a new training framework that first learns latent successor-measure representations from reward-free offline data, followed by contrastive search and fine-tuning using preference data. Through extensive experiments and ablations, we show that our method achieves superior preference efficiency over offline PbRL baselines. This work is the first to connect RFRL with PbRL, highlighting its potential as a feedback-efficient solution. Our code is publicly available at https://github.com/rl-bandits-lab/FB-PbRL.

Abstract

Preference-Based RL

Explore similar work

From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning

PrefO2O: Zero-Shot Preference-Based Reinforcement Learning via Offline-to-Online Latent Preference Transfer

Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation