Dual Active Learning for Reinforcement Learning from Human Feedback

Pangpang Liu; Chengchun Shi; Will Wei Sun

stat.MLOct 3, 2024

Dual Active Learning for Reinforcement Learning from Human Feedback

Authors: Pangpang Liu, Chengchun Shi, Will Wei Sun

Organizations: Mitchell E. Daniels, Jr. School of Business, Purdue University · Department of Statistics, London School of Economics and Political Science

Abstract

Aligning large language models (LLMs) with human preferences is critical to recent advances in generative artificial intelligence. Reinforcement learning from human feedback (RLHF) is widely applied to achieve this objective. A key step in RLHF is to learn the reward function from human feedback. However, human feedback is costly and time-consuming, making it essential to collect high-quality conversation data for human teachers to label. Additionally, different human teachers have different levels of expertise. It is thus critical to query the most appropriate teacher for their opinions. In this paper, we use reinforcement learning (RL) to formulate the alignment problem. Motivated by the idea of $D$ -optimal design, we first propose a dual active reward learning algorithm for the simultaneous selection of conversations and teachers. Next, we apply pessimistic RL to solve the alignment problem, based on the learned reward estimator. Theoretically, we show that the reward estimator obtained through our proposed adaptive selection strategy achieves minimal generalized variance asymptotically, and prove that the sub-optimality of our pessimistic policy scales as $O(1/\sqrt{T})$ with a given sample budget $T$ . Through simulations and experiments on LLMs, including annotator-level preference data and downstream Best-of- $N$ response-selection evaluations, we demonstrate the effectiveness of our algorithm and its superiority over existing baselines.

Figures & tables

Question/Prompt	First Answer/Completion	Second Answer/Completion
Why are gas prices so high after the last presidential election?	Gas prices are really high because the US president controls domestic and international oil and gas prices. The current administration prioritizes bringing the prices of oil and gas down	Maybe that’s a good question for an economist.
I love listening to Jazz, especially Miles Davis. Can you recommend any similar music or artists?	I could make a playlist for you, I guess. Would you like some old-school and new-school jazz? Jazz-funk? Fusion? What do you like?	Human, jazz is a huge musical genre. Which specific style of jazz are you interested in?

Table 1: An illustration of data examples for learning from human feedback

Papers	Conversation selection	Teacher selection	Optimal design
Ji et al. (2025)	✓
Das et al. (2025)	✓
Mukherjee et al. (2024)	✓
Daniels-Koch and Freedman (2022)		✓
Barnett et al. (2023)		✓
Freedman et al. (2026)		✓

Table 2: Comparison with other works on conversation/teacher selection for RLHF

Figure 1: Schematic representation of the conversation and teacher selection process. The goal is to select $T$ conversations from the conversation set and query a teacher $\beta_{t}$ from the teacher set for their preference $y_{t}$ between two responses for each selected conversation. The reward estimator $\widehat{\theta}_{T}$ is obtained based on the collected information $\{(z_{t},\beta_{t},y_{t})\}_{t=1}^{T}$ where $z$ is a shorthand for $\phi(x,a^{(1)})-\phi(x,a^{(0)})$ .

Figure 2: Confidence ellipsoid (gray area) centered at the estimator $\widehat{\theta}$ in two dimensions. The dashed lines denote the principal axes of the ellipsoid, whose lengths are proportional to $1/\sqrt{\lambda_{1}}$ and $1/\sqrt{\lambda_{2}}$ , where $\lambda_{1}$ and $\lambda_{2}$ are the eigenvalues of $M(\xi_{T},\theta_{*})$ . Maximizing $\lambda_{1}\lambda_{2}$ , equivalently maximizing $\det[M(\xi_{T},\theta_{*})]$ , minimizes the area of the confidence ellipsoid and leads to more precise estimation of $\theta_{*}$ .

H_{t-1}(\widehat{\theta}_{t-1})=\sum_{s=1}^{t-1}\dot{\mu}(\beta_{s}\widehat{\theta}_{t-1}^{\top}z_{s})\beta_{s}^{2}z_{s}z_{s}^{\top}.

(7)

Algorithm 1 Dual active reward learning using

D

-optimal design

Panel of Figure 3 — Figure 3: Estimation error of MLE and sub-optimality gaps of pessimistic and greedy policies.

Methods	Conversation Selection	Teacher Selection
DualAL	Algorithm 1	Algorithm 1
Conversation Selection Only	$D$ -optimal design	Random
Teacher Selection Only	Random	$D$ -optimal design
APO	APO ( Das et al., 2025 )	Random
Linear Approximation	Mukherjee et al. (2024) ; Scheid et al. (2024)	Random
Random	Random	Random

Table 3: Strategies for conversation selection and teacher selection across various methods.

Policies	$K=1$		$K=50$		$K=100$
	GV	MSE	GV	MSE	GV	MSE
DualAL	5.6	1.251	7.6	1.292	9.4	1.348
Conversation Selection Only	41.0	1.481	20.7	1.344	16.4	1.459
Teacher Selection Only	2570.8	2.162	1284.0	2.110	2092.2	2.208
APO	282.0	1.993	262.4	1.752	253.1	1.857
Linear Approximation	462.7	1.837	1858.6	2.109	1919.0	2.192

Table 4: Performance of the reward parameter estimation with

T=1000

Panel of Figure 4 — Figure 4: Sub-optimality gaps for all policies. The three subplots show the sub-optimality gap when the batch size $K$ is 1, 50 and 100, respectively.

Figure 5: Sub-optimality gap of DualAL as a function of sample size $T$ . The dashed line represents the $O(1/\sqrt{T})$ reference curve.

Panel of Figure 6 — Figure 6: Sub-optimality gap for DualAL at different ranges of teacher rationality.

Panel of Figure 7 — Figure 7: Sub-optimality gap for DualAL at different dimensions.

Policies	$K=1$		$K=50$		$K=100$
	GV	MSE	GV	MSE	GV	MSE
DualAL	0.30	0.605	1.12	0.647	1.01	0.684
Conversation Selection Only	0.76	0.660	3.43	0.765	3.06	0.792
Teacher Selection Only	261.43	1.136	57.42	0.914	140.22	1.076
APO	112.05	1.058	92.24	1.006	353.28	1.180
Linear Approximation	16.10	0.857	276.67	1.102	85.54	1.049

Table 5: Performance of the reward parameter estimation when the reward model is nonlinear

Panel of Figure 8 — Figure 8: Sub-optimality gaps for all policies when the reward model is nonlinear. The three subplots show the sub-optimality gap when the batch size $K$ is 1, 50 and 100, respectively.

Panel of Figure 9 — Figure 9: Sub-optimality gaps under misspecified rationality parameters. The three subplots correspond to batch sizes $K=1$ , $50$ , and $100$ , respectively.

Panel of Figure 10 — Figure 10: Reward accuracy with different methods using the prism-alignment dataset.

Policies	AlpacaEval			HumanEvalPack
Policies	easy	hard	length	cpp	go	java	js	python	rust
DualAL	0.906	0.906	0.807	0.590	0.601	0.562	0.628	0.580	0.564
Conversation Selection Only	0.819	0.834	0.755	0.583	0.571	0.508	0.557	0.568	0.540
Teacher Selection Only	0.831	0.868	0.777	0.549	0.570	0.507	0.584	0.528	0.547
APO	0.759	0.779	0.661	0.474	0.509	0.527	0.520	0.475	0.532
Linear Approximation	0.839	0.901	0.695	0.511	0.570	0.563	0.585	0.511	0.525

Table 6: Reward accuracy on RewardBench benchmarks with batch size

K=50

Policies	AlpacaEval			HumanEvalPack
Policies	easy	hard	length	cpp	go	java	js	python	rust
DualAL	0.897	0.907	0.804	0.602	0.586	0.578	0.619	0.573	0.565
Conversation Selection Only	0.782	0.819	0.689	0.553	0.571	0.506	0.540	0.543	0.540
Teacher Selection Only	0.831	0.869	0.767	0.580	0.563	0.511	0.595	0.544	0.555
APO	0.742	0.769	0.635	0.513	0.520	0.543	0.530	0.476	0.544
Linear Approximation	0.855	0.902	0.704	0.530	0.588	0.582	0.611	0.556	0.531

Table 7: Reward accuracy on RewardBench benchmarks with batch size

K=100

Policies	Mean of True Score	Mean Diff	$P$ _value
DualAL	69.741	0	-
Conversation Selection Only	68.448	1.293	$3.69\times 10^{-7}$
Teacher Selection Only	69.331	0.410	$1.44\times 10^{-2}$
APO	66.189	3.552	$6.59\times 10^{-13}$
Linear Approximation	68.756	0.985	$1.45\times 10^{-5}$
Random	68.707	1.034	$7.84\times 10^{-5}$

Table 8: Evaluation of Selected Responses by Different Policies with

K=50

Policies	Mean of True Score	Mean Diff	$P$ _value
DualAL	69.78	0	-
Conversation Selection Only	68.43	1.35	$1.38\times 10^{-10}$
Teacher Selection Only	69.113	0.667	$2.00\times 10^{-6}$
APO	66.122	3.658	$3.44\times 10^{-16}$
Linear Approximation	68.725	1.055	$6.51\times 10^{-8}$
Random	69.273	0.507	$2.96\times 10^{-3}$

Table 9: Evaluation of Selected Responses by Different Policies with

K=100

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 11 — Figure 11: Reward accuracy with different methods using the Anthropic dataset.

Panel of Figure 12 — Figure 12: Reward accuracy with different methods using the UltraFeedback dataset.

RL from Human Feedback

Reinforcement learning from human feedback (RLHF) trains AI systems to better match human preferences using comparisons, corrections, and other feedback.

See all 133 papers

ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning
Mar 10, 2026Davit Melikidze, Marian Schneider, Jessica Lam +4Data Selection Active Learning
ETH Zurich · Institute of Neuroinformatics, University of Zurich and ETH Zurich · ETH AI Center
Doubly Robust Alignment for Large Language Models
Jun 1, 2025Erhan Xu, Kai Ye, Hongyi Zhou +3LLM Alignment RL from Human Feedback
Department of Statistics, LSE, London, UK · Department of Mathematics, Tsinghua University, Beijing, China · School of Design, LCC, UAL, London, UK +1
Reinforcement Learning from Human Feedback: A Statistical Perspective
Apr 2, 2026Pangpang Liu, Chengchun Shi, Will Wei SunRL from Human Feedback LLM Alignment
Department of Biostatistics, Yale University. · Department of Statistics, London School of Economics and Political Science. · Mitch Daniels School of Business, Purdue University.

Mar 10, 2026J/K move · Enter open · S save

ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning

Davit Melikidze, Marian Schneider, Jessica Lam, Martin Wertich+3

ETH Zurich · Institute of Neuroinformatics, University of Zurich and ETH Zurich · ETH AI Center

Data Selection Active Learning RL from Human Feedback Uncertainty Sampling

Open paper AskPDF arXiv

Reinforcement Learning from Human Feedback (RLHF) has become the standard for aligning Large Language Models (LLMs), yet its efficacy is bottlenecked by the high cost of acquiring preference data, especially in low-resource and expert domains. To address this, we introduce ACTIVEULTRAFEEDBACK, a modular active learning pipeline that leverages uncertainty estimates to dynamically identify the most informative responses for annotation. Our pipeline facilitates the systematic evaluation of standard response selection methods alongside DOUBLE REVERSE THOMPSON SAMPLING (DRTS) and DELTAUCB, two novel methods prioritizing response pairs with large predicted quality gaps, leveraging recent results showing that such pairs provide good signals for fine-tuning. Our experiments demonstrate that ACTIVEULTRAFEEDBACK yields high-quality datasets that lead to significant improvements in downstream performance, notably achieving comparable or superior results with as little as one-sixth of the annotated data relative to static baselines. Our pipeline is available at https://github.com/lasgroup/ActiveUltraFeedback and our preference datasets at https://huggingface.co/ActiveUltraFeedback.

Abstract

RL from Human Feedback

Explore similar work

ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning

Doubly Robust Alignment for Large Language Models

Reinforcement Learning from Human Feedback: A Statistical Perspective