AERO: Adaptive Ensemble-Disagreement Routing for Oracle Feedback in Sample-Efficient Online RLHF

Shrey Singhal; Nannan Huang; Haytham M. Fayek

cs.LGNov 7, 2026

AERO: Adaptive Ensemble-Disagreement Routing for Oracle Feedback in Sample-Efficient Online RLHF

Authors: Shrey Singhal, Nannan Huang, Haytham M. Fayek

Organizations: RMIT University Melbourne, VIC, Australia

Abstract

Aligning Large Language Models (LLMs) with human preferences via Reinforcement Learning from Human Feedback (RLHF) has driven substantial improvements across a wide range of tasks, but the cost of acquiring preference labels remains the dominant bottleneck in the alignment pipeline. While many recent alignment methods improve sample efficiency through active exploration over candidate responses, they typically query the oracle uniformly across contexts, even for contexts where the learned reward estimates are already reliable enough to provide synthetic labels. We propose Adaptive Ensemble-disagreement Routing for Oracle feedback (AERO), a fully online RLHF algorithm built on a Dyna-style model-based RL architecture, in which an ensemble-based Epistemic Reward Model (ERM) serves both as a reward posterior for exploration and as a learned preference-feedback model for synthetic labeling. AERO uses ensemble vote entropy, a Query-by-Committee disagreement measure, combined with an adaptive thresholding mechanism to decide which contexts require oracle feedback and which can instead receive ERM-based synthetic preference labels, concentrating expensive labels where they are most informative. Empirical results across different model families and scales show that AERO achieves highly competitive alignment performance while reducing the required oracle-query budget compared to online DPO and strong active-exploration baselines.

Figures & tables

Figure 1: Overview of AERO. Contexts with high ensemble disagreement are routed to the oracle, while the rest receive synthetic labels from the ERM. $\mathbb{P}$ denotes the Preference Oracle, $\mathcal{R}_{\Phi}$ the ERM, $\pi_{\theta}$ the policy. Thompson Sampling explores the exact disagreement measured by vote entropy. High vote entropy ensures $y$ is drawn from severe disagreement.

Panel of Figure 2 — Figure 2: Oracle-budget sample efficiency across three seeds. We plot win rate against cumulative oracle queries for (a) Llama-3.2-1B-Instruct and (b) Qwen2.5-1.5B-Instruct on UltraFeedback. Otsu-based adaptive routing automatically determines the routing ratio, which yields different total $B$ across models. The matched budget range is chosen based on AERO’s full $B$ throughout training in each setting.

Model	Qwen2.5-1.5B-Instruct				Llama-3-8B-Instruct				Llama-3.2-1B-Instruct
Model	LC	WR	Len	$B$	LC	WR	Len	$B$	LC	WR	Len	$B$
Base	10.14	7.38	1395	—	29.71	30.15	1994	—	8.06	9.23	2010	—
DPO 1	11.46	8.24	1389	12.8k	37.01	32.64	1766	12.8k	8.92	10.03	2019	12.8k
DPO 2	12.34	8.62	1404	38.4k	39.48	34.66	1743	25.6k	10.16	10.99	1966	38.4k
DPO 3	12.85	8.89	1369	64.0k	40.47	34.59	1725	38.4k	8.81	9.60	1969	64.0k
COPO 1	12.10	8.89	1422	12.8k	38.64	33.36	1726	12.8k	8.61	9.72	2018	12.8k

Table 1: AlpacaEval 2.0 results across three model settings. Final LC win rates for 1B and 1.5B class models are mean

\pm

SE over three seeds for AERO and the two active-exploration baselines; other evaluations are single-run.

B

denotes cumulative oracle queries. Iteration indices 1–3 correspond to iterations 100/300/500 for the 1B & 1.5B class models and 100/200/300 for the 8B model.

Panel of Figure 3 — Figure 3: Ablations on Llama-3.2-1B-Instruct on UltraFeedback. Win rate relative to the initial instruction-tuned model is shown over cumulative oracle-query consumption $B$ for AERO and four single-component ablation variants.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Iteration-wise win rate against the initial instruction-tuned policy. Checkpoints are evaluated every 25 online training iterations.

Figure 5: Aleatoric filtering activity across training. Number of candidate pairs dropped per iteration by the aleatoric filter (Eq 5 ,

\alpha

= 0.6) using Llama-3.2-1B-Instruct on UltraFeedback, averaged across three seeds. The first 8 iterations are a burn-in period during which the filter is disabled.

Figure 6: Oracle win rate against estimated cumulative GPU-hours per seed on Llama-3.2-1B-Instruct and UltraFeedback.

Method	Generations	ERM-routed	Synthetic labels	Oracle queries	GPU-h	Wall-clock
AERO	3.840M	102.0K	99.7K	81.8K	$\sim$ 160	$\sim$ 27 h
SEA	3.840M	56.1K	56.1K	135.9K	$\sim$ 170	$\sim$ 28 h
COPO	0.576M	0	0	192.0K	$\sim$ 100	$\sim$ 17 h
Online DPO	0.384M	0	0	192.0K	$\sim$ 67	$\sim$ 11 h

Appendix

Table 2: Total resource usage for the primary three-seed Llama-3.2-1B-Instruct experiment. Generation, erm-routing, synthetic-label, oracle-query, and GPU-hour counts are totals across three seeds, whereas wall-clock time is per seed on two A100 40GB GPUs.

Model	Early	Middle	Final
Llama-3.2-1B	0.60	1.00	0.40
Qwen2.5-1.5B	0.40	1.00	1.00
Llama-3-8B	0.80	0.60	0.80

Appendix

Table 3: Within-stage Spearman agreement between win-rate gains evaluated using the Skywork-Reward-Llama-3.1-8B RM and AlpacaEval 2.0 length-controlled win-rate gains.

Method	Raw WR	Style-controlled WR
Initial policy	18.0	17.0 ( $-1.2/+1.7$ )
Online DPO	18.2	17.7 ( $-1.2/+1.3$ )
SEA	16.6	15.8 ( $-1.5/+1.3$ )
COPO	15.4	15.1 ( $-1.5/+1.2$ )
AERO	19.0	17.2 ( $-1.4/+1.8$ )

Appendix

Table 4: Arena-Hard results for the final Llama-3-8B checkpoints.

Figure 7: Adaptive versus fixed-quantile routing. Comparison of AERO’s adaptive Otsu routing with fixed-quantile routing using an oracle-routing ratio of

q=0.46

, evaluated on Llama-3.2-1B-Instruct with UltraFeedback and averaged across three seeds.

Iteration	Fixed $q=0.15$		Fixed $q=0.25$		AERO (Otsu)		Fixed $q=0.65$
Iteration	WR	$B$	WR	$B$	WR	$B$	WR	$B$
100	57.84	2.8K	59.88	3.9K	56.67–58.20	5.2–5.7K	58.91	8.4K
200	64.71	4.7K	62.47	7.1K	61.00–62.07	10.8–11.4K	63.24	16.4K
300	63.65	6.6K	64.10	10.3K	62.42–63.75	16.1–16.9K	63.59	24.2K
400	64.77	8.5K	65.43	13.5K	63.29–65.27	21.6–22.2K	64.56	32.1K
500	66.19	10.4K	65.58	16.7K	63.75–64.61	26.9–27.6K	66.80	39.8K

Appendix

Table 5: Single-seed sweep over fixed routing fractions

q

on Llama-3.2-1B-Instruct with UltraFeedback. As in Figure 2 , WR (%) is the held-out win rate against the initial policy judged by Skywork-Reward-Llama-3.1-8B , and

B

is the cumulative oracle queries. For AERO (Otsu, average fraction

\approx 0.46

) we report the range over three seeds. Per-iteration WR differs by only 1–2 points with no consistent ordering across

q

, while

B

scales with

q

; confirming this trend requires additional seeds.

Figure 8: Comparison of fixed-CFN and rolling-CFN variants of the COPO-adapted baseline on Llama-3.2-1B-Instruct with UltraFeedback. The fixed-CFN version is used in the main results because it performs slightly better in our fine-grained online setting.

Basic Parameters
Fine-tuning strategy	LoRA [ Hu et al., 2022 ] ( Llama-3-8B-Instruct ), Full parameter fine tuning ( Qwen2.5-1.5B-Instruct , Llama-3.2-1B-Instruct )
LoRA target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
LoRA $r$	64
LoRA alpha	64
LoRA dropout	0.05
Learning rate	$5\times 10^{-7}$ ( Qwen2.5-1.5B-Instruct , Llama-3.2-1B-Instruct ), $2\times 10^{-5}$ ( Llama-3-8B-Instruct )

Appendix

Table 14

Panel of Figure 9 — Figure 9: Single-seed results on TL;DR summarization [ Völske et al., 2017 ] using a 1B model from the Pythia family [ Biderman et al., 2023 ] . AERO performs comparably to SEA by iteration and achieves a higher win rate at matched oracle budgets. The oracle-budget plot shows that Otsu-based routing selects a smaller oracle budget in this easier summarization setting, suggesting that AERO adapts by routing more low-disagreement contexts to ERM synthetic labeling when the ensemble reaches consensus.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 628 papers

Mar 10, 2026

ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning

Reinforcement Learning from Human Feedback (RLHF) has become the standard for aligning Large Language Models (LLMs), yet its efficacy is bottlenecked by the high cost of acquiring preference data, especially in low-resource and expert domains. To address this, we introduce ACTIVEULTRAFEEDBACK, a modular active learning pipeline that leverages uncertainty estimates to dynamically identify the most informative responses for annotation. Our pipeline facilitates the systematic evaluation of standard response selection methods alongside DOUBLE REVERSE THOMPSON SAMPLING (DRTS) and DELTAUCB, two novel methods prioritizing response pairs with large predicted quality gaps, leveraging recent results showing that such pairs provide good signals for fine-tuning. Our experiments demonstrate that ACTIVEULTRAFEEDBACK yields high-quality datasets that lead to significant improvements in downstream performance, notably achieving comparable or superior results with as little as one-sixth of the annotated data relative to static baselines. Our pipeline is available at https://github.com/lasgroup/ActiveUltraFeedback and our preference datasets at https://huggingface.co/ActiveUltraFeedback.

Davit Melikidze, Marian Schneider, Jessica Lam +4

ETH Zurich · Institute of Neuroinformatics, University of Zurich and ETH Zurich · ETH AI Center

Data Selection Active Learning RL from Human Feedback Uncertainty Sampling

Ask PDF

Feb 24, 2026

Oracle-Robust Online Alignment for Large Language Models

We study online alignment of large language models under misspecified preference feedback, where the observed preference oracle deviates from an ideal but unknown ground-truth oracle. The online LLM alignment problem is a bi-level reinforcement problem due to the coupling between data collection and policy updates. Recently, the problem has been reduced to a tractable single-level objective in the SAIL (Self-Improving Efficient Online Alignment) framework. In this paper, we introduce a pointwise oracle uncertainty set in this problem and formulate an oracle-robust online alignment objective as a worst-case optimization problem. For log-linear policies, we show that this robust objective admits an exact closed-form decomposition into the original loss function plus an explicit sensitivity penalty. In the finite-response log-linear setting, we develop projected stochastic composite updates for the resulting weakly convex objective and prove

\widetilde{O}(\varepsilon^{-2})

oracle complexity for reaching approximate stationarity.

Zimeng Li, Mudit Gaur, Vaneet Aggarwal

Purdue University

LLM Alignment Preference-Based RL Language Model Robustness Robust Optimization

Ask PDF

Oct 3, 2024

Dual Active Learning for Reinforcement Learning from Human Feedback

Aligning large language models (LLMs) with human preferences is critical to recent advances in generative artificial intelligence. Reinforcement learning from human feedback (RLHF) is widely applied to achieve this objective. A key step in RLHF is to learn the reward function from human feedback. However, human feedback is costly and time-consuming, making it essential to collect high-quality conversation data for human teachers to label. Additionally, different human teachers have different levels of expertise. It is thus critical to query the most appropriate teacher for their opinions. In this paper, we use reinforcement learning (RL) to formulate the alignment problem. Motivated by the idea of

D

-optimal design, we first propose a dual active reward learning algorithm for the simultaneous selection of conversations and teachers. Next, we apply pessimistic RL to solve the alignment problem, based on the learned reward estimator. Theoretically, we show that the reward estimator obtained through our proposed adaptive selection strategy achieves minimal generalized variance asymptotically, and prove that the sub-optimality of our pessimistic policy scales as

O(1/\sqrt{T})

with a given sample budget

T

. Through simulations and experiments on LLMs, including annotator-level preference data and downstream Best-of-

N

response-selection evaluations, we demonstrate the effectiveness of our algorithm and its superiority over existing baselines.

Pangpang Liu, Chengchun Shi, Will Wei Sun

Mitchell E. Daniels, Jr. School of Business, Purdue University · Department of Statistics, London School of Economics and Political Science

RL from Human Feedback Preference Learning for LLMs Reward Modeling

Ask PDF

Abstract

LLM Alignment

Explore similar work

ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning

Oracle-Robust Online Alignment for Large Language Models

Dual Active Learning for Reinforcement Learning from Human Feedback