AERO: Adaptive Ensemble-Disagreement Routing for Oracle Feedback in Sample-Efficient Online RLHF

Shrey Singhal; Nannan Huang; Haytham M. Fayek

cs.LGNov 7, 2026

AERO: Adaptive Ensemble-Disagreement Routing for Oracle Feedback in Sample-Efficient Online RLHF

Authors: Shrey Singhal, Nannan Huang, Haytham M. Fayek

Organizations: RMIT University Melbourne, VIC, Australia

Abstract

Aligning Large Language Models (LLMs) with human preferences via Reinforcement Learning from Human Feedback (RLHF) has driven substantial improvements across a wide range of tasks, but the cost of acquiring preference labels remains the dominant bottleneck in the alignment pipeline. While many recent alignment methods improve sample efficiency through active exploration over candidate responses, they typically query the oracle uniformly across contexts, even for contexts where the learned reward estimates are already reliable enough to provide synthetic labels. We propose Adaptive Ensemble-disagreement Routing for Oracle feedback (AERO), a fully online RLHF algorithm built on a Dyna-style model-based RL architecture, in which an ensemble-based Epistemic Reward Model (ERM) serves both as a reward posterior for exploration and as a learned preference-feedback model for synthetic labeling. AERO uses ensemble vote entropy, a Query-by-Committee disagreement measure, combined with an adaptive thresholding mechanism to decide which contexts require oracle feedback and which can instead receive ERM-based synthetic preference labels, concentrating expensive labels where they are most informative. Empirical results across different model families and scales show that AERO achieves highly competitive alignment performance while reducing the required oracle-query budget compared to online DPO and strong active-exploration baselines.

Figures & tables

Figure 1: Overview of AERO. Contexts with high ensemble disagreement are routed to the oracle, while the rest receive synthetic labels from the ERM. $\mathbb{P}$ denotes the Preference Oracle, $\mathcal{R}_{\Phi}$ the ERM, $\pi_{\theta}$ the policy. Thompson Sampling explores the exact disagreement measured by vote entropy. High vote entropy ensures $y$ is drawn from severe disagreement.

Panel of Figure 2 — Figure 2: Oracle-budget sample efficiency across three seeds. We plot win rate against cumulative oracle queries for (a) Llama-3.2-1B-Instruct and (b) Qwen2.5-1.5B-Instruct on UltraFeedback. Otsu-based adaptive routing automatically determines the routing ratio, which yields different total $B$ across models. The matched budget range is chosen based on AERO’s full $B$ throughout training in each setting.

Model	Qwen2.5-1.5B-Instruct				Llama-3-8B-Instruct				Llama-3.2-1B-Instruct
Model	LC	WR	Len	$B$	LC	WR	Len	$B$	LC	WR	Len	$B$
Base	10.14	7.38	1395	—	29.71	30.15	1994	—	8.06	9.23	2010	—
DPO 1	11.46	8.24	1389	12.8k	37.01	32.64	1766	12.8k	8.92	10.03	2019	12.8k
DPO 2	12.34	8.62	1404	38.4k	39.48	34.66	1743	25.6k	10.16	10.99	1966	38.4k
DPO 3	12.85	8.89	1369	64.0k	40.47	34.59	1725	38.4k	8.81	9.60	1969	64.0k
COPO 1	12.10	8.89	1422	12.8k	38.64	33.36	1726	12.8k	8.61	9.72	2018	12.8k

Table 1: AlpacaEval 2.0 results across three model settings. Final LC win rates for 1B and 1.5B class models are mean

\pm

SE over three seeds for AERO and the two active-exploration baselines; other evaluations are single-run.

B

denotes cumulative oracle queries. Iteration indices 1–3 correspond to iterations 100/300/500 for the 1B & 1.5B class models and 100/200/300 for the 8B model.

Panel of Figure 3 — Figure 3: Ablations on Llama-3.2-1B-Instruct on UltraFeedback. Win rate relative to the initial instruction-tuned model is shown over cumulative oracle-query consumption $B$ for AERO and four single-component ablation variants.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Iteration-wise win rate against the initial instruction-tuned policy. Checkpoints are evaluated every 25 online training iterations.

Figure 5: Aleatoric filtering activity across training. Number of candidate pairs dropped per iteration by the aleatoric filter (Eq 5 ,

\alpha

= 0.6) using Llama-3.2-1B-Instruct on UltraFeedback, averaged across three seeds. The first 8 iterations are a burn-in period during which the filter is disabled.

Figure 6: Oracle win rate against estimated cumulative GPU-hours per seed on Llama-3.2-1B-Instruct and UltraFeedback.

Method	Generations	ERM-routed	Synthetic labels	Oracle queries	GPU-h	Wall-clock
AERO	3.840M	102.0K	99.7K	81.8K	$\sim$ 160	$\sim$ 27 h
SEA	3.840M	56.1K	56.1K	135.9K	$\sim$ 170	$\sim$ 28 h
COPO	0.576M	0	0	192.0K	$\sim$ 100	$\sim$ 17 h
Online DPO	0.384M	0	0	192.0K	$\sim$ 67	$\sim$ 11 h

Appendix

Table 2: Total resource usage for the primary three-seed Llama-3.2-1B-Instruct experiment. Generation, erm-routing, synthetic-label, oracle-query, and GPU-hour counts are totals across three seeds, whereas wall-clock time is per seed on two A100 40GB GPUs.

Model	Early	Middle	Final
Llama-3.2-1B	0.60	1.00	0.40
Qwen2.5-1.5B	0.40	1.00	1.00
Llama-3-8B	0.80	0.60	0.80

Appendix

Table 3: Within-stage Spearman agreement between win-rate gains evaluated using the Skywork-Reward-Llama-3.1-8B RM and AlpacaEval 2.0 length-controlled win-rate gains.

Method	Raw WR	Style-controlled WR
Initial policy	18.0	17.0 ( $-1.2/+1.7$ )
Online DPO	18.2	17.7 ( $-1.2/+1.3$ )
SEA	16.6	15.8 ( $-1.5/+1.3$ )
COPO	15.4	15.1 ( $-1.5/+1.2$ )
AERO	19.0	17.2 ( $-1.4/+1.8$ )

Appendix

Table 4: Arena-Hard results for the final Llama-3-8B checkpoints.

Figure 7: Adaptive versus fixed-quantile routing. Comparison of AERO’s adaptive Otsu routing with fixed-quantile routing using an oracle-routing ratio of

q=0.46

, evaluated on Llama-3.2-1B-Instruct with UltraFeedback and averaged across three seeds.

Iteration	Fixed $q=0.15$		Fixed $q=0.25$		AERO (Otsu)		Fixed $q=0.65$
Iteration	WR	$B$	WR	$B$	WR	$B$	WR	$B$
100	57.84	2.8K	59.88	3.9K	56.67–58.20	5.2–5.7K	58.91	8.4K
200	64.71	4.7K	62.47	7.1K	61.00–62.07	10.8–11.4K	63.24	16.4K
300	63.65	6.6K	64.10	10.3K	62.42–63.75	16.1–16.9K	63.59	24.2K
400	64.77	8.5K	65.43	13.5K	63.29–65.27	21.6–22.2K	64.56	32.1K
500	66.19	10.4K	65.58	16.7K	63.75–64.61	26.9–27.6K	66.80	39.8K

Appendix

Table 5: Single-seed sweep over fixed routing fractions

q

on Llama-3.2-1B-Instruct with UltraFeedback. As in Figure 2 , WR (%) is the held-out win rate against the initial policy judged by Skywork-Reward-Llama-3.1-8B , and

B

is the cumulative oracle queries. For AERO (Otsu, average fraction

\approx 0.46

) we report the range over three seeds. Per-iteration WR differs by only 1–2 points with no consistent ordering across

q

, while

B

scales with

q

; confirming this trend requires additional seeds.

Figure 8: Comparison of fixed-CFN and rolling-CFN variants of the COPO-adapted baseline on Llama-3.2-1B-Instruct with UltraFeedback. The fixed-CFN version is used in the main results because it performs slightly better in our fine-grained online setting.

Basic Parameters
Fine-tuning strategy	LoRA [ Hu et al., 2022 ] ( Llama-3-8B-Instruct ), Full parameter fine tuning ( Qwen2.5-1.5B-Instruct , Llama-3.2-1B-Instruct )
LoRA target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
LoRA $r$	64
LoRA alpha	64
LoRA dropout	0.05
Learning rate	$5\times 10^{-7}$ ( Qwen2.5-1.5B-Instruct , Llama-3.2-1B-Instruct ), $2\times 10^{-5}$ ( Llama-3-8B-Instruct )

Appendix

Table 14

Panel of Figure 9 — Figure 9: Single-seed results on TL;DR summarization [ Völske et al., 2017 ] using a 1B model from the Pythia family [ Biderman et al., 2023 ] . AERO performs comparably to SEA by iteration and achieves a higher win rate at matched oracle budgets. The oracle-budget plot shows that Otsu-based routing selects a smaller oracle budget in this easier summarization setting, suggesting that AERO adapts by routing more low-disagreement contexts to ERM synthetic labeling when the ensemble reaches consensus.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 628 papers

ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning
Mar 10, 2026Davit Melikidze, Marian Schneider, Jessica Lam +4Data Selection Active Learning
ETH Zurich · Institute of Neuroinformatics, University of Zurich and ETH Zurich · ETH AI Center
Oracle-Robust Online Alignment for Large Language Models
Feb 24, 2026Zimeng Li, Mudit Gaur, Vaneet AggarwalLLM Alignment Preference-Based RL
Purdue University
Dual Active Learning for Reinforcement Learning from Human Feedback
Oct 3, 2024Pangpang Liu, Chengchun Shi, Will Wei SunRL from Human Feedback Preference Learning for LLMs
Mitchell E. Daniels, Jr. School of Business, Purdue University · Department of Statistics, London School of Economics and Political Science

Mar 10, 2026J/K move · Enter open · S save

ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning

Davit Melikidze, Marian Schneider, Jessica Lam, Martin Wertich+3

ETH Zurich · Institute of Neuroinformatics, University of Zurich and ETH Zurich · ETH AI Center

Data Selection Active Learning RL from Human Feedback Uncertainty Sampling

Open paper AskPDF arXiv

Reinforcement Learning from Human Feedback (RLHF) has become the standard for aligning Large Language Models (LLMs), yet its efficacy is bottlenecked by the high cost of acquiring preference data, especially in low-resource and expert domains. To address this, we introduce ACTIVEULTRAFEEDBACK, a modular active learning pipeline that leverages uncertainty estimates to dynamically identify the most informative responses for annotation. Our pipeline facilitates the systematic evaluation of standard response selection methods alongside DOUBLE REVERSE THOMPSON SAMPLING (DRTS) and DELTAUCB, two novel methods prioritizing response pairs with large predicted quality gaps, leveraging recent results showing that such pairs provide good signals for fine-tuning. Our experiments demonstrate that ACTIVEULTRAFEEDBACK yields high-quality datasets that lead to significant improvements in downstream performance, notably achieving comparable or superior results with as little as one-sixth of the annotated data relative to static baselines. Our pipeline is available at https://github.com/lasgroup/ActiveUltraFeedback and our preference datasets at https://huggingface.co/ActiveUltraFeedback.

Abstract

LLM Alignment

Explore similar work

ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning

Oracle-Robust Online Alignment for Large Language Models

Dual Active Learning for Reinforcement Learning from Human Feedback