cs.LGMar 10, 2026

ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning

Authors: Davit Melikidze, Marian Schneider, Jessica Lam, Martin Wertich, Ido Hakimi, Barna Pásztor, Andreas Krause

Organizations: ETH Zurich · Institute of Neuroinformatics, University of Zurich and ETH Zurich · ETH AI Center

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reinforcement Learning from Human Feedback (RLHF) has become the standard for aligning Large Language Models (LLMs), yet its efficacy is bottlenecked by the high cost of acquiring preference data, especially in low-resource and expert domains. To address this, we introduce ACTIVEULTRAFEEDBACK, a modular active learning pipeline that leverages uncertainty estimates to dynamically identify the most informative responses for annotation. Our pipeline facilitates the systematic evaluation of standard response selection methods alongside DOUBLE REVERSE THOMPSON SAMPLING (DRTS) and DELTAUCB, two novel methods prioritizing response pairs with large predicted quality gaps, leveraging recent results showing that such pairs provide good signals for fine-tuning. Our experiments demonstrate that ACTIVEULTRAFEEDBACK yields high-quality datasets that lead to significant improvements in downstream performance, notably achieving comparable or superior results with as little as one-sixth of the annotated data relative to static baselines. Our pipeline is available at https://github.com/lasgroup/ActiveUltraFeedback and our preference datasets at https://huggingface.co/ActiveUltraFeedback.

Figures & tables

Figure 1: Comparison of response pair selection methods on downstream and reward model benchmarks deployed in ActiveUltraFeedback . The scores have been averaged over four datasets (see Section 5.4 ) of different scales, and indicate improvement over the base model. * denotes an existing dueling bandit method and † indicates our novel active delta learning methods.
Figure 2: The ActiveUltraFeedback pipeline. For each prompt, responses are generated from a large pool of LLMs, the rewards for the responses are predicted with corresponding uncertainties, and a pair of responses is selected for preference annotation. Each new batch of preference data is used to train the reward model, improving the accuracy of reward and uncertainty estimates for subsequent iterations. The displayed procedure is performed in a looping manner until all prompts have been processed.
Methods# Responses to Annotate
Baseline Heuristics 
Random2
MaxMinmm
UltraFeedback ( Cui et al., 2024 )4
DeltaQwen ( Geng et al., 2025 )0
Dueling Bandit Methods 
Table 1: Overview of response pair selection methods and the number of responses that need to be annotated per prompt. † indicates the methods that we propose.
MethodGSM8KIFEvalTruthfulQAAlpacaEval 2MeanRewardBench 2
Base Model+ 0.758+ 0.713+ 0.468+ 0.083+ 0.506+ 0.290
Original+0.039+0.025+0.055+0.030+0.037+0.295
Random+0.024+0.028+0.056+0.077+0.046+0.278
UltraFeedback+0.037-0.001+0.039+0.072+0.036+0.287
MaxMin+0.022-0.016+0.150+0.289+0.111+0.318
DeltaQwen+0.055+0.047+0.130+0.316+0.137+0.100
Table 2: Comparison between all response pair selection methods, based on the reward model and fine-tuned model (DPO) performance after training the same base model on each generated dataset. The base model score is given for reference, and all scores are reported as relative deltas to it, with higher values indicating better performance. We also provide the deltas achieved with the original response pairs in UltraFeedback . †\dagger denotes our proposed methods. Best scores are marked in bold.
Figure 3: Mean performance trajectories for fine-tuned and reward models as a function of consumed samples on the ActiveUltraFeedback prompt pool using DPO. All curves share the same prompts and differ only in the response pair selection strategy. For readability, we subset the selection strategies; for a full comparison, see Figure 10 in the Appendix. We compare datasets generated via ActiveUltraFeedback using various selection methods, and also report the score achieved by the original UltraFeedback dataset ( Cui et al., 2024 ) with its original response pairs.
Figure 4: Benchmarking of downstream and reward model performance across input prompt datasets, increasing in scale from left to right. Scores are reported as relative deltas to the base model. We provide the scores achieved using the original preference dataset instead of just the prompts with ActiveUltraFeedback for reference.
Figure 5: Mean performance trajectories for our fine-tuned models using IPO ( Figure 5(a) ) and SimPO ( Figure 5(b) ) as a function of consumed samples on datasets generated using ActiveUltraFeedback based on UltraFeedback prompts. For readability, we subset the selection strategies, for a full comparison, see Figure 11 in the Appendix. We provide the scores achieved using the original preference dataset instead of just the prompts with ActiveUltraFeedback for reference.
Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Model# ParametersLicense
Qwen/Qwen2.5-0.5B-Instruct0.5BApache 2.0
Qwen/Qwen2.5-72B-Instruct72BQwen
Qwen/Qwen3-0.6B0.6BApache 2.0
Qwen/Qwen3-1.7B1.7BApache 2.0
Qwen/Qwen3-14B14BApache 2.0
Qwen/Qwen3-30B-A3B30BApache 2.0
Appendix
Table 3: The 30 models used for response generation with their total number of parameters (in billions) and licenses. Separators are placed between models from different families.
ConfigurationWin RateTie RateParse Errors
Probabilistic Scoring76.70%0.0%0
Discrete Generation75.36%14.7%275
Probabilistic Scoring + Explicit Reasoning73.54%11.3%120
Discrete Generation + Explicit Reasoning73.37%12.1%20,181
Appendix
Table 4: Comparison of the four experimental judging configurations using the Qwen/Qwen3-235B-A22B model on the UltraFeedback dataset ( N=60′829N=60^{\prime}829 ). Win Rate measures the percentage of samples where the judge assigned a strictly higher overall score to the preferred response. Ties occur when the calculated overall score is identical for both responses. The Probabilistic Scoring configuration (without reasoning) was selected for the final annotation pipeline due to its superior alignment, reliability, and speed.
ConfigurationTie RateMean
Probabilistic Scoring0.8330.744
Discrete Scoring0.7290.698
Appendix
Table 5: Comparison of Probabilistic vs Discrete Scoring using Qwen/Qwen3-235B-A22B on RewardBench 2 ( Malik et al., 2025 ) .
ConfigurationSpearman’s rrHuman Agreement
Probabilistic Scoring0.6762.7%
Discrete Scoring0.7060.4%
Appendix
Table 6: Comparison of Probabilistic vs Discrete Scoring using Qwen/Qwen3-235B-A22B on AlpacaEval 2.0 ( Dubois et al., 2024 ) .
ModelFactualityFocusMathPrecise IFSafetyTiesMean
Qwen3-32B0.7870.8400.7100.3430.8440.8630.731
Qwen3-235B-A22B0.8510.7920.6890.3690.9310.8330.744
Llama-3.3-70B-Instruct0.6920.7530.6830.4370.8060.8660.706
Skywork-Reward-V2-Llama-3.1-8B0.8440.9830.7700.6560.9670.8120.839
Appendix
Table 7: Rewardbench 2 scores for our judge using different models as judge models. With this comparison, we aim to cover a wide range of model sizes to examine how model size affects annotation quality. We also added Skywork-Reward-V2-Llama-3.1-8B, the current rank 1 on RewardBench 2, as a reference.
MethodGSM8KIFEvalTruthfulQAAlpacaEval 2MeanRewardBench 2
DRTS+0.057±0.009+0.057_{\pm 0.009}+0.025±0.017+0.025_{\pm 0.017}+0.132±0.010+0.132_{\pm 0.010}+0.246±0.007+0.246_{\pm 0.007}+0.114±0.006+0.114_{\pm 0.006}+0.277±0.025+0.277_{\pm 0.025}
DeltaUCB+0.058±0.009+0.058_{\pm 0.009}+0.017±0.009+0.017_{\pm 0.009}+0.103±0.007+0.103_{\pm 0.007}+0.230±0.012+0.230_{\pm 0.012}+0.101±0.006+0.101_{\pm 0.006}+0.282±0.011+0.282_{\pm 0.011}
Appendix
Table 8: Stability of ActiveUltraFeedback across 5 different random seeds with two response pair selection methods. We report the mean and standard deviation for each benchmark. Scores are reported as relative deltas to the base model.
MetricGSM8KIFEvalTruthfulQAAlpacaEval 2Mean
Mean+0.039+0.020+0.056+0.028+0.035
Std. Dev.0.0050.0060.0010.0060.003
Appendix
Table 9: Training stability across 5 different random seeds. We report the mean and standard deviation for each benchmark. Scores are reported as relative deltas to the base model.
MetricGSM8KIFEvalTruthfulQAAlpacaEval 2Mean
Mean+0.044+0.020+0.054+0.030+0.035
Std. Dev.0.0030.0110.0000.0080.004
Appendix
Table 10: Training stability across 5 runs with a fixed seed (Seed 42), assessing system-level non-determinism. Scores are reported as relative deltas to the base model.
MetricFactualityFocusMathPrecise IFSafetyTiesMean
Mean+0.344+0.495+0.145+0.095+0.453+0.253+0.298
Std. Dev.0.0190.0290.0300.0310.0360.0720.011
Appendix
Table 11: Reward Model training stability across 5 different random seeds. Scores are reported as relative deltas to the base model.
MetricFactualityFocusMathPrecise IFSafetyTiesMean
Mean+0.363+0.444+0.145+0.128+0.546+0.252+0.292
Std. Dev.0.0050.0060.0070.0070.0060.0080.004
Appendix
Table 12: Reward Model stability across 5 runs with a fixed seed (Seed 42). Scores are reported as relative deltas to the base model.
BenchmarkGSM8KIFEvalTruthfulQAAlpacaEvalMean
Mean Delta+0.033+0.019+0.058+0.273+0.095
Std. Dev.0.0110.0090.0060.0360.015
Appendix
Table 13: Stability analysis of our SimPO algorithms setup. We report the Mean and Standard Deviation across 5 different random seeds. Scores are reported as relative deltas to the base model.
BenchmarkGSM8KIFEvalTruthfulQAAlpacaEvalMean
Mean Delta+0.048+0.035+0.040+0.304+0.106
Std. Dev.0.0040.0050.0050.0360.011
Appendix
Table 14: Stability analysis of our IPO algorithms setup. We report the Mean and Standard Deviation across 5 different random seeds. Scores are reported as relative deltas to the base model.
HyperparameterResponse GenerationPreference Annotation
Temperature1.00.0
Top- pp1.0–
Max Response Tokens409616
Appendix
Table 15: Sampling parameters for Response Generation and Preference Annotation in ActiveUltraFeedback .
HyperparameterValue
Number of MLP heads20
Number of layers per MLP head2
Hidden size of each MLP head128
Appendix
Table 16: Hyperparameters for the ENN architecture.
HyperparameterValue
Max Length (Prompt + Response)4096
Batch Size, ∣B∣\lvert\mathcal{B}\rvert64
Train Steps100
Initial Regularization, ζ\zeta1.0
Reward Centering Coefficient, γ\gamma0.01
Learning Rate5×10−55\times 10^{-5}
Appendix
Table 17: Fixed hyperparameters used across experiments for ENN training.
HyperparameterGrid ValuesInfoMaxDTSMaxMinLCBDRTSDeltaUCB
Optimal for DPO Fine-Tuning
Beta β\beta[1, 2]21112
Regularization Decay[0.9, 0.99, 0.999]0.990.990.990.9990.999
Replay Buffer Size Factor, ρ\rho[100, 1000]10001000100010001000
Optimal for Reward Modeling
Beta β\beta[1, 2]21211
Appendix
Table 18: ENN training hyperparameters and sweep ranges for each active response pair selection method. Separate optimal values were chosen based on performance after DPO fine-tuning and on RewardBench 2.
HyperparameterGrid ValuesChosen Value
For DPO
Learning Rate[ 1×10−61\times 10^{-6} , 2×10−52\times 10^{-5} , 5×10−45\times 10^{-4} ]2×10−52\times 10^{-5}
Lambda λ\lambda[0.1, 0.01]0.1
Epochs[1, 3]3
For IPO
Learning Rate[ 5×10−65\times 10^{-6} , 1×10−51\times 10^{-5} , 2×10−52\times 10^{-5} , 5×10−55\times 10^{-5} ]5×10−65\times 10^{-6}
Appendix
Table 19: Optimal hyperparameters for our DPO, IPO, and SimPO fine-tuning, selected based on evaluation performance.
HyperparameterGrid ValuesChosen Value
Learning Rate[ 3×10−63\times 10^{-6} , 5×10−65\times 10^{-6} , 2×10−52\times 10^{-5} ]2×10−52\times 10^{-5}
Epochs[1, 2, 3]2
Appendix
Table 20: Optimal hyperparameters for reward model training, selected based on RewardBench 2 performance.
HyperparametersChosen Value
Rank64
Alpha16
Dropout0.1
Target Modulesall-linear
Appendix
Table 21: Hyperparameters for our LoRA setup.
StepEstimated GPU Hours
Response Generation600
Annotation600
Active Learning Loop32
Appendix
Table 22: Compute estimates for each step of ActiveUltraFeedback , estimated in GPU hours.
MethodMean Chosen ScoreMean Rejected ScoreMean Score
Random4.5223.5644.043
UltraFeedback4.7473.8104.279
MaxMin4.9251.6053.625
DeltaQwen4.5492.9243.736
InfoMax3.6663.1563.411
DTS4.8554.5844.720
Appendix
Table 23: Mean score of the chosen, rejected, and overall responses from different response pair selection methods on the UltraFeedback prompts.
Figure 6: Comparison between the number of times each model from our model pool ( Section B.1 ) has been selected as chosen and rejected model on the UltraFeedback prompts for all response pair selection methods we consider.
Figure 7: Comparison between the number of times each model from our model pool ( Section B.1 ) has been selected as chosen and rejected model on the UltraFeedback prompts for all response pair selection methods we consider.
Figure 8: Comparison between the number of times each model from our model pool ( Section B.1 ) has been selected as chosen and rejected model on the UltraFeedback prompts for all response pair selection methods we consider.
Figure 9: Results for the sample efficiency experiment ( Section 5.3 ). We compare the aggregate scores with and without AlpacaEval 2 to demonstrate how its larger magnitude dominates the mean across all benchmarks.
MethodGSM8KIFEvalTruthfulQAAlpacaEval 2MeanRewardBench 2
Base Model+ 0.758+ 0.713+ 0.468+ 0.083+ 0.506+ 0.290
UltraFeedback Prompts
Original+0.039+0.025+0.055+0.030+0.037+0.295
Random+0.024+0.028+0.056+0.077+0.046+0.278
UltraFeedback+0.037-0.001+0.039+0.072+0.036+0.287
MaxMin+0.022-0.016+0.150+0.289+0.111+0.318
Appendix
Table 24: Results of ActiveUltraFeedback on downstream and reward model benchmarks using different prompt input datasets and response pair selection methods. All scores are given as relative deltas to the base model’s scores for readability. Best scores are in bold. We furthermore show the scores obtained by training on the actual UltraFeedback , Skywork , and Tulu 3 preference mixture datasets.
AlgorithmMethodGSM8KIFEvalTruthfulQAAlpacaEval 2Mean
–Base Model+ 0.758+ 0.713+ 0.468+ 0.083+ 0.506
DPORandom+0.024+0.028+0.056+0.077+0.046
UltraFeedback+0.037-0.001+0.039+0.072+0.036
MaxMin+0.022-0.016+0.150+0.289+0.111
DeltaQwen+0.055+0.047+0.130+0.316+0.137
InfoMax+0.011+0.019+0.018+0.020+0.016
Appendix
Table 25: Results of ActiveUltraFeedback on downstream benchmarks using different preference tuning algorithms and response pair selection methods. All scores are given as relative deltas to the base model’s scores for readability. Best score highlighted in bold.
Figure 10: Mean performance trajectories for fine-tuned and reward models as a function of consumed samples on the ActiveUltraFeedback prompt pool using DPO. All curves share the same prompts and differ only in the response pair selection strategy. We compare datasets generated via ActiveUltraFeedback using various selection methods, and also report the score achieved by the original UltraFeedback dataset ( Cui et al., 2024 ) with its original response pairs.
Figure 11: Mean performance trajectories for our fine-tuned models using IPO ( Figure 11(a) ) and SimPO ( Figure 11(b) ) as a function of consumed samples on datasets generated using ActiveUltraFeedback based on UltraFeedback prompts. We provide the scores achieved using the original preference dataset instead of just the prompts with ActiveUltraFeedback for reference.
Model Pool CompositionGSM8KIFEvalTruthfulQAAlpacaEval 2Mean
All 30 models+0.060+0.019+0.129+0.240+0.112
15 largest models only+0.011-0.001-0.003+0.059+0.017
15 smallest models only+0.050+0.036+0.100+0.105+0.073
5 largest + 5 smallest models+0.070+0.025+0.114+0.152+0.090
Appendix
Table 26: Sensitivity of DRTS to the size and diversity of the response model pool. Scores are reported as deltas relative to the base model after DPO training.
More in this topic

Data Selection

Data selection chooses informative, representative, or reliable examples to improve model training and evaluation while reducing data, compute, or labeling costs.

See all 79 papers

Explore similar work

CardsList

  1. CoAct: Co-Active LLM Preference Learning with Human-AI Synergy

    Apr 19, 2026Ruiyao Xu, Mihir Parmar, Tiankai Yang +3RL for Language ModelsLLM Alignment

    Northwestern University · Google · University of Southern California +1

  2. Reinforcement Learning from Human Feedback: A Statistical Perspective

    Apr 2, 2026Pangpang Liu, Chengchun Shi, Will Wei SunRL from Human FeedbackLLM Alignment

    Department of Biostatistics, Yale University. · Department of Statistics, London School of Economics and Political Science. · Mitch Daniels School of Business, Purdue University.

  3. Your Mouse and Eyes Secretly Leak Your Preference: LLM Alignment using Implicit Feedback from Users

    Jun 18, 2026Haw-Shiuan Chang, Jeffrey Gomez, Mehul Patwari +2Reward ModelingLLM Alignment

    University of Massachusetts, Amherst, USA · York University, Canada