Efficient Exploration at Scale

Seyed Mohammad Asghari; Chris Chute; Vikranth Dwaracherla; Xiuyuan Lu; Mehdi Jafarnia; Victor Minden; Zheng Wen; Benjamin Van Roy

cs.LGMar 18, 2026

Efficient Exploration at Scale

Authors: Seyed Mohammad Asghari, Chris Chute, Vikranth Dwaracherla, Xiuyuan Lu, Mehdi Jafarnia, Victor Minden, Zheng Wen, Benjamin Van Roy

Organizations: The Efficient Agent Team, Google DeepMind

Abstract

We develop an online learning algorithm that dramatically improves the data efficiency of reinforcement learning from human feedback (RLHF). Our algorithm incrementally updates reward and language models as choice data is received. The reward model is fit to the choice data, while the language model is updated by a variation of reinforce, with reinforcement signals provided by the reward model. Several features enable the efficiency gains: a small affirmative nudge added to each reinforcement signal, an epistemic neural network that models reward uncertainty, and information-directed exploration. With Gemma large language models (LLMs), our algorithm matches the performance of offline RLHF trained on 200K labels using fewer than 20K labels, representing more than a 10x gain in data efficiency. Extrapolating from our results, we expect our algorithm trained on 1M labels to match offline RLHF trained on 1B labels. This represents a 1,000x gain. To our knowledge, these are the first results to demonstrate that such large improvements are possible.

Figures & tables

Figure 1: The plots are of performance, in terms of the win rate over a baseline policy, as functions of the amount of human feedback, in terms of the number of choices observed. Efficient exploration shifts the scaling law.

Figure 2: Feedback in the form of a choice between two responses produced is used to improve the policy.

Figure 3: Given responses generated by competing and baseline policies, the human feedback simulator produces a preference probability.

Figure 4: Strong performance of online RLHF relies on a reward function (left) and an affirmative nudge (right).

Panel of Figure 5 — Figure 5: A neural network reward model versus an epistemic neural network reward model.

Figure 6: Inference pathway for the point estimate ( $Z=0$ ).

Figure 7: Inference pathway for an ensemble particle ( $Z=1,\ldots,N$ ).

Figure 8: Win rate against the baseline policy as a function of the amount of feedback collected to adapt the language model.

Panel of Figure 9 — Figure 9: (a) Extrapolation of win rates attained by efficient exploration and offline RLHF. We fit a function of the form $w(n)=1-0.5(n/a)^{-b}$ , with parameters $a$ and $b$ . (b) Projected gain in data efficiency relative to offline RLHF. The projected gain is computed using extrapolations from (a). With 1M labels, the efficient exploration algorithm is predicted to attain a 1,000x gain in data efficiency.

RL from Human Feedback

Reinforcement learning from human feedback (RLHF) trains AI systems to better match human preferences using comparisons, corrections, and other feedback.

See all 122 papers

Data-dependent Exploration for Online Reinforcement Learning from Human Feedback
May 6, 2026Zhen-Yu Zhang, Yuting Tang, Jiandong Zhang +2RL Exploration Sample-Efficient RL
Center for Advanced Intelligence Project, RIKEN · Graduate School of Frontier Sciences, The University of Tokyo · Northeastern University +1
Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learning
Mar 4, 2026Lei Huang, Xiang Cheng, Chenxiao Zhao +6RL Exploration RL for Language Models
Harbin Institute of Technology · Xiaohongshu Inc.
ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning
Mar 10, 2026Davit Melikidze, Marian Schneider, Jessica Lam +4Data Selection Active Learning
ETH Zurich · Institute of Neuroinformatics, University of Zurich and ETH Zurich · ETH AI Center

May 6, 2026J/K move · Enter open · S save

Data-dependent Exploration for Online Reinforcement Learning from Human Feedback

Zhen-Yu Zhang, Yuting Tang, Jiandong Zhang, Lanjihong Ma+1

Center for Advanced Intelligence Project, RIKEN · Graduate School of Frontier Sciences, The University of Tokyo · Northeastern University+1

RL Exploration Sample-Efficient RL Preference-Based RL RL from Human Feedback

Open paper AskPDF arXiv

Online reinforcement learning from human feedback (RLHF) has emerged as a promising paradigm for aligning large language models (LLMs) by continuously collecting new preference feedback during training. A foundational challenge in this setting is exploration, which requires algorithms that enable the LLMs to generate informative comparisons that improve sample-efficiency in online RLHF. Existing exploration strategies often derive bonuses via on-policy expectations, which are difficult to estimate reliably from the limited historical preference data available during training; as a result, the policy can prematurely down-weight under-explored regions that may contain high-value behaviors. In this paper, we propose data-dependent exploration for preference optimization (DEPO), a simple and scalable method that leverages historical data to construct an extra uncertainty bonus for high-uncertainty regions, encouraging exploration toward potentially high-value data. Theoretically, we provide a data-dependent regret bound for the proposed algorithm, showing that it adapts to the hardness of the learning task itself and can be tighter than worst-case bounds in practice. Empirically, the proposed method consistently outperforms strong baselines across benchmarks, demonstrating improved sample efficiency.

Abstract

RL from Human Feedback

Explore similar work

Data-dependent Exploration for Online Reinforcement Learning from Human Feedback

Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learning

ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning