cs.LGMar 19, 2026

Best-of-Both-Worlds Multi-Dueling Bandits: Unified Algorithms for Stochastic and Adversarial Preferences under Condorcet and Borda Objectives

Authors: S Akash, Pratik Gajane, Jawar Singh

Organizations: Indian Institute of Technology Patna, India · University of Orléans, France

Abstract

Multi-dueling bandits, where a learner selects m≥2m \geq 2 arms per round and observes only the winner, arise naturally in many applications including ranking and recommendation systems, yet a fundamental question has remained open: can a single algorithm perform optimally in both stochastic and adversarial environments, without knowing which regime it faces? We answer this affirmatively, providing the first best-of-both-worlds algorithms for multi-dueling bandits under both Condorcet and Borda objectives. For the Condorcet setting, we propose MetaDueling\texttt{MetaDueling}, a black-box reduction that converts any dueling bandit algorithm into a multi-dueling bandit algorithm by transforming multi-way winner feedback into an unbiased pairwise signal. Instantiating our reduction with Versatile-DB\texttt{Versatile-DB} yields the first best-of-both-worlds algorithm for multi-dueling bandits: it achieves O(KT)O(\sqrt{KT}) pseudo-regret against adversarial preferences and the instance-optimal O(∑i≠a⋆log⁡TΔi)O\left(\sum_{i \neq a^\star} \frac{\log T}{Δ_i}\right) pseudo-regret under stochastic preferences, both simultaneously and without prior knowledge of the regime. For the Borda setting, we propose SA-MiDEX\texttt{SA-MiDEX}, a stochastic-and-adversarial algorithm that achieves O(K2log⁡KT+Klog⁡2T+∑i:ΔiB>0Klog⁡KT(ΔiB)2)O\left(K^2 \log KT + K \log^2 T + \sum_{i: Δ_i^{\mathrm{B}} > 0} \frac{K\log KT}{(Δ_i^{\mathrm{B}})^2}\right) regret in stochastic environments and O(KTlog⁡KT+K1/3T2/3(log⁡K)1/3)O\left(K \sqrt{T \log KT} + K^{1/3} T^{2/3} (\log K)^{1/3}\right) regret against adversaries, again without prior knowledge of the regime. We complement our upper bounds with matching lower bounds for the Condorcet setting. For the Borda setting, our upper bounds are near-optimal with respect to the lower bounds (within a factor of KK) and match the best-known results in the literature.

Explore similar work

May 3, 2026

Multi-User Dueling Bandits: A Fair Approach using Nash Social Welfare

Learning from human preference data is becoming a useful tool, from fine-tuning large language models to training reinforcement learning agents. However, in most scenarios, the model is trained on the average preference of all human evaluators, which, under large variations of preferences, can be unfair to minority groups. In this work, we consider fairness in dueling bandits, a standard framework for online learning from preference data. We assume that each user has a (potentially distinct) Condorcet winner, which is an arm preferred to every other arm. Using these user-specific Condorcet winners as reference points, we evaluate and score arms according to their performance relative to the corresponding winner. To promote fairness across heterogeneous users, we adopt the well-established Nash Social Welfare objective, which maximizes the product of user utilities, thereby inherently penalizing inequality and preventing the marginalization of any single user. Within this framework, we construct a hard instance to establish a regret lower bound of Ω(T2/3min⁡(K,D)13)Ω(T^{2/3}\min(K,D)^\frac{1}{3}) for a time horizon TT, KK arms, and DD users, which, to the best of our knowledge, is the first result quantifying the cost of fairness in dueling bandits with heterogeneous preferences. We then present the Fair-Explore-Then-Commit and Fair-εε-Greedy algorithms with a Condorcet winner identification phase. We further derive their regret upper bounds that match the lower-bound dependence on TT up to logarithmic factors.
Apr 9, 2026

Analysis of Search Heuristics in the Multi-Armed Bandit Setting

We consider the classic Multi-Armed Bandit setting to understand the exploration/exploitation tradeoffs made by different search heuristics. Since many search heuristics work by comparing different options (in evolutionary algorithms called "individuals"; in the Bandit literature called "arms"), we work with the "Dueling Bandits" setting. In each iteration, a comparison between different arms can be made; in the binary stochastic setting, each arm has a fixed winning probability against any other arm. A Condorcet winner is any arm that beats every other arm with a probability strictly higher than 1/21/2. We show that evolutionary algorithms are rather bad at identifying the Condorcet winner: Even if the Condorcet winner beats every other arm with a probability 1−p1-p, the (1+1) EA, in its stationary distribution, chooses the Condorcet winner only with constant probability if p=Ω(1/n)p=Ω(1/n). By contrast, we show that a simple EDA (based on the Max-Min Ant System with iteration-best update) will choose the Condorcet winner in its maintained distribution with probability 1−Θ(p)1-Θ(p). As a remedy for the (1+1) EA, we show how repeated duels can significantly boost the probability of the Condorcet winner in the stationary distribution.
Jun 1, 2026

Tree-Guided Identify-Then-Exploit: A Unified Framework of Best Arm Identification and Regret Minimization for Dueling Bandits

We study NN-armed stochastic dueling bandits under the Condorcet-winner assumption, where three widely adopted objectives are considered: best-arm identification (BAI), weak regret, and strong regret. We propose Tree-Guided Identify-Then-Exploit (TG-ITE), the first unified framework to tackle all these objectives to our knowledge. Without requiring stronger assumptions, we propose a shared tree-guided identification approach to find a high-confidence incumbent within O(N)O(N) comparisons. We further propose varied exploitation strategies to utilize this warm-start stage to optimize the specific objectives at hand. This methodology enables our approach to (1) achieve O(N)O(N) sample complexity in BAI without commonly adopted stronger assumptions; (2) build the first winner-stays-style algorithm to achieve O(N)O(N) weak regret; (3) enjoy the same O(Nlog⁡T)O(N \log T) guarantee as specialized strong-regret approaches; (4) realize the joint optimization of BAI and weak regret with O(N)O(N) guarantees for both, eliminating the sub-optimal gap of O(log⁡N)O(\log N) in the existing approach. Our results provide evidence that the trade-off between BAI and regret minimization is relatively benign in dueling bandits.