Auction-Based Online Policy Adaptation for Evolving Objectives

Guruprerana Shabadi; Kaushik Mallik

cs.LGApr 2, 2026

Auction-Based Online Policy Adaptation for Evolving Objectives

Authors: Guruprerana Shabadi, Kaushik Mallik

Organizations: University of Pennsylvania, United States · IMDEA Software Institute, Spain

Abstract

We consider multi-objective reinforcement learning problems where objectives come from an identical family -- such as the class of reachability objectives -- and may appear or disappear at runtime. Our goal is to design adaptive policies that can efficiently adjust their behaviors as the set of active objectives changes. To solve this problem, we propose a modular framework where each objective is supported by a selfish local policy, and coordination is achieved through a novel auction-based mechanism: policies bid for the right to execute their actions, with bids reflecting the urgency of the current state. The highest bidder selects the action, enabling a dynamic and interpretable trade-off among objectives. Going back to the original adaptation problem, when objectives change, the system adapts by simply adding or removing the corresponding policies. Moreover, as objectives arise from the same family, identical copies of a parameterized policy can be deployed, facilitating immediate adaptation at runtime. We show how the selfish local policies can be computed by turning the problem into a general-sum Markov game, where the policies compete against each other to fulfill their own objectives. To succeed, each policy must not only optimize its own objective, but also reason about the presence of other goals and learn to produce calibrated bids that reflect relative priority. Under mild assumptions, we prove the existence of Nash equilibria where dishonest bidding leads to suboptimal outcome, and the most urgent objectives win control automatically. In our implementation, the policies are trained concurrently using proximal policy optimization (PPO). We evaluate on two Atari games and a gridworld-based path-planning task with dynamic targets. Our method achieves substantially better performance than monolithic policies trained with PPO.

Figures & tables

Panel of Figure 2 — Figure 2: Comparison between the standard monolithic setting and our auction-based framework.

Panel of Figure 3 — Figure 3: Learning curves for different methods on Cat Feeder (left), Atari Assault (center), and Atari Air-Raid (right). We plot mean over 10 seeds at each evaluation iteration and the shaded region is $\pm$ standard deviation across seeds.

Method	Cat Feeder	Atari Assault	Atari Air-raid
Winner-Pays	$49.44$ $(4.19)$	$556.05$ $(29.23)$	3902.55 $(90.44)$
w. No Attn. Pool.	$51.72$ $(3.97)$	—	—
All-Pay	72.52 $(4.23)$	576.64 $(43.19)$	3870.45 $(76.72)$
w. No Attn. Pool.	$37.72$ $(50.32)$	—	—
DWN	-10.27 $(2.38)$	$310.80$ $(159.54)$	$2438.38$ $(625.97)$
Single-Policy PPO	—	358.60 $(25.99)$	3549.33 $(1170.14)$

Table 1: Scores (mean and standard deviation across 10 seeds) averaged over each seed’s last 5 evaluation iterations. Each evaluation is run every 10 PPO iterations using 20 rollouts. In Cat Feeder, score is the number of feeding requests completed minus the number that expired. Atari Assault and Atari Air-raid scores are the game scores.

Panel of Figure 4 — Figure 4: Comparison of bid distributions across environments and methods. Bars show aggregate bid counts for each bid value across all policies.

Panel of Figure 5 — Figure 5: Scaling number of targets and ablations on the Cat Feeder environment.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Control distributions across environments and methods. The control distribution is the number of timesteps controlled by each policy.

Panel of Figure 8 — Figure 8: Effect of the action-window size on performance.

Parameter	Value
Grid size	$30\times 30$
Number of agents / targets	8
Target reward	50.0
Target expiry steps	200
Target expiry penalty	50.0
Max episode steps	2000

Appendix

Table 2: Cat Feeder shared (training) environment parameters.

Auction
Parameter	Value
Bid upper bound	6
Bid penalty	0.1
Action window	5
Distance reward scale	0.6
Training

Appendix

Table 3: Cat Feeder multi-agent PPO hyperparameters ( All-Pay and Winner-Pays ).

Training
Parameter	Value
Iterations	400
Parallel environments	4096
Steps per rollout	256
Minibatches	512
PPO epochs	8

Appendix

Table 4: Cat Feeder single-agent PPO baseline hyperparameters. Hyperparameters were tuned using Optuna with 40 trials. The nearest-target shaping variant is identical but uses a distance reward scale of 0.6 with shaping toward the nearest target.

Training
Parameter	Value
Total timesteps	$5\times 10^{8}$
Parallel environments	256
Discount $\gamma$	0.99
Replay buffer size	$1\times 10^{6}$
Batch size	256

Appendix

Table 5: Cat Feeder DWN baseline hyperparameters.

Parameter	Value
Number of agents	3
Max enemies	3
Max episode steps	10000
Enemy destroy reward	10.0
Life loss penalty	10.0
Fire-while-hot penalty	8.0

Appendix

Table 6: Assault shared environment parameters.

Training
Parameter	Value
Iterations	150
Parallel environments	128
Steps per rollout	512
Minibatches	8
PPO epochs	8

Appendix

Table 7: Assault multi-agent PPO hyperparameters ( All-Pay and Winner-Pays ).

Parameter	Value
Learning rate	$2.5\times 10^{-4}$ (annealed)
Clip coefficient	0.1
Entropy coefficient	0.01
Target KL	—

Appendix

Table 8: Assault single-agent PPO baseline hyperparameters. Entries list differences from the multi-agent PPO setting; all omitted parameters are shared. Target KL early stopping is disabled for this baseline.

Training
Parameter	Value
Total timesteps	$1\times 10^{7}$
Parallel environments	8
Discount $\gamma$	0.99
Replay buffer size	$5\times 10^{5}$
Batch size	256

Appendix

Table 9: Assault DWN baseline hyperparameters.

Multi-Objective Reinforcement Learning

Multi-objective reinforcement learning studies how agents can balance competing goals and learn policies for different trade-offs.

See all 144 papers

Jun 27, 2026

A3M: Adaptive, Adversarial and Multi-Objective Learning for Strategic Bidding in Repeated Auctions

Learning to bid in repeated multi-unit auctions with bandit feedback poses a fundamental challenge. Existing methods often rely on rigid explore-then-exploit schedules, assume stationary adversaries, and optimize solely for bidder utility, thereby limiting adaptability and strategic robustness. To address these limitations, we introduce the A3M framework, which integrates adaptive deep reinforcement learning (DRL), explicit adversarial reasoning, and principled multi-objective reward design for online auction strategy optimization. A3M employs an actor-critic DRL backbone to dynamically balance exploration and exploitation, an opponent model for fictitious play against non-stationary adversaries, and a composite reward function to jointly maximize utility, auctioneer revenue, and fairness. We provide the first comprehensive empirical evaluation of this integrated approach against established baselines in both discriminatory and uniform price auctions. Results show that A3M reduces final regret by 30--40% in standard settings, maintains robust performance against adversarial strategy shifts, scales favorably with the number of units

K

, and enables tunable multi-objective trade-offs. An extensive ablation study confirms the necessity of each core component. Our work establishes A3M as a powerful and flexible framework for learning in complex auction environments.

Junhan Li, Yuxin Zhang, Haoran Wang +1

Department of Computer Science Nanjing University

Multi-Objective Reinforcement Learning Game Theory Automated Bidding Reward Design for RL

Ask PDF

Feb 24, 2026

Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI Feedback

Reward design has been one of the central challenges for real world reinforcement learning (RL) deployment, especially in settings with multiple objectives. Preference-based RL offers an appealing alternative by learning from human preferences over pairs of behavioural outcomes. More recently, RL from AI feedback (RLAIF) has demonstrated that large language models (LLMs) can generate preference labels at scale, mitigating the reliance on human annotators. However, existing RLAIF work typically focuses only on single-objective tasks, leaving the open question of how RLAIF handles systems that involve multiple objectives. In such systems trade-offs among conflicting objectives are difficult to specify, and policies risk collapsing into optimizing for a dominant goal. In this paper, we explore the extension of the RLAIF paradigm to multi-objective self-adaptive systems. We show that multi-objective RLAIF can produce policies that yield balanced trade-offs reflecting different user priorities without laborious reward engineering. We argue that integrating RLAIF into multi-objective RL offers a scalable path toward user-aligned policy learning in domains with inherently conflicting objectives.

Chenyang Zhao, Vinny Cahill, Ivana Dusparic

Trinity College Dublin, Dublin, Ireland

Multi-Objective Reinforcement Learning Preference-Based RL

Ask PDF

Oct 26, 2026

Multi-Objective Reinforcement Learning Using Routed Ensembles and Trajectory Attribution

Multi-objective reinforcement learning is a fundamental problem for modern AI, with applications spanning post-training, reasoning models, and robotic control. A central challenge is to learn policies that balance competing objectives, such as helpfulness versus honesty in safety and alignment, or speed versus energy in robotics. The optimal policy can vary sharply with objective-weighting combinations, and the extent of this variation depends crucially on the degree of conflict among objectives, making per-preference policy learning sample-inefficient. In this paper, we introduce objective-conflict coefficients and derive near-optimal regret bounds for online settings using this measure, thereby quantifying the impact of misalignment on learning efficiency. Then, we design routed ensemble policy learning, an algorithm that maintains a small set of basis policies and dynamically routes trajectories to the policy with the highest potential reward. Finally, we consider trajectory selection as another approach to mitigate objective conflict, and design a kernel surrogate trajectory attribution algorithm that estimates how feedback signals from a set of trajectories influence policy learning on other trajectories. The kernel can capture nonlinear interactions such as synergy and antagonism between trajectories. We evaluate our approach across diverse environments and benchmarks: routed ensembles improve average performance by 4.3% over baselines for online environments from Meta-World and MO-Gymnasium, and kernel surrogate attribution improves correlation with leave-one-out analysis over existing baselines by 50%. When applied to trajectory selection, kernel-based attribution yields a 27% improvement in reward over baseline attribution methods.

Zhenshuo Zhang, Ruoxuan Xiong, Hongyang R. Zhang

Northeastern University, Boston, MA · Emory University, Atlanta, GA

Multi-Objective Reinforcement Learning Reinforcement Learning

Ask PDF

Abstract

Multi-Objective Reinforcement Learning

Explore similar work

A3M: Adaptive, Adversarial and Multi-Objective Learning for Strategic Bidding in Repeated Auctions

Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI Feedback

Multi-Objective Reinforcement Learning Using Routed Ensembles and Trajectory Attribution