We consider multi-objective reinforcement learning problems where objectives come from an identical family -- such as the class of reachability objectives -- and may appear or disappear at runtime. Our goal is to design adaptive policies that can efficiently adjust their behaviors as the set of active objectives changes. To solve this problem, we propose a modular framework where each objective is supported by a selfish local policy, and coordination is achieved through a novel auction-based mechanism: policies bid for the right to execute their actions, with bids reflecting the urgency of the current state. The highest bidder selects the action, enabling a dynamic and interpretable trade-off among objectives. Going back to the original adaptation problem, when objectives change, the system adapts by simply adding or removing the corresponding policies. Moreover, as objectives arise from the same family, identical copies of a parameterized policy can be deployed, facilitating immediate adaptation at runtime. We show how the selfish local policies can be computed by turning the problem into a general-sum Markov game, where the policies compete against each other to fulfill their own objectives. To succeed, each policy must not only optimize its own objective, but also reason about the presence of other goals and learn to produce calibrated bids that reflect relative priority. Under mild assumptions, we prove the existence of Nash equilibria where dishonest bidding leads to suboptimal outcome, and the most urgent objectives win control automatically. In our implementation, the policies are trained concurrently using proximal policy optimization (PPO). We evaluate on two Atari games and a gridworld-based path-planning task with dynamic targets. Our method achieves substantially better performance than monolithic policies trained with PPO.
Figures & tables
Figure 1: Cat Feeder env.
Figure 2: Comparison between the standard monolithic setting and our auction-based framework.
Figure 3: Learning curves for different methods on Cat Feeder (left), Atari Assault (center), and Atari Air-Raid (right). We plot mean over 10 seeds at each evaluation iteration and the shaded region is ± standard deviation across seeds.
Method
Cat Feeder
Atari Assault
Atari Air-raid
Winner-Pays
49.44(4.19)
556.05(29.23)
3902.55 (90.44)
w. No Attn. Pool.
51.72(3.97)
—
—
All-Pay
72.52 (4.23)
576.64 (43.19)
3870.45 (76.72)
w. No Attn. Pool.
37.72(50.32)
—
—
DWN
-10.27 (2.38)
310.80(159.54)
2438.38(625.97)
Single-Policy PPO
—
358.60 (25.99)
3549.33 (1170.14)
Table 1: Scores (mean and standard deviation across 10 seeds) averaged over each seed’s last 5 evaluation iterations. Each evaluation is run every 10 PPO iterations using 20 rollouts. In Cat Feeder, score is the number of feeding requests completed minus the number that expired. Atari Assault and Atari Air-raid scores are the game scores.
Figure 4: Comparison of bid distributions across environments and methods. Bars show aggregate bid counts for each bid value across all policies.
Figure 5: Scaling number of targets and ablations on the Cat Feeder environment.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Atari Assault environment.
Figure 7: Control distributions across environments and methods. The control distribution is the number of timesteps controlled by each policy.
Figure 8: Effect of the action-window size on performance.
Table 4: Cat Feeder single-agent PPO baseline hyperparameters. Hyperparameters were tuned using Optuna with 40 trials. The nearest-target shaping variant is identical but uses a distance reward scale of 0.6 with shaping toward the nearest target.
Table 8: Assault single-agent PPO baseline hyperparameters. Entries list differences from the multi-agent PPO setting; all omitted parameters are shared. Target KL early stopping is disabled for this baseline.