cs.MAOct 4, 2026

Orchestrating Level-KK Policies Against Unknown Opponents in Partially-Observable Dynamic Games

Authors: Addison Kalanther, Sanika Bharvirkar, Daniel Bostwick, Chinmay Maheshwari, Shankar Sastry

Organizations: EECS, University of California, Berkeley, CA, USA · ECE, Johns Hopkins University, Baltimore, MD, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Level-KK reasoning generates a hierarchy of policies specialized to opponents with different reasoning levels. When an opponent's level is unknown, a common deployment rule estimates that level and selects the corresponding response. In a dynamic, partially-observed game, this selection is repeated, with each choice shaping subsequent states and observations. The response associated with the most likely opponent level need not maximize expected return from the current history. We formulate this deployment problem as dynamic orchestration of a fixed, pretrained policy library in a partially observable Markov game. We compare classification-based orchestrators (CBOs) trained using offline data or on-policy data aggregation with a reinforcement-learning-based orchestrator (RLBO) trained to maximize expected discounted return. In pursuit-evasion experiments, on-policy training improves classification and return, yet RLBO achieves higher return than the on-policy and offline CBOs. Given a pretrained library, RLBO also reaches performance comparable to a policy trained directly over the pursuers' action space with fewer training timesteps. These findings support treating a hierarchy of level-KK policies as a resource for orchestration, not a prescription for deployment.

Figures & tables

 k^t∈arg⁡max⁡kqψ(k∣ht).\hat{k}_{t}\in\arg\max_{k}q_{\psi}(k\mid h_{t}). 
Fig. 1 : On-policy opponent classifier training using DAgger-style dataset aggregation.
Fig. 2 : Partially observable pursuit–evasion environment. Pink stars denote possible evader origin and destination locations. The HLP, LLP, and EDR are shown by a blue diamond, blue triangle, and red triangle, respectively. Shaded regions denote their fields of view. Gray, orange, yellow, and green cells correspond to obstacle Types I–IV.
Fig. 3 : Expected discounted return across orchestration strategies.
Fig. 4 : Expected discounted return as a function of orchestration training timesteps.
Fig. 5 : Opponent-level classification accuracy as a function of trajectory timestep.
Fig. 6 : Frequency with which RLBO selects the expert originally trained as a best response to the true opponent level. The dashed line denotes uniform random expert selection.
Fig. 7 : Expected discounted return of the Time-Delayed Oracle as a function of the opponent-level reveal time.
Fig. 8 : Pursuer win rates for different pursuer–evader policy pairs.

Explore similar work

CardsList

  1. Minimax-Optimal Policy Regret in Partially Observable Markov Games

    Jun 1, 2026Raman AroraRegret Minimization in RLImperfect-Information Games

    Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA.

  2. Hierarchical Belief Modeling for Zero-Shot Opponent Adaptation in Partially Observable Multi-Agent Navigation

    Sep 11, 2026Kowei Shih, Lu Cheng, Zeyu Wang +2Opponent ModelingPartially Observable RL

    Tsinghua University, Beijing, China · Stevens Institute of Technology, New Jersey, USA · University of California, Los Angeles, Los Angeles, USA +2

  3. Learning Robust Penetration Testing Policies under Partial Observability: A systematic evaluation

    Sep 24, 2025Raphael Simon, Pieter Libin, Wim MeesReinforcement LearningPartially Observable RL

    Cyber Defence Lab, CISS Department Royal Military Academy · AI Lab, Department of Computer Science Vrije Universiteit Brussel