cs.AIAug 15, 2026

UC-PSRO: Utility-Conditioned Policy-Space Response Oracles with a Communication-Dropout Curriculum for Game-Theoretic Course-of-Action Generation in Adversarial Swarms

Authors: Phillip Jiang

Organizations: Appsofa LLC

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We study generating game-theoretically optimized Courses of Action (COAs) for a Blue UAS swarm against an adaptive Red adversary in a communication-degraded environment, motivated by (but not derived from) a public U.S. Air Force SBIR solicitation. We propose UC-PSRO (Utility-Conditioned Policy-Space Response Oracles with a Communication-Dropout Curriculum), combining three mechanisms: (i) PSRO self-play, so Blue and Red policies train as approximate best responses to each other rather than one side against a fixed scripted opponent; (ii) FiLM conditioning of the Blue policy on a Commander's-Intent weight vector, sampled from a Dirichlet distribution during training, so one trained policy is re-steerable at execution time without retraining; and (iii) a curriculum annealing communication-graph edge dropout during training, so the swarm learns decentralized, peer-to-peer fallback instead of depending on full connectivity. We evaluate on a synthetic, unclassified stand-in for the solicitation's maritime scenario, with 5 seeds at N=25 Blue agents and a scalability sweep to N=200. We find a genuine trade-off, not a uniform win: the communication-dropout curriculum alone gives the strongest, most robust mission-completion rates of any learned method, improving counter-intuitively as denial increases (35% to 62% success as dropout rises from 0 to 0.75); adding utility-conditioning and PSRO self-play substantially slows convergence within a fixed budget, and we find no reliable exploitability advantage for self-play over a fixed-opponent policy, both statistically indistinguishable from a small, near-zero gap. We report this honestly as a convergence cost not yet offset by a demonstrated robustness benefit, rather than overstating one method as dominant, and provide a fully vectorized, open environment training at N=200 agents in single-digit milliseconds per step on a single consumer GPU.

Figures & tables

Methodpdrop=0p_{\text{drop}}{=}0pdrop=0.25p_{\text{drop}}{=}0.25pdrop=0.5p_{\text{drop}}{=}0.5pdrop=0.75p_{\text{drop}}{=}0.75
Rule-based (1)1.00 ±\pm 0.001.00 ±\pm 0.001.00 ±\pm 0.001.00 ±\pm 0.00
MAPPO (2)0.27 ±\pm 0.290.23 ±\pm 0.310.28 ±\pm 0.280.35 ±\pm 0.25
MAPPO+CDC (3)0.35 ±\pm 0.310.40 ±\pm 0.300.50 ±\pm 0.290.62 ±\pm 0.27
MAPPO+Utility (4)0.00 ±\pm 0.000.00 ±\pm 0.000.00 ±\pm 0.000.04 ±\pm 0.08
UC-PSRO full (5)0.00 ±\pm 0.000.01 ±\pm 0.020.00 ±\pm 0.000.01 ±\pm 0.02
Table 1: Mission success rate (mean ±\pm std over 5 seeds) vs. test-time communication-edge dropout probability, headline scenario ( N=25N{=}25 , M=6M{=}6 ).
Figure 1: Mission success rate and swarm survivability vs. test-time communication dropout, headline scenario. MAPPO+CDC is the strongest learned method and, notably, its success rate rises with dropout rather than degrading.
NN102550100200
Red count26122448
Train wall-clock (h)0.510.811.132.025.06
Inference latency (ms/step)1.201.512.053.206.82
Table 2: UC-PSRO scalability: single-consumer-GPU training wall-clock and per-step inference latency vs. swarm size NN (Red count and map area scaled proportionally).
Figure 2: UC-PSRO scalability: inference latency and training wall-clock vs. NN , single consumer GPU.
Blue policyExploitability gap
MAPPO (trained vs. fixed scripted Red)−0.39±1.08-0.39\pm 1.08
UC-PSRO full (trained via self-play)+0.24±0.31+0.24\pm 0.31
Table 3: Exploitability gap: mean reward ±\pm std (over 20 episodes) achieved by a dedicated Red best response trained from scratch against a frozen Blue policy. Lower is less exploitable; 95% CIs (not shown) for both rows span or sit close to zero.
Figure 3: Exploitability gap under the fixed reward: both Blue policies are close to a small, near-zero gap, and the two are not clearly separated once training-time-consistent policies are compared.
IntentSurvivabilityTime-to-completeRed neutralizedSuccess rate
Balanced0.80200.00.780.00
Prioritize survivability0.81200.00.720.00
Prioritize speed0.81200.00.750.00
Prioritize neutralization0.78200.00.760.00
Table 4: Steerability of a single trained UC-PSRO policy, sweeping named Commander’s-Intent weight vectors without retraining.
Figure 4: Steerability sweep: survivability and Red-neutralized fraction across four named Commander’s-Intent vectors on a single trained UC-PSRO policy, no retraining. Differences across intents are present but small at this training budget.
Methodpdrop=0p_{\text{drop}}{=}0pdrop=0.25p_{\text{drop}}{=}0.25pdrop=0.5p_{\text{drop}}{=}0.5pdrop=0.75p_{\text{drop}}{=}0.75
MAPPO (2)0.36 ±\pm 0.430.34 ±\pm 0.400.36 ±\pm 0.370.44 ±\pm 0.37
MAPPO+CDC (3)0.71 ±\pm 0.310.71 ±\pm 0.270.79 ±\pm 0.170.89 ±\pm 0.15
MAPPO+Utility (4)0.00 ±\pm 0.000.00 ±\pm 0.000.00 ±\pm 0.000.00 ±\pm 0.00
UC-PSRO full (5)0.00 ±\pm 0.000.00 ±\pm 0.000.00 ±\pm 0.000.00 ±\pm 0.00
Table 5: Ablation table: component contributions on the headline comm-denial metric (mission success rate, mean ±\pm std over 5 seeds), an independent campaign from Table 1 run to check reproducibility.
Figure 5: Ablation campaign (independent 5 seeds from Figure 1 ): MAPPO+CDC reproduces its lead by a wide margin; both utility-conditioned methods reproduce exactly 0%0\% success at every dropout level.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Hidden dimension128
Learning rate3×10−43\times 10^{-4}
Discount γ\gamma0.99
GAE λ\lambda0.95
PPO clip ϵ\epsilon0.2
PPO epochs per update4
Actor minibatch size512
Appendix
Table 6: MAPPO / PSRO hyperparameters (defaults used for all reported experiments).
Blue agents / Red assets25 / 6 (3 IADS, 3 interceptors, 2 jammers)
Map size100×100100\times 100 (synthetic units)
Episode horizon200 steps
Sensor / comm range12 / 15
Blue engagement radius, neutralize prob.3, 0.5
IADS detection / engagement radius, kill prob.10 / 6, 0.08
Interceptor detection / engagement radius, kill prob.14 / 4, 0.12
Appendix
Table 7: Headline scenario parameters.
Methodpdrop=0p_{\text{drop}}{=}0pdrop=0.25p_{\text{drop}}{=}0.25pdrop=0.5p_{\text{drop}}{=}0.5pdrop=0.75p_{\text{drop}}{=}0.75
Rule-based (1)0.860.870.860.84
MAPPO (2)0.880.880.880.87
MAPPO+CDC (3)0.860.870.860.88
MAPPO+Utility (4)0.880.870.900.88
UC-PSRO full (5)0.840.850.890.90
Appendix
Table 8: Swarm survivability (fraction of initial Blue roster alive at episode end, mean over 5 seeds), headline scenario. Unlike mission success (Table 1 ), survivability is similar across all learned methods once the reward fix of § 7 is applied, since risk-avoidance is no longer the dominant term separating methods’ behavior.
Updates0–100100–200200–300300–400400–500500–600
Original reward−- 0.28−- 0.10−- 0.06−- 0.02−- 0.01++ 0.04
Fixed reward, τ=0.5\tau{=}0.5−- 0.29−- 0.17−- 0.01++ 0.01++ 0.06++ 0.00
Fixed reward, τ=0.3\tau{=}0.3−- 0.29−- 0.17−- 0.01 ∗++ 0.01 ∗++ 0.06 ∗++ 0.00 ∗
Appendix
Table 9: Single-seed diagnostic runs on the headline scenario, chunked mean scalar reward over training. “Original reward” is the pre-fix decomposition (§ 7 ); “fixed reward, τ=0.5\tau{=}0.5 ” adds the target-relative observation and dense at-target bonus but keeps the original 50% survivor threshold; “fixed reward, τ=0.3\tau{=}0.3 ” additionally relaxes the threshold. Mission success rate (not shown) was exactly 0 in every chunk of the first two rows and became nonzero only in the last row, peaking at 2.6% in updates 400–500.
NN102550100200
Before (ms/step)—7.3343.81309.842851.5
After (ms/step)1.201.512.053.206.82
Speedup—4.9 ×\times21.4 ×\times96.8 ×\times418 ×\times
Appendix
Table 10: Per-step environment latency before and after vectorizing the communication-graph, engagement-roll, and observation-construction hot paths from nested Python loops to batched NumPy array operations (§ 6.2 ), single consumer GPU host (env stepping itself is CPU-bound regardless of GPU).
More in this topic

Multi-Agent Coordination

Multi-agent coordination studies how autonomous agents align actions, share information, and divide work to achieve individual or collective goals.

See all 517 papers

Explore similar work

CardsList
May 18, 2026

Shared Backbone PPO for Multi-UAV Communication Coverage with Connection Preservation

This paper proposes a Shared Backbone Proximal Policy Optimization (Shared Backbone PPO) algorithm. By sharing the base module between the Actor and Critic networks, the algorithm achieves efficient training and improved performance. The algorithm is implemented in a connectivity-preserving multi-UAV swarm communication coverage task and compared with the standard PPO algorithm. Experimental results demonstrate that the proposed method achieves superior performance. Furthermore, a graph information aggregation module is incorporated into the model architecture to accommodate the communication conditions among agents. With the integration of this module, the algorithm remains effective, and the trained agent swarm exhibits a higher level of cooperation.

Z. Jiang

Zhejiang University

Multi-Robot SystemsProximal Policy OptimizationCooperative MARLMulti-Agent Reinforcement Learning
AskPDF
Mar 17, 2026

Communication-Aware Multi-Agent Reinforcement Learning for Decentralized Cooperative UAV Deployment

Autonomous Unmanned Aerial Vehicle (UAV) swarms are increasingly used as rapidly deployable aerial relays and sensing platforms, yet practical deployments must operate under partial observability and intermittent peer-to-peer connectivity. We present a graph-based multi-agent reinforcement learning framework trained under centralized training with decentralized execution (CTDE): a centralized critic and global state are available only during training, while each UAV executes a shared policy using local observations and messages from nearby neighbors. Under restricted communication, neighbor relations are induced by an SNR-threshold connectivity graph. Our architecture encodes local agent state and nearby entities with an agent-entity attention module and aggregates inter-UAV messages with neighbor self-attention over a signal-quality-limited communication graph defined by a channel model. We evaluate the framework on a cooperative relay-deployment task, DroneConnect. Experimental results show that the proposed method achieves an approximately 12% increase in target coverage over MAPPO under restricted communication and partial observability, while remaining competitive with a mixed-integer linear programming (MILP)-based offline upper bound with full node observability.

Enguang Fan, Yifan Chen, Zihan Shan +3

University of Illinois at Urbana-Champaign · Boeing Research and Technology

Cooperative MARLDecentralized MARLMulti-Agent CommunicationPartially Observable RL
AskPDF
Sep 11, 2026

Curriculum-Based Adversarial Heterogeneous Agent Reinforcement Learning for Autonomous Quad-Copter Landing in Maritime Settings

Recovering unmanned aerial vehicles (UAVs) in maritime environments is challenging due to wind turbulence and ship-deck motion, making it a valuable test case for alternative control and learning approaches as conventional landing approaches often become unreliable. We study simulated mid-air capture of quadrotor UAVs by a ship-mounted robotic arm, learning robust cooperative control policies with Heterogeneous-Agent Proximal Policy Optimization (HAPPO) Reinforcement Learning. We train with HAPPO using a curriculum and an adversarial wind agent (HARL-AC) in NVIDIA Isaac Lab, and compare the obtained control policies against those generated through curriculum-based domain randomization and a benchmark trained on a single sea state. In-distribution evaluation on sea states 0/4/50/4/5 shows comparable success for HARL-AC and domain randomization of up to 97.5%97.5\%. On out-of-distribution sea states 7/8/107/8/10, HARL-AC generalizes better, achieving up to 16%16\% higher median success rate at sea state 10, and substantially lower crash rates of up to 14%14\% compared to the domain randomization policy. Furthermore, we show that the adversarially trained policy shows more cautious behavior, slightly increasing timeouts by <3%<3\%, but yields safer recovery behavior in severe, unseen conditions.

Allan Minh-Tam Nguyen, Sree Showrya Kotala, Stefan Banioi-Crijman +2

Department of Advanced Computing Sciences, Maastricht University, Minderbroedersberg 4-6, 6211 LK Maastricht, Netherlands

Adversarial TrainingAerial RoboticsUAV Landing on Moving PlatformsQuadrotor ControlRobust RLHeterogeneous MARL
AskPDF