Game-Theory-Assisted Reinforcement Learning for Border Defense: Early Termination based on Analytical Solutions

Goutam Das; Michael Dorothy; Kyle Volle; Daigo Shishika

cs.LGMar 16, 2026

Game-Theory-Assisted Reinforcement Learning for Border Defense: Early Termination based on Analytical Solutions

Authors: Goutam Das, Michael Dorothy, Kyle Volle, Daigo Shishika

Organizations: School of Aeronautics and Astronautics, Purdue University, 610 Purdue Mall, West Lafayette, IN 47907, USA · Army Research Directorate, DEVCOM Army Research Laboratory, APG, MD 21005, USA · Munitions Directorate, Air Force Research Laboratory, Eglin AFB, FL 32542, USA · Department of Mechanical Engineering, George Mason University, 4400 University Dr, Fairfax, VA 22030, USA

Abstract

Game theory provides the gold standard for analyzing adversarial engagements, offering strong optimality guarantees. However, these guarantees often become brittle when assumptions such as perfect information are violated. Reinforcement learning (RL), by contrast, is adaptive but can be sample-inefficient in large, complex domains. This paper introduces a hybrid approach that leverages game-theoretic insights to improve RL training efficiency. We study a border defense game with limited perceptual range, where defender performance depends on both search and pursuit strategies, making classical differential game solutions inapplicable. Our method employs the Apollonius Circle (AC) to compute equilibrium in the post-detection phase, enabling early termination of RL episodes without learning pursuit dynamics. This allows RL to concentrate on learning search strategies while guaranteeing optimal continuation after detection. Across single- and multi-defender settings, this early termination method yields 10-20% higher rewards, faster convergence, and more efficient search trajectories. Extensive experiments validate these findings and demonstrate the overall effectiveness of our approach.

Figures & tables

Fig. 1: Illustration of the border-defense game environment. An attacker (

A

) spawns in the designated area at the top and attempts to reach the target line (

\mathcal{T}

) at the bottom. The defender team, composed of agents with varying sensing (

D_{1}

D_{2}

) and capture (

D_{2}

D_{3}

) capabilities, must coordinate to intercept the attacker.

Fig. 2: Geometric representation of the optimal capture strategy in the deterministic pursuit phase (Phase II). The Apollonius Circle (

\mathcal{A}

) defines the boundary of the attacker’s dominance region. The optimal capture point is the location on the circle with the minimum y-coordinate, and the optimal trajectories for both agents converge at this point.

Fig. 3: Optimal capture point computation for a multi-defender scenario. The attacker’s reachable region is the intersection of the individual Apollonius Circles (

\mathcal{A}_{1},\mathcal{A}_{2}

). The optimal interception point, marked by the red star, corresponds to the minimum y-coordinate within this intersection, representing the game-theoretic payoff.

Fig. 4: Value function level-set for optimal defender placement. The contours illustrate the Nash equilibrium payoff ( $J^{*}$ ) as a function of a third defender’s position, given two fixed defenders. Blue regions indicate configurations that yield a better payoff for the attacker, while red regions are more favorable for the defenders.

Parameter	Value
Domain	$\Omega=[0,1]\times[0,1]$
Target boundary	$\mathcal{T}=\{(x,y)\in\Omega\mid y=0\}$
Attacker spawn region	$\Omega_{A}=[0,1]\times[0.9,1.0]$
Speed ratio	$\nu=v_{D_{i}}/v_{A}=3.33$
Sensing radius	$\rho_{s}\in\{0.15,0.3,0.35\}$
Capture radius	$\rho_{c}=0.07$

TABLE I: Environment Parameters

Fig. 5: Training performance in the 1v1 homogeneous scenario. The GT-assisted policy (blue) converges faster and to a significantly higher mean reward (

\mu=0.605

) than the end-to-end baseline (red,

\mu=0.545

), demonstrating a 10% performance improvement.

Fig. 6: Training performance in the 3v1 homogeneous scenario. The GT-assisted approach (blue) again shows superior performance, achieving a stable mean reward of

\mu=0.745

with lower variance. The end-to-end method (red) converges to a lower reward of

\mu=0.626

Fig. 7: Evaluation of spatial configuration quality for the 3v1 homogeneous scenario (

\rho_{s}=0.15

) over 1,000 test episodes. (a) The GT-assisted policy achieves a much higher sensing rate (67.1%) compared to the baseline (45.2%). (b) This leads to superior cumulative performance. (c) For episodes where sensing occurred, the GT-assisted policy achieves a distribution of Apollonius payoffs with a significantly higher median (

0.768

vs.

0.701

), indicating that it learns to position defenders more strategically at the moment of sensing.

Game-Theoretic RL

See all 28 papers

PolicyGuard: Towards Test-time and Step-level Adversary (Backdoor) Defense for Reinforcement Learning Agent
Jun 11, 2026Junfeng Guo Heng HuangBackdoor Attacks Backdoor Defense
Computer Science, University of Maryland College Park, USA.
Shielded Analysis: Certification and Characterization of Defensibility in Systems under Adversarial Interaction
Jun 11, 2026Achraf Hsain, Sultan AlmuhammadiAutonomous Cyber Defense Adversarial Robustness
Information and Computer Science Department King Fahd University of Petroleum and Minerals Dhahran, Saudi Arabia
Plan2Cleanse: Test-Time Backdoor Defense via Monte-Carlo Planning in Deep Reinforcement Learning
May 10, 2026Sze-Ann Chen, Zhi-Yi Chin, Kui-Yuan Chen +2Backdoor Detection Backdoor Defense
National Yang Ming Chiao Tung University, Hsinchu, Taiwan · CISPA Helmholtz Center for Information Security

Jun 11, 2026J/K move · Enter open · S save

PolicyGuard: Towards Test-time and Step-level Adversary (Backdoor) Defense for Reinforcement Learning Agent

Junfeng Guo Heng Huang

Computer Science, University of Maryland College Park, USA.

Backdoor Attacks Backdoor Defense Robust RL

Open paper AskPDF arXiv

While real-world applications of reinforcement learning (RL) are becoming increasingly popular, the security of RL systems deserve more attention and exploration. In particular, recent work has revealed that RL agents are vulnerable to backdoor attacks, where a victim agent behaves normally under standard conditions but executes malicious actions when a specific trigger is activated. Existing backdoor defenses for RL either require access to the agent's internal parameters, operate only at the model or trajectory level, or are limited to specific attack types. To ensure the security of RL agents, we propose PolicyGuard, a test-time step-level backdoor defense which leverages Gaussian Process (GP) posterior variance and adapts pseudo trajectories to enable uncertainty computation for individual time step. Besides, we also provide theoretical foundations to explain the efficacy of GP posterior variance. Extensive experiments across seven RL games demonstrate that PolicyGuard achieves state-of-the-art detection performance in most cases, with average AUROC of 0.856 for perturbation-based attacks and 0.859 for adversary-agent attacks.

Abstract

Game-Theoretic RL

Explore similar work

PolicyGuard: Towards Test-time and Step-level Adversary (Backdoor) Defense for Reinforcement Learning Agent

Shielded Analysis: Certification and Characterization of Defensibility in Systems under Adversarial Interaction

Plan2Cleanse: Test-Time Backdoor Defense via Monte-Carlo Planning in Deep Reinforcement Learning