Game-Theory-Assisted Reinforcement Learning for Border Defense: Early Termination based on Analytical Solutions
Authors: Goutam Das, Michael Dorothy, Kyle Volle, Daigo Shishika
Organizations: School of Aeronautics and Astronautics, Purdue University, 610 Purdue Mall, West Lafayette, IN 47907, USA · Army Research Directorate, DEVCOM Army Research Laboratory, APG, MD 21005, USA · Munitions Directorate, Air Force Research Laboratory, Eglin AFB, FL 32542, USA · Department of Mechanical Engineering, George Mason University, 4400 University Dr, Fairfax, VA 22030, USA
Game theory provides the gold standard for analyzing adversarial engagements, offering strong optimality guarantees. However, these guarantees often become brittle when assumptions such as perfect information are violated. Reinforcement learning (RL), by contrast, is adaptive but can be sample-inefficient in large, complex domains. This paper introduces a hybrid approach that leverages game-theoretic insights to improve RL training efficiency. We study a border defense game with limited perceptual range, where defender performance depends on both search and pursuit strategies, making classical differential game solutions inapplicable. Our method employs the Apollonius Circle (AC) to compute equilibrium in the post-detection phase, enabling early termination of RL episodes without learning pursuit dynamics. This allows RL to concentrate on learning search strategies while guaranteeing optimal continuation after detection. Across single- and multi-defender settings, this early termination method yields 10-20% higher rewards, faster convergence, and more efficient search trajectories. Extensive experiments validate these findings and demonstrate the overall effectiveness of our approach.
Figures & tables
Fig. 1: Illustration of the border-defense game environment. An attacker ( A ) spawns in the designated area at the top and attempts to reach the target line ( T ) at the bottom. The defender team, composed of agents with varying sensing ( D1 , D2 ) and capture ( D2 , D3 ) capabilities, must coordinate to intercept the attacker.
Fig. 2: Geometric representation of the optimal capture strategy in the deterministic pursuit phase (Phase II). The Apollonius Circle ( A ) defines the boundary of the attacker’s dominance region. The optimal capture point is the location on the circle with the minimum y-coordinate, and the optimal trajectories for both agents converge at this point.
Fig. 3: Optimal capture point computation for a multi-defender scenario. The attacker’s reachable region is the intersection of the individual Apollonius Circles ( A1,A2 ). The optimal interception point, marked by the red star, corresponds to the minimum y-coordinate within this intersection, representing the game-theoretic payoff.
Fig. 4: Value function level-set for optimal defender placement. The contours illustrate the Nash equilibrium payoff ( J∗ ) as a function of a third defender’s position, given two fixed defenders. Blue regions indicate configurations that yield a better payoff for the attacker, while red regions are more favorable for the defenders.
Parameter
Value
Domain
Ω=[0,1]×[0,1]
Target boundary
T={(x,y)∈Ω∣y=0}
Attacker spawn region
ΩA=[0,1]×[0.9,1.0]
Speed ratio
ν=vDi/vA=3.33
Sensing radius
ρs∈{0.15,0.3,0.35}
Capture radius
ρc=0.07
TABLE I: Environment Parameters
Fig. 5: Training performance in the 1v1 homogeneous scenario. The GT-assisted policy (blue) converges faster and to a significantly higher mean reward ( μ=0.605 ) than the end-to-end baseline (red, μ=0.545 ), demonstrating a 10% performance improvement.
Fig. 6: Training performance in the 3v1 homogeneous scenario. The GT-assisted approach (blue) again shows superior performance, achieving a stable mean reward of μ=0.745 with lower variance. The end-to-end method (red) converges to a lower reward of μ=0.626 .
Fig. 7: Evaluation of spatial configuration quality for the 3v1 homogeneous scenario ( ρs=0.15 ) over 1,000 test episodes. (a) The GT-assisted policy achieves a much higher sensing rate (67.1%) compared to the baseline (45.2%). (b) This leads to superior cumulative performance. (c) For episodes where sensing occurred, the GT-assisted policy achieves a distribution of Apollonius payoffs with a significantly higher median ( 0.768 vs. 0.701 ), indicating that it learns to position defenders more strategically at the moment of sensing.
While real-world applications of reinforcement learning (RL) are becoming increasingly popular, the security of RL systems deserve more attention and exploration. In particular, recent work has revealed that RL agents are vulnerable to backdoor attacks, where a victim agent behaves normally under standard conditions but executes malicious actions when a specific trigger is activated. Existing backdoor defenses for RL either require access to the agent's internal parameters, operate only at the model or trajectory level, or are limited to specific attack types. To ensure the security of RL agents, we propose PolicyGuard, a test-time step-level backdoor defense which leverages Gaussian Process (GP) posterior variance and adapts pseudo trajectories to enable uncertainty computation for individual time step. Besides, we also provide theoretical foundations to explain the efficacy of GP posterior variance. Extensive experiments across seven RL games demonstrate that PolicyGuard achieves state-of-the-art detection performance in most cases, with average AUROC of 0.856 for perturbation-based attacks and 0.859 for adversary-agent attacks.
Junfeng Guo Heng Huang
Computer Science, University of Maryland College Park, USA.
Formal safety analysis determines whether a system admits a safe defense; adaptive evaluation characterizes the operating quality sustained under adversarial interaction. Both answers matter because systems with the same safety verdict can impose very different operational burdens. We introduce shielded analysis, a design-time framework that derives these answers from one encoded system while keeping the safety requirement and admissible threat model independently variable. It returns a defensibility certificate and a four-axis defensibility fingerprint spanning structural margin, shield latitude, and adaptive operating quality. Each axis is informative in its own right; their relationships show whether formal and operational assessments agree, diverge, or respond differently to system changes. We instantiate the framework for network defense on a reference segment and four controlled perturbations spanning topology, safety requirements, and adversary capabilities. Every configuration is certified defensible, yet two topology variants with nearly identical structural profiles sustain mean clean-host fractions of 22.7% and 80.7% under adaptive pressure. Shielded analysis turns a safety-game solution into a comparative instrument: it determines whether a defense exists, characterizes what that defense requires, and identifies which system changes strengthen it.
Achraf Hsain, Sultan Almuhammadi
Information and Computer Science Department King Fahd University of Petroleum and Minerals Dhahran, Saudi Arabia
Ensuring the security of reinforcement learning (RL) models is critical, particularly when they are trained by third parties and deployed in real-world systems. Attackers can implant backdoors into these models, causing them to behave normally under typical conditions, but execute malicious behaviors when specific triggers are activated. In this work, we propose Plan2Cleanse, a test-time detection and mitigation framework that adapts Monte Carlo Tree Search to efficiently identify and neutralize RL backdoor attacks without requiring model retraining. Our approach recasts backdoor detection as a planning problem, enabling systematic exploration of temporally extended trigger sequences while maintaining black-box access to the target policy. By leveraging the detection results, Plan2Cleanse can further achieve efficient mitigation through tree-search preventive replanning. We evaluated our method in competitive MuJoCo environments, simulated O-RAN wireless networks, and Atari games. Plan2Cleanse achieves substantial improvements, increasing trigger detection success rates by more than 61.4 percentage points in stealthy O-RAN scenarios and improving win rates from 35% to 53% in competitive Humanoid environments. These results demonstrate the effectiveness of our test-time defense approach and highlight the importance of proactive defenses against backdoor threats in RL deployments. Our implementation is publicly available at https://github.com/rl-bandits-lab/RL-Backdoor.
Sze-Ann Chen, Zhi-Yi Chin, Kui-Yuan Chen +2
National Yang Ming Chiao Tung University, Hsinchu, Taiwan · CISPA Helmholtz Center for Information Security