Information-Guided Safe Reinforcement Learning for Autonomous Gas Source Localization using sUAS
Authors: Sachin Giri, Thomas Zhao, Matthew Huynh, YangQuan Chen
Organizations: Electrical Engineering & Computer Science Graduate Program, University of California, Merced, 5200 N. Lake Rd, Merced, CA 95343, USA · Dept. of Mechanical Engineering, University of California, Merced, 5200 N. Lake Rd, Merced, CA 95343, USA
The autonomous localization of fugitive gas emissions using small Unmanned Aircraft Systems (sUAS) constitutes a fundamentally ill-posed inverse problem. In turbulent atmospheric boundary layers, highly intermittent scalar concentration fields violate the assumptions of classical gradient-based navigation, causing data-driven estimators to suffer from severe noise and spurious local minima. To address these challenges, we introduce an Information-Guided Safe Reinforcement Learning framework evaluated within a custom, GPU-accelerated 3D simulation environment coupling an Eulerian wind solver with a Lagrangian puff dispersion model. We identify a critical vulnerability in deterministic information-seeking planners - a Gramian bias where agents act greedily upon flawed early estimates, starving the estimator of spatial diversity. To systematically break this degeneracy, our architecture integrates a classical empirical observability Gramian (EMGR) planner with a learned Soft Actor-Critic (SAC) exploratory policy. A deterministic meta-supervisor actively monitors estimator reliability via Kullback-Leibler (KL) divergence, dynamically blending deterministic exploitation with learned exploration to steer the sUAS into high-information zones. Trained via a progressive curriculum and safeguarded by a strictly enforced Robust Control Barrier Function (RCBF), our RL framework achieves nearly 80% localization success on complex, mobile sources - drastically outperforming classical baselines (~30%) - while ensuring zero safety violations.
Figures & tables
Fig. 1: 3D simulation environment built with NVIDIA Warp, demonstrating the Lagrangian puff dispersion model (color-coded by concentration) navigating around physical obstacles.
Fig. 2: 2D cross-sections of the coupled simulation environment.
Fig. 3: Sequential execution pipeline. Sensor data feeds parallel estimators whose fused estimate drives both the classical EMGR planner and the learned SAC policy. The meta-supervisor allocates blending authority ωkact ; the safety layer enforces collision avoidance before actuation.
Method
Success
Med. Dist. [5th, 95th]
Med. Steps
MLE+EMGR
30.4%
24.3 [4.7, 135.8] m
1173
MLE+EMGR+PF+KL
32.1%
25.0 [4.8, 159.3] m
2000
Stage 1 (small)
98.8%
4.9 [4.7, 5.0] m
256
Stage 2 (full)
79.3%
4.9 [4.7, 43.8] m
310
Stage 3 (mobile)
78.6%
4.8 [4.5, 59.2] m
276
TABLE I: Localization Performance (100 episodes, dconv=5 m)
Fig. 4: Median distance to source with interquartile bands. RL agents converge below dconv=5m within 300 steps; baselines stagnate due to Gramian bias.
Fig. 5: Cumulative success rate over tracking steps.
We address robust separation assurance for small Unmanned Aircraft Systems (sUAS) under GPS degradation and spoofing via Multi-Agent Reinforcement Learning (MARL). In cooperative surveillance, each aircraft (or agent) broadcasts its GPS-derived position; when such position broadcasts are corrupted, the entire observed air traffic state becomes unreliable. We cast this state observation corruption as a zero-sum game between the agents and an adversary: with probability R, the adversary perturbs the observed state to maximally degrade each agent's safety performance. We derive a closed-form expression for this adversarial perturbation, bypassing the iterative inner optimization of adversarial training entirely and enabling linear-time evaluation in the state dimension. We show that this expression approximates the exact minimizer of the value function over the modeled uncertainty set with second-order accuracy. We further bound the safety performance gap between clean and corrupted observations, showing that it degrades at most linearly with the corruption probability under Kullback-Leibler regularization. Finally, we integrate the closed-form adversarial policy into a MARL policy gradient algorithm to obtain a robust counter-policy for the agents. In a high-density sUAS simulation, we observe near-zero collision rates under corruption levels up to 35%, outperforming a baseline policy trained without adversarial perturbations.
Alex Zongo, Filippos Fotiadis, Ufuk Topcu +1
A. Zongo and P. Wei are with the Department of Mechanical and Aerospace Engineering, George Washington University, Washington, D.C. 20052, USA. · F. Fotiadis and U. Topcu are with the Oden Institute for Computational Engineering & Sciences, University of Texas at Austin, Austin, TX 78712, USA.
Undocumented orphaned wells pose significant health and environmental risks to nearby communities by releasing toxic gases and contaminating water sources, with methane emissions being a primary concern. Traditional survey methods such as magnetometry often fail to detect older wells effectively. In contrast, aerial in-situ sensing using unmanned aerial vehicles (UAVs) offers a promising alternative for methane emission detection and source localization. This study presents a robust and efficient framework based on a multi-agent deep reinforcement learning (MARL) algorithm for the chemical plume source localization (CPSL) problem. The proposed approach leverages virtual anchor nodes to coordinate UAV navigation, enabling collaborative sensing of gas concentrations and wind velocities through onboard and shared measurements. Source identification is achieved by analyzing the historical trajectory of anchor node placements within the plume. Comparative evaluations against the fluxotaxis method demonstrate that the MARL framework achieves superior performance in both localization accuracy and operational efficiency.
Zhirun Li, Derek Hollenbeck, Ruikun Wu +4
Department of Electrical and Computer Engineering, University of New Mexico, Albuquerque, NM 87131 USA · Department of Mechanical Engineering, University of California, Merced, CA 95343 USA · Department of Electrical Engineering, Colorado School of Mines, Golden, CO 80401 USA +3
With the rapid development of autonomous aerial systems, Unmanned Aerial Vehicles (UAVs) are increasingly deployed in applications such as inspection, environmental monitoring, and rescue, creating growing demand for reliable autonomous navigation. However, autonomous UAV navigation in dense environments remains challenging under sparse perception and dynamic constraints. Most reinforcement learning (RL) methods lack explicit safety mechanisms, leading to unsafe exploration, unstable training, and risky behaviors, especially during high-speed flight. Even in safe RL approaches, safety is often enforced by projecting policy outputs onto a safe action set, which may introduce instability. Meanwhile, many learning-based methods rely on dense inputs or large networks, increasing computational burden and limiting lightweight onboard deployment. Facing the above challenges, we propose a safety-constrained perception-control integrated framework for UAV navigation. A lightweight network encodes sparse observations into collision-risk-aware features using asymmetric and depthwise separable convolutions. We formulate the task as a constrained Markov decision process within a hierarchical control architecture and solve it using a Lagrangian-based safe PPO algorithm. Curriculum learning further improves training stability. Experiments with varying obstacle densities and flight speeds demonstrate higher success rates, improved safety, and better efficiency than existing reinforcement learning baselines.