Smart Commander: A Hierarchical Reinforcement Learning Framework for Fleet-Level PHM Decision Optimization

Yong Si; Mingfei Lu; Jing Li; Yang Hu; Guijiang Li; Yueheng Song; Zhaokui Wang

cs.LGApr 8, 2026

Smart Commander: A Hierarchical Reinforcement Learning Framework for Fleet-Level PHM Decision Optimization

Authors: Yong Si, Mingfei Lu, Jing Li, Yang Hu, Guijiang Li, Yueheng Song, Zhaokui Wang

Organizations: School of Aerospace Engineering, Tsinghua University, China · College of Information and Control Engineering, Xi’an University of Architecture and Technology, China · Hangzhou International Innovation Institute, Beihang University, China · First Aircraft Institute of Aviation Industry Corporation of China, China ·

Abstract

Decision-making in military aviation Prognostics and Health Management (PHM) faces significant challenges due to the "curse of dimensionality" in large-scale fleet operations, combined with sparse feedback and stochastic mission profiles. To address these issues, this paper proposes Smart Commander, a novel Hierarchical Reinforcement Learning (HRL) framework designed to optimize sequential maintenance and logistics decisions. The framework decomposes the complex control problem into a two-tier hierarchy: a strategic General Commander manages fleet-level availability and cost objectives, while tactical Operation Commanders execute specific actions for sortie generation, maintenance scheduling, and resource allocation. The proposed approach is validated within a custom-built, high-fidelity discrete-event simulation environment that captures the dynamics of aircraft configuration and support logistics.By integrating layered reward shaping with planning-enhanced neural networks, the method effectively addresses the difficulty of sparse and delayed rewards. Empirical evaluations demonstrate that Smart Commander significantly outperforms conventional monolithic Deep Reinforcement Learning (DRL) and rule-based baselines. Notably, it achieves a substantial reduction in training time while demonstrating superior scalability and robustness in failure-prone environments. These results highlight the potential of HRL as a reliable paradigm for next-generation intelligent fleet management.

Figures & tables

Figure 1. Hierarchical decision-making architecture of the Smart Commander framework. The General Commander operates at the strategic level, coordinating global fleet objectives, while Operation Commanders function at the tactical level, managing specific subsystems (flight operations, maintenance, and resources).

Figure 2. Training dynamics under nominal conditions. The Smart Commander (HRL, purple) converges faster than DRL (orange) across all metrics. Top row: availability rate (

r_{ab}

) and cost-benefit ratio (

r_{cb}

). Middle row: mission success rate (

r_{ms}

) and sortie success rate (

r_{ss}

). Bottom row: total cost (

C_{\text{total}}

) and virtual cost-benefit ratio (

r_{vcb}

Figure 3. Training rewards under nominal conditions. Top: General Commander reward (

R^{G}

). Bottom: Operation Commanders’ rewards (

R^{F}

R^{M}

R^{R}

). The HRL framework achieves quicker convergence and higher rewards compared to DRL.

Metric	Rule-Based	DRL	HRL (Ours)
$r_{ab}$ (%)	$92.3\pm 2.1$	$94.5\pm 1.8$	$\mathbf{96.2\pm 0.9}$
$r_{ms}$ (%)	$81.6\pm 3.5$	$87.3\pm 2.2$	$\mathbf{92.1\pm 1.3}$
$r_{ss}$ (%)	$86.9\pm 2.8$	$90.7\pm 1.9$	$\mathbf{93.5\pm 1.1}$
$ttc$ (k)	$1150\pm 210$	$1890\pm 723$	$\mathbf{1230\pm 109}$
$r_{cb}$	$2.3\pm 0.03$	$1.22\pm 0.02$	$\mathbf{0.75\pm 0.02}$
$r_{vcb}$	–	$4.35\pm 0.25$	$\mathbf{0.05\pm 0.01}$

Table 1. Performance comparison under nominal conditions. Values are mean

\pm

std over 5 random seeds. Best results in bold .

Figure 4. Scalability analysis under varying system complexity (

\lambda

). The Smart Commander (HRL, purple) maintains high performance as complexity increases, while DRL (orange) and Rule-Based (green) methods degrade significantly.

Figure 5. Robustness analysis under varying failure intensities (

\varepsilon

). The Smart Commander (HRL, purple) maintains stable performance across all failure rates, while DRL (orange) and Rule-Based (green) methods degrade significantly in harsh environments (

\varepsilon<1.0

). Layout same as Figure 4 .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6. Simulation model architecture with mission, fleet-health, and support/logistics modules.

Figure 7. Simulation flow of fleet operations in each DES cycle.

Parameter	Symbol	Value/Range
Fleet size	$N_{L}$	12 aircraft
Simulation time horizon	$T_{H}$	720 hours
Simulation time step	$\Delta t$	1 hour
Number of maintenance bays	$N_{B}$	6
Number of spare component types	$N_{C}$	5
Number of suppliers per component	$N^{i}_{S}$	3

Appendix

Table 2. Fleet-level simulation parameters (nominal configuration).

Parameter	Unit	AVI	FCS	POW	STR	MEC
mfhbf	hours	120	300	250	500	100
failure_prob	–	0.10	0.10	0.20	0.15	0.20
repair_time	hours	24	24	120	60	36
repair_cost	k$	5	7	20	15	10
detection_delay	hours	2	2	3	3	2
predict_lead	hours	0	0	80	100	40

Appendix

Table 3. Component-level PHM parameters for military aircraft (nominal configuration).

Parameter	General	Flight	Maintenance	Resource
Action dimension $d_{a}$	$\|\mathcal{M}\|$	$N_{L}\times 3$	$N_{B}\times 2$	$N_{C}\times N_{S}\times 2$
Hidden layers	[256, 256]	[128, 128]	[128, 128]	[128, 128]
Activation function	ReLU	ReLU	ReLU	ReLU
Batch size	64	128	128	128
Learning rate $\alpha$	$10^{-4}$	$10^{-3}$	$10^{-3}$	$10^{-3}$
Discount factor $\gamma$	0.99	0.95	0.95	0.95

Appendix

Table 4. Network architecture and training hyperparameters.

Commander	Parameter	Symbol	Value
Flight	Fleet availability weight	$\alpha$	2.0
Flight	Mission failure penalty	–	$-2\times re_{i}$
Maintenance	Time-to-cost conversion	$\beta$	0.2
Resource	Lead time weight	$\gamma$	0.5
Resource	Inventory holding weight	$\eta$	1.0
General	Flight performance weight	$\tau_{F}$	1.0

Appendix

Table 5. Reward function parameters.

Figure 8. Rule-based policy: mission selection and fleet state evolution under nominal conditions.

Figure 9. Flat DRL policy: mission selection and fleet state evolution under nominal conditions.

Figure 10. HRL Smart Commander: mission selection and fleet state evolution under nominal conditions.

Hierarchical RL

Hierarchical reinforcement learning studies policies that organize decisions into reusable skills, subgoals, or actions operating at different timescales.

See all 158 papers

Intent-Context Synergy Reinforcement Learning for Autonomous UAV Decision-Making in Air Combat
Mar 1, 2026Jiahao Fu, Feng YangReinforcement Learning Partially Observable RL
National Elite Institute of Engineering,Northwestern Polytechnical University, Xi’an, Shanxi, China · School of Automation, Northwestern Polytechnical University, Xi’an, Shanxi, China
Vehicle-as-Prompt: A Unified Deep Reinforcement Learning Framework for Heterogeneous Fleet Vehicle Routing Problem
Apr 6, 2026Shihong Huang, Shengjie Wang, Lei Gao +4Vehicle Routing RL for Combinatorial Optimization
Polytechnic Institute, Zhejiang University, 310015, Hangzhou, China · S.F. Technology Co., Ltd., 518054, Shenzhen, China · School of Management, Zhejiang University, 310058, Hangzhou, China
UAV-MARL: Multi-Agent Reinforcement Learning for Time-Critical and Dynamic Medical Supply Delivery
Mar 11, 2026Islam Guven, Mehmet ParlakMulti-Agent Coordination Multi-Agent Reinforcement Learning
ICTEAM, Université catholique de Louvain Ottignies-Louvain-la-Neuve, Belgium

Mar 1, 2026J/K move · Enter open · S save

Intent-Context Synergy Reinforcement Learning for Autonomous UAV Decision-Making in Air Combat

Jiahao Fu, Feng Yang

National Elite Institute of Engineering,Northwestern Polytechnical University, Xi’an, Shanxi, China · School of Automation, Northwestern Polytechnical University, Xi’an, Shanxi, China

Reinforcement Learning Partially Observable RL Deep Q-Learning Hierarchical RL

Open paper AskPDF arXiv

Autonomous UAV infiltration in dynamic contested environments remains a significant challenge due to the partially observable nature of threats and the conflicting objectives of mission efficiency versus survivability. Traditional Reinforcement Learning (RL) approaches often suffer from myopic decision-making and struggle to balance these trade-offs in real-time. To address these limitations, this paper proposes an Intent-Context Synergy Reinforcement Learning (ICS-RL) framework. The framework introduces two core innovations: (1) An LSTM-based Intent Prediction Module that forecasts the future trajectories of hostile units, transforming the decision paradigm from reactive avoidance to proactive planning via state augmentation; (2) A Context-Analysis Synergy Mechanism that decomposes the mission into hierarchical sub-tasks (safe cruise, stealth planning, and hostile breakthrough). We design a heterogeneous ensemble of Dueling DQN agents, each specialized in a specific tactical context. A dynamic switching controller based on Max-Advantage values seamlessly integrates these agents, allowing the UAV to adaptively select the optimal policy without hard-coded rules. Extensive simulations demonstrate that ICS-RL significantly outperforms baselines (Standard DDQN) and traditional methods (PSO, Game Theory). The proposed method achieves a mission success rate of 88% and reduces the average exposure frequency to 0.24 per episode, validating its superiority in ensuring robust and stealthy penetration in high-dynamic scenarios.

Abstract

Hierarchical RL

Explore similar work

Intent-Context Synergy Reinforcement Learning for Autonomous UAV Decision-Making in Air Combat

Vehicle-as-Prompt: A Unified Deep Reinforcement Learning Framework for Heterogeneous Fleet Vehicle Routing Problem

UAV-MARL: Multi-Agent Reinforcement Learning for Time-Critical and Dynamic Medical Supply Delivery