eess.SYOct 20, 2026

Brittle by Design? Multi-Scenario Reinforcement Learning for Robust Mission Coordination in Systems of Systems

Authors: İbrahim Oğuz Çetinkaya, Sajad Khodadadian, Taylan Topcu

Organizations: Grado Department of Industrial and Systems Engineering, Virginia Tech, Blacksburg, VA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

As systems engineering (SE) shifts from monolithic systems to Systems of Systems, Mission Engineering (ME) has emerged as a new line of thinking in SE. Mission environments are uncertain and dynamic, and outcomes depend on how mission assets interact with them, rendering static architectures brittle and calling for flexible yet analytically rigorous ME approaches. We propose an intelligent mission coordination methodology integrating high-fidelity digital mission models with Reinforcement Learning (RL) for adaptive task allocation. Using a Digital Engineering (DE) infrastructure comprising a digital mission model and agent-based simulation, we formulate mission tactics management as a Markov Decision Process and train an RL coordinator with Proximal Policy Optimization to map mission states to tactical actions from realized outcomes. An aerial firefighting mission with three scenarios demonstrates the approach. First, all trained coordinators significantly outperform random and heuristic baselines. Second, single-scenario coordinators can be brittle: the coordinator trained on the more forgiving scenario degrades to random tactics performance in more demanding scenarios, whereas the demanding-scenario coordinator retains most of its advantage when transferred. Third, within the same training budget, a coordinator trained on both scenarios concurrently matches the scenario-specific coordinators in their own scenarios and significantly outperforms every other policy. Sensitivity analyses on ignition location, weather, scenario, and training curricula show that the variability represented during training shapes post-deployment performance. This proof of concept shows DE-enabled mission simulations combined with RL offer a reusable methodology for improving ME practice, which can be extended to fleet design and selection from a mission-first perspective.

Figures & tables

Figure 1: Methodological Overview
ParameterSeaplaneeVTOLParameterSeaplaneeVTOL
Energy sourceFuelBatteryCruise speed (m/s)10090
Takeoff modeRunway or waterVerticalCruise altitude (m)3,000457
Span (m)186Takeoff distance (m)2,7600
Payload capacity (kg)1,300360Takeoff climb rate (m/s)3.810.51
Endurance (min)42027Cruise climb rate (m/s)6.963.56
Range (km)2,500150   
Table 1: Characteristics of the seaplane and eVTOL platforms.
Figure 2: Mission scenarios and the variable ignition regions for (a) Palisades and (b) Pyrenees.
ParameterPalisadesPyrenees
Temperature ( ∘ C)10–2310–25
Relative humidity (%)22–9020–70
Wind run (km/day)320350
Wind speed (m/s)2.6–5.42.8–5.9
Wind direction ( ∘ )15 ±\pm 45300 ±\pm 30
Table 2: Nominal weather ranges of the Palisades and Pyrenees scenarios.
Figure 3: Operational workflow of the simulation environment for training the RL-based intelligent mission coordinator
Figure 4: Training performance of PA and PY over 150,000 episodes
Figure 5: MoE distributions of PY, PA and baselines across Palisades, and Pyrenees scenarios
 PalisadesPyrenees
PolicyContained (%)Mean MoEContained (%)Mean MoE
PPO Palisades-only (PA)98.10.9726.2−0.113-0.113
PPO Pyrenees-only (PY)95.50.94315.50.017
Heuristics87.40.8537.5−0.088-0.088
Random Tactics87.90.8605.5−0.122-0.122
Table 3: Containment rate (share of the 1,000 evaluation episodes in which the fire did not escape the region) and mean MoE of each coordinator in each scenario
Figure 6: Comparative training performance of PA, PY, and MS over 150,000 episodes
Figure 7: Comparative training performance of PA, PY, and MS over 150,000 episodes in (a) Palisades and (b) Pyrenees scenarios
Figure 8: MoE distribution of MS, PY, PA, and baselines across (a) Palisades, (b) Pyrenees scenarios, and (c) Pooled across both over 1,000 evaluation shared episodes on each scenario
A vs BΔ\Delta MoE95% CIpHolmp_{\mathrm{Holm}}
Palisades ( 1,0001{,}000 simulations)
MS vs PA+0.0010{+}0.0010[ −0.0099-0.0099 , +0.0120{+}0.0120 ]11
MS vs PY+0.0299{+}0.0299[ +0.0137{+}0.0137 , +0.0460{+}0.0460 ]0.00120.0012
PA vs PY+0.0288{+}0.0288[ +0.0131{+}0.0131 , +0.0446{+}0.0446 ]0.00120.0012
PY vs Heuristics+0.0899{+}0.0899[ +0.0684{+}0.0684 , +0.1113{+}0.1113 ]3.6×10−153.6\times 10^{-15}
PY vs Random Tactics+0.0826{+}0.0826[ +0.0593{+}0.0593 , +0.1058{+}0.1058 ]2.8×10−112.8\times 10^{-11}
Table 4: Pairwise statistical comparisons of coordinators’ MoE in Palisades, Pyrenees, and both scenarios pooled using paired tt -tests with Holm correction
Figure 9: MoE regret of coordinators in (a) Palisades, (b) Pyrenees, and (c) pooled across both scenarios (overall regret).
Figure 10: (a) Training MoE of PY and PY fix over 150,000 simulations, with the Random Tactics baseline for reference. (b) Mean evaluation MoE of PA, PY, MS, PY fix , and the baselines under varying and fixed ignition (each dumbbell connects a policy’s scores in the two settings).
Figure 11: Mean evaluation MoE of coordinators and the baselines under adverse and mild weather conditions (each dumbbell connects a coordinator’s scores in the three settings).
 Mean MoE
CurriculumPalisadesPyreneesPooled
Sequential, even (75k →\rightarrow 75k)0.9409−- 0.00620.4673
Sequential, warm-up (25k →\rightarrow 125k)0.94800.01500.4815
Concurrent (44% / 56%)0.97260.01820.4954
Table 5: Mean MoE of multi-scenario coordinators trained under three curricula that divide the same 150,000-simulation budget differently between Palisades and Pyrenees.
Figure 12: Salamis scenario
Figure 13: MoE distributions of MS, PY, PA, Heuristics and Random Tactics over 1,000 shared episodes in Salamis
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Variable Name (Number of Variables)DescriptionUnit of Measurement
Scenario identifier † (3)Identifies the mission scenario in a one-hot encoded wayDimensionless
Time since detectionDuration since the detection of the fireMinutes
TemperatureCurrent environmental temperatureCelsius degrees
HumidityCurrent environmental humidityPercentage
Wind speed and direction (2)Current wind speed and directionm/s and degrees
Time to sunsetDuration until sunsetMinutes
Appendix
Table 6: Description of State Variables
Figure 14: Simulation progression, and two example track and suppress operations
Scenario iiMax Burnt Areai\text{Max Burnt Area}_{i} (ha)Max Costi\text{Max Cost}_{i} (M€)Max Emissionsi\text{Max Emissions}_{i} (tonnes)Max Casualtiesi\text{Max Casualties}_{i} (persons)
Palisades9,087191,106131,224300,000
Pyrenees9,93817,5092,364,06410,000
Salamis4,14613,993714,0096,000
Appendix
Table 7: Per-scenario normalization constants of the MoE (Eq. (3)): the maximum damage each term can inflict, bounded by the region’s characteristics.
More in this topic

Multi-Agent Coordination

Multi-agent coordination studies how autonomous agents align actions, share information, and divide work to achieve individual or collective goals.

See all 520 papers

Explore similar work

CardsList
Mar 12, 2026

The price of decentralization in managing engineering systems through multi-agent reinforcement learning

Inspection and maintenance (I&M) planning involves sequential decision making under uncertainties and incomplete information, and can be modeled as a partially observable Markov decision process (POMDP). While single-agent deep reinforcement learning provides approximate solutions to POMDPs, it does not scale well in multi-component systems. Scalability can be achieved through multi-agent deep reinforcement learning (MADRL), which decentralizes decision-making across multiple agents, locally controlling individual components. However, this decentralization can induce cooperation pathologies that degrade the optimality of the learned policies. To examine these effects in I&M planning, we introduce a set of deteriorating systems in which redundancy is varied systematically. These benchmark environments are designed such that computation of centralized (near-)optimal policies remains tractable, enabling direct comparison of solution methods. We implement and benchmark a broad set of MADRL algorithms spanning fully centralized and decentralized training paradigms, from value-factorization to actor-critic methods. Our results show a clear effect of redundancy on coordination: MADRL algorithms achieve near-optimal performance in series-like settings, whereas increasing redundancy amplifies coordination challenges and can lead to optimality losses. Nonetheless, decentralized agents learn structured policies that consistently outperform optimized heuristic baselines, highlighting both the promise and current limitations of decentralized learning for scalable maintenance planning.

Prateek Bhustali, Pablo G. Morato, Konstantinos G. Papakonstantinou +1

Faculty of Architecture and the Built Environment, Delft University of Technology, Delft, 2628 BL, The Netherlands · Engineering Risk Analysis Group, Technical University of Munich, Munich, 80333, Germany · Department of Civil & Environmental Engineering, The Pennsylvania State University, University Park, PA, 16802, USA

Decentralized MARLMulti-Agent CoordinationCooperative MARL
AskPDF
Jun 29, 2026

Sampling-Based Coordination-Informed Multi-Objective Multi-Robot Reinforcement Learning

Multi-robot systems must simultaneously optimize competing objectives while maintaining coordinated behavior. Existing multi-agent reinforcement learning approaches often rely on fixed or centralized coordination, which limits adaptability and violates distributed constraints. This work introduces the Coordination-Informed Multi-Objective Reinforcement Learning (CIMORL) framework, integrating a distributed weight prediction mechanism, a privileged expert training strategy, and theoretical guarantees for Pareto-optimal solutions. We present the base CIMORL method alongside two sampling-based variants, CIMORL-TS (Tree Search) and CIMORL-MPPI (MPPI), which leverage privileged global information during training to enable fully decentralized deployment. Experimental validation in cooperative and adversarial scenarios demonstrates a 21.2%21.2\% hypervolume improvement and superior policy stability compared to state-of-the-art baselines. Real-world experiments with Crazyflie drones further validate the framework's robustness in resource allocation and multi-attacker multi-defend scenarios under partial observability.

Antonio Marino, Esteban Restrepo, Soon-jo Chung +2

University of Cambridge · CNRS, Univ Rennes, Inria, IRISA · Division of Engineering and Applied Science, California Institute of Technology

Multi-Objective Reinforcement LearningMulti-Robot CoordinationDecentralized MARLCooperative MARLMulti-Agent Reinforcement Learning
AskPDF
Dec 1, 2025

Bayesian Ambiguity Contraction-based Adaptive Robust Markov Decision Processes for Adversarial Surveillance Missions

Collaborative Combat Aircraft (CCAs) are envisioned to enable autonomous Intelligence, Surveillance, and Reconnaissance (ISR) missions in contested environments, where adversaries may act strategically to deceive or evade detection. These missions pose challenges due to model uncertainty and the need for safe, real-time decision-making. Robust Markov Decision Processes (RMDPs) provide worst-case guarantees but are limited by static ambiguity sets that capture initial uncertainty without adapting to new observations. This paper presents an adaptive RMDP framework tailored to ISR missions with CCAs. We introduce a mission-specific formulation in which the CCA alternates between movement and sensing states. Adversarial uncertainty is represented by finite threat prototypes defining observation, exposure, and state-action transition models. Our approach incrementally refines policies by eliminating inconsistent threat models, allowing the CCA to move from worst-case protection toward more efficient mission policies while maintaining robustness. We establish fixed point stability, agreement with nominal planning as credible sets identify the true node-specific threat prototypes, and a conservative expected value guarantee while those prototypes remain credible. Experiments under Gaussian and non-Gaussian threat models across diverse network topologies show higher mission rewards and fewer exposure events compared to nominal and static robust planners.

Jimin Choi, Max Z. Li

University of Michigan, Ann Arbor, MI 48109, USA

Robust Markov Decision ProcessesModel-Based Planning
AskPDF