Organizations: College of Intelligence Science and Technology, National University of Defense Technology, Changsha, 410073, China · College of Mechanical and Vehicle Engineering, Hunan University, Changsha, 410082, China · School of Mechatronic Engineeringand Automation, Shanghai University, Shanghai 200444, China · Department of Computer Science, The University of Manchester, 2628 CD Manchester, U.K.
Distributed learning control for multirobot systems (MRS) offers significant flexibility in presence of uncertainties but lacks provable performance guarantees. A promising direction involves integrating reinforcement learning (RL) into distributed model predictive control (DMPC), leveraging the strengths of RL in nonlinear policy design and the receding-horizon replanning capabilities of DMPC. However, ensuring secure control within such a learning framework under malicious cyber attacks, particularly stealthy ones, remains a critical challenge, because the distributed policies generation depends on information exchange among neighbors, where compromised agents can rapidly influence the behavior of others through the communication network. This article proposes a distributed secure learning control (DSLC) framework for large-scale MRS under malicious, stealthy actuator attacks. Our framework offers two key features: (i) a unified approach that enables secure learning control across various coordination scenarios and (ii) a game-theoretic distributed learning-based predictive control strategy that learns how to balance the attacker and defender through a differential-game based DMPC framework. Specifically, DSLC employs a distributed attacker-actor-critic architecture to learn the optimal defense and attack policies online within each prediction interval. Unlike numerical optimization-based controllers that calculate open-loop control sequences, our method simultaneously generates adversarial attack policies and corresponding defense policies in analytical closed-loop form. The defense policies could be directly generalized to MRS with varying scales and diverse actuator attack probabilities. The effectiveness and scalability of DSLC are validated through comprehensive simulations and real-world experiments in multiple wheeled robots via various control tasks.
Figures & tables
Fig. 1: The scheme diagram of the distributed secure learning control (DSLC) framework. The attackers that deteriorate the control performance and the defenders that secure the closed-loop system are modeled as multi-players in a differential-game based DMPC framework. The resulting problem is solved by a computationally efficient attacker-actor-critic learning algorithm, which generates the closed-loop control policies for the attacker and defender. βt=(βt,1,⋯,βt,M) .
Attacks
Methods
Two Robots
Four Robots
Six Robots
IAE
ITAE
Jc
IAE
ITAE
Jc
IAE
ITAE
Jc
β=0.1
DSLC
0.57
0.47
3.35
0.82
0.8
4.13
0.93
1.22
7.09
DRHRL [ 8 ]
7.04
46.14
44.45
11.49
101.41
39.52
25.65
255.22
249.43
DRMPC [ 6 ]
19.56
206.85
23.68
38.51
546.78
127.14
89.83
1014.86
470.44
β=0.2
DSLC
0.65
0.5
3.36
0.73
1.42
3.84
0.78
1.15
6.01
DRHRL [ 8 ]
17.63
172.64
283.94
28.45
214.12
187.36
24.34
241.9
277.96
TABLE I: Criterion for the Accessment of the Distributed Control Performance under Different Attacks.
Fig. 2: Communication graphs of multirobots, where Robot 0 is the virtual leader.
Fig. 3: The curves of the cost to go Jp=∫τ=0tsimr(τ)dτ for formation control under different attack probabilities, where tsim is the terminal time of the simulation period.
Fig. 4: The trajectories of six-robot formation with different random attack probabilities under DRHRL [ 8 ] , DRMPC [ 6 ] , and DSLC.
Fig. 5: The curves of rˉi=∥xNi∥Qi2+∥us,i∥Rs,i2 for general formation of 100 and 1000 robots under different random probabilistic attacks using the policies learned from two robots.
Fig. 6: Schematic diagram of the MRS experimental platform under additive deception actuator attacks. (a) The scene of the experimental platform; (b) The information flow for MRS under attacks.
Fig. 7: In Scenario I, trajectories and snapshots of 4 robots without attacks ( β=0 ) under DSLC.
Fig. 8: In Scenario II, trajectories and snapshots of 4 robots without attacks ( β=0 ) under DSLC.
Fig. 9: Scenario I: trajectories and snapshots of 6 robots without attacks ( β=0 ) under DSLC.
Fig. 10: Scenario II: trajectories and snapshots of 6 robots without attacks ( β=0 ) under DSLC.
Fig. 11: Scenario I: trajectories and snapshots of 4 robots with an attack probability of β=0.5 . (a) DSLC; (b) DRHRL. The robots under DSLC formed and maintained a stable rectangular formation, whereas the formation under DRHRL was disrupted by the attack.
Fig. 12: Scenario II: trajectories and snapshots of 4 robots with an attack probability of β=0.5 . (a) DSLC; (b) DRHRL. The robots under DSLC formed and maintained a stable rectangular formation, whereas the formation under DRHRL was disrupted by the attack.
Fig. 13: Scenario I: trajectories and snapshots of 6 robots with an attack probability of β=0.5 . (a) DSLC; (b) DRHRL. The robots under DSLC formed and maintained a stable rectangular formation, whereas the formation under DRHRL was disrupted by the attack.
Fig. 14: Scenario II: trajectories and snapshots of 6 robots with an attack probability of β=0.5 . (a) DSLC; (b) DRHRL. The robots under DSLC formed and maintained a stable rectangular formation, whereas the formation under DRHRL was disrupted by the attack.
Fig. 15: Scenario I: trajectories and snapshots of 4 robots with an attack probability of β=1 . (a) DSLC; (b) DRHRL. The robots under DSLC formed and maintained a stable rectangular formation, whereas the formation under DRHRL was disrupted by the attack.
Fig. 16: Scenario II: trajectories and snapshots of 4 robots with an attack probability of β=1 . (a) DSLC; (b) DRHRL. The robots under DSLC formed and maintained a stable rectangular formation, whereas the formation under DRHRL was disrupted by the attack.
Fig. 17: Scenario I: trajectories snapshots of 6 robots with an attack probability of β=1 . (a) DSLC; (b) DRHRL. The robots under DSLC formed and maintained a stable rectangular formation, whereas the formation under DRHRL was disrupted by the attack.
Fig. 18: Scenario II: trajectories and snapshots of 6 robots with an attack probability of β=1 . (a) DSLC; (b) DRHRL. The robots under DSLC formed and maintained a stable rectangular formation, whereas the formation under DRHRL was disrupted by the attack.
Attack probabilities
Approaches
4 Robots
6 Robots
Scenario (i)
Scenario (ii)
Scenario (i)
Scenario (ii)
β=0.5
DSLC
1.6
7.36
1.83
15.17
DRHRL [ 8 ]
1.97
8.46
5.09
20.47
β=1
DSLC
1.85
7.22
2.19
14.92
DRHRL [ 8 ]
3.22
8.27
7.3
16.12
TABLE II: Control Performance under Attacks with Different Probabilities for General Formation in the Experiments
Fig. 19: Trajectories and snapshots for affine formation with an attack probability of β=0.5 . (a) DSLC. (b) DRHRL. The robots under DSLC formed a stable rectangular formation and dynamically expanded or contracted it to avoid collisions, whereas the robots under DRHRL failed to avoid the obstacles due to the attack.
Fig. 20: Trajectories and snapshots for affine formation with an attack probability of β=1 . (a) DSLC. (b) DRHRL. The robots under DSLC formed a stable rectangular formation and dynamically expanded or contracted it to avoid collisions, whereas the robots under DRHRL failed to avoid the obstacles due to the attack.
Fig. 21: Trajectories and snapshots for containment with an attack probability of β=0.5 . (a) DSLC. (b) DRHRL. The robots under DSLC formed a stable containment shape and successfully reconfigured to accommodate a newly joined robot. In contrast, the follower robot under DRHRL failed to effectively enter the convex hull and maintain safe distances, resulting in collisions.
Fig. 22: Trajectories and snapshots for containment with an attack probability of β=0.5 . (a) DSLC. (b) DRHRL. The robots under DSLC formed a stable containment shape and successfully reconfigured to accommodate a newly joined robot. In contrast, the follower robot under DRHRL failed to effectively enter the convex hull and maintain safe distances, resulting in collisions.
Tasks
Approaches
β=0.5
β=1
Affine Formation
DSLC
4.41
4.06
DRHRL [ 8 ]
6.54
6.31
Containment
DSLC
21.44
15.84
DRHRL [ 8 ]
31.66
40
TABLE III: Performance Assessment of Affine Formation and Containment under Different Attacks in the Experiments
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 23: The trajectories of six-robot formation with different random attack probabilities across robots: (a), β=(0.4,0.8,0.6,0.6,0.4,0.4) ; (b), β=(0.3,0.2,0.3,0.5,0.2,0.6) ; (c), β=(0.3,0.5,0.4,1,0.9,0.3) ; (d), β=(0.9,0.3,0.5,0.1,0.1,0.9) , where β=(β1,⋯,β6).
Fig. 24: The trajectories of two-robot formation with different attack probabilities under DRHRL [ 8 ] , DRMPC [ 6 ] , and DSLC.
Fig. 25: The trajectories of four-robot formation with different attack probabilities under DRHRL [ 8 ] , DRMPC [ 6 ] , and DSLC.
Fig. 26: The trajectories of a six-robot formation illustrating the deployment of a defense policy trained under a specific attack probability to scenarios with different attack probabilities.
Robot Number
β
Average Computation Time (ms)
DSLC
DRMPC
2
0.1
1.38
25.6
0.4
1.53
16.3
0.7
1.28
37.4
4
0.2
2.28
56.2
0.5
2.00
54.2
Appendix
TABLE IV: Average Computational Time Comparison between DSLC and DRMPC
Task
Robot Number
β
Completion Time (s)
DSLC
DRHRL
Containment
5
0.5
25.02
36.11
1
25.02
34.28
Affine Formation
4
0.5
10.01
10.01
1
10.02
10.02
General Formation
4 (Scenario I)
0.5
11.11
11.18
Appendix
TABLE V: Task Completion Time Comparison
Fig. 27: The trajectories of a six-robot formation under DSLC in the presence of a destabilizing attack policy, i.e., ua,i=Kxi,∀i∈N1M , where K=[0.5,1,0.1,0.5;0.2,0.7,0.3,1] is a randomly generated matrix.
Multi-robot systems (MRS) are essential for large-scale applications such as disaster response, material transport, and warehouse logistics, yet ensuring robust, safety-aware formation control in cluttered and dynamic environments remains a major challenge. Existing model predictive control (MPC) approaches suffer from limitations in scalability and provable safety, while control barrier functions (CBFs), though principled for safety enforcement, are difficult to handcraft for large-scale nonlinear systems. This paper presents FORMULA, a safe distributed, learning-enhanced predictive control framework that integrates MPC with Control Lyapunov Functions (CLFs) for stability and neural network-based CBFs for decentralized safety, eliminating manual safety constraint design. This scheme maintains formation integrity during obstacle avoidance, resolves deadlocks in dense configurations, and reduces online computational load. Simulation results demonstrate that FORMULA enables scalable, safety-aware, formation-preserving navigation for multi-robot teams in complex environments.
Qintong Xie, Weishu Zhan, Peter Chin
Thayer School of Engineering, Dartmouth College, NH 03784. · Department of Computer Science, The University of Manchester, Manchester M13 9PL, United Kingdom.
This work studies resilient output containment for heterogeneous linear multi-agent systems with actuator cyber-attacks over directed network topologies. The leaders generate bounded locally absolutely continuous trajectories; however, their dynamics, velocity bounds, and motion envelopes are undisclosed to the followers. The cyber-attack model includes state- and input-correlated, as well as bounded exogenous actuator false-data terms. A continuous two-layer adaptive control architecture is proposed. The first layer is a virtual-actuator reconfiguration layer that uses partial state measurements to compensate for actuator attacks in the local tracking-error dynamics. The second layer is a network interface that generates task-space commands via an adaptive interaction protocol. This protocol uses only neighbor-exchanged network-interface states whose dimensions match those of the plant output, and it does not require global graph knowledge for parameter tuning. For directed graphs, under a leader-rooted united spanning-tree condition, a nonsmooth Lyapunov analysis yields asymptotic containment at the command level. The physical outputs then converge to the leader convex hull up to a residual determined by the command-tracking local controllers. Simulation results using a network of quadrotors with damped suspended loads illustrate the performance of attack recovery and containment tracking.
Department of Electrical and Computer Engineering, Concordia University, Montreal, Canada. · Department of Electrical Engineering, Qatar University, Doha, Qatar.
This paper develops a safe and fully decentralized multi-agent reinforcement learning (MARL) algorithm to solve a class of discrete-time control problems on networks, including the persistent monitoring problem. Fully decentralized control of agents, while offering numerous benefits, faces issues such as exponentially increasing sample complexity, lack of global information about the system, and challenges in coordinating between agents. To address these issues, this paper introduces a fully decentralized multi-agent reinforcement learning algorithm that integrates deep reinforcement learning with safety considerations. This method feeds a history of local observations of the network's state into two parallel neural-network branches: the graph encoder, which adds structural information and correlations among nodes, and a state estimator, which predicts the uncertainty at each node in the graph. Additionally, the result of feeding that input into an actor-critic network is passed through a discrete-time control barrier heuristic to reduce the likelihood that any node will be neglected. This approach enables teams of fully decentralized agents to solve challenging problems by increasing system awareness and incorporating built-in safety measures to prevent the adoption of potentially harmful control policies. Numerical results from a custom simulation environment demonstrate that the proposed algorithm achieves 26.3 percent lower average uncertainty than a centralized control policy and is within 1 percent of the uncertainty performance of a more computationally complex algorithm with added attention layers.
Theodore Rogalski, Shirantha Welikala
Electrical & Computer Engineering Department Stevens Institute of Technology Hoboken, USA