There is a well-established body of work demonstrating that individual machine learning (ML) algorithms can be manipulated or deceived. However, in many real-world systems, multiple ML algorithms are composed to enable automated decision-making, particularly in Cyber-Physical Systems (CPS). Often such systems use a composition of vision, planning, and other ML models to affect their internal state and ultimate take actions in the physical world. While there is a huge literature on the (in)security of ML algorithms, it is not immediately clear how an attacker might leverage their ability to trick individual learners when attacking the composition as a whole. We develop methods for creating an attack policy (a mapping from state to attack) for a data-manipulating attacker affecting a CPS or other state-machine-based AI compositions. We describe our framework, a basic threat model and corresponding methods for computing an optimal attack policy, as well as planned extensions. The goal of this work-in-progress paper is to stimulate conversation and forge new collaborations regarding the security of compositional learning.
Figures & tables
Figure 1: Self Driving Car Example. The Stopped (S) state represents the car stopped, awaiting instruction. The Accelerating (A) state represents a car that is increasing its velocity. When moving at the desired speed in the desired lane with no perceived obstacles, the car is in the Lane Keeping (LK) state. The Decelerating (D) and Passing (PS) states represent reactive maneuvers to environmental hazards, while Parking (PK) represents a transition to a terminal safety state. In our example, the attacker’s objective is to reach the target state s†=S ; the optimal adversarial policy shifts between exploiting the D or PS pathways depending on the defender’s transition probabilities, defined by the parameters αs,s′ (transition probability from state s to state s′ ). For figure clarity we have removed the self-loops of every state, representing the fact that the car may remain in a state across multiple time steps.