In this paper we illustrate a novel architecture generating interpretable behavior and explanations. We refer to this architecture as the Mirror Agent Model because it defines the observer model, that is the target of explicit and implicit communications, as a mirror of the agent's. With the goal of providing a general understanding of this work, we firstly show prior relevant results addressing the informative communication of agents intentions and the production of legible behavior. In the second part of the paper we furnish the architecture with novel capabilities for explanations through off-the-shelf saliency methods, followed by preliminary qualitative results.
Figures & tables
Figure 1: PR : an agent interacting with its environment. PRH : expectations inside the mind of the observer about the agent. The agent models these expectations through a second-order theory of mind. The general goal of interpretable behavior is to keep the distance ∣PR−PRH∣ low, signifying that the agent’s behavior effectively matches the observer’s expectations.
Figure 2: Mirror agent models for a BDI agent and a RL agent. PR : agent model. PRH : observer model. The nodes in gray indicate the variables that are observed by the observer.
Figure 3: Graphical user interface utilized during the user study on informative communication. The interface shows the positions of robot, objects and doors (left). For every step of a scenario the participants could select a prediction about the robot’s goal (right) given a sequence of verbalizations (bottom).
Figure 4: Hit ratio of the participants of the user study as a function of percentage of communicated actions.
α=0
α=0.1
α=0.5
α=1
α=2
α=5
C0
0.8
0.8
0.76
0.76
0.77
0.75
C1..3
0.29
0.21
0.15
0.14
0.13
0.11
Success
0.99
0.95
0.96
0.95
0.92
0.87
Table 1: Average accumulated reward ration by the policies for color C0 and colors C1..3 for increasing values of α . The row Success indicates the probability of completing a tunnel without hitting obstacles.
α=0
α=0.1
α=0.5
α=1
α=2
α=5
L
0.30
0.36
0.44
0.48
0.48
0.51
Lgain
1
1.2
1.46
1.6
1.6
1.7
Table 2: Legibility gain for increasing values of α . The legibility gain is measured as the ratio between the legibility of the legible and optimal policies.
Figure 5: Qualitative results for the production of legible behavior. The figure illustrates the agent’s behavior for increasing levels of α on three different positions of reward regions. The higher the regularization, the more the trajectories are skewed to avoid reward regions not rewarding the agent’s current intention of walking over red cells, but rather rewarding other intentions.
Figure 6: The explanation model defines how agent and observer respectively select and infer policies using the current state and saliency masks, while the function H measures the distance between these two processes. Blue nodes represent the newly introduced explanation model while gray nodes are the observer’s observations.
α
Walk
Saliency
0
1
0
1
Figure 7: Agent’s trajectory and saliency maps for two sampled tunnel environments. α is the magnitude of regularization for legibility. The agent selects to communicate its regions of reward (red) and the obstacles (grey). The agent does not select regions of other colors because their communication would make the observer model to infer the other policies that are rewarded to pass on those cells.