cs.AIJul 16, 2025
SaveA Survey of Explainable Reinforcement Learning: Targets, Methods and Needs
Organizations: ∗ IMT Mines Albi, Albi, France
Abstract
The success of recent Artificial Intelligence (AI) models has been accompanied by the opacity of their internal mechanisms, due notably to the use of deep neural networks. In order to understand these internal mechanisms and explain the output of these AI models, a set of methods have been proposed, grouped under the domain of eXplainable AI (XAI). This paper focuses on a sub-domain of XAI, called eXplainable Reinforcement Learning (XRL), which aims to explain the actions of an agent that has learned by reinforcement learning. We propose an intuitive taxonomy based on two questions "What" and "How". The first question focuses on the target that the method explains, while the second relates to the way the explanation is provided. We use this taxonomy to provide a state-of-the-art review of over 250 papers. In addition, we present a set of domains close to XRL, which we believe should get attention from the community. Finally, we identify some needs for the field of XRL.
Figures & tables
Figure 1. Our taxonomy for Explainable Reinforcement Learning.
Figure 2. Detailed taxonomy for policy-level methods.
| Type | Refs | |
| Inherently Understandable (54) | Hierarchical Policy (20) | Y. Jiang et al. 2019 ; Shu et al. 2018 ; D. Xu and Fekri 2021 |
| Chang et al. 2020 ; Lyu et al. 2019 ; B. Wu et al. 2019 | ||
| Beyret et al. 2019 ; Jin et al. 2022 ; Rossum et al. 2021 | ||
| Andreas et al. 2017 ; Rietz et al. 2023 ; Tasse et al. 2020 | ||
| Furelos-Blanco et al. 2021 ; Leonetti et al. 2016 ; Sridharan et al. 2019 | ||
| Eysenbach et al. 2019 ; F. Yang, Lyu, et al. 2018a ; A. Zhang et al. 2018a | ||
Table 1. Interpretable Policy works.
Figure 3. A discrete decision tree for the CartPole environment (Silva et al. 2019 ). Each internal node of the tree (in blue) represents a condition on a feature of the agent’s state, and each leaf of the tree (in green) corresponds to an action.
Figure 4. Example of Abstract Policy Graph (Topin and M. Veloso 2019 ). Each vertex is an abstract state with an associated action, and the edges between two vertices represent transitions (weighted by their probability of occurring).
| Type | Refs |
|---|---|
| Sequences (13) | D. Amir and O. Amir 2018 ; Huber, Weitz, et al. 2021 ; Septon et al. 2023 |
| Amitai, Septon, et al. 2024 ; Feit et al. 2022 ; Sequeira and Gervasio 2020 | |
| S. H. Huang, Held, et al. 2019 ; Lage et al. 2019a ; 2019b ; Sequeira and Gervasio 2023 | |
| Altmann et al. 2024 ; Frost el al. 2022 ; Gottesman et al. 2020 | |
| Critical States (8) | Dao, Huff, et al. 2021 ; S. H. Huang, Bhatia, et al. 2018 ; Rupprecht et al. 2020 |
| Dao, I. Mishra, et al. 2018 ; Dinu et al. 2020 , I. Mishra et al. 2018 |
Table 2. Policy Summary works.
Figure 5. Landmarks and their ordering obtained using TLdR (Sreedharan, Srivastava, and Kambhampati 2020 ) for the exploding blocks-world problem (Younes 2005 ). Vertices are landmarks and each blue edge defines an order between two landmarks. The green nodes are goals that the agent must reach.
Figure 6. SHAP for a set of states along an episode for a machine control use case (Y. Wang et al. 2020 ). The evolution of the importance score of each feature is represented by a colored curve in the chart, this importance varying over time.
| Type | Refs | |
|---|---|---|
| Inherently Understandable (33) | MDP Representation (20) | Bica et al. 2021 ; Jenner and Gleave 2022 ; Saldiran et al. 2024 |
| Devidze et al. 2021 ; Srinivasan and Doshi-Velez 2020 | ||
| Bewley and Lécué 2022 ; Camacho et al. 2019 | ||
| Gaon and Brafman 2020 ; Icarte, Klassen, et al. 2018 | ||
| Kaiser et al. 2019 ; Kasenberg and Scheutz 2017 ; X. Li et al. 2017a | ||
| Degris et al. 2006 ; Goel et al. 2018 ; Metzen 2013 | ||
Table 3. Human-readable MDP works.
Figure 7. A reward function as a binary tree structure for the original RoboCar environment consisting of a 4-wheeled vehicle whose goal is to reach a certain area while avoiding obstacles (Bewley and Lécué 2022 ). In this reward function, each internal node represents a test on a feature of an agent state, and each leaf a reward. The color of the leaves reflects the amount of reward the agent can obtain (blue means a high amount, red a low one).
Figure 8. Explanation via ASAP (Bekkemoen and Langseth 2023 ) of the agent’s policy for the Mountain Car environment. The first line shows a representative state for each hyperstate with its associated action. The second line is divided into two parts. On the left, a policy graph models the policy based on the hyperstates. Each vertex is a hyperstate and each edge represents a transition between two hyperstates (weighted by its probability of occurrence). On the right, the attention maps of each hyperstate are displayed (where an attention score is associated with each feature, determining the impact of the features on the agent’s decision-making).
| Type | Refs |
|---|---|
| Visual Toolkit (5) | Jaunet et al. 2020 ; A. Mishra et al. 2021 ; J. Wang et al. 2019a |
| W. He et al. 2020 ; McGregor et al. 2017 | |
| Latent Space Visualization (3) | Annasamy and Sycara 2019 ; Ben-Zrihem et al. 2016 ; Zahavy et al. 2016 |
| Inspection (2) | Dethise et al. 2019 ; Ullauri et al. 2022 |
Table 4. Visual Analysis works.
Figure 9. PolicyExplainer visual interface (A. Mishra et al. 2021 ). The different tools are divided into three parts. Panels A-C provide a global view of the frequency of actions and rewards, the policy, the value function. Panels D-G provide a detailed analysis of particular states and their associated Q-value, or a particular trajectory. Panel H answers the questions ‘Why?’, ‘Why not?’ and ‘When?’ using state features.
Figure 10. Semi-MDP built on top of a t-SNE map for the Atari 2600 Breakout environment (Ben-Zrihem et al. 2016 ). The t-SNE map is the set of points on which the Semi-MDP is based. It is made up of states (or clusters of MDP states), linked by transitions weighted by their probability of occurrence. Each cluster is modelled by an image of the game where the position of the ball in each state of the cluster is displayed in red.
Figure 11. Detailed taxonomy for sequence-level methods.
| Type | Refs | |
| Counterfactual Sequence (6) | Alabdulkarim and Riedl 2022 ; Waa et al. 2018 | |
| Gajcin and Dusparic 2024a ; Tsirtsis, De, et al. 2021 | ||
| Stefik et al. 2021 ; Tsirtsis and Rodriguez 2023 | ||
| Important Elements (4) | W. Guo, X. Wu, et al. 2021b ; H. Liu et al. 2023 | |
| Saulières et al. 2023a ; 2024 | ||
| Human-readable MDP (3) | Surrogate Model (2) | Soni 2024 ; Sreedharan, Soni, et al. 2022 |
Table 5. Sequence-level works.
Figure 12. Possible explanations for the sequence of actions that the agent (represented by a robot in cell (3,1)) performs in a 2D environment to reach the green cell (Alabdulkarim and Riedl 2022 ). The user’s question is written in red. Explanation A (in black and marked by a red cross) is based solely on the agent’s policy, while explanation B (in blue marked with a correct tick) uses influence predictors. According to authors, explanation B is of better quality.
Figure 13. An explanation of an episode from a game of Atari 2600 Pong (W. Guo, X. Wu, et al. 2021b ). The objective is to determine the important time-steps for the defeat of the agent controlling the green pad. The colored bar below each state of the episode represents its level of importance (yellow represents a low level, orange a medium level and red a high level).
Figure 14. Detailed taxonomy for action-level methods.
| Type | Refs | |
| Saliency Maps (32) | Attention Mechanism (15) | Itaya et al. 2021 ; Mott et al. 2019 ; Nikulin et al. 2019 |
| Y. Liu et al. 2022 ; Ou et al. 2024 ; Shi et al. 2022 | ||
| Bao et al. 2021 ; Josef and Degani 2020 ; Y. Tang et al. 2020 | ||
| X. Wang, Yuan, et al. 2019b ; Q. Yang et al. 2024 ; Z. Yang et al. 2018c | ||
| Huber, Schiller, et al. 2019 ; Z. Wang, and S. Wang 2022 | ||
| Q. Zhang 2021b | ||
Table 6. Feature Importance works.
Figure 15. Two saliency maps for a state-action pair from the Ms PacMan environment (Iyer et al. 2018 ). The action is ‘move right’ and the sub-figure ‘a’ is the state from which a pixel saliency map (sub-figure ‘b’) and an object saliency map (sub-figure ‘c’) are generated. The saliency of an element is represented using a grey scale, where the darker the element, the more salient it is. In sub-figure ‘c’, we can see that the agent has focused on Ms PacMan and the pills in the right path to choose the ‘move right’ action. Conversely, it did not focus on the part of the image that symbolises 1 life for the player, as shown by the white square in the object saliency map.
Figure 16. Comparison of counterfactual states in the Atari 2600 Ms PacMan environment (Bellemare et al. 2013 ) generated on the basis of a state-action pair (on the left) (Huber, Demmler, et al. 2023 ). The agent’s action is ‘move right’ and the counterfactual one is ‘move left’ . The state in the middle, where Ms PacMan is missing, is generated using (Olson et al. 2021 ) and the state on the right, where Ms PacMan is safe in the left part of the map, is generated using Huber, Demmler, et al. The first counterfactual state is not a convincing one as it shows that the agent chooses ‘move left’ while Ms PacMan does not even appear, whereas in the second one, Ms PacMan is (partially) visible.
| Type | Refs | |
| State (16) | Causal Lens (5) | Madumal et al. 2020b ; 2020a ; Volodin 2021 |
| Herlau and Larsen 2022 ; Yu et al. 2023 | ||
| Probability (4) | Cruz, Dazeley, and Vamplew 2019 ; Cruz, Dazeley, Vamplew, and Moreira 2023 | |
| Dodson et al. 2011 ; Khan et al. 2009 | ||
| Contrastive (4) | Barros et al. 2020 ; Z. Lin et al. 2021 ; Stein 2021 ; Tsuchiya et al. 2023 | |
| Others (3) | Fukuchi et al. 2017a ; 2017b ; Pan et al. 2019 | |
Table 7. Expected Outcomes works.
Figure 17. Causal model for the LunarLander environment (Yu et al. 2023 ). The features of a state are in blue, the actions in orange and the outcomes in green. An arrow between two elements describes a causal relation.
Figure 18. RDX for a Human-Robot collaboration task (Iucci et al. 2021 ). Two actions (related to robot speed) are compared across four different reward types (on the x-axis of the chart). The y-axis shows the loss and gain for each type of reward when performing the agent’s action rather than the other one.
Figure 19. Explanation of each action in an episode in the Taxi environment (Yau et al. 2020 ). The episode is described on two lines, starting on the left of the first line. In each state, the agent’s future positions according to the belief map are displayed by colored cells. The color intensity reflects the agent’s confidence in accessing the position (the more intense the colour, the more confident the agent). In this episode, the Taxi picks up the passenger at the position marked with a ‘G’ and drops him off at the position marked with a ‘B’.
More in this topic
Interpretability in RL
Interpretability in reinforcement learning makes learned policies, decisions, and internal representations understandable and auditable.
See all 70 papers