Deep Reinforcement Learning (RL) plays a crucial role in tackling complex tasks, yet its reliance on deep neural networks makes the learned policies opaque. To enhance the trustworthiness of RL policies, existing work (e.g., the SILVER framework [Li et al., 2025]) has focused on Shapley-based interpretability, but is typically restricted to low-dimensional state spaces, binary-action settings, and a single class of linear surrogate. We propose ReShap, a policy-distillation framework that extends Shapley-based interpretability to high-dimensional, multi-action environments. ReShap extracts compact feature representations from raw image observations, performs SHAP-based feature attribution to characterize how each feature contributes to the policy's decisions, and distills the deep RL policy into an interpretable surrogate trained on the full set of (state, policy-action) pairs along the policy's visited distribution. We instantiate the surrogate with three interpretable model families (decision trees, linear regression, and logistic regression) and compare their fidelity and task return. We evaluate ReShap on Atari and Box2D environments using three deep RL algorithms and conduct a human-subject study to assess the clarity and trustworthiness of the resulting policies. Results show that ReShap preserves competitive task performance, that decision trees generally outperform linear and logistic regression, and that linear regression is weakest in most settings.
Figures & tables
Figure 1: ReShap overview: the deep RL policy π is rolled out to collect the states it visits together with the actions it takes; each state is encoded by a CNN feature extractor; SHAP attributes the contribution of every feature; and an interpretable surrogate (decision tree, linear regression, or logistic regression) is distilled on the full set of collected (state, action) pairs and evaluated by fidelity and episode return.
Figure 2: Mean absolute SHAP value of each feature for MsPacman under different algorithms (DQN, A2C, PPO). The A2C rollout is single-action, so its SHAP attributions are negligible (note the axis scale).
Figure 3: Episode return of the interpretable surrogates as a percentage of the original RL policy’s return, under full-distribution distillation, for MsPacman, RoadRunner, and LunarLander. Bars are grouped by RL algorithm (DQN, A2C, PPO), one bar per surrogate; the A2C logistic-regression bar is omitted where the rollout contains fewer than two distinct actions.
Schemes
Target Environments
Action Space
Decision Boundary Modeling
Action Labeling
Policy-explanation Alignment
Causal Lens Madumal et al. [2020]
Low-dimensional control and box2d (e.g., Mountain Car, Lunar Lander)
Discrete (low-cardinality)
None (Structural causal model only)
Not applicable
Not applicable
SVERL Beechey et al. [2023]
Low-dimensional discrete states (e.g., Taxi, Minesweeper)
Discrete (low-cardinality)
None (Shapley attribution only)
Not applicable
Not applicable
DISTILL2EXPLAIN Gokhale et al. [2024]
Low-dimensional control (e.g., Simulation of home energy management)
Discrete (low-cardinality)
None (Policy distillation only)
Not applicable
Behaviorally aligned
SILVER Li et al. [2025]
Low-dimensional control (e.g., CartPole, Mountain Car)
Figure 4: Gameplay clips of PPO agents in Atari environments
Figure 5: CNN-based feature extractor
Figure 6: MsPacman feature maps: the 16 CNN features for ten states sampled along each RL policy’s rollout, annotated with the policy’s action.
Figure 7: RoadRunner feature maps: the 16 CNN features for ten states sampled along each RL policy’s rollout, annotated with the policy’s action.
Figure 8: Performance comparison: SILVER vs ReShap
Algorithm
Decision Boundary
DQN
f(x,x˙,θ,θ˙)=−0.5x−0.687x˙−1.09θ−θ˙−0.018
PPO
f(x,x˙,θ,θ˙)=−0.193x−0.523x˙−θ−θ˙+0.0014
A2C
f(x,x˙,θ,θ˙)=−0.4875x−0.9811x˙−1.09θ−θ˙
Table 2: CartPole: interpretable policy boundary for SILVER Li et al. [2025]
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Mean absolute SHAP value of each feature for RoadRunner under different algorithms (DQN, A2C, PPO). The A2C rollout is single-action, so its SHAP attributions are negligible (note the axis scale).
Figure 10: SHAP value comparison of LunarLander under different algorithms (DQN, A2C, PPO)
Figure 11: Fidelity (action agreement with the RL policy) of the three interpretable surrogates (decision tree, linear regression, logistic regression) under full-distribution distillation, for each RL algorithm (DQN, A2C, PPO) and environment. Bars are grouped by algorithm; within each group one bar per surrogate. The A2C logistic-regression bar is omitted in MsPacman and RoadRunner because the collected rollouts contain fewer than two distinct actions.
Figure 12: Participants’ education level and experience in RL
Figure 13: Distribution of responses from Q1 to Q4
Figure 14: Participants’ trust ratings
Figure 15: Distribution of time cost from Q1 to Q4
Table 6: LunarLander: interpretable policy boundary for linear regression. Regressors are the physical state variables: position (x,y) , velocity (vx,vy) , angle θ , angular velocity ω , and leg-contact flags (cL,cR) .
Table 7: MsPacman PPO: interpretable policy boundary for logistic regression
The MsPacman A2C policy is single-action in the held-out rollout (it always selects action 8 ), so only one class is present. Logistic regression requires at least two distinct classes and is therefore undefined for this configuration.
Appendix
Table 8: MsPacman A2C: logistic regression is undefined; the rollout is single-action (always action 8 ).
Table 10: RoadRunner PPO: interpretable policy boundary for logistic regression
The RoadRunner A2C policy is single-action in the held-out rollout (it always selects action 4 ), so only one class is present. Logistic regression requires at least two distinct classes and is therefore undefined for this configuration.
Appendix
Table 11: RoadRunner A2C: logistic regression is undefined; the rollout is single-action (always action 4 )
Deep reinforcement learning (RL) achieves remarkable performance but lacks interpretability, limiting trust in policy behavior. The existing SILVER framework (Li, Siddique, and Cao 2025) explains RL policy via Shapley-based regression but remains restricted to low-dimensional, binary-action domains. We propose SILVER with RL-guided labeling, an enhanced variant that extends SILVER to multi-action and high-dimensional environments by incorporating the RL policy's own action outputs into the boundary points identification. Our method first extracts compact feature representations from image observations, performs SHAP-based feature attribution, and then employs RL-guided labeling to generate behaviorally consistent boundary datasets. Surrogate models, such as decision trees and regression-based functions, are subsequently trained to interpret RL policy's decision structure. We evaluate the proposed framework on two Atari environments using three deep RL algorithms and conduct human-subject study to assess the clarity and trustworthiness of the derived interpretable policy. Results show that our approach maintains competitive task performance while substantially improving transparency and human understanding of agent behavior. This work advances explainable RL by transforming SILVER into a scalable and behavior-aware framework for interpreting deep RL agents in high-dimensional, multi-action settings.
Explainability remains a key issue in reinforcement learning (RL). Distilling an interpretable policy from an agent trained in a complex environment is particularly challenging when the action space is continuous. We introduce ORCAID, a novel method for extracting interpretable rule-based policies from RL agents operating in mixed continuous-discrete environments with continuous action spaces. Our main contribution is an efficient oblique decision tree training algorithm that partitions the state space by hyperplanes and fits local linear models. The key idea lies in a three-stage split search: efficient random initialization, local refinement, and backward elimination. Finally, adjacent leaves are merged to yield a concise set of interpretable rules describing a given deep RL policy. We evaluate ORCAID across multiple RL environments, demonstrating that the extracted rule-based policies maintain strong performance with a low number of parameters and can even be used to improve the performance of the original deep RL policy.
Ignacio D. Lopez-Miguel, Ezio Bartocci, Thomas Eiter +1
Reinforcement learning policies are difficult to inspect, but interpreting them is a prerequisite for trustworthiness. Converting a trained policy into explicit decision-tree rules improves transparency and the resulting artifacts often remain too complex for human understanding. We present a pruning process that simplifies such rule-based policies while preserving task performance and making edits to the policy auditable. The process defines a small set of structural and usage-aware operators and evaluates candidate edits by re-executing the policy to measure return and interpretability proxies. This exposes an transformation process from complex to compact policy structures. We investigate this approach on classic control and MuJoCo benchmarks, where pruning traces reveal consistent interpretability improvements while maintaining high performance.
Mark Leon Ringer, Michel Tokic
Faculty of Mathematics, Informatics and Statistics, Ludwig-Maximilians-University Munich, Munich, Germany · Siemens AG, Data & Artificial Intelligence, Otto-Hahn-Ring 6, 81739 Munich, Germany