ReShap: Interpretable Reinforcement Learning for High-Dimensional Multi-Action Settings
Organizations: RMIT University · Damiao Tech · University of California, Irvine
Abstract
Deep Reinforcement Learning (RL) plays a crucial role in tackling complex tasks, yet its reliance on deep neural networks makes the learned policies opaque. To enhance the trustworthiness of RL policies, existing work (e.g., the SILVER framework [Li et al., 2025]) has focused on Shapley-based interpretability, but is typically restricted to low-dimensional state spaces, binary-action settings, and a single class of linear surrogate. We propose ReShap, a policy-distillation framework that extends Shapley-based interpretability to high-dimensional, multi-action environments. ReShap extracts compact feature representations from raw image observations, performs SHAP-based feature attribution to characterize how each feature contributes to the policy's decisions, and distills the deep RL policy into an interpretable surrogate trained on the full set of (state, policy-action) pairs along the policy's visited distribution. We instantiate the surrogate with three interpretable model families (decision trees, linear regression, and logistic regression) and compare their fidelity and task return. We evaluate ReShap on Atari and Box2D environments using three deep RL algorithms and conduct a human-subject study to assess the clarity and trustworthiness of the resulting policies. Results show that ReShap preserves competitive task performance, that decision trees generally outperform linear and logistic regression, and that linear regression is weakest in most settings.
Figures & tables
| Schemes | Target Environments | Action Space | Decision Boundary Modeling | Action Labeling | Policy-explanation Alignment |
|---|---|---|---|---|---|
| Causal Lens Madumal et al. [2020] | Low-dimensional control and box2d (e.g., Mountain Car, Lunar Lander) | Discrete (low-cardinality) | None (Structural causal model only) | Not applicable | Not applicable |
| SVERL Beechey et al. [2023] | Low-dimensional discrete states (e.g., Taxi, Minesweeper) | Discrete (low-cardinality) | None (Shapley attribution only) | Not applicable | Not applicable |
| DISTILL2EXPLAIN Gokhale et al. [2024] | Low-dimensional control (e.g., Simulation of home energy management) | Discrete (low-cardinality) | None (Policy distillation only) | Not applicable | Behaviorally aligned |
| SILVER Li et al. [2025] | Low-dimensional control (e.g., CartPole, Mountain Car) | Discrete (binary) | Geometric distance in Shapley space | Implicit cluster-based | Potentially misaligned |
| ReShap (Ours) | High-dimensional visual (e.g., MsPacman, RoadRunner) | Discrete (high-cardinality) | Full-distribution policy distillation | Direct policy actions | Behaviorally consistent |
| Algorithm | Decision Boundary |
|---|---|
| DQN | |
| PPO | |
| A2C |
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
| Algorithm | Decision Boundary |
|---|---|
| PPO | |
| A2C | (constant; the A2C rollout is single-action) |
| DQN |
| Algorithm | Decision Boundary |
|---|---|
| PPO | |
| A2C | (constant; the A2C rollout is single-action) |
| DQN |
| Algorithm | Decision Boundary |
|---|---|
| PPO | |
| A2C | |
| DQN |
| Action | |
|---|---|
| 0 | |
| 6 | |
| 7 | |
| 8 |
| The MsPacman A2C policy is single-action in the held-out rollout (it always selects action ), so only one class is present. Logistic regression requires at least two distinct classes and is therefore undefined for this configuration. |
| Action | |
|---|---|
| 1 | |
| 3 | |
| 4 | |
| 5 | |
| 6 | |
| 7 |
| Action | |
|---|---|
| 2 | |
| 4 | |
| 5 | |
| 6 | |
| 7 | |
| 8 |
| The RoadRunner A2C policy is single-action in the held-out rollout (it always selects action ), so only one class is present. Logistic regression requires at least two distinct classes and is therefore undefined for this configuration. |
| Action | |
|---|---|
| 0 | |
| 2 | |
| 3 | |
| 4 | |
| 5 | |
| 7 |
| Action | |
|---|---|
| 0 | |
| 1 | |
| 2 | |
| 3 |
| Action | |
|---|---|
| 0 | |
| 1 | |
| 2 | |
| 3 |
| Action | |
|---|---|
| 0 | |
| 1 | |
| 2 | |
| 3 |
Interpretability in RL
Interpretability in reinforcement learning makes learned policies, decisions, and internal representations understandable and auditable.
See all 79 papers