Interpret Policies in Deep Reinforcement Learning using SILVER with RL-Guided Labeling: A Model-level Approach to High-dimensional and Multi-action Environments
Abstract
Deep reinforcement learning (RL) achieves remarkable performance but lacks interpretability, limiting trust in policy behavior. The existing SILVER framework (Li, Siddique, and Cao 2025) explains RL policy via Shapley-based regression but remains restricted to low-dimensional, binary-action domains. We propose SILVER with RL-guided labeling, an enhanced variant that extends SILVER to multi-action and high-dimensional environments by incorporating the RL policy's own action outputs into the boundary points identification. Our method first extracts compact feature representations from image observations, performs SHAP-based feature attribution, and then employs RL-guided labeling to generate behaviorally consistent boundary datasets. Surrogate models, such as decision trees and regression-based functions, are subsequently trained to interpret RL policy's decision structure. We evaluate the proposed framework on two Atari environments using three deep RL algorithms and conduct human-subject study to assess the clarity and trustworthiness of the derived interpretable policy. Results show that our approach maintains competitive task performance while substantially improving transparency and human understanding of agent behavior. This work advances explainable RL by transforming SILVER into a scalable and behavior-aware framework for interpreting deep RL agents in high-dimensional, multi-action settings.
Figures & tables
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
| Aspect | SILVER ( Li, Siddique, and Cao 2025 ) | SILVER with RL-Guided Labeling |
|---|---|---|
| Target environments | Restricted to binary-action, low-dimensional tasks (e.g., CartPole, MountainCar) | Scalable to high-dimensional, multi-action environments (up to 18 discrete actions) |
| Boundary identification | Determined purely by geometric distances between action clusters in Shapley vectors | Retains geometric identification but assigns action labels to each boundary state based on the output of RL policy |
| Labeling mechanism | Implicitly determined by nearest cluster pairs in Shapley vectors | Explicitly determined by querying the RL policy for action labels |
| Resulting data | Boundary samples may be ambiguous or misaligned with the actual RL policy behavior | Behaviorally consistent dataset enabling reliable interpretable models training |
| Feature 1 | Feature 2 | Feature 3 | Feature 4 | Feature 5 | Feature 6 | Feature 7 | Feature 8 |
|---|---|---|---|---|---|---|---|
| 0.053718 | -0.58133 | -1.57969 | -10.0231 | 10.97137 | -3.29043 | -0.34815 | -0.07359 |
| -0.13582 | 0.166289 | 17.79752 | -8.14196 | 16.23918 | 0.061392 | 0.271136 | 362.5037 |
| -0.06903 | -0.93946 | 21.95477 | 194.3563 | 54.4827 | -3.42446 | -0.39898 | 104.8359 |
| 0.147134 | -0.61962 | 21.12769 | 11.61575 | 41.8084 | -5.48569 | -0.59579 | 61.4086 |
| -0.08426 | -0.2127 | -3.38793 | -5.58269 | -26.1117 | -0.91717 | -0.01381 | -45.8628 |
| -0.09114 | -0.1284 | 18.9745 | -8.89167 | 29.9608 | -2.01041 | -0.14159 | 269.8545 |
| Feature 1 | Feature 2 | Feature 3 | Feature 4 | Feature 5 | Feature 6 | Feature 7 | Feature 8 |
|---|---|---|---|---|---|---|---|
| 0 | 0 | 0 | 0 | 55.7541 | 0 | 0 | 721.5164 |
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 55.56659 |
| 0 | 0 | 0 | 501.7624 | 0 | 0 | 0 | 336.9333 |
| 0 | 0 | 0 | 85.61454 | 0 | 0 | 0 | 431.6569 |
| 0 | 0 | 0 | 0 | 489.7511 | 0 | 0 | 1842.243 |
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 137.5602 |
| Feature 1 | Feature 2 | Feature 3 | Feature 4 | Feature 5 | Feature 6 | Feature 7 | Feature 8 |
|---|---|---|---|---|---|---|---|
| 0 | -0.00083 | -0.47892 | 1.12E-05 | 3.01E-05 | 0 | 0 | 0.058596 |
| 0 | 0.000949 | -0.27889 | -1.53E-05 | 1.98E-05 | 0 | 0 | -0.1481 |
| 0 | -0.00135 | 0.477397 | -5.52E-08 | 6.72E-06 | 0 | 0 | 0.135079 |
| 0 | -0.00022 | 0.081397 | -2.44E-06 | -1.36E-05 | 0 | 0 | -0.29276 |
| 0 | -0.00043 | -0.24212 | 1.18E-05 | 7.31E-05 | 0 | 0 | -0.29675 |
| 0 | 0.000124 | 2.800792 | -2.25E-05 | -8.00E-05 | 0 | 0 | 0.189882 |
| Feature 1 | Feature 2 | Feature 3 | Feature 4 | Feature 5 | Feature 6 | Feature 7 | Feature 8 |
|---|---|---|---|---|---|---|---|
| 0 | 0 | 623.6633 | 0 | 0 | 0 | 0 | 532.5274 |
| 0 | 0 | 653.3041 | 0 | 0 | 0 | 0 | 107.4317 |
| 0 | 0 | 1341.658 | 0 | 0 | 0 | 0 | 727.1529 |
| 0 | 0 | 1061.923 | 0 | 0 | 0 | 0 | 494.4895 |
| 0 | 0 | 616.0728 | 0 | 0 | 0 | 0 | 259.9639 |
| 0 | 0 | 478.1401 | 0 | 0 | 0 | 0 | 696.2931 |
| Feature 1 | Feature 2 | Feature 3 | Feature 4 | Feature 5 | Feature 6 | Feature 7 | Feature 8 |
|---|---|---|---|---|---|---|---|
| 0.000 | -0.28516 | 0.000 | 0.072798 | 0.000265 | 0.000 | 0.000 | 0.000 |
| 0.000 | -0.08226 | 0.000 | 0.378046 | -0.00056 | 0.000 | 0.000 | 0.000 |
| 0.000 | 0.86831 | 0.000 | 0.295693 | -0.00029 | 0.000 | 0.000 | 0.000 |
| 0.000 | -0.58381 | 0.000 | 0.084519 | -0.000063 | 0.000 | 0.000 | 0.000 |
| 0.000 | -0.37113 | 0.000 | -0.08261 | -0.00043 | 0.000 | 0.000 | 0.000 |
| 0.000 | -0.23301 | 0.000 | -0.11092 | -0.00054 | 0.000 | 0.000 | 0.000 |
| Feature 1 | Feature 2 | Feature 3 | Feature 4 | Feature 5 | Feature 6 | Feature 7 | Feature 8 |
|---|---|---|---|---|---|---|---|
| 0.0 | 5753.206 | 0.0 | 530.9578 | 0.0 | 0.0 | 0.0 | 0.0 |
| 0.0 | 6628.804 | 0.0 | 1914.645 | 0.0 | 0.0 | 0.0 | 0.0 |
| 0.0 | 0.000 | 0.0 | 2528.653 | 0.0 | 0.0 | 0.0 | 0.0 |
| 0.0 | 4611.498 | 0.0 | 550.1152 | 0.0 | 0.0 | 0.0 | 0.0 |
| 0.0 | 8737.567 | 0.0 | 0.000 | 0.0 | 0.0 | 0.0 | 0.0 |
| 0.0 | 7480.565 | 0.0 | 0.000 | 0.0 | 0.0 | 0.0 | 0.0 |
| Feature 1 | Feature 2 | Feature 3 | Feature 4 | Feature 5 | Feature 6 | Feature 7 | Feature 8 |
|---|---|---|---|---|---|---|---|
| 5.474527 | -0.81353 | 67.99976 | -0.18912 | -26.918 | -0.15756 | -1.73123 | -10.2803 |
| 2.992853 | 2.733483 | -3.56914 | -0.13995 | 5.06779 | 0.207614 | -3.64042 | 25.05078 |
| 1.108827 | -0.73966 | -10.4527 | -0.04402 | 8.2174 | -0.04362 | -2.02183 | -9.78841 |
| 12.07903 | -0.79765 | 190.9837 | -0.37814 | 3.13685 | -0.28556 | -2.23423 | 106.7969 |
| 10.21576 | 3.761232 | 68.81969 | -0.36309 | 4.5211 | -0.27273 | -3.14379 | 86.7971 |
| 10.2965 | -0.78144 | 74.93028 | -0.32361 | 4.95804 | -0.19189 | -2.76325 | 86.90049 |
| Feature 1 | Feature 2 | Feature 3 | Feature 4 | Feature 5 | Feature 6 | Feature 7 | Feature 8 |
|---|---|---|---|---|---|---|---|
| 0.000 | 0.000 | 188.7657 | 0.000 | 208.6209 | 0.000 | 0.000 | 0.000 |
| 0.000 | 13.94768 | 50.96701 | 0.000 | 18.80776 | 8.201801 | 0.000 | 60.99874 |
| 0.000 | 0.000 | 35.64004 | 0.000 | 105.0581 | 0.000 | 0.000 | 0.000 |
| 0.000 | 0.000 | 227.0123 | 0.000 | 95.6398 | 0.000 | 0.000 | 127.3856 |
| 0.000 | 14.1228 | 113.9336 | 0.000 | 49.6401 | 0.000 | 0.000 | 106.4597 |
| 0.000 | 0.000 | 138.5666 | 0.000 | 23.21588 | 0.000 | 0.000 | 110.8463 |
| Feature 1 | Feature 2 | Feature 3 | Feature 4 | Feature 5 | Feature 6 | Feature 7 | Feature 8 |
|---|---|---|---|---|---|---|---|
| -0.0016 | 0.012969 | 0.002574 | 0.006918 | 0.090507 | 0.004858 | 0.344943 | 4.097872 |
| 0.067005 | 0.111029 | -0.00269 | 0.056097 | 0.010497 | -0.0101 | 0.396967 | 2.144845 |
| 0.168133 | 0.205574 | -0.00202 | 0.046697 | -0.085069 | -0.0076 | 0.334608 | 3.086863 |
| 0.251399 | 0.11519 | -0.00046 | 0.061054 | -1.47655 | 0.007191 | 0.419466 | 2.611407 |
| 0.172695 | 0.072814 | 0.001032 | 0.126251 | 2.13337 | 0.092112 | 0.115391 | 4.472525 |
| 0.117376 | 0.144224 | -0.00195 | 0.032066 | -2.19087 | 0.00218 | 0.313983 | 2.340529 |
| Feature 1 | Feature 2 | Feature 3 | Feature 4 | Feature 5 | Feature 6 | Feature 7 | Feature 8 |
|---|---|---|---|---|---|---|---|
| 161.505 | 0.000 | 0.000 | 0.000 | 849.9095 | 0.000 | 0.000 | 1037.385 |
| 95.19944 | 0.000 | 0.000 | 0.000 | 470.2562 | 0.000 | 0.000 | 0.000 |
| 66.96087 | 0.000 | 0.000 | 0.000 | 172.4873 | 0.000 | 6.232745 | 0.000 |
| 51.1582 | 50.33921 | 0.000 | 0.000 | 62.84172 | 237.3442 | 33.74626 | 936.9337 |
| 549.525 | 101.9358 | 0.000 | 0.000 | 104.4528 | 0.000 | 0.000 | 0.000 |
| 89.66468 | 0.000 | 0.000 | 0.000 | 93.95311 | 0.000 | 35.43744 | 0.000 |
| Feature 1 | Feature 2 | Feature 3 | Feature 4 | Feature 5 | Feature 6 | Feature 7 | Feature 8 |
|---|---|---|---|---|---|---|---|
| -4.60653 | 0.296192 | 0.969315 | 0.000136 | 0.08256 | 0.000 | 2.431848 | 0.011123 |
| 0.727034 | 0.678398 | 1.124768 | -0.000008 | 1.15713 | 0.000 | 0.155846 | -0.00254 |
| 0.177717 | 0.13062 | -0.09505 | -0.00013 | 0.01159 | 0.000 | 0.778338 | -0.00021 |
| 0.301813 | 0.12316 | 0.271282 | -0.00018 | 0.03311 | 0.000 | 0.486024 | 0.011636 |
| 0.414713 | -0.20925 | 1.222974 | 0.000054 | 0.10786 | 0.000 | 0.251595 | -0.01727 |
| 1.700075 | 0.199318 | 0.225426 | -0.000032 | 0.01006 | 0.000 | 0.652782 | 0.030314 |
| Feature 1 | Feature 2 | Feature 3 | Feature 4 | Feature 5 | Feature 6 | Feature 7 | Feature 8 |
|---|---|---|---|---|---|---|---|
| 393.7799 | 0.000 | 710.9127 | 0.000 | 0.000 | 0.000 | 706.2158 | 0.000 |
| 0.000 | 16696.2 | 19701.97 | 0.000 | 0.000 | 0.000 | 1955.914 | 0.000 |
| 1478.477 | 0.000 | 2021.436 | 0.000 | 0.000 | 0.000 | 727.3788 | 60.47329 |
| 1949.78 | 0.000 | 440.0298 | 0.000 | 404.4525 | 0.000 | 883.1434 | 0.000 |
| 1911.436 | 380.9045 | 242.8682 | 0.000 | 440.9319 | 0.000 | 1777.836 | 0.000 |
| 2592.608 | 0.000 | 1427.355 | 0.000 | 0.000 | 0.000 | 1441.367 | 0.000 |
| Algorithm | Decision Boundary |
|---|---|
| PPO | |
| A2C | |
| DQN |
| Algorithm | Decision Boundary |
|---|---|
| PPO | |
| A2C | |
| DQN |
| Action | |
|---|---|
| 5 | |
| 6 | |
| 7 | |
| 8 |
| Action | |
|---|---|
| 1 | |
| 5 | |
| 7 | |
| 8 |
| Action | |
|---|---|
| 0 | |
| 1 | |
| 2 | |
| 3 | |
| 5 | |
| 6 |
| Action | |
|---|---|
| 2 | |
| 7 | |
| 9 | |
| 12 | |
| 17 |
| Action | |
|---|---|
| 2 | |
| 4 | |
| 5 | |
| 7 | |
| 9 |
| Action | |
|---|---|
| 2 | |
| 3 | |
| 5 | |
| 6 | |
| 7 | |
| 8 |
Interpretability in RL
Interpretability in reinforcement learning makes learned policies, decisions, and internal representations understandable and auditable.
See all 79 papers