Hierarchical Support Vector State Partitioning for Distilling Black Box Reinforcement Learning Policies

Senne Deproost; Mehrdad Asadi; Ann Nowé

cs.LGMay 5, 2026

Hierarchical Support Vector State Partitioning for Distilling Black Box Reinforcement Learning Policies

Authors: Senne Deproost, Mehrdad Asadi, Ann Nowé

Organizations: aVrije Universiteit Brussel · bFlanders Make

Abstract

We introduce State Vector Space Partitioning (SVSP), a novel method to mimic a black box reinforcement learning policy using a set of human-interpretable subpolicies. By partitioning a distillation dataset of state action pairs with linear support vector machine splits, SVSP constructs a compact and structured representation of the original policy. Our method improves mean return by +7.4% over previous critic driven state partitioning attempts such as Voronoi State Partitioning (VSP) and +2.8% over the original TD3 policy, while reducing the number of required subpolicies against VSP by 82.1%. Our results pave the path towards a more flexible form of distillation where both the decision boundary and surrogate models can be chosen within a margin of the original black box behavior.

Explore similar work

Critic-Driven Voronoi-Quantization for Distilling Deep RL Policies to Explainable Models
May 14, 2026Senne Deproost, Denis Steckelmacher, Ann NowéCritic Learning Voronoi Regions
Self-Distilled Policy Gradient
Jun 2, 2026Yifeng Liu, Shiyuan Zhang, Yifan Zhang +1Unsupervised On-Policy Self-Distillation On-Policy Distillation
Trust-Region Behavior Blending for On-Policy Distillation
May 29, 2026Daniil Plyusov, Alexey Gorbatovski, Alexey Malakhov +4On-Policy Distillation Trust Region

May 14, 2026 · cs.LGJ/K move · Enter open · S save

Critic-Driven Voronoi-Quantization for Distilling Deep RL Policies to Explainable Models

Senne Deproost, Denis Steckelmacher, Ann Nowé

Despite many successful attempts at explaining Deep Reinforcement Learning policies using distillation, it remains difficult to balance the performance-interpretability trade-off and select a fitting surrogate model. In addition to this, traditional distillation only minimizes the distance between the behavior of the original and the surrogate policy while other RL-specific components such as action value are disregarded. To solve this, we introduce a new model-agnostic method called Critic-Driven Voronoi State Partitioning, which partitions a black box control policy into regions where a simple class of model can be optimized using gradient descent. By exploiting the critic value network of the original policy, we iteratively introduce new subpolicies in regions with insufficient value, standing in for a measure of policy complexity. The partitioning, a Voronoi quantizer, uses nearest neighbor lookups to assign a linear function to each point in the state space resulting in a cell-like diagram. We validate our approach on several well known benchmarks and proof that this distillation approaches the original policy using a reasonable sized set of linear functions.

Critic Learning Voronoi Regions

Open paper Ask about it PDF

Abstract

Explore similar work

Critic-Driven Voronoi-Quantization for Distilling Deep RL Policies to Explainable Models

Self-Distilled Policy Gradient

Trust-Region Behavior Blending for On-Policy Distillation