In locomotion control tasks, Deep Reinforcement Learning (DRL) has demonstrated high performance; however, the decision-making process of the learned policy remains a black box, making it difficult for humans to understand. On the other hand, in periodic motions such as walking, it is well known that implicit motion phases exist, such as the stance phase and the swing phase. Focusing on this point, this study hypothesizes that a policy trained for locomotion control may also represent a phase structure that is interpretable by humans. To examine this hypothesis in a controlled setting, we consider a locomotion task that is amenable to observing whether a policy autonomously acquires temporally structured phases through interaction with the environment. To verify this hypothesis, in the MuJoCo locomotion benchmark HalfCheetah-v5, the state transition sequences acquired by a policy trained for walking control through interaction with the environment were aggregated into semantic phases based on state similarity and consistency of subsequent transitions. As a result, we demonstrated that the state sequences generated by the trained policy exhibit periodic phase transition structures as well as phase branching. Furthermore, by approximating the states and actions corresponding to each semantic phase using Explainable Boosting Machines (EBMs), we analyzed phase-dependent decision making-namely, which state features the policy function attends to and how it controls action outputs in each phase. These results suggest that neural network-based policies, which are often regarded as black boxes, can autonomously acquire interpretable phase structures and logical branching mechanisms.
Figures & tables
Figure 1: Overview of the analysis of a locomotion-control policy function. This analysis is composed of two stages: (Left) identification of temporally semantic phases by embedding state sequences and clustering with transition entropy minimization, and (Right) phase-wise surrogate modeling using an Explainable Boosting Machine (EBM) to reveal state–action contribution rules within each phase.
Item
Specification
Simulator
MuJoCo
Benchmark Environment
HalfCheetah-v5
State Space Dimension
17 (Continuous)
Action Space Dimension
6 (Continuous Torque)
RL Algorithm
TD3
Policy Architecture
MLP (2 hidden layers with 256 units)
Table 1: Experimental settings and specifications.
Figure 2: Visualization of the state space projected by UMAP. Each dot represents a state, colored by its assigned cluster (phase). The blue edges connect each state to its successor state at the next time step. To clarify the dominant flow of phase transitions, white arrows are overlaid on the edges corresponding to transitions that cumulatively account for more than 70% of observed transitions from each source cluster (see Table 2), indicating the direction of those transitions.
N
Target Cluster
0
1
2
3
4
5
6
7
8
9
Source Cluster
0
764
137
563
3
0
0
0
16
44
0
0
1
622
0
5
610
5
0
0
0
1
0
0
2
645
6
0
20
612
5
0
0
0
2
0
3
621
1
0
0
3
609
7
0
0
0
0
4
615
77
0
0
0
0
537
0
0
0
0
Table 2: Transition Matrix of Source vs. Target Clusters. Entries corresponding to destination clusters that cumulatively account for more than 70% of transitions from each source cluster are shown in bold.
Figure 3: Randomly sampled rendered states from two phase–transition sequences. The upper row shows examples drawn from the transition cycle Cluster0→1→2→3→4→5→0 (Pattern 1), while the lower row shows examples drawn from the longer cycle Cluster0→1→2→3→4→5→6→7→8→9→0 (Pattern 2). For each cluster in the respective sequence, two states were randomly sampled from the corresponding state trajectory and rendered to visualize representative postures along the phase transitions.
K
0
1
2
3
4
5
6
7
8
9
R2
0.76
0.61
0.70
0.65
0.72
0.90
0.68
0.68
0.85
0.82
Table 3: Determination coefficient ( R2 ) of the EBM approximation for each phase.
Figure 4: Feature attribution heatmaps for Phase 0 approximated by EBM, together with representative robot postures for each phase. Red frames indicate the state–action pairs whose attribution values exceed 95% of the maximum heatmap value.
Figure 5: Feature attribution heatmaps for Phase 1, 2 and 3 approximated by EBM, together with representative robot postures for each phase. Red frames indicate the state–action pairs whose attribution values exceed 95% of the maximum heatmap value.
Figure 6: Feature attribution heatmaps for Phase 4 and 5 approximated by EBM, together with representative robot postures for each phase. Red frames indicate the state–action pairs whose attribution values exceed 95% of the maximum heatmap value.
Figure 7: Feature attribution heatmaps for Phase 6 to 9 approximated by EBM, together with representative robot postures for each phase. Red frames indicate the state–action pairs whose attribution values exceed 95% of the maximum heatmap value.
Figure 8: Comparison of feature attributions in the branching phase (Phase 5). The upper heatmap shows the contributions for transitions to Phase 0, while the lower heatmap shows the contributions for transitions to Phase 6. Note the distinct high contribution of Pitch Angular Velocity (Vel Ang) in the transition to Phase 6 (red box), indicating a momentum-based decision rule.
Figure 9: Actuated joints of the HalfCheetah robot. The six joints used for control are the front and back thighs, shins, and feet.
Index
Variable Name
Physical Meaning
Unit
0
Root Z
Root (torso) z position (height)
m
1
Root Ang
Root pitch angle
rad
2
B-Thigh
Back thigh angle
rad
3
B-Shin
Back shin angle
rad
4
B-Foot
Back foot angle
rad
5
F-Thigh
Front thigh angle
rad
Table 4: Details of the observation space in HalfCheetah-v5 (Dimension: 17)
Index
Joint Name
Controlled Target
0
B-Thigh
Torque applied to back thigh rotor
1
B-Shin
Torque applied to back shin rotor
2
B-Foot
Torque applied to back foot rotor
3
F-Thigh
Torque applied to front thigh rotor
4
F-Shin
Torque applied to front shin rotor
5
F-Foot
Torque applied to front foot rotor
Table 5: Details of the action space in HalfCheetah-v5 (Dimension: 6, Range: [−1,1] )
Deep reinforcement learning (DRL) has been shown to achieve high performance on locomotion control tasks in MuJoCo benchmarks such as HalfCheetah, Ant, and Walker2D. However, visualizing the motion structures internally obtained by a trained policy function implemented as a deep neural network remains challenging. It is known from biomechanics and related fields that locomotion control is realized through the repetition of motion phases such as the stance phase and swing phase. In this study, we propose a framework for uncovering latent motion phase structures from trajectories generated by locomotion control policies through interaction with the environment. The proposed method extends the clustering features from state observations alone to augmented features including actions, next states, and next actions, and introduces a method for determining the number of clusters that suppresses self-transitions. Applying the proposed method to three environments -- Ant-v5, HalfCheetah-v5, and Walker2D-v5 -- we successfully identified phase structures with clearer and more regular transition rules than those obtained by the existing method.
Daisuke Yasui, Toshitaka Matuki, Hiroshi Sato
Mathematics and Computer Science National Defense Academy of Japan Yokosuka, Japan
Deep reinforcement learning (DRL) agents achieve strong performance in complex environments, yet their decision-making processes remain difficult to interpret. We introduce SPOT (Sampling Policy Observation Tree), a novel model-agnostic, sampling-based framework for interpreting DRL policies. Given access to the policy and an environment simulator, SPOT constructs an interpretable finite-horizon tree by sampling actions and recursively simulating the resulting successor states. The tree provides an empirical representation of the policy's action preferences and their possible downstream evolution. We provide formal guarantees establishing SPOT's asymptotic recovery of the policy's unique most probable action and characterizing its disagreement behavior under high-entropy policies. We demonstrate SPOT in the SUMO-RL traffic-signal control domain. The case study illustrates how its tree-based representation can be used to inspect policy preferences, compare alternative future trajectories, and reveal downstream behaviors that are not visible through single-timestep feature-attribution methods.
Tamar Gozlan, Claudia V. Goldman
Benin School of Computer Science and Engineering, The Hebrew University of Jerusalem · Hebrew University Business School, Data Science Department, The Hebrew University of Jerusalem
Reinforcement learning has become the leading paradigm in legged locomotion, enabling complex behaviors from backflips to parkour via massively parallel simulation. Under PPO's non-stationarity, shallow networks remain the de facto architecture, supported by carefully staged environments; nevertheless, the representations learned by these policies remain poorly understood, specifically the relation to motion produced on hardware. In this work, we empirically study locomotion policies through the effective rank of the network Jacobian and show that conditioning rank on gait phase exposes representation structures invisible from a global rank. In particular, we find that architectural choices, namely observation and layer normalization with residual connections, allocate more than one additional dimension of effective rank to swing-dominant states than to stance-dominant states, which is mostly absent in vanilla MLPs. Building on this, we propose practical recommendations to convert these into diagnostics for healthier and smoother policies. In practice, this results in lower joint jitter, up to 2x, not only in simulation, but also on a physical Spot. More broadly, our findings show that how a policy distributes its capacity across the gait cycle is closely tied to its health and sim-to-real transfer.
Felipe Tommaselli, Ricardo V. Godoy, Thiago H. Segreto +2