In locomotion control tasks, Deep Reinforcement Learning (DRL) has demonstrated high performance; however, the decision-making process of the learned policy remains a black box, making it difficult for humans to understand. On the other hand, in periodic motions such as walking, it is well known that implicit motion phases exist, such as the stance phase and the swing phase. Focusing on this point, this study hypothesizes that a policy trained for locomotion control may also represent a phase structure that is interpretable by humans. To examine this hypothesis in a controlled setting, we consider a locomotion task that is amenable to observing whether a policy autonomously acquires temporally structured phases through interaction with the environment. To verify this hypothesis, in the MuJoCo locomotion benchmark HalfCheetah-v5, the state transition sequences acquired by a policy trained for walking control through interaction with the environment were aggregated into semantic phases based on state similarity and consistency of subsequent transitions. As a result, we demonstrated that the state sequences generated by the trained policy exhibit periodic phase transition structures as well as phase branching. Furthermore, by approximating the states and actions corresponding to each semantic phase using Explainable Boosting Machines (EBMs), we analyzed phase-dependent decision making-namely, which state features the policy function attends to and how it controls action outputs in each phase. These results suggest that neural network-based policies, which are often regarded as black boxes, can autonomously acquire interpretable phase structures and logical branching mechanisms.
Figures & tables
Figure 1: Overview of the analysis of a locomotion-control policy function. This analysis is composed of two stages: (Left) identification of temporally semantic phases by embedding state sequences and clustering with transition entropy minimization, and (Right) phase-wise surrogate modeling using an Explainable Boosting Machine (EBM) to reveal state–action contribution rules within each phase.
Item
Specification
Simulator
MuJoCo
Benchmark Environment
HalfCheetah-v5
State Space Dimension
17 (Continuous)
Action Space Dimension
6 (Continuous Torque)
RL Algorithm
TD3
Policy Architecture
MLP (2 hidden layers with 256 units)
Table 1: Experimental settings and specifications.
Figure 2: Visualization of the state space projected by UMAP. Each dot represents a state, colored by its assigned cluster (phase). The blue edges connect each state to its successor state at the next time step. To clarify the dominant flow of phase transitions, white arrows are overlaid on the edges corresponding to transitions that cumulatively account for more than 70% of observed transitions from each source cluster (see Table 2), indicating the direction of those transitions.
N
Target Cluster
0
1
2
3
4
5
6
7
8
9
Source Cluster
0
764
137
563
3
0
0
0
16
44
0
0
1
622
0
5
610
5
0
0
0
1
0
0
2
645
6
0
20
612
5
0
0
0
2
0
3
621
1
0
0
3
609
7
0
0
0
0
4
615
77
0
0
0
0
537
0
0
0
0
Table 2: Transition Matrix of Source vs. Target Clusters. Entries corresponding to destination clusters that cumulatively account for more than 70% of transitions from each source cluster are shown in bold.
Figure 3: Randomly sampled rendered states from two phase–transition sequences. The upper row shows examples drawn from the transition cycle Cluster0→1→2→3→4→5→0 (Pattern 1), while the lower row shows examples drawn from the longer cycle Cluster0→1→2→3→4→5→6→7→8→9→0 (Pattern 2). For each cluster in the respective sequence, two states were randomly sampled from the corresponding state trajectory and rendered to visualize representative postures along the phase transitions.
K
0
1
2
3
4
5
6
7
8
9
R2
0.76
0.61
0.70
0.65
0.72
0.90
0.68
0.68
0.85
0.82
Table 3: Determination coefficient ( R2 ) of the EBM approximation for each phase.
Figure 4: Feature attribution heatmaps for Phase 0 approximated by EBM, together with representative robot postures for each phase. Red frames indicate the state–action pairs whose attribution values exceed 95% of the maximum heatmap value.
Figure 5: Feature attribution heatmaps for Phase 1, 2 and 3 approximated by EBM, together with representative robot postures for each phase. Red frames indicate the state–action pairs whose attribution values exceed 95% of the maximum heatmap value.
Figure 6: Feature attribution heatmaps for Phase 4 and 5 approximated by EBM, together with representative robot postures for each phase. Red frames indicate the state–action pairs whose attribution values exceed 95% of the maximum heatmap value.
Figure 7: Feature attribution heatmaps for Phase 6 to 9 approximated by EBM, together with representative robot postures for each phase. Red frames indicate the state–action pairs whose attribution values exceed 95% of the maximum heatmap value.
Figure 8: Comparison of feature attributions in the branching phase (Phase 5). The upper heatmap shows the contributions for transitions to Phase 0, while the lower heatmap shows the contributions for transitions to Phase 6. Note the distinct high contribution of Pitch Angular Velocity (Vel Ang) in the transition to Phase 6 (red box), indicating a momentum-based decision rule.
Figure 9: Actuated joints of the HalfCheetah robot. The six joints used for control are the front and back thighs, shins, and feet.
Index
Variable Name
Physical Meaning
Unit
0
Root Z
Root (torso) z position (height)
m
1
Root Ang
Root pitch angle
rad
2
B-Thigh
Back thigh angle
rad
3
B-Shin
Back shin angle
rad
4
B-Foot
Back foot angle
rad
5
F-Thigh
Front thigh angle
rad
Table 4: Details of the observation space in HalfCheetah-v5 (Dimension: 17)
Index
Joint Name
Controlled Target
0
B-Thigh
Torque applied to back thigh rotor
1
B-Shin
Torque applied to back shin rotor
2
B-Foot
Torque applied to back foot rotor
3
F-Thigh
Torque applied to front thigh rotor
4
F-Shin
Torque applied to front shin rotor
5
F-Foot
Torque applied to front foot rotor
Table 5: Details of the action space in HalfCheetah-v5 (Dimension: 6, Range: [−1,1] )
Benin School of Computer Science and Engineering, The Hebrew University of Jerusalem · Hebrew University Business School, Data Science Department, The Hebrew University of Jerusalem