Reinforcement learning algorithms have been widely used in dynamic and control systems. However, interpreting their internal learning behavior remains a challenge. In the authors' previous work, a critic match loss landscape visualization method was proposed to study critic training. This study extends that method into a framework which provides a multi-perspective view of the learning dynamics, clarifying how value estimation, policy optimization, and temporal-difference (TD) signals interact during training. The proposed framework includes four complementary components; a three-dimensional reconstruction of the critic match loss surface that shows how TD targets shape the optimization geometry; an actor loss landscape under a frozen critic that reveals how the policy exploits that geometry; a trajectory combining time, Bellman error, and policy weights that indicates how updates move across the surface; and a state-TD map that identifies the state regions that drive those updates. The Action-Dependent Heuristic Dynamic Programming (ADHDP) algorithm for spacecraft attitude control is used as a case study. The framework is applied to compare several ADHDP variants and shows how training stabilizers and target updates change the optimization landscape and affect learning stability. Therefore, the proposed framework provides a systematic and interpretable tool for analyzing reinforcement learning behavior across algorithmic designs.
Figures & tables
Visualization index
Input data
Output figure
Interpretation focus
Critic match loss landscape
Recorded critic weights; states-action samples and TD targets during training
3-D loss surface over critic weight space
Evolution of critic fitting behavior during training
Actor loss landscape
Recorded actor weights; frozen critic as cost function
3-D loss surface over actor weight space
Geometry and quality of the learned policy
Time–TD–actor weight trajectory
Actor weights and TD errors recorded across training steps
3-D trajectory of actor weight vs. TD vs. time
Coupling between policy drift and Bellman error
State–TD plot
System states and TD errors recorded during simulation
2-D plot of state vs. TD
State regions that dominate TD fluctuations
Table 1: Summary of visualization indices in the proposed framework.
Figure 1: Structure of ADHDP
No.
ADHDP variant
Target network
Numeric cost scaling
Huber critic loss
Target-policy smoothing (TPS)
1
Basic version
–
–
–
–
2
With target network
✓
–
–
–
3
With training stabilizers + TPS
✓
✓
✓
✓
4
With training stabilizers only
✓
✓
✓
–
Table 2: DRL techniques applied in each ADHDP variant. The numbering corresponds to the order of variants discussed in the text.
Figure 2: State trajectory, basic ADHDP
Figure 3: Loss surface, basic ADHDP
Figure 4: TD trajectory, basic ADHDP
Figure 5: State trajectory, ADHDP with target net
Figure 6: Loss surface, ADHDP with target net
Figure 7: TD trajectory, ADHDP with target net
Figure 8: ADHDP with training stabilizers (Huber critic loss and numeric cost scaling) and target policy smoothing
Figure 9: Loss surface, ADHDP with training stabilizers (Huber critic loss and numeric cost scaling) and target policy smoothing
Figure 10: TD trajectory, ADHDP with training stabilizers (Huber critic loss and numeric cost scaling) and target policy smoothing
Figure 11: State trajectory, ADHDP with training stabilizers(Huber critic loss and numeric cost scaling), no target policy smoothing
Figure 12: Loss surface, ADHDP with training stabilizers(Huber critic loss and numeric cost scaling), no target policy smoothing
Figure 13: TD trajectory, ADHDP with training stabilizers(Huber critic loss and numeric cost scaling), no target policy smoothing
Reinforcement learning has proven its power on various occasions. However, its performance is not always guaranteed when system dynamics change. Instead, it largely relies on users' empirical experience. For reinforcement learning algorithms with an actor-critic structure, the critic neural network reflects the approximation and optimization process in the RL algorithm. Analyzing the performance of the critic neural network helps to understand the mechanism of the algorithm. To support systematic interpretation of such algorithms in dynamic control problems, this work proposes a critic match loss landscape visualization method for online reinforcement learning. The method constructs a loss landscape by projecting recorded critic parameter trajectories onto a low-dimensional linear subspace. The critic match loss is evaluated over the projected parameter grid using fixed reference state samples and temporal-difference targets. This yields a three-dimensional loss surface together with a two-dimensional optimization path that characterizes critic learning behavior. To extend analysis beyond visual inspection, quantitative landscape indices and a normalized system performance index are introduced, enabling structured comparison across different training outcomes. The approach is demonstrated using the Action-Dependent Heuristic Dynamic Programming algorithm on cart-pole and spacecraft attitude control tasks. Comparative analyses across projection methods and training stages reveal distinct landscape characteristics associated with stable convergence and unstable learning. The proposed framework enables both qualitative and quantitative interpretation of critic optimization behavior in online reinforcement learning.
Jingyi Liu, Jian Guo, Eberhard Gill
Faculty of Aerospace Engineering, Delft University of Technology, Kluyverweg 1, Delft, 2629 HS, The Netherlands
This work extends an established critic match loss landscape visualization method from online to off-policy reinforcement learning (RL), aiming to reveal the optimization geometry behind critic learning. Off-policy RL differs from stepwise online actor-critic learning in its replay-based data flow and target computation. Based on these two structural differences, the critic match loss landscape visualization method is adapted to the Soft Actor-Critic (SAC) algorithm by aligning the loss evaluation with its batch-based data flow and target computation, using a fixed replay batch and precomputed critic targets from the selected policy. Critic parameters recorded during training are projected onto a principal component plane, where the critic match loss is evaluated to form a 3-D landscape with an overlaid 2-D optimization path. Applied to a spacecraft attitude control problem, the resulting landscapes are analyzed both qualitatively and quantitatively using sharpness, basin area, and local anisotropy metrics, together with temporal landscape snapshots. Comparisons between convergent SAC, divergent SAC, and divergent Action-Dependent Heuristic Dynamic Programming (ADHDP) cases reveal distinct geometric patterns and optimization behaviors under different algorithmic structures. The results demonstrate that the adapted critic match loss visualization framework serves as a geometric diagnostic tool for analyzing critic optimization dynamics in replay-based off-policy RL-based control problems.
Jingyi Liu, Jian Guo, Eberhard Gill
Delft University of Technology, Delft, 2629 HS, the Netherlands
Deep reinforcement learning has the potential to solve attitude control problems more adaptively, precisely, and robustly by handling nonlinear dynamics, uncertainties, and failure cases more effectively than traditional attitude control approaches. We explore reinforcement learning (RL) for attitude control in spacecraft re-entry. An industry-standard proportional-integral-derivative controller with gain scheduling serves as a strong baseline for model-free RL and hybrid controllers that combine these two approaches. We formalize the application in the RL framework to apply continuous, off-policy RL. State-of-the-art RL achieves comparable performance to traditional control approaches in this domain. However, its out-of-distribution generalization is not sufficient. Hence, we use dynamics randomization to introduce challenging task variations during training and enforce generalization in a predefined operational envelope. Finally, we assess the best obtained RL-based controllers with application-specific metrics to show superior performance in comparison to traditional controllers in the operational envelope, that is, hybrid controllers are able to track the angle of attack better and are more robust under variations of mass, inertia tensor, and flap actuator bandwidth.
Alexander Fabisch, Melvin Laux, Mariela De Lucas Álvarez +2
Robotics Innovation Center, German Research Center for Artificial Intelligence (DFKI GmbH), Germany · 2Airbus Defence and Space GmbH, Germany