Adapting Critic Match Loss Landscape Visualization to Off-policy Reinforcement Learning

Jingyi Liu; Jian Guo; Eberhard Gill

cs.LGMar 15, 2026

Adapting Critic Match Loss Landscape Visualization to Off-policy Reinforcement Learning

Authors: Jingyi Liu, Jian Guo, Eberhard Gill

Organizations: Delft University of Technology, Delft, 2629 HS, the Netherlands

Abstract

This work extends an established critic match loss landscape visualization method from online to off-policy reinforcement learning (RL), aiming to reveal the optimization geometry behind critic learning. Off-policy RL differs from stepwise online actor-critic learning in its replay-based data flow and target computation. Based on these two structural differences, the critic match loss landscape visualization method is adapted to the Soft Actor-Critic (SAC) algorithm by aligning the loss evaluation with its batch-based data flow and target computation, using a fixed replay batch and precomputed critic targets from the selected policy. Critic parameters recorded during training are projected onto a principal component plane, where the critic match loss is evaluated to form a 3-D landscape with an overlaid 2-D optimization path. Applied to a spacecraft attitude control problem, the resulting landscapes are analyzed both qualitatively and quantitatively using sharpness, basin area, and local anisotropy metrics, together with temporal landscape snapshots. Comparisons between convergent SAC, divergent SAC, and divergent Action-Dependent Heuristic Dynamic Programming (ADHDP) cases reveal distinct geometric patterns and optimization behaviors under different algorithmic structures. The results demonstrate that the adapted critic match loss visualization framework serves as a geometric diagnostic tool for analyzing critic optimization dynamics in replay-based off-policy RL-based control problems.

Figures & tables

$\zeta$	Chosen center point in the landscape graph
$\bm{\delta},\bm{\eta}$	Selected directions to visualize the landscape
$\alpha,\beta$	Distances from the center point along directions $\delta$ and $\eta$
$f(\alpha,\beta)$	Loss function evaluated at perturbed parameters $(\alpha,\beta)$
$\mathcal{L}\!$	Loss function operator for the visualization
$J(t)$	Cost function at time step
$r(t)$	Reinforcement signal (reward) at time step $t$

Table 1

Panel of Figure 2 — Figure 2: SAC convergent control results for spacecraft attitude system

Panel of Figure 3 — Figure 3: Final 3-D and 2-D loss landscape of convergent spacecraft attitude SAC control

Panel of Figure 4 — Figure 4: 3-D and 2-D loss landscape of spacecraft attitude ADHDP control

Metric	SAC	ADHDP
Sharpness ( $\mathrm{Sharp}_{\epsilon}$ )	0.7255963360245651	0.10840051628952453
Basin ( $A_{\rho}$ )	1252.2596440947382	Non-existent
Local anisotropy ( $\log\kappa$ )	3.718889422035786	10.39062500455904

Table 2: Quantitative loss landscape comparison between SAC and ADHDP on attitude control task.

Panel of Figure 5 — Figure 5: Divergent control results for spacecraft attitude system under SAC with divergent critic

Panel of Figure 6 — Figure 6: Critic and actor weight with training steps for SAC with divergent critic

Panel of Figure 7 — Figure 7: 3-D and 2-D loss landscape of divergent spacecraft control under SAC with divergent critic

Panel of Figure 8 — Figure 8: Snapshots during training of 3-D and 2-D loss landscape of divergent spacecraft attitude control under SAC with divergent critic

Panel of Figure 9 — Figure 9: Divergent control results for spacecraft attitude system under SAC with convergent critic

Panel of Figure 10 — Figure 10: 3-D and 2-D loss landscape of divergent spacecraft attitude control under SAC with convergent critic

Panel of Figure 11 — Figure 11: Snapshots during training of 3-D and 2-D loss landscape of divergent spacecraft attitude control under SAC with convergent critic

Panel of Figure 12 — Figure 12: SAC control results for cart-pole system

Panel of Figure 13 — Figure 13: 3-D and 2-D loss landscape of spacecraft attitude SAC control

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 14 — Figure 14: ADHDP control results for cart-pole system

Loss Landscape Geometry

Loss landscape geometry studies how the shape of learning objectives affects optimization, convergence, and generalization.

See all 34 papers

Visualizing Critic Match Loss Landscapes for Interpretation of Online Reinforcement Learning Control Algorithms
Mar 15, 2026Jingyi Liu, Jian Guo, Eberhard GillRL Control Actor-Critic Methods
Faculty of Aerospace Engineering, Delft University of Technology, Kluyverweg 1, Delft, 2629 HS, The Netherlands
A Loss Landscape Visualization Framework for Interpreting Reinforcement Learning: An ADHDP Case Study
Mar 15, 2026Jingyi Liu, Jian Guo, Eberhard GillLoss Landscape Interpretability in RL
Faculty of Aerospace Engineering, Delft University of Technology, Kluyverweg 1, Delft, 2629 HS, The Netherlands
Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learning
Apr 21, 2026Yuan Zhuang, Yuexin Bian, Sihong He +7Low-Rank Adaptation Off-Policy RL
University of Connecticut · University of California San Diego · University of Texas at Arlington +1

Mar 15, 2026J/K move · Enter open · S save

Visualizing Critic Match Loss Landscapes for Interpretation of Online Reinforcement Learning Control Algorithms

Jingyi Liu, Jian Guo, Eberhard Gill

Faculty of Aerospace Engineering, Delft University of Technology, Kluyverweg 1, Delft, 2629 HS, The Netherlands

RL Control Actor-Critic Methods

Open paper AskPDF arXiv

Reinforcement learning has proven its power on various occasions. However, its performance is not always guaranteed when system dynamics change. Instead, it largely relies on users' empirical experience. For reinforcement learning algorithms with an actor-critic structure, the critic neural network reflects the approximation and optimization process in the RL algorithm. Analyzing the performance of the critic neural network helps to understand the mechanism of the algorithm. To support systematic interpretation of such algorithms in dynamic control problems, this work proposes a critic match loss landscape visualization method for online reinforcement learning. The method constructs a loss landscape by projecting recorded critic parameter trajectories onto a low-dimensional linear subspace. The critic match loss is evaluated over the projected parameter grid using fixed reference state samples and temporal-difference targets. This yields a three-dimensional loss surface together with a two-dimensional optimization path that characterizes critic learning behavior. To extend analysis beyond visual inspection, quantitative landscape indices and a normalized system performance index are introduced, enabling structured comparison across different training outcomes. The approach is demonstrated using the Action-Dependent Heuristic Dynamic Programming algorithm on cart-pole and spacecraft attitude control tasks. Comparative analyses across projection methods and training stages reveal distinct landscape characteristics associated with stable convergence and unstable learning. The proposed framework enables both qualitative and quantitative interpretation of critic optimization behavior in online reinforcement learning.

Abstract

Loss Landscape Geometry

Explore similar work

Visualizing Critic Match Loss Landscapes for Interpretation of Online Reinforcement Learning Control Algorithms

A Loss Landscape Visualization Framework for Interpreting Reinforcement Learning: An ADHDP Case Study

Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learning