Soft Actor-Critic (SAC) is a widely used deep reinforcement learning algorithm that combines entropy regularization with an actor-critic architecture. Standard implementations rely on one-step temporal-difference learning of Q-functions, yet alternative critic design choices remain underexplored. In this work, we revisit three such choices: learning soft advantages instead of soft Q-values, the use of multi-step return targets, and the training of multi-step critics from replay buffer data. Keeping the policy objective fixed, we compare the six critic variants these choices generate: (i) advantage and (ii) dueling-advantage critics learned from one-step transitions, and λ-return and soft Generalized Advantage Estimation (GAE) critics learned either (iii, iv) on-policy or (v, vi) from replayed trajectory segments. Each variant is evaluated across an entire random hyperparameter search on fourteen tasks, treating the hyperparameters as part of the algorithm. Learning soft advantages instead of soft Q-values does not improve performance, and the soft GAE critic adds nothing over the simpler λ-return critic. What does improve performance, most clearly on continuous control, is training a λ-return critic on replayed trajectory segments, even without importance weighting or trace cutting. This extends earlier evidence that uncorrected multi-step targets work well under replay to the full soft λ-return, entropy terms included. By design, the underlying ideas studied here extend beyond SAC, and our empirical findings aim to encourage their application to a broader class of reinforcement learning algorithms.
Figures & tables
Figure 1: Design choices and resulting variants. The critic is trained on replayed or on-policy data, with one-step or multi-step λ -return targets, and predicts soft Q-values Qαπ or soft advantages Aαπ . SAC occupies the one-step, replay-based Q-value cell; the other five cells are the variants we study.
Figure 2: Training pipelines of the SAC variants. (a) SAC-A and SAC-DA keep SAC’s replay-based one-step pipeline but learn soft advantages. (b) SAC trains its Q-network on replayed transitions, SAC- λ on parallel on-policy rollouts, and SAC- λ -R on replayed trajectory segments. (c) SAC-GAE and SAC-GAE-R learn a soft value network, on rollouts or replayed segments, and compute soft GAEs for the policy update.
Figure 3: Performance profiles and peak performance. (a) & (b) Performance profiles: the share of configurations reaching normalized score τ (Eq. 15 ). (c) & (d) Peak score against sensitivity Φ ; dashed lines mark constant relative sensitivity Φ/Peak , and the lower right corner is best. Bands and bars are 95% stratified bootstrap confidence intervals. PQN is only defined for discrete action spaces and is omitted for continuous action spaces.
Figure 4: Hyperparameter transfer performance and effect of the horizon. (a) & (b) Best per-task configuration against the best configuration transferred from all other tasks. (c) & (d) Normalized score against target horizon N . The on-policy variants (solid) are sampled at N≤32 , the replay-based ones (dashed) at N≥32 . Bands and bars are 95% stratified bootstrap confidence intervals.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Variance decomposition. Share of total variance attributable to task, configuration, training seed and evaluation episode, over all configurations (a, c) and over each method’s top-quartile configurations (b, d), on discrete (a, b) and continuous (c, d) tasks.
Figure 6: Per-task ranks (1 = best). Left: by peak score maxhxh,m ; right: by IQMh(xh,m) . PQN is not defined for continuous action spaces.
Figure 7: Random-search budget curves. Expected normalized score of the best shared configuration among n sampled ones. Bands are 95% bootstrap confidence intervals.
Figure 8: Probability of improvement at N=32 of the replay-based over the on-policy variant, per task and aggregated. (a): SAC-GAE-R vs. SAC-GAE; (b): SAC- λ -R vs. SAC- λ ; the dashed line marks 0.5 . Bars are 95% stratified bootstrap confidence intervals.
Figure 9: Effect of the sampled λ . normalized score against λ , binned at bin centers, for every method that samples it. The λ -return critics are sampled on [0.50,0.95] and the GAE critics on [0.85,0.99] , so the two families occupy different parts of the axis and should not be compared by slope across families. Bands are 95% bootstrap confidence intervals; slopes are given in Table 1 .
Marginal
Partial
Method
slope
lower
upper
slope
lower
upper
Discrete
SAC- λ
+0.3
−1.3
+2.0
+0.3
−1.1
+1.8
SAC- λ -R
−2.0
−4.1
+0.1
−2.2
−3.6
−0.6
SAC-GAE
−2.6
−8.2
+3.1
−3.2
−6.9
+0.4
SAC-GAE-R
−0.8
−7.3
+5.2
−0.9
−5.5
+3.5
Appendix
Table 1: Effect of λ on performance. Slopes of the task-averaged normalized score of a configuration (in percentage points) per +0.1 in λ , from weighted least squares. The marginal slope regresses on λ alone; the partial slope is the λ coefficient in a regression on all hyperparameters sampled for the method, with learning rates, sizes and counts on a log scale. For the GAE critics, λ is the GAE coefficient, sampled on [0.85,0.99] ; the λ -return critics sample λ∈[0.50,0.95] . Lower and upper bounds are 95% bootstrap confidence intervals; bold marks intervals that exclude zero.
Figure 10: Per-task performance profiles : share of configurations and seeds reaching normalized score τ on each task (Eq. 15 with M=1 ).
Hmin
Hmax
S
Training runs
Method
Brax
CC
MinAtar
Brax
CC
MinAtar
Brax
CC
MinAtar
Brax
CC
MinAtar
PPO
64
64
64
64
64
64
3
5
3
1152
1280
768
PQN
–
64
64
–
64
64
–
5
3
–
960
768
SAC
64
58
64
64
65
64
3
5
3
1152
1255
768
SAC-A
64
64
64
64
64
64
3
5
3
1152
1280
768
SAC-DA
64
64
64
64
64
64
3
5
3
1152
1280
768
Appendix
Table 2: Configurations per task ( H ; minimum and maximum across the tasks of a suite), training seeds per configuration ( S ) and resulting training runs per suite.
Hyperparameter
Shared by all methods
Discount factor γ
[0.95,0.995]
Actor learning rate
[5×10−5,3×10−3] , log
Critic learning rate
[5×10−5,3×10−3] , log
Temperature learning rate
[5×10−5,3×10−3] , log
One-step replay critics (SAC, SAC-A, SAC-DA)
Appendix
Table 3: Sampled hyperparameter ranges. Continuous ranges are given as intervals, discrete ones as sets; “log” denotes log-uniform sampling.
95th percentile
Maximum
Task
Floor
value
lower
upper
value
lower
upper
Discrete
CartPole
0
500
500
500
500
500
500
Acrobot
−500
−78.0
−78.5
−77.5
−65.6
−69.4
−65.6
MountainCar
−200
−99.3
−100.2
−98.9
−95.2
−95.9
−95.2
MinAsterix
0
25.3
22.9
28.1
57.4
45.3
57.4
Appendix
Table 4: Reference returns used for normalization. The floor is the minimum achievable return and maps to 0 ; the 95 th percentile of per-run mean returns, pooled over all algorithms, maps to 1 . The pooled maximum is given for reference. Lower and upper bounds are 95% bootstrap confidence intervals.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.