Soft Actor-Critic

Also known as SAC

Overview

Soft Actor-Critic (SAC) learns continuous actions by combining reward with an entropy bonus, encouraging exploration, while replay buffers and twin critics support sample-efficient off-policy training. Research builds on this core with distributional critics, ensembles, hierarchical policies, safety constraints, and guidance to improve value estimates, exploration, and long-horizon planning. SAC is studied in settings including robotic locomotion and manipulation, building control, and autonomous driving, with sim-to-real transfer an important goal. A central tension is improving data efficiency and scaling to complex tasks without unstable or overestimated critics, while choosing enough exploration to learn useful behavior.

Latest papers

31 papers2 papers in the last four weeks, down 60% on the four weeks before. 0.0% of all new papers.

Jul 21, 2024

Enhancing Hardware Fault Tolerance in Machines with Reinforcement Learning Policy Gradient Algorithms

Industry is moving toward autonomous, network-connected machines that detect and adapt to changing conditions, including hardware faults. Conventional fault-tolerant design duplicates hardware and reroutes control logic; reinforcement learning (RL) offers a learning-based alternative. This paper presents the first systematic comparison of two RL algorithms -- Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) -- for integrating fault tolerance into control. Beyond algorithm choice, we investigate four knowledge-transfer strategies: retaining or discarding model parameters, and retaining or discarding storage contents. Performance is evaluated in two Gymnasium environments: Ant-v5 and FetchReachDense-v3. Results show rapid, fault-specific recovery with clear trade-offs. In Ant-v5, retaining PPO's parameters boosts early returns and remains the safest choice across all faults, while retaining SAC's parameters yields mixed outcomes. SAC's early performance further depends on whether the replay buffer is retained: beneficial when prior experiences match current dynamics, but harmful when they diverge. In FetchReachDense-v3, discarding both PPO's and SAC's parameters was most effective under sensor corruption. Across tasks, both algorithms recover near-normal performance within minutes in low-dimensional settings and within days in high-dimensional settings, highlighting a clear trade-off between adaptation speed and asymptotic performance. These findings demonstrate that RL can deliver robust fault tolerance and offer practical guidelines.