Efficient Soft Actor-Critic with LLM-Based Action-Level Guidance for Continuous Control

Hao Ma; Zhiqiang Pu; Xiaolin Ai; Huimu Wang

cs.LGMar 18, 2026

Efficient Soft Actor-Critic with LLM-Based Action-Level Guidance for Continuous Control

Authors: Hao Ma, Zhiqiang Pu, Xiaolin Ai, Huimu Wang

Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · JD.com, Beijing, China

Abstract

We present GuidedSAC, a novel reinforcement learning (RL) algorithm that facilitates efficient exploration in vast state-action spaces. GuidedSAC leverages large language models (LLMs) as intelligent supervisors that provide action-level guidance for the Soft Actor-Critic (SAC) algorithm. The LLM-based supervisor analyzes the most recent trajectory using state information and visual replays, offering action-level interventions that enable targeted exploration. Furthermore, we provide a theoretical analysis of GuidedSAC, proving that it preserves the convergence guarantees of SAC while improving convergence speed. Through experiments in both discrete and continuous control environments, including toy text tasks and complex MuJoCo benchmarks, we demonstrate that GuidedSAC consistently outperforms standard SAC and state-of-the-art exploration-enhanced variants (e.g., RND, ICM, and E3B) in terms of sample efficiency and final performance.

Figures & tables

Figure 1: The framework of GuidedSAC. GuidedSAC leverages an LLM-based supervisor to analyze the last trajectory and determine whether intervention is necessary. If intervention is triggered, a residual action $\Delta a$ is added to the original action $a$ , resulting in the intervened action $\widetilde{a}$ . This intervened action is then stored in the replay buffer, facilitating the discovery of high-value trajectories.

Figure 2: Illustration of the LLM-based supervisor’s cooperation details.

Panel of Figure 3 — Figure 3: Performance comparison on toy text. Training curves comparing GuidedSAC with RND, E3B, and ICM across four toy text environments. The shaded regions represent the predefined intervals where intervention can occur.

Panel of Figure 4 — Figure 4: Training curves for MountainCar and Humanoid. Shaded regions indicate intervention periods for GuidedSAC. For MountainCar the intervention occurs between steps 50k and 53k while for Humanoid it occurs between steps 700k and 800k. MountainCar reward of 100 indicates successful goal achievement. Humanoid values above 5000 represent robust bipedal locomotion.

Figure 5: Policy landscape evolution in MountainCar. Columns show snapshots at 0k, 50k intervention start, 53k intervention end, 60k, 80k, and 100k training steps. Horizontal axis shows car position with $x\in[-1.3,0.7]$ . Vertical axis shows velocity with $v\in[-0.07,0.07]$ . Color indicates most probable action $\arg\max_{a}\pi(a|s)$ . Dotted lines intersect at initial state with $x=-0.5$ and $v=0$ . The intervention window causes immediate policy reconfiguration, demonstrating efficient knowledge transfer from $\pi_{\text{interv}}$ to $\pi_{\phi}$ .

Panel of Figure 6 — Figure 6: Visualization of final policies in Humanoid task. Left: GuidedSAC produces coordinated bipedal running with sinusoidal leg motion patterns. Right: SAC converges to an unnatural gait—supporting on one leg while performing rapid small-step hops. Both achieve reward, but GuidedSAC’s locomotion aligns more closely with natural human movement.

Panel of Figure 7 — Figure 7: Impact of intervention timing and duration. Left side shows intervention start point $s$ tested at 5k, 10k, 20k, and 40k steps. Right side shows intervention duration $l$ tested at 1k, 3k, 5k, and 10k steps. Optimal performance requires early intervention with moderate duration. If duration is too short it dilutes impact, while too long hinders autonomous exploration.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	Guided Window	End Step	Batch Size	$\gamma$	$\tau$	$\alpha$	Start
Blackjack	2000	5000	256	0.5	0.005	auto	1000
CliffWalking	2000	10000	1024	0.99	0.005	0.01	0
FrozenLake	100	5000	64	0.99	0.005	0.01	0
Taxi	2000	5000	256	0.99	0.005	0.01	0
MountainCar	1000	3000	256	0.99	0.005	auto	0
Humanoid	1000	900,000	256	0.99	0.005	auto	100

Appendix

Table 1: Hyperparameter settings for GuidedSAC across different environments.

Figure 8: Prompts for the advisor and coder LLMs.

LLM-Guided RL

Large language model-guided reinforcement learning uses language models to shape rewards, exploration, planning, and policy learning.

See all 140 papers

Learning Agentic Policy from Action Guidance
May 12, 2026Yuxiang Ji, Zengbin Wang, Yong Wang +6Agentic RL Reinforcement Learning
1Xiamen University · 2AMAP, Alibaba Group · Work done during internship at AMAP, Alibaba Group +1
Memory-Based Advantage Shaping for LLM-Guided Reinforcement Learning
Feb 20, 2026Narjes Nourzad, Carlee Joe-WongMemory-Augmented RL Reinforcement Learning
University of Southern California · Carnegie Mellon University
LaGO: Latent Action Guidance for Online Reinforcement Learning
Jun 23, 2026Kuan-Yen Liu, Ren-Jyun Huang, Ti-Rong WuLatent Action Learning LLM-Guided RL
Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, USA · Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan · Institute of Information Science, Academia Sinica, Taiwan.

May 12, 2026J/K move · Enter open · S save

Learning Agentic Policy from Action Guidance

Yuxiang Ji, Zengbin Wang, Yong Wang, Shidong Yang+5

1Xiamen University · 2AMAP, Alibaba Group · Work done during internship at AMAP, Alibaba Group+1

Agentic RL Reinforcement Learning LLM-Guided RL LLM Agent Training Policy Learning

Open paper AskPDF arXiv

Agentic reinforcement learning (RL) for Large Language Models (LLMs) critically depends on the exploration capability of the base policy, as training signals emerge only within its in-capability region. For tasks where the base policy cannot reach reward states, additional training or external guidance is needed to recover effective learning signals. Rather than relying on costly iterative supervised fine tuning (SFT), we exploit the abundant action data generated in everyday human interactions. We propose ActGuide-RL, which injects action data as plan-style reference guidance, enabling the agentic policy to overcome reachability barriers to reward states. Guided and unguided rollouts are then jointly optimized via mixed-policy training, internalizing the exploration gains back into the unguided policy. Motivated by a theoretical and empirical analysis of the benefit-risk trade-off, we adopt a minimal intervention principle that invokes guidance only as an adaptive fallback, matching task difficulty while minimizing off-policy risk. On search-agent benchmarks, ActGuide-RL substantially improves over zero RL (+10.7 pp on GAIA and +19 pp on XBench with Qwen3-4B), and performs on par with the SFT+RL pipeline without any cold start. This suggests a new paradigm for agentic RL that reduces the reliance on heavy SFT data by using scalable action guidance instead.

Abstract

LLM-Guided RL

Explore similar work

Learning Agentic Policy from Action Guidance

Memory-Based Advantage Shaping for LLM-Guided Reinforcement Learning

LaGO: Latent Action Guidance for Online Reinforcement Learning