S3AC: Stabilized, Selective, and Scalable Actor-Critic for LLM Reasoning

Shuo Shen; Guangran Cheng; Chengqi Lyu; Xiangsen Wang; Yanan Sun; Wenwei Zhang

cs.LGOct 26, 2026

S3AC: Stabilized, Selective, and Scalable Actor-Critic for LLM Reasoning

Authors: Shuo Shen, Guangran Cheng, Chengqi Lyu, Xiangsen Wang, Yanan Sun, Wenwei Zhang

Organizations: Shanghai AI Laboratory · Independent Researcher

Abstract

Critic-free methods are widely used in reinforcement learning with verifiable rewards (RLVR) for long chain-of-thought (CoT) reasoning, avoiding difficult and costly critic training. Common alternatives lack token-level credit assignment and typically rely on grouped rollouts to estimate prompt-specific baselines. A critic can address both limitations, but long-CoT critic training is hindered by insufficient coverage of trajectories with diverse lengths and early gradients dominated by the newly initialized value head. We introduce S3AC (Stabilized, Selective, and Scalable Actor-Critic) for LLM reasoning, built around a practical critic-training recipe. The trained critic provides token-specific advantages and prompt-specific baselines. Using the prompt-level baseline to estimate difficulty, we propose Difficulty-Aware Prompt Sampling (DAPS). DAPS combines this estimate with outcome history to select prompts before generation, reducing rollout waste without offline difficulty initialization. S3AC improves group-rollout performance across dense and mixture-of-experts models from 1.5B to 397B parameters, outperforming a modern GRPO variant (GRPO-X). At 35B, it sustains single-rollout training while tested critic-free baselines deteriorate after initial gains.

Figures & tables

Figure 1: Overview of S3AC for long-CoT RLVR. (a) Critic value pretraining and online updates (§ 3.3 ). (b) DAPS selects prompts using critic values and outcome history (§ 3.4 ). (c) S3AC uses token-level GAE advantages; critic-free methods share one advantage across the response (§ 3.2 ).

Figure 2: Prompt-conditioned values and advantages (35B single rollout). (a) Initial value versus held-out return. (b) Long failures (

>20

K tokens), using length-adaptive GAE with

\alpha=1.5

Figure 3: Critic values on Intern-S2-Preview-35B rise as the response identifies candidate values (A) and determines the requested term, $4/5$ (B). Expanded examples are in Appendix C.2 .

Table 4

Figure 4: Training dynamics on Intern-S2-Preview-35B: (a) seven-benchmark average, (b) rollout entropy, and (c) response length. S3AC uses DAPS.

Figure 5: Coverage pretraining improves value prediction and learning progress on Intern-S2-Preview-35B, reaching higher final performance within the same actor-update budget (group rollout).

		Value loss		Grad	Clip
Config	Grad. steps	first 20	last 20	first 20	(%)
Head init. std. (coverage data, clip 10)
$1/(H{+}1)$	342	0.45	0.21	4.3	0.3
$0$	342	0.47	0.22	5.8	2.0
$0.1/(H{+}1)$	342	0.47	0.23	5.2	1.8
$0.01$	342	0.72	0.24	17.5	26.3

Table 4: Value-pretraining ablations on Intern-S2-Preview-35B. Loss: mean training loss over the first/last 20 critic gradient steps. Clip: percentage of gradient steps exceeding the clip threshold.

Setting	Prompts $\times K$	Grad. step budget	Uniform	+DAPS
35B group	$512\!\times\!8$	1,520	79.04	80.49
35B single	$4096\!\times\!1$	1,520	74.40	76.94
R1D group	$256\!\times\!16$	1,024	30.87	31.08
R1D single	$4096\!\times\!1$	1,024	30.16	30.36

Table 5: DAPS with group and single rollout: best checkpoint scores within the listed gradient-step budgets. Curves are in Appendix D .

Figure 6: DAPS sampling on Intern-S2-Preview-35B. (a) Trajectories discarded per rollout batch by the group filter. (b) Single-rollout training share on prompts with

0.2<p_{\mathrm{ref}}\leq 0.8

		Training (80 gradient steps)					Evaluation
Method	Rollout	Grad. steps	Entropy	Length	Reward	EV	Length	Trunc. (%)
Qwen2.5-Math-1.5B (eval: gradient step 1,024, $\leq$ 4,096 tokens)
GRPO-X	group	952–1,024	0.15	993	-0.32	–	921	3.1
S3AC	group	952–1,024	0.07	965	-0.06	0.35	912	3.7
REINFORCE++	single	952–1,024	1.50	1535	-0.40	–	1723	28.9
FlashREINFORCE	single	952–1,024	0.17	1540	-0.35	–	1951	41.1

Table 7: Training and evaluation statistics near the end of each run. Training metrics average 10 batch-level records over 80 actor gradient steps. A dash indicates an unavailable statistic. † REINFORCE++ uses a post-collapse window.

Policy loss	AIME24	AIME25	GPQA-D	IMO- AnswerBench	MMMU-Pro	MathVision	MathVista	Avg
CISPO + IcePoP + seq. mask	92.0	89.9	74.8	66.2	75.3	75.3	87.9	80.2
DIS	91.3	89.2	75.1	64.8	78.5	74.0	88.2	80.1
DIS + seq. mask	91.2	88.7	73.7	63.4	78.5	74.5	88.3	79.8

Table 8: Policy-loss variants for single-rollout Intern-S2-Preview-35B; all settings other than the policy loss are identical. Top: benchmark scores averaged over gradient steps 1,360, 1,440, 1,520. Bottom: means over the 10 logged points at gradient steps 1,448–1,520 of rollout entropy, response length (K tokens), actor-target EV, and sequence mask rate (Mask).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Response-length distribution (a) and accuracy by length (b) for the initial policy and the policy at gradient step 1,600 on Intern-S2-Preview-35B. Bins span 2K tokens.

Setting	Prompts $\times K$	$a_{\min}$
35B group	$512\times 8$	0.3
35B single	$4096\times 1$	0.1
R1D group	$256\times 16$	0.3
R1D single	$4096\times 1$	0.3

Appendix

Table 9: Prompt batch sizes and minimum acceptance probabilities for the four DAPS comparisons.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training
Sep 29, 2026Hongyang Li, Xiao Li, Caesar Wu +3Reinforcement Learning Value Function Estimation
University of Luxembourg · Seafill Open-Source Community · Université Paris-Saclay
Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning
Apr 8, 2026Rohan Surana, Gagan Mundada, Xunyi Jiang +19RL for Language Model Reasoning RL for Language Models
University of California, San Diego · University of Toronto · UIUC +3
Trust the Critic More
Sep 30, 2026Kaiyue Wen, Luke Bailey, Arvind Mahankali +1RL for Language Model Reasoning Credit Assignment in RL
Stanford University

Sep 29, 2026J/K move · Enter open · S save

Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training

Hongyang Li, Xiao Li, Caesar Wu, Said Mammar+2

University of Luxembourg · Seafill Open-Source Community · Université Paris-Saclay

Reinforcement Learning Value Function Estimation RL for Language Model Reasoning RL Post-Training Policy Learning

Open paper AskPDF arXiv

Recent approaches to reinforcement learning (RL) post-training for large language models increasingly remove the critic to reduce training instability and memory overhead. Even where a critic is trained, it is discarded once training ends, although it has learned to predict outcomes. We revisit this trend and show that a pretrained critic's ability to predict future outcomes can make it a valuable asset for efficient long-horizon reasoning. First, we find that instability in critic-based RL for long chain-of-thought reasoning is largely an optimization artifact: keeping policy updates small and low in variance restores stable convergence. Second, a well-pretrained critic estimates the posterior probability of eventual success from later trajectory states and unfinished prefixes. Its predictions provide outcome-derived, dense, per-prefix learning signals that, during policy optimization, require neither completed rollouts, step-level annotations, nor external reward labels. Building on this insight, we introduce Reward-Free Policy Optimization (RFPO), which repurposes a single calibrated, frozen critic as a rollout-level reward, a value baseline for generalized advantage estimation, and a success forecaster for unfinished prefixes. We further show that binarizing the debiased score stops the policy from exploiting the critic's length bias. Binarized, RFPO matches supervised PPO without a single label in the training loop, while cutting compute and memory overhead. This makes RFPO well suited to long-horizon reasoning tasks, where outcomes arrive late and generation dominates cost: because rollouts can be rewarded before they finish, training no longer has to pay for waiting on every trajectory to complete. Our findings challenge the prevailing critic-free paradigm and establish critic-based, reward-free optimization as a scalable and computationally efficient path for LLM post-training.

Abstract

RL for Language Model Reasoning

Explore similar work

Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training

Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning

Trust the Critic More