Exploration via Activation-Space Steering for Reinforcement Learning with Verifiable Rewards

Yunan Zeng; Jinhao Jing; Xi Cheng; Yuwang Wang

cs.LGOct 19, 2026

Exploration via Activation-Space Steering for Reinforcement Learning with Verifiable Rewards

Authors: Yunan Zeng, Jinhao Jing, Xi Cheng, Yuwang Wang

Organizations: Fudan University · The Chinese University of Hong Kong, Shenzhen · Kunlun Inc. · Tsinghua University

Abstract

Reinforcement learning with verifiable rewards (RLVR) trains a language model by sampling responses, scoring them with an automatic verifier, and updating the policy from the resulting rewards. The policy therefore receives direct learning signals only from trajectories that appear in its rollouts. Yet repeated sampling from the same policy can concentrate on similar reasoning paths, leaving useful alternatives unexplored under a fixed rollout budget. We introduce Activation-Space Exploration (ASE), an online method for diversifying training rollouts. By clustering activation differences between correct and incorrect responses to the same problem, ASE builds a structured bank of latent exploration vectors. Different vectors are then applied to subsequent rollouts to explore alternative reasoning paths. On Qwen3-1.7B-Base and Qwen3-8B-Base trained on DAPO-Math-17k, ASE improves average Mean@8 across six mathematical reasoning benchmarks over GRPO by 4.87 and 1.87 percentage points, respectively, under the same rollout budget.

Figures & tables

Figure 1: Motivation and high-level overview of activation-space exploration. Standard GRPO samples all responses from one proposal distribution, whereas ASE uses outcome feedback to maintain multiple activation-space directions, routes responses through different proposals, and updates the actor and vector bank online.

Figure 2: ASE pipeline. We replay successful and unsuccessful responses to read out hidden activations, cluster their pairwise differences to form exploration directions, match and smooth the directions across updates, and inject one selected direction into the residual stream for each routed response.

	Mathematical benchmarks							OOD benchmarks
Method	MATH500	Minerva	AIME24	AIME25	AMC24	Olympiad	Math Avg.	MMLU-Pro	GPQA-D	BBH	OOD Avg.
GRPO	88.17 / 94.87	43.98 / 55.76	26.39 / 48.89	21.39 / 37.78	58.06 / 82.22	55.80 / 70.18	48.96 / 64.95	67.62 / 85.64	49.45 / 79.80	76.73 / 91.19	64.60 / 85.54
DAPO	86.09 / 93.20	42.56 / 54.17	21.53 / 41.11	18.06 / 30.00	48.98 / 74.07	51.38 / 66.82	44.77 / 59.89	61.28 / 83.38	46.70 / 80.81	76.71 / 91.32	61.56 / 85.17
Dr.GRPO	87.83 / 95.87	43.20 / 59.44	25.42 / 47.78	20.69 / 37.78	55.56 / 77.78	57.78 / 74.78	48.41 / 65.57	65.20 / 86.00	48.34 / 84.85	77.79 / 91.99	63.78 / 87.61
GSPO	88.05 / 95.00	43.01 / 55.51	25.00 / 53.33	22.50 / 36.67	58.33 / 80.00	56.71 / 72.40	48.94 / 65.49	64.94 / 84.08	47.47 / 80.81	75.74 / 91.20	62.72 / 85.36
RePO	86.09 / 94.53	41.57 / 54.29	22.64 / 42.22	18.61 / 32.22	53.06 / 81.48	53.34 / 71.36	45.89 / 62.69	61.21 / 84.40	43.96 / 81.31	67.92 / 91.35	57.69 / 85.69

Table 1: Main results for Qwen3-8B-Base trained on DAPO-Math-17k. Scores are reported as Mean@8 / Pass@8 (in percent). Reported scores are averaged over three runs. Each

\Delta

row reports the absolute difference from the corresponding baseline in percentage points.

	Mathematical benchmarks							OOD benchmarks
Method	MATH500	Minerva	AIME24	AIME25	AMC24	Olympiad	Math Avg.	MMLU-Pro	GPQA-D	BBH	OOD Avg.
GRPO	65.59 / 83.47	27.37 / 43.50	6.39 / 14.44	5.42 / 22.22	18.24 / 40.74	29.82 / 50.59	25.47 / 42.50	35.46 / 69.09	26.45 / 79.12	44.70 / 84.15	35.54 / 77.45
DAPO	68.02 / 84.20	27.94 / 45.22	10.00 / 26.67	5.00 / 18.89	21.20 / 43.70	30.87 / 50.49	27.17 / 44.86	39.22 / 69.27	29.12 / 76.60	50.43 / 85.28	39.59 / 77.05
Dr.GRPO	68.16 / 84.53	30.39 / 45.59	12.08 / 23.33	2.92 / 12.22	21.39 / 40.74	31.43 / 51.68	27.73 / 43.02	40.54 / 70.33	30.30 / 77.61	51.99 / 85.43	40.94 / 77.79
GSPO	68.42 / 84.80	28.71 / 44.36	9.58 / 22.22	4.86 / 18.89	18.24 / 39.26	32.00 / 53.81	26.97 / 43.89	38.38 / 70.15	27.90 / 78.96	47.02 / 84.75	37.77 / 77.95
RePO	65.42 / 84.67	26.19 / 44.61	7.08 / 22.22	4.31 / 15.56	17.78 / 42.22	29.60 / 51.88	25.06 / 43.53	36.15 / 69.45	26.54 / 76.26	44.36 / 84.96	35.68 / 76.89

Table 2: Main results for Qwen3-1.7B-Base trained on DAPO-Math-17k. Scores are reported as Mean@8 / Pass@8 (in percent). Reported scores are averaged over three runs. Each

\Delta

row reports the absolute difference from the corresponding baseline in percentage points.

Table 6

Method	MATH500	Minerva	AIME24	AIME25	AMC24	Olympiad	Math Avg.	MMLU-Pro	GPQA-D	BBH	OOD Avg.
	Mathematical benchmarks							OOD benchmarks
Dr.GRPO	68.27 / 84.60	31.25 / 47.06	12.08 / 20.00	2.92 / 16.67	20.83 / 40.00	31.36 / 52.67	27.79 / 43.50	40.36 / 70.62	29.86 / 77.27	52.03 / 85.61	40.75 / 77.83
ASE-Dr.GRPO	69.40 / 86.20	30.06 / 47.43	11.25 / 26.67	5.83 / 16.67	24.17 / 44.44	33.70 / 52.67	29.07 / 45.68	40.45 / 69.23	31.06 / 77.78	51.81 / 84.95	41.11 / 77.32
$\Delta$ vs. Dr.GRPO	+1.13 / +1.60	-1.19 / +0.37	-0.83 / +6.67	+2.91 / +0.00	+3.34 / +4.44	+2.34 / +0.00	+1.28 / +2.18	+0.09 / -1.39	+1.20 / +0.51	-0.22 / -0.66	+0.36 / -0.51
GSPO	69.00 / 86.20	28.72 / 43.75	8.75 / 23.33	5.00 / 23.33	19.17 / 42.22	32.64 / 54.30	27.21 / 45.52	38.15 / 70.48	26.96 / 80.81	46.91 / 84.55	37.34 / 78.61
ASE-GSPO	69.80 / 87.00	30.19 / 47.43	10.00 / 23.33	6.25 / 13.33	23.33 / 48.89	32.60 / 54.75	28.70 / 45.79	40.48 / 72.52	28.98 / 78.79	48.80 / 85.53	39.42 / 78.95

Table 5: ASE-Dr.GRPO and ASE-GSPO on Qwen3-1.7B-Base. Entries are Mean@8 / Pass@8 (%); Math Avg. and OOD Avg. cover six mathematical and three OOD benchmarks, respectively.

Model	Method	Step time (s/step) $\downarrow$	Training throughput (k tokens/s/GPU) $\uparrow$	Rollout latency ( $\mu$ s/output token) $\downarrow$	Actor-update time ( $\mu$ s/valid token) $\downarrow$
1.7B	GRPO	92.21	2.845	52.91	29.15
1.7B	ASE-GRPO	93.05	2.666	54.35	33.60
1.7B	$\Delta$ vs. GRPO	+0.91%	$-$ 6.29%	+2.72%	+15.27%
8B	GRPO	136.10	1.041	60.08	49.35
8B	ASE-GRPO	145.88	0.974	61.59	56.08
8B	$\Delta$ vs. GRPO	+7.19%	$-$ 6.44%	+2.51%	+13.64%

Table 6: Training-time overhead on matched GRPO and ASE-GRPO runs. Throughput uses all prompt and response tokens, rollout latency is normalized by output tokens, and actor-update latency is normalized by all valid tokens.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	Qwen3-1.7B-Base	Qwen3-8B-Base
Training data	DAPO-Math-17k	DAPO-Math-17k
Training epochs	4	4
Train/actor mini-batch	64/64	64/64
Rollouts per problem	8	8
Max prompt/response length	1024/4096	1024/4096
Optimizer	AdamW	AdamW

Appendix

Table 7: Common settings for the two model sizes.

Method	PPO clipping	Loss aggregation / policy loss	Extra processing
GRPO	(0.2, 0.2)	token-mean / vanilla GRPO	none
DAPO	(0.2, 0.28)	token-mean / vanilla GRPO	dynamic sampling; overlong buffer
Dr.GRPO	(0.2, 0.2)	seq-mean-token-sum-norm / vanilla GRPO	response normalization
GSPO	( $3\times 10^{-4}$ , $4\times 10^{-4}$ )	seq-mean-token-mean / GSPO	sequence-level loss
RePO	(0.2, 0.2)	seq-mean-token-mean / vanilla GRPO	replay cache 16; samples 8
ASE-GRPO	(0.2, 0.2)	token-mean / vanilla GRPO	routed activation steering

Appendix

Table 8: Method-specific configurations.

Configuration	Value
Intervention layer	Transformer block 0
Vector construction	CAA-style clustered positive-minus-negative differences
Number of routes $K$	3
Route allocation	balanced uniform
Clean response bank	enabled
Readout position function $p$	response token at 25% of response length

Appendix

Table 9: ASE-GRPO configuration.

$K$	Math Avg.	OOD Avg.
2	29.60/45.74	40.08/78.44
3	29.64/47.02	39.62/76.52
4	27.01/44.63	39.40/78.24
5	28.04/46.38	38.82/78.26
6	28.04/44.54	39.54/77.72

Appendix

Table 10: Sensitivity to the number of clusters

K

. For this ASE sweep, we fix

\alpha=0.4

and

\beta=0.5

$\beta$	Math Avg.	OOD Avg.
0.0	26.48/42.63	35.84/72.25
0.1	28.29/45.41	38.02/72.10
0.3	26.27/46.10	35.33/76.78
0.5	30.21/46.44	40.79/79.71
0.7	29.41/48.83	37.14/79.14
0.9	29.35/47.81	39.89/78.35

Appendix

Table 11: Sensitivity to the EMA coefficient

\beta

. For this ASE sweep, we fix

K=3

and

\alpha=0.2

$\alpha$	Math Avg.	OOD Avg.
0.2	30.21/46.44	40.79/79.71
0.3	28.38/48.05	38.75/77.62
0.4	29.64/47.02	39.62/76.52
0.5	30.31/48.67	41.35/79.97
0.6	29.21/44.63	39.43/76.84

Appendix

Table 12: Sensitivity to the steering strength

\alpha

. For this ASE sweep, we fix

K=3

and

\beta=0.5

Readout fraction $f$	Math Avg.	OOD Avg.
0.25	28.93/46.79	38.24/77.59
0.50	27.89/43.94	39.68/76.98
0.75	27.59/44.95	40.00/78.36

Appendix

Table 13: Sensitivity to the response-relative activation readout position. We fix

K=3

\alpha=0.2

, and

\beta=0.7

; scores are Mean@8 / Pass@8 (%). Math Avg. and OOD Avg. use the same benchmark groups as the main tables.

Method	Math Avg.
GRPO	26.00/43.91
Unstructured random steering	26.65/41.59
ASE-GRPO	30.31/48.67

Appendix

Table 14: Six-benchmark Math Avg. (Mean@8 / Pass@8, %). GRPO and ASE-GRPO use the first evaluation at their selected checkpoints.

Quantity	Values or definition
Models	Base; standard GRPO; ASE-GRPO
Number of routes $K$	$1,2,3,4,5,6,7$
Capture fraction $f$	$0.25,0.50,0.75$
Strength $\alpha$	$0.25,0.50,0.75,1.00$
Conditions per model	$7\times 3\times 4=84$
Repeats	Three independent evaluations per cell

Appendix

Table 15: Inference-time sweep grid and reporting units.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,087 papers

LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models
Jul 30, 2026Shuang Liang, Haoyang Zhou, Yifan Gong +2Numerical Reasoning in Language Models Prompt Selection for RL
Renmin University of China · AntGroup
ExTra: Exploratory Trajectory Optimization for Language Model Reinforcement Learning
Jun 23, 2026Wenyang Hu, Junxiang Jia, Zhen Shu +3RL for Language Models RL for Language Model Reasoning
1National University of Singapore · 2SAP
Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR
May 15, 2026Chanuk Lee, Sangwoo Park, Minki Kang +1RL for Language Model Reasoning RL Exploration
KAIST

Jul 30, 2026J/K move · Enter open · S save

LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models

Shuang Liang, Haoyang Zhou, Yifan Gong, Guowei Wang+1

Renmin University of China · AntGroup

Numerical Reasoning in Language Models Prompt Selection for RL Reinforcement Learning with Verifiable Rewards Sample-Efficient RL

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models, but prompt groups with identical rollout rewards consume generation budget without effective learning signals. Pre-rollout prompt selection can reduce this waste by screening prompts before rollout generation. However, existing pre-rollout methods struggle to balance exploitation and exploration: repeatedly exploiting historically informative prompts can narrow training coverage, whereas broader exploration can lower the fraction of informative prompts. To address these limitations, we introduce LEEPS, a Latent-Guided Explore--Exploit Prompt Sampler that adaptively balances the reuse of previously observed informative prompts with continued exploration of uncertain ones. LEEPS partitions candidates into exploit and explore portfolios and adaptively allocates rollout budget according to their recent non-trivial ratios. It further uses representation-space neighbors and historical rollout outcomes to prioritize uncertain prompts likely to yield non-zero reward variance, thereby making exploration more targeted without additional rollouts. Across six mathematical reasoning benchmarks, LEEPS achieves the highest average score at both model scales, with relative gains of 2.6% and 3.7% over the strongest baseline for Qwen2.5-Math-1.5B and 7B, respectively, and generally improves faster during the training process. It also achieves the highest average score across the three evaluated OOD general-reasoning benchmarks at both model scales and adds only about 2 seconds of online sampling overhead per training step. Code is available at https://github.com/ShuangLiangX/LEEPS.

Abstract

RL for Language Model Reasoning

Explore similar work

LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models

ExTra: Exploratory Trajectory Optimization for Language Model Reinforcement Learning

Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR