Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target

Du Cheng; Junyu Lu; Jingfan Zhang; Yuzhuo Zhang; Haoxuan Jia; Shichao Weng

cs.LGOct 9, 2026

Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target

Authors: Du Cheng, Junyu Lu, Jingfan Zhang, Yuzhuo Zhang, Haoxuan Jia, Shichao Weng

Abstract

Reasoning post-training draws on offline demonstrations and policy-generated rollouts, yet conventional supervised fine-tuning (SFT) followed by reinforcement learning (RL) separates their objectives and limits continued reuse of collected responses. We propose Cross-Behavior Policy Optimization (CBPO), which makes these sources compatible with one regression loss through a fixed-reference target. Because the generating policy does not enter this target, positive and negative offline completions, reference responses, and current and historical rollouts can share a common reward-labeled record format and replay buffer. As new responses enter the buffer, optimization retains the same regression objective without coordinating separate SFT and RL objectives or requiring stored behavior-policy probabilities, importance weighting, or a learned critic. Restricting the available sources gives offline, on-policy, and off-policy variants, while CBPO-Unified combines them. Across two model scales and four mathematical reasoning benchmarks, CBPO delivers leading avg@32 and pass@32 performance across benchmarks and data regimes. On Qwen3-1.7B, CBPO-Unified improves benchmark-averaged avg@32 and pass@32 over SFT DAPO by 2.33 and 4.44 percentage points, respectively. In the replay ablation, retaining historical rollouts improves these metrics by 1.87 and 3.72 percentage points over the no-history configuration.

Figures & tables

Figure 1: CBPO-Unified/SFT $\rightarrow$ DAPO: $L/D=1.0205/1.2207$ , with total turn $=58.20^{\circ}/368.75^{\circ}$ .

Figure 2: SFT

\rightarrow

RLVR vs. shared-target CBPO-Unified on offline and on-/off-policy data.

Regime	Methods	AIME25		MinervaMath		AMC23		MATH500		Avg.
Regime	Methods	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32
Offline	SFT	14.23 ± 0.06	29.33 ± 1.49	19.29 ± 0.02	33.53 ± 0.16	48.77 ± 0.03	75.50 ± 1.12	64.99 ± 0.01	77.96 ± 0.09	36.82 ± 0.02	54.08 ± 0.11
	DPO	14.50 ± 0.05	29.33 ± 1.49	19.32 ± 0.01	33.68 ± 0.20	48.97 ± 0.03	74.50 ± 1.12	65.34 ± 0.01	78.16 ± 0.09	37.03 ± 0.02	53.92 ± 0.41
	CBPO-Offline	15.81 ± 0.05	29.33 ± 1.49	20.29 ± 0.01	35.22 ± 0.16	50.11 ± 0.04	78.00 ± 1.12	66.99 ± 0.01	80.44 ± 0.09	38.30 ± 0.02	55.75 ± 0.54
On-policy	PPO	17.48 ± 0.05	32.67 ± 1.49	21.35 ± 0.01	38.82 ± 0.20	56.47 ± 0.03	84.50 ± 1.12	69.71 ± 0.01	83.68 ± 0.11	41.25 ± 0.02	59.92 ± 0.42
	DAPO	17.27 ± 0.05	38.00 ± 1.83	21.58 ± 0.01	39.78 ± 0.16	56.23 ± 0.03	85.50 ± 1.12	70.38 ± 0.01	84.64 ± 0.09	41.37 ± 0.01	61.98 ± 0.67

Table 1: Qwen3-1.7B performance across data regimes and joint SFT/RL over five training seeds.

Regime	Methods	AIME25		MinervaMath		AMC23		MATH500		Avg.
Regime	Methods	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32
Offline	SFT	33.17 ± 0.06	38.00 ± 1.83	45.20 ± 0.01	52.87 ± 0.16	83.16 ± 0.04	87.00 ± 1.12	84.61 ± 0.02	88.56 ± 0.09	61.53 ± 0.02	66.61 ± 0.65
	DPO	33.52 ± 0.05	37.33 ± 1.49	45.60 ± 0.01	53.38 ± 0.16	83.27 ± 0.03	90.50 ± 1.12	85.11 ± 0.01	89.76 ± 0.09	61.87 ± 0.02	67.74 ± 0.38
	CBPO-Offline	34.04 ± 0.05	38.00 ± 1.83	47.20 ± 0.01	55.29 ± 0.20	84.13 ± 0.03	89.50 ± 1.12	85.61 ± 0.02	90.64 ± 0.09	62.74 ± 0.02	68.36 ± 0.39
On-policy	PPO	41.27 ± 0.05	47.33 ± 1.49	50.31 ± 0.01	58.68 ± 0.20	89.72 ± 0.04	95.50 ± 1.12	89.40 ± 0.01	96.44 ± 0.09	67.67 ± 0.01	74.49 ± 0.43
	DAPO	39.40 ± 0.05	42.67 ± 1.49	49.08 ± 0.01	58.16 ± 0.16	88.53 ± 0.03	97.00 ± 1.12	89.60 ± 0.01	96.76 ± 0.09	66.65 ± 0.01	73.65 ± 0.66

Table 2: Qwen3-8B performance across data regimes and joint SFT/RL over five training seeds.

Figure 3: Qwen3-1.7B avg@32 / pass@32 trajectories for twelve methods on four benchmarks.

Methods	AIME25		MinervaMath		AMC23		MATH500		Avg.
Methods	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32
SFT $\rightarrow$ DAPO	18.31 ± 0.05	42.67 ± 1.49	21.98 ± 0.01	40.88 ± 0.16	57.28 ± 0.03	87.00 ± 1.12	71.38 ± 0.02	85.52 ± 0.11	42.24 ± 0.02	64.02 ± 0.36
CBPO w/o History (PosCoT)	18.17 ± 0.06	42.67 ± 1.49	22.33 ± 0.02	40.29 ± 0.20	57.33 ± 0.03	84.50 ± 1.12	71.67 ± 0.01	85.88 ± 0.11	42.37 ± 0.02	63.34 ± 0.43
CBPO w/o History	18.77 ± 0.05	42.67 ± 1.49	22.52 ± 0.02	41.25 ± 0.16	57.98 ± 0.03	89.00 ± 1.37	71.54 ± 0.01	86.04 ± 0.09	42.70 ± 0.01	64.74 ± 0.64
CBPO-Unified FIFO-1	18.98 ± 0.05	44.67 ± 1.83	22.85 ± 0.01	41.25 ± 0.16	58.55 ± 0.04	93.50 ± 1.37	71.96 ± 0.01	86.16 ± 0.09	43.08 ± 0.02	66.39 ± 0.61
CBPO-Unified	20.67 ± 0.06	49.33 ± 1.49	24.47 ± 0.01	42.35 ± 0.16	60.17 ± 0.03	94.50 ± 1.12	72.99 ± 0.01	87.64 ± 0.09	44.57 ± 0.02	68.46 ± 0.59

Table 3: Qwen3-1.7B comparison of SFT

\rightarrow

DAPO and CBPO variants over five training seeds.

Methods	AIME25		MinervaMath		AMC23		MATH500		Avg.
Methods	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32
CBPO-Offline	15.83	30.00	20.29	35.29	50.08	77.50	66.99	80.40	38.30	55.80
CBPO w/o History	18.75	43.33	22.52	41.18	57.97	90.00	71.54	86.00	42.70	65.13
CBPO-Unified FIFO-1	18.96	43.33	22.85	41.18	58.52	92.50	71.96	86.20	43.07	65.80
CBPO-Unified FIFO-4	19.27	46.67	23.10	41.18	58.91	92.50	71.91	86.20	43.30	66.64
CBPO-Unified FIFO-16	19.69	46.67	23.30	41.54	59.14	92.50	72.27	86.40	43.60	66.78

Table 4: Qwen3-1.7B with no history, FIFO-

K

, or full replay.

Methods	$\boldsymbol{L}$	$\boldsymbol{D}$	$\boldsymbol{L/D}$	Cumulative turn
2 components: 82.8744%
SFT $\rightarrow$ DAPO	2.1000	1.7203	1.2207	$368.75^{\circ}$
CBPO-Unified	1.7631	1.7278	1.0205	$\boldsymbol{58.20^{\circ}}$
5 components: 91.6378%
SFT $\rightarrow$ DAPO	2.2498	1.8869	1.1923	$295.67^{\circ}$
CBPO-Unified	1.7688	1.7349	1.0196	$\boldsymbol{69.76^{\circ}}$

Table 5: Geometry of the projected optimization trajectories for 2, 5, and 10 components.

$q_{\mathrm{neg}}$	AIME25		MinervaMath		AMC23		MATH500		Avg.
$q_{\mathrm{neg}}$	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32
$0\%$	19.58	46.67	23.56	41.54	59.22	92.50	71.49	86.60	43.46	66.83
$33.3\%$	20.63	50.00	24.47	42.28	60.16	95.00	72.99	87.60	44.56	68.72
$66.7\%$	19.06	43.33	23.17	41.18	59.14	92.50	71.38	86.80	43.19	65.95
$100\%$	18.65	43.33	22.94	40.44	58.13	92.50	70.28	85.60	42.50	65.47

Table 6: CBPO-Unified

q_{\mathrm{neg}}

sensitivity with three trajectories per prompt.

$\boldsymbol{K_{\mathrm{ref}}}$	avg@32	pass@32
8	42.36	66.52
16	44.15	69.09
24	44.56	68.72
32	45.27	70.15

Table 7: Left:

K_{\mathrm{ref}}

sensitivity with fixed training records, update timing,

M

, and

R_{\mathrm{roll}}

. Right: Current-policy sample size with fixed reference statistics and update timing.

$\boldsymbol{K_{\mathrm{ref}}}$	$\boldsymbol{M}$	$\boldsymbol{R_{\mathrm{roll}}}$	avg@32	pass@32
8	5	32	37.23	60.57
20	8	20	40.65	64.49
24	10	16	44.56	68.72
32	20	8	42.26	66.97

Table 8: Budget-coupled

K_{\mathrm{ref}},M,R_{\mathrm{roll}}

sensitivity.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 12 — Figure B.1: Two-message chat prompt with the system instruction and a row-specific math problem.

Problem provenance	User message	Ground truth
Math12k	What is the least positive integer greater than 1 that leaves a remainder of 1 when divided by each of 2, 3, 4, 5, 6, 7, 8 and 9?	2521
Math12k	Find the remainder when $6x^{4}-14x^{3}-4x^{2}+2x-26$ is divided by $2x-6.$	52
AMC/AIME	The workers in a factory produce widgets and whoosits. For each product, production time is constant and identical for all workers, but not necessarily equal for the two products. In one hour, $100$ workers can produce $300$ widgets and $200$ whoosits. In two hours, $60$ workers can produce $240$ widgets and $300$ whoosits. In three hours, $50$ workers can produce $150$ widgets and $m$ whoosits. Find $m$ .	450
AMC/AIME	How many whole numbers between 1 and 1000 do not contain the digit 1? (A) 512, (B) 648, (C) 720, (D) 728, or (E) 800.	728
Olympiads	It is known that the difference between the variable quantities $z$ and $y$ is proportional to the quantity $x$ , and the difference between the quantities $x$ and $z$ is proportional to the quantity $y$ . The coefficient of proportionality is the same and is a positive integer $k$ . A certain value of the quantity $z$ is $5/3$ times the difference of the corresponding values of $x$ and $y$ . Find the numerical value of the coefficient $k$ .	3
Olympiads	Mr. and Mrs. Zeta want to name baby Zeta so that the first letters of the baby’s first, middle, and last names are in alphabetical order and the letters are distinct. The number of possible names is (A) 276, (B) 300, (C) 552, (D) 600, or (E) 15600.	300

Appendix

Table B.1: Representative RLVR-Math-16k prompts with source provenance and verifier targets.

Regime	Methods	Qwen3-1.7B	Qwen3-8B
Offline	SFT	102.23	413.92
	DPO	106.69	430.28
	CBPO-Offline	105.94	421.76
On-policy	PPO	232.84	901.36
	DAPO	118.43	473.72
	GSPO	118.65	475.28

Appendix

Table B.2: Per-run training cost in GPU hours on two NVIDIA B200 GPUs.

Check	Candidate pairs	Candidate test questions	Same-problem pairs	Matched test questions
Exact-string matching	0	0	0	0
Mathematical-expression normalization	1	1	1	1
Approximate-text matching	90	52	6	5
Semantic-similarity retrieval	106	77	13	12
Deduplicated union	151	97	14	13

Appendix

Table B.3: Training–evaluation overlap screening before decontamination. Counts are deduplicated within each row. Checks can retrieve the same pair or question, so their counts are not additive.

Benchmark	Evaluation questions	Same problem	Changed answer request	Shared setting, different question
AIME25	30	0	0	0
MinervaMath	272	0	0	0
AMC23	40	0	0	0
MATH500	500	13	2	8

Appendix

Table B.4: Evaluation questions matched to the unfiltered training prompt pool. Each category counts distinct evaluation questions. A question may occur in more than one category.

Unit	Before filtering	Removed	Retained
Training-question texts	16,268	14	16,254
Evaluation questions	842	0	842

Appendix

Table B.5: Training and evaluation question counts before and after excluding same-problem training questions.

Setting	Value
Method scope	Core Qwen3-1.7B CBPO configuration
Base and reference model	Qwen3-1.7B
Online prompt pool	Decontaminated RLVR-Math-16k, 16,254 prompts
Held-out evaluation	842 prompts across AIME25, MinervaMath, AMC23, and MATH500
Evaluation samples	32 responses per prompt
Target prompt batch size	$P=512$

Appendix

Table C.1: Core Qwen3-1.7B CBPO settings for data, generation, optimization, and hardware.

Source	Methods	AIME25		MinervaMath		AMC23		MATH500		Avg.
Source	Methods	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32
R1-Qwen-1.5B	BRIDGE	15.21	26.67	21.76	40.07	54.30	75.00	70.23	82.00	40.37	55.94
R1-Qwen-1.5B	CBPO-Unified	18.33	33.33	23.43	41.18	58.13	82.50	71.31	85.20	42.80	60.55
R1-Qwen-7B	BRIDGE	17.60	33.33	22.55	40.81	56.41	82.50	71.01	82.60	41.89	59.81
R1-Qwen-7B	CBPO-Unified	19.90	43.33	23.95	41.54	58.98	87.50	72.05	86.20	43.72	64.64
R1-Qwen-32B	BRIDGE	18.33	43.33	23.01	41.18	57.66	85.00	71.31	83.00	42.58	63.13

Appendix

Table D.1: Qwen3-1.7B performance under three individual teacher sources and the union of all three sources on the common retained prompt set. The mixed condition combines the complete teacher-specific datasets for each method.

Correct responses	Prompts	Proportion (%)	Reward 0 (%)	Reward 0.1 (%)
0–1	5,963	36.69	77.30	20.33
2–5	3,157	19.42	63.56	22.98
6–11	937	5.76	52.19	13.06
12–23	3,110	19.13	17.38	1.94
24	3,087	18.99	0.00	0.00

Appendix

Table D.2: Reference-bank composition by reference success count. Reward frequencies are computed over the recorded reference responses in each group.

	Mean $u_{K}$				90th percentile $u_{K}$
Correct reference responses	$K=8$	$K=16$	$K=24$	$K=32$	$K=8$	$K=16$	$K=24$	$K=32$
0–1	3.9088	3.3017	2.9156	2.6821	4.0156	3.4453	3.0912	2.8242
2–5	2.6713	1.7165	1.3281	1.0938	4.0156	2.1142	1.6655	1.3406
6–11	1.6336	0.9610	0.7252	0.5992	2.5843	1.1732	0.8989	0.7624
12–23	0.7080	0.4044	0.3010	0.2564	1.1295	0.6222	0.4872	0.3940
24	0.4572	0.2288	0.1526	0.1282	0.4572	0.2288	0.1526	0.1282

Appendix

Table D.3: Pointwise 95% exact trinomial bounds on absolute log-normalizer estimation error by reference-sample budget

K

, under independent sampling from

\pi_{\mathrm{ref}}

Figure D.1: Mean avg@32 gain over the reference-policy baseline at each reference-sample budget. The five curves correspond to difficulty groups defined by the number of correct responses in the 24-response reference bank.

Negatives	AIME25		MinervaMath		AMC23		MATH500		Avg.
Negatives	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32
0	19.38	46.67	23.23	41.18	58.91	92.50	71.11	86.40	43.16	66.69
1	20.63	50.00	24.47	42.28	60.16	95.00	72.99	87.60	44.56	68.72
2	20.10	46.67	24.21	41.91	59.69	92.50	72.45	87.00	44.11	67.02

Appendix

Table D.4: Negative-count comparison within CBPO-Unified with two fixed positive offline trajectories per prompt. All other data sources and training settings are held fixed.

Negative type	AIME25		MinervaMath		AMC23		MATH500		Avg.
Negative type	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32
No negative	19.38	46.67	23.23	41.18	58.91	92.50	71.11	86.40	43.16	66.69
Reward 0	19.48	46.67	23.10	40.81	59.14	92.50	71.03	86.40	43.19	66.59
Reward 0.1	20.94	50.00	24.76	42.65	60.39	95.00	73.11	88.00	44.80	68.91
Clearly invalid	18.85	40.00	22.87	40.44	58.52	85.00	70.33	85.20	42.64	62.66
Near-correct	21.35	53.33	24.89	43.01	60.86	97.50	73.62	88.60	45.18	70.61

Appendix

Table D.5: Negative-type comparisons with training seed 42 and fixed positive demonstrations on the same 16,254 prompts. Both groups share the no-negative reference from Table D.4 , and each negative condition uses one response per prompt.

Verifier reward	Response content
Verifier reward	Clearly invalid	Near-correct
0	$40.32\pm 0.07$	$19.35\pm 0.04$
0.1	$24.19\pm 0.05$	$16.13\pm 0.03$

Appendix

Table D.6: Proportion (%) of selected negative offline trajectories by verifier reward and response-content category over the five training seeds of Table 1 . Each seed contributes 16,254 negative trajectories.

Sampling rule	AIME25		MinervaMath		AMC23		MATH500		Avg.
Sampling rule	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32
Age-balanced	20.52	50.00	24.31	42.28	60.23	95.00	72.91	87.40	44.49	68.67
Source-balanced	20.31	46.67	24.57	42.65	60.08	92.50	73.12	88.00	44.52	67.45
Prompt-balanced	20.42	50.00	24.51	42.28	60.16	95.00	72.94	87.60	44.51	68.72
Response-balanced	20.63	50.00	24.47	42.28	60.16	95.00	72.99	87.60	44.56	68.72

Appendix

Table D.7: Qwen3-1.7B replay-sampling comparisons across complete training runs with matched generation and optimizer-step budgets.

Age	Residual level	Residual Abs Avg.	Log prob Avg.	Reward-1 prop.(%)	AIME25		MinervaMath		AMC23		MATH500
Age					avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32
2–4	Low	1.53	-181.75	86.57	20.42	50.00	24.15	41.91	60.16	95.00	72.75	87.00
2–4	High	7.06	-266.46	20.19	20.63	50.00	24.41	42.28	60.16	95.00	73.00	87.60
5–16	Low	2.18	-200.82	62.81	20.31	46.67	24.00	41.91	59.92	92.50	72.60	86.40
5–16	High	8.43	-223.48	51.22	20.83	53.33	24.60	42.65	60.39	95.00	73.30	88.00
$>16$	Low	4.24	-239.89	38.23	20.10	46.67	23.49	41.54	59.38	90.00	72.00	86.00

Appendix

Table D.8: Replay statistics aggregated over training and benchmark performance by policy age and residual level. Age is measured in policy-version intervals at the time of replay.

Figure D.2: Checkpoint-wise turning of CBPO-Unified and SFT

\rightarrow

DAPO trajectories.

Figure D.3: Optimization trajectories with five PCA components.

Figure D.4: Optimization trajectories with ten PCA components.

Method	$\boldsymbol{L/D}$		Cumulative turn
	Raw	Normalized	Raw	Normalized
2 components: 82.8744% / 82.8637%
SFT $\rightarrow$ DAPO	1.2207	1.2267	$368.75^{\circ}$	$368.96^{\circ}$
CBPO-Unified	1.0205	1.0214	$58.20^{\circ}$	$58.96^{\circ}$
5 components: 91.6378% / 91.6359%
SFT $\rightarrow$ DAPO	1.1923	1.1980	$295.67^{\circ}$	$296.59^{\circ}$

Appendix

Table D.9: Layer-scale sensitivity of the checkpoint trajectories. Each block reports cumulative explained variance for raw and layer-normalized displacements, respectively.

Method	PCA components	$\boldsymbol{L/D}$	Cumulative turn	Avg.
Method				avg@32	pass@32
SFT $\rightarrow$ CBPO-OnPolicy	2	1.1695	$355.74^{\circ}$	42.71	65.22
	5	1.1083	$285.91^{\circ}$
	10	1.0847	$279.33^{\circ}$
CBPO-Unified	2	1.0205	$58.20^{\circ}$	44.56	68.72
	5	1.0196	$69.76^{\circ}$

Appendix

Table D.10: SFT

\rightarrow

CBPO-OnPolicy versus CBPO-Unified. Cumulative explained variance for raw displacements is 83.4765%, 92.8952%, and 96.2445% for 2, 5, and 10 PCA components, respectively.

$\boldsymbol{\beta}$	avg@32	pass@32
0.1	41.84	66.59
0.2	44.56	68.72
0.4	43.15	67.45
0.8	42.60	66.29

Appendix

Table D.11: CBPO-Unified reward-temperature sensitivity for

\beta\in\{0.1,0.2,0.4,0.8\}

Policy Optimization

Policy optimization develops algorithms for improving decision rules in reinforcement learning, language models, and sequential decision systems.

See all 221 papers

Jan 17, 2026

R $^2$ PO: Decoupling Rollout and Inference Policies for LLM Reasoning

Existing reinforcement learning methods for LLM reasoning implicitly assume that the policy generating training trajectories should coincide with the one producing inference responses. We argue that this is a misleading inductive bias: the optimization-optimal trajectory distribution favors informative gradients, whereas the inference-optimal response distribution emphasizes accuracy and consistency. Forcing both into a single policy entangles their gradients and suppresses exploration. We propose R

^2

PO (Residual Rollout Policy Optimization), which attaches a lightweight Residual Rollout-Head atop the policy to decouple training trajectories from inference responses, diversifying rollouts during training while keeping inference generation intact. Experiments show that R

^2

PO consistently outperforms baselines, with average accuracy gains of 3.4% on MATH-500 and 1.3% on APPS, alongside more diverse rollouts and reduced length bias. Our code is available at https://github.com/RRPO-ARR/Code.

Jingchu Wang, Bingbing Xu, Yige Yuan +4

State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · National University of Singapore

RL for Language Model Reasoning Policy Optimization RL Exploration RL Post-Training Policy Learning

Ask PDF

Jun 3, 2026

SocraticPO: Policy Optimization via Interactive Guidance

Reinforcement learning (RL) for large language models usually supervises reasoning with scalar outcome rewards, such as binary correctness. Such rewards provide an optimization direction but rarely explain how a model should revise its mistaken reasoning, which can encourage shortcut learning and brittle policies. We propose SocraticPO (Socratic Policy Optimization), a policy-optimization framework that augments RL rollouts with Socratic-style natural-language guidance. During rollout, the student first answers independently; if the answer is incorrect, a teacher diagnoses the attempt and provides concise corrective guidance, after which the student continues under the expanded context. Crucially, this guidance is paired with reward decay: correct answers obtained after teacher intervention only receive decayed rewards, preventing the policy from treating teacher help as a free path to reward. Since SocraticPO only modifies the rollout process while leaving the standard expected-reward objective intact, it can be plugged into existing policy-gradient backends such as Reinforce++. Moreover, because the teacher provides only text-level guidance, SocraticPO can leverage stronger black-box teacher models without requiring access to logits or distribution matching. On undergraduate-level scientific reasoning benchmarks from SciKnowEval, SocraticPO improves over strong RL and self-distillation baselines. Ablations show that both targeted guidance and reward decay are necessary, with reward decay mitigating reliance on assisted correction.

Zirui Liu, Jie Ouyang, Qi Liu +8

State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · iFLYTEK AI Research (Central China), iFLYTEK Co., Ltd

RL for Language Model Reasoning LLM-Guided RL Scientific Reasoning in Language Models Teacher-Student Learning Policy Learning Socratic Tutoring

Ask PDF

Apr 4, 2026

Design Conditions for Intra-Group Learning of Sequence-Level Rewards: Token Gradient Cancellation

Reinforcement learning for multi-step reasoning with large language models (LLMs) typically relies on sparse terminal rewards, which creates a poorly conditioned credit-assignment problem: the final feedback is propagated uniformly across all intermediate decisions. This leads to high gradient variance, unstable training, and many ineffective updates, ultimately limiting sustained model improvement. We propose a counterfactual-comparison framework for credit assignment. For each input, the framework samples multiple reasoning trajectories and treats their differences as implicit approximations to alternative decisions. This yields an implicit process-level advantage estimator that converts sparse terminal rewards into step-sensitive learning signals. Building on this framework, we introduce Implicit Behavior Policy Optimization (IBPO), which substantially improves training stability and the performance ceiling on mathematical and code-reasoning benchmarks. Our results point to a promising direction for unlocking the reasoning potential of LLMs.

Fei Ding, Yongkang Zhang, youwei wang +1

Alibaba Group · Tsinghua University

RL for Language Model Reasoning Step-Level Credit Assignment in RL Sparse-Reward RL Counterfactual Credit Assignment

Ask PDF

Abstract

Policy Optimization

Explore similar work

R2^22PO: Decoupling Rollout and Inference Policies for LLM Reasoning

SocraticPO: Policy Optimization via Interactive Guidance

Design Conditions for Intra-Group Learning of Sequence-Level Rewards: Token Gradient Cancellation

R $^2$ PO: Decoupling Rollout and Inference Policies for LLM Reasoning