Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target

Du Cheng; Junyu Lu; Jingfan Zhang; Yuzhuo Zhang; Haoxuan Jia; Shichao Weng

cs.LGOct 9, 2026

Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target

Authors: Du Cheng, Junyu Lu, Jingfan Zhang, Yuzhuo Zhang, Haoxuan Jia, Shichao Weng

Abstract

Reasoning post-training draws on offline demonstrations and policy-generated rollouts, yet conventional supervised fine-tuning (SFT) followed by reinforcement learning (RL) separates their objectives and limits continued reuse of collected responses. We propose Cross-Behavior Policy Optimization (CBPO), which makes these sources compatible with one regression loss through a fixed-reference target. Because the generating policy does not enter this target, positive and negative offline completions, reference responses, and current and historical rollouts can share a common reward-labeled record format and replay buffer. As new responses enter the buffer, optimization retains the same regression objective without coordinating separate SFT and RL objectives or requiring stored behavior-policy probabilities, importance weighting, or a learned critic. Restricting the available sources gives offline, on-policy, and off-policy variants, while CBPO-Unified combines them. Across two model scales and four mathematical reasoning benchmarks, CBPO delivers leading avg@32 and pass@32 performance across benchmarks and data regimes. On Qwen3-1.7B, CBPO-Unified improves benchmark-averaged avg@32 and pass@32 over SFT DAPO by 2.33 and 4.44 percentage points, respectively. In the replay ablation, retaining historical rollouts improves these metrics by 1.87 and 3.72 percentage points over the no-history configuration.

Figures & tables

Figure 1: CBPO-Unified/SFT $\rightarrow$ DAPO: $L/D=1.0205/1.2207$ , with total turn $=58.20^{\circ}/368.75^{\circ}$ .

Figure 2: SFT

\rightarrow

RLVR vs. shared-target CBPO-Unified on offline and on-/off-policy data.

Regime	Methods	AIME25		MinervaMath		AMC23		MATH500		Avg.
Regime	Methods	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32
Offline	SFT	14.23 ± 0.06	29.33 ± 1.49	19.29 ± 0.02	33.53 ± 0.16	48.77 ± 0.03	75.50 ± 1.12	64.99 ± 0.01	77.96 ± 0.09	36.82 ± 0.02	54.08 ± 0.11
	DPO	14.50 ± 0.05	29.33 ± 1.49	19.32 ± 0.01	33.68 ± 0.20	48.97 ± 0.03	74.50 ± 1.12	65.34 ± 0.01	78.16 ± 0.09	37.03 ± 0.02	53.92 ± 0.41
	CBPO-Offline	15.81 ± 0.05	29.33 ± 1.49	20.29 ± 0.01	35.22 ± 0.16	50.11 ± 0.04	78.00 ± 1.12	66.99 ± 0.01	80.44 ± 0.09	38.30 ± 0.02	55.75 ± 0.54
On-policy	PPO	17.48 ± 0.05	32.67 ± 1.49	21.35 ± 0.01	38.82 ± 0.20	56.47 ± 0.03	84.50 ± 1.12	69.71 ± 0.01	83.68 ± 0.11	41.25 ± 0.02	59.92 ± 0.42
	DAPO	17.27 ± 0.05	38.00 ± 1.83	21.58 ± 0.01	39.78 ± 0.16	56.23 ± 0.03	85.50 ± 1.12	70.38 ± 0.01	84.64 ± 0.09	41.37 ± 0.01	61.98 ± 0.67

Table 1: Qwen3-1.7B performance across data regimes and joint SFT/RL over five training seeds.

Regime	Methods	AIME25		MinervaMath		AMC23		MATH500		Avg.
Regime	Methods	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32
Offline	SFT	33.17 ± 0.06	38.00 ± 1.83	45.20 ± 0.01	52.87 ± 0.16	83.16 ± 0.04	87.00 ± 1.12	84.61 ± 0.02	88.56 ± 0.09	61.53 ± 0.02	66.61 ± 0.65
	DPO	33.52 ± 0.05	37.33 ± 1.49	45.60 ± 0.01	53.38 ± 0.16	83.27 ± 0.03	90.50 ± 1.12	85.11 ± 0.01	89.76 ± 0.09	61.87 ± 0.02	67.74 ± 0.38
	CBPO-Offline	34.04 ± 0.05	38.00 ± 1.83	47.20 ± 0.01	55.29 ± 0.20	84.13 ± 0.03	89.50 ± 1.12	85.61 ± 0.02	90.64 ± 0.09	62.74 ± 0.02	68.36 ± 0.39
On-policy	PPO	41.27 ± 0.05	47.33 ± 1.49	50.31 ± 0.01	58.68 ± 0.20	89.72 ± 0.04	95.50 ± 1.12	89.40 ± 0.01	96.44 ± 0.09	67.67 ± 0.01	74.49 ± 0.43
	DAPO	39.40 ± 0.05	42.67 ± 1.49	49.08 ± 0.01	58.16 ± 0.16	88.53 ± 0.03	97.00 ± 1.12	89.60 ± 0.01	96.76 ± 0.09	66.65 ± 0.01	73.65 ± 0.66

Table 2: Qwen3-8B performance across data regimes and joint SFT/RL over five training seeds.

Figure 3: Qwen3-1.7B avg@32 / pass@32 trajectories for twelve methods on four benchmarks.

Methods	AIME25		MinervaMath		AMC23		MATH500		Avg.
Methods	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32
SFT $\rightarrow$ DAPO	18.31 ± 0.05	42.67 ± 1.49	21.98 ± 0.01	40.88 ± 0.16	57.28 ± 0.03	87.00 ± 1.12	71.38 ± 0.02	85.52 ± 0.11	42.24 ± 0.02	64.02 ± 0.36
CBPO w/o History (PosCoT)	18.17 ± 0.06	42.67 ± 1.49	22.33 ± 0.02	40.29 ± 0.20	57.33 ± 0.03	84.50 ± 1.12	71.67 ± 0.01	85.88 ± 0.11	42.37 ± 0.02	63.34 ± 0.43
CBPO w/o History	18.77 ± 0.05	42.67 ± 1.49	22.52 ± 0.02	41.25 ± 0.16	57.98 ± 0.03	89.00 ± 1.37	71.54 ± 0.01	86.04 ± 0.09	42.70 ± 0.01	64.74 ± 0.64
CBPO-Unified FIFO-1	18.98 ± 0.05	44.67 ± 1.83	22.85 ± 0.01	41.25 ± 0.16	58.55 ± 0.04	93.50 ± 1.37	71.96 ± 0.01	86.16 ± 0.09	43.08 ± 0.02	66.39 ± 0.61
CBPO-Unified	20.67 ± 0.06	49.33 ± 1.49	24.47 ± 0.01	42.35 ± 0.16	60.17 ± 0.03	94.50 ± 1.12	72.99 ± 0.01	87.64 ± 0.09	44.57 ± 0.02	68.46 ± 0.59

Table 3: Qwen3-1.7B comparison of SFT

\rightarrow

DAPO and CBPO variants over five training seeds.

Methods	AIME25		MinervaMath		AMC23		MATH500		Avg.
Methods	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32
CBPO-Offline	15.83	30.00	20.29	35.29	50.08	77.50	66.99	80.40	38.30	55.80
CBPO w/o History	18.75	43.33	22.52	41.18	57.97	90.00	71.54	86.00	42.70	65.13
CBPO-Unified FIFO-1	18.96	43.33	22.85	41.18	58.52	92.50	71.96	86.20	43.07	65.80
CBPO-Unified FIFO-4	19.27	46.67	23.10	41.18	58.91	92.50	71.91	86.20	43.30	66.64
CBPO-Unified FIFO-16	19.69	46.67	23.30	41.54	59.14	92.50	72.27	86.40	43.60	66.78

Table 4: Qwen3-1.7B with no history, FIFO-

K

, or full replay.

Methods	$\boldsymbol{L}$	$\boldsymbol{D}$	$\boldsymbol{L/D}$	Cumulative turn
2 components: 82.8744%
SFT $\rightarrow$ DAPO	2.1000	1.7203	1.2207	$368.75^{\circ}$
CBPO-Unified	1.7631	1.7278	1.0205	$\boldsymbol{58.20^{\circ}}$
5 components: 91.6378%
SFT $\rightarrow$ DAPO	2.2498	1.8869	1.1923	$295.67^{\circ}$
CBPO-Unified	1.7688	1.7349	1.0196	$\boldsymbol{69.76^{\circ}}$

Table 5: Geometry of the projected optimization trajectories for 2, 5, and 10 components.

$q_{\mathrm{neg}}$	AIME25		MinervaMath		AMC23		MATH500		Avg.
$q_{\mathrm{neg}}$	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32
$0\%$	19.58	46.67	23.56	41.54	59.22	92.50	71.49	86.60	43.46	66.83
$33.3\%$	20.63	50.00	24.47	42.28	60.16	95.00	72.99	87.60	44.56	68.72
$66.7\%$	19.06	43.33	23.17	41.18	59.14	92.50	71.38	86.80	43.19	65.95
$100\%$	18.65	43.33	22.94	40.44	58.13	92.50	70.28	85.60	42.50	65.47

Table 6: CBPO-Unified

q_{\mathrm{neg}}

sensitivity with three trajectories per prompt.

$\boldsymbol{K_{\mathrm{ref}}}$	avg@32	pass@32
8	42.36	66.52
16	44.15	69.09
24	44.56	68.72
32	45.27	70.15

Table 7: Left:

K_{\mathrm{ref}}

sensitivity with fixed training records, update timing,

M

, and

R_{\mathrm{roll}}

. Right: Current-policy sample size with fixed reference statistics and update timing.

$\boldsymbol{K_{\mathrm{ref}}}$	$\boldsymbol{M}$	$\boldsymbol{R_{\mathrm{roll}}}$	avg@32	pass@32
8	5	32	37.23	60.57
20	8	20	40.65	64.49
24	10	16	44.56	68.72
32	20	8	42.26	66.97

Table 8: Budget-coupled

K_{\mathrm{ref}},M,R_{\mathrm{roll}}

sensitivity.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 12 — Figure B.1: Two-message chat prompt with the system instruction and a row-specific math problem.

Problem provenance	User message	Ground truth
Math12k	What is the least positive integer greater than 1 that leaves a remainder of 1 when divided by each of 2, 3, 4, 5, 6, 7, 8 and 9?	2521
Math12k	Find the remainder when $6x^{4}-14x^{3}-4x^{2}+2x-26$ is divided by $2x-6.$	52
AMC/AIME	The workers in a factory produce widgets and whoosits. For each product, production time is constant and identical for all workers, but not necessarily equal for the two products. In one hour, $100$ workers can produce $300$ widgets and $200$ whoosits. In two hours, $60$ workers can produce $240$ widgets and $300$ whoosits. In three hours, $50$ workers can produce $150$ widgets and $m$ whoosits. Find $m$ .	450
AMC/AIME	How many whole numbers between 1 and 1000 do not contain the digit 1? (A) 512, (B) 648, (C) 720, (D) 728, or (E) 800.	728
Olympiads	It is known that the difference between the variable quantities $z$ and $y$ is proportional to the quantity $x$ , and the difference between the quantities $x$ and $z$ is proportional to the quantity $y$ . The coefficient of proportionality is the same and is a positive integer $k$ . A certain value of the quantity $z$ is $5/3$ times the difference of the corresponding values of $x$ and $y$ . Find the numerical value of the coefficient $k$ .	3
Olympiads	Mr. and Mrs. Zeta want to name baby Zeta so that the first letters of the baby’s first, middle, and last names are in alphabetical order and the letters are distinct. The number of possible names is (A) 276, (B) 300, (C) 552, (D) 600, or (E) 15600.	300

Appendix

Table B.1: Representative RLVR-Math-16k prompts with source provenance and verifier targets.

Regime	Methods	Qwen3-1.7B	Qwen3-8B
Offline	SFT	102.23	413.92
	DPO	106.69	430.28
	CBPO-Offline	105.94	421.76
On-policy	PPO	232.84	901.36
	DAPO	118.43	473.72
	GSPO	118.65	475.28

Appendix

Table B.2: Per-run training cost in GPU hours on two NVIDIA B200 GPUs.

Check	Candidate pairs	Candidate test questions	Same-problem pairs	Matched test questions
Exact-string matching	0	0	0	0
Mathematical-expression normalization	1	1	1	1
Approximate-text matching	90	52	6	5
Semantic-similarity retrieval	106	77	13	12
Deduplicated union	151	97	14	13

Appendix

Table B.3: Training–evaluation overlap screening before decontamination. Counts are deduplicated within each row. Checks can retrieve the same pair or question, so their counts are not additive.

Benchmark	Evaluation questions	Same problem	Changed answer request	Shared setting, different question
AIME25	30	0	0	0
MinervaMath	272	0	0	0
AMC23	40	0	0	0
MATH500	500	13	2	8

Appendix

Table B.4: Evaluation questions matched to the unfiltered training prompt pool. Each category counts distinct evaluation questions. A question may occur in more than one category.

Unit	Before filtering	Removed	Retained
Training-question texts	16,268	14	16,254
Evaluation questions	842	0	842

Appendix

Table B.5: Training and evaluation question counts before and after excluding same-problem training questions.

Setting	Value
Method scope	Core Qwen3-1.7B CBPO configuration
Base and reference model	Qwen3-1.7B
Online prompt pool	Decontaminated RLVR-Math-16k, 16,254 prompts
Held-out evaluation	842 prompts across AIME25, MinervaMath, AMC23, and MATH500
Evaluation samples	32 responses per prompt
Target prompt batch size	$P=512$

Appendix

Table C.1: Core Qwen3-1.7B CBPO settings for data, generation, optimization, and hardware.

Source	Methods	AIME25		MinervaMath		AMC23		MATH500		Avg.
Source	Methods	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32
R1-Qwen-1.5B	BRIDGE	15.21	26.67	21.76	40.07	54.30	75.00	70.23	82.00	40.37	55.94
R1-Qwen-1.5B	CBPO-Unified	18.33	33.33	23.43	41.18	58.13	82.50	71.31	85.20	42.80	60.55
R1-Qwen-7B	BRIDGE	17.60	33.33	22.55	40.81	56.41	82.50	71.01	82.60	41.89	59.81
R1-Qwen-7B	CBPO-Unified	19.90	43.33	23.95	41.54	58.98	87.50	72.05	86.20	43.72	64.64
R1-Qwen-32B	BRIDGE	18.33	43.33	23.01	41.18	57.66	85.00	71.31	83.00	42.58	63.13

Appendix

Table D.1: Qwen3-1.7B performance under three individual teacher sources and the union of all three sources on the common retained prompt set. The mixed condition combines the complete teacher-specific datasets for each method.

Correct responses	Prompts	Proportion (%)	Reward 0 (%)	Reward 0.1 (%)
0–1	5,963	36.69	77.30	20.33
2–5	3,157	19.42	63.56	22.98
6–11	937	5.76	52.19	13.06
12–23	3,110	19.13	17.38	1.94
24	3,087	18.99	0.00	0.00

Appendix

Table D.2: Reference-bank composition by reference success count. Reward frequencies are computed over the recorded reference responses in each group.

	Mean $u_{K}$				90th percentile $u_{K}$
Correct reference responses	$K=8$	$K=16$	$K=24$	$K=32$	$K=8$	$K=16$	$K=24$	$K=32$
0–1	3.9088	3.3017	2.9156	2.6821	4.0156	3.4453	3.0912	2.8242
2–5	2.6713	1.7165	1.3281	1.0938	4.0156	2.1142	1.6655	1.3406
6–11	1.6336	0.9610	0.7252	0.5992	2.5843	1.1732	0.8989	0.7624
12–23	0.7080	0.4044	0.3010	0.2564	1.1295	0.6222	0.4872	0.3940
24	0.4572	0.2288	0.1526	0.1282	0.4572	0.2288	0.1526	0.1282

Appendix

Table D.3: Pointwise 95% exact trinomial bounds on absolute log-normalizer estimation error by reference-sample budget

K

, under independent sampling from

\pi_{\mathrm{ref}}

Figure D.1: Mean avg@32 gain over the reference-policy baseline at each reference-sample budget. The five curves correspond to difficulty groups defined by the number of correct responses in the 24-response reference bank.

Negatives	AIME25		MinervaMath		AMC23		MATH500		Avg.
Negatives	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32
0	19.38	46.67	23.23	41.18	58.91	92.50	71.11	86.40	43.16	66.69
1	20.63	50.00	24.47	42.28	60.16	95.00	72.99	87.60	44.56	68.72
2	20.10	46.67	24.21	41.91	59.69	92.50	72.45	87.00	44.11	67.02

Appendix

Table D.4: Negative-count comparison within CBPO-Unified with two fixed positive offline trajectories per prompt. All other data sources and training settings are held fixed.

Negative type	AIME25		MinervaMath		AMC23		MATH500		Avg.
Negative type	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32
No negative	19.38	46.67	23.23	41.18	58.91	92.50	71.11	86.40	43.16	66.69
Reward 0	19.48	46.67	23.10	40.81	59.14	92.50	71.03	86.40	43.19	66.59
Reward 0.1	20.94	50.00	24.76	42.65	60.39	95.00	73.11	88.00	44.80	68.91
Clearly invalid	18.85	40.00	22.87	40.44	58.52	85.00	70.33	85.20	42.64	62.66
Near-correct	21.35	53.33	24.89	43.01	60.86	97.50	73.62	88.60	45.18	70.61

Appendix

Table D.5: Negative-type comparisons with training seed 42 and fixed positive demonstrations on the same 16,254 prompts. Both groups share the no-negative reference from Table D.4 , and each negative condition uses one response per prompt.

Verifier reward	Response content
Verifier reward	Clearly invalid	Near-correct
0	$40.32\pm 0.07$	$19.35\pm 0.04$
0.1	$24.19\pm 0.05$	$16.13\pm 0.03$

Appendix

Table D.6: Proportion (%) of selected negative offline trajectories by verifier reward and response-content category over the five training seeds of Table 1 . Each seed contributes 16,254 negative trajectories.

Sampling rule	AIME25		MinervaMath		AMC23		MATH500		Avg.
Sampling rule	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32
Age-balanced	20.52	50.00	24.31	42.28	60.23	95.00	72.91	87.40	44.49	68.67
Source-balanced	20.31	46.67	24.57	42.65	60.08	92.50	73.12	88.00	44.52	67.45
Prompt-balanced	20.42	50.00	24.51	42.28	60.16	95.00	72.94	87.60	44.51	68.72
Response-balanced	20.63	50.00	24.47	42.28	60.16	95.00	72.99	87.60	44.56	68.72

Appendix

Table D.7: Qwen3-1.7B replay-sampling comparisons across complete training runs with matched generation and optimizer-step budgets.

Age	Residual level	Residual Abs Avg.	Log prob Avg.	Reward-1 prop.(%)	AIME25		MinervaMath		AMC23		MATH500
Age					avg@32	pass@32	avg@32	pass@32	avg@32	pass@32	avg@32	pass@32
2–4	Low	1.53	-181.75	86.57	20.42	50.00	24.15	41.91	60.16	95.00	72.75	87.00
2–4	High	7.06	-266.46	20.19	20.63	50.00	24.41	42.28	60.16	95.00	73.00	87.60
5–16	Low	2.18	-200.82	62.81	20.31	46.67	24.00	41.91	59.92	92.50	72.60	86.40
5–16	High	8.43	-223.48	51.22	20.83	53.33	24.60	42.65	60.39	95.00	73.30	88.00
$>16$	Low	4.24	-239.89	38.23	20.10	46.67	23.49	41.54	59.38	90.00	72.00	86.00

Appendix

Table D.8: Replay statistics aggregated over training and benchmark performance by policy age and residual level. Age is measured in policy-version intervals at the time of replay.

Figure D.2: Checkpoint-wise turning of CBPO-Unified and SFT

\rightarrow

DAPO trajectories.

Figure D.3: Optimization trajectories with five PCA components.

Figure D.4: Optimization trajectories with ten PCA components.

Method	$\boldsymbol{L/D}$		Cumulative turn
	Raw	Normalized	Raw	Normalized
2 components: 82.8744% / 82.8637%
SFT $\rightarrow$ DAPO	1.2207	1.2267	$368.75^{\circ}$	$368.96^{\circ}$
CBPO-Unified	1.0205	1.0214	$58.20^{\circ}$	$58.96^{\circ}$
5 components: 91.6378% / 91.6359%
SFT $\rightarrow$ DAPO	1.1923	1.1980	$295.67^{\circ}$	$296.59^{\circ}$

Appendix

Table D.9: Layer-scale sensitivity of the checkpoint trajectories. Each block reports cumulative explained variance for raw and layer-normalized displacements, respectively.

Method	PCA components	$\boldsymbol{L/D}$	Cumulative turn	Avg.
Method				avg@32	pass@32
SFT $\rightarrow$ CBPO-OnPolicy	2	1.1695	$355.74^{\circ}$	42.71	65.22
	5	1.1083	$285.91^{\circ}$
	10	1.0847	$279.33^{\circ}$
CBPO-Unified	2	1.0205	$58.20^{\circ}$	44.56	68.72
	5	1.0196	$69.76^{\circ}$

Appendix

Table D.10: SFT

\rightarrow

CBPO-OnPolicy versus CBPO-Unified. Cumulative explained variance for raw displacements is 83.4765%, 92.8952%, and 96.2445% for 2, 5, and 10 PCA components, respectively.

$\boldsymbol{\beta}$	avg@32	pass@32
0.1	41.84	66.59
0.2	44.56	68.72
0.4	43.15	67.45
0.8	42.60	66.29

Appendix

Table D.11: CBPO-Unified reward-temperature sensitivity for

\beta\in\{0.1,0.2,0.4,0.8\}

Policy Optimization

Policy optimization develops algorithms for improving decision rules in reinforcement learning, language models, and sequential decision systems.

See all 221 papers

R $^2$ PO: Decoupling Rollout and Inference Policies for LLM Reasoning
Jan 17, 2026Jingchu Wang, Bingbing Xu, Yige Yuan +4RL for Language Model Reasoning Policy Optimization
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · National University of Singapore
SocraticPO: Policy Optimization via Interactive Guidance
Jun 3, 2026Zirui Liu, Jie Ouyang, Qi Liu +8RL for Language Model Reasoning LLM-Guided RL
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · iFLYTEK AI Research (Central China), iFLYTEK Co., Ltd
Design Conditions for Intra-Group Learning of Sequence-Level Rewards: Token Gradient Cancellation
Apr 4, 2026Fei Ding, Yongkang Zhang, youwei wang +1RL for Language Model Reasoning Step-Level Credit Assignment in RL
Alibaba Group · Tsinghua University

Jan 17, 2026J/K move · Enter open · S save

R $^2$ PO: Decoupling Rollout and Inference Policies for LLM Reasoning

Jingchu Wang, Bingbing Xu, Yige Yuan, Dan Zhang+3

State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · National University of Singapore

RL for Language Model Reasoning Policy Optimization RL Exploration RL Post-Training Policy Learning

Open paper AskPDF arXiv

Existing reinforcement learning methods for LLM reasoning implicitly assume that the policy generating training trajectories should coincide with the one producing inference responses. We argue that this is a misleading inductive bias: the optimization-optimal trajectory distribution favors informative gradients, whereas the inference-optimal response distribution emphasizes accuracy and consistency. Forcing both into a single policy entangles their gradients and suppresses exploration. We propose R

^2

PO (Residual Rollout Policy Optimization), which attaches a lightweight Residual Rollout-Head atop the policy to decouple training trajectories from inference responses, diversifying rollouts during training while keeping inference generation intact. Experiments show that R

^2

PO consistently outperforms baselines, with average accuracy gains of 3.4% on MATH-500 and 1.3% on APPS, alongside more diverse rollouts and reduced length bias. Our code is available at https://github.com/RRPO-ARR/Code.

Abstract

Policy Optimization

Explore similar work

R2^22PO: Decoupling Rollout and Inference Policies for LLM Reasoning

SocraticPO: Policy Optimization via Interactive Guidance

Design Conditions for Intra-Group Learning of Sequence-Level Rewards: Token Gradient Cancellation

R $^2$ PO: Decoupling Rollout and Inference Policies for LLM Reasoning