SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning

Jialong Liu; Yuling Shi; Ning Yang; Xiaodong Gu; Zuchao Li

cs.AIAug 24, 2026

SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning

Authors: Jialong Liu, Yuling Shi, Ning Yang, Xiaodong Gu, Zuchao Li

Organizations: School of artificial intelligence, Wuhan University, Wuhan, China · School of computer science, Shanghai Jiao Tong University, Shanghai, China · Institute of automation, Chinese Academy of Sciences, Beijing, China

Abstract

Self-reflection is a powerful mechanism for credit assignment in human learning, converting sparse outcome feedback into actionable guidance. However, its potential for post-training Large Language Models (LLMs) remains underexplored. We propose Self-Reflective Policy Optimization (SRPO), a framework that internalizes this capability. SRPO enables LLMs to analyze their own completed trajectories, synthesize errors into concise "reflection patches," and use reflection-conditioned teacher scores on student on-policy rollouts as dense token-level training signals. This process effectively transforms sparse terminal supervision into dense, token-level learning signals without requiring external critics, separate reward models, or larger teacher models. We demonstrate that SRPO achieves state-of-the-art performance across mathematical reasoning and long-horizon agentic benchmarks with exceptional data efficiency. Using a Qwen3-8B base model, SRPO attains 73.3% on AIME'24 using only 8% (0.08x) of the training FLOPs required by scaled supervised fine-tuning, while significantly improving success rates on WebShop (64.7%), ALFWorld (76.8%), and SWE-Bench-Lite (31.2%). Code is available at https://github.com/Galleons2029/SRPO

Figures & tables

Figure 1: Overview of the SRPO Framework. The process consists of two stages using the same model

\pi_{\theta}

. Stage 1: Given a prompt

x

and a sparse outcome

o

from an initial attempt, the model generates a concise Reflection Patch (

p

). This patch is prepended to the prompt (Reset-with-Memory) to guide the model in generating a high-quality “Teacher” distribution

\pi_{\theta}(\cdot\mid[p;x])

. Stage 2: The base model (Student), seeing only the original prompt

x

, generates on-policy rollouts. It is optimized to minimize the Reverse KL divergence between its output distribution and the Teacher’s distribution, effectively converting sparse outcomes into dense, token-level supervision.

Figure 2: Performance comparison on mathematical reasoning benchmarks. All methods start from the same SFT-400K checkpoint. SRPO achieves the best performance across all benchmarks with significantly fewer training FLOPs (0.08

\times

). † Inference-time methods with 3 refinement iterations. Detailed numerical results are in Table 5 .

Figure 3: Performance on long-horizon agentic benchmarks. (a) Success rate comparison across WebShop, ALFWorld, and SWE-Bench-Lite. (b) Average episode steps showing execution efficiency. SRPO achieves the highest success rates while maintaining the shortest episode length. Detailed numerical results are in Table 6 .

Figure 4: Inference-time scaling on AIME’24. (a) Performance vs. number of refinement iterations for different methods. SRPO-trained models show better scaling with additional compute (+1.8%/iter vs +1.2%/iter for GRPO). (b) Performance vs. total inference FLOPs, demonstrating Pareto efficiency.

Figure 5: Comparison of full fine-tuning vs. LoRA adaptation for SRPO. (a) Performance on AIME’24 showing LoRA-128 achieves 97.8% of full fine-tuning accuracy. (b) GPU memory requirements demonstrating LoRA’s efficiency with 72–73% reduction. Detailed numerical results are in Table 7 .

Stage 1: Reflection Design
Configuration	AIME’24	WebShop
SRPO (Full)	73.3	64.7
w/o reflection (direct retry)	65.8	54.2
w/ verbose reflection ( $>$ 10 points)	70.0	60.3
w/ outcome-only feedback	67.2	56.8
w/ external teacher reflection	71.5	62.4

Table 1: Ablation study on AIME’24 and WebShop. Each row removes or modifies one component from the full SRPO system.

Figure 6: Scaling behavior of SRPO. (a) Performance vs. model size shows consistent improvements across scales, with larger gains for smaller models (+7.8 for 1.5B vs +3.8 for 32B). (b) Performance vs. training compute demonstrates SRPO reaches 70% performance with

\sim

\times

fewer FLOPs than GRPO.

Model Size	SFT	GRPO	SRPO	$\Delta$ vs GRPO
Qwen3-1.5B	38.2	44.5	52.3	+7.8
Qwen3-8B	60.0	68.0	73.3	+5.3
Qwen3-32B	71.4	78.2	82.0	+3.8

Table 2: Scaling analysis across model sizes on AIME’24. SRPO shows consistent improvements across all scales, with the relative gain increasing for smaller models.

Figure 7: Analysis of reflection quality. (a) Distribution of reflection helpfulness scores rated by GPT-4: 67% of reflections receive scores

\geq

4 (useful), while only 8% receive scores

\leq

2 (unhelpful). (b) Strong correlation (

r=0.72

) between reflection quality and improvement in rethinking rollout success rate.

Student	Teacher	AIME’24	Teacher FLOPs
Qwen3-8B	None (GRPO)	68.0	0
	Qwen3-32B	70.0	4.0 $\times$
	Qwen3-72B	72.5	9.0 $\times$
	Self (SRPO)	73.3	1.0 $\times$
Qwen3-1.5B	Qwen3-8B	48.7	5.3 $\times$
	Qwen3-32B	51.2	21.3 $\times$

Table 3: Comparison of self-distillation (SRPO) vs. distillation from larger teachers. Self-distillation achieves competitive performance without requiring access to larger models.

Method	Math (Before)	Code (After)	Math (After)	Retention
SFT	60.0	28.4	48.2	80.3%
GRPO	68.0	26.7	59.3	87.2%
SRPO	73.3	31.2	69.8	95.2%

Table 4: Continual learning evaluation. Models are first trained on math (AIME’24) then adapted to code (SWE-Lite). SRPO better preserves original capabilities while acquiring new skills.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Method	AIME’24	MATH-500	GSM8K	DeepScaleR	Train FLOPs
SFT-400K (init)	45.0 $\pm$ 1.9	52.4 $\pm$ 1.0	69.2 $\pm$ 0.6	28.3 $\pm$ 1.4	–
PPO	55.3 $\pm$ 2.1	65.8 $\pm$ 1.0	80.7 $\pm$ 0.5	41.2 $\pm$ 1.3	$1.2\times$
GRPO	68.0 $\pm$ 1.7	72.2 $\pm$ 0.8	88.4 $\pm$ 0.4	53.8 $\pm$ 1.1	$1.0\times$
OPD (Qwen3-32B teacher)	70.0 $\pm$ 1.5	78.6 $\pm$ 0.6	92.1 $\pm$ 0.3	55.4 $\pm$ 1.0	$0.11\times$
OPD (Qwen3-72B teacher)	72.5 $\pm$ 1.4	80.1 $\pm$ 0.5	93.0 $\pm$ 0.3	55.8 $\pm$ 1.0	$0.23\times$
GRPO + prompt reflection	69.5 $\pm$ 1.6	73.4 $\pm$ 0.8	89.0 $\pm$ 0.4	54.1 $\pm$ 1.1	$1.0\times$

Appendix

Table 5: Performance comparison on mathematical reasoning benchmarks. All methods start from the same SFT-400K checkpoint and are trained with identical prompt template, sampling temperature, max tokens, batch size, rollouts per prompt, and learning schedule (Appendix B.8 ); the only variable is the teacher distribution. We report mean over 5 independent seeds; the

\pm

values are paired-bootstrap standard deviations over 10 000 resamples. “Train FLOPs” is the full Stage 1+Stage 2 cost (Appendix B.4 ), normalized to GRPO. † Inference-time methods with 3 refinement iterations.

Method	WebShop	ALFWorld	SWE-Lite	LiveCodeBench	Avg. Steps
Qwen3-8B (base)	42.3 $\pm$ 1.4	51.2 $\pm$ 1.7	18.7 $\pm$ 1.3	22.4 $\pm$ 1.2	12.4
SFT (expert traj.)	56.8 $\pm$ 1.2	68.4 $\pm$ 1.4	24.3 $\pm$ 1.2	26.1 $\pm$ 1.1	10.8
GRPO	51.2 $\pm$ 1.1	62.7 $\pm$ 1.5	22.1 $\pm$ 1.2	28.4 $\pm$ 1.0	11.6
PPO	49.8 $\pm$ 1.3	60.3 $\pm$ 1.7	21.5 $\pm$ 1.3	27.0 $\pm$ 1.1	11.9
OPD (Qwen3-32B teacher)	57.3 $\pm$ 0.9	69.2 $\pm$ 1.3	26.4 $\pm$ 1.1	31.2 $\pm$ 0.9	11.0
OPD (Qwen3-72B teacher)	61.8 $\pm$ 0.9	72.4 $\pm$ 1.2	28.6 $\pm$ 1.0	32.4 $\pm$ 0.9	10.7

Appendix

Table 6: Performance on long-horizon agentic and code-reasoning benchmarks. Success rate / pass@1 (%) is reported. LiveCodeBench ( Jain et al., 2024 ) is evaluated on a contamination-free Aug 2025 – Jan 2026 slice (240 problems). Mean over 5 seeds;

\pm

are paired-bootstrap standard deviations.

Model	Method	AIME’24	MATH-500	Params	Memory
Qwen3-8B	Full FT	73.3	81.2	8.0B	64GB
	LoRA-64	70.0	79.4	52M	18GB
	LoRA-128	71.7	80.1	104M	22GB
Qwen3-32B	Full FT	82.0	87.3	32B	256GB
	LoRA-64	78.3	85.1	210M	68GB
	LoRA-128	79.7	86.0	420M	78GB

Appendix

Table 7: Comparison of full fine-tuning vs. LoRA adaptation for SRPO. LoRA shows competitive performance with significantly reduced memory footprint, though full fine-tuning maintains an advantage on the most challenging benchmarks.

Stage 1 (one-time preprocessing)
Component	FLOPs ( $\times 10^{18}$ )	% of SRPO total
Initial rollout generation	1.2	22.2%
Reflection patch generation	0.6	11.1%
Rethinking rollout (quality validation)	1.2	22.2%
Stage 2 (500 iterations)
Student on-policy rollout	1.2	22.2%

Appendix

Table 8: Complete FLOPs breakdown for SRPO (Qwen3-8B, AIME’24 training). Stage 1 is one-time, embarrassingly parallel preprocessing that can be cached across epochs; only Stage 2 is repeated each training run. The rethinking rollout

\tilde{y}

is reported for full transparency but is not consumed as a training target in Stage 2, which uses only teacher-forced scoring of student tokens (see Eq. 6 in the main paper). Removing this row would cut Stage 1 cost by 22.2% without altering the Stage 2 objective.

Benchmark	Instances	8-gram overlaps	Overlap rate
AIME’24	30	0	0.0%
MATH-500	500	3	0.6% (common phrases)
GSM8K	1 319	0	0.0%
DeepScaleR	1 200	0	0.0%
GPQA Diamond	448	0	0.0%
LogiQA 2.0	1 572	0	0.0%

Appendix

Table 9: Data contamination audit. 8-gram overlap between OpenThoughts-3 training data and each evaluation benchmark. Numbers in parentheses for MATH-500 indicate the 3 flagged 8-grams, which on manual inspection were common mathematical phrases (e.g. “find the value of

x

such that”), not problem-level contamination.

Variable	SRPO & OPD	Aligned?
Prompt template	system + task	✓
Sampling temperature	0.7	✓
Top- $p$	0.95	✓
Max tokens	4 096	✓
Rollouts per prompt	4 (best-of-4)	✓
Batch size	256	✓

Appendix

Table 10: Experimental controls for the SRPO vs. external-teacher OPD comparison. Every variable other than the teacher distribution is held identical across runs.

Method	Unique Paths / Problem	Novel Correct (%)
OPD (Qwen3-72B teacher)	2.3	8.4
SRPO (self-reflection)	4.1	15.7

Appendix

Table 11: Solution-diversity probe on the 30 AIME’24 problems (10 samples per problem). The reflection-conditioned self-teacher expands the student’s reasoning-path entropy rather than contracting it, in contrast to a larger external teacher.

Figure 8: Reflection semantics control experiment. (a) Performance comparison across reflection conditions: mismatched reflections (from unrelated problems) and template-only reflections yield performance comparable to no reflection, while matched SRPO shows significant gains. (b) Token length distributions are matched across conditions, isolating the effect of semantic content.

Semantic Disruption
Condition	AIME’24	WebShop	Avg. Tokens	$\Delta$ vs Full
SRPO (Full, Matched)	73.3 $\pm$ 0.8	64.7 $\pm$ 1.5	92.4	–
Shuffled Words	66.2 $\pm$ 1.2	55.4 $\pm$ 1.9	92.4	-7.1 / -9.3
Shuffled Sentences	66.8 $\pm$ 1.1	56.1 $\pm$ 1.7	92.4	-6.5 / -8.6
Mismatched Reflection	66.5 $\pm$ 1.0	55.8 $\pm$ 1.8	91.8	-6.8 / -8.9
Content Reduction

Appendix

Table 12: Comprehensive reflection semantics ablation results. Performance drops sharply when reflections lack task-specific semantic content, regardless of format or token count preservation.

Figure 9: Extended reflection semantics analysis. (a) Comprehensive ablation on AIME’24 showing that all semantic disruption conditions converge to baseline performance. (b) WebShop results confirm the pattern generalizes to agentic tasks. (c) Training dynamics reveal that semantic-aligned reflections enable faster convergence. (d) Content analysis shows mismatched reflections lack task-relevant information despite identical format.

Method	Before Adaptation	After Code Adaptation
	Math	Code	Math (Retention)
SFT	60.0	28.4	48.2 (80.3%)
GRPO	68.0	26.7	59.3 (87.2%)
SRPO	73.3	31.2	69.8 (95.2%)

Appendix

Table 13: Continual learning evaluation. Models are first trained on math (AIME’24) then adapted to code (SWE-Lite). SRPO better preserves original capabilities while acquiring new skills.

Explore similar work

SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation
Sep 29, 2026Zhenrui Yue, Huimin Zeng, Yueqi Wang +8RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
University of Illinois · UC San Diego · Rochester Institute of Technology +2
3SPO: State-Score-Supervised Policy Optimization for LLM Agents
Jun 8, 2026Yu Han, Kailing Li, Yang Jiao +4Credit Assignment in RL Step-Level Credit Assignment in RL
School of Computer Science and Technology, East China Normal University · Fudan University · KAUST +1
AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignment
May 18, 2026Zhenlin Wei, Pu Jian, Yingzhuo Deng +6Token-Level Credit Assignment RL for Language Model Reasoning
Meituan Beijing, China · University of Chinese Academy of Sciences

Sep 29, 2026J/K move · Enter open · S save

SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation

Zhenrui Yue, Huimin Zeng, Yueqi Wang, Yaokun Liu+7

University of Illinois · UC San Diego · Rochester Institute of Technology+2

RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards On-Policy Distillation Knowledge Distillation

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for improving large language models (LLMs) on various tasks, yet its sparse outcome rewards lack token-level credit assignment for intermediate steps. To address this, on-policy self-distillation (OPSD) leverages a self-teacher with privileged context to provide additional dense learning signals. However, because the self-teacher is often overconfident and imposes excessive penalties on long reasoning trajectories, OPSD frequently struggles in practice. To mitigate this, we propose self-instructing policy optimization (SIPO) with a contrastive self-teacher to provide dense credit. At each iteration, SIPO samples multiple rollouts per prompt from the current policy, scores them with environment rewards, and constructs two teacher contexts for each rollout by pairing the reference answer with mistakes made within the group. The model then re-evaluates its own responses under both contexts, using the difference between the two teacher log-probabilities as token-level feedback, so that biases shared by both contexts are expected to largely cancel. The resulting objective yields a token-level advantage for every rollout: the reward still sets the main direction of each update while the self-teacher redistributes credit across tokens. Even in groups where every rollout fails and group-relative advantages vanish, SIPO still provides a learning signal. By preserving direct optimization of the task reward while providing dense, token-level feedback, this approach bridges reinforcement learning and on-policy self-distillation. Extensive experiments across multiple reasoning and code-generation benchmarks demonstrate that SIPO outperforms both RLVR and OPSD baselines without an external teacher or additional generation.

Abstract

Explore similar work

SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation

3SPO: State-Score-Supervised Policy Optimization for LLM Agents

AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignment