Lifted State Policy Optimization: Internalizing Self-Correction through Transition Geometry

Ren Zhuang; Ben Wang; Shuifa Sun

cs.AIOct 14, 2026

Lifted State Policy Optimization: Internalizing Self-Correction through Transition Geometry

Authors: Ren Zhuang, Ben Wang, Shuifa Sun

Organizations: Hangzhou Normal University

Abstract

Preference optimization ranks final answers and trajectory RL maximizes aggregate returns, yet neither credits the local change that turns a flawed reasoning state into a better one. We introduce Lifted State Policy Optimization (LSPO), which establishes this finite, cost-adjusted transition as the primary unit of learning. Each textual answer is augmented with a learned continuous coordinate, forming a lifted state that an energy function evaluates and orders. A shaped reward drives energy descent net of edit and step costs, and an internalization loss absorbs selected low-energy answers into the direct generator. On Qwen3.5-27B, the resulting single-pass policy reaches 71.8% aggregate accuracy across mathematics, science, code, and logic benchmarks, matching test-time reflection at 71.9% with 28% lower latency and 25.8% fewer tokens. The pattern holds across dense and mixture-of-experts backbones, and matched controls confirm that transition credit and the lifted coordinate each contribute, as does answer internalization. Our code is available at https://github.com/Venomrko/lspo.

Figures & tables

Figure 1: Improvements found during training persist in direct generation. On Qwen3.5-27B, the LSPO policy attains the highest aggregate direct pass@1 and pass@16 accuracy across benchmarks in the left panels, selected training endpoints align closely with deployed outputs in the top right, and final energy separates correct from incorrect answers in the bottom right.

Figure 2: Comparison of training paradigms. Preference learning in the top left panel optimizes static final answers, while trajectory RL in the top right panel maximizes returns across full rollouts. LSPO in the bottom panel assigns credit to local transitions and internalizes selected endpoints.

Model	Method	MATH-500		AIME25		GPQA		LCB		ZebraLogic		MMLU-Pro		Average
		@1	@16	@1	@16	@1	@16	@1	@16	@1	@16	@1	@16	@1	@16
Qwen3.5-9B	Base	84.8	93.6	39.6	57.2	45.8	61.4	41.9	48.6	60.2	73.4	56.3	68.8	54.8	67.2
	DPO	85.6	94.1	40.9	58.3	46.6	62.3	43.1	50.1	61.2	74.3	57.2	69.7	55.8	68.1
	Step-DPO	86.4	94.7	42.7	60.5	47.5	63.5	44.4	52.0	62.4	75.3	58.1	70.8	56.9	69.5
	GRPO	87.6	95.3	46.0	65.0	49.1	66.4	46.6	55.0	64.4	77.0	59.8	72.7	58.9	71.9
	GDPO	88.0	95.5	46.6	65.9	50.0	67.8	47.4	55.9	65.1	77.5	60.6	73.7	59.6	72.7

Table 1: Main results across Qwen3.5 backbones. LSPO achieves the highest aggregate accuracy among the listed baselines that generate directly under both pass@1 and pass@16 evaluation, with frontier models included as scale references.

Figure 3: Quality, sampling, revision, and token cost on Qwen3.5-27B. LSPO reaches 71.8% accuracy at 10.9 s latency in a single pass; test-time reflection adds 0.1 points but costs 4.3 s more and 440 additional tokens. LSPO remains above the pass@ $k$ curve, most often converts incorrect answers to correct ones, and avoids the long latency tail of repeated refinement.

Method	Net gain	Direct $E$	Terminal $E$
Base	0.0	0.468	N/A
DPO	1.8	0.450	N/A
Step-DPO	2.5	0.438	N/A
GRPO	3.3	0.420	0.203
GDPO	4.0	0.411	0.201
SCoRe	3.6	0.411	0.197

Table 2: Preservation of improvements on Qwen3.5-27B. LSPO yields the highest net gain and lowest energy across all methods.

Method	Agg. ${}_{\pm\text{SD}}$	Latency (s)	Tokens
Select then SFT	70.6 ±0.17	10.58	1218
SI only	70.8 ±0.16	10.74	1246
GRPO	70.0 ±0.18	12.04	1370
SCoRe	70.7 ±0.18	13.12	1501
Reflection	71.9 ±0.19	15.18	1705
LSPO	71.8 ±0.15	10.93	1265

Table 3: Training alternatives and deployment cost. Test-time reflection matches LSPO accuracy at higher deployment cost.

Figure 4: Training dynamics across transition stability, net gain, and direct accuracy. Transition stability and net gain plateau before direct accuracy, consistent with the geometric landscape forming before the generator absorbs it.

Variant	Agg.	Net gain	Latency (s)	Tokens	GPU hrs
LSPO full	71.8	4.7	10.93	1265	2983
- w/o lifting	70.9	3.2	11.70	1316	2804
- w/o progress reward	70.0	1.6	12.35	1366	2715
- w/o internalization	70.3	2.4	12.02	1341	2566
GRPO	70.0	3.3	12.04	1370	2386
DPO	66.9	1.8	10.42	1196	1218

Table 4: Component ablations on Qwen3.5-27B. The full LSPO stack leads in aggregate score while controlling latency, output length, and training cost.

Panel of Figure 5 — Figure 5: Policy-training energy and task-level net gains. The left panel shows recorded energy across policy updates, where the curve tracks the cumulative minimum and points mark new minima. The right panel independently reports net correction gains across benchmarks.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Figure B.1: Benchmark profile for Qwen3.5-9B. LSPO improves direct accuracy while preserving substantial sampled pass@16 headroom in the smaller dense backbone.

Figure B.2: Benchmark profile for Qwen3.5-27B. LSPO achieves consistent direct gains while preserving the sampled pass@16 envelope.

Figure B.3: Benchmark profile for Qwen3.5-35B-A3B. LSPO remains the strongest direct policy as available performance headroom narrows in the MoE backbone.

Method	MATH-500	GPQA	LCB	Aggregate	Latency (s)
Base	87.6	48.7	47.1	61.1	6.81
DPO	88.3	49.4	48.0	61.9	7.04
GRPO	89.2	50.1	49.4	62.9	7.18
LSPO	90.7	51.3	52.0	64.7	7.31

Appendix

Table B.1: Transfer across backbones to Gemma-4-E4B-it. LSPO attains the highest aggregate score on this distinct architecture while incurring only a minor latency overhead relative to direct preference baselines.

Figure C.1: A single LSPO training update. Move shaping in the lifted space and answer internalization act concurrently on the selected candidate.

Component	Architecture	Input / output	Key setting
Lift projection $g_{\psi}$	2-layer MLP, width 1024, GELU	Hidden state $\rightarrow z\in\mathbb{R}^{8}$	Input LayerNorm
Energy head $\tilde{E}$	3-layer MLP, width 1024, SiLU	Pooled text, $z$ , task embedding $\rightarrow$ scalar	Pairwise margin 0.20
Transition block	2 residual blocks, width 1024, SiLU	Text delta, $b_{t}$ , action embedding $\rightarrow\Delta z$	$z_{t+1}=\mathrm{LN}(b_{t}+\Delta z)$
Action embedding	Lookup table, width 128	Revise, Switch, Stop $\rightarrow$ vector	Frozen transition conditioning
Value head	2-layer MLP, width 512, SiLU	Transition state $\rightarrow$ scalar value	GAE 0.95, clip 0.2

Appendix

Table C.1: Architectural specifications for the primary Qwen3.5-27B LSPO implementation.

Task family	Public corpus source	Samples
Math	open-r1/OpenR1-Math-220k 1 1 1 https://huggingface.co/datasets/open-r1/OpenR1-Math-220k	38,000
Code	BAAI/TACO 2 2 2 https://huggingface.co/datasets/BAAI/TACO	24,000
Science	allenai/sciq 3 3 3 https://huggingface.co/datasets/allenai/sciq	18,000
Logic	longface/logicLM 4 4 4 https://huggingface.co/datasets/longface/logicLM	12,000

Appendix

Table C.2: Training corpus mixture and sample counts shared by the compared methods.

Measure / phase	Revise (%)	Switch (%)	Stop (%)	Reach $T_{\max}$ (%)
Overall	49.8	25.9	24.3	11.8
Energy decrease rate	70.8	63.4	–	–
Early 0–30%	56.1	30.8	13.1	21.4
Late 70–100%	42.6	20.2	37.2	5.9

Appendix

Table C.3: Observed action use across LSPO training phases on Qwen3.5-27B. Revise and Switch transitions are also reported by their energy decrease rates.

Figure C.2: Annotated reward and objective for LSPO. The assignment of credit per step, the regularization of the policy, and direct training on the answer occupy distinct terms within the update.

Figure C.3: Training alternatives on Qwen3.5-27B. LSPO achieves accuracy comparable to test-time reflection with lower deployment cost and gains from constructive state changes.

Method	Agg. $\pm$ SD	LSPO diff. (95% CI)	Inference contract
Select then SFT	70.6 $\pm$ 0.17	+1.2 [0.7, 1.6]	direct
SI only	70.8 $\pm$ 0.16	+1.0 [0.6, 1.4]	direct
Without lifting	70.9 $\pm$ 0.18	+0.9 [0.5, 1.3]	direct
Without progress reward	70.0 $\pm$ 0.19	+1.8 [1.3, 2.3]	direct
Return-matched terminal reward	69.7 $\pm$ 0.20	+2.1 [1.5, 2.6]	direct
Test-time reflection	71.9 $\pm$ 0.19	–	critique + revision

Appendix

Table C.4: Matched training alternatives on Qwen3.5-27B. Evaluation examples and nominal budgets are shared; Table C.7 specifies candidate pools, reward definitions, and SI/SFT matching. SI only uses a fixed LSPO candidate pool.

Family	Variant	Aggregate drop	Net gain
Lift	$z$ dimension = 64	0.6	3.9
Lift	$z$ dimension = 2	0.8	3.7
Lift	No lift	0.9	3.2
Lift	No switch	1.4	3.1
Lift	Revise only	1.7	2.7
Lift	Switch only	2.0	2.5

Appendix

Table C.5: Extended ablations on Qwen3.5-27B. Perturbations to the lifted coordinate and corruption of energy degrade distinct metrics, separating the role of state geometry from that of local reward shaping.

Energy signal	Agg.	Drop	Net gain	AUC	Pairwise
Clean LSPO energy	71.8	0.0	4.7	0.792	0.734
Gaussian noise	70.3	1.5	2.9	0.731	0.681
Label flip or shuffle	68.6	3.2	0.7	0.508	0.506
Terminal only	69.7	2.1	2.2	0.681	0.641

Appendix

Table C.6: Sensitivity to energy corruption on Qwen3.5-27B. Gaussian noise weakens but preserves part of the transition gain, whereas flipped or shuffled ordering drives rank metrics toward chance.

Variant	Candidate pool	State space	Target selector	Controller RL	Reward signal	Transfer target
Full LSPO	Dynamic rollout	Lifted $(x,z)$	Frozen $\tilde{E}$	Dense policy	$\Delta\tilde{E}-\text{costs}$	$\min\tilde{E}$ (SI)
Return-matched terminal reward	Dynamic rollout	Lifted $(x,z)$	Frozen $\tilde{E}$	Delayed policy	Trajectory $R(\tau)$	$\min\tilde{E}$ (SI)
SI only on fixed pool	Fixed Full pool	Lifted $(x,z)$	Frozen $\tilde{E}$	None	None	$\min\tilde{E}$ (SI)
Select then SFT	Fixed Full pool	Surface text	Task verifier	None	None	Verifier-best (SFT)
Without lifting	Dynamic rollout	Surface text	Surface $E$	Dense policy	$\Delta E-\text{costs}$	$\min E$ (SI)
Without progress reward	Dynamic rollout	Lifted $(x,z)$	Frozen $\tilde{E}$	Dense policy	$-\text{costs only}$	$\min\tilde{E}$ (SI)

Appendix

Table C.7: Control specifications for training alternatives. Nominal budgets specify candidate limits, update counts, and sampling settings; they do not imply identical realized token counts or GPU hours.

Input condition	Action acc. $\pm$ SD	NLL $\pm$ SD	Energy AUC $\pm$ SD
Text only	62.8 $\pm$ 0.7	0.842 $\pm$ 0.011	0.681 $\pm$ 0.010
Text plus $z$	68.9 $\pm$ 0.6	0.768 $\pm$ 0.010	0.732 $\pm$ 0.009
Text plus shuffled $z$	60.9 $\pm$ 0.8	0.874 $\pm$ 0.013	0.641 $\pm$ 0.012
Text plus zero $z$	63.4 $\pm$ 0.7	0.831 $\pm$ 0.012	0.687 $\pm$ 0.011

Appendix

Table C.8: Frozen probe for information in the lifted coordinate on transitions held out from fitting. Adding the learned coordinate improves action prediction and energy ordering; shuffled coordinates remove the association.

Variant	Agg. $\pm$ SD	Net gain	LSPO $-$ variant (95% CI)
Full LSPO	71.8 $\pm$ 0.15	4.7	–
Reference baseline ( Table 4 )
Current no-lift baseline	70.9 $\pm$ 0.18	3.2	+0.9 [0.5, 1.3]
Parameter-matched coordinate controls
Same-parameter no-lift MLP	70.8 $\pm$ 0.18	3.0	+1.0 [0.6, 1.4]
Random $z$ , matched dim.	70.3 $\pm$ 0.19	2.8	+1.5 [1.1, 1.9]

Appendix

Table C.9: Parameter-matched controls for the lifted coordinate on Qwen3.5-27B. All variants use the same candidate budget, training budget, and three-seed protocol. The final column reports the paired aggregate gap between full LSPO and each control.

Variant	Agg.	Gain	Latency (s)	Tokens	GPU hours
LSPO full	72.6	4.4	8.30	1097	2410
Without lifting	71.6	2.9	8.88	1141	2265
Without progress reward	70.8	1.3	9.38	1185	2193
Without internalization	71.1	2.1	9.13	1163	2073
Outcome RL	71.5	2.7	9.33	1200	1940
DPO	68.8	1.5	8.08	1048	990

Appendix

Table C.10: Component ablation on the Qwen3.5-35B-A3B MoE backbone. The dense 27B model remains the primary attribution setting; this table tests whether the component pattern transfers to the MoE model.

Figure D.1: Energy discriminates correct from incorrect endpoints across scales. Correct outputs consistently concentrate in regions with lower energy.

Figure D.2: Training endpoints align with direct outputs after LSPO. The divergence between terminal states selected during alignment and direct generations is minimal under the LSPO objective.

Figure D.3: Step diagnostics across scales. LSPO preserves the strongest monotonicity in energy descent, correct local structure, and stop optimality across the evaluated backbones.

Figure D.4: Transition gains across scales. LSPO achieves the largest surplus of conversions from incorrect to correct on every backbone while keeping conversions from correct to incorrect modest.

Method	Initial $p_{V}$	$V\to V$	$I\to V$	$V\to I$	Net gain
Full LSPO	40.1	95.8	29.5	4.2	+16.0
Same-parameter no-lift MLP	39.7	92.1	21.6	7.9	+9.9
Without progress reward	39.8	89.8	18.7	10.2	+7.2
Terminal-only reward	39.1	90.3	16.9	9.7	+6.5
GRPO	39.4	90.9	18.6	9.1	+7.7

Appendix

Table D.1: Span validity transitions. Initial validity, preservation, repair, and damage are percentages; net gain is in percentage points. Preservation and damage condition on initially valid spans; repair conditions on initially invalid spans. Net gain weights these rates by each method’s initial valid fraction.

Figure E.1: Deployment cost distributions. LSPO stays within the latency and length ranges established by preference optimization, remaining below baselines that use extensive search.

Figure E.2: Frozen energy ordering under distribution shift. On tasks held out from fitting (GSM8K, ARC-Challenge, BBH, HumanEval, and MBPP), the LSPO energy head outperforms outcome only, shuffled, and random controls in area under the curve, Spearman correlation, and pairwise accuracy.

Score	AUC	Spearman	Kendall	Pairwise	Bins	ECE
Task families used for fitting include math, code, logic, and science
Random energy	0.502	0.01	0.01	0.502	2	0.241
Outcome only score	0.681	0.31	0.22	0.641	4	0.132
LSPO energy	0.792	0.48	0.35	0.734	5	0.071
Shuffled LSPO energy	0.508	0.01	0.01	0.506	2	0.235
Task families held out from fitting include GSM8K, ARC-C, BBH, HumanEval, and MBPP

Appendix

Table E.1: Frozen energy ordering under distribution shift. LSPO yields the strongest AUC, rank correlation, and pairwise preference accuracy on both the fitted task families and tasks held out from fitting.

Method	GSM8K	ARC-C	BBH	HumanEval	MBPP	Avg. $\pm$ SD
Base	88.4	84.7	66.2	55.5	62.4	71.4 $\pm$ 0.22
DPO	89.0	85.3	67.1	56.4	63.0	72.2 $\pm$ 0.20
GRPO	90.6	86.7	69.0	58.6	64.8	73.9 $\pm$ 0.22
SCoRe	91.1	87.1	69.3	59.1	65.0	74.3 $\pm$ 0.24
LSPO	91.5	87.4	70.4	60.1	65.2	74.9 $\pm$ 0.20

Appendix

Table E.2: Direct generation on benchmark suites held out from policy training, energy fitting, and endpoint selection. LSPO improves over Base on every benchmark.

$T_{\max}$	$K$	Agg. $\pm$ SD	Gain $\pm$ SD	Path length	Unique ratio
2	3	70.9 $\pm$ 0.18	3.2 $\pm$ 0.30	1.6	0.82
4	5	71.5 $\pm$ 0.16	4.2 $\pm$ 0.26	2.7	0.76
6	7	71.8 $\pm$ 0.15	4.7 $\pm$ 0.23	3.4	0.71
8	9	71.8 $\pm$ 0.16	4.8 $\pm$ 0.24	3.7	0.66

Appendix

Table E.3: Joint candidate and horizon sensitivity on Qwen3.5-27B.

K

counts the maximum number of retained states per input, including the initial direct state

s_{0}

. Values are means

\pm

sample SD across three independent training seeds. Aggregate accuracy reaches 71.8 at

(T_{\max},K)=(6,7)

and remains at that value for

(8,9)

Method	WritingBench L&A	$\Delta$ Base	MT-Bench Roleplay	$\Delta$ Base	avg.
Base	7.42 $\pm$ 0.08	0.00	7.61 $\pm$ 0.11	0.00	7.52
DPO	7.50 $\pm$ 0.07	+0.08	7.68 $\pm$ 0.10	+0.07	7.59
GRPO	7.36 $\pm$ 0.09	-0.06	7.53 $\pm$ 0.12	-0.08	7.45
SCoRe	7.32 $\pm$ 0.09	-0.10	7.49 $\pm$ 0.12	-0.12	7.41
DRO-adapted	7.53 $\pm$ 0.08	+0.11	7.70 $\pm$ 0.10	+0.09	7.62
LSPO	7.48 $\pm$ 0.08	+0.06	7.66 $\pm$ 0.10	+0.05	7.57

Appendix

Table E.4: Policy transfer pilot on subjective tasks. LSPO maintains positive transfer over Base across both benchmarks without requiring test-time rubrics, reference answers, or verifier loops.

Subset	Spearman $\rho$	Kendall $\tau$	Pairwise AUC	$\Delta$
WritingBench Literature and Arts	0.34 $\pm$ 0.04	0.24 $\pm$ 0.03	0.675 $\pm$ 0.018	+0.42
MT-Bench Roleplay	0.29 $\pm$ 0.06	0.20 $\pm$ 0.04	0.642 $\pm$ 0.026	+0.35

Appendix

Table E.5: Frozen energy ordering in open form transfer. Energy is evaluated as an ordering of seven candidates from each input; judge scores are used only for evaluation.

Subset	Strongest alignment	Weakest alignment	Failure slice
WritingBench Literature and Arts	Instruction fit, coherence	Creativity, originality	Prompts requiring high creativity
MT-Bench Roleplay	Character consistency, instruction fit	Expressive richness	Playful improvisation

Appendix

Table E.6: Alignment dimensions and failure slices for the open form audit. The frozen energy ordering is most reliable on constrained dimensions, while open-ended creativity and expressive richness remain its main failure modes.

Method	Peak mem (GB)	Elapsed time (h)	GPU hrs	Pilot GPU hours	Latency (s)	Output tokens
DPO	57.7	152.2	1217.6	146.1	10.42	1196
Step-DPO	57.7	203.1	1624.8	195.0	10.73	1242
GRPO	57.7	298.3	2386.4	286.4	12.04	1370
GDPO	57.7	335.6	2684.8	322.2	11.54	1324
SCoRe	57.7	358.2	2865.7	343.9	13.12	1501
LSPO	61.9	372.9	2983.2	358.0	10.93	1265

Appendix

Table E.7: Compute and deployment accounting for Qwen3.5-27B. Training uses

8\times

NVIDIA A100 80GB GPUs. The LSPO cost of 2983.2 GPU-hours is the mean per-run cost across three seeds.

$K_{\mathrm{sel}}$	Mean $\tilde{E}$	Energy drop	MATH-500 $\Delta$ EM (pp)	WritingBench L&A	MT-Bench Roleplay
0	$0.398\pm 0.0041$	0.0%	0.0	$7.47\pm 0.077$	$7.65\pm 0.104$
1	$0.322\pm 0.0044$	19.1%	+0.2	$7.50\pm 0.083$	$7.68\pm 0.109$
2	$0.268\pm 0.0047$	32.7%	+0.6	$7.53\pm 0.091$	$7.71\pm 0.115$
4	$0.224\pm 0.0049$	43.7%	+0.4	$7.53\pm 0.087$	$7.71\pm 0.121$
8	$0.195\pm 0.0052$	51.0%	+0.2	$7.50\pm 0.096$	$7.68\pm 0.128$
16	$0.171\pm 0.0057$	57.0%	-0.4	$7.45\pm 0.111$	$7.62\pm 0.141$

Appendix

Table E.8: Energy ordering under increasing selection budgets with one fixed Full LSPO checkpoint.

K_{\mathrm{sel}}

counts additional candidates beyond the direct output; selection uses all

K_{\mathrm{sel}}+1

candidates. Energy and open-form scores are means

\pm

bootstrap SD over evaluation inputs. Mean energy averages one selected state per input. Energy drop and MATH-500 change are relative to direct generation; open-form scores are absolute.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,087 papers

May 8, 2026

Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States

Reinforcement learning with verifiable rewards (RLVR) for Large Reasoning Models rests on variance reduction, which requires both a reliable baseline and high prompt diversity within each training batch. This is especially difficult in multi-domain training for general reasoning models, where prompts from different tasks induce highly diverse gradient signals. Existing approaches fall short in different ways: GRPO estimates its baseline as the group mean over rollouts from the same prompt, so an accurate baseline leaves fewer distinct prompts in the batch, while PPO avoids this trade-off by training a policy scale critic, roughly doubling the cost of training. We introduce POISE (Policy Optimization with Internal State Value Estimation), a reinforcement learning algorithm that turns the model's internal states into a value model. A lightweight probe reads the signals already computed during the forward pass to predict the baseline, and is trained online alongside the policy. To preserve gradient unbiasedness, we introduce a cross-rollout construction that predicts each rollout's value from an independent rollout's internal states. On Qwen3-4B and OLMo3-7B-Instruct-DPO across a six-domain verifiable-reward corpus, POISE outperforms other RLVR baselines while achieving more stable training. Moreover, the probe matches a separate LLM-scale value model, generalizes to various tasks, and remains accurate as the policy scales. By leveraging the model's internal representations, POISE enables stable policy optimization.

Yunho Choi, Jongwon Lim, Woojin Ahn +3

Graduate School of Data Science Seoul National University · Computer Science and Engineering Seoul National University

Value Function Estimation RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards Actor-Critic Methods Policy Learning

Ask PDF

Jun 3, 2026

SocraticPO: Policy Optimization via Interactive Guidance

Reinforcement learning (RL) for large language models usually supervises reasoning with scalar outcome rewards, such as binary correctness. Such rewards provide an optimization direction but rarely explain how a model should revise its mistaken reasoning, which can encourage shortcut learning and brittle policies. We propose SocraticPO (Socratic Policy Optimization), a policy-optimization framework that augments RL rollouts with Socratic-style natural-language guidance. During rollout, the student first answers independently; if the answer is incorrect, a teacher diagnoses the attempt and provides concise corrective guidance, after which the student continues under the expanded context. Crucially, this guidance is paired with reward decay: correct answers obtained after teacher intervention only receive decayed rewards, preventing the policy from treating teacher help as a free path to reward. Since SocraticPO only modifies the rollout process while leaving the standard expected-reward objective intact, it can be plugged into existing policy-gradient backends such as Reinforce++. Moreover, because the teacher provides only text-level guidance, SocraticPO can leverage stronger black-box teacher models without requiring access to logits or distribution matching. On undergraduate-level scientific reasoning benchmarks from SciKnowEval, SocraticPO improves over strong RL and self-distillation baselines. Ablations show that both targeted guidance and reward decay are necessary, with reward decay mitigating reliance on assisted correction.

Zirui Liu, Jie Ouyang, Qi Liu +8

State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · iFLYTEK AI Research (Central China), iFLYTEK Co., Ltd

RL for Language Model Reasoning LLM-Guided RL Scientific Reasoning in Language Models Teacher-Student Learning Policy Learning Socratic Tutoring

Ask PDF

Aug 24, 2026

SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning

Self-reflection is a powerful mechanism for credit assignment in human learning, converting sparse outcome feedback into actionable guidance. However, its potential for post-training Large Language Models (LLMs) remains underexplored. We propose Self-Reflective Policy Optimization (SRPO), a framework that internalizes this capability. SRPO enables LLMs to analyze their own completed trajectories, synthesize errors into concise "reflection patches," and use reflection-conditioned teacher scores on student on-policy rollouts as dense token-level training signals. This process effectively transforms sparse terminal supervision into dense, token-level learning signals without requiring external critics, separate reward models, or larger teacher models. We demonstrate that SRPO achieves state-of-the-art performance across mathematical reasoning and long-horizon agentic benchmarks with exceptional data efficiency. Using a Qwen3-8B base model, SRPO attains 73.3% on AIME'24 using only 8% (0.08x) of the training FLOPs required by scaled supervised fine-tuning, while significantly improving success rates on WebShop (64.7%), ALFWorld (76.8%), and SWE-Bench-Lite (31.2%). Code is available at https://github.com/Galleons2029/SRPO

Jialong Liu, Yuling Shi, Ning Yang +2

School of artificial intelligence, Wuhan University, Wuhan, China · School of computer science, Shanghai Jiao Tong University, Shanghai, China · Institute of automation, Chinese Academy of Sciences, Beijing, China

RL for Language Model Reasoning Policy Optimization RL Post-Training RL for LLM Agents LLM Reasoning LLM Self-Refinement

Ask PDF

Abstract

RL for Language Model Reasoning

Explore similar work

Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States

SocraticPO: Policy Optimization via Interactive Guidance

SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning