cs.LGOct 25, 2026
SaveSelf-Evolution Reasoner: Continuous Optimization via Policy-Intrinsic Exploration and Exploitation
Organizations: Renmin University of China · Peking University · Tencent
Abstract
Reinforcement learning with verifiable rewards (RLVR) enables large language models (LLMs) to continuously self-improve their reasoning capabilities. Compared to RLVR's scalar rewards, recent self-supervised on-policy distillation methods leverage denser supervision signals. However, both paradigms suffer from exploration bottlenecks, struggling to escape local optima when the policy fails to sample valid solutions. Relying on external demonstrations can bypass this issue; however, the off-policy nature hinders genuine self-improvement and the mismatched-distribution reasoning can not be fully exploited. In this work, we propose Self-Evolution Policy Optimization (SEPO), a framework that drives continuous reasoning capability evolution via guided exploration and in-distribution exploitation. Specifically, SEPO utilizes labels to elicit correct reasoning from the policy, unlocking the exploration potential. By treating current policy conditioned on this in-distribution reasoning as a self-teacher, SEPO distills the newly discovered solution back into the policy. Through this iterative cycle of policy-intrinsic exploration and exploitation, SEPO achieves robust and continuous capability evolution. Across scientific, tool-use, and complex logical reasoning domains, SEPO consistently achieves the best performance under identical training time or steps, demonstrating superior efficiency and convergence ceilings. Notably, SEPO maintains stable improvements even in scenarios where baselines severely underperform or fail entirely, such as on complex tasks or with small-scale models. Our code is available at https://github.com/Eric8932/SEPO.
Figures & tables
Figure 1: Overview of Self-Evolution Policy Optimization (SEPO) framework. SEPO couples guided exploration with in-distribution exploitation, overcoming the exploration bottlenecks and distribution-mismatched exploitation of prior methods, enabling more sustained self-evolution.
| Chemistry | Biology | Physics | Material | Tool use | ||||||
| 1h | 10h | 1h | 10h | 1h | 10h | 1h | 10h | 1h | 10h | |
| Qwen3-8B | 41.5 | 30.5 | 58.6 | 59.8 | 57.1 | |||||
| + GRPO (off-policy) | 48.0 | 73.9 | 30.5 | 68.1 | 59.5 | 74.8 | 70.9 | 76.1 | 60.9 | 67.6 |
| + OPCD ( Self ) | 66.4 | 81.4 | 49.4 | 67.9 | 62.2 | 77.0 | 67.6 | 74.5 | 63.6 | 65.7 |
| + OPCD ( Expert ) | 72.5 | 78.4 | 50.1 | 57.4 | 63.6 | 73.8 | 72.5 | 76.7 | 63.3 | 64.6 |
| + OPCD ( Label ) | 58.8 | 78.5 | 38.5 | 66.9 | 59.9 | 68.5 | 72.9 | 77.8 | 62.5 | 62.5 |
Table 1: Performance comparison on science QA and tool use benchmarks, reporting the highest avg@16 achieved within 1 hour and 10 hours of training time. Bold and underlined denote the best and second-best performance, respectively. SEPO consistently achieves the optimal or near-optimal performance at both time marks, demonstrating superior training efficiency and performance ceilings.
Figure 2: Training process on (a) Chemistry and (b) Arrow Maze tasks using Qwen3-8B. Shaded regions show the standard deviation across 3 runs. SEPO consistently maintains top performance throughout training, achieving effective improvements even on Arrow Maze where baselines fail.
Figure 3: Highest avg@16 on Chemistry after 10h training across Qwen3 scales. SEPO consistently performs best, uniquely outperforming GRPO at 0.6B and 1.7B.
Figure 4: ( Left ) Response length progression. ( Right ) Best@ across different at the highest avg@16 checkpoint. Both evaluations use Qwen3-8B on Chemistry task. SEPO compresses reasoning while achieving superior best@ , maintaining exploration diversity.
Figure 5: An exemplary reasoning process of SEPO on a Chemistry case, showing a natural self-reflective reasoning pattern.
Figure 6: Impact of loss divergence and top- truncation on 10-hour performance. JSD performs best and top-100 matches full-distribution distillation.
Figure 7: Impact of rollout group size on peak 1-hour and 10-hour performance. Smaller size accelerates training but lower the ceiling, yet any configurations outperform other methods.
| Method | Chem. | Bio. | Tool Use |
| SEPO (Dynamic) | 47.5 | 45.2 | 61.4 |
| OPCD (Self, Fixed) | 78.5 | 64.3 | 63.7 |
| SEPO (Fixed) | 80.6 | 65.1 | 66.9 |
| OPCD (Self, EMA) | 81.4 | 67.9 | 65.7 |
| SEPO (EMA) | 82.9 | 69.0 | 68.2 |
Table 2: Impact of teacher-update strategy on peak 10-hour performance. EMA performs best, while SEPO outperforms OPCD (Self) under both fixed and EMA teachers.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Given a question and four options, please select the right answer. Respond in the following format: <reasoning> … </reasoning> <answer> … </answer> For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output: <answer>A</answer> {Question} Please reason step by step. |
Table 3: Base Meta-Prompt utilized for standard student rollouts.
| {Base Meta-Prompt} (Correct answer: {label}. Reason as you naturally would without this hint — arrive at the correct answer through genuine understanding, not by working backward from it.) |
Table 4: Label-Conditioned Reasoning Prompt utilized for teacher exploration.
| {Base Meta-Prompt} Here is a reference solution: {solution} Now, solve the problem on your own. |
Table 5: Contextual Distillation Prompt utilized during the self-distillation phase.
| Chemistry | Biology | Physics | Material | Tool use | ||||||
| 1h | 10h | 1h | 10h | 1h | 10h | 1h | 10h | 1h | 10h | |
| Qwen3-8B | 41.5 | 30.5 | 58.6 | 59.8 | 57.1 | |||||
| + GRPO (off-policy) | 48.0 | 73.9 | 30.5 | 68.1 | 59.5 | 74.8 | 70.9 | 76.1 | 60.9 | 67.6 |
| + GRPO (on-policy) | 42.5 | 70.5 | 32.4 | 58.0 | 60.5 | 70.2 | 69.3 | 78.1 | 59.3 | 66.5 |
| Olmo3-7B-Instruct | 25.1 | 13.0 | 37.5 | 37.6 | 45.3 | |||||
| + GRPO (off-policy) | 25.4 | 54.7 | 26.0 | 50.6 | 40.5 | 53.2 | 61.1 | 73.9 | 52.7 | 63.4 |
Table 6: Supplementary performance comparison between the off-policy and on-policy GRPO variants, extending the results from Table 1 .
| Chemistry | Biology | Physics | Material | Tool use | ||||||
| 50 | 400 | 50 | 400 | 50 | 400 | 50 | 400 | 50 | 400 | |
| Qwen3-8B | 41.5 | 30.5 | 58.6 | 59.8 | 57.1 | |||||
| + GRPO (off-policy) | 63.2 | 73.6 | 51.7 | 68.1 | 67.5 | 76.1 | 72.9 | 77.1 | 60.9 | 68.0 |
| + OPCD ( Self ) | 76.1 | 81.7 | 55.0 | 67.9 | 68.2 | 77.0 | 70.1 | 74.5 | 63.4 | 65.7 |
| + OPCD ( Expert ) | 76.2 | 78.3 | 49.2 | 57.4 | 68.3 | 73.8 | 73.0 | 78.1 | 63.7 | 64.6 |
| + OPCD ( Label ) | 74.2 | 78.9 | 48.2 | 66.9 | 67.3 | 68.5 | 75.7 | 77.8 | 62.5 | 62.5 |
Table 7: Performance comparison on science QA and tool-use benchmarks, reporting the highest avg@16 achieved within 50 and 400 training steps. Bold and underlined values denote the best and second-best performance, respectively.
| Chemistry | Biology | Physics | Material | Tool use | ||||||
| 1h | 10h | 1h | 10h | 1h | 10h | 1h | 10h | 1h | 10h | |
| Qwen3-8B | – | – | – | – | – | |||||
| + GRPO (off-policy) | 0.42 | 0.27 | 1.05 | 0.41 | 0.82 | 0.44 | 0.96 | 1.05 | 0.85 | 0.49 |
| + OPCD ( Self ) | 0.31 | 0.67 | 0.92 | 0.55 | 1.01 | 0.53 | 0.88 | 0.42 | 0.62 | 0.94 |
| + OPCD ( Expert ) | 0.56 | 1.02 | 0.45 | 0.88 | 0.94 | 0.47 | 1.12 | 0.55 | 1.08 | 0.51 |
| + OPCD ( Label ) | 0.48 | 1.13 | 0.68 | 0.62 | 0.55 | 0.92 | 0.76 | 0.39 | 0.93 | 0.45 |
Table 8: Standard deviations across three independent runs corresponding to the main results in Table 1 .
| Chemistry (Thinking) | Physics(Thinking) | Dyck Language | ||||
| 1h | 10h | 1h | 10h | 1h | 10h | |
| Qwen3-8B | 56.3 | 70.2 | 12.5 | |||
| + GRPO (off-policy) | 57.3 | 62.5 | 71.3 | 72.6 | 15.0 | 73.8 |
| + OPCD ( Self ) | 58.3 | 78.3 | 72.2 | 76.3 | 19.8 | 96.7 |
| + OPCD ( Expert ) | 59.8 | 65.8 | 71.4 | 75.7 | 18.8 | 86.9 |
| + OPCD ( Label ) | 57.2 | 63.4 | 70.5 | 74.3 | 14.3 | 64.4 |
Table 9: Performance comparison when models are in thinking mode on science tasks, alongside results on the Dyck Language logical reasoning task.
| Question How many hydrogen bond acceptors are there in the molecule with the canonical SMILES representation ’C=CNCC(=O)N’? A: 2 B: 8 C: 6 D: 9 (answer: A=2) |
| Complete teacher reasoning To determine the number of hydrogen bond acceptors in the molecule represented by the SMILES string ’C=CNCC(=O)N’, we need to identify all atoms that can accept hydrogen bonds. Hydrogen bond acceptors are typically atoms that have lone pairs of electrons and are capable of forming hydrogen bonds with hydrogen atoms. These include: Oxygen (O), Nitrogen (N), Fluorine (F), Sulfur (S) In the given molecule, the atoms of interest are: 1. The nitrogen in the ’C=CN’ group: This is a nitrogen atom in a double bond and has a lone pair, making it a hydrogen bond acceptor. 2. The oxygen in the ’C(=O)N’ group: This is an oxygen double-bonded to carbon and has two lone pairs, making it a hydrogen bond acceptor. 3. The nitrogen in the ’N’ at the end: This is a terminal nitrogen atom that has a lone pair and acts as a hydrogen bond acceptor. No other atoms in the molecule are capable of hydrogen bonding, as the other carbon and hydrogen atoms cannot act as hydrogen bond acceptors. Thus, there are a total of 3 hydrogen bond acceptors in the molecule: 1 nitrogen, 1 oxygen, and 1 nitrogen. However, the terminal nitrogen (the last N in C(=O)N) is not a hydrogen-bond acceptor in this context. Let’s re-identify the atoms: the non-amide nitrogen in the C=CN group has an available lone pair and can act as a hydrogen-bond acceptor; the oxygen in the C(=O)N group is a double-bonded oxygen with lone pairs and can act as a hydrogen bond acceptor; the carbon and hydrogen atoms can not act as hydrogen bond acceptors. Thus, there are a total of 2 hydrogen bond acceptors: one nitrogen and one oxygen. |
Table 10: A complete retained intrinsic teacher trace for the Chemistry question.
| Method | Output |
| GRPO | To calculate the molar mass of a molecule given its SMILES string ’C1CC1C(=O)NCCC=O’, we need to determine the molecular formula and sum the atomic masses: 1. Decompose the SMILES string: - C1CC1: This is a cyclopropane ring (C3H6). - C(=O): This is a carbonyl group (C=O). - N: A single nitrogen atom. - CCC=O: This is an ethyl group with a double bond to oxygen (C2H5O). 2. Break it down into atoms: Carbon (C) : There are 5 carbon atoms in the ring (C1CC1) + 1 from C(=O) + 2 from CCC=O = 8 C atoms. Hydrogen (H) : There are 6 H atoms in the ring (C1CC1) + 2 H atoms in CCC=O = 8 H atoms. Oxygen (O) : There is 1 O from C(=O) and 1 O from CCC=O = 2 O atoms. Nitrogen (N) : 1 atom. 3. Atomic masses (g/mol) :- Carbon (C) = 12.011. - Hydrogen (H) = 1.008. - Oxygen (O) = 16.00.- Nitrogen (N) = 14.007. 4. Calculate molar mass: (8\times 12.011)+(8\times 1.008)+(2\times 16.00)+(1\times 14.007)=96.088+8.064+32.00+14.007={\color[rgb]{0.6719,0.3047,0.293}150.159} g/mol (rounded to 3 decimal places). \n Thus, the correct answer is C: 159.180 . |
| Self | The SMILES representation ’C1CC1C(=O)NCCC=O’ creates a molecule that contains: A five-membered ring with two carbon atoms, one nitrogen, and one oxygen atom (C2N1O1) , A side chain of three carbon atoms, one hydrogen atom, and two oxygen atoms (C3H1O2), A carbonyl group and a nitrogen group, thus summing up the full molecular composition. After combining these, the molecular formula is correctly determined as C 6 H 11 N 2 O 3 , reflecting the accurate count of atoms and their sources. Here are the calculations: Carbon (C): 6 × 12.01 = 72.06 g/mol; Hydrogen (H): 11 × 1.008 = 11.09 g/mol; Nitrogen (N): 2 × 14.01 = 28.02 g/mol; Oxygen (O): 3 × 16.00 = 48.00 g/mol. Total molar mass = 72.06 + 11.09 + 28.02 + 48.00 = 159.17 g/mol . This matches Option C (159.180 g/mol), which is the only option that closely aligns with the calculated value and corresponds to the correct molecular formula. Hence, Option C is the most accurate answer. |
| Expert | The SMILES string ’C1CC1C(=O)NCCC=O’ represents a molecule with a cyclobutane ring and a side chain. By breaking down the structure, we determine the molecular formula to be C8H12NO2 . Calculating the molar mass: Carbon: 8 carbons * 12.01 g/mol = 96.08 g/mol, Hydrogen: 12 hydrogens * 1.008 g/mol = 12.096 g/mol, Nitrogen: 1 nitrogen * 14.01 g/mol = 14.01 g/mol, Oxygen: 2 oxygens * 16.00 g/mol = 32.00 g/mol. Total molar mass = 96.08 + 12.096 + 14.01 + 32.00 = 154.186 g/mol . After careful analysis, the molecular formula calculation results in a molar mass consistent with the option C: 159.18 g/mol . |
| Label | Let’s break down the SMILES string and identify the atoms and bonds in the molecule. - ’C1CC1’: This forms a cyclopropane ring with two carbon atoms connected to the ring. - ’C(=O)’: This is a carbonyl group, indicating a carbon double-bonded to an oxygen. - ’N’: This is a nitrogen atom. - ’CCC=O’: This is a chain of three carbon atoms ending in a carbonyl group (C=O). So the molecule is a cyclopropane ring (C3), with a carbonyl group (C=O), a nitrogen (N), and a chain of three carbons ending in a carbonyl group. Now, we count each atom and calculate the total molecular weight based on standard atomic weights: Carbon: 8 × 12.011 = 96.088 g/mol, Oxygen: 2 × 16.00 = 32.00 g/mol, Nitrogen: 1 × 14.007 = 14.007 g/mol. Total molecular weight: 96.088 + 32.00 + 14.007 = 142.095 g/mol But wait—this calculation assumes the structure is that simple. […] Now, repeating the molecular weight calculation: […] Total molecular weight: 96.088 + 32.00 + 14.007 = 142.095 g/mol [Wait—this still doesn’t match the correct answer. Let’s go back and accurately analyze the SMILES again.] *4 Adding gives: 96.088 + 32.00 + 14.007 = 142.095 g/mol . So the correct answer is C . |
| SEPO | The SMILES string ’C1CC1C(=O)NCCC=O’ describes a molecule with a six-membered ring containing a nitrogen atom, a carbonyl group, and a terminal carbonyl group. A careful count of the atoms gives:- 12 carbon atoms: the ring has 5 carbons, and the remaining 7 carbons from the side chain and other groups. - 10 hydrogen atoms: the hydrogens are located on the ring and the substituents. - 1 nitrogen atom: part of the ring structure. - 2 oxygen atoms: one from the carbonyl group and another from the amide group. Using the standard atomic weights: Carbon: g/mol, Hydrogen: g/mol, Oxygen: g/mol, Nitrogen: g/mol. Adding these: g/mol However, after revisiting the SMILES, C1CC1 is a three-membered cyclopropane ring. Recounting gives 7 carbon atoms, 11 hydrogen atoms, 1 nitrogen atom, and 2 oxygen atoms. Thus, the molar mass is 141.168 g/mol , aligning with option B. |
Table 11: Responses from different methods after 100 training steps to the question: “What is the molar weight (g/mol) of the molecule with the canonical SMILES representation C1CC1C(=O)NCCC=O? A: 172.270, B: 141.170, C: 159.180, D: 183.200.” The correct answer is B. Red marks erroneous structural claims, calculations, or an incorrect final answer. Blue marks superficial mechanical reasoning in GRPO, and repeated reflection in Label when the derived answer is inconsistent with the options. Green denotes spontaneous and corrective reflection in SEPO.
More in this topic
Language Model Self-Improvement
Language model self-improvement studies how models use their own outputs, feedback, and experience to improve with less external supervision.
See all 171 papers