cs.AISep 27, 2026
SaveSeOPD: Self-Evolving LLMs via Online Policy Distillation from Self-Generated Chain-of-Thought
Organizations: National University of Defense Technology
Abstract
Recent advances in online policy self-distillation (OPSD) have demonstrated that large language models (LLMs) can improve their capabilities by leveraging external privileged information (PI), such as manual annotations or feedback from external environments. However, obtaining accurate annotations and constructing sophisticated environments often require substantial human effort and computation, limiting the scalability of OPSD. While a few recent studies have explored self-improvement without external PI, the resulting gains remain limited. In this work, we explore whether LLMs can achieve comparable self-improvement without external PI. Our key observation is that a single LLM can support multiple reasoning modes, such as deep-thinking and non-thinking modes, with deep thinking generating additional information during reasoning. Based on this observation, we propose Self-Evolving Online Policy Distillation (SeOPD), which enables LLMs to distill and internalize information generated by their own chain of thought (CoT). Specifically, it (1) generates CoT with the deep-thinking mode, (2) produces responses with the non-thinking mode, and (3) uses the generated CoT as PI to provide token-level supervision for the non-thinking response, allowing new information inferred during reasoning to guide the non-thinking mode and be internalized into the shared model parameters, thereby improving both non-thinking and deep-thinking capabilities. Extensive experiments across LLMs and tasks demonstrate the effectiveness of SeOPD.
Figures & tables
Figure 1: Motivation of SeOPD: Internalizing new knowledge from self-generated CoT for self-evolving LLMs, thereby reducing reliance on expensive external privileged information.
Figure 2: Overview of the SeOPD training framework
| Model | Mode | Science Q&A | Tool Use | Coding | ||||
| Chemistry | Physics | Biology | Materials | Mean | ToolAlpaca | LCBv6 | ||
| Qwen3-4B | Thinking | 45.9 | 56.2 | 29.0 | 73.1 | 51.1 | 57.3 | 41.7 |
| Non-Thinking | 42.9 | 60.6 | 33.9 | 62.6 | 50.0 | 58.4 | 46.6 | |
| + SeOPD | Thinking | 59.3 | 68.1 | 39.9 | 72.8 | 60.0 | 59.5 | 63.5 |
| Non-Thinking | 58.7 | 61.7 | 40.1 | 75.7 | 59.7 | 62.0 | 53.4 | |
| Qwen3-8B | Thinking | 47.5 | 58.0 | 26.9 | 62.8 | 48.8 | 58.4 | 43.1 |
Table 1: Performance comparison across science Q&A, tool use, and coding tasks. indicates an improvement over the corresponding base model.
| Method | LCBv6 | IFEval | MATH-500 | |||
| Thinking | Non-Thinking | Thinking | Non-Thinking | Thinking | Non-Thinking | |
| Qwen3-8B | 43.1 | 51.7 | 85.0 | 83.1 | 97.1 | 83.8 |
| SeOPD (LCBv6) | 64.5 | 56.4 | 84.6 | 81.5 | 96.4 | 84.2 |
Table 2: Performance after training only on LCBv6. IFEval and MATH-500 are hold-out tasks.
Figure 5: Per-step CoT perplexity under the frozen reference model. Rising PPL across domains indicates that SeOPD internalizes new knowledge and, during reasoning, uses it to produce additional knowledge that the initial model cannot understand.
| Privileged Information | Science Q&A | Tool Use | Coding | ||||
| Chemistry | Physics | Biology | Materials | Mean | ToolAlpaca | LCBv6 | |
| External privileged information | |||||||
| Answer (SDPO) | 80.1 | 77.3 | 60.0 | 78.0 | 73.9 | – | – |
| Environment feedback (SDPO) | – | – | – | – | – | 64.71 | 62.7 |
| Few-shot (SDFT) | 75.9 | 73.2 | 53.5 | 76.5 | 69.8 | 65.07 | 53.4 |
| System Prompt (OPCD) | 47.0 | 57.4 | 37.8 | 68.2 | 52.6 | 57.0 | 52.5 |
Table 3: Comparison of external and internal PI on Science Q&A, tool use, and coding. All experiments are conducted on Qwen3-8B. “–” indicates that the method is not applicable to the corresponding setting.
Figure 6: Comparison of different CoT for SeOPD on Science Q A. Experiments are conducted on Qwen3-8B.
| Method | Science Q&A | Tool Use | Coding |
| Reference Model (Qwen3-8B) | 48.8 | 58.4 | 43.1 |
| After SeOPD | 52.4 | 58.9 | 45.8 |
Table 4: Comparison between the reference model and SeOPD using the same CoT from the reference model.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameters | Science QA | ToolUse | LCB |
| Data | |||
| Max prompt length | 2048 | 2048 | 2048 |
| Max response length | 13312 | 13312 | 24576 |
| Batching | |||
| Question batch size | 32 | 32 | 32 |
| Mini batch size | 32 | 32 | 32 |
Table 5: Hyperparameters for SeOPD training.
Figure 7: Mean response length of non-thinking during the first 40 SeOPD training steps. Each panel corresponds to one task. LCBv6 is shown up to step 19, which is the length of that run.
| How many heavy atoms are there in the molecule with the IUPAC name ‘(3R)-3-aminobutanoate’? A: 7 B: 30 C: 31 D: 9 |
| CoT before SeOPD (Number of tokens = 4612): … Each carbon atom can have atoms attached. For atom count, I need to count all heavy atoms, probably meaning atoms other than hydrogen, but in chemistry, "heavy atoms" often refer to carbon, nitrogen, oxygen, etc., excluding hydrogen. In the context of this question, "heavy atoms" likely means the carbon atoms, as is common in some contexts, but the options include numbers like 7, 30, 31, 9, so it might not be just carbon. Perhaps "heavy atoms" includes all atoms except hydrogen. But let’s think about the molecule. In "3-aminobutanoate", the carbon chain has 4 carbons. The amino group is -NH2, so there is one nitrogen and two hydrogens. But the carbon from the amino group might be considered part of the molecule, but "3-aminobutanoate" likely means the amino is attached directly to the carbon, so the carbon is still part of the chain. Perhaps "3-aminobutanoate" is a way to name an amino acid, like 3-aminobutanoic acid. But "butanoate" might imply the acid form. I think "3-aminobutanoate" could be 3-aminobutanoic acid, which has four carbons in the chain. But "butanoate" might be ambiguous; perhaps it’s a four-carbon chain with a carboxylic acid. But let’s count the atoms. Perhaps "3-aminobutanoate" is the same as 3-aminobutanoic acid, which has molecular formula C4H9NO2. So, carbon atoms: 4 Hydrogen: 9 Nitrogen: 1 Oxygen: 2 But "heavy atoms" – if it’s carbon, then 4, but 4 is not in the options. Options are 7, 30, 31, 9. … Oh! 7 is option A. Perhaps "heavy atoms" means atoms other than hydrogen. … So, I’ll go with A. |
| CoT After SeOPD (Number of tokens = 2023): … In chemistry, heavy atoms often refer to atoms with atomic number greater than hydrogen, so carbon, nitrogen, oxygen, etc. In this case, for C4H9NO2, the heavy atoms are C4, N1, O2, so 4+1+2=7. Yes, 7 heavy atoms. Option A is 7. … So, answer is A. |
Table 6: Case Study Q1: Knowledge Internalization on Science Q&A.
| Context: Your task is to answer the user’s question using available tools. You have access to the following tools: generateAvatar: Create a unique avatar based on the user’s preferences. … Question: My Discord server’s current icon is getting old. I’d like to get a fresh look. Please generate a minimal styled server icon for my server with the ID "987654321" in JPG format. |
| CoT before SeOPD (Number of tokens = 1100): Okay, the user wants a fresh server icon for their Discord server with ID "987654321". They specifically asked for a minimal style in JPG format. First, I need to verify if the server ID is correct. Since the user provided it directly, I’ll assume it’s accurate unless instructed otherwise. The request is straightforward—they want a new icon generated using the available tools. The tool generateServerIcon is perfect for this. It takes the server ID, style preference, and format. The user specified "minimal" style and JPG format, so I’ll map those directly to the parameters. I wonder if "minimal" is clear enough for the tool’s interpretation. Maybe the user expects something like clean lines or a simple geometric shape. But since they didn’t provide more details, I’ll stick to their exact request. The output should be a URL, which they can use directly in Discord. No need for additional steps unless they ask for something else. This seems like a simple, one-step task. The user knows what they want, and the tools can deliver it. No need to overcomplicate things here. |
| CoT After SeOPD (Number of tokens = 350): The user wants a new server icon for their Discord server. The tool generateServerIcon is perfect for this. I need to use the server ID they provided and specify the style and format. They mentioned "minimal styled," so I’ll use "minimal" as the style. The format should be JPG. No other details are given, so I’ll keep it simple. |
Table 7: Case Study Q2: Procedural Efficiency on ToolAlpaca.
| Question: Given three integers , determine whether they can be permuted so that . |
| CoT before SeOPD (Number of tokens = 2162): We are given three integers A1, A2, and A3, and we need to determine whether there exists a permutation (B1, B2, B3) such that B1 * B2 = B3. Since there are three elements, there are 3! = 6 possible permutations. One straightforward approach is to explicitly enumerate all six permutations and check whether the first two elements multiply to the third. … Alternatively, because multiplication is commutative, we can reduce the six permutations to three distinct cases: A1 * A2 = A3, A1 * A3 = A2, or A2 * A3 = A1. … However, we should carefully verify that these three conditions indeed cover all possible permutations. For example, if the permutation is (A2, A1, A3), the condition becomes A2 * A1 = A3, which is equivalent to A1 * A2 = A3. Similarly, (A3, A1, A2) is equivalent to A1 * A3 = A2, and (A3, A2, A1) is equivalent to A2 * A3 = A1. … We also need to consider whether duplicate values cause any issue. The problem allows the input values to be equal, so a permutation should still use each occurrence exactly once. For example, for (2, 2, 4), the permutation (2, 2, 4) satisfies 2 * 2 = 4. Thus, duplicates do not change the validity of the three-condition formulation. … We could therefore either enumerate all permutations using itertools.permutations or directly check the three arithmetic conditions. Since the input contains only three integers, both approaches are easily fast enough. … Let me further verify the cases using the examples. For (3, 15, 5), we have 3 * 5 = 15, so the answer is Yes. For (5, 3, 2), none of the three products equals the remaining number, so the answer is No. … Therefore, the direct condition ab == c or ac == b or b*c == a is sufficient. … |
| CoT After SeOPD (Number of tokens = 1100): We need to determine whether the three given integers can be reordered so that the first two multiply to the third. The key observation is that the permutation only determines which number serves as the product and which two numbers serve as its factors. Therefore, instead of explicitly constructing all six permutations, we can characterize the valid permutations by the three possible choices of the product element. If A3 is the product, we need A1 * A2 = A3. If A2 is the product, we need A1 * A3 = A2. If A1 is the product, we need A2 * A3 = A1. Because multiplication is commutative, swapping the two factors does not create a new condition. Thus, these three checks completely characterize whether a valid permutation exists. … For example, with 3, 15, and 5, the third number can be viewed as the product because 3 * 5 = 15, so the corresponding permutation (3, 5, 15) satisfies the requirement. For 5, 3, and 2, none of the three possible product assignments is valid. … The input values are positive integers bounded by 100, so ordinary integer multiplication is sufficient. There are only three arithmetic comparisons, so the computation takes constant time and constant space. … Thus, the solution can directly evaluate: a * b == c or a * c == b or b * c == a If any of these conditions holds, we print "Yes"; otherwise, we print "No". The important part is that the permutation structure has been reduced to the underlying multiplicative relationship, so there is no need to explicitly enumerate or repeatedly verify the individual permutations. … |
Table 8: Case Study Q3: Structural Internalization on LCBv6.