Recently, reinforcement learning~(RL) has become an important approach for improving the capabilities of large language models~(LLMs). In particular, reinforcement learning from verifiable rewards~(RLVR) has emerged as a promising paradigm for reasoning tasks. However, existing RL-based training still remains only a rough approximation to human learning. Human learners leverage both external and internal experience to guide exploration and gradually internalize useful trajectories into stable knowledge. Motivated by this gap, we ask: how can LLMs better utilize and internalize experience during RLVR training? To answer this question, we propose Dual Guidance Optimization~(DGO), a unified framework that leverages external and internal experience to improve training effectiveness. Specifically, DGO first constructs an experience bank from previously explored trajectories. The policy then performs exploration under the joint guidance of the experience bank and the model's internal knowledge. The resulting trajectories are further used to refine the experience bank and optimize model parameters, forming a closed loop of experience utilization and internalization. Experiments show that DGO consistently outperforms baseline methods, suggesting that better utilization and internalization of experience lead to more effective reasoning.
Figures & tables
Figure 1: Comparison of DGO and baseline methods on six in-domain and out-of-domain benchmarks using Qwen3-4B/8B/14B-Base.
Method
Optimization Paradigm
Enhanced Capability
Non-parameter
Parameter
Utilization
Internalization
SFT
✗
✓
✗
✓
GRPO
✗
✓
✗
✓
On-Policy Self-Distillation
✓
✓
✗
✓
Skill-Augmented RL
✓
✓
✓
✗
DGO (Ours)
✓
✓
✓
✓
Table 1: Comparison of representative methods in terms of optimization form and their primary emphasis on experience utilization and internalization. DGO integrates parametric and non-parametric optimization within a unified framework that considers both aspects.
Figure 2: Our framework consists of three components: 1) Experience Construction, 2) Joint Trajectory–Policy Refinement, and 3) Experience Internalization, forming a closed-loop process that enhances experience utilization and internalization. I denotes the training iteration, and the final model is the RL policy from the last iteration.
Model
Method
In-domain (%)
Out-of-domain (%)
Avg
AIME24
AIME25
HMMT25
MATH500
GPQA-D
HLE
Qwen3-4B-Base
EGI
1.46
1.46
0.00
23.52
12.12
1.31
6.64
SFT
5.83
5.62
1.04
31.52
22.92
2.56
11.58
GRPO
26.88
22.50
8.75
72.75
42.55
2.73
29.36
DAPO
26.04
22.29
9.17
71.35
41.79
2.34
28.83
DGO (zero)
27.29
23.54
11.67
72.60
43.56
3.37
30.34
Table 2: Performance comparison of our method with other approaches across three model scales on both in-domain and out-of-domain benchmarks. Bold indicates the best result within each model, and underlined indicates the second-best. DGO (zero) and DGO (TTS) denote intrinsic inference and test-time scaling, respectively.
Figure 3: Accuracy of Qwen3-8B-Base across successive TTS rounds on AIME24 and AIME25, compared with DGO and other baseline methods.
Method
In-domain (%)
Out-of-domain (%)
Avg
AIME24
AIME25
HMMT25
MATH500
GPQA-D
HLE
DGO (zero)
27.08
21.04
9.17
70.08
40.47
2.99
28.47
w/o EA
19.38
17.50
7.71
68.08
38.64
3.24
25.76
w/o ER
19.58
18.75
7.50
68.53
38.57
2.90
25.97
w/o TR
22.08
17.50
6.04
66.65
39.90
2.47
25.77
w/o PD
23.75
20.83
7.71
67.73
38.26
2.79
26.84
Table 3: Ablation study on Qwen-4B-Base. EA , ER , TR , and PD denote Experience Annealing , Experience Renewal , Trajectory Rephrasing , and Policy Distillation respectively. Experiments are conducted between iteration 1 and 2.
Figure 4: Accuracy curves of two models on AIME25 across training iterations.
Figure 5: Experience utilization under three settings on AIME24. DGO benefits more from relevant experience and is more robust to irrelevant experience. Numbers denote relative improvement over zero experience.
Figure 6: t-SNE visualization of generated trajectories on AIME24 for the DGO- and GRPO-trained models.
Figure 7: Comparison of the DGO- and GRPO-trained models on AIME24 in terms of Pass@K Accuracy.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Hyper-parameter
Value
Learning Rate
1e-6
Total Steps
200
Batch Size
128
Mini Batch Size
32
KL Loss Coefficient
0.0
Clip Higher
(0.28, 0.2)
Appendix
Table 4: RL Training hyper-parameters.
Hyper-parameter
Value
Learning Rate
1e-6
Total Epochs
4
Batch Size
128 for 8B/14B; 64 for 4B
Maximum Length
6144
Appendix
Table 5: SFT Training hyper-parameters.
Hyper-parameter
Value
Learning Rate
1e-5
Total Epochs
4
Batch Size
128
Maximum Length
16384
Appendix
Table 6: Experience Generator Training hyper-parameters.
### You are an expert knowledge extractor specializing in converting problem-solving trajectories into abstract, reusable reasoning rules that preserve crucial mathematical specificity and improve future problem solving efficiency. Your task is to analyze a given problem and a candidate solution trajectory. Based on logical consistency, mathematical validity, and alignment with the problem requirements, judge whether the solution is likely correct or incorrect, and then extract one concise general rule as reusable experience. The solution may be correct or incorrect. You must infer this by critically examining the reasoning and results. #### Instructions: 1. Input Analysis: problem: The original problem statement. solution: A full solution trajectory attempting to solve the problem. 2. Experience Extraction Guidelines: - Self-judgment: Evaluate whether the solution is likely correct by checking: (a) whether the final answer satisfies the problem conditions, and (b) whether each reasoning step is logically and mathematically valid. - If the solution appears correct: Do NOT merely restate why it works. Instead, focus on extracting optimization experience, such as: • how the solution could be made shorter or more direct, • which intermediate steps, case splits, or calculations are redundant, • which key insight could be used earlier to avoid detours, • how to reformulate the approach into a cleaner standard method. Summarize this as a rule that helps future solvers reach the answer faster and with less complexity. - If the solution appears incorrect: Identify the most critical error, unjustified assumption, or logical gap. Extract an avoiding-errors experience that explains: • what warning sign to look for, and • what should be done instead to prevent this type of mistake. - One Output: Generate exactly one rule that reflects either: (a) an optimization rule distilled from a correct but improvable solution, or (b) an avoiding-errors rule distilled from an incorrect solution. - Abstraction with Specificity: Generalize the rule while retaining essential mathematical structures, formulas, constraints, or thresholds that make the experience precise and actionable. Use LaTeX for mathematical expressions where appropriate. 3. Rule Format Requirement: Output exactly one rule in the following structure, with no extra text: WHEN <Broad domain or phase>. IF <Specific condition, scenario, or inefficiency / potential error>. THEN <Action to take or avoid, emphasizing simplification, direct strategy, or error prevention, with key formulas if needed>. # Input: # problem:\n{problem}\n # solution:\n{solution}\n # Output: WHEN <Broad domain or phase>. IF <Specific condition, scenario, or inefficiency / potential error>. THEN <Action to take or avoid>.
Appendix
Table 7: Prompt template for experience extraction.
Please reason step by step, and put your final answer within \boxed{{}}. {problem}
Appendix
Table 8: Prompt template for reasoning without experience.
Please reason step by step, and put your final answer within \boxed{{}}. You will be given some reusable problem-solving experiences, and you can refer to them during the reasoning. [Experiences] {experiences} [Problem] {problem}
Appendix
Table 9: Prompt template for reasoning with experience.
You are an expert mathematician. You are given a math problem, some background [Experience Content], and a [Reference Solution] which may have used those experiences explicitly. Your task is to re-author a "Gold Standard" solution. This solution must be entirely self-contained, as if you solved the problem using only your innate mathematical intuition. Strict requirements: 1. Total De-Reference: You MUST NOT mention, quote, or even subtly refer to the [Experience Content] provided. Do not use phrases like "By logic of...", "Following the principle of...", or any terminology that implies you are looking at an external guide. The reader should have no idea that any "Experience Content" was provided to you. 2. Invisible Integration: Instead of "using" the experience, you must "be" the expert who already knows it. If the experience suggests a shortcut or a formula, justify its use through first-principles reasoning within the flow of the solution. For example, do not say "Using the formula for X," say "Since the problem requires [reason], we can derive that [logical step]..." 3. Silent Audit & Correction: Treat the [Reference Solution] as a potentially flawed scratchpad. Verify every calculation and logical leap. If it is wrong, fix it silently. Your final output must be a clean, error-free, and authoritative proof. Do NOT mention that the reference was wrong. 4. Strategic Fidelity: Follow the general strategy of the [Reference Solution] (e.g., if it uses energy conservation, you should too), but execute it with perfect precision and without any meta-talk or conversational fillers. 5. Final Answer Consistency: The final numerical/symbolic answer must exactly match the [Reference Solution] and be enclosed in \boxed{{}}. [Experience Content] {experiences} [Problem] {problem} [Reference Solution] {solution} [Your Solution]
Appendix
Table 10: Prompt template for solution rewriting.
{ "problem": "Let b≥2 be an integer. Call a positive integer nb-eautiful if it has exactly two digits when expressed in base b and these two digits sum to n. For example, 81 is 13-eautiful because 81=6313 and 6+3=81. Find the least integer b≥2 for which there are more than ten b-eautiful integers.", "experiences": [ "WHEN solving problems about numbers with given digit sums and square roots in a base representation.\nIF setting up an equation from the condition that the sum of digits equals the square root of the number,\nTHEN ensure the correct conversion from base b to decimal: if the two-digit representation is adb with digits a (tens) and d (units), then n=ab+d, and the condition is a+d=ab+d. Square both sides to get (a+d)ˆ2=ab+d, then rearrange to analyze integer solutions, noting digit constraints 1≤a≤b−1, 0≤d≤b−1. Avoid incorrectly writing n=b(a+1) which omits the units digit.", "WHEN solving for numbers with digit sum equal to the square root in a given base.\nIF the condition leads to n=kˆ2 and d1=b−1k(k−1) where d1 is the leading digit in base b.\nTHEN use the divisibility condition (b−1)∣k(k−1) to find all k for each b by factoring b−1, which is more efficient than brute-forcing over the range of k.", "WHEN solving for the smallest base b with a specified count of numbers satisfying a digit-sum and square root condition.\nIF you have derived the relation d1=b−1S(S−1) with S=d1+d2, and b−1≤Sˆ2<bˆ2.\nTHEN iterate over integer S≥2, for each S compute all divisors d of S(S−1), set b=d+1, and check if b−1≤Sˆ2<bˆ2; if so, record (b,S). Then collect all b and their counts to efficiently find the smallest b with the required count.", "WHEN solving problems about numbers with digit conditions in a given base.\nIF the condition leads to an equation of the form d1=b−1kˆ2−k with k=d1+d2 and d1 must be an integer between 1 and b−1,\nTHEN systematically check all integers k from ⌈b−1⌉ to b−1, and verify that (b−1)∣k(k−1); this ensures a complete count without missing valid solutions." ] }
Appendix
Table 11: An example of retrieved experiences associated with a problem.
[Problem] Let \omega\neq 1 be a 13th root of unity. Find the remainder when ∏k=0ˆ12(2−2ωˆk+ωˆ2k) is divided by 1000. [Experiences] [1] WHEN solving set-counting problems with multiple sets and given counts of exactly k-set intersections. IF the total number of elements and the sizes of each set are known, and the goal is to find the intersection of all sets. THEN compute the total number of set memberships T=∑∣Ai∣ and equate it to ∑kk⋅nk, where nk is the number of elements in exactly k sets, leading to a linear equation for the unknown intersection count.
Appendix
Table 12: Comparison of two trajectory excerpts under the same retrieved experiences. Both trajectories receive the same irrelevant experiences, but the left one stays problem-driven while the right one makes an invalid polynomial substitution.
[Problem] There exist real numbers x and y, both greater than 1, such that logx(yˆx)=logy(xˆ4y)=10. Find xy.
Appendix
Table 13: Comparison of two trajectory excerpts without retrieved experience. The DGO-trained model directly invokes a compact logarithmic relation and reaches the correct result, while the GRPO-trained model fails to internalize the same simplification pattern and falls into unstable symbolic manipulation.
[Problem] Let b≥2 be an integer. Call a positive integer nb-eautiful if it has exactly two digits when expressed in base b and these two digits sum to n. For example, 81 is 13-eautiful because 81=6313 and 6+3=81. Find the least integer b≥2 for which there are more than ten b-eautiful integers. [Round 0] WHEN solving for numbers with digit sum equal to the square root in a given base. IF the condition leads to n=kˆ2 and d1=b−1k(k−1) where d1 is the leading digit in base b. THEN use the divisibility condition (b−1)∣k(k−1) to find all k for each b by factoring b−1, which is more efficient than brute-forcing over the range of k. [Round 1] WHEN solving problems about digit conditions in a given base. IF the condition is a+d=ab+d with a the leading digit, 1≤a≤b−1, 0≤d≤b−1. THEN set k=a+d so n=kˆ2, derive a=b−1k(k−1), and find b such that b−1∣k(k−1) for multiple k, ensuring a and d=k−a satisfy digit bounds, to avoid inefficient brute-force enumeration. [Round 2] WHEN solving problems about numbers with two digits in base b satisfying a digit-sum condition involving n. IF you have derived that n=ab+d with a+d=k=n and obtained the relation a=b−1kˆ2−k. THEN note that k(k−1) must be divisible by b−1, which is equivalent to k≡0 or 1(modb−1); use this congruence to restrict k to two arithmetic progressions within the range ⌈b⌉≤k≤b−1, drastically reducing the number of candidates to check.
Appendix
Table 14: The problem and one representative extracted experience from each round.
Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for improving the reasoning capabilities of large language models (LLMs). However, existing RLVR methods typically rely on on-policy optimization from scratch, resulting in high sampling costs and inefficient utilization of accumulated experience. As model capabilities and policy behaviors evolve during training, recent attempts to reuse experience via fixed reasoning trajectories further suffer from policy mismatch. Motivated by these limitations, we argue that experience in RLVR should not be reused as fixed reasoning trajectories, but instead expressed in a policy-adaptive manner. In this work, we propose Experience-Augmented Policy Optimization (EAPO), which leverages a prior RL-optimized policy as an action-level experience prior and selectively injects experience at critical decision points during rollout. To ensure stable and unbiased learning from experience-augmented rollouts, EAPO further incorporates an adapted importance sampling scheme. Experiments on using Qwen-2.5-math 7b and Qwen-3-8B on five different benchmarks demonstrate that EAPO consistently improves reasoning performance over state-of-the-art RLVR methods.
Jinda Lu, Kexin Huang, Junkang Wu +7
University of Science and Technology of China · 2Peking University · 3Dartmouth College +1
Reinforcement Learning with Verifiable Rewards (RLVR) has become a widely adopted technique for enhancing the reasoning ability of Large Language Models (LLMs). However, the effectiveness of RLVR strongly depends on the capability of base models. This issue arises because it requires the model to have sufficient capability to perform high-quality exploration, which involves both effectiveness and diversity. Unfortunately, existing methods address this issue by imitating expert trajectories, which improve effectiveness but neglect diversity. To address this, we argue that the expert only needs to provide guidance only at critical decision points rather than the entire reasoning path. Based on this insight, we propose MENTOR: Mixed-policy Expert Navigation for Token-level Optimization of Reasoning, a framework that provides expert guidance only at critical decision points to perform effective and diverse exploration in RLVR. Extensive experiments show that MENTOR enables models capture the essence of expert strategies rather than surface imitation, thereby performing high-quality exploration and achieving superior overall performance. Our code is available online.
Zishang Jiang, Jinyi Han, Tingyun Li +7
School of Data Science, Fudan University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University +1
Reinforcement Learning (RL) with rubric-based rewards has recently shown remarkable progress in enhancing general reasoning capabilities of Large Language Models (LLMs), yet still suffers from ineffective exploration confined to curent policy distribution. In fact, RL optimization can be viewed as steering the policy toward an ideal distribution that maximizes the rewards, while effective exploration should align efforts with desired target. Leveraging this insight, we propose HeRL, a Hindsight experience guided Reinforcement Learning framework to bootstrap effective exploration by explicitly telling LLMs the desired behaviors specified in rewards. Concretely, HeRL treats failed trajectories along with their unmet rubrics as hindsight experience, which serves as in-context guidance for the policy to explore desired responses beyond its current distribution. Additionally, we introduce a bonus reward to incentivize responses with greater potential for improvement under such guidance. HeRL facilitates effective learning from desired high quality samples without repeated trial-and-error from scratch, yielding a more accurate estimation of the expected gradient theoretically. Extensive experiments across various benchmarks demonstrate that HeRL achieves superior performance gains over baselines, and can further benefit from experience guided self-improvement at test time. Our code is available at https://github.com/sikelifei/HeRL.
Wenjian Zhang, Kongcheng Zhang, Jiaxin Qi +2
Dalian University of Technology · Zhejiang University · Chinese Academy of Sciences