Towards Effective Experiential Learning: Dual Guidance for Utilization and Internalization

Fei Bai; Zhipeng Chen; Chuan Hao; Ming Yang; Ran Tao; Bryan Dai; Wayne Xin Zhao; Jian Yang

cs.LGMar 25, 2026

Towards Effective Experiential Learning: Dual Guidance for Utilization and Internalization

Authors: Fei Bai, Zhipeng Chen, Chuan Hao, Ming Yang, Ran Tao, Bryan Dai, Wayne Xin Zhao, Jian Yang,

Organizations: Gaoling School of Artificial Intelligence, Renmin University of China · IQuest Research · Beihang University

Abstract

Recently, reinforcement learning~(RL) has become an important approach for improving the capabilities of large language models~(LLMs). In particular, reinforcement learning from verifiable rewards~(RLVR) has emerged as a promising paradigm for reasoning tasks. However, existing RL-based training still remains only a rough approximation to human learning. Human learners leverage both external and internal experience to guide exploration and gradually internalize useful trajectories into stable knowledge. Motivated by this gap, we ask: how can LLMs better utilize and internalize experience during RLVR training? To answer this question, we propose Dual Guidance Optimization~(DGO), a unified framework that leverages external and internal experience to improve training effectiveness. Specifically, DGO first constructs an experience bank from previously explored trajectories. The policy then performs exploration under the joint guidance of the experience bank and the model's internal knowledge. The resulting trajectories are further used to refine the experience bank and optimize model parameters, forming a closed loop of experience utilization and internalization. Experiments show that DGO consistently outperforms baseline methods, suggesting that better utilization and internalization of experience lead to more effective reasoning.

Figures & tables

Figure 1: Comparison of DGO and baseline methods on six in-domain and out-of-domain benchmarks using Qwen3-4B/8B/14B-Base.

Method	Optimization Paradigm		Enhanced Capability
Method	Non-parameter	Parameter	Utilization	Internalization
SFT	✗	✓	✗	✓
GRPO	✗	✓	✗	✓
On-Policy Self-Distillation	✓	✓	✗	✓
Skill-Augmented RL	✓	✓	✓	✗
DGO (Ours)	✓	✓	✓	✓

Table 1: Comparison of representative methods in terms of optimization form and their primary emphasis on experience utilization and internalization. DGO integrates parametric and non-parametric optimization within a unified framework that considers both aspects.

Figure 2: Our framework consists of three components: 1) Experience Construction, 2) Joint Trajectory–Policy Refinement, and 3) Experience Internalization, forming a closed-loop process that enhances experience utilization and internalization. $I$ denotes the training iteration, and the final model is the RL policy from the last iteration.

Model	Method	In-domain (%)				Out-of-domain (%)		Avg
Model	Method	AIME24	AIME25	HMMT25	MATH500	GPQA-D	HLE	Avg
Qwen3-4B-Base	EGI	1.46	1.46	0.00	23.52	12.12	1.31	6.64
	SFT	5.83	5.62	1.04	31.52	22.92	2.56	11.58
	GRPO	26.88	22.50	8.75	72.75	42.55	2.73	29.36
	DAPO	26.04	22.29	9.17	71.35	41.79	2.34	28.83
	DGO (zero)	27.29	23.54	11.67	72.60	43.56	3.37	30.34

Table 2: Performance comparison of our method with other approaches across three model scales on both in-domain and out-of-domain benchmarks. Bold indicates the best result within each model, and underlined indicates the second-best. DGO (zero) and DGO (TTS) denote intrinsic inference and test-time scaling, respectively.

Panel of Figure 3 — Figure 3: Accuracy of Qwen3-8B-Base across successive TTS rounds on AIME24 and AIME25, compared with DGO and other baseline methods.

Method	In-domain (%)				Out-of-domain (%)		Avg
Method	AIME24	AIME25	HMMT25	MATH500	GPQA-D	HLE	Avg
DGO (zero)	27.08	21.04	9.17	70.08	40.47	2.99	28.47
w/o EA	19.38	17.50	7.71	68.08	38.64	3.24	25.76
w/o ER	19.58	18.75	7.50	68.53	38.57	2.90	25.97
w/o TR	22.08	17.50	6.04	66.65	39.90	2.47	25.77
w/o PD	23.75	20.83	7.71	67.73	38.26	2.79	26.84

Table 3: Ablation study on Qwen-4B-Base. EA , ER , TR , and PD denote Experience Annealing , Experience Renewal , Trajectory Rephrasing , and Policy Distillation respectively. Experiments are conducted between iteration 1 and 2.

Panel of Figure 4 — Figure 4: Accuracy curves of two models on AIME25 across training iterations.

Panel of Figure 5 — Figure 5: Experience utilization under three settings on AIME24. DGO benefits more from relevant experience and is more robust to irrelevant experience. Numbers denote relative improvement over zero experience.

Panel of Figure 6 — Figure 6: t-SNE visualization of generated trajectories on AIME24 for the DGO- and GRPO-trained models.

Panel of Figure 7 — Figure 7: Comparison of the DGO- and GRPO-trained models on AIME24 in terms of Pass@K Accuracy.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Hyper-parameter	Value
Learning Rate	1e-6
Total Steps	200
Batch Size	128
Mini Batch Size	32
KL Loss Coefficient	0.0
Clip Higher	(0.28, 0.2)

Appendix

Table 4: RL Training hyper-parameters.

Hyper-parameter	Value
Learning Rate	1e-6
Total Epochs	4
Batch Size	128 for 8B/14B; 64 for 4B
Maximum Length	6144

Appendix

Table 5: SFT Training hyper-parameters.

Hyper-parameter	Value
Learning Rate	1e-5
Total Epochs	4
Batch Size	128
Maximum Length	16384

Appendix

Table 6: Experience Generator Training hyper-parameters.

### You are an expert knowledge extractor specializing in converting problem-solving trajectories into abstract, reusable reasoning rules that preserve crucial mathematical specificity and improve future problem solving efficiency. Your task is to analyze a given problem and a candidate solution trajectory. Based on logical consistency, mathematical validity, and alignment with the problem requirements, judge whether the solution is likely correct or incorrect, and then extract one concise general rule as reusable experience. The solution may be correct or incorrect. You must infer this by critically examining the reasoning and results. #### Instructions: 1. Input Analysis: problem: The original problem statement. solution: A full solution trajectory attempting to solve the problem. 2. Experience Extraction Guidelines: - Self-judgment: Evaluate whether the solution is likely correct by checking: (a) whether the final answer satisfies the problem conditions, and (b) whether each reasoning step is logically and mathematically valid. - If the solution appears correct: Do NOT merely restate why it works. Instead, focus on extracting optimization experience, such as: • how the solution could be made shorter or more direct, • which intermediate steps, case splits, or calculations are redundant, • which key insight could be used earlier to avoid detours, • how to reformulate the approach into a cleaner standard method. Summarize this as a rule that helps future solvers reach the answer faster and with less complexity. - If the solution appears incorrect: Identify the most critical error, unjustified assumption, or logical gap. Extract an avoiding-errors experience that explains: • what warning sign to look for, and • what should be done instead to prevent this type of mistake. - One Output: Generate exactly one rule that reflects either: (a) an optimization rule distilled from a correct but improvable solution, or (b) an avoiding-errors rule distilled from an incorrect solution. - Abstraction with Specificity: Generalize the rule while retaining essential mathematical structures, formulas, constraints, or thresholds that make the experience precise and actionable. Use LaTeX for mathematical expressions where appropriate. 3. Rule Format Requirement: Output exactly one rule in the following structure, with no extra text: WHEN <Broad domain or phase>. IF <Specific condition, scenario, or inefficiency / potential error>. THEN <Action to take or avoid, emphasizing simplification, direct strategy, or error prevention, with key formulas if needed>. # Input: # problem:\n{problem}\n # solution:\n{solution}\n # Output: WHEN <Broad domain or phase>. IF <Specific condition, scenario, or inefficiency / potential error>. THEN <Action to take or avoid>.

Appendix

Table 7: Prompt template for experience extraction.

Please reason step by step, and put your final answer within \boxed{{}}. {problem}

Appendix

Table 8: Prompt template for reasoning without experience.

Please reason step by step, and put your final answer within \boxed{{}}. You will be given some reusable problem-solving experiences, and you can refer to them during the reasoning. [Experiences] {experiences} [Problem] {problem}

Appendix

Table 9: Prompt template for reasoning with experience.

You are an expert mathematician. You are given a math problem, some background [Experience Content], and a [Reference Solution] which may have used those experiences explicitly. Your task is to re-author a "Gold Standard" solution. This solution must be entirely self-contained, as if you solved the problem using only your innate mathematical intuition. Strict requirements: 1. Total De-Reference: You MUST NOT mention, quote, or even subtly refer to the [Experience Content] provided. Do not use phrases like "By logic of...", "Following the principle of...", or any terminology that implies you are looking at an external guide. The reader should have no idea that any "Experience Content" was provided to you. 2. Invisible Integration: Instead of "using" the experience, you must "be" the expert who already knows it. If the experience suggests a shortcut or a formula, justify its use through first-principles reasoning within the flow of the solution. For example, do not say "Using the formula for X," say "Since the problem requires [reason], we can derive that [logical step]..." 3. Silent Audit & Correction: Treat the [Reference Solution] as a potentially flawed scratchpad. Verify every calculation and logical leap. If it is wrong, fix it silently. Your final output must be a clean, error-free, and authoritative proof. Do NOT mention that the reference was wrong. 4. Strategic Fidelity: Follow the general strategy of the [Reference Solution] (e.g., if it uses energy conservation, you should too), but execute it with perfect precision and without any meta-talk or conversational fillers. 5. Final Answer Consistency: The final numerical/symbolic answer must exactly match the [Reference Solution] and be enclosed in \boxed{{}}. [Experience Content] {experiences} [Problem] {problem} [Reference Solution] {solution} [Your Solution]

Appendix

Table 10: Prompt template for solution rewriting.

{ "problem": "Let

b\ge 2

be an integer. Call a positive integer

n

b\text-\textit{eautiful}

if it has exactly two digits when expressed in base

b

and these two digits sum to

\sqrt n

. For example,

81

13\text-\textit{eautiful}

because

81 = \underline{6}\ \underline{3}_{13}

and

6 + 3 = \sqrt{81}

. Find the least integer

b\ge 2

for which there are more than ten

b\text-\textit{eautiful}

integers.", "experiences": [ "WHEN solving problems about numbers with given digit sums and square roots in a base representation.\nIF setting up an equation from the condition that the sum of digits equals the square root of the number,\nTHEN ensure the correct conversion from base

b

to decimal: if the two-digit representation is

\overline{ad}_b

with digits

a

(tens) and

d

(units), then

n = ab + d

, and the condition is

a + d = \sqrt{ab + d}

. Square both sides to get

(a+d)ˆ2 = ab + d

, then rearrange to analyze integer solutions, noting digit constraints

1 \le a \le b-1

0 \le d \le b-1

. Avoid incorrectly writing

n = b(a+1)

which omits the units digit.", "WHEN solving for numbers with digit sum equal to the square root in a given base.\nIF the condition leads to

n = kˆ2

and

d_1 = \frac{k(k-1)}{b-1}

where

d_1

is the leading digit in base

b

.\nTHEN use the divisibility condition

(b-1) \mid k(k-1)

to find all

k

for each

b

by factoring

b-1

, which is more efficient than brute-forcing over the range of

k

.", "WHEN solving for the smallest base

b

with a specified count of numbers satisfying a digit-sum and square root condition.\nIF you have derived the relation

d_1 = \frac{S(S-1)}{b-1}

with

S = d_1 + d_2

, and

b-1 \le Sˆ2 < bˆ2

.\nTHEN iterate over integer

S \ge 2

, for each

S

compute all divisors

d

S(S-1)

, set

b = d+1

, and check if

b-1 \le Sˆ2 < bˆ2

; if so, record

(b,S)

. Then collect all

b

and their counts to efficiently find the smallest

b

with the required count.", "WHEN solving problems about numbers with digit conditions in a given base.\nIF the condition leads to an equation of the form

d_1 = \frac{kˆ2 - k}{b-1}

with

k = d_1 + d_2

and

d_1

must be an integer between

1

and

b-1

,\nTHEN systematically check all integers

k

from

\lceil \sqrt{b-1} \rceil

b-1

, and verify that

(b-1) \mid k(k-1)

; this ensures a complete count without missing valid solutions." ] }

Appendix

Table 11: An example of retrieved experiences associated with a problem.

[Problem] Let \omega\neq 1 be a 13th root of unity. Find the remainder when

\prod_{k=0}ˆ{12}(2-2\omegaˆk+\omegaˆ{2k})

is divided by 1000. [Experiences] [1] WHEN solving set-counting problems with multiple sets and given counts of exactly k-set intersections. IF the total number of elements and the sizes of each set are known, and the goal is to find the intersection of all sets. THEN compute the total number of set memberships

T = \sum |A_i|

and equate it to

\sum_k k \cdot n_k

, where

n_k

is the number of elements in exactly

k

sets, leading to a linear equation for the unknown intersection count.

Appendix

Table 12: Comparison of two trajectory excerpts under the same retrieved experiences. Both trajectories receive the same irrelevant experiences, but the left one stays problem-driven while the right one makes an invalid polynomial substitution.

[Problem] There exist real numbers

x

and

y

, both greater than 1, such that

\log_x\left(yˆx\right)=\log_y\left(xˆ{4y}\right)=10.

Find

xy

Appendix

Table 13: Comparison of two trajectory excerpts without retrieved experience. The DGO-trained model directly invokes a compact logarithmic relation and reaches the correct result, while the GRPO-trained model fails to internalize the same simplification pattern and falls into unstable symbolic manipulation.

[Problem] Let

b\ge 2

be an integer. Call a positive integer

n

b\text-\textit{eautiful}

if it has exactly two digits when expressed in base

b

and these two digits sum to

\sqrt n

. For example,

81

13\text-\textit{eautiful}

because

81 = \underline{6}\ \underline{3}_{13}

and

6 + 3 = \sqrt{81}

. Find the least integer

b\ge 2

for which there are more than ten

b\text-\textit{eautiful}

integers. [Round 0] WHEN solving for numbers with digit sum equal to the square root in a given base. IF the condition leads to

n = kˆ2

and

d_1 = \frac{k(k-1)}{b-1}

where

d_1

is the leading digit in base

b

. THEN use the divisibility condition

(b-1) \mid k(k-1)

to find all

k

for each

b

by factoring

b-1

, which is more efficient than brute-forcing over the range of

k

. [Round 1] WHEN solving problems about digit conditions in a given base. IF the condition is

a+d = \sqrt{ab+d}

with

a

the leading digit,

1 \le a \le b-1

0 \le d \le b-1

. THEN set

k = a+d

n = kˆ2

, derive

a = \frac{k(k-1)}{b-1}

, and find

b

such that

b-1 \mid k(k-1)

for multiple

k

, ensuring

a

and

d = k-a

satisfy digit bounds, to avoid inefficient brute-force enumeration. [Round 2] WHEN solving problems about numbers with two digits in base

b

satisfying a digit-sum condition involving

\sqrt{n}

. IF you have derived that

n = a b + d

with

a+d = k = \sqrt{n}

and obtained the relation

a = \frac{kˆ2 - k}{b-1}

. THEN note that

k(k-1)

must be divisible by

b-1

, which is equivalent to

k \equiv 0

1 \pmod{b-1}

; use this congruence to restrict

k

to two arithmetic progressions within the range

\lceil \sqrt{b} \rceil \le k \le b-1

, drastically reducing the number of candidates to check.

Appendix

Table 14: The problem and one representative extracted experience from each round.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

Experience Augmented Policy Optimization for LLM Reasoning
Jun 29, 2026Jinda Lu, Kexin Huang, Junkang Wu +7Reinforcement Learning RL for Language Model Reasoning
University of Science and Technology of China · 2Peking University · 3Dartmouth College +1
Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
Oct 5, 2025Zishang Jiang, Jinyi Han, Tingyun Li +7RL for Language Model Reasoning LLM-Guided RL
School of Data Science, Fudan University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University +1
Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMs
Mar 20, 2026Wenjian Zhang, Kongcheng Zhang, Jiaxin Qi +2Hindsight Experience Replay RL for Language Model Reasoning
Dalian University of Technology · Zhejiang University · Chinese Academy of Sciences

Jun 29, 2026J/K move · Enter open · S save

Experience Augmented Policy Optimization for LLM Reasoning

Jinda Lu, Kexin Huang, Junkang Wu, Shuo Yang+6

University of Science and Technology of China · 2Peking University · 3Dartmouth College+1

Reinforcement Learning RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for improving the reasoning capabilities of large language models (LLMs). However, existing RLVR methods typically rely on on-policy optimization from scratch, resulting in high sampling costs and inefficient utilization of accumulated experience. As model capabilities and policy behaviors evolve during training, recent attempts to reuse experience via fixed reasoning trajectories further suffer from policy mismatch. Motivated by these limitations, we argue that experience in RLVR should not be reused as fixed reasoning trajectories, but instead expressed in a policy-adaptive manner. In this work, we propose Experience-Augmented Policy Optimization (EAPO), which leverages a prior RL-optimized policy as an action-level experience prior and selectively injects experience at critical decision points during rollout. To ensure stable and unbiased learning from experience-augmented rollouts, EAPO further incorporates an adapted importance sampling scheme. Experiments on using Qwen-2.5-math 7b and Qwen-3-8B on five different benchmarks demonstrate that EAPO consistently improves reasoning performance over state-of-the-art RLVR methods.

Abstract

RL for Language Model Reasoning

Explore similar work

Experience Augmented Policy Optimization for LLM Reasoning

Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs

Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMs