Self-Evolving Agents through Privileged Retrospective Distillation

Yuhang Zhou; Yifan Wu; Bo Peng; Mingyi Wang; Jiayi Liu; Xiangjun Fan; Lizhu Zhang; Zhuokai Zhao

cs.LGOct 20, 2026

Self-Evolving Agents through Privileged Retrospective Distillation

Authors: Yuhang Zhou, Yifan Wu, Bo Peng, Mingyi Wang, Jiayi Liu, Xiangjun Fan, Lizhu Zhang, Zhuokai Zhao

Organizations: Meta AI

Abstract

Self-evolving agents aim to improve from their own experience rather than remain fixed after deployment, but naturally collected trajectories often lack ground-truth solutions or verifier-derived rewards. We introduce Retrospective Self-Distillation (RSD), a multi-round framework that turns such experience into dense supervision using privileged retrospective context. For each on-policy trajectory, the same underlying model acts as both student and teacher: the teacher re-scores the student's realized actions while conditioning on cross-task precedents and lessons, together with turn-level reflection and future hindsight from the completed trajectory. The resulting teacher distribution is distilled back into the policy through on-policy self-distillation. Across rounds, the experience bank grows and the retrospective context is refreshed, allowing the model, its data, and its supervision to co-evolve. We further introduce frontier-guided context-policy evolution, where a stronger frontier model proposes constrained edits to how privileged context is constructed. Across ALFWorld, ScienceWorld, and SearchQA with Qwen3-8B and Qwen3-14B, RSD improves performance by up to 59.7% relative to the base model, while frontier-guided context-policy evolution adds up to a further 44.7% relative improvement over a fixed context policy. These results show that naturally accumulated experience can be converted into persistent model capability without ground-truth trajectories or verifier-derived rewards for the core policy update.

Figures & tables

Figure 1: Overview of RSD . At self-evolution round $r$ , the current policy $\pi^{(r)}$ first collects on-policy interaction experience, which is incorporated into the accumulated experience bank. Privileged retrospective context is then constructed from past experience: precedents and lessons, together with reflection and future hindsight from the completed current trajectory. The same underlying model retrospectively re-scores the student’s realized actions under this privileged view, and the resulting distribution is distilled back into the student to obtain $\pi^{(r+1)}$ . Repeating this process jointly evolves the policy, accumulated experience, and retrospective supervision. In the stronger-access extension, a frontier model additionally adapts the context-construction policy $\phi^{(r)}$ across rounds.

	ALFWorld (SR)		ScienceWorld (SR)		SearchQA (EM)
Method	Qwen3-8B	Qwen3-14B	Qwen3-8B	Qwen3-14B	Qwen3-8B	Qwen3-14B
Base	62.9	69.3	15.6	12.9	35.4	35.4
RFT	66.9	71.0	12.7	14.3	35.4	36.3
GT-SFT	34.1	42.7	11.6	13.4	34.7	35.2
STaR	65.3	77.8	12.7	13.6	34.4	37.6
BoN-16+SFT	71.6	77.7	20.8	20.4	29.7	35.5

Table 1: Main results across ALFWorld, ScienceWorld, and SearchQA. For ALFWorld, SR is averaged over the valid-seen and valid-unseen splits, while for SearchQA, EM is averaged across the evaluated sub-datasets. Results are averaged over three seeds. Bold denotes the best result and underline denotes the second-best result in each setting.

Variant	ALFWorld	ScienceWorld
Full	76.6	23.4
$-$ Precedents	75.0	15.1
$-$ Lessons	71.6	22.7
$-$ Reflection	71.4	21.8
$-$ Future	71.4	17.6

Table 2: Ablation of privileged-context components on 8B. Different tasks rely on different privileged signals: generated hindsight components dominate on ALFWorld, whereas precedents are essential on ScienceWorld.

Figure 2: Performance of fixed, random, and frontier-guided context-policy evolution on ALFWorld and ScienceWorld. The frontier variant uses a frontier model to propose context-policy edits, with candidate selection guided by a no-training privileged-context-at-inference probe. We report success rate over self-evolution rounds for each task and model scale.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

\mathcal{D}^{(r)}\leftarrow\left\{\tau\sim\pi^{(r)}(\cdot\mid x):x\in\mathcal{X}^{(r)}\right\}

Appendix

Algorithm 1 RSD : Self-Evolution through Privileged Retrospective Distillation

	ALFWorld (SR)		ScienceWorld (SR)		SearchQA (EM)
Method	8B	14B	8B	14B	8B	14B
Reflexion@1	66.1	69.5	12.9	13.6	34.8	36.9
Reflexion@2	76.0	76.7	17.8	18.5	37.1	38.1
Reflexion@3	80.7	79.6	20.9	24.1	41.3	43.6
RSD	76.6	79.6	23.4	20.6	37.9	38.7

Appendix

Table 3: Test-time scaling with Reflexion across one, two, and three sequential attempts. RSD is included as a reference and is evaluated under the standard single-attempt protocol.

Figure 3: Internalization of privileged supervision across tasks and model scales. The blue bars show the internalized gain, measured as Evolved(no-C)

-

Base(C), while the orange bars show the remaining harness utility, measured as Evolved(C)

-

Evolved(no-C). Positive blue values indicate that the evolved model without privileged context outperforms the base model equipped with the harness at inference, whereas positive orange values indicate complementary information that remains useful after evolution.

Round Pair	$\Delta\log p$ on Disfavored Actions
ALFWorld $r1\!\rightarrow\!r2$	$-0.384$
ALFWorld $r2\!\rightarrow\!r3$	$-0.063$
ScienceWorld $r1\!\rightarrow\!r2$	$-0.696$
ScienceWorld $r5\!\rightarrow\!r6$	$+0.006$

Appendix

Table 4: Directional uptake of privileged corrections across consecutive evolution rounds. Negative values indicate that the subsequent policy assigns lower probability to realized actions disfavored by the privileged teacher. Detailed analyses are provided in Appendix D.5 .

Panel of Figure 4 — Figure 4: Self-evolution trajectories of Qwen3-8B and Qwen3-14B over ten rounds. Round 0 denotes the corresponding base model, and dashed lines indicate the RFT baseline. Each panel shows a single-seed trajectory; stars denote the best round within the displayed run. ALFWorld reports unseen SR, while the main table reports the mean over seen and unseen splits. Canonical larger-scale evaluations are reported in Table 1 .

Lesson Source	ALFWorld	ScienceWorld
Successful only	69.4	22.5
Failed only	72.6	22.5
Successful + Failed (default)	76.6	23.4

Appendix

Table 5: Ablation on the source of accumulated lessons. We report success rate (%) with Qwen3-8B. The default configuration summarizes both successful and failed trajectories and reports the main-result reference.

Figure 5: Context evolution on ALFWorld-8B. Refreshing privileged context across self-evolution rounds substantially outperforms freezing the round-1 context, with lesson updates providing particularly strong gains.

Method	Probe	SR
Fixed	–	23.4
Random	–	23.2
Frontier (diagnostics)	No	23.9
Frontier + Probe	Yes	27.5

Appendix

Table 6: Probe ablation on ScienceWorld-8B. The diagnostics-only frontier receives the same training digest but does not observe the no-training candidate probe.

Context Edit	Adopted@Peak	Ablation $\Delta$ SR
Precedents $\uparrow$	4/4	$-8.3$
Lessons off	3/4	$-0.7$
Future off	1/4	$-5.8$

Appendix

Table 7: Context-policy edits discovered by the frontier model on ScienceWorld-8B. Adopted@Peak reports how many independent runs contain the edit in their peak-performing configuration. Ablation

\Delta

SR denotes the performance drop when the corresponding component is removed from the canonical context.

Round	Context Before	Proposed Change	SR	Decision
$r_{2}$	Prec.=0, Lessons=on, Future=on	Prec. $0\!\rightarrow\!2$	0.180	Revert
$r_{3}$	Prec.=0, Lessons=on, Future=on	Prec. $0\!\rightarrow\!1$	0.258	Keep
$r_{4}$	Prec.=1, Lessons=on, Future=on	Future $\text{on}\!\rightarrow\!\text{off}$	0.221	Revert
$r_{5}$	Prec.=1, Lessons=on, Future=on	No change	0.216	Hold
$r_{6}$	Prec.=1, Lessons=on, Future=on	Prec. $1\!\rightarrow\!2$	0.243	Revert
$r_{7}$	Prec.=1, Lessons=on, Future=on	No change	0.227	Hold

Appendix

Table 8: Representative trajectory of frontier-guided privileged-context evolution on ScienceWorld (Qwen3-8B). “Context Before” denotes the last accepted configuration. The frontier may either propose a constrained context-policy edit or retain the current configuration when no candidate appears beneficial. SR reports the resulting checkpoint performance at each round; proposed edits are subsequently kept or reverted according to their post-training performance.

Teacher Schedule	R1	R2	R3	R4	R5	R6	R7	R8	R9	R10	Peak	Final
Frozen	65.0	68.2	63.5	65.7	63.5	65.7	62.4	67.5	67.1	67.9	68.2	67.9
Fully evolving	64.6	71.6	78.5	79.2	78.1	79.6	80.3	76.2	39.4	0.0	80.3	0.0

Appendix

Table 9: ALFWorld teacher-schedule ablation with Qwen3-8B under the fixed canonical context policy. We report mean success rate (%) over seen and unseen splits for each self-evolution round. Results are intended as a controlled diagnostic of teacher dynamics.

Precedent Count	ALFWorld	ScienceWorld
$n=1$	75.6	24.9
$n=2$ (default)	76.6	23.4
$n=4$	78.8	20.3

Appendix

Table 10: Sensitivity to the number of retrieved precedents. We report success rate (%) for Qwen3-8B.

n=2

denotes the default configuration and reports the default main-result reference.

Field	Prompt Content
System prompt	You are an embodied agent solving a household task in a text-based simulator (AlfWorld). You receive an Observation describing what you can see and a list of Admissible commands you may execute. On every turn you must reply with exactly two lines: Thought: <one short sentence of reasoning> Action: <one command, copied verbatim from the Admissible list when possible> Plan briefly, then issue the next action. Only the line beginning with ’Action:’ is used to step the environment. Do not add any other text after the Action line.
One-shot demonstration	A one-shot demonstration.
User turn	Task: {task_goal} Observation: {scene_description} Admissible: cmd1; cmd2; ...; cmd30 (+N more)
Assistant turn	Thought: <reasoning> Action: <command>

Appendix

Table 11: Prompt template for ALFWorld. A one-shot demonstration is provided before the user turn.

Field	Prompt Content
System prompt	You are an embodied agent solving a science task in the ScienceWorld text-based simulator. You receive an Observation describing what you can see and an Inventory listing what you carry. On every turn you must reply with exactly two lines: Thought: <one short sentence of reasoning> Action: <one valid action> Plan briefly, then issue the next action. Only the line beginning with ’Action:’ is used to step the environment.
One-shot demonstration	A single generic ReAct trajectory.
User turn	Task: {task_description} Observation: {observation} Inventory: {inventory or "(empty)"}
Assistant turn	Thought: <reasoning> Action: <valid action>

Appendix

Table 12: Prompt template for ScienceWorld. A single generic one-shot ReAct trajectory is provided before the user turn.

Field	Prompt Content
System prompt	You are an expert open-domain question-answering agent. You answer questions by repeatedly issuing wikipedia search queries, reading the returned snippets, and finally producing a concise short answer. Output protocol -- on every turn you must reply with exactly two blocks: <thought>... brief reasoning (1-2 sentences) ...</thought> <action>VERB[PAYLOAD]</action> where VERB[PAYLOAD] is exactly one of: search[your concise wikipedia query, 3-8 words] to fetch top-3 snippets answer[your final short answer] to terminate the episode. Use search[...] until you have enough evidence, then answer[...]. Final answers should be short and direct (a single entity, date, or short phrase). Do not output anything after the </action> tag.
One-shot demonstration	A fixed one-shot demonstration.
User turn	Question: {question} <information>1) ... 2) ... 3) ...</information>
Assistant turn	<thought>...</thought> <action>search[query]</action> or <thought>...</thought> <action>answer[final]</action>

Appendix

Table 13: Prompt template for SearchQA. The agent repeatedly issues Wikipedia search queries and terminates with a concise final answer.

Component	Prompt
Lessons	Successful and failed attempts at ’{task_type}’ ALFWorld tasks: {demos} In 3 short bullet lessons, give the GENERAL strategy for this task type: the correct sub-goal order, and the common mistakes seen in the FAILED attempts to AVOID.
Reflection	You attempted an ALFWorld task. TASK: {goal} YOUR ATTEMPT (step: action -> observation): {ftxt} OUTCOME: {environment outcome} after {N} steps. Using hindsight, identify the SINGLE earliest step whose action was the pivotal mistake, and the better action there. Respond EXACTLY as: STEP: <int> ACTION: <action>

Appendix

Table 14: Prompts used to generate privileged retrospective information. The lessons summary is generated once per task type and round from selected successful and failed trajectories, while reflection is generated retrospectively for the current completed trajectory.

Stage	Construction Template
Forward context	[GENERAL STRATEGY across tasks] {behavior} [LESSONS for this task type] {lessons summary} [PRECEDENT 1 --- a different, similar task solved successfully] 1) act; 2) act; ... [PRECEDENT 2 --- ...] 1) ...
Reflection	[SELF-REFLECTION on your current attempt] {reflection}
Future hindsight	[HINDSIGHT --- with knowledge of the future: ...] {future trajectory information}
Final privileged note	[PRIVILEGED] [LESSONS for this task type] ... [PRECEDENT 1 ...] ... [PRECEDENT 2 ...] ... [SELF-REFLECTION on your current attempt] ... [HINDSIGHT --- with knowledge of the future: ...]
Default injection	The privileged note is appended to the teacher’s initial user turn ( user_end ). Alternative supported positions are system , user_begin , last_user , and trailing .

Appendix

Table 15: Construction of the privileged note used for teacher re-scoring. Components are enabled or disabled by the corresponding context-policy configuration. The default configuration uses lessons, two precedents, reflection, and future hindsight.

Frontier hypothesis-generation prompt (P1)

You are a SCIENTIST tuning the privileged context policy (the ’harness’) for on-policy self-distillation (OPSD). A teacher rescores the student’s OWN rollouts under a note built from precedents + lessons + behavior + future + reflection; the harness config controls that note. GOAL: maximize downstream dev SR (task success) of the next model. PRINCIPLES (general, NOT task-specific): - MORE IS NOT BETTER. Extra/again-predicted context adds tokens+noise without helping; ’remove/simplify’ and turning a component OFF are as valid as ’add’. EVERY whitelisted knob is fair game --- precedent count AND retrieval AND lessons/behavior/future/reflection/selection/etc. Do NOT fixate on precedent count. - You are optimizing PROXY DEV SR measured by a NO-TRAIN harness@inference eval: each candidate’s precedents are injected into the model’s OWN context and we read dev SR. This probe reflects the FORWARD-CONTEXT knobs (precedent count/retrieval) --- it is VALIDATED to track true full-round SR for those. It does NOT move for hindsight/training-mechanism knobs (future/lessons/reflection/teacher_temp/selection): those have no inference analog. You MAY still propose them, but the probe will read

\sim

flat for them and they will be judged only by the real training round (the retrospective gate), not this probe. - Prefer edits your evidence says raise SR. - Form hypotheses, test them, and revise. ===== ROUND DIAGNOSTICS ===== {digest} ===== EDITABLE KNOBS (whitelist) ===== {whitelist} ===== PROBE EVIDENCE SO FAR THIS ROUND (harness@inference dev SR, no-train) ===== {accumulated evidence, or ‘‘(none yet --- this is the first iteration)’’} Propose {NCAND} DISTINCT candidate edits to TEST now (each changes AT MOST {MAX_EDITS} whitelisted knobs, in-bounds). Cover DIFFERENT knob families (include >=1 subtractive/OFF and >=1 touching a generated component: lessons/behavior/future/reflection). Print EXACTLY {NCAND} lines, nothing else: CAND: <key>=<val>[, <key2>=<val2>]; HYP: <why this should raise dev SR>

Appendix

Table 16: Frontier-model prompt for proposing candidate context-policy edits. The frontier model receives round diagnostics, the current editable configuration, and any no-training probe evidence accumulated during the current search round.

Frontier candidate-selection prompt (P2)

Candidate edits tested this iteration (harness@inference dev SR, no-train; higher is better): {cand1: <edit> -> proxy_dev_SR=...; cand2: ...; ...} CURRENT (unedited) harness under the SAME no-train probe: proxy_dev_SR={CUR_SR} Read the proxy SRs as EVIDENCE of downstream task success. Choose the action: - CHOOSE cand<i> --- if a candidate’s proxy SR is convincingly the best (and beats leaving the harness as-is). - CHOOSE none --- if none convincingly beats the current harness. - REHYPOTHESIZE --- if the evidence suggests a better untested direction; you will get another probe round. Print EXACTLY one line: ‘CHOOSE: cand<i>’ or ‘CHOOSE: none’ or ‘REHYPOTHESIZE’ Then one line: ‘WHY: <cite the proxy-SR evidence>’

Appendix

Table 17: Frontier-model prompt for selecting among candidate context-policy edits after no-training probe evaluation.

Self-Evolving Agents

Self-evolving agents improve their skills, memory, tools, policies, or environments by learning from interaction and evaluation feedback.

See all 416 papers

Aug 13, 2026

Latent On-Policy Self-Distillation

Enabling agents to learn from experience and internalize it into their policy has become a central problem in self-evolving AI. On-policy self-distillation (OPSD) offers an effective pathway by using a privileged self-teacher to provide dense supervision on the student's own trajectories; however, existing methods still rely heavily on designer-specified privileged artifacts (e.g., answers, feedback, skills, or trajectories), limiting the end-to-end learnability and scalability required for continual self-improvement. In this work, we introduce Latent On-Policy Self-Distillation (LOPD), which, rather than proposing another hand-crafted OPSD variant with a newly prescribed form of privileged context, makes the teacher's privileged context itself learnable end-to-end from experience. Technically, LOPD retrieves relevant experiences and composes them into continuous latent tokens that condition a self-teacher, while the student generates trajectories from the task and interaction history and receives dense token-level supervision at every visited prefix. We further introduce a privileged-margin objective to stabilize and regulate the learning of latent context. Empirically, LOPD demonstrates (I) strong performance, outperforming RLVR and representative OPSD methods including OPSD, SDPO, and Skill-SD across both agentic tool use and code generation; and (II) high learning efficiency, surpassing GRPO and Skill-SD with less than 30% of their rollout budget. Ablation studies further provide direct evidence that making privileged context learnable is necessary for realizing these gains. Together, these results position LOPD as a step toward a more scalable and self-directed paradigm for agent evolution.

Guibin Zhang, Jiayang Lyu, Ran Sun +4

1National University of Singapore · 2Beijing University of Posts and Telecommunications · 3Shanghai Jiao Tong University

On-Policy Self-Distillation Knowledge Distillation Self-Evolving Agents

Ask PDF

Oct 8, 2026

Environmental Feedback Modeling Matters: Rethinking Feedback Treatment in Agentic Hindsight Self-Distillation

Reinforcement learning is commonly used to train language agents in interactive environments, but cannot be directly applied when rewards are unavailable. Recent methods use environmental feedback as privileged context for hindsight self-distillation, but our analysis suggests that simply conditioning the teacher on feedback is insufficient, motivating us to rethink how environmental feedback is used in agentic self-distillation. Given that environmental feedback contains rich supervision for modeling how the environment responds to agent actions, we introduce agentic SElf-distilLation with environmental Feedback modeling (SELF), a framework that jointly optimizes environmental feedback modeling and hindsight self-distillation. SELF learns to predict environmental responses while distilling guidance from a feedback-conditioned self-teacher into the policy. Our analysis reveals a mutually reinforcing mechanism: environmental feedback modeling strengthens hindsight supervision and policy learning, while self-distillation enhances the model's ability to model environmental feedback. With Qwen3-8B, SELF outperforms SDPO and GRPO by 6.4 and 4.1 percentage points in

τ

-bench success rate, and by 10.71 and 3.57 percentage points in AppWorld task goal completion, respectively. These results show that SELF uses environmental feedback more effectively within agentic self-distillation, improving agent capabilities.

Hangxi Guo, Fengyuan Liu, Yue Wang +4

The Chinese University of Hong Kong, Shenzhen · Shanghai AI Laboratory · University of Science and Technology of China +2

LLM Agent Training Self-Distillation Self-Training for Language Models

Ask PDF

Jun 16, 2026

OPD-Evolver: Cultivating Holistic Agent Evolver via On-Policy Distillation

Memory has become a standard substrate for self-evolving agents, yet retaining experience is not the same as learning how to evolve through it. Existing memory agents can store trajectories, retrieve reflections, or accumulate skills, but often lack the holistic competence to select useful experience, act on it, write reusable knowledge, and maintain a growing repository. We introduce OPD-Evolver, a slow-fast co-evolution framework that cultivates such an agent evolver through on-policy self-distillation. In the fast loop, OPD-Evolver interacts with a four-level memory hierarchy to read, use, write, and maintain experience for rapid test-time evolution. In the slow loop, outcome-calibrated memory attribution and privileged hindsight distill these four abilities into the deployable policy. Across multi-domain benchmarks, OPD-Evolver surpasses memory systems such as ReasoningBank by up to 11.5%, and training-based methods such as Skill0 by ~5.8%. Further analysis shows that OPD-Evolver internalizes high-value experience and memory management, enabling OPD-Evolver-9B to challenge giant counterparts such as Qwen3.5-397B-A17B and Step-3.5-Flash, pointing beyond memory-augmented agents toward genuinely qualified agent evolvers.

Guibin Zhang, Xun Xu, Yanwei Yue +4

LV-NUS Lab · FDU · PKU +1

LLM Agent Memory Memory-Augmented Agents Self-Evolving Agents

Ask PDF

Abstract

Self-Evolving Agents

Explore similar work

Latent On-Policy Self-Distillation

Environmental Feedback Modeling Matters: Rethinking Feedback Treatment in Agentic Hindsight Self-Distillation

OPD-Evolver: Cultivating Holistic Agent Evolver via On-Policy Distillation