Learning to Self-Evolve

Xiaoyin Chen; Canwen Xu; Yite Wang; Boyi Liu; Zhewei Yao; Yuxiong He

cs.CLMar 19, 2026

Learning to Self-Evolve

Authors: Xiaoyin Chen, Canwen Xu, Yite Wang, Boyi Liu, Zhewei Yao, Yuxiong He

Organizations: Mila – Quebec AI Institute · University of Montreal · Snowflake

Abstract

We introduce Learning to Self-Evolve (LSE), a reinforcement learning framework that trains large language models (LLMs) to improve their own contexts at test time. We situate LSE in the setting of test-time self-evolution, where a model iteratively refines its context from feedback on seen problems to perform better on new ones. Existing approaches rely entirely on the inherent reasoning ability of the model and never explicitly train it for this task. LSE reduces the multi-step evolution problem to a single-step RL objective, where each context edit is rewarded by the improvement in downstream performance. We pair this objective with a tree-guided evolution loop. On Text-to-SQL generation (BIRD) and general question answering (MMLU-Redux), a 4B-parameter model trained with LSE outperforms self-evolving policies powered by GPT-5 and Claude Sonnet 4.5, as well as prompt optimization methods including GEPA and TextGrad, and transfers to guide other models without additional training. Our results highlight the effectiveness of treating self-evolution as a learnable skill.

Figures & tables

Figure 1: Overview of Learning to Self-Evolve (LSE). Left: Tree-guided self-evolution at test time. Upper Confidence Bound (UCB) selection chooses a context from the evolution tree; the action model generates outputs for a new batch of problems; the self-evolving policy receives the performance summary and proposes a revised context. Right: LSE trains the self-evolving policy via RL with an improvement-based reward computed as the difference between post-edit and pre-edit performance.

Method	Financial	Toxicology	Codebase	Formula 1	Card Games	Avg.
Seed prompt	51.0	60.3	63.7	54.5	56.5	57.2
Qwen3-4B-Instruct	63.7	60.3	70.2	56.0	61.0	62.2
Claude Sonnet 4.5	70.8	63.8	67.8	57.3	63.0	64.5
GPT-5	70.8	65.8	72.0	54.3	63.3	65.2
GEPA	64.0	62.0	72.0	54.0	62.0	62.8
TextGrad	60.3	66.0	71.5	56.5	61.3	63.1

Table 1: Text-to-SQL results on BIRD. All methods use Qwen3-4B-Instruct as the action policy

\pi_{\theta}

. We report execution accuracy (%). Best result per column in bold .

Method	Bus. Eth.	Phil.	Prof. Acc.	Econ.	Anat.	Security	Virol.	Moral Sc.	Glob. Facts	Prof. Law	Avg.
Seed prompt	70.5	76.0	67.7	76.2	76.0	68.2	72.5	59.7	55.5	54.0	67.6
Qwen3-4B-Instruct	75.5	79.2	73.0	83.0	82.3	68.8	77.0	59.7	56.0	57.3	71.2
Claude Sonnet 4.5	75.3	79.0	75.5	81.0	79.8	69.3	77.5	67.3	57.0	58.5	72.0
GPT-5	77.5	80.0	73.0	83.5	81.0	70.0	82.0	64.8	55.5	58.0	72.5
GEPA	76.0	80.0	78.0	84.0	82.0	70.0	78.0	62.0	56.0	64.0	73.0
TextGrad	74.0	79.0	67.8	73.3	80.0	67.8	77.3	62.5	51.3	58.3	69.1

Table 2: Question-answering results on MMLU-Redux. All methods use Qwen3-4B-Instruct as the action policy

\pi_{\theta}

. We report accuracy (%). Best result per column in bold .

Panel of Figure 2 — Figure 2: Ablation studies on reward design and search strategy. (a) $A_{\mathrm{GRPO}}$ uses $\bar{R}(c_{1})$ with GRPO’s group-based advantage; $A_{\mathrm{LSE}}$ uses the improvement-based reward $r_{\mathrm{LSE}}=\bar{R}(c_{1})-\bar{R}(c_{0})$ (Eq. 7 ). (b) Tree search (UCB) vs. linear chain (always extends the most recent node), both with the untrained Qwen3-4B-Instruct as $f_{\psi}$ .

Figure 3: Per-round average accuracy on the BIRD Card Games database. The linear chain cannot recover from bad edits, while tree search (UCB) backtracks to higher-scoring ancestors.

Variant	Financial	Toxicology	Codebase	Formula 1	Card Games	Avg.
Seed prompt	56.8	54.5	65.3	52.3	59.5	57.7
+ LSE evolution	68.3	62.3	71.5	57.0	63.0	64.4

Table 3: Test-time self-evolution of a specialized action model on BIRD. Arctic-Text2SQL-R1-7B ( Yao et al., 2025 ) , an RL-tuned text-to-SQL model, serves as the action policy

\pi_{\theta}

. The self-evolving policy

f_{\psi}

is the LSE-trained Qwen3-4B-Instruct from the main experiments, applied without further training. We report execution accuracy (%).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Domain	# Problems
BIRD	Financial	106
	Toxicology	145
	Card Games	191
	Formula 1	174
	Codebase	186
MMLU-Redux	Business Ethics	100

Appendix

Table 4: Number of evaluation problems per domain.

Figure 4: Search strategy ablation on MMLU-Redux, complementing Figure 2(b) . Both variants use the untrained Qwen3-4B-Instruct as the self-evolving policy

f_{\psi}

. Tree search improves the average accuracy from 69.0% to 71.2%.

Explore similar work

On the Generalization Gap in Self-Evolving Language Model Reasoning
May 31, 2026Zhenting Qi, Susanna Maria Baby, Stefanie Anna Baby +5LLM Self-Refinement Language Modeling
Google Research · Harvard University · Google +1
Confidence-Orchestrated Self-Evolution against Uncertain LLM Feedback
May 27, 2026Bowen Wei, Nan Wang, Yuqing Zhou +2Self-Training for Language Models RL for Language Model Reasoning
George Mason University · Amazon
Self-Improving Large Language Models via Progressive Experience Evolution
Aug 3, 2026Shijie Ren, Xiting Wang, Meng Li +8RL for Language Model Reasoning Language Model Self-Improvement
Gaoling School of Artificial Intelligence, Renmin University of China · Weixin AI, Tencent Inc, China

May 31, 2026J/K move · Enter open · S save

On the Generalization Gap in Self-Evolving Language Model Reasoning

Zhenting Qi, Susanna Maria Baby, Stefanie Anna Baby, Kan Yuan+4

Google Research · Harvard University · Google+1

LLM Self-Refinement Language Modeling Language Model Self-Improvement LLM Reasoning

Open paper AskPDF arXiv

Recent work suggests that large language models (LLMs) can improve through self-evolution (SE), using supervision signals generated by the model itself. In this work, we ask: under a strict closed-loop setup, where the self-evolution algorithm has access only to an unlabeled prompt set and a base model, how close can internally generated supervision come to oracle-supervised training? We analyze four representative strategies in a unified offline self-evolution framework: single-round verification, multi-turn revision with feedback, iterative training, and curriculum learning. Our primary experiments use Knights and Knaves (KK) logical reasoning tasks, which provide deterministic solutions, controlled difficulty levels, and a clean testbed for easy-to-hard generalization. We first show that self-evolution consistently improves over the base model, but plateaus after excessive training compute is invested, and eventually still leaves a non-trivial gap to oracle supervision. We find that multi-turn critic-revision with large models can reach strong self-evolution performance, with Gemma 12B nearly matching oracle-supervised training. Beyond Knights and Knaves, we also evaluate self-evolution on real-world reasoning benchmarks, where gains are also modest. Overall, our results characterize when closed-loop self-evolution can help and show how internally generated supervision remains insufficient under this minimal formulation.

Abstract

Explore similar work

On the Generalization Gap in Self-Evolving Language Model Reasoning

Confidence-Orchestrated Self-Evolution against Uncertain LLM Feedback

Self-Improving Large Language Models via Progressive Experience Evolution