Evolutionary System Prompt Learning for Reinforcement Learning in LLMs

Lunjun Zhang; Ryan Chen; Bradly C. Stadie

cs.AIFeb 16, 2026

Evolutionary System Prompt Learning for Reinforcement Learning in LLMs

Authors: Lunjun Zhang, Ryan Chen, Bradly C. Stadie

Organizations: Department of Computer Science, University of Toronto · Department of Statistics and Data Science, Northwestern University · Bridgewater AIA Labs

Abstract

Building agentic systems that can autonomously self-improve from experience is a longstanding goal of AI. Large language models (LLMs) today primarily self-improve via two mechanisms: self-reflection for context updates, and reinforcement learning (RL) for weight updates. In this work, we propose Evolutionary System Prompt Learning (E-SPL), a method for jointly improving model contexts and model weights. In each RL iteration, E-SPL samples trajectories under multiple system prompts in parallel, then jointly applies RL updates to LLM weights and evolutionary updates to system prompts. System prompts evolve via mutation and crossover, two genetic operators driven by LLM self-reflection; selection is based on relative performance ratings updated across RL iterations. E-SPL encourages a natural division between declarative knowledge encoded in prompts and procedural knowledge encoded in weights, resulting in improved performance across reasoning and agentic tasks. For instance, in an easy-to-hard (AIME $\rightarrow$ BeyondAIME) generalization setting, E-SPL improves RL success rate from 38.8% $\rightarrow$ 45.1% while also outperforming reflective prompt evolution (40.0%). Overall, our results demonstrate that RL and system prompt evolution are deeply synergistic, and combining the two yields consistent gains in sample efficiency and generalization. Code: https://github.com/LunjunZhang/E-SPL

Figures & tables

Figure 1: Evolving System Prompts helps RL generalize better .

Figure 2: Evolutionary System Prompt Learning (E-SPL) jointly optimizes model contexts and model weights to enhance LLM self-improvement. Evolution updates system prompts; RL updates weights. The learned system prompts can encode declarative knowledge via articulated principles and strategies, while RL gradients can further hone the model’s procedural knowledge for reliable execution.

Figure 3: Evolutionary trees of E-SPL . During RL, E-SPL creates an evolutionary tree of system prompts, by re-suing the same data already generated by RL with minimal additional computational overhead. Each genetic operator (mutation or crossover) only requires a sampling server for LLM self-reflection with different context construction strategies, which can be concurrent with RL gradient updates.

Panel of Figure 4 — Figure 4: Discovered strategies in learned System Prompts for solving math problems. Those explicit behavior specifications include: useful heuristics and tips for various categories of problems, self-verification strategies such as checking for consistency and plausibility, a list of common failure modes to avoid, etc. Note that RL is done under diverse system prompts, and does not overfit to any particular one.

Figure 5: Mutation operator in E-SPL. The highest-performing prompt in each iteration undergoes LLM self-reflection on group-wise agent trajectories and their outcomes. An LLM-generated $\mathrm{diff}$ edits the parent into a child system prompt, removing ineffective rules and converting observed mistakes into improved declarative instructions, yielding a new prompt that enters the evolutionary population.

Figure 6: Crossover operator in E-SPL. System prompts are compared based on their problem-wise performance within the current RL batch. Guided by these differential strengths and weaknesses, an LLM self-reflection process selectively recombines the most effective complementary segments from multiple parent prompts, yielding a new child prompt that enters the evolutionary population.

Panel of Figure 7 — Figure 7: An example of the system prompt discovered by E-SPL for RL with search engines. Here, the discovered system prompt specifies a workflow that includes: generative self-verification, detailed instructions on tool use, critical formatting rules, heuristics for determining the reliability of various information sources, actionable guidance for avoiding common failure modes, etc.

Figure 8: TrueSkill scores learned from train-time rankings are predictive of relative test-time performance of system prompts. E-SPL treats each RL iteration as a set of relative comparisons between system prompts for TrueSkill message passing. The visualization shows that E-SPL effectively aggregates noisy train-time comparisons into a stable and informative criterion for evolutionary selection.

Panel of Figure 9 — Figure 9: An example of how the RL model learns to utilize learned system prompts (numbered G1 $\dots$ G30) to solve problems.

Panel of Figure 10 — Figure 10: An example of how the RL model learns to utilize learned system prompts to solve problems.

Figure 11: Results on mathematical reasoning datasets using DeepSeek-v3.1. Evolution + RL (our E-SPL method) performs best compared to self-reflection (prompting only), evolution (prompting only), and RL (weight update only). The results show that coupling RL with evolution on system prompts unlocks a synergistic form of self-improvement that neither approach achieves in isolation.

Figure 12: E-SPL learns faster, and achieves better asymptotic performance that neither RL nor prompt evolution alone could achieve. By studying the learned system prompts (e.g. Figure 4 and Figure 9 ), we hypothesize that E-SPL achieves this by encouraging a natural division between declarative knowledge (factual information or global heuristics that can be verbally explained) in prompts and procedural knowledge (practical, instinctive “know-how” that relies on intuition developed through repetition) in weights.

Figure 13: E-SPL outperforms both (prompt) Evolution and RL on agentic search tasks with openai gpt-oss-120b model.

	Mistakes in Learned E-SPL System Prompts
Discovered	For a polynomial $P(n)={\textstyle\prod}_{i=1}^{m}(n-a_{i})$ , the largest integer $d$ that divides $P(n)$ for all integers $n$ is $d={\textstyle\prod}_{p}p^{e_{p}}$ , where $e_{p}=\min_{r\bmod p}\bigl(\#\{\,i:a_{i}\equiv r\!\!\pmod{p}\,\}\bigr).$ Only primes $p\leq\max(a_{i})$ need be considered, as for larger $p$ , the $a_{i}$ are distinct modulo $p$ , so $e_{p}=0$ . For primes where the function mapping $i$ to $a_{i}\bmod p$ is a permutation (e.g., if $a_{i}=i^{3}$ and $p\equiv 2\bmod 3$ ), $e_{p}={{\left\lfloor\frac{m}{p}\right\rfloor}}$ .
Corrected	For a polynomial $P(n)=\prod_{i=1}^{m}(n-a_{i})$ , the largest integer $d$ that divides $P(n)$ for all integers $n$ is $d=\prod_{p}p^{e_{p}}$ , where $e_{p}=\min_{n\in\mathbb{Z}}v_{p}\!\bigl(P(n)\bigr).$ Moreover, one always has the lower bound $e_{p}\geq\min_{r\bmod p}\bigl(\#\{\,i:a_{i}\equiv r\!\!\pmod{p}\,\}\bigr).$ In particular, for this mod- $p$ bound it suffices to consider primes $p\leq m$ , since if $p>m$ then some residue class occurs zero times and the minimum is $0$ . For primes where the function mapping $i$ to $a_{i}\bmod p$ is a permutation (e.g., if $a_{i}=i^{3}$ and $p\equiv 2\bmod 3$ ), the bound gives $e_{p}\boldsymbol{\geq}\left\lfloor\frac{m}{p}\right\rfloor.$
Discovered	For graphs with odd-degree vertices, find a minimum $T$ -join by pairing odd vertices along efficient paths . In symmetric graphs such as grids, pair boundary vertices optimally to minimize total length.
Corrected	For graphs with odd-degree vertices, find a minimum $T$ -join by pairing odd vertices via a minimum-weight perfect matching on shortest-path distances . In symmetric graphs such as grids, restrict the matching to symmetry-respecting pairings to minimize total length.

Table 1: The discovered system prompts sometimes include heuristics that are not always true but still helpful to problem-solving . Unlike RL updates to model weights, the learned system prompts are interpretable and thus can be monitored and corrected. Here we showcase two subtle mistakes in two discovered principles, and what their corrected versions should be (see § C.1 for explanations).

Figure 14: Crossover operator in evolution speeds up early learning , but obtains mixed results in terms of final performance. On AIME

\rightarrow

BeyondAIME, Mutation + RL gets to

45.1\%

success rate, while Mutation + Crossover + RL gets to

42.5\%

; on HMMT, using Crossover operator results in slightly better performance (

52.7\%

51.6\%

). In either case, Evolution + RL outperforms RL-only.

Figure 15: Using a fixed reference policy rather than the online RL policy for prompt evolution is important to performance.

Panel of Figure 16 — Figure 16: An example of how the RL model learns to utilize learned system prompts to solve problems (also see Figure 22 ).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 17: Detailed overview of the E-SPL pipeline , which illustrates how Figure 2 , Figure 5 , and Figure 6 fit into a single algorithm.

Figure 18: Self-Reflection Module in Mutation (where the best-performing system prompt

\mathbf{s}_{k}

is edited) consists of four sequential modules: trajectory summarization (§ A.2 ), self-critique (§ A.3 ), local edit proposal (§ A.4 ), local-to-global aggregation (§ A.5 ).

ID	Learned Principles in System Prompt (AIME)
G0	When optimizing sums, set parameters to cancel dominant terms. For quadratic forms, choose parameters that minimize the leading coefficients, reducing the problem to analyzing remainder terms.
G1	For combinatorial counting problems, use generating functions with geometric series expansions. Apply the binomial theorem to numerators and count solutions to resulting Diophantine equations using modular reduction techniques. Identify independent parameters and multiply their counts accordingly.
G2	When counting nonnegative solutions to a linear Diophantine equation $a_{1}x_{1}+\cdots+a_{n}x_{n}=N$ , use modular arithmetic to reduce variables. For a prime $p$ dividing some coefficients, take the equation modulo $p$ to constrain one variable, substitute $x_{i}=c+pt$ , simplify, and divide by common factors. Repeat until the equation is solvable by iteration or direct counting.
G3	After computations, verify results for consistency across domains: check denominators, symmetry, and physical plausibility. Use alternative methods to recompute critical quantities and detect patterns such as periodicity.
G4	Avoid detours and unjustified guesses, including contest heuristics without validation. Trust sound derivations; if inconsistencies persist, explore alternative configurations or identities.
G5	Reformulate problems by identifying a key parameter or ratio that simplifies constraints. Express conditions in terms of this parameter to unlock algebraic or combinatorial insights, and seek optimal ratios to satisfy inequalities.

Appendix

Table 20

ID	(Continued) Learned Principles in System Prompt (AIME)
G19	For bipartite grid graphs with residue constraints, first count valid assignments on one part using multinomial coefficients. Then determine the number of valid assignments on the other part, which is often constant due to symmetry.
G20	For problems with equal angles from boundary points to an interior point, use rotations to express the interior point as a rotated vector from each boundary point. Equating these expressions yields solvable relations via spiral similarity.
G21	For tilted polyhedra with given vertex heights and a horizontal water surface, use vector representations. Express heights via dot products with the vertical vector and compute submerged volume by integrating over the unit cube or via geometric dissection.
G22	In modular arithmetic and sums, decompose expressions using residues and periodicity. Apply the Chinese Remainder Theorem for coprime moduli and multiplicativity for square-free cases.
G23	For circle geometry, use power of a point, perpendicular bisectors, and homothety. Exploit symmetry to simplify equations and avoid sign errors.
G24	In symmetric probabilistic problems, use complementary probability and symmetry to reduce cases. Verify that no hidden asymmetries remain and compute failure probabilities carefully.

Appendix

Table 21

ID	Learned Principles in System Prompt (DAPO)
G0	In symmetric problems, assume an ordering of variables to reduce cases. Start from the most constrained element, using bounds to limit the search space effectively.
G1	When a bound is stated as “does not exceed $M$ ,”, default to $0\leq x\leq M$ unless specified otherwise. This resolves common contest-style ambiguity.
G2	For assignment problems with constraints, enumerate valid distributions first. Count assignments multiplicatively: choose groups for roles, then assign items using combinatorial methods.
G3	For subset problems with coprimality conditions, construct maximal invalid sets. Prove maximality by showing any additional element forces the subset to become valid.
G4	For geometric problems with integer coordinates, translate conditions into algebraic equations. Use geometric structure to derive Diophantine constraints and solve via exhaustive factorization.
G5	To verify modular injectivity, factor differences and check that the cofactor is nonzero modulo the modulus. For complex cases, test small primes; derivative-based criteria can be sufficient but are not necessary.

Appendix

Table 22

ID	Learned Principles in System Prompt (HMMT)
G0	When dealing with geometric constraints, model positions using vectors or coordinates to interpret distances and directions. Exploit symmetry in coordinate placement (e.g., centroid at the origin) only when justified; otherwise, use general orientations with parameters.
G1	Before solving equations or counting problems, express the target in terms of fundamental components (variables, prime factors, or combinatorial elements). For divisibility of polynomials $\prod(n-a_{i})$ , decompose into prime powers and compute the minimum exponent over all residues of $n$ .
G2	When deriving polynomial identities from functional equations, arrange expressions as $g(x)Q(x)-P(x)=k\prod(x-r_{i})$ to maintain consistent signs and avoid coefficient errors.
G3	If a proportionality constant scales all coefficients equally, it cancels in ratios of coefficients. Compute such ratios directly without solving for the constant.
G4	For graphs with odd-degree vertices, find a minimum $T$ -join by pairing odd vertices along efficient paths. In symmetric graphs such as grids, pair boundary vertices optimally to minimize total length.
G5	When decomposing angles at a vertex with multiple rays, express unknown angles using auxiliary variables tied to geometric constraints (e.g., isosceles triangles). Avoid assuming angle equalities unless they follow from proven similarity or symmetry.

Appendix

Table 23

ID	(Continued) Learned Principles in System Prompt (HMMT)
G19	In grid path problems, decompose the grid into diagonals (sets with constant $i+j$ ). Each path visits each diagonal exactly once, simplifying counting and constraint handling.
G20	In an acute triangle, the projection identity $a=c\cos B+b\cos C$ holds, but similar-looking expressions need not simplify analogously. Always verify projection formulas using a non-symmetric example.
G21	For placements in symmetric geometric structures, classify positions by symmetry type (corners, edges, faces). Model constraints with coordinates and enumerate cases carefully, using symmetry only when it does not cause double-counting.
G22	For symmetric configurations (e.g., roots of unity), identify the symmetry group preserving the condition. Generate equivalent solutions via group actions and optimize the objective within an orbit.
G23	For sums involving periodic functions, use binary expansions and pairing arguments. If the number of terms is odd, isolate the unpaired term. Validate results with small cases or known identities.
G24	For a chord intersected by circles, place a coordinate along the chord and use feet of perpendiculars from centers as reference points. Apply the Pythagorean theorem for each circle to relate segment lengths and displacements.

Appendix

Table 24

Panel of Figure 19 — Figure 19: Example of failure modes in E-SPL.

Panel of Figure 20 — Figure 20: An example of how the RL model learns to utilize learned system prompts to solve problems.

Panel of Figure 21 — Figure 21: Example of failure modes in E-SPL.

Panel of Figure 22 — Figure 22: An example of how the RL model learns to utilize learned system prompts to solve problems.

Language Model Self-Improvement

Language model self-improvement studies how models use their own outputs, feedback, and experience to improve with less external supervision.

See all 171 papers

Mar 19, 2026

Learning to Self-Evolve

We introduce Learning to Self-Evolve (LSE), a reinforcement learning framework that trains large language models (LLMs) to improve their own contexts at test time. We situate LSE in the setting of test-time self-evolution, where a model iteratively refines its context from feedback on seen problems to perform better on new ones. Existing approaches rely entirely on the inherent reasoning ability of the model and never explicitly train it for this task. LSE reduces the multi-step evolution problem to a single-step RL objective, where each context edit is rewarded by the improvement in downstream performance. We pair this objective with a tree-guided evolution loop. On Text-to-SQL generation (BIRD) and general question answering (MMLU-Redux), a 4B-parameter model trained with LSE outperforms self-evolving policies powered by GPT-5 and Claude Sonnet 4.5, as well as prompt optimization methods including GEPA and TextGrad, and transfers to guide other models without additional training. Our results highlight the effectiveness of treating self-evolution as a learnable skill.

Xiaoyin Chen, Canwen Xu, Yite Wang +3

Mila – Quebec AI Institute · University of Montreal · Snowflake

Language Model Self-Improvement RL for Language Models

Ask PDF

Jun 2, 2026

EvoTrainer: Co-Evolving LLM Policies and Training Harnesses for Autonomous Agentic Reinforcement Learning

Autonomous LLM training is often framed as recipe search, which leaves the training harness largely static. This limitation sharpens in agentic RL, where shifting bottlenecks and scalar rewards mask diverse failure modes. We introduce EvoTrainer, an autonomous training framework that co-evolves LLM policies and training-side harnesses through empirical feedback: it diagnoses rollout-level evidence, revises diagnostics, backtests interventions, and accumulates reusable skills. Evaluated on mathematical reasoning, competitive-programming code generation, and repository-level software engineering, EvoTrainer matches or exceeds the human-engineered RL references under the same data, codebase, and evaluation protocol, with the largest gain on long-horizon agentic SWE. Trajectory analyses show that retained strategies diverge across domains, evolving diagnostics prevent invalid high-scoring branches from being promoted, and reusable skills shape later search. Autonomous LLM RL should move beyond recipe search toward joint evolution of policies and the training harnesses that interpret them.

Guhong Chen, Yingcheng Shi, Yongbin Li +6

1Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · 2Tongyi Lab , Alibaba Group · 3Alibaba Group +1

RL for Code Generation Agentic RL Reinforcement Learning Agent Harness Optimization LLM Agent Skill Learning LLM Agent Training+1

Ask PDF

Jun 3, 2026

SePO: Self-Evolving Prompt Agent for System Prompt Optimization

System prompt optimization improves agent behavior without modifying the underlying model, yielding human-readable, model-agnostic instructions. Existing methods build a prompt agent that refines task agents' system prompts, yet leave the prompt agent's own system prompt hand-engineered and fixed. We propose Self-Evolving Prompt Optimization (SePO), which treats the prompt agent's own system prompt as an optimization target alongside task agents' system prompts. SePO adopts a self-referential design. A single prompt agent improves both task agents' system prompts and its own under an open-ended evolutionary search that maintains an archive of candidate prompts as stepping stones. Training proceeds in two stages: pre-training evolves the prompt agent on a multi-task pool, and fine-tuning then applies it to a target task. Across five benchmarks spanning math (AIME'25), abstract reasoning (ARC-AGI-1), graduate-level science (GPQA), code generation (MBPP), and logic puzzles (Sudoku), SePO consistently outperforms Manual-CoT, TextGrad, and MetaSPO, improving the average accuracy by 4.49 points compared to Manual-CoT. The prompt optimization skill from pre-training also generalizes to tasks beyond the pre-training mixture, rather than memorizing per-task prompts.

Wangcheng Tao, Han Wu, Weng-Fai Wong

National University of Singapore · City University of Hong Kong

LLM Agent Self-Improvement Automatic Prompt Optimization LLM Agents Prompt Optimization

Ask PDF

Abstract

Language Model Self-Improvement

Explore similar work

Learning to Self-Evolve

EvoTrainer: Co-Evolving LLM Policies and Training Harnesses for Autonomous Agentic Reinforcement Learning

SePO: Self-Evolving Prompt Agent for System Prompt Optimization