Predict Before You Modify: Learning Improvement Dynamics for Self-Evolving Agents

Zixuan Xie; Chang Xu; Yuhua Jiang; Li Zhao

cs.LGSep 26, 2026

Predict Before You Modify: Learning Improvement Dynamics for Self-Evolving Agents

Authors: Zixuan Xie, Chang Xu, Yuhua Jiang, Li Zhao

Organizations: University of Virginia · Microsoft Research · Tsinghua University

Abstract

Self-evolving agents improve through repeated cycles of modification and evaluation. However, existing methods receive feedback on modifications only after costly evaluations, and aggregate scores may obscure distinctions between failures that require different repairs. We introduce predictive self-evolution through an editable Improvement Dynamics Model (IDM). Before evaluation, the IDM predicts task-level outcomes of a proposed modification and guides an optional revision. After evaluation, past modifications and their observed outcomes are used to improve the IDM, allowing predictive knowledge to accumulate across generations. The system can thus improve how it solves tasks, proposes modifications, and predicts their effects. Building on HyperAgents, we evaluate this approach across multilingual coding, paper review, robotics reward design, and Olympiad-level mathematical grading. Our approach outperforms HyperAgents across all four domains, with up to 46% higher average performance over generations. Gains also persist under matched token budgets that include both agent evolution and task evaluation, and extend to held-out tasks. Forecast accuracy improves as the IDM evolves, and the learned predictive knowledge transfers to improve evolution in a new domain.

Figures & tables

Figure 1: An Athena for self-evolution. Like Athena guiding Odysseus, the IDM anticipates task-level outcomes to guide an optional revision before evaluation. Only the resulting child’s task agent is evaluated. Final forecasts and observed outcomes provide evidence for improving the IDM across generations.

Figure 2: Pipeline of predictive self-evolution. Upper: the IDM forecasts a draft and guides at most one revision; revised diffs are forecast again before evaluation. The final forecast and observed outcomes form a record, and the child enters the archive. Lower: accumulated records support candidate IDM updates, which replace the current IDM only after validation on subsequently collected records.

Figure 3: More effective coding evolution with gains on held-out tasks. (a) AUBC as the generation budget increases. (b) AUBC under matched token budgets, including tokens used for both agent evolution and task evaluation. (c) Improvement in resolve rate over the common initial agent on 166 held-out coding tasks, measured in percentage points (pp). The initial agent’s resolve rate is 16.27%. Curves and bars show medians over five runs. Shading and error bars indicate 95% run-level bootstrap confidence intervals, computed pointwise in (a).

Figure 4: Predictive guidance improves joint evolution across different task objectives. AUBC under (a) a budget of 40 generations and (b) matched token budgets. Paper and Robotics track the best evaluated descendant for each domain. Joint tracks the highest joint score attained by a single evaluated descendant. Bars show medians over five runs, and error bars indicate 95% run-level bootstrap confidence intervals. AUBC values are multiplied by 100. The vertical axes are truncated.

Figure 5: IDM predictions improve during evolution. (a) Candidate and current IDMs are compared on subsequent validation records in 28 comparisons across five Polyglot runs, including rejected candidates. Blue points indicate lower candidate Brier loss. (b) Successive deployed IDMs are evaluated on fixed modifications from generations 1–5 of selected runs, with initial loss normalized to 100. Markers indicate each version’s first deployment. Mathematical grading uses Selective IDM.

Figure 6: Predictive experience transfers to mathematical grading. AUBC under (a) 40 generations and (b) a matched token budget for mathematical grading. (c) Accuracy on 100 held-out tasks for agents selected by validation performance. Bars show medians over five runs, with 95% run-level bootstrap confidence intervals. AUBC values are multiplied by 100. The vertical axes are truncated.

Figure 7: Effects of predictive feedback, IDM updates, and revision strategy. Curves show median AUBC over five runs on Polyglot as the generation budget increases. The left panel shows the complete trajectory; the right panel enlarges the outlined region for budgets of 20–40 generations.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Train	Validation	Held-out	Evolution score
Polyglot coding	60	–	166	Resolve rate
Paper review	100	100	–	Validation accuracy
Robotics reward design	6	–	–	Training fitness
Mathematical grading	100	100	100	Validation accuracy

Appendix

Table 1: Task splits and scores used in our experiments. Robotics counts refer to repeated reward-design trials on the same control task.

Setting	Task agent	Meta agent and IDM
Polyglot coding	o3-mini	Claude Sonnet 4.6
Joint: paper review	GPT-4o	GPT-5.4
Joint: robotics	GPT-5.4	GPT-5.4
Mathematical grading	GPT-4o	GPT-5.4

Appendix

Table 2: Initial LLM assignments across experimental settings.

Method	IDM feedback	Revision rule	IDM updates
HyperAgents	No	None	–
Reflective HyperAgents	No	Always	–
IDM Reflection	Yes	Always	Yes
Frozen Selective IDM	Yes	Selective	No
Selective IDM	Yes	Selective	Yes

Appendix

Table 3: Baselines and variants compared on Polyglot. “Always” requests one revision for every draft. “Selective” requests one revision according to the revision rule.

Method	Mean resolve rate (%)	Full-evaluation generations (%)
HyperAgents	17.00	5.0
Reflective HyperAgents	29.25	35.0
IDM Reflection	36.50	57.5
Frozen Selective IDM	28.25	30.0
Selective IDM	29.50	45.0

Appendix

Table 5: Quality of generated coding descendants. Values are percentages, reported as medians over five runs. Each metric is computed within a run over all 40 generations.

Setting	Method	Mean tokens (M)
Coding	HyperAgents	49.45
	Reflective HyperAgents	138.56
	Frozen Selective IDM	134.62
	IDM Reflection	207.58
	Selective IDM	170.22
Joint	HyperAgents	101.67

Appendix

Table 6: Token costs of complete evolution runs. Values are mean estimated token counts over five 40-generation runs, in millions. Initialization, independent held-out evaluation, and post-hoc analyses are excluded.

Figure 8: Selective IDM retains its advantage under a matched token budget. Bars show median AUBC over five runs on Polyglot at a common budget of 145.79M tokens. The vertical axis is truncated.

Figure 9: Individual-run results for coding. AUBC on Polyglot under (a) 40 generations and (b) a common budget of 145.79M estimated tokens. The token budget is the median cost of five complete Reflective HyperAgents runs and is shared by all five methods. AUBC values are multiplied by 100. The vertical axes are truncated.

Comparator	40 generations	Matched tokens
Reflective HyperAgents	$4.42\;[-6.54,\,9.25]$	$1.21\;[-6.32,\,4.79]$
Frozen Selective IDM	$2.67\;[-3.29,\,7.62]$	$0.92\;[-4.73,\,2.64]$
IDM Reflection	$1.71\;[-14.17,\,5.46]$	$1.13\;[-9.78,\,2.85]$

Appendix

Table 7: Median coding AUBC differences: Selective IDM minus each comparator. Each entry is Selective IDM minus the comparator, followed by its 95% run-level bootstrap interval. Values are absolute AUBC differences multiplied by 100, not relative percentage gains. The token budget is 145.79M for all methods.

Figure 10: Individual-run results for joint evolution. AUBC for paper review, robotics, and the joint objective under (a) 40 generations and (b) a common budget of 111.98M estimated tokens. AUBC values are multiplied by 100. The vertical axes are truncated.

Figure 11: Individual-run results for mathematical grading. AUBC under (a) 40 generations and (b) a common budget of 69.36M estimated tokens. AUBC values are multiplied by 100. The vertical axes are truncated.

Figure 12: Modification information improves IDM forecasts. Relative reduction in mean prediction loss compared with No-diff, which withholds the diff and modified source code while retaining parent code, parent outcomes, and historical records. Comparisons use the same deployed IDM for each of 15 evaluated modifications from five joint evolution runs. Joint loss weights the two domains equally before computing the relative reduction.

Figure 13: Task strategies and forecasting rules evolve within the same run. (a) Best and mean archive scores, together with the lineage leading to the final best agent. (b) Mean online prediction loss during each IDM version’s deployment period. Annotations summarize selected changes to the task agent and the IDM.

Figure 14: Forecast-guided revision across tasks. In Coding, a forecast of widespread incomplete execution prompts syntax repair. In Paper review, a forecast of limited applicability is followed by an expanded preprocessing change.

Figure 15: Learning to use historical outcomes in forecasting. The evolved IDM adds per-task outcome summaries to its forecast inputs; the quotation shows their use in a later online forecast. The lower panel separately compares the original and updated IDMs on the same historical modification. The updated IDM’s highest-probability prediction matches the observed test failure.

LLM Agent Self-Improvement

Large language model agent self-improvement studies how agents use experience and feedback to improve their behavior, tools, and workflows.

See all 257 papers

The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators
Jun 24, 2026Alex Iacob, Andrej Jovanović, William F. Shen +12Evolutionary Optimization AI Agent Evaluation
University of Cambridge · Flower Labs · NVIDIA +2
Self-Evolving Harness on Multiple Tasks with the Agent as Its Own Optimizer
Sep 29, 2026Qiankai XuContinual Learning for LLM Agents AI Agent Evaluation
Nanjing University
Self-Improvements in Modern Agentic Systems: A Survey
Jul 14, 2026Zhe Ren, Yimeng Chen, Dandan Guo +9LLM Agent Self-Improvement Self-Improving Agents
School of Artificial Intelligence, Jilin University · King Abdullah University of Science and Technology (KAUST) · Independent Researcher +2

Jun 24, 2026J/K move · Enter open · S save

The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators

Alex Iacob, Andrej Jovanović, William F. Shen, Daniel Burkhardt+11

University of Cambridge · Flower Labs · NVIDIA+2

Evolutionary Optimization AI Agent Evaluation Self-Improving Agents Automated Peer Review Recursive Self-Improvement

Open paper AskPDF arXiv

Self-improving agents are state-of-the-art on agentic coding benchmarks, yet their search methods assume a stationary evaluation criterion. This ignores a central feature of evolution: species adapt as their environments change with them. We introduce the Red Queen Gödel Machine (RQGM), an evolutionary framework for recursive self-improvement under non-stationary utilities. This allows learned evaluators to improve alongside the agents they guide. On DeepSWE, the RQGM improves over its fixed-evaluator baseline by adding a complementary agent-as-a-judge code-review signal: a co-evolved reviewer grades coder patches to guide search. At low reasoning effort, the RQGM coder passes 82.1% of held-out tasks against the baseline's 75.0%, and nearly matches the GPT-6 Astra model at high effort. In scientific paper writing and reviewing, and Olympiad-level proof writing and grading, co-evolved evaluators provide an evaluation criterion. Anchored to human IMO grades, a co-evolved grader writes its own milestone rubric and exceeds static baselines at a 3x lower search cost, driving the prover to the best mean score. Since the RQGM can modify the search objective across epochs, it can regularize the search. For example, the RQGM reduces self-preference bias via an additional adversarial objective to discover reviewers equally stringent on AI and human work. Guided by these calibrated reviewers, co-evolved writers reach 1.78x-1.86x higher acceptance rates than the baseline under an agent-as-a-judge panel. The RQGM enables self-improving systems where agents and evaluators recursively bootstrap each other beyond static evaluation.

Abstract

LLM Agent Self-Improvement

Explore similar work

The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators

Self-Evolving Harness on Multiple Tasks with the Agent as Its Own Optimizer

Self-Improvements in Modern Agentic Systems: A Survey