RSI-Opt: Recursive Self-Improvement Optimizer Agents For Model Training

Binghang Lu; Yige Sun; Guang Lin

cs.AIOct 23, 2026

RSI-Opt: Recursive Self-Improvement Optimizer Agents For Model Training

Authors: Binghang Lu, Yige Sun, Guang Lin

Organizations: Purdue University · Independent Researcher

Abstract

Conventional neural-network training typically follows a predefined optimization recipe, motivating a LLM agent that monitors training dynamics and adjusts optimizer settings as training progresses, but a fixed agent policy does not accumulate experience across runs. We introduce RSI-Opt, a plug-and-play Recursive Self-Improvement Optimizer Agents which evolves the knowledge that guides this process: a prompt, a playbook of procedures, a library of tested optimizer code, and a decision cadence. A meta-agent rewrites this policy using evidence from earlier runs, and an archive retains evaluated policies for further evolution. Across 5 generations on three training tasks, the meta-agents develop reusable procedures and code and conduct their own CPU proxy experiments. RSI-Opt substantially improves optimizer-agent performance, with representative raw-space summary $z$ -scores increasing from $+10.48$ to $+14.33$ on the evolution suite with Opus 5.5 and from $-0.36$ to $+3.11$ on held-out tasks with GPT-5.5 through cross-model policy transfer. On the Burgers PDE, it reduces mean relative $L_2$ error by approximately 2.7-fold with Opus 5.5 and 8.8-fold with GPT-6.1 over the fixed-prompt agent baseline, although PDE transfer varies across LLMs and tasks. Controlled comparisons show that the evolved policy combines reusable optimization knowledge with adaptation to each task: initial decisions capture most observed gains on seen tasks, while continued adaptation improves outcomes on unseen tasks. These findings demonstrate how procedures and executable tools accumulated across training runs can improve optimizer-agent performance and support transfer to scientific machine learning.

Figures & tables

Figure 1: RSI-Opt evolves reusable knowledge for an optimizer agent. (a) Across runs, a meta-agent samples a parent policy and run evidence, rewrites and validates a child, and evaluates it on the training suite. Every valid evaluated child enters the archive with its score and run evidence. (b) Within a run, an agent implemented reads losses and diagnostics at scheduled decisions and rewrites the optimizer and diagnostic programs. The policy conditions these decisions; the model, data, loss, and update budget remain fixed. Final metrics are withheld from the in-run agent. Held-out tasks are reserved for transfer evaluation; Appendix A documents the access protocol and a historical exposure during evolution.

Group	Task	Static recipe	Metric	Static result
Training suite	ResNet-18 / CIFAR-10	SGD-Nesterov, one-cycle	test acc. $\uparrow$	94.60 $\pm$ 0.16
	GPT-34M / FineWeb	AdamW, warmup-cosine	val. loss $\downarrow$	4.480 $\pm$ 0.008
	Flow UNet / MNIST	AdamW, cosine, EMA	EMA val. MSE $\downarrow$	0.1700 $\pm$ 0.0004
Held-out ML	ViT / CIFAR-100	AdamW, warmup-cosine	test acc. $\uparrow$	55.84 $\pm$ 0.34
	LSTM / WikiText-2	Adam, step decay	test loss $\downarrow$	5.386 $\pm$ 0.064
	ResNet-18 / noisy CIFAR-10	AdamW, warmup-cosine	test acc. $\uparrow$	73.76 $\pm$ 1.71

Table 1: Tasks. Static: the task’s own recipe, mean

\pm

std over 5 seeds of the final metric.

Figure 2: Left: policies evaluated during evolution on training-suite seeds 100 and 101. Gray lines connect children to parents; the red line shows the best archive score so far. Right: per-task z-scores of generation 0 and the selected final policy on these evolution seeds. Fresh-seed comparisons are reported separately.

Model	Method	Train	Held-out ML	Held-out PDE	All held-out
–	Static, tuned LR (oracle)	+3.46	+0.49	+1.11	+0.80
Opus 5.5	Single-agent	+10.48	+8.71	+1.72	+5.22
	RSI-Opt	+14.33	+8.23	+1.51	+4.87
	wins	3/3	2/3	2/3	4/6
GPT-5.5	Single-agent	+4.33	+1.18	-1.90	-0.36
	RSI-Opt	+14.10	+6.54	-0.32	+3.11

Table 3: Task-balanced raw-space summary

z

-scores recomputed from the printed means and static standard deviations in Table 2 ; positive is better than the static recipe. Wins count tasks on which RSI-Opt has the better reported mean, not individual seeds. Scores are descriptive and subject to input rounding.

Task	Opus 5.5	GPT-5.5	GPT-6.1
ResNet-18 / CIFAR-10	+0.23	+0.19	+0.45
GPT-34M / FineWeb	+0.025	+0.099	+0.005
Flow UNet / MNIST	+0.0028	+0.0063	+0.0038
Training suite: mean $\Delta\bar{z}^{\mathrm{raw}}$	+3.85	+9.77	+4.31
ViT / CIFAR-100	-1.10	+4.42	+6.38
LSTM / WikiText-2	+0.001	-0.030	-0.124

Table 4: Signed differences between the reported mean metrics in Table 2 , positive when RSI-Opt is better. Task rows use accuracy points, loss units, or raw relative

L_{2}

error; PDE rows are not differences of per-seed log errors. Group rows are differences of the raw-space summary

z

-scores in Table 3 , computed before rounding. Individual-seed win counts and paired bootstrap intervals are not inferred from aggregate means and standard deviations.

Figure 3: Opus training-suite and held-out ML score trajectories. Baseline and final endpoints are raw-space summary scores recomputed from Table 2 ; intermediate checkpoints retain their originally reported scores and precision. Policies are selected using training-suite scores, not the held-out outcomes shown here. PDE checkpoint aggregates are omitted because their per-task raw metrics are unavailable for this recomputation.

Condition	Train	Held-out ML
Static recipe	+0.00	+0.00
Static, tuned LR (oracle)	+3.46	+0.49
Frozen library recipe (no agent)	+9.71	-21.80
Single-agent: first change, then frozen	+10.72	+6.35
Single-agent: one decision	+9.43	+6.12
Single-agent: full agent (20 decisions)	+10.48	+8.71

Table 5: Reusable code and online adaptation with Opus 5.5. Full-agent and tuned-static rows use the raw-space summary scores recomputed from Table 2 ; other ML control scores retain their original reported values and precision. Frozen library: unedited evolved updater defaults, no agent. First change, then frozen: replay the first modified updater without further decisions. One decision: ask the agent once. PDE and combined held-out control aggregates are omitted because their per-task raw metrics are unavailable for consistent recomputation.

Policy	Train	Held-out ML
Single-agent (fixed prompt)	+10.48	+8.71
RSI-Opt (full policy)	+14.33	+8.23
$-$ library	+14.45	+8.00
$-$ playbook	+13.60	+8.06
$-$ evolved prompt	–	–

Table 6: Ablations of the evolved policy with Opus 5.5. Full-policy and baseline scores are recomputed from Table 2 ; ablation ML scores retain their original reported values and precision. Unavailable measurements are marked –. PDE and combined held-out aggregate columns are omitted because consistent raw-space scores for the ablation conditions cannot be reconstructed from the available summaries.

Component	Single-agent	RSI-Opt
Muon / orthogonalised update	35%	79%
Kronecker-factored (Shampoo / SOAP)	15%	10%
Second-order / L-BFGS-like	11%	10%
Clipping or update-size control	60%	79%
Per-update loss feedback (ctx.loss)	20%	10%
Uses a module of the evolved library	0%	82%

Table 7: What the agents wrote: fraction of final-evaluation runs (all models and tasks) whose final optimizer program contains each component (detected from the source), and decision statistics.

Model	Method	Input tok. (M)	Cached (M)	Output tok. (k)	Changing decisions
Opus 5.5	Single-agent	5.63	5.63	68	9.1
	RSI-Opt	4.02	4.02	40	3.5
GPT-5.5	Single-agent	8.90	8.49	79	11.6
	RSI-Opt	4.61	4.24	42	5.0
GPT-6.1	Single-agent	6.99	6.99	58	13.9
	RSI-Opt	3.65	3.65	24	4.2

Table 8: Agent usage per training run (mean over runs of the final evaluation): model calls are aggregated as input tokens (including cache reads), output tokens and decisions that changed the program.

Large Language Model-Guided Optimization

Large Language Model-Guided Optimization uses language models to propose, steer, and refine solutions to hard search and design problems.

See all 608 papers

Sep 21, 2026

RRSI: Regularized Recursive Self-Improvement of Agent Harnesses

An LLM agent's capability is largely magnified by its harness, namely the prompts, control flow, tooling, memory, and context management surrounding the frozen backbone model. Recent methods increasingly automate this process by iteratively proposing and selecting component-wise edits of an agent harness, practically establishing a form of recursive self-improvement (RSI) at the agent-system level. However, such recursive evolution may overfit by memorizing the training tasks, showing large in-distribution gains that shrink or even vanish on out-of-distribution benchmarks. We introduce Regularized Recursive Self-Improvement of Agent Harnesses (RRSI), which incorporates the principles of regularizations into harness self-improvement by constraining the evolution candidate proposal and selection. The proposer operates with a temporally annealed budget, limiting how many edits a candidate can bundle, and it encourages unexplored trajectories based on evolution history. The selector is equipped with a critic and a pruner: the critic screens benchmark-specific proposals, while the pruner, removes changes that are too small, too expensive, or no longer useful. Together these constraints favor reusable agent mechanisms over benchmark-specific ones or even noises. Across eight benchmarks spanning coding, agentic workspace and engineering design tasks, RRSI gains up to 6.0 points on the split it evolves against and up to 4.3 points on the five out-of-distribution benchmarks, while producing a harness that runs on 30% fewer policy tokens than the unregularized evolution. Code is available at https://github.com/google-research/rrsi and project page is https://regularized-rsi.com/.

Peng Xia, Rujun Han, Zifeng Wang +11

Google Cloud AI Research · UNC-Chapel Hill · Stanford University +1

OOD Generalization Agent Harness Optimization LLM Agent Self-Improvement Recursive Self-Improvement

Ask PDF

Oct 2, 2026

VERSE: Verified Self-Evolving Optimizer for Agent Harnesses

Harness evolution improves an LLM agent's prompts, tools, and workflow, while the optimizer's own tools and procedures often remain fixed. We study whether an optimizer can improve another agent more effectively by also improving how it diagnoses failures, develops edits, and tests their effects. Two observations guide our design. In a controlled study, optimizer self-evolution fails to improve performance without execution-based verification, but achieves the best result of that study when verification is available. Across five executors, self-evolving optimizers build their own tools for failure analysis, verification, training audits, and workflow control. Motivated by these findings, we introduce VERSE, a Verified Self-Evolving optimizer for agent harnesses. VERSE lets the optimizer test draft edits, replay failures, and perturb suspected steps before submission, while tracking fixes and regressions across rounds. Using this feedback, the optimizer revises both the executor harness and its own prompts, skills, tools, hooks, and notes, while the weights of the optimizer and executor models stay fixed. Under a shared protocol with disjoint training, validation, and test tasks, VERSE improves all four evaluated harness optimizers on held-out SWE-rebench tasks and newer out-of-distribution tasks in five languages. Its best validation-selected harness reaches 42.3% and 37.7% accuracy, respectively, against 39.2% and 29.3% for the strongest baselines. Code is available at https://github.com/wzekai/VERSE.

Zekai Wang, Yingqiang Ge, Zekun Wang +6

MIT · Amazon

Agent Harness Optimization Self-Evolving Agents Software Engineering Agents LLM Agent Verification

Ask PDF

Oct 26, 2026

Skill $^{\infty}$ : Recursive Self-Improvement for Agent Skill Optimization

Recursive self-improvement (RSI), in which a system also improves the way it improves, is a long-standing aspiration on the path to AGI. Language agents increasingly store procedural knowledge in reusable skills. Optimizing skills offers a direct way to achieve RSI, because the optimization procedure itself can be encoded as a skill. Existing skill optimizers, however, follow a hand-designed procedure. Even methods that evolve a meta-skill confine it to a fixed pipeline and relearn it for each task. We present Skill

^{\infty}

, which makes skill optimization recursively self-improving: a meta agent follows a meta-skill to optimize task skills and optimizes the meta-skill itself using the same procedure. The optimization is agent-led. The meta agent diagnoses execution traces, proposes candidates, requests independent reviews, and allocates the search budget. The meta-skill defines these steps, allowing self-improvement to rewrite them. This meta-skill is also decoupled from task skills, refined on multiple skill optimization tasks and then reused to optimize task skills across diverse domains. As in deployment, skills at both levels are standard packages loaded through native skill discovery. On 83 SkillsBench tasks, Skill

^{\infty}

reaches a pass rate of 60.7%, compared with 56.4% for human-curated skills and 54.5% for SkillEvolver.

Wangcheng Tao, Zhanglu Yan, Han Wu +1

National University of Singapore · City University of Hong Kong

LLM Agent Self-Improvement LLM Agent Skill Learning Recursive Self-Improvement

Ask PDF

Abstract

Large Language Model-Guided Optimization

Explore similar work

RRSI: Regularized Recursive Self-Improvement of Agent Harnesses

VERSE: Verified Self-Evolving Optimizer for Agent Harnesses

Skill∞^{\infty}∞: Recursive Self-Improvement for Agent Skill Optimization

Skill $^{\infty}$ : Recursive Self-Improvement for Agent Skill Optimization