Which Self-Improvements Should We Trust? Reliable Self-Improvement When Agents Reuse Their Benchmarks

Xiaojing Sun; Yuhan Zeng; Zihua She; Xiao Wang

stat.MLSep 27, 2026

Which Self-Improvements Should We Trust? Reliable Self-Improvement When Agents Reuse Their Benchmarks

Authors: Xiaojing Sun, Yuhan Zeng, Zihua She, Xiao Wang

Organizations: Purdue University

Abstract

As recursive self-improvement (RSI) rapidly advances, reliable evaluation becomes critical for guiding adaptive search. RSI typically relies on finite evaluation resources, such as fixed benchmarks, to determine which modifications are retained and what is proposed next. However, when these finite resources are repeatedly reused, new candidates are proposed based on feedback from the same evaluation set, so the search trajectory can adaptively overfit and empirical improvement may not reflect genuine population improvement on the underlying task distribution. Some existing methods account for multiple comparisons but assume that candidates are chosen independently of the evaluation set, and therefore do not control this adaptive dependence. To address this, we propose REUSE (Risk-controlled Evaluation Under Sequential Evolution), a certified evaluation and promotion framework that allows a fixed evaluation set to support repeated adaptive decisions while providing statistical guarantees. For a user-specified error level $α$ , with probability at least $1-α$ , every promoted modification is a genuine population improvement on the underlying task distribution. REUSE achieves this by strictly limiting the evaluation feedback returned to the search process and accounting for possible promotion histories within the error budget. We develop detailed statistical theory for RSI evaluation in this setting, including simultaneous error control, valid lower bounds on cumulative improvement, and a characterization of the fundamental limits of adaptive evaluation reuse. In live self-improvement experiments, REUSE commits substantially fewer false promotions than evaluation frameworks from current RSI systems and error-controlled baselines, reducing the proportion of false promotions from up to 20.7% to 0%, while achieving final true population performance comparable to the best baselines.

Figures & tables

Figure 1: Overview of Reuse . (a) At round $t$ , the proposer generates $K$ candidate modifications using development data and earlier promotion decisions. (b) Reuse compares each candidate with the current system $A_{t-1}$ on a fixed evaluation set $S$ of $n$ tasks that is reused in every round. (c) Only the promotion decision $y_{t}$ is returned to the proposer. With probability at least $1-\alpha$ , every promotion is a true improvement in population performance, $V(A_{t})>V(A_{t-1})$ , simultaneously over the entire run, and the cumulative improvement $V(A_{t})-V(A_{0})$ is certified to be at least $C_{t}$ .

Method	Select.	Repeat.	Adapt.	Pop. Impr. $\uparrow$	Prom.	False $\downarrow$	Opt. Gap $\downarrow$	Pop. Impr. $\uparrow$	Prom.	False $\downarrow$	Opt. Gap $\downarrow$
				$n=2{,}000$				$n=10{,}000$
Empirical best-of- $K$	✗	✗	✗	7.04 $\,\pm\,$ 0.08	13.0	75	0.43 $\,\pm\,$ 0.11	7.19 $\,\pm\,$ 0.06	14.2	71	0.15 $\,\pm\,$ 0.06
SICA-style	✗	✗	✗	6.36 $\,\pm\,$ 0.18	12.3	72	0.34 $\,\pm\,$ 0.10	6.56 $\,\pm\,$ 0.16	12.5	62	0.14 $\,\pm\,$ 0.05
DGM-style	✗	✗	✗	3.89 $\,\pm\,$ 0.17	6.4	34	0.06 $\,\pm\,$ 0.11	4.03 $\,\pm\,$ 0.18	7.2	18	0.01 $\,\pm\,$ 0.05
Elite	✗	✗	✗	6.72 $\,\pm\,$ 0.14	14.3	89	0.47 $\,\pm\,$ 0.12	6.85 $\,\pm\,$ 0.12	13.7	59	0.16 $\,\pm\,$ 0.05
Niche-elite	✗	✗	✗	6.46 $\,\pm\,$ 0.15	12.3	71	0.40 $\,\pm\,$ 0.13	6.37 $\,\pm\,$ 0.15	12.4	44	0.13 $\,\pm\,$ 0.05

Table 1: Live self-improvement on Covertype over 30 seeds at each evaluation-set size, with all methods run on the same seeds. Pop. Impr. is the final population improvement of the deployed system over starting system

A_{0}

in percentage points (pp), estimated on the held-out set

H

and reported as mean

\pm

standard error. Prom. is the mean number of promotions per run, and False is the number of promotions with non-positive population improvement, summed over the 30 runs. Opt. Gap is the final improvement measured on

S

minus the population improvement measured on

H

, which measures how much the reused evaluation set overstates the final improvement. Select., Repeat., and Adapt. indicate whether a rule controls errors from within-round selection, repeated testing, and adaptive dependence, the three sources of false promotion discussed in Section 1 .

Figure 2: Population improvement of the deployed system over starting system

A_{0}

across rounds, estimated on

H

and averaged over 30 seeds, in percentage points (pp). The left panel uses

n=2{,}000

and the right panel uses

n=10{,}000

Figure 3: Population improvement of every promoted modification over the system it replaces, estimated on $H$ and pooled over 30 seeds, in percentage points (pp). Each point is one promotion, and red triangles mark false promotions, whose population improvement is at most zero. Within each panel, the left column uses $n=2{,}000$ and the right column uses $n=10{,}000$ .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Data
Quantity	Setting
Training set $D_{\mathrm{tr}}$	50,000 examples
Development set $D_{\mathrm{dev}}$	20,000 examples
Evaluation pool $\mathcal{E}$	60,000 examples
Held-out set $H$	50,000 examples
Evaluation-set size $n$	$2{,}000$ or $10{,}000$

Appendix

Table 2: Protocol shared by all methods in the live experiments.

Information in the prompt	Compared methods	Reuse
Parent configuration	✓	✓
Accuracy and slice error rates on $D_{\mathrm{dev}}$	✓	✓
Accuracies of past proposals on $D_{\mathrm{dev}}$	✓	✓
Evaluation accuracies on $S$	✓	✗
Promotion decisions	✓	✓
Development-draft updates and submissions	✗	✓

Appendix

Table 3: Information in the prompt of the proposer. Only Reuse withholds evaluation accuracies on

S

, so its proposer learns about

S

solely through promotion decisions. SICA-style additionally sees up to ten archive members with the highest evaluation accuracy.

Method	Feedback from $S$	Promotion rule	Select.	Repeat.	Adapt.
Empirical best-of- $K$	scores	largest positive gain on $S$	✗	✗	✗
SICA-style	scores	largest positive gain on $S$	✗	✗	✗
DGM-style	scores	largest positive gain on $S$	✗	✗	✗
Elite	scores	largest positive gain on $S$	✗	✗	✗
Niche-elite	scores	largest positive gain on $S$	✗	✗	✗
PACE-style	scores	McNemar test at level $0.05$	✗	✗	✗

Appendix

Table 4: Compared methods in the live experiments. Select., Repeat., and Adapt. indicate whether a rule controls errors from within-round selection, repeated testing, and adaptive dependence, as in Table 1 .

	Diff. vs Reuse		Worst $\uparrow$
Method	$n=2{,}000$	$n=10{,}000$	$n=2{,}000$	$n=10{,}000$
Empirical best-of- $K$	$+0.02$ $\,\pm\,$ 0.25	$+0.00$ $\,\pm\,$ 0.15	$-0.89$	$-0.61$
SICA-style	$-0.66$ $\,\pm\,$ 0.29	$-0.63$ $\,\pm\,$ 0.19	$-0.63$	$-0.42$
DGM-style	$-3.13$ $\,\pm\,$ 0.31	$-3.16$ $\,\pm\,$ 0.23	$-1.44$	$-0.21$
Elite	$-0.30$ $\,\pm\,$ 0.28	$-0.34$ $\,\pm\,$ 0.18	$-1.12$	$-0.44$
Niche-elite	$-0.56$ $\,\pm\,$ 0.34	$-0.82$ $\,\pm\,$ 0.19	$-1.13$	$-0.41$

Appendix

Table 5: Additional summaries of the live experiments over 30 seeds at each evaluation-set size, in percentage points. Diff. is the seed-matched difference in final population improvement between a method and Reuse , reported as mean

\pm

standard error, so a positive value means the method ends higher. Worst is the smallest population improvement among all promotions of a method, pooled over runs.

	$S$ tests	Prom.	False /	Pop. Impr.	Diff.	Changed	$C_{T}$ at default’s
Setting	per run	per run	all	(pp)	(pp)	runs	promotions (pp)
Default	4.30 $\,\pm\,$ 0.37	2.90 $\,\pm\,$ 0.28	0 /29	6.38 $\,\pm\,$ 0.23	N/A	N/A	0.74 $\,\pm\,$ 0.11
Split $\rho$
$\rho=0.1$	4.00 $\,\pm\,$ 0.30	3.00 $\,\pm\,$ 0.15	0 /30	7.23 $\,\pm\,$ 0.36	$+0.85$ $\,\pm\,$ 0.40	9/10	0.89 $\,\pm\,$ 0.11
$\rho=0.25$	3.90 $\,\pm\,$ 0.18	2.60 $\,\pm\,$ 0.16	0 /26	6.52 $\,\pm\,$ 0.27	$+0.14$ $\,\pm\,$ 0.37	4/10	0.84 $\,\pm\,$ 0.11
$\rho=0.75$	4.80 $\,\pm\,$ 0.33	3.00 $\,\pm\,$ 0.26	0 /30	6.52 $\,\pm\,$ 0.35	$+0.14$ $\,\pm\,$ 0.29	1/10	0.59 $\,\pm\,$ 0.10

Appendix

Table 6: Sensitivity of Reuse to its allocation ( 4 ) at

n=2{,}000

over 10 new seeds. We change one of

\rho

w_{t}

v_{p}

at a time from the default. The variable

t

indexes the rounds where

S

is queried and the harmonic weight uses

c\approx 0.590747

. The columns for

S

tests, Prom., Pop. Impr. and

C_{T}

are means

\pm

standard error over seeds. The False / all column sums false and all promotions over the 10 runs. Diff. is the seed-matched difference from the default run. Changed counts the runs in which the default allocation would have made at least one querying or promotion decision differently.

C_{T}

re-prices the promotions of the same-seed default run at the certificate level

(1-\rho)\delta_{t,p}

. All improvements are in percentage points.

Method	Prom.	False $\downarrow$	Runs $\downarrow$	Prom.	False $\downarrow$	Runs $\downarrow$
	$n=2{,}000$			$n=10{,}000$
Empirical best-of- $K$	13.0	75	30/30	14.2	71	28/30
SICA-style	12.3	72	30/30	12.5	62	26/30
DGM-style	6.4	34	22/30	7.2	18	15/30
Elite	14.3	89	27/30	13.7	59	25/30
Niche-elite	12.3	71	28/30	12.4	44	24/30

Appendix

Table 7: False promotions of the alternative Reuse variant and of the seven compared methods over the 30 seeds at each evaluation set size. Prom. and False are identical to the metrics in Table 1 . Runs is the number of runs out of 30 with at least one false promotion.

Figure 4: Performance of Reuse with and without its test on

S

n=10{,}000

over 10 seeds using a 500 example development set and pre-screening threshold

z_{\mathrm{dev}}>0

. Each point is a submitted candidate placed at its gain on

S

and its population improvement over the incumbent estimated on

H

. The shaded band marks population improvement at most zero and the dashed line is equality. The left panel shows Reuse whose test on

S

promotes the filled points and rejects the open circles. The right panel shows the same rule without the test which promotes every submitted candidate. Red triangles are false promotions.

	Empirical		SICA-		DGM-				Niche-		PACE-		Bonferroni-		Reuse
Seed	best-of- $K$		style		style		Elite		elite		style		style		(ours)
3	7.14	( 1 /9)	5.76	( 3 /11)	4.65	( 1 /8)	7.14	( 3 /13)	7.17	( 3 /20)	7.22	(0/9)	0.00	(0/0)	7.24	(0/3)
4	7.25	( 2 /12)	7.24	( 3 /14)	5.26	(0/5)	7.25	(0/9)	7.24	( 2 /9)	5.75	(0/5)	3.21	(0/2)	5.73	(0/3)
5	7.06	( 3 /14)	7.41	( 3 /16)	2.77	( 2 /7)	4.65	( 4 /15)	6.81	( 1 /14)	3.79	(0/5)	0.00	(0/0)	7.26	(0/4)
6	6.70	( 3 /16)	5.55	( 2 /11)	2.33	( 2 /4)	6.70	( 5 /19)	6.70	( 6 /19)	6.72	( 1 /8)	4.11	(0/2)	6.70	(0/4)
7	7.14	( 3 /15)	5.92	( 2 /14)	3.21	( 1 /9)	5.92	( 5 /18)	6.50	( 3 /13)	4.65	(0/4)	0.86	(0/1)	10.00	(0/5)

Appendix

Table 8: Per-seed results at

n=2{,}000

for the 30 seeds of Table 1 . For each of the eight methods, a cell gives the final population improvement of the deployed system over

A_{0}

in percentage points, estimated on the held-out set

H

, followed in parentheses by the run’s false promotions and all promotions; nonzero false counts are in red. Mean averages the final population improvement over the 30 seeds and Total sums the promotion counts. Mean and the false counts in Total equal the corresponding columns of Table 1 , and the total promotions divided by 30 give its Prom. column.

	Empirical		SICA-		DGM-				Niche-		PACE-		Bonferroni-		Reuse
Seed	best-of- $K$		style		style		Elite		elite		style		style		(ours)
3	7.17	( 1 /17)	4.65	(0/5)	2.92	( 1 /7)	7.41	( 4 /20)	7.24	( 2 /10)	7.24	(0/9)	7.10	(0/7)	6.70	(0/5)
4	7.37	( 1 /13)	7.44	( 2 /14)	5.27	( 1 /8)	7.27	( 3 /16)	5.61	(0/7)	7.37	(0/7)	7.24	(0/7)	7.25	(0/8)
5	6.64	( 2 /14)	7.37	( 2 /13)	2.19	(0/5)	6.45	( 2 /8)	5.80	(0/9)	7.24	(0/9)	7.10	(0/7)	7.37	(0/6)
6	7.41	( 2 /15)	7.34	( 2 /13)	5.46	(0/7)	7.24	( 1 /9)	6.57	( 2 /20)	7.27	(0/10)	4.65	(0/4)	7.34	(0/7)
7	7.41	( 3 /17)	5.92	( 1 /13)	4.65	( 1 /8)	6.62	( 4 /21)	5.18	(0/13)	7.24	(0/9)	7.05	(0/7)	7.24	(0/6)

Appendix

Table 9: Per-seed results at

n=10{,}000

for the 30 seeds of Table 1 . For each of the eight methods, a cell gives the final population improvement of the deployed system over

A_{0}

in percentage points, estimated on the held-out set

H

Figure 5: Mean population improvement of the deployed system over

A_{0}

by round, estimated on

H

and averaged over 30 seeds, with one panel per method; dashed lines use

n=2{,}000

and solid lines use

n=10{,}000

. Shaded bands show

\pm 1

standard error across the 30 seeds.

Figure 6: Final population improvement over

A_{0}

against the number of promotions in the run, estimated on

H

, one point per run with small horizontal jitter, for the 30 runs of each method at each evaluation-set size. Light gray points are the other benchmark-driven frameworks (SICA-style, DGM-style, Elite and Niche-elite), and outlined markers place Empirical best-of-

K

, PACE-style, Bonferroni-style and Reuse at their median number of promotions and median final population improvement. The arrow joins the medians of Empirical best-of-

K

and Reuse .

$n$	Final population improvement	Promotions per run	Running certificate $C_{T}$	Direct certificate $D_{T}$	Valid $C_{T}$ / $D_{T}$
$2{,}000$	7.02	3.2	0.92 [0.13, 1.76]	3.10 [1.16, 5.49]	30 / 30
$10{,}000$	7.19	5.9	2.67 [2.06, 3.88]	5.33 [4.34, 6.92]	30 / 30

Appendix

Table 10: Certificates of cumulative improvement for Reuse , over the 30 runs at each evaluation-set size. The final population improvement over

A_{0}

is the mean over runs, estimated on

H

as in Table 1 .

C_{T}

and

D_{T}

are reported as the mean over runs with the range across runs in brackets. Valid gives, for

C_{T}

and for

D_{T}

, the number of runs (of 30) in which the certificate is at most the realized final population improvement. The final population improvement,

C_{T}

and

D_{T}

are in percentage points.

Explore similar work

Jul 8, 2026cs.AI

Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops

AI systems increasingly participate in their own improvement: revising their outputs, adapting their own harnesses during deployment, training on data they generate, and, increasingly, conducting AI research itself. This literature is described under a vocabulary ("self-refine," "self-reward," "self-play," "self-evolve") that conflates fundamentally different ambitions. We survey 1,250 arXiv papers (2024-2026) along two axes: what the system improves -- its behavior in deployment, its policy through training, its evaluator, or the research process itself -- and the degree of loop closure (human-in-the-loop to fully closed). The taxonomy separates bounded self-refinement -- convergent, evaluable, and already industrial practice -- from open-ended recursive self-improvement (RSI), which remains bounded by grounding requirements, collapse dynamics, and compute constraints on every measured axis. Its distinctive feature is a dedicated category for self-evaluation: every improvement loop is a claim that some signal can substitute for human judgment. We survey the evaluator design space -- judges, process reward models, verifiers, rubrics, meta-evaluation -- order the signals into a verification hierarchy from formal verifiers (strongest) to intrinsic self-assessment (weakest), and observe that demonstrated self-improvement strength tracks this hierarchy, that its failure modes (self-confirming loops, model collapse, diversity collapse) follow from its violations, and that the "research direction-setting" bottleneck keeping humans in the loop sits at the top of that hierarchy. We connect the technical literature to the theory of RSI limits and to the safety and governance questions raised by frontier-lab accounts of closing the loop, and identify governance-grade measurement of self-improvement as the field's most underpopulated niche.

Mingguang Chen, Licheng Wang, Bo Qu

University of California, Riverside (UCR) · AlphaAvatar · Illinois Institute of Technology (IIT)

Recursive Self-Improvement Self-Evaluation Artificial Intelligence Research Human-In-The-Loop Refinement Loop+1

Ask PDF

Jun 24, 2026cs.LG

The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators

Self-improving agents are state-of-the-art (SOTA) on agentic coding benchmarks and have recently been extended to general domains. However, their search methods generally assume a stationary evaluation criterion: a fixed verifier, benchmark, or labeled dataset that remains valid as the agent improves. This ignores a central feature of evolution: species adapt as their environments change with them. We aim to bring the same principle to recursive self-improvement, making evaluation part of the improvement loop and opening search to evolving evaluators, adversarial objectives, and dynamic utilities that may surpass static benchmarks. We introduce the Red Queen Godel Machine (RQGM), an evolutionary framework for recursive self-improvement under non-stationary utilities. The RQGM makes this possible through controlled utility evolution: search is organized into epochs with a fixed within-epoch evaluation criterion, while the utility can be updated at epoch boundaries, so self-improvement guarantees hold per epoch as the objective evolves across them. We begin by showing that even on verifiable coding tasks, the RQGM improves test pass rate over the prior SOTA by adding a complementary agent-as-a-judge code-review signal. This signal is cheaper and the RQGM uses 1.35x-1.72x fewer tokens. We then turn to scientific paper writing and reviewing, and Olympiad-level proof writing and grading, where the RQGM improves performance over prior self-improving agents: co-evolved writers reach 1.78x-1.86x higher acceptance rates under a diverse agent-as-a-judge panel, while co-evolved graders reach 9% higher ground-truth accuracy. In paper reviewing, the strongest baseline reviewer over-accepts AI-generated papers at up to 1.91x the human rate. The RQGM corrects this by introducing an adversarial objective that discovers reviewers equally stringent on AI and human work.

Alex Iacob, Andrej Jovanović, William F. Shen +10

University of Cambridge · 3Flower Labs · 2NVIDIA +2

Self-Improving Agents Self-Evolving Agents Agentic Evaluations Counter Turing Test Code Quality Epoch

Ask PDF

Sep 6, 2026cs.LG

MetaRSI / RSI2: A Meta-Recursive Self-Improving System for Recursive Self-Improving Systems Themselves

Recursive self-improvement (RSI) lets a system improve the model-building machinery from its own failures, so every later model inherits the gain. Yet RSI has been validated almost exclusively on coding and formal benchmarks such as science QA and mathematics. This format bound limits RSI to improvement within a machine-checkable slice, not general capability where questions are open and correctness is settled by argument, replication, or measurement. We argue RSI must next operate across real, diverse scientific, engineering, and meta-scientific domains, not where formal evaluation is merely tractable. To that end we present MetaRSI-v1, where improvement is the scheduled composition of three typed operators over one unified paradigm. Data-RSI amplifies existing competence and marks its boundary; Harness-RSI edits a five-slot scaffold without touching weights; Model-RSI internalizes capability into parameters through bounded training. Sharing one loop kernel and artifact vocabulary, they make data, scaffold, and model changes composable rather than exclusive. A two-axis optimizer jointly decides operator order and each operator's proposal policy, while a meta-level policy revises the schedule across terms. We validate MetaRSI-v1 under the field's standard evaluations, on code and closed-form science, with no external teacher: the target model plays every role in its own loop. MetaRSI-v1 reframes self-improvement from a single-surface edit to a composition across the full model-production pipeline, opening two paths: a model route internalizing capability through training, and a harness route leaving weights untouched and thus extending self-improvement to any model reachable through an interface, with Data-RSI redefined as the shared substrate feeding both. The framework further yields refutable laws on where loops exist, how operators compose, and what supervision buys.

Zihan Tan, Leixin Sun, Zitong Shi +28

Hugging Face

Recursive Self-Improvement Recursive Models Meta-Learning Mle-Bench Lite Scientific Discovery Dream

Ask PDF