Reliable Self-Evolution with Imperfect Proxy Rewards

Kangjun Noh; Soyu Kim; Kyungwoo Song

cs.AIOct 2, 2026

Reliable Self-Evolution with Imperfect Proxy Rewards

Authors: Kangjun Noh, Soyu Kim, Kyungwoo Song

Organizations: Yonsei University

Abstract

Large language model (LLM)-based self-evolving search is a promising approach to scientific discovery. However, high-fidelity evaluation of every candidate is prohibitively expensive in some domains. Self-evolving systems in such settings therefore rely on low-cost but imperfect proxy rewards, which may assign high scores to infeasible candidates. These false positives may contaminate both the final output and the feedback used to guide subsequent generations. This motivates statistically calibrated reward intervals for more reliable self-evolving search. We propose Conformal Interval-Driven Self-Evolution (CISE), which constructs candidate-specific reward intervals using conditional conformal inference and iteration-wise online density-ratio estimation. CISE uses conservative interval-based rewards for evolutionary feedback and returns candidates only when all required property intervals lie entirely within their respective feasible regions. We derive fixed-iteration coverage results under explicit assumptions of independence and covariate shift. We evaluate CISE on three self-evolving search tasks in materials science. In our experiments, all candidates returned by CISE are true positives under high-fidelity evaluation, whereas the baselines return more candidates but include false positives. These results highlight the value of a smaller, more precise shortlist when downstream validation budgets are limited. Our repository is available at https://github.com/MLAI-Yonsei/CISE.git.

Figures & tables

Figure 1: Overview of CISE. Offline, we fit the conditional proxy-error quantile using $\mathcal{D}_{\mathrm{fit}}$ , $\mathcal{D}_{\mathrm{cal}}$ provides calibration scores, and $\mathcal{D}_{\mathrm{src}}$ provides the source reference for density-ratio estimation. During evolution, unlabeled probes are used to estimate the current distribution shift, which is incorporated into conditional conformal inference to construct candidate-specific reward intervals. These intervals guide subsequent evolution and candidates are returned only when the entire interval satisfies the task constraint.

	Wide-bandgap (WBG)			Solid-state electrolytes (SSE)			Photovoltaic absorbers (PV)
Method	Output	TP	FP	Output	TP	FP	Output	TP	FP
MatterGen	3/200	2 (0.67)	1 (0.33)	2/200	1 (0.50)	1 (0.50)	0/200	–	–
LLMatDesign	37/200	0 (0.00)	37 (1.00)	2/200	0 (0.00)	2 (1.00)	18/200	0 (0.00)	18 (1.00)
LLEMA	102/200	94 (0.92)	8 (0.08)	104/200	71 (0.68)	33 (0.32)	60/200	39 (0.65)	21 (0.35)
CISE	41/200	41 (1.00)	0 (0.00)	51/200	51 (1.00)	0 (0.00)	6/200	6 (1.00)	0 (0.00)

Table 1: Comparison of output reliability across three material-discovery tasks. Each method generates 200 candidate structures. Output denotes candidates satisfying the method-specific output criterion. TP and FP denote output candidates that satisfy or violate the target constraints under DFT evaluation, respectively. Values in parentheses denote the fraction among output candidates.

Figure 2: Evolution of formation energy on the solid-state electrolyte (SSE) task, where success requires $E_{\mathrm{form}}\leq-1.0$ eV/atom. (a) Proxy rewards for LLEMA and reward intervals for CISE; vertical blue bars denote CISE intervals and the dashed blue curve shows their mean upper bound. (b) Corresponding DFT evaluations. Circles and crosses denote selected and non-selected candidates, respectively, and solid curves show the 20-iteration output mean. In later iterations, LLEMA produces a pronounced concentration of candidates above the DFT success threshold, visible in the upper-right of (b), including selected false positives. In contrast, CISE shows no comparable accumulation and its output mean remains within the feasible region throughout evolution.

	LLEMA: top-6 closest to $1.2$ eV				CISE: six returned candidates
Rank	Material	Proxy $E_{g}$	DFT $E_{g}$	Result	Material	CISE interval	DFT $E_{g}$	Result
1	SrO	1.24	2.99	FP	RbLiO	$[0.73,\,1.98]$	1.24	TP
2	ZnF 2	1.25	2.32	FP	RbNa 2 OF	$[1.34,\,1.51]$	1.32	TP
3	MgTiO 3	1.37	1.65	TP	KNa 2 OF	$[1.19,\,1.62]$	1.50	TP
4	MgO	1.54	1.39	TP	Na 3 Li 3 P 2	$[0.92,\,1.72]$	1.08	TP
5	MgZrO 3	1.57	1.07	TP	NaLi(MgP) 2	$[0.83,\,1.86]$	1.66	TP

Table 2: Equal post-evaluation budget comparison on the photovoltaic absorber (PV) task. To match the six candidates returned by CISE, we select six candidates from the LLEMA output whose proxy-predicted band gaps are closest to the target value of

1.2

eV. Importantly, this ranking is performed before DFT evaluation and therefore does not use the TP/FP labels. Under the same validation budget, three of the six LLEMA candidates are false positives under DFT, whereas all six candidates returned by CISE are true positives.

Notation	Description
Material search and rewards
$\mathcal{X}$ ; $X,x$ ; $X^{\prime}$	Space of material candidates; a random candidate and its realization; a fresh candidate at the current iteration.
$\hat{r}(x)$ ; $r(x)$	Computationally inexpensive proxy reward and expensive true reward of candidate $x$ .
$\mathcal{H}_{t}$	Archive after iteration $t$ , containing candidates and their stored rewards. Algorithm 2 stores proxy rewards, while Algorithm 1 stores rewards evaluated conservatively over prediction intervals.
$\pi_{t}(\cdot\mid\mathcal{H}_{t})$ ; $\mathcal{S}_{t}$	Parent-selection policy applied to $\mathcal{H}_{t}$ and the resulting set of $K$ parents used to guide the following iteration.
$\mathcal{C}_{t}$ ; $\mathcal{O}_{t}$	Candidate batch proposed at iteration $t$ and cumulative output set through that iteration.

Table 3: Notation used for proxy-reward-driven material evolution and conditional conformal inference.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Qualitative comparison of proxy-driven evolution and CISE on the photovoltaic absorber task, with target band gap $0.7\leq E_{g}\leq 2.0$ eV. (a) LLEMA reuses proxy-qualified candidates along the parent–child lineage $\mathrm{Na_{2}O}\rightarrow\mathrm{Li_{2}O}\rightarrow\mathrm{Na_{2}O}$ . All three candidates satisfy the point-estimate constraint but fail DFT validation (FP). The recorded LLM design justifications illustrate how previous surrogate evaluations inform subsequent proposals, allowing false positives to become reusable search feedback. The displayed CISE intervals extend beyond the target range, so these candidates would be excluded by the interval-based output rule. (b) A $\mathrm{RbLiO}$ candidate from a separate CISE run has its entire band-gap interval within the target range and also satisfies the constraint under DFT evaluation. Arrows indicate parent–child relations; checkmarks and crosses indicate acceptance and rejection, respectively.

Explore similar work

When Is Enough Enough in Self-Evolving LLM Systems?
Oct 3, 2026Enoch Yin, Bin Liu, Zhengling QiEarly Stopping Language Model Self-Improvement
Department of Statistics, The George Washington University · Department of Statistics and Data Science, School of Management, Fudan University · Department of Decision Sciences, School of Business, The George Washington University
Confidence-Orchestrated Self-Evolution against Uncertain LLM Feedback
May 27, 2026Bowen Wei, Nan Wang, Yuqing Zhou +2Self-Training for Language Models RL for Language Model Reasoning
George Mason University · Amazon
SMCEvolve: Principled Scientific Discovery via Sequential Monte Carlo Evolution
May 14, 2026Jiachen Jiang, Huminhao Zhu, Zhihui ZhuLLM-Based Program Synthesis Large Language Model-Guided Scientific Discovery
Department of Computer Science and Engineering The Ohio State University

Oct 3, 2026J/K move · Enter open · S save

When Is Enough Enough in Self-Evolving LLM Systems?

Enoch Yin, Bin Liu, Zhengling Qi

Department of Statistics, The George Washington University · Department of Statistics and Data Science, School of Management, Fudan University · Department of Decision Sciences, School of Business, The George Washington University

Early Stopping Language Model Self-Improvement LLM Evaluation Change-Point Detection Sequential Hypothesis Testing

Open paper AskPDF arXiv

Self-evolving large language model (LLM) systems repeatedly propose, evaluate, and incorporate updates to prompts, skills, or other persistent artifacts. Despite their growing effectiveness, these systems typically operate under a predetermined iteration or compute budget, without a principled criterion to determine when further evolution is no longer worthwhile. This can lead to two undesirable consequences: unnecessary computation after performance has saturated and the risk of returning late updates that overfit or exploit the evaluation signal. These issues motivate us to study two fundamental questions: when should a self-evolving system stop, and what should it output once it stops? We address the first by formulating an online sequential testing problem and constructing an anytime-valid restart detector using the per-item paired evaluation outcomes already produced by self-evolving LLM systems. We address the second by formulating a change-point estimation problem and using the estimated transition to select an earlier artifact for output. The resulting procedure is plug-and-play and requires no modification of the underlying self-evolving algorithms. Across two self-evolving frameworks, three LLM model families, and five benchmarks, our method substantially reduces computation costs while maintaining comparable unseen-test performance. For example, on SearchQA with SkillOpt and DeepSeek V4 Flash, our method stops at round 4 rather than the full budget of 40, reducing token usage by 91.6% while achieving 82.43% unseen-test accuracy versus 82.00% under the full-budget run.

Abstract

Explore similar work

When Is Enough Enough in Self-Evolving LLM Systems?

Confidence-Orchestrated Self-Evolution against Uncertain LLM Feedback

SMCEvolve: Principled Scientific Discovery via Sequential Monte Carlo Evolution