When Better Gets Worse: Improvement Fidelity for Self-Improving Agents in Adaptive Worlds

Ke Wang; Zijie Zhao; Zhiyi Yuan; Changlun Li

cs.AISep 26, 2026

When Better Gets Worse: Improvement Fidelity for Self-Improving Agents in Adaptive Worlds

Authors: Ke Wang, Zijie Zhao, Zhiyi Yuan, Changlun Li

Organizations: University of Cambridge · Georgia Institute of Technology · Massachusetts Institute of Technology · University of Hong Kong ·

Abstract

Self-improving agents increasingly rely on proxy verifiers to choose policy updates, yet deployment can change the world in which those updates are evaluated. An update that looks better to the verifier can therefore become worse after deployment even when the verifier ranks policies well overall. We formalize this gap as Improvement Fidelity, which asks whether proxy improvements preserve the sign and ordering of deployment improvements over the updates an improvement process actually proposes. We show that global policy accuracy need not guarantee update fidelity: operator shift and deployment response can create update-level errors, while candidate margins determine whether those errors change the replacement decision. We introduce PIVOT-KG, a paired, decision-aware validator that allocates scarce high-fidelity evaluation according to the expected reduction in selection regret per unit cost. Across 90 held-out roots in Leduc, Kuhn, and Melting Pot, proxy and deployment optimal sets are disjoint in 51 cases. In an eight-candidate HighwayEnv stress test, PIVOT-KG reduces mean improvement-selection regret from 0.0435 under the exact Uniform validation rule to 0.0055 at the primary budget. Together, these results show why reliable self-improvement should evaluate proposed improvements in the worlds they induce, while providing a practical rule for allocating scarce deployment evidence when it can affect the replacement decision.

Figures & tables

Figure 1: Proxy improvements can reverse after deployment. Each point is a proposed update; lower-right points are proxy-positive but deployment-negative.

Figure 2: Operator shift exposes update-level failure. The horizontal axis is the design intensity

s^{2}

, not a measured

\chi^{2}

divergence. Global Spearman rank is a fixed calibration reference.

Figure 3: Decision relevance and HF resolvability. (a) Disjoint optimal sets. (b) Winner-relative correction-to-margin ratio versus normalized ISR reduction. (c) Model-assumed observation SD relative to the proxy top-2 margin versus normalized ISR reduction; filled markers hit the ex-post decision-critical pair.

World / response	HF queries	Uniform ISR	PIVOT-KG ISR	Uniform $-$ PIVOT [95% CI]
Leduc, long	2	0.046	0.009	0.036 [0.014, 0.064]
Kuhn, long	2	0.058	0.056	0.002 [0.000, 0.006]
Melting Pot, long	2	1.742	0.836	0.906 [-0.784, 3.401]

Table 1: Primary held-out fixed-budget comparisons (30 paired roots per row). Positive Uniform

-

PIVOT means lower ISR for PIVOT-KG; brackets give 95% root-bootstrap CIs.

Figure 4: Legacy batch-selector diagnostics at matched HF query counts in (a) Performative Control and (b) Congestion Resource. Lower single-decision ISR is better; the all-HF reference uses only fully queried rows. Intervals bootstrap seeds.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Response mechanisms. (a) Response-disabled proxy, actor, and strategic evaluations. (b) Strategic-minus-actor effects by opponent family. (c) Actor versus strategic improvement. Intervals use 30 matched seeds.

Figure 6: Leduc response-horizon confirmation. Reversal rates are computed among proxy-positive updates within each root, where a reversal means negative deployment gain. Bars show means across 30 roots under one-step and eight-step adaptation, with 95% root-bootstrap CIs.

World / response	HF queries	Uniform ISR	PIVOT-KG ISR	Uniform $-$ PIVOT [95% CI]
Melting Pot, long	1	5.283	1.859	3.423 [0.390, 6.944]
Melting Pot, short	5	1.035	1.610	-0.575 [-1.042, -0.180]

Appendix

Table 2: Secondary held-out comparisons. Positive Uniform

-

PIVOT means lower ISR for PIVOT-KG; brackets give 95% root-bootstrap CIs.

Figure 7: Leduc confirmation-cohort component contrasts under long response (30 roots; primary budget 16,384 HF hands). Positive values favor the first term. The expected-Uniform comparison is prespecified primary; all others are secondary with unadjusted 95% CIs.

Figure 8: Legacy forced-replacement trajectories. Top: cumulative true improvement; bottom: candidate-only

\mathrm{CISR}_{C}

, excluding no-update. Thin curves are seeds; markers summarize batch decision rules.

Figure 9: Legacy batch-selector Monte Carlo diagnostics on 24 test groups. Panels show query agreement across sample counts and regret/query variation under common cost scaling with different random draws; this is not an identified cost-misspecification effect.

Cohort	$B$	Roots	Uniform ISR	PIVOT-KG ISR	Uniform $-$ PIVOT [95% CI]
Original	1	60	0.0492	0.0718	$-0.0226$ $[-0.0439,-0.0008]$
Original	2	60	0.0422	0.0374	$0.0049$ $[-0.0088,0.0199]$
Original	4	60	0.0144	0.0029	$0.0115$ $[0.0008,0.0237]$
Replication	2	60	0.0254	0.0261	$-0.0006$ $[-0.0143,0.0145]$
Replication	4	60	0.0037	0.0004	$0.0034$ $[-0.0007,0.0099]$
Redesign (8)	2	120	0.0613	0.0157	$0.0456$ $[0.0343,0.0587]$

Appendix

Table 3: HighwayEnv cohorts. Positive Uniform

-

PIVOT means lower ISR for PIVOT-KG; brackets give paired 95% bootstrap CIs. Bold budgets mark the primary comparison within each cohort.

Figure 10: MetaDrive stress test as a boundary case. The prespecified

B=2

comparison remains unresolved:

+1.89\,[-1.29,5.49]

, with the 95% root-bootstrap CI crossing zero. (a) Selected deployment gain (12 profiles, 30 held-out roots; marginal 95% root-bootstrap intervals). (b) Paired PIVOT-KG–Uniform selected deployment gain contrasts;

B=1,4

are secondary.

Self-Improving Agents

Self-improving agents use experience and feedback to update their models, memories, skills, or workflows for better future task performance.

See all 167 papers

Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents
Jul 27, 2026Diandian Guo, Cong Cao, Fangfang Yuan +3AI Agent Evaluation LLM Agent Self-Improvement
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
Self-Improvements in Modern Agentic Systems: A Survey
Jul 14, 2026Zhe Ren, Yimeng Chen, Dandan Guo +9LLM Agent Self-Improvement Self-Improving Agents
School of Artificial Intelligence, Jilin University · King Abdullah University of Science and Technology (KAUST) · Independent Researcher +2
Which Self-Improvements Should We Trust? Reliable Self-Improvement When Agents Reuse Their Benchmarks
Sep 27, 2026Xiaojing Sun, Yuhan Zeng, Zihua She +1Benchmark Contamination Automated Evaluation
Purdue University

Jul 27, 2026J/K move · Enter open · S save

Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents

Diandian Guo, Cong Cao, Fangfang Yuan, Yingqi Wang+2

Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China

AI Agent Evaluation LLM Agent Self-Improvement LLM Agent Verification Self-Improving Agents

Open paper AskPDF arXiv

Self-improving agents accumulate capability by repeatedly rewriting procedural policies, controllers, or heuristic rules. They typically rely on self-authored tests or metrics to decide whether to accept subsequent edits. The agent controls both the optimized object and its verifier. As a result, self-assigned scores can remain near perfect while real deployment performance degrades or stays low. We study this problem through the verifier--deployment gap. This gap refers to the discrepancy between an agent's self-authored verification signal and a sealed deployment evaluation that the agent cannot observe or access. We ask how self-authored verification fails under iterative policy-and-test rewriting, how the failure changes with capability, and how little exogenous trust is sufficient to prevent real regressions from being deployed. To address this problem, we introduce a Sealed Exogenous Acceptance Loop (SEAL). SEAL retains self-authored tests but compares each candidate with the incumbent through a fixed harness-side audit. The agent cannot author or inspect the audit, receives only accept/reject, and the whole incumbent state is retained after a clear regression. Our experiments show that this problem often appears in heuristic learning settings. These settings require trial-and-error discovery of the target objective. We further find that failures of self-written verification are stratified by capability. Weaker agents tend to damage previously acquired strategies behind easy self-tests. Stronger agents are more stable, but they still mismeasure the deployment distribution. Standard self-written constraints do not reliably close this gap. In contrast, SEAL outperforms unprotected baselines across six models and three random seeds. Reliable self-improvement need not abandon self-verification, but it requires at least one deployment-acceptance signal outside the agent's control.

Abstract

Self-Improving Agents

Explore similar work

Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents

Self-Improvements in Modern Agentic Systems: A Survey

Which Self-Improvements Should We Trust? Reliable Self-Improvement When Agents Reuse Their Benchmarks