Trustworthy Recursive Self-Improvement in the Era of Large Models: Foundation, Risks, Evaluation and Open Challenges

Jiayue Pu; Ang Gao; Jinyi Bai; Qipeng Wang; Tongyu Wen; Jiahao Zhao; Keyu Qu; Junhao Zhang

cs.AIOct 26, 2026

Trustworthy Recursive Self-Improvement in the Era of Large Models: Foundation, Risks, Evaluation and Open Challenges

Authors: Jiayue Pu, Ang Gao, Jinyi Bai, Qipeng Wang, Tongyu Wen, Jiahao Zhao, Keyu Qu, Junhao Zhang,

Organizations: Gaoling School of Artificial Intelligence, Renmin University of China · University of Chinese Academy of Sciences

Abstract

AI systems built around large models increasingly use experience and feedback to make persistent changes to models, prompts, memories, tools, and workflows. Recursive self-improvement (RSI) extends this process to inductive biases and to the mechanisms that govern future updates, which can themselves be revised and reused across improvement cycles. Yet self-modification alone does not guarantee trustworthy improvement. Reported gains may reflect flawed evaluations, previously acquired capabilities may deteriorate, and compliance with behavioral constraints may weaken. Compromised artifacts or sensitive information may also persist in successor systems. Changes to improvement mechanisms may invalidate earlier verification results and make human oversight less effective. This survey presents an operational definition and process framework for RSI and organizes the requirements for trustworthy RSI into four interconnected themes: genuine and sustained improvement; behavioral constraints, alignment, and fairness; system security and privacy; and transparency and controllability. We examine these themes from three perspectives: Improvement Validity, Constraint Preservation, and Effective Oversight. This analysis covers improvement diagnosis, proposal generation, proposal evaluation, and update implementation (S1--S4), including the recursive application of these stages to improvement mechanisms (RI). We review risks, safeguards, and evaluation evidence for individual updates, effects inherited by successor systems, and revisions to improvement mechanisms, while distinguishing direct RSI findings from evidence in related systems. Finally, we examine evaluation settings, identify requirements for independent and longitudinal assessment, and discuss applications and open research problems.

Figures & tables

Figure 1: Overview of the RSI Process and Trustworthiness Framework. The numbered labels (§2–§8) indicate the corresponding sections of this survey. The central panel (Section 2) presents the definition and four-stage improvement cycle of RSI (S1–S4), including recursive revision and reuse of the bias update function (RI). Four surrounding trustworthiness themes are organized under three complementary analytical roles: Improvement Validity, Constraint Preservation, and Effective Oversight. Section 7 examines evaluation designs and evidence requirements at the individual-update, successor, and meta-improvement levels, while §8 covers representative applications and open research problems.

Survey	Main analytical focus	MT	IL	MR	LA	JE
Trustworthiness-oriented surveys
Li et al. (2023)	Trustworthiness principles and practices for AI	$\checkmark$	—	—	—	$\triangle$
Liu et al. (2023)	Trustworthiness requirements and methods for LLMs	$\checkmark$	—	—	—	$\triangle$
Yu et al. (2025)	Trustworthiness of LLM agents	$\checkmark$	—	—	—	$\triangle$
Qi et al. (2026)	Trustworthiness of agentic workflows	$\checkmark$	$\triangle$	—	$\triangle$	$\triangle$
Self-evolution and RSI-oriented surveys

Table 1: Comparison with representative surveys of trustworthy AI, self-evolution, RSI, and evolutionary safety. Symbols indicate the depth of analysis defined in the legend.

Approach	Who determines the inductive bias $B$ ?	Is $B$ revised autonomously?	Is the bias update function revised and reused?
Conventional machine learning	Human designer	✗	✗
Reinforcement learning	Human designer	✗	✗ (learning rule fixed)
Meta-learning	Meta-learner	✓	✗ (meta-update rule fixed)
AutoML	Automated search	✓	✗ (search rule fixed)
RSI	Bias update function $\mathcal{U}_{t}$	✓	✓ (the revised $\mathcal{U}_{t}$ guides later bias updates)

Table 2: Comparison of RSI with other machine learning approaches in their standard forms.

Figure 3: The recursive self-improvement (RSI) process. The core improvement cycle (S1–S4) identifies limitations, generates and evaluates candidate revisions to the policy $h_{t}$ or inductive bias $B_{t}$ , and incorporates accepted changes into the successor system. Recursive improvement (RI) applies the same four stages to the bias update function $\mathcal{U}_{t}$ itself. The resulting $\mathcal{U}_{t+1}$ is retained for subsequent improvement cycles, while components without accepted revisions remain unchanged.

Analytical role	Theme	Requirement groups	RSI-specific question
Improvement Validity	Genuine and sustained improvement Section 3	(i) Validity, reliability, robustness, and truthfulness	Are improvements genuine and sustained, including after revisions to the update mechanism?
Constraint Preservation	Behavioral constraints, alignment, and fairness Section 4	(ii) Behavioral safety and value alignment; (iii) fairness	Do behavioral requirements remain satisfied across successive revisions?
Constraint Preservation	System security, resilience, privacy, and data governance Section 5	(iv) System security and resilience; (v) privacy and data governance	Are inherited components and information protected during subsequent use?
Effective Oversight	Transparency and controllability Section 6	(vi) Transparency and interpretability; (vii) human control and corrigibility; (viii) accountability and governance	Can revisions, including changes to the update mechanism, be reviewed and corrected?

Table 3: Mapping the eight trustworthiness requirement groups to four RSI themes. The three analytical roles are complementary and do not represent separate stages of the improvement process.

Figure 4: Genuine and Sustained Improvement Across S1–S4 and RI in the Process of RSI. Stage-specific risks and checks address biased diagnosis, shortcut search, misleading acceptance scores, and ineffective updates. The lower panel compares the gains, retention, and transfer of later updates produced by the original and revised improvement mechanisms, starting from matched initial states under common tests and resource budgets.

Study and setting	Modified object / reuse	Horizon	Comparison and resource evidence
SICA ( Robeyns et al., 2025 ) Claude 3.5 / o3-mini; SWE-bench, LiveCodeBench	Agent code , prompts, tools; $\checkmark$ revised agents write later edits (archive selection fixed)	15 self-edits	Initial vs. later agents: 17% $\rightarrow$ 53% at peak on 50 SWE-bench tasks; $\sim$ $7k API; no fixed-updater ablation
Agent Plasticity ( Singh et al., 2026 ) Frontier LMs; chess, Go, Hex, NetHack	Tools, skills, notes inherited; – improvement protocol fixed (weights fixed)	20/40 rounds	Held-out checkpoint tests; gain per dollar , including reflection and rejected edits
MASS ( Lee et al., 2026a ) Qwen3.6-27B; finance, robotics, pharmacy	Workflows and weights ; $\checkmark$ updated model reused as optimizer and evaluator (outer loop fixed)	2 cycles	Base vs. trained models; external judgments; token efficiency reported, training cost unmatched
Hyperagents ( Zhang et al., 2026f ) Claude 4.5; paper review, robotics rewards	Task/meta-agent code ; $\checkmark$ revised meta-agent reused across domains	100 searches	Fixed-meta ablation and search controls; $\sim$ 33M self-edit tokens per 100 iterations, evaluation excluded
Evo 2 ( Xu et al., 2026b ) Qwen3.8-27B; Polyglot, SWE-bench, LoCoMo	Harness / Proposer–Reflector weights ; $\checkmark$ trained updater reused in later searches	Repeated ; total not reported	Trained vs. untrained evolver ; parents, executor, search fixed; training excluded from inference cost

Table 4: Representative evidence on improvement and mechanism reuse. Horizons use study-specific units;

\checkmark

marks a revised updater reused in later updates, and – marks artifact reuse under a fixed protocol.

Figure 5: Behavioral Constraints, Alignment, and Fairness Across S1–S4 and RI in the Process of RSI. Behavioral requirements, authorized human goals, and fairness criteria apply throughout the four improvement stages. Each stage pairs a potential failure with a corresponding safeguard. The lower panel examines whether changes to the improvement mechanism remain authorized and whether later successors continue to satisfy the applicable behavioral, alignment, and fairness requirements.

Study	Tested setting	Horizon / controls	Key finding
Behavioral constraints
Qi et al. (2024)	GPT-3.5 Turbo / Llama-2-7B; 330 prompts, 11 prohibited categories	1 epoch ; before/after on the same safety test and scoring rule	Harmful compliance $\uparrow$ after benign tuning
Zhao et al. (2026b)	GPT-4o web agent; BrowserART / WebArena	106 steps ; weights and tests fixed; execution / refusal memories varied	Unsafe actions $\downarrow$ , but over-refusal $\uparrow$
Alignment
Langosco et al. (2022)	CoinRun; relocated coin vs. level completion	Zero-shot ; trained policy and reward fixed	Navigation retained — relocated coin often passed
Khalaf et al. (2025)	GPQA; proxy reward vs. answer accuracy	Up to 32 samples ; questions, response pool, reward model fixed	Accuracy peaks then declines with stronger selection

Table 5: Representative behavioral, alignment, and fairness findings. Studies are grouped by tested property; horizons and fixed conditions follow the original experiments.

Figure 6: System Security, Resilience, Privacy, and Data Governance Across S1–S4 and RI in the Process of RSI. The two rows distinguish security and resilience from privacy and data governance. Stage-specific risks and safeguards cover compromised experience, malicious or sensitive derived assets, unreliable admission evidence, and inherited threats or information exposure. The lower panel outlines independent approval, renewed protection checks, and tests of successors after the improvement mechanism is revised and reused.

Figure 7: Transparency and Controllability Across S1–S4 and RI in the Process of RSI. The four stages connect testable diagnoses, proposed interventions, independent evaluation, and enforceable correction. Cross-stage oversight requires testable explanations, traceable sources and revisions, and independent review. Authorized interventions must reach affected inherited components and active processes, while separate tests establish their actual effects. The lower panel extends these checks to monitoring and control after improvement-mechanism revision and reuse.

Retained form	Trace	$\rightarrow$ Control test
Textual memories and skills	Source , active text, retrieval / selection, actual use	Edit / exclude ; test influence remaining in copies
Loadable adapters	Source / generator / teacher version ; base and active adapters	Swap / load / scale / ablate under composition
Merged parameters	Contributors / merge ; active and derived weights	Verify correction in merged and derived state

Table 6: Traceability and intervention checks for retained experience.

Attribution	Question	Evidence
Technical contribution	Which changes affected the outcome?	Component ablations ; interactions across revisions
Operational responsibility	Which duties applied / were fulfilled?	Assigned roles ; available information; recovery actions
Governance accountability	Who must explain and remedy?	Authorization / delegation ; review and action records

Table 7: Three forms of attribution and their evidence.

Panel of Figure 8 — Figure 8: Evidence coverage in trustworthy RSI studies. Studies are grouped by their primary use: update-level , successor-level , and meta-improvement . Rows cover the three overarching trustworthiness concerns and supporting evaluation designs. The upper rows group selected evidence by the three analytical roles; the lower rows record supporting evaluation designs. Outlined cells indicate discussion or illustrative cases; gray cells indicate not evaluated; and question marks indicate unclear evidence. Task performance includes benign utility, and the final row records updates produced by a retained, revised updater. Coding is at the study level. Appendix A.4 and the audit files provide coding criteria, paper versions, source locations, and verification status.

Figure 9: Three-level comparison design for longitudinal RSI evaluation. Update-level comparisons separate development, acceptance, and independent reporting after S4; successor-level comparisons track inherited outcomes across versions; and meta-improvement comparisons use paired continuations with fixed updaters. All three assess task performance, constraint satisfaction, and oversight effectiveness.

Requirement group	Core signal	Across versions
Improvement validity	Independent gains ; variance	Retention / shift ; report accuracy
Behavioral safety and alignment	Violation severity	Conflicting or changed goals
Fairness	Worst group ; subgroup gains / harms	Population coverage across versions
Security and resilience	Attack success ; benign utility	Inherited compromise ; recovery
Privacy and data governance	Disclosure / inference / extraction	Residual exposure after deletion
Transparency and interpretability	Reconstructable changes	Explanation fidelity after inheritance

Table 8: Eight longitudinal trustworthiness checks. For each claim level, report versions, scope, threat model, uncertainty, and evaluation status.

Figure 10: Representative application settings for trustworthy RSI.

System setting	What is modified?	Is feedback independent?	What persists after acceptance?	Main evidence gap
Coding agents ( Zhang et al., 2026e ; Guo et al., 2026 )	Agent code, prompts, tests, and scaffolds.	Independence depends on test ownership; protected assessment is needed.	Code, instructions, and generated artifacts.	Safety under the intended deployment conditions.
Memory / search agents ( Zhang et al., 2025a ; Lee et al., 2026b )	Memory contents and designs, retrieval policies, and indexes.	Independence requires protected relevance and coverage checks beyond search feedback.	Memories, indexes, and query behavior.	Privacy, coverage, and resistance to poisoning.
Agents with evolving safeguards ( Rosser and Foerster, 2025 ; Cheng et al., 2026a )	Candidates, policies, harnesses, and guards.	Reassess feedback independence after safeguards change.	Accepted candidates and safety mechanisms.	Protection retained across later versions.
Meta-agents (Hyperagents) ( Zhang et al., 2026f )	The mechanism $\mathcal{U}_{t}$ governing later updates.	Assessment must remain independent of the editable updater.	The revised process for producing future changes.	Quality of subsequent updates under the revised updater.
Clinical agents (MedRSI) ( Wu et al., 2026b )	Tools, registration decisions, and workflows.	Cohort-based registration; independent clinical validation is still required.	Registered tools and workflows; clinical effects if deployed.	Patient-level validity and clinical deployment safety.
Embodied agents (RoboFoundry) ( Liang et al., 2026 )	Skills, context, and recovery procedures.	Tested transfer; independent physical safety and intervention checks are required.	Retained skills and procedures; physical effects when executed.	Physical safety beyond the tasks and real-world trials tested.

Table 9: Representative system settings and their evidence requirements. The feedback column summarizes evaluation arrangements and required independent checks; the final column lists the evidence needed for broader or longer-term claims.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Source	Full venue or source name	References
Preprints and online sources
arXiv	arXiv preprints	198 (65.78%)
Web	Online model announcements and research essays	3 (1.00%)
OpenReview	OpenReview records without a named proceedings venue	1 (0.33%)
Preprints.org	Preprints.org preprints	1 (0.33%)
Subtotal		203 (67.44%)

Appendix

Table 10: Publication sources of the 301 references cited in this survey.

Recursive Self-Improvement

Recursive self-improvement studies AI systems that use feedback from their own work to modify and evaluate future versions.

See all 84 papers

Sep 10, 2026

The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement

Recursive self-improvement (RSI) enables AI systems to turn experience and feedback into persistent changes that improve both their capabilities and the process of future improvement. We first use the Headroom-Closed Index (HCI) to reveal the problems of existing LLMs, then introduce the RSI concept and its development roadmap: from improvement-execution autonomy, improvement-strategy autonomy, experience-acquisition autonomy, and environment-adaptation autonomy, to recursive meta-improvement. Next we examine RSI across scenarios (e.g., scientific discovery, embodied intelligence, software engineering), highlighting their distinct requirements and development speeds. Drawing on diverse industry practices and preliminary empirical evidence, we connect RSI research with practical systems and identify key challenges to achieving genuine RSI.

Yi Duan, Ying Liu, Zirui Tang +32

Shanghai Jiao Tong University · Theseus Labs · Agent-Native Research Lab +6

Self-Improving Agents Language Model Self-Improvement Recursive Self-Improvement

Ask PDF

Jul 8, 2026

Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops

AI systems increasingly participate in their own improvement: revising their outputs, adapting their own harnesses during deployment, training on data they generate, and, increasingly, conducting AI research itself. This literature is described under a vocabulary ("self-refine," "self-reward," "self-play," "self-evolve") that conflates fundamentally different ambitions. We survey 1,250 arXiv papers (2024-2026) along two axes: what the system improves -- its behavior in deployment, its policy through training, its evaluator, or the research process itself -- and the degree of loop closure (human-in-the-loop to fully closed). The taxonomy separates bounded self-refinement -- convergent, evaluable, and already industrial practice -- from open-ended recursive self-improvement (RSI), which remains bounded by grounding requirements, collapse dynamics, and compute constraints on every measured axis. Its distinctive feature is a dedicated category for self-evaluation: every improvement loop is a claim that some signal can substitute for human judgment. We survey the evaluator design space -- judges, process reward models, verifiers, rubrics, meta-evaluation -- order the signals into a verification hierarchy from formal verifiers (strongest) to intrinsic self-assessment (weakest), and observe that demonstrated self-improvement strength tracks this hierarchy, that its failure modes (self-confirming loops, model collapse, diversity collapse) follow from its violations, and that the "research direction-setting" bottleneck keeping humans in the loop sits at the top of that hierarchy. We connect the technical literature to the theory of RSI limits and to the safety and governance questions raised by frontier-lab accounts of closing the loop, and identify governance-grade measurement of self-improvement as the field's most underpopulated niche.

Mingguang Chen, Licheng Wang, Bo Qu

University of California, Riverside (UCR) · AlphaAvatar · Illinois Institute of Technology (IIT)

LLM Self-Refinement Human-in-the-Loop Evaluation Self-Improving Agents Language Model Self-Improvement Recursive Self-Improvement Autonomous Research Agents

Ask PDF

Oct 20, 2026

Auditing the Product, Not the Recursion: The Evaluation Evidence Behind Recursive Self-Improvement Claims

Recursive self-improvement (RSI) promises AI systems, such as self-evolving LLM agents, that become better at producing further improvements. Evaluating this promise requires separating product claims about the final system from process claims about the improvement process itself. Existing surveys specify the evidence these claims require, but how often studies report that evidence remains unmeasured. We examine this question through full-text coding of 114 eligible studies in a reproducible arXiv search frame frozen in October 2026, including 90 that describe their own systems as RSI. The codebook links claims to evaluation designs. Independent human coding, third-reader adjudication, and a separate blind coding of improver comparisons assess coding reliability. Among the 90 RSI studies, 90% report product evaluation beyond development tasks under a broad criterion that includes partial or unstated separation, while 60% report evaluation on tasks unused for selection. Only 22% audit two or more reruns of the full loop. In a human-coded random sample, the RSI studies show the same contrast: 45% evaluate on tasks unused for selection and 14% audit two or more reruns. Of the 26 studies claiming a better improvement procedure, 23 report a related comparison, but only 4 compare earlier and later procedures from identical starting artifacts at a matched budget. Across these results and a comparison set of established works, evidence for repeatability and improved procedures remains less developed than product evaluation. Three propositions explain how selection effects, shared evaluator errors, and changes in system state limit what product evaluations can establish. Together, the findings support a claim-indexed reporting standard that connects each claim to the evaluation design needed to test it.

Zihan Chen, Di Zhu, Lei Nico Zheng

Stevens Institute of Technology Hoboken, NJ, USA · University of Massachusetts Boston Boston, MA, USA

Recursive Self-Improvement AI Agent Auditing

Ask PDF

Abstract

Recursive Self-Improvement

Explore similar work

The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement

Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops

Auditing the Product, Not the Recursion: The Evaluation Evidence Behind Recursive Self-Improvement Claims