Trustworthy Recursive Self-Improvement in the Era of Large Models: Foundation, Risks, Evaluation and Open Challenges

Jiayue Pu; Ang Gao; Jinyi Bai; Qipeng Wang; Tongyu Wen; Jiahao Zhao; Keyu Qu; Junhao Zhang

cs.AIOct 26, 2026

Trustworthy Recursive Self-Improvement in the Era of Large Models: Foundation, Risks, Evaluation and Open Challenges

Authors: Jiayue Pu, Ang Gao, Jinyi Bai, Qipeng Wang, Tongyu Wen, Jiahao Zhao, Keyu Qu, Junhao Zhang,

Organizations: Gaoling School of Artificial Intelligence, Renmin University of China · University of Chinese Academy of Sciences

Abstract

AI systems built around large models increasingly use experience and feedback to make persistent changes to models, prompts, memories, tools, and workflows. Recursive self-improvement (RSI) extends this process to inductive biases and to the mechanisms that govern future updates, which can themselves be revised and reused across improvement cycles. Yet self-modification alone does not guarantee trustworthy improvement. Reported gains may reflect flawed evaluations, previously acquired capabilities may deteriorate, and compliance with behavioral constraints may weaken. Compromised artifacts or sensitive information may also persist in successor systems. Changes to improvement mechanisms may invalidate earlier verification results and make human oversight less effective. This survey presents an operational definition and process framework for RSI and organizes the requirements for trustworthy RSI into four interconnected themes: genuine and sustained improvement; behavioral constraints, alignment, and fairness; system security and privacy; and transparency and controllability. We examine these themes from three perspectives: Improvement Validity, Constraint Preservation, and Effective Oversight. This analysis covers improvement diagnosis, proposal generation, proposal evaluation, and update implementation (S1--S4), including the recursive application of these stages to improvement mechanisms (RI). We review risks, safeguards, and evaluation evidence for individual updates, effects inherited by successor systems, and revisions to improvement mechanisms, while distinguishing direct RSI findings from evidence in related systems. Finally, we examine evaluation settings, identify requirements for independent and longitudinal assessment, and discuss applications and open research problems.

Figures & tables

Figure 1: Overview of the RSI Process and Trustworthiness Framework. The numbered labels (§2–§8) indicate the corresponding sections of this survey. The central panel (Section 2) presents the definition and four-stage improvement cycle of RSI (S1–S4), including recursive revision and reuse of the bias update function (RI). Four surrounding trustworthiness themes are organized under three complementary analytical roles: Improvement Validity, Constraint Preservation, and Effective Oversight. Section 7 examines evaluation designs and evidence requirements at the individual-update, successor, and meta-improvement levels, while §8 covers representative applications and open research problems.

Survey	Main analytical focus	MT	IL	MR	LA	JE
Trustworthiness-oriented surveys
Li et al. (2023)	Trustworthiness principles and practices for AI	$\checkmark$	—	—	—	$\triangle$
Liu et al. (2023)	Trustworthiness requirements and methods for LLMs	$\checkmark$	—	—	—	$\triangle$
Yu et al. (2025)	Trustworthiness of LLM agents	$\checkmark$	—	—	—	$\triangle$
Qi et al. (2026)	Trustworthiness of agentic workflows	$\checkmark$	$\triangle$	—	$\triangle$	$\triangle$
Self-evolution and RSI-oriented surveys

Table 1: Comparison with representative surveys of trustworthy AI, self-evolution, RSI, and evolutionary safety. Symbols indicate the depth of analysis defined in the legend.

Approach	Who determines the inductive bias $B$ ?	Is $B$ revised autonomously?	Is the bias update function revised and reused?
Conventional machine learning	Human designer	✗	✗
Reinforcement learning	Human designer	✗	✗ (learning rule fixed)
Meta-learning	Meta-learner	✓	✗ (meta-update rule fixed)
AutoML	Automated search	✓	✗ (search rule fixed)
RSI	Bias update function $\mathcal{U}_{t}$	✓	✓ (the revised $\mathcal{U}_{t}$ guides later bias updates)

Table 2: Comparison of RSI with other machine learning approaches in their standard forms.

Figure 3: The recursive self-improvement (RSI) process. The core improvement cycle (S1–S4) identifies limitations, generates and evaluates candidate revisions to the policy $h_{t}$ or inductive bias $B_{t}$ , and incorporates accepted changes into the successor system. Recursive improvement (RI) applies the same four stages to the bias update function $\mathcal{U}_{t}$ itself. The resulting $\mathcal{U}_{t+1}$ is retained for subsequent improvement cycles, while components without accepted revisions remain unchanged.

Analytical role	Theme	Requirement groups	RSI-specific question
Improvement Validity	Genuine and sustained improvement Section 3	(i) Validity, reliability, robustness, and truthfulness	Are improvements genuine and sustained, including after revisions to the update mechanism?
Constraint Preservation	Behavioral constraints, alignment, and fairness Section 4	(ii) Behavioral safety and value alignment; (iii) fairness	Do behavioral requirements remain satisfied across successive revisions?
Constraint Preservation	System security, resilience, privacy, and data governance Section 5	(iv) System security and resilience; (v) privacy and data governance	Are inherited components and information protected during subsequent use?
Effective Oversight	Transparency and controllability Section 6	(vi) Transparency and interpretability; (vii) human control and corrigibility; (viii) accountability and governance	Can revisions, including changes to the update mechanism, be reviewed and corrected?

Table 3: Mapping the eight trustworthiness requirement groups to four RSI themes. The three analytical roles are complementary and do not represent separate stages of the improvement process.

Figure 4: Genuine and Sustained Improvement Across S1–S4 and RI in the Process of RSI. Stage-specific risks and checks address biased diagnosis, shortcut search, misleading acceptance scores, and ineffective updates. The lower panel compares the gains, retention, and transfer of later updates produced by the original and revised improvement mechanisms, starting from matched initial states under common tests and resource budgets.

Study and setting	Modified object / reuse	Horizon	Comparison and resource evidence
SICA ( Robeyns et al., 2025 ) Claude 3.5 / o3-mini; SWE-bench, LiveCodeBench	Agent code , prompts, tools; $\checkmark$ revised agents write later edits (archive selection fixed)	15 self-edits	Initial vs. later agents: 17% $\rightarrow$ 53% at peak on 50 SWE-bench tasks; $\sim$ $7k API; no fixed-updater ablation
Agent Plasticity ( Singh et al., 2026 ) Frontier LMs; chess, Go, Hex, NetHack	Tools, skills, notes inherited; – improvement protocol fixed (weights fixed)	20/40 rounds	Held-out checkpoint tests; gain per dollar , including reflection and rejected edits
MASS ( Lee et al., 2026a ) Qwen3.6-27B; finance, robotics, pharmacy	Workflows and weights ; $\checkmark$ updated model reused as optimizer and evaluator (outer loop fixed)	2 cycles	Base vs. trained models; external judgments; token efficiency reported, training cost unmatched
Hyperagents ( Zhang et al., 2026f ) Claude 4.5; paper review, robotics rewards	Task/meta-agent code ; $\checkmark$ revised meta-agent reused across domains	100 searches	Fixed-meta ablation and search controls; $\sim$ 33M self-edit tokens per 100 iterations, evaluation excluded
Evo 2 ( Xu et al., 2026b ) Qwen3.8-27B; Polyglot, SWE-bench, LoCoMo	Harness / Proposer–Reflector weights ; $\checkmark$ trained updater reused in later searches	Repeated ; total not reported	Trained vs. untrained evolver ; parents, executor, search fixed; training excluded from inference cost

Table 4: Representative evidence on improvement and mechanism reuse. Horizons use study-specific units;

\checkmark

marks a revised updater reused in later updates, and – marks artifact reuse under a fixed protocol.

Figure 5: Behavioral Constraints, Alignment, and Fairness Across S1–S4 and RI in the Process of RSI. Behavioral requirements, authorized human goals, and fairness criteria apply throughout the four improvement stages. Each stage pairs a potential failure with a corresponding safeguard. The lower panel examines whether changes to the improvement mechanism remain authorized and whether later successors continue to satisfy the applicable behavioral, alignment, and fairness requirements.

Study	Tested setting	Horizon / controls	Key finding
Behavioral constraints
Qi et al. (2024)	GPT-3.5 Turbo / Llama-2-7B; 330 prompts, 11 prohibited categories	1 epoch ; before/after on the same safety test and scoring rule	Harmful compliance $\uparrow$ after benign tuning
Zhao et al. (2026b)	GPT-4o web agent; BrowserART / WebArena	106 steps ; weights and tests fixed; execution / refusal memories varied	Unsafe actions $\downarrow$ , but over-refusal $\uparrow$
Alignment
Langosco et al. (2022)	CoinRun; relocated coin vs. level completion	Zero-shot ; trained policy and reward fixed	Navigation retained — relocated coin often passed
Khalaf et al. (2025)	GPQA; proxy reward vs. answer accuracy	Up to 32 samples ; questions, response pool, reward model fixed	Accuracy peaks then declines with stronger selection

Table 5: Representative behavioral, alignment, and fairness findings. Studies are grouped by tested property; horizons and fixed conditions follow the original experiments.

Figure 6: System Security, Resilience, Privacy, and Data Governance Across S1–S4 and RI in the Process of RSI. The two rows distinguish security and resilience from privacy and data governance. Stage-specific risks and safeguards cover compromised experience, malicious or sensitive derived assets, unreliable admission evidence, and inherited threats or information exposure. The lower panel outlines independent approval, renewed protection checks, and tests of successors after the improvement mechanism is revised and reused.

Figure 7: Transparency and Controllability Across S1–S4 and RI in the Process of RSI. The four stages connect testable diagnoses, proposed interventions, independent evaluation, and enforceable correction. Cross-stage oversight requires testable explanations, traceable sources and revisions, and independent review. Authorized interventions must reach affected inherited components and active processes, while separate tests establish their actual effects. The lower panel extends these checks to monitoring and control after improvement-mechanism revision and reuse.

Retained form	Trace	$\rightarrow$ Control test
Textual memories and skills	Source , active text, retrieval / selection, actual use	Edit / exclude ; test influence remaining in copies
Loadable adapters	Source / generator / teacher version ; base and active adapters	Swap / load / scale / ablate under composition
Merged parameters	Contributors / merge ; active and derived weights	Verify correction in merged and derived state

Table 6: Traceability and intervention checks for retained experience.

Attribution	Question	Evidence
Technical contribution	Which changes affected the outcome?	Component ablations ; interactions across revisions
Operational responsibility	Which duties applied / were fulfilled?	Assigned roles ; available information; recovery actions
Governance accountability	Who must explain and remedy?	Authorization / delegation ; review and action records

Table 7: Three forms of attribution and their evidence.

Panel of Figure 8 — Figure 8: Evidence coverage in trustworthy RSI studies. Studies are grouped by their primary use: update-level , successor-level , and meta-improvement . Rows cover the three overarching trustworthiness concerns and supporting evaluation designs. The upper rows group selected evidence by the three analytical roles; the lower rows record supporting evaluation designs. Outlined cells indicate discussion or illustrative cases; gray cells indicate not evaluated; and question marks indicate unclear evidence. Task performance includes benign utility, and the final row records updates produced by a retained, revised updater. Coding is at the study level. Appendix A.4 and the audit files provide coding criteria, paper versions, source locations, and verification status.

Figure 9: Three-level comparison design for longitudinal RSI evaluation. Update-level comparisons separate development, acceptance, and independent reporting after S4; successor-level comparisons track inherited outcomes across versions; and meta-improvement comparisons use paired continuations with fixed updaters. All three assess task performance, constraint satisfaction, and oversight effectiveness.

Requirement group	Core signal	Across versions
Improvement validity	Independent gains ; variance	Retention / shift ; report accuracy
Behavioral safety and alignment	Violation severity	Conflicting or changed goals
Fairness	Worst group ; subgroup gains / harms	Population coverage across versions
Security and resilience	Attack success ; benign utility	Inherited compromise ; recovery
Privacy and data governance	Disclosure / inference / extraction	Residual exposure after deletion
Transparency and interpretability	Reconstructable changes	Explanation fidelity after inheritance

Table 8: Eight longitudinal trustworthiness checks. For each claim level, report versions, scope, threat model, uncertainty, and evaluation status.

Figure 10: Representative application settings for trustworthy RSI.

System setting	What is modified?	Is feedback independent?	What persists after acceptance?	Main evidence gap
Coding agents ( Zhang et al., 2026e ; Guo et al., 2026 )	Agent code, prompts, tests, and scaffolds.	Independence depends on test ownership; protected assessment is needed.	Code, instructions, and generated artifacts.	Safety under the intended deployment conditions.
Memory / search agents ( Zhang et al., 2025a ; Lee et al., 2026b )	Memory contents and designs, retrieval policies, and indexes.	Independence requires protected relevance and coverage checks beyond search feedback.	Memories, indexes, and query behavior.	Privacy, coverage, and resistance to poisoning.
Agents with evolving safeguards ( Rosser and Foerster, 2025 ; Cheng et al., 2026a )	Candidates, policies, harnesses, and guards.	Reassess feedback independence after safeguards change.	Accepted candidates and safety mechanisms.	Protection retained across later versions.
Meta-agents (Hyperagents) ( Zhang et al., 2026f )	The mechanism $\mathcal{U}_{t}$ governing later updates.	Assessment must remain independent of the editable updater.	The revised process for producing future changes.	Quality of subsequent updates under the revised updater.
Clinical agents (MedRSI) ( Wu et al., 2026b )	Tools, registration decisions, and workflows.	Cohort-based registration; independent clinical validation is still required.	Registered tools and workflows; clinical effects if deployed.	Patient-level validity and clinical deployment safety.
Embodied agents (RoboFoundry) ( Liang et al., 2026 )	Skills, context, and recovery procedures.	Tested transfer; independent physical safety and intervention checks are required.	Retained skills and procedures; physical effects when executed.	Physical safety beyond the tasks and real-world trials tested.

Table 9: Representative system settings and their evidence requirements. The feedback column summarizes evaluation arrangements and required independent checks; the final column lists the evidence needed for broader or longer-term claims.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Source	Full venue or source name	References
Preprints and online sources
arXiv	arXiv preprints	198 (65.78%)
Web	Online model announcements and research essays	3 (1.00%)
OpenReview	OpenReview records without a named proceedings venue	1 (0.33%)
Preprints.org	Preprints.org preprints	1 (0.33%)
Subtotal		203 (67.44%)

Appendix

Table 10: Publication sources of the 301 references cited in this survey.

Recursive Self-Improvement

Recursive self-improvement studies AI systems that use feedback from their own work to modify and evaluate future versions.

See all 84 papers

The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement
Sep 10, 2026Yi Duan, Ying Liu, Zirui Tang +32Self-Improving Agents Language Model Self-Improvement
Shanghai Jiao Tong University · Theseus Labs · Agent-Native Research Lab +6
Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops
Jul 8, 2026Mingguang Chen, Licheng Wang, Bo QuLLM Self-Refinement Human-in-the-Loop Evaluation
University of California, Riverside (UCR) · AlphaAvatar · Illinois Institute of Technology (IIT)
Auditing the Product, Not the Recursion: The Evaluation Evidence Behind Recursive Self-Improvement Claims
Oct 20, 2026Zihan Chen, Di Zhu, Lei Nico ZhengRecursive Self-Improvement AI Agent Auditing
Stevens Institute of Technology Hoboken, NJ, USA · University of Massachusetts Boston Boston, MA, USA

Sep 10, 2026J/K move · Enter open · S save

The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement

Yi Duan, Ying Liu, Zirui Tang, Haodong Chen+31

Shanghai Jiao Tong University · Theseus Labs · Agent-Native Research Lab+6

Self-Improving Agents Language Model Self-Improvement Recursive Self-Improvement

Open paper AskPDF arXiv

Recursive self-improvement (RSI) enables AI systems to turn experience and feedback into persistent changes that improve both their capabilities and the process of future improvement. We first use the Headroom-Closed Index (HCI) to reveal the problems of existing LLMs, then introduce the RSI concept and its development roadmap: from improvement-execution autonomy, improvement-strategy autonomy, experience-acquisition autonomy, and environment-adaptation autonomy, to recursive meta-improvement. Next we examine RSI across scenarios (e.g., scientific discovery, embodied intelligence, software engineering), highlighting their distinct requirements and development speeds. Drawing on diverse industry practices and preliminary empirical evidence, we connect RSI research with practical systems and identify key challenges to achieving genuine RSI.

Abstract

Recursive Self-Improvement

Explore similar work

The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement

Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops

Auditing the Product, Not the Recursion: The Evaluation Evidence Behind Recursive Self-Improvement Claims