Self-Improvement of Large Language Models: A Technical Overview and Future Outlook

Haoyan Yang; Mario Xerri; Solha Park; Huajian Zhang; Yiyang Feng; Sai Akhil Kogilathota; Jiawei Zhou

cs.CLMar 26, 2026

Self-Improvement of Large Language Models: A Technical Overview and Future Outlook

Authors: Haoyan Yang, Mario Xerri, Solha Park, Huajian Zhang, Yiyang Feng, Sai Akhil Kogilathota, Jiawei Zhou

Organizations: Zesearch NLP Lab, Stony Brook University

Abstract

As large language models (LLMs) continue to advance, improving them solely through human supervision is becoming increasingly costly and limited in scalability. As models approach human-level capabilities in certain domains, human feedback may no longer provide sufficiently informative signals for further improvement. At the same time, the growing ability of models to make autonomous decisions and execute complex actions naturally enables abstractions in which components of the model development process can be progressively automated. Together, these challenges and opportunities have driven increasing interest in self-improvement, where models autonomously generate data, evaluate outputs, and iteratively refine their own capabilities. In this paper, we present a system-level perspective on self-improving language models and introduce a unified framework that organizes existing techniques. We conceptualize the self-improvement system as a closed-loop lifecycle, consisting of four tightly coupled processes: data acquisition, data selection, model optimization, and inference refinement, along with an autonomous evaluation layer throughout the process. Within this framework, the model itself plays a central role in driving each stage: collecting or generating data, selecting informative signals, updating its parameters, and refining outputs, while the autonomous evaluation layer continuously monitors progress and guides the improvement cycle across stages. Following this lifecycle perspective, we systematically review and analyze representative methods for each component from a technical standpoint. We further examine current limitations, potential risks, and prominent applications, and outline our vision for future research toward fully self-improving LLMs.

Figures & tables

Figure 1: Vision of self-improving language models. Humans only bootstrap the system, after which the model autonomously performs repeated operations such as acquiring data, reflecting on its outputs, and iteratively refining its capabilities to improve itself, potentially enabling the system to evolve beyond human-level intelligence.

Figure 2: Overview of the proposed self-improvement system. The system consists of five interconnected stages: (i) Data Acquisition collects or generates candidate training data from external sources, environments, or the model itself, producing raw acquired data; (ii) Data Selection filters, ranks, or constructs high-quality subsets from the acquired data, producing selected training data; (iii) Model Optimization updates the model using the selected data, producing an improved model; (iv) Inference Refinement improves model outputs during inference through search, feedback, or self-correction, producing refined responses; and (v) Autonomous Evaluation continuously assesses the model, data, and outputs, producing feedback signals that guide long-term iterative self-improvement.

Panel of Figure 3 — Figure 3: A taxonomy of the self-improvement system of LLMs.

Figure 4: Overview of data acquisition in the self-improvement system. This stage focuses on how a model autonomously acquires raw data or experiences that can be used for self-improvement. (i) Static Curation: The model collects and filters information from pre-existing datasets or knowledge sources to construct curated training data. (ii) Environment Interaction: The model actively interacts with external environments, issuing actions and receiving observations to form interaction trajectories. (iii) Synthetic Generation: The model generates new training data directly from its own parameters through prompting, transformation, or multi-model interaction.

Environment	Methods
Web Browsing	WebGPT ( Nakano et al., 2022 ) , Toolformer ( Schick et al., 2023 ) , Go-Browse ( Gandhi & Neubig, 2025 ) , BrowserAgent ( Yu et al., 2025f ) , InSTA ( Trabucco et al., 2025 ) , EnvScaler ( Song et al., 2026a )
Code Execution	TRACED ( Ding et al., 2023 ) , RLEF ( Gehring et al., 2025 ) , CWM ( FAIR CodeGen team et al., 2025 ) , CodeRL+ ( Jiang et al., 2025d ) , AgentFounder ( Su et al., 2025b ) , Learn-by-Interact ( Su et al., 2025a )
Game Environments	Supervise Thyself ( Racah & Pal, 2019 ) , Generative Agents ( Park et al., 2023 ) , ALYMPICS ( Mao et al., 2025 ) , FAIRGAME ( Huynh et al., 2025 )

Table 1: Environment interaction methods categorized by environment type. We group representative methods based on the external environments they interact with, including web browsing, code execution, and game environments.

Generation Mechanism	Methods
Prompt-Based	TinyStories ( Eldan & Li, 2023 ) , Phi-1 ( Gunasekar et al., 2023 ) , Phi-1.5 ( Li et al., 2023c ) , Self-Instruct ( Wang et al., 2023c ) , AttrPrompt ( Yu et al., 2023 ) , WizardLM ( Xu et al., 2024a ) , Phi-3 ( Abdin et al., 2024a ) , Phi-4 ( Abdin et al., 2024b ) , CodecLM ( Wang et al., 2024i ) , Cosmopedia ( Ben Allal et al., 2024 ) , Constraint-Based ( Fedoseev et al., 2024 ) , CoT-Self-Instruct ( Yu et al., 2025d ) , MIND ( Akter et al., 2025a ) , SAO ( Yin et al., 2026 ) , Autodata ( Kulikov et al., 2026 )
Transformation-Based	WRAP ( Maini et al., 2024 ) , Instruction Pre-Training ( Cheng et al., 2024 ) , SBP ( Yang et al., 2025c ) , SynthLLM ( Qin et al., 2025 ) , Gradient Matching ( Nguyen et al., 2025 )
Interaction-Based	Shah et al. (2018) , CICERO ( FAIR Diplomacy Team et al., 2023 ) , Baize ( Xu et al., 2023 ) , SPIN ( Chen et al., 2024j ) , ALAS ( Atreja, 2025 ) , LSP ( Kuba et al., 2025 ) , Multi-Agent Dialogues ( Ueda et al., 2025 ) , Math Gen ( Wan et al., 2025 ) , EigenData ( Chen et al., 2026b ) , R-Zero ( Huang et al., 2026a )

Table 2: Synthetic data generation methods categorized by generation mechanism. We group representative methods based on how synthetic data is produced, including prompt-based, transformation-based, and interaction-based approaches.

Figure 5: Overview of data selection in the self-improvement system. This stage focuses on how a model evaluates and selects high-quality data from raw data pools to support effective self-improvement. (i) Metric-Guided Scoring: The model applies predefined scoring metrics derived from model signals to rank and filter data. One-shot scoring computes scores once before training, while iterative re-scoring periodically refreshes scores as the model evolves. (ii) Adaptive Selection: A learnable selector dynamically chooses training data through an iterative loop of selection, evaluation, and selector update, continuously adapting the training distribution based on feedback.

Model Metric	One-Shot Scoring (Offline)	Iterative Re-scoring (Online)
Perplexity	Marion et al. (2023) , NUGGETS ( Li et al., 2024k ) , Perplexity-based Pruning ( Ankner et al., 2025 ) , Perplexity Correlations ( Thrush et al., 2025 )	SST ( hattami et al., 2025 ) , PREPO ( Sun et al., 2025b )
Loss	DoReMi ( Xie et al., 2023a ) , S2L ( Yang et al., 2024c )	RHO-LOSS ( Mindermann et al., 2022 )
Gradient	LESS ( Xia et al., 2024 ) , IProX ( Chen et al., 2026c )	GREATS ( Wang et al., 2024d ) , LearnAlign ( Li et al., 2025d )
Difficulty	Li et al. (2024e) , Learning Percentage ( Mekala et al., 2024 ) , T-SHIRT ( Fu et al., 2025 )	IT2ACL ( Huang & Xiong, 2024 ) , P3 ( Yang et al., 2024b ) , AdaRFT ( Shi et al., 2025a ) , AdaSTaR ( Koh et al., 2025 ) , Bae et al. (2026)
Uncertainty	SelectIT ( Liu et al., 2024d ) , UniMax ( Han et al., 2025 )	Recency Bias ( Song et al., 2020 ) , Active Instruction Tuning ( Kung et al., 2023 )
Similarity	DEFT ( Ivison et al., 2023 ) , DSIR ( Xie et al., 2023b )	DiverseEvol ( Wu et al., 2023 ) , Balanced LSH ( Phan et al., 2025 )

Table 3: Metric-guided scoring methods based on model metrics. This table summarizes representative methods according to the metrics they use for data selection (e.g., perplexity, loss, gradient, difficulty, uncertainty, similarity, and external evaluators), and distinguishes two settings in metric-guided scoring: one-shot scoring (offline) and iterative re-scoring (online).

Method	Unit	Signal	Update	Coupling
MATES ( Yu et al., 2024d )	Instance	Influence	Supervised	Coupled
RAISE ( Lv et al., 2025 )	Instance	Loss	RL	Coupled
ScalingRL ( Chen et al., 2025i )	Instance	Composite	Learned reweighting	Coupled
APO ( Das et al., 2025 )	Instance	Uncertainty	Bandit	Coupled
SEAL ( Shen et al., 2025 )	Instance	Loss	Bilevel	Decoupled
Multi-Actor ( Bai et al., 2025 )	Instance	Composite	Learned reweighting	Decoupled

Table 4: Adaptive selection methods and their design dimensions. This table summarizes representative methods by their selection unit, the signals used, the update mechanism and whether the selection process is coupled or decoupled with model training.

Training Phase	Methods
Pre-training	C4 ( Raffel et al., 2020 ) , CCNet ( Wenzek et al., 2020 ) , The Pile ( Gao et al., 2020 ) , GPT-3 ( Brown et al., 2020 ) , GLaM ( Du et al., 2022 ) , RHO-LOSS ( Mindermann et al., 2022 ) , PaLM ( Chowdhery et al., 2023 ) , DSIR ( Xie et al., 2023b ) , DoReMi ( Xie et al., 2023a ) , GREATS ( Wang et al., 2024d ) , DATAMASK ( Fan et al., 2026 )
Post-training	Recency Bias ( Song et al., 2020 ) , DEFT ( Ivison et al., 2023 ) , NUGGETS ( Li et al., 2024k ) , S2L ( Yang et al., 2024c ) , LESS ( Xia et al., 2024 ) , Li et al. (2024e) , SelectIT ( Liu et al., 2024d ) , AlpaGasus ( Chen et al., 2024d ) , DEITA ( Liu et al., 2024g ) , GREATS ( Wang et al., 2024d ) , IT2ACL ( Huang & Xiong, 2024 ) , P3 ( Yang et al., 2024b ) , SST ( hattami et al., 2025 ) , PREPO ( Sun et al., 2025b ) , AdaRFT ( Shi et al., 2025a ) , AdaSTaR ( Koh et al., 2025 ) , LearnAlign ( Li et al., 2025d ) , DUMP ( Wang et al., 2025j ) , SEC ( Chen et al., 2025f ) , RAISE ( Lv et al., 2025 ) , SPaCe ( Do et al., 2025 ) , ScalingRL ( Chen et al., 2025i ) , SEAL ( Shen et al., 2025 ) , DynamixSFT ( Shin et al., 2026 )

Table 5: Representative data selection methods across different training stages. This table summarizes representative methods applied during pre-training and post-training, highlighting how data selection strategies are used at different stages of model development.

Figure 6: Overview of model optimization in the self-improvement system. We organize model optimization into three paradigms: (i) Direct Optimization , a single offline update on a fixed corpus; (ii) Self-Generated Optimization (SGO) , a closed loop of generation , reward , and optimization where the model learns from self-produced experience; and (iii) Self-Evolving Optimization , which improves the optimization process itself, from the update rule to the agent-level scaffolding.

Reward Type	Granularity	Methods
Implicit Reward	—	STaR ( Zelikman et al., 2022 ) , LMSI ( Huang et al., 2023a ) , TRIPOST ( Yu et al., 2024c ) , RISE ( Qu et al., 2024 ) , SELF ( Lu et al., 2024c ) , SPIN ( Chen et al., 2024j ) , V-STaR ( Hosseini et al., 2024 ) , Goedel-Prover ( Lin et al., 2025b ) , IWSI ( Jiang et al., 2025a ) , STaPLe ( Ramji et al., 2025 ) , CRESCENT ( Sun et al., 2025c ) , ReGenesis ( PENG et al., 2025 ) , Subramaniam et al. (2025) , STaSC ( Moskvoretskii et al., 2025 ) , SimRAG ( Xu et al., 2025a ) , SiriuS ( Zhao et al., 2025c ) , SOL-VER ( Lin et al., 2025c ) , Semantic Voting ( Jiang et al., 2025b )
Explicit Reward	Binary	Bensal et al. (2025) , SeRL ( Fang et al., 2025c ) , TBV ( Liu et al., 2025d ) , PAG ( Jiang et al., 2025e ) , Self-Challenging ( Zhou et al., 2025 ) , SSR ( Wei et al., 2025b ) , EvoCoT ( Liu et al., 2026a )
	Scalar	SIRLC ( Pang et al., 2024 ) , ReST-MCTS ∗ ( Zhang et al., 2024a ) , SCoRe ( Kumar et al., 2025 ) , Xiong et al. (2025) , DTE ( Srivastava et al., 2025 ) , RLSR ( Simonds et al., 2025 ) , CURE ( Wang et al., 2025h ) , Absolute Zero ( Zhao et al., 2025a ) , SPICE ( Liu et al., 2025a ) , MAE ( Chen et al., 2025h ) , ReVeal ( Jin et al., 2026 ) , RESTRAIN ( Yu et al., 2026b ) , R-Zero ( Huang et al., 2026a ) , Dr. Zero ( Yue et al., 2026 )
	Ordinal	Yuan et al. (2024b) , ScPO ( Prasad et al., 2025 ) , SynPO ( Dong et al., 2025 ) , SPHERE ( Singh et al., 2025 ) , Ji et al. (2025) , Wu et al. (2025c) , DNPO ( Yang et al., 2026a )
	Probability	SPPO ( Wu et al., 2025g ) , RSPO ( Tang et al., 2025b )

Table 6: Reward types used in model optimization for self-improvement. We categorize reward signals into implicit and explicit types. Explicit rewards are further divided into binary, scalar, ordinal, and probability-based signals, reflecting different levels of supervision granularity.

Low Automation	Intermediate Automation	High Automation
STaR ( Zelikman et al., 2022 ) TRIPOST ( Yu et al., 2024c ) RISE ( Qu et al., 2024 ) ReST-MCTS ∗ ( Zhang et al., 2024a ) SPIN ( Chen et al., 2024j ) V-STaR ( Hosseini et al., 2024 ) SCoRe ( Kumar et al., 2025 ) Xiong et al. (2025) SiriuS ( Zhao et al., 2025c ) IWSI ( Jiang et al., 2025a ) ReGenesis ( PENG et al., 2025 ) STaSC ( Moskvoretskii et al., 2025 ) Bensal et al. (2025) SPHERE ( Singh et al., 2025 ) TBV ( Liu et al., 2025d ) PAG ( Jiang et al., 2025e ) RSPO ( Tang et al., 2025b ) DNPO ( Yang et al., 2026a ) EvoCoT ( Liu et al., 2026a )	LMSI ( Huang et al., 2023a ) SIRLC ( Pang et al., 2024 ) SELF ( Lu et al., 2024c ) ScPO ( Prasad et al., 2025 ) DTE ( Srivastava et al., 2025 ) Goedel-Prover ( Lin et al., 2025b ) SPPO ( Wu et al., 2025g ) STaPLe ( Ramji et al., 2025 ) Subramaniam et al. (2025) SimRAG ( Xu et al., 2025a ) Semantic Voting ( Jiang et al., 2025b ) CURE ( Wang et al., 2025h ) Wu et al. (2025c) ReVeal ( Jin et al., 2026 ) RESTRAIN ( Yu et al., 2026b )	Yuan et al. (2024b) CRESCENT ( Sun et al., 2025c ) SynPO ( Dong et al., 2025 ) RLSR ( Simonds et al., 2025 ) Ji et al. (2025) SeRL ( Fang et al., 2025c ) SOL-VER ( Lin et al., 2025c ) Self-Challenging ( Zhou et al., 2025 ) SSR ( Wei et al., 2025b ) Absolute Zero ( Zhao et al., 2025a ) SPICE ( Liu et al., 2025a ) MAE ( Chen et al., 2025h ) R-Zero ( Huang et al., 2026a ) Dr. Zero ( Yue et al., 2026 )

Table 7: Degree of automation in model optimization with respect to data dependency. Methods are categorized by their reliance on externally collected prompts and ground-truth annotations: high automation requires neither, intermediate uses either one, and low automation depends on both.

Figure 7: Overview of inference refinement in the self-improvement system. This stage focuses on how the model improves output quality during inference without permanently updating its parameters. (i) Decoding Strategies: The model improves generation by modifying decoding procedures, such as sampling control, logit modification, tree search, or efficiency-oriented decoding methods. (ii) Reasoning-Based Improvement: The model refines reasoning trajectories through iterative processes such as generate–execute–refine loops, feedback-based revision, or collaborative reasoning among multiple agents. (iii) Agentic System-Based Improvement: The model operates within agentic systems that coordinate prompts, tools, workflows, and memory modules to enhance task performance. (iv) Test-Time Training: The model temporarily adapts its parameters during inference by generating feedback signals and updating the model before producing the final solution.

Decoding Strategy	Methods
Self-Consistency	SC ( Wang et al., 2023b ) , USC ( Chen et al., 2024g ) , CISC ( Taubenfeld et al., 2025 ) , CER ( Razghandi et al., 2025 )
Best-of-N	Stiennon et al. (2020) , Cobbe et al. (2021) , MAV ( Lifshitz et al., 2025 ) , UnCert-CoT ( Zhu et al., 2025b ) , Self-Certainty ( Kang et al., 2025b )
Synthesis	LLM-Blender ( Jiang et al., 2023 ) , LMCor ( Vernikos et al., 2024 ) , GSR ( Wang et al., 2025c ) , MoA ( Wang et al., 2025a ) , CoT-based Synthesizer ( Zhang et al., 2025b )
Beam Search	Xie et al. (2023c) , DBS ( Zhu et al., 2024 ) , AlphaGeometry ( Trinh et al., 2024 ) , PRM-BAS ( Hu et al., 2025b ) , AdaBeam ( Quamar et al., 2025 )
Tree Search	ToT ( Yao et al., 2023b ) , TS-LLM ( Feng et al., 2024 ) , MCTSr ( Zhang et al., 2024b ) , PPO-MCTS ( Liu et al., 2024b )
Reward Guidance	ARGS ( Khanov et al., 2024 ) , Controlled Decoding ( Mudgal et al., 2024 ) , DeAL ( Huang et al., 2025b ) , GenARM ( Xu et al., 2025c ) , PAD ( Chen et al., 2025b )

Table 8: Summary of decoding-based inference-time methods that can serve self-improvement. Early approaches improve generation by modifying the decoding process to enhance output quality and efficiency without updating model parameters.

Method	Feedback Source	Feedback Type
Self-Refine ( Madaan et al., 2023 )	Base LLM	Textual Critique
Reflexion ( Shinn et al., 2023 )	External Evaluator	Score $\rightarrow$ Textual Critique
RCI ( Kim et al., 2023 )	Base LLM or Verifier	Textual Critique or Verification
LEVER ( Ni et al., 2023 )	Verifier	Score
Self-Debugging ( Chen et al., 2024h )	Base LLM or Verifier	Textual Critique or Verification
IoE Prompt ( Li et al., 2024d )	Base LLM	Internal Confidence

Table 9: Summary of feedback-based inference-time refinement methods. These approaches improve generation through multi-step feedback loops, leveraging diverse feedback sources to iteratively refine outputs without updating model parameters.

Figure 8: Overview of autonomous evaluation in the self-improvement system. This stage focuses on how the model evaluates its performance and obtains feedback signals to support continuous self-improvement. (i) Dynamic Benchmarking: The evaluation set is continuously updated using data from evolving or real-world sources, allowing benchmarks to adapt to distribution shifts and emerging tasks. (ii) Interactive Environment Evaluation: The model’s capabilities are assessed through interactions with an environment, where performance is measured based on task completion, feedback signals, or environment-derived rewards.

Benchmark	Update Mode	Data Origin
RealTime QA ( Kasai et al., 2023 )	Live	News
KoLA ( Yu et al., 2024b )	Live	Multiple Datasets
EvoCodeBench ( Li et al., 2024c )	Live	Git/Code
FreshQA ( Vu et al., 2024 )	Live	Web
AntiLeakBench ( Wu et al., 2025e )	Live	Wikipedia
DeepScholar-Bench ( Patel et al., 2025 )	Live	arXiv

Table 10: Overview of dynamic evaluation benchmarks. Each benchmark name is hyperlinked to its official repository or project website. Benchmarks are compared by update mode and data origin.

Benchmark / Environment	Type	Task Domain	Evaluation Metrics
TextWorld ( Côté et al., 2018 )	Outcome	Games	Task success
Jericho ( Hausknecht et al., 2020 )	Outcome	Text / Games	Episode reward, game completion
ScienceWorld ( Wang et al., 2022 )	Outcome	Science	Task success, interaction reward
WebShop ( Yao et al., 2023a )	Outcome	Web & Information	Task success, efficiency
WebArena ( Zhou et al., 2023b )	Outcome	Web & Information	Task success, efficiency
GAIA ( Mialon et al., 2023 )	Outcome	General Assistant	Task success

Table 11: Overview of interactive evaluation benchmarks and environments. Each entry is hyperlinked to its official repository or project website, and compared by evaluation type, task domain, and metrics across diverse interactive settings.

Panel of Figure 9 — Figure 9: A taxonomy of discussions on the self-improvement system of LLMs. It includes challenges and limitations, safety risks, applications, and future outlook.

Figure 10: Challenges and limitations in the self-improvement system . (i) Data Autophagy: Iterative synthetic data loops degrade information quality and diversity. (ii) Flawed Feedback Signals: Self-generated evaluation signals are inherently biased and unstable. (iii) Optimization-Driven Failures: Training-time optimization against proxy rewards distorts model behavior. (iv) Ineffective Self-Refinement: Inference-time feedback loops fail to consistently improve outputs. (v) Evaluation Bottlenecks: Flawed benchmarks, metrics, and evaluators undermine reliable measurement of self-improvement. (vi) Supervision Bottlenecks: External oversight fails to effectively control self-improving systems. (vii) Budget Constraints: Iterative self-improvement repeatedly consumes compute across data generation, selection, optimization, and evaluation, while reliance on stronger models further amplifies cost.

Figure 11: Taxonomy of safety risks in self-improving systems. We categorize the risks into four dimensions: (i) Loss of Control , including oversight loss and deceptive alignment; (ii) High-Stakes Harm , including individual harm and systemic failure; (iii) Social and Ethical Risks , including social disruption and ethical breakdown; and (iv) Misuse Risks , including unintended damage and deliberate misuse. The central loop highlights that these risks can be amplified by autonomous and recursive self-improvement.

Figure 12: Applications of the self-improvement system. We highlight representative applications of self-improving LLMs and their extensions to self-evolving agents across six major domains: Code , Math , Healthcare , Finance , Science , and Auto Research .

Domain	Methods
Code	SATLUTION ( Yu et al., 2025a ) , CURE ( Wang et al., 2025h ) , LLMLOOP ( Ravi et al., 2025 ) , SSR ( Wei et al., 2025b ) , ReVeal ( Jin et al., 2026 ) , ACE ( Zhang et al., 2026c )
Math	SIAM ( Yu et al., 2024a ) , Goedel-Prover ( Lin et al., 2025b ) , OpenSIR ( Kwan et al., 2025 ) , SPHERE ( Singh et al., 2025 ) , rStar-Math ( Guan et al., 2025 ) , Xiong et al. (2025)
Healthcare	Agent Hospital ( Li et al., 2025c ) , HealthFlow ( Zhu et al., 2025a ) , MDTeamGPT ( Chen et al., 2025a ) , MedAgentSim ( Almansoori et al., 2025 ) , EvoClinician ( He et al., 2026 ) , MedReflect ( Huang et al., 2026b )
Finance	FINMEM ( Li et al., 2024a ) , FinAgent ( Zhang et al., 2024d ) , AlphaAgents ( Zhao et al., 2025b ) , FinRS ( Liu & Dang, 2025 ) , QuantAgents ( Li et al., 2025e ) , Chaudhari & Charate (2025)
Science	MatPilot ( Ni et al., 2024 ) , SciAgents ( Ghafarollahi & Buehler, 2025 ) , The AI Cosmologist I ( Moss, 2025 ) , Knowledge-extractor ( Yao et al., 2025 ) , AlphaEvolve ( Novikov et al., 2025 ) , Gödel Agent ( Yin et al., 2025 ) , Self-Developing ( Ishibashi et al., 2025 ) , SEAL ( Zweiger et al., 2025 ) , ADAS ( Hu et al., 2025c ) , AgentEvolver ( Zhai et al., 2025 ) , DGM ( Zhang et al., 2026a )
Auto Research	The AI Scientist-v2 ( Yamada et al., 2025 ) , Agent Laboratory ( Schmidgall et al., 2025 ) , InternAgent ( Team et al., 2025 ) , AI co-scientist ( Gottweis et al., 2026 ) , Agon ( Sun et al., 2026b ) , S1-NexusAgent ( Team, 2026 ) , FARS ( Analemma, 2026 ) , AutoResearchClaw ( Liu et al., 2026b ) , PaperOrchestra ( Song et al., 2026b ) , Paper2Rebuttal ( Ma et al., 2026 )

Table 12: Overview of domain-specific self-improving models and self-evolving agents. Approaches are organized by application domains, including code, math, healthcare, finance, science, and auto research.

Language Model Self-Improvement

Language model self-improvement studies how models use their own outputs, feedback, and experience to improve with less external supervision.

See all 170 papers

Oct 16, 2026

Language Models Learn to Improve Language Models

Large language models (LLMs) are increasingly used to automate AI research, including language-model (LM) development, offering a potential path toward recursive self-improvement. However, turning experimental feedback into a learned ability to propose effective improvements across models faces two challenges. First, training such an ability with real feedback is expensive: reinforcement learning (RL) requires evaluating tens of thousands of proposals, each potentially requiring a full model-training run. Even at ten GPU-hours per run, reward collection alone would consume hundreds of thousands of GPU-hours, motivating a cheaper surrogate. Second, surrogate feedback can become unreliable as policy optimization favors overestimated proposals when the proposal distribution shifts. To address these challenges, we propose LM2LM, which trains LLMs with RL to generate effective LM improvement proposals. To provide scalable feedback, we use a research world model (RWM), a frozen LLM that predicts proposal gains from real experimental records provided in context, as the reward model for RL. To maintain feedback reliability, we periodically execute current-policy proposals ranked highly by the RWM and add their measured outcomes to its context. Across eight autoregressive and diffusion LM development environments, real-execution evaluations on held-out tasks show that LM2LM enables Qwen3.8-27B to outperform GPT-6-sol, even when GPT-6-sol is given 300 real experimental records in context, while requiring 33% lower data-generation cost. LM2LM also generalizes to a model family unseen during training.

Minhua Lin, Zijun Wang, Zewen Liu +7

The Pennsylvania State University · UC Santa Cruz · Emory University +2

Language Model Self-Improvement Large Language Model-Guided Optimization RL for Language Models Reward Modeling

Ask PDF

Jul 5, 2026

Self-Reference in Large Language Models: The Introspection Threshold for Recursive Self-Improvement

The pursuit of self-evolving AI raises a critical question: when is autonomous self-improvement sustainable rather than degenerative? Drawing an analogy to von Neumann's complexity threshold for self-reproducing automata, we argue that sustainable recursive self-improvement in Large Language Models (LLMs) requires a functional analogue: introspection -- the system's capacity to simulate its own operations and target modifications. Grounded in Kleene's Second Recursion Theorem, we demonstrate the theoretical existence of such introspective programs. However, an empirical review reveals that while current LLMs exhibit quasi-introspection (e.g., partial metacognition), they fall short of true introspection due to structural bottlenecks: a lack of complete self-access, the feedforward nature of the Transformer, and computational class constraints that prevent fixed-point iteration. We conclude by outlining architectural paths to cross this complexity threshold and discussing the associated safety implications.

Jiang Zhang, Bing Yuan, Qian Zhang

School of Systems Science, Beijing Normal University · Swarma Research

Language Model Introspection Language Model Self-Improvement Recursive Self-Improvement Metacognition in Language Models

Ask PDF

Oct 3, 2026

When Is Enough Enough in Self-Evolving LLM Systems?

Self-evolving large language model (LLM) systems repeatedly propose, evaluate, and incorporate updates to prompts, skills, or other persistent artifacts. Despite their growing effectiveness, these systems typically operate under a predetermined iteration or compute budget, without a principled criterion to determine when further evolution is no longer worthwhile. This can lead to two undesirable consequences: unnecessary computation after performance has saturated and the risk of returning late updates that overfit or exploit the evaluation signal. These issues motivate us to study two fundamental questions: when should a self-evolving system stop, and what should it output once it stops? We address the first by formulating an online sequential testing problem and constructing an anytime-valid restart detector using the per-item paired evaluation outcomes already produced by self-evolving LLM systems. We address the second by formulating a change-point estimation problem and using the estimated transition to select an earlier artifact for output. The resulting procedure is plug-and-play and requires no modification of the underlying self-evolving algorithms. Across two self-evolving frameworks, three LLM model families, and five benchmarks, our method substantially reduces computation costs while maintaining comparable unseen-test performance. For example, on SearchQA with SkillOpt and DeepSeek V4 Flash, our method stops at round 4 rather than the full budget of 40, reducing token usage by 91.6% while achieving 82.43% unseen-test accuracy versus 82.00% under the full-budget run.

Enoch Yin, Bin Liu, Zhengling Qi

Department of Statistics, The George Washington University · Department of Statistics and Data Science, School of Management, Fudan University · Department of Decision Sciences, School of Business, The George Washington University

Early Stopping Language Model Self-Improvement LLM Evaluation Change-Point Detection Sequential Hypothesis Testing

Ask PDF

Abstract

Language Model Self-Improvement

Explore similar work

Language Models Learn to Improve Language Models

Self-Reference in Large Language Models: The Introspection Threshold for Recursive Self-Improvement

When Is Enough Enough in Self-Evolving LLM Systems?