Language Models Learn to Improve Language Models

Minhua Lin; Zijun Wang; Zewen Liu; Zhaotian Weng; Bing He; Zhan Shi; Benoit Dumoulin; Xiang Zhang

cs.CLOct 16, 2026

Language Models Learn to Improve Language Models

Authors: Minhua Lin, Zijun Wang, Zewen Liu, Zhaotian Weng, Bing He, Zhan Shi, Benoit Dumoulin, Xiang Zhang,

Organizations: The Pennsylvania State University · UC Santa Cruz · Emory University · UC Santa Barbara ·

Abstract

Large language models (LLMs) are increasingly used to automate AI research, including language-model (LM) development, offering a potential path toward recursive self-improvement. However, turning experimental feedback into a learned ability to propose effective improvements across models faces two challenges. First, training such an ability with real feedback is expensive: reinforcement learning (RL) requires evaluating tens of thousands of proposals, each potentially requiring a full model-training run. Even at ten GPU-hours per run, reward collection alone would consume hundreds of thousands of GPU-hours, motivating a cheaper surrogate. Second, surrogate feedback can become unreliable as policy optimization favors overestimated proposals when the proposal distribution shifts. To address these challenges, we propose LM2LM, which trains LLMs with RL to generate effective LM improvement proposals. To provide scalable feedback, we use a research world model (RWM), a frozen LLM that predicts proposal gains from real experimental records provided in context, as the reward model for RL. To maintain feedback reliability, we periodically execute current-policy proposals ranked highly by the RWM and add their measured outcomes to its context. Across eight autoregressive and diffusion LM development environments, real-execution evaluations on held-out tasks show that LM2LM enables Qwen3.8-27B to outperform GPT-6-sol, even when GPT-6-sol is given 300 real experimental records in context, while requiring 33% lower data-generation cost. LM2LM also generalizes to a model family unseen during training.

Figures & tables

Figure 1: Overview of automated LM development.

Model	Experimental Gain			Pairwise Win Rate (%) $\uparrow$
Model	Avg@4 $\uparrow$	Best@4 $\uparrow$	Neg. (%) $\downarrow$	Row vs. Base	LM2LM vs. Row
AR families
Base	$-7.94$	$11.39$	$70.4$	–	$93.3$
Base-ICL	$4.04$	$16.54$	$40.8$	$81.7$	$66.7$
GPT-6-sol	$6.51$	$8.79$	$52.5$	$83.3$	$56.7$
GPT-6-sol-ICL	$6.15$	$11.27$	$\mathbf{29.6}$	$78.3$	$51.7$

Table 1: Comparison with baselines on held-out research states. Experimental gains are reported in

10^{-3}

BPB (AR) and

10^{-2}

nats per token (diffusion). Best results in Avg@4, Best@4, Neg., and Row vs. Base are in bold .

Panel of Figure 3 — Figure 3: Real experimental gains (Avg@4) versus data-generation cost and time on held-out research states. (a, b) show AR tasks in $10^{-3}$ BPB, while (c, d) show diffusion tasks in $10^{-2}$ nats per token.

Model	Gain $\uparrow$		Win Rate (%) $\uparrow$
Model	Avg@4	Best@4	Row vs. Base	LM2LM vs. Row
Base	$3.53$	$31.39$	–	$87.5$
Base-ICL	$2.97$	$25.22$	$58.3$	$100.0$
GPT-6-sol	$20.19$	$22.75$	$66.7$	$62.5$
GPT-6-sol-ICL	$21.63$	$25.27$	$75.0$	$62.5$
LM2LM	$\mathbf{53.72}$	$\mathbf{86.69}$	$\mathbf{87.5}$	–

Table 2: Generalization to the held-out Qwen3 family. Gains are reported in

10^{-3}

BPB. Best results in Avg@4, Best@4, and Row vs. Base are in bold .

Panel of Figure 4 — Figure 4: Ablation of on-policy calibration on held-out AR research states at step 96.

Panel of Figure 5 — Figure 5: RWM feedback reliability on held-out AR research states .

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

			Held-out	Records
Family	Training tasks	Held-out tasks (states)	states	$M_{0}$	Pool
AR families
OLMo-3	100m-4c , 100m-8c , 190m-1c , 190m-2c , 370m-1c , 370m-2c , 760m-1c	100m-1c (17), 100m-2c (8), 190m-4c (4)	29	16	47
Qwen3	q06b-1b , q06b-2b , q17b-1b , q17b-2b , q152m-8b , q267m-1b , q267m-2b	q152m-2b (8), q267m-4b (4)	12	25	62
NanoChat	d8-8k-0p82b , d8-8k-3p28b , d12-8k-8p59b , d16-8k-4p58b , d20-8k-8p49b	d12-8k-2p15b (17)	17	15	79
Marin	153m-2p9b , 153m-8b , 267m-2p9b , 267m-8b , 486m-2p9b	486m-8b (2)	2	11	30

Appendix

Table 3: Training and held-out tasks, evaluation states, and experimental records by model family . Parentheses indicate the number of states per held-out task;

M_{0}

and Pool denote the initial RWM records and the 300-record baseline dataset, respectively.

Family	Task	Parameters	Budget	$b_{e}$	$\sigma_{e}$
OLMo-3	100m-1c	153M	2.0B tokens	1.3750	0.0011
OLMo-3	100m-2c	153M	4.0B tokens	1.2939	0.0056
OLMo-3	190m-4c	267M	15.2B tokens	1.1275	0.0017
Qwen3	q152m-2b	153M	2.0B tokens	1.3641	0.0025
Qwen3	q267m-4b	267M	4.0B tokens	1.1683	0.0042
NanoChat	d12-8k-2p15b	286M	2.15B tokens	0.8135	0.0007

Appendix

Table 4: Held-out tasks, model sizes, training budgets, and reference statistics . Parameters are total model parameters; budgets are training tokens (AR) or FLOPs with corresponding tokens (diffusion).

b_{e}

and

\sigma_{e}

are the mean and standard deviation of reference validation loss over three runs, in BPB (AR) and nats per token (diffusion).

Direction	Example changes	AR	Diffusion
Attention reach	RoPE fraction and base, attention windows, token-mixer layout	42	–
Decay shape	learning-rate decay tail, warmup, weight averaging, $\beta_{2}$	34	–
Update geometry	Muon iterations and scaling, $\beta_{2}$ , weight decay, gradient clipping	22	9
Capacity allocation	per-layer FFN widths, attention–FFN parameter split	29	–
KV / state budget	placement and slots of context-memory layers, KV heads	23	–
Loss geometry	confidence-weighted loss, label smoothing, position-scheduled loss	23	–

Appendix

Table 5: Distribution of pre-collected experimental records across mechanism directions for AR and diffusion families . Each proposal is counted once under its primary direction. Other includes head shape and context-window curriculum.

Panel of Table 6 — Table 6: A representative mechanism card from the library .

Model	Experimental Gain $\uparrow$		Pairwise Win Rate $\uparrow$
Model	Avg@4	Best@4	Row vs. Base	LM2LM vs. Row
OLMo-3
Base (Qwen3.8-27B)	$-10.72$	$10.48$	–	$0.862$
Base-ICL	$-3.58$	$10.97$	$0.724$	$0.759$
GPT-6-sol	$5.54$	$8.26$	$0.828$	$0.586$
GPT-6-sol-ICL	$-1.16$	$6.21$	$0.655$	$0.655$

Appendix

Table 7: Per-family results on held-out AR tasks using real execution . Avg@4 and Best@4 are reported in

10^{-3}

BPB. Win rate compares the first model against the second, with values above 0.5 favoring the first. Best results in each family are in bold .

Model	Experimental Gain $\uparrow$		Pairwise Win Rate $\uparrow$
Model	Avg@4	Best@4	Row vs. Base	LM2LM vs. Row
E2D2
Base (Qwen3.8-27B)	$-5.37$	$7.65$	–	$1.000$
Base-ICL	$3.28$	$6.88$	$0.667$	$0.667$
GPT-6-sol	$3.23$	$6.12$	$0.667$	$0.667$
GPT-6-sol-ICL	$3.72$	$4.98$	$0.833$	$0.833$

Appendix

Table 8: Per-family results on held-out diffusion tasks using real execution. Avg@4 and Best@4 are reported in

10^{-2}

nats per token. Win rate compares the first model against the second, with values above 0.5 favoring the first. Best results in each family are in bold .

	AR				Diffusion
Model	Avg@4 $\uparrow$	Best@4 $\uparrow$	$>2\sigma_{e}$ (%) $\uparrow$	Rel. (%) $\uparrow$	Avg@4 $\uparrow$	Best@4 $\uparrow$	$>2\sigma_{e}$ (%) $\uparrow$	Rel. (%) $\uparrow$
Base	$-8.83$	$5.65$	$22.5$	$-0.75$	$-31.61$	$54.25$	$34.7$	$-1.25$
Base-ICL	$0.73$	$8.67$	$30.8$	$0.25$	$28.10$	$62.66$	$73.6$	$1.22$
GPT-6-sol	$3.51$	$4.94$	$35.4$	$0.47$	$28.61$	$43.02$	$61.1$	$0.60$
GPT-6-sol-ICL	$4.91$	$6.58$	$40.0$	$0.46$	$33.99$	$37.52$	$\mathbf{100.0}$	$1.49$
SFT	$-11.84$	$2.24$	$11.3$	$-1.01$	$-11.71$	$24.93$	$31.9$	$-0.90$

Appendix

Table 9: Proposal-level results of Table 1 , normalized by task-specific seed noise

\sigma_{e}

. Avg@4 and Best@4 are reported in units of

\sigma_{e}

;

>2\sigma_{e}

denotes the fraction of proposals with gains exceeding

2\sigma_{e}

, and Rel. denotes Avg@4 as a percentage of reference loss. Best results are in bold .

		Data generation	Proposer training		Total
Method		$C_{\$ } $($ k)	GPU-h	Cost ($k)	($k)
ICL ( $K=100$ – $300$ )		$19.8$ – $60.0$	$0$	$0$	$19.8$ – $60.0$
SFT		$0$	$1.6$	$<0.1$	$<0.1$
RFT-Real		$60.0$	$0.8$	$<0.1$	$60.0$
LM2LM	$M_{0}$	$23.9$	$152$	$2.2$	$26.1$
	$M_{1}$	$28.3$	$434$	$6.2$	$34.5$

Appendix

Table 10: Total cost : data-generation cost (Table 11 ) plus the GPU cost of proposer training, in B200 GPU-hours (H200 converted at a ratio of 1.198) and thousands of dollars at $14.208 per GPU-hour.

		Data-generation cost						Gain $\uparrow$
Method	$K$	Real runs	GPU-h	Tokens (M)	$C_{\$ }$	$T_{\mathrm{run}}$ (h)	$T_{\mathrm{RWM}}$ (h)	Avg@4	Best@4
Base-ICL	$0$	$0$	$0$	$0.0$	$0$	$0.0$	$0.0$	$-7.94$	$11.39$
	$100$	$100$	$1{,}396$	$0.0$	$19{,}839$	$168.7$	$0.0$	$-7.64$	$9.10$
	$150$	$150$	$2{,}040$	$0.0$	$28{,}986$	$246.8$	$0.0$	$-2.12$	$10.99$
	$200$	$200$	$2{,}780$	$0.0$	$39{,}495$	$334.2$	$0.0$	$-1.98$	$11.06$
	$300$	$300$	$4{,}223$	$0.0$	$60{,}001$	$510.0$	$0.0$	$4.04$	$16.54$

Appendix

Table 11: Data-generation cost, time, and real experimental gains on held-out AR states (Fig. 3 ).

T_{\mathrm{run}}

and

T_{\mathrm{RWM}}

denote cumulative real-execution and RWM prediction time. Gains are reported in

10^{-3}

BPB.

Panel of Figure 6 — Figure 6: Real experimental gains (Best@4) versus data-generation cost and time on held-out research states. (a, b) show AR tasks in $10^{-3}$ BPB, while (c, d) show diffusion tasks in $10^{-2}$ nats per token.

Panel of Figure 7 — Figure 7: M-UPD-72 under top-rated and random calibration. (a) Fraction of NanoChat training proposals that apply it. (b) Mean RWM reward of these proposals in the training reward scale; the dashed line is their real reward on held-out states. C1–C3 mark the three calibrations.

Panel of Figure 8 — Figure 8: OLMo token-mixing change under top-rated and random calibration. (a) Fraction of OLMo training proposals that apply it. (b) Mean RWM reward of these proposals in the training reward scale. C1–C3 mark the three calibrations.

Panel of Figure 9 — Figure 9: Additional feedback diagnostics on held-out AR states through step 128. Each variant is evaluated using its own training RWM. (d) reports the mean real advantage of proposals with positive predicted advantage. Error bars indicate 95% bootstrap confidence intervals over states.

Figure 10: Pairwise accuracy of the same proposals of LM2LM scored with the initial records

M_{0}

and with the latest calibrated records (

M_{2}

up to step 64,

M_{3}

at step 96, and

M_{4}

at step 128).

Figure 11: Predicted versus real advantages of the proposer’s own proposals on held-out AR states, at steps 16, 64, and 128. Top: static variant scored by

W(M_{0})

; bottom: LM2LM scored by the RWM in force. Points on the dotted diagonal are predicted exactly; the shaded region contains wrongly reinforced proposals (positive predicted, negative real advantage). Each panel reports the pairwise accuracy (acc) and wrong-reinforcement rate (wrong). Values are clipped to

\pm 4.5

for display.

Panel of Figure 12 — Figure 12: Case study of the NanoChat change M-UPD-72 (tokens per optimizer update) for the static variant and LM2LM. (a) Fraction of NanoChat training proposals that apply it. (b) Mean reward the training RWM assigns to these proposals; the dashed line is their real reward on held-out states. (c) Fraction of held-out NanoChat proposals that apply it. C1 and C2 mark the first two calibrations of LM2LM.

Base model	Qwen3.8-27B
Adaptation	LoRA, rank 16, $\alpha=32$ , no dropout
LoRA targets	all linear layers of attention and MLP blocks
Training pairs (AR / diffusion)	SFT 218 / 82; RFT-Real 76 / 38
Loss	token-level NLL on proposal tokens only
Optimizer	AdamW, $\beta=(0.9,0.999)$ , no weight decay
Learning rate	$10^{-4}$ , 3% linear warmup, cosine decay to 10%

Appendix

Table 12: SFT and RFT-Real training configuration.

Panel of Table 13 — Table 13: RWM prompt. Slots in {braces} are filled per model family, e.g., for OLMo-3 the experiment type is “language-model pretraining” and the gain is the reference ten-task macro BPB minus the proposal’s.

Panel of Table 14 — Table 14: Builder prompt (abbreviated). The instructions are followed by the build task, the task’s typed configuration space and default configuration, and, for implementation, the editable runtime files, the overlays already built for the family, and the contract of the targeted setting.

Language Model Self-Improvement

Language model self-improvement studies how models use their own outputs, feedback, and experience to improve with less external supervision.

See all 171 papers

Chain-of-Experience for Continual LLM Improvement
Aug 18, 2026Haoqin Tu, Yunhao Fang, Yizhong Wang +2Language Model Self-Improvement Continual Learning for LLMs
UC Santa Cruz · Bytedance Seed
Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation
Sep 18, 2025Yujun Zhou, Zhenwen Liang, Haolin Liu +7Language Model Self-Improvement RL for Language Models
Tencent AI Lab · University of Notre Dame · University of Virginia
Self-Improvement of Large Language Models: A Technical Overview and Future Outlook
Mar 26, 2026Haoyan Yang, Mario Xerri, Solha Park +4Language Model Self-Improvement LLM Self-Refinement
Zesearch NLP Lab, Stony Brook University

Aug 18, 2026J/K move · Enter open · S save

Chain-of-Experience for Continual LLM Improvement

Haoqin Tu, Yunhao Fang, Yizhong Wang, Cihang Xie+1

UC Santa Cruz · Bytedance Seed

Language Model Self-Improvement Continual Learning for LLMs LLM Evaluation

Open paper AskPDF arXiv

Humans continuously learn from experience, whereas conventional large language model (LLM) evaluations ignore the models' ability to improve through inference-time interaction. In this paper, we study how LLMs learn from iterative experience at test time, a setting we refer to as Chain-of-Experience (CoE), where models accumulate experiential traces through iterative interactions with self or environmental feedback to form a continual improvement loop beyond zero-shot inference. We instantiate CoE with diverse feedback mechanisms, including model self-feedback and environmental signals such as correctness or public coding test pass rates, and evaluate across math, coding, and knowledge domains using 8 LLMs, including GPT-5, Gemini-2.5 Pro, Claude-4.5 Sonnet. Our study shows that leveraging iterative experience consistently outperforms feedback-free baselines, achieving substantial gains with self feedback alone, alongside a 5.6% overall improvement and 19% lower API cost across tasks and models. We further show that combining complementary feedback channels (e.g., model and correctness signals) yields additional gains, and that CoE delivers higher accuracy per token than existing test-time strategies. We observe a positive correlation between LLM base ability and improvement capacity, and show that models remain robust under weak or spurious feedback, with different feedback contributing to distinct improvement aspects and most gains emerging early in the iterations.

Abstract

Language Model Self-Improvement

Explore similar work

Chain-of-Experience for Continual LLM Improvement

Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation

Self-Improvement of Large Language Models: A Technical Overview and Future Outlook