Chain-of-Experience for Continual LLM Improvement

Haoqin Tu; Yunhao Fang; Yizhong Wang; Cihang Xie; Shen Yan

cs.CLAug 18, 2026

Chain-of-Experience for Continual LLM Improvement

Authors: Haoqin Tu, Yunhao Fang, Yizhong Wang, Cihang Xie, Shen Yan

Organizations: UC Santa Cruz · Bytedance Seed

Abstract

Humans continuously learn from experience, whereas conventional large language model (LLM) evaluations ignore the models' ability to improve through inference-time interaction. In this paper, we study how LLMs learn from iterative experience at test time, a setting we refer to as Chain-of-Experience (CoE), where models accumulate experiential traces through iterative interactions with self or environmental feedback to form a continual improvement loop beyond zero-shot inference. We instantiate CoE with diverse feedback mechanisms, including model self-feedback and environmental signals such as correctness or public coding test pass rates, and evaluate across math, coding, and knowledge domains using 8 LLMs, including GPT-5, Gemini-2.5 Pro, Claude-4.5 Sonnet. Our study shows that leveraging iterative experience consistently outperforms feedback-free baselines, achieving substantial gains with self feedback alone, alongside a 5.6% overall improvement and 19% lower API cost across tasks and models. We further show that combining complementary feedback channels (e.g., model and correctness signals) yields additional gains, and that CoE delivers higher accuracy per token than existing test-time strategies. We observe a positive correlation between LLM base ability and improvement capacity, and show that models remain robust under weak or spurious feedback, with different feedback contributing to distinct improvement aspects and most gains emerging early in the iterations.

Figures & tables

Figure 1: Summarized results on four benchmarks across math, code, and knowledge over four LLMs. Iterative problem solving under CoE provides three benefits: upper: by incorporating feedback in CoE, the average performance across four LLMs outperforms other test-time augmentations; lower left: models are capable of achieving the better performance with lower API cost with feedback ; lower right: LLMs that perform better on the task display better improving capabilities through CoE with moderate to strong Pearson correlation. We present more explanations regarding baselines in Section 4.1 .

Figure 2: An overview of our studies iterative improvement loop for LMs. The model $\mathcal{M}$ learns by repeatedly interacting with the environment $\mathcal{E}$ ( e.g. , model simulation or coding environment). In each cycle, $\mathcal{M}$ generates actions conditioned on the input and accumulated experience, then receives feedback from $\mathcal{E}$ to form a new experience. We investigate four specific feedback in this study: none, execution, model, and correctness feedback.

Figure 3: Results of five state-of-the-art LLMs on six benchmarks using different generation techniques. Models under CoE with different levels of feedback ( correctness/executor , self , and binary executor feedback) generally perform better than the baseline strategies ( no feedback , DC , ICL , and the ones with different reasoning efforts ). Results are averaged over 3 runs and we shade the standard deviation with a lighter color and plot bars showing model performance under different reasoning efforts. The full results are in Appendix H .

Figure 4: Total cost of each model over task completion vs. its best performance within 20 iterations. LLMs with feedback generally achieve higher scores with fewer costs (at the upper left), while iterative experience without feedback generally falls behind (at lower right). We provide full results over six benchmarks in Appendix H .

Figure 5: Zero-shot performance of models (Base Capacity) and the learning gain show positive Pearson correlations (r), indicating that better-performing LLMs possess greater improving capability. Scores are averaged across 3 runs.

Figure 6: Percentages of different reasons for LLMs’ improvement patterns from 6,630 incorrect to correct response pairs. We employ GPT-5 to conduct this automatic analysis.

	AIME 2025		LiveBench (Code)		OmniMath
Setting	Acc	Best R	Acc	Best R	Acc	Best R
Dual (Model + Corr/Exec)	76.7	R19	81.2	R15	73.5	R17
Correctness / Executor	70.0	R13	78.1	R15	74.5	R17
Binary Executor	–	–	71.9	R13	–	–
Model	60.0	R6	57.8	R17	50.5	R9
Model + DC	50.0	R8	51.6	R15	46.0	R10

Table 1: Dual feedback and principled experience selection results using Claude 4.5 Sonnet. Dual feedback combines model feedback with correctness (math) or executor (code) signals. Memory-based methods (DC, SimpleMem) are applied within-task with no cross-task leakage. Acc: best accuracy (%) over 20 iterations; Best R: iteration achieving best performance.

Feedback	AIME 2025		GPQA Diamond
	GPT-5 mini	o4 mini	GPT-5 mini	o4 mini
Self	93.3	91.1	79.9	78.8
SelMV Self	91.1	88.9	80.4	79.5
All Correct	90.0	73.3	79.3	75.8
SelMV Correct	93.3	73.3	79.3	76.3
Incorrect	91.7	83.3	79.3	72.7

Table 2: The best performance over 20 iterations under constant correct ” or incorrect ” feedback ( e.g. , “the answer is correct”). Selective majority voting (SelMV) helps LLMs maintain performance. Results are averaged over 3 runs with best scores emphasized .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Method	AIME 2025	LiveCodeBench (V6)	LiveBench (Code)	OmniMath	GPQA Diamond	EvaLearn
ICL	71.83%	62.50%	65.46%	53.12%	78.45%	40.99%
ACE	71.98%	66.94%	69.38%	50.33%	76.58%	42.54%
DC	73.33%	63.59%	68.58%	48.64%	79.56%	42.68%
w/o Feedback	77.78%	72.57%	60.16%	65.17%	80.02%	44.91%
Reasoning-high	69.05%	70.63%	55.46%	61.81%	76.21%	39.58%
Reasoning-low	60.48%	61.03%	55.38%	50.60%	72.92%	29.34%

Appendix

Table 3: Average performance comparison (%) across different LLMs on different datasets. For baselines, ICL, ACE, DC stands for few-shot in-context learning, agentic context engineering, and dynamic cheatsheet, respectively.

Figure 7: Model performance using constant “incorrect” and “correct” feedback. By leveraging the selective majority voting, LLMs show decent performance when facing spurious feedback on math tasks.

Figure 8: Performance of GPT-5 mini, o4-mini, and o3-mini with extended iterations of experience to 50 on math tasks. We mark performance points within the first 20 iterations of experience in blue, and those from iterations 20 to 50 in green.

Figure 9: Model performance of GPT-5 and GPT-5 mini with external model feedback on two math tasks.

Category	Agreement (%)	Cohen’s $\kappa$
Feedback Fidelity	84.0	0.81
Self Reflection	72.0	0.71
Specification Recall	80.0	0.78
Random	68.0	0.63
Overall	76.0	0.768

Appendix

Table 4: Cohen’s Kappa agreement between GPT-5 judge and human annotators across four improvement categories on 100 sampled trajectories. The overall

\kappa

of 0.768 indicates substantial agreement [ landis1977measurement ] .

Figure 10: On BrowseComp-Plus, self-feedback models fall behind as the task requires external search-based knowledge.

Figure 11: Full results of eight state-of-the-art LLMs on six benchmarks incorporating different generation techniques. Model accuracies with different levels of feedback ( correctness/executor , self , and binary executor feedback) generally perform better than the baseline strategies.

Figure 12: Full results of total API cost (in dollar) vs. best model performance over 20 iterations. LLMs with detailed feedback ( i.e. , self feedback ) achieves decent results with fewer costs, while CoE without feedback generally falls behind (at lower right).

Dataset	Method	Tokens	Acc (%)
AIME 2025	CEF	108,734	84.6
	SF	108,231	83.8
	NF	106,825	74.1
	DC	11,233	74.7
OmniMath	CEF	176,412	74.2
	SF	175,806	72.1

Appendix

Table 5: Token complexity vs. accuracy across methods. CEF: Correctness/Executor Feedback; SF: Self Feedback; NF: No Feedback; DC: Dynamic CheatSheet. Token counts are aggregated across all iterations.

Language Model Self-Improvement

Language model self-improvement studies how models use their own outputs, feedback, and experience to improve with less external supervision.

See all 171 papers

Language Models Learn to Improve Language Models
Oct 16, 2026Minhua Lin, Zijun Wang, Zewen Liu +7Language Model Self-Improvement Large Language Model-Guided Optimization
The Pennsylvania State University · UC Santa Cruz · Emory University +2
Continual Learning Exam: Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents
Oct 25, 2026Haoyu Zhao, Sitian Tu, Yuxuan Huang +16Continual Learning for LLM Agents Long-Horizon Agent Evaluation
Rethinking Continual Experience Internalization for Self-Evolving LLM Agents
Jun 3, 2026Jingwen Chen, Wenkai Yang, Shengda Fan +7Continual Learning for LLM Agents Continual Learning for LLMs
1Gaoling School of Artificial Intelligence, Renmin University of China · School of Software, Beihang University · 3Meituan

Oct 16, 2026J/K move · Enter open · S save

Language Models Learn to Improve Language Models

Minhua Lin, Zijun Wang, Zewen Liu, Zhaotian Weng+6

The Pennsylvania State University · UC Santa Cruz · Emory University+2

Language Model Self-Improvement Large Language Model-Guided Optimization RL for Language Models Reward Modeling

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used to automate AI research, including language-model (LM) development, offering a potential path toward recursive self-improvement. However, turning experimental feedback into a learned ability to propose effective improvements across models faces two challenges. First, training such an ability with real feedback is expensive: reinforcement learning (RL) requires evaluating tens of thousands of proposals, each potentially requiring a full model-training run. Even at ten GPU-hours per run, reward collection alone would consume hundreds of thousands of GPU-hours, motivating a cheaper surrogate. Second, surrogate feedback can become unreliable as policy optimization favors overestimated proposals when the proposal distribution shifts. To address these challenges, we propose LM2LM, which trains LLMs with RL to generate effective LM improvement proposals. To provide scalable feedback, we use a research world model (RWM), a frozen LLM that predicts proposal gains from real experimental records provided in context, as the reward model for RL. To maintain feedback reliability, we periodically execute current-policy proposals ranked highly by the RWM and add their measured outcomes to its context. Across eight autoregressive and diffusion LM development environments, real-execution evaluations on held-out tasks show that LM2LM enables Qwen3.8-27B to outperform GPT-6-sol, even when GPT-6-sol is given 300 real experimental records in context, while requiring 33% lower data-generation cost. LM2LM also generalizes to a model family unseen during training.

Abstract

Language Model Self-Improvement

Explore similar work

Language Models Learn to Improve Language Models

Continual Learning Exam: Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents

Rethinking Continual Experience Internalization for Self-Evolving LLM Agents