Do LLM Agents Still Need to Be Told How to Self-Improve?
Organizations: MIT CSAIL · University of Tübingen
Abstract
LLM agents increasingly drive their own learning. Self-improvement methods delegate various components of model optimization to the models themselves. As models become more capable, this motivates a basic question: do these increasingly capable models still need us to specify how they should self-improve? In this paper we study an autonomous paradigm in which, given basic primitives, resources, and a general goal for self-improvement, agents construct their own self-learning algorithm for context-based optimization. Using the setting of gameplay as a testbed, a frozen LLM agent is given a compute budget and tools for collecting experience and updating text documents, and allowed to freely experiment without an externally imposed loop or access to real-world data. We find that sufficiently capable models do self-improve under these conditions, demonstrating strategies that align with classic cognitive frameworks for self-regulated learning and constructs from machine learning literature on self-generated curricula. Furthermore, we study the extent to which the learned text artifacts transfer to external benchmarks, observing improvement across different game horizons, mechanics, and 3D navigation.
Figures & tables
| Player | Text artifact source | Win rate (%) |
|---|---|---|
| Opus 5 | None | 25.4 3.6 |
| GPT-6 Astra | 47.5 4.5 | |
| GPT-6 Astra | None | 67.1 3.8 |
| Opus 5 | 72.1 4.0 |
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | # Games | Turns ( ) | Tokens / game | Eval rollouts / game |
|---|---|---|---|---|
| PuzzleScript-t20 | 20 | 20 | 25,000 | 4 |
| PuzzleScript-t30 | 20 | 30 | 31,250 | 4 |
| ARC-AGI-3 (L1) | 17 | 30 | 31,250 | 4 |
| MazeBench | 16 | 100 | 75,000 | 2 |
| Setting | Default Value |
|---|---|
| Teacher update budget | 20 |
| Teacher gameplay budget | 150 |
| Max turns per game | {20,30} |
| Max output tokens per game (summed over turns) | {25000,31250} |
| Output-token limit per model response (teacher and player) | 20000 |
| Reasoning effort (teacher and player) | low |
| Game level | Baseline | Learned |
|---|---|---|
| Bricks and Hammer (L10) | 100.0 | 100.0 0.0 |
| Lights Out (L0) | 100.0 | 100.0 0.0 |
| Philosophy (L61) | 100.0 | 91.7 7.2 |
| Power Problems (L0) | 100.0 | 100.0 0.0 |
| Puzzle Script Tutorial (L15) | 100.0 | 100.0 0.0 |
| STUCK IN A HOLE (L10) | 100.0 | 100.0 0.0 |
LLM Agent Self-Improvement
Large language model agent self-improvement studies how agents use experience and feedback to improve their behavior, tools, and workflows.
See all 296 papers