cs.LGMar 25, 2026
SaveUnderstanding the Challenges in Iterative Generative Optimization with LLMs
Organizations: Google DeepMind · CNRS · Stanford University · Carnegie Mellon University ·
Abstract
Generative optimization uses large language models (LLMs) to iteratively improve artifacts (such as code, workflows or prompts) using execution feedback. It is a promising approach to building self-improving agents, yet in practice remains brittle: despite active research, only 9% of surveyed agents used any automated optimization. We argue that this brittleness arises because, to set up a learning loop, an engineer must make ``hidden'' design choices: What can the optimizer edit and what is the "right" learning evidence to provide at each update? We investigate three factors that affect most applications: the starting artifact, the credit horizon for execution traces, and batching trials and errors into learning evidence. Through case studies in MLAgentBench, Atari, and BigBench Extra Hard, we find that these design decisions can determine whether generative optimization succeeds, yet they are rarely made explicit in prior work. Different starting artifacts determine which solutions are reachable in MLAgentBench, truncated traces can still improve Atari agents, and larger minibatches do not monotonically improve generalization on BBEH. We conclude that the lack of a simple, universal way to set up learning loops across domains is a major hurdle for productionization and adoption. We provide practical guidance for making these choices.
Figures & tables
Figure 1: The learning loop of generative optimization.
Figure 2: Three Key Decisions for Implementing a Learning Loop . To set up iterative generative optimization, an agent engineer must make three core decisions. For the initial system: What (1) starting artifacts ( instructions, files, specs) to provide? Different initializations can lead to different solution spaces. For the learning context ( ): What learning evidence to provide to the LLM optimizer – (2) how many steps to include per trial ( credit horizon ) and (3) how many trials to batch together ( experience batching ).
Figure 3: Starting artifacts. The one-function and many-function initializations provide equivalent docstring information but differ in pipeline modularization.
Figure 4: Left: Kaggle leaderboard performance, reported as the percentile of the trained ML model submissions (higher is better). Right: MLAgentBench results. We run both systems 5 times and report the average and best Kaggle submission score across runs. One and Multi refer to a one-function and many-functions workflow design for the learned ML pipeline (Figure 3 ). Both systems use the same LLM API.
Figure 5: Credit Horizon Comparison Across Games . Performance of code agents optimized with one-step (immediate reward) vs multi-step (full rollout) credit horizons across 5 trials. Both setups use agents with the same starting artifacts. We see that observing full execution traces (multi-step) is only useful in discovering better code in 4 out of 8 games, suggesting credit horizon is a design choice and can be tuned to each task.
Figure 6: Training Efficiency Comparison . Deep RL training builds on CleanRL ( Huang et al., 2022 ) ; PPO uses ten parallel environments for sampling while DQN and LLM use a single environment. Both PPO and DQN use batch training on a single A100 GPU. All algorithms use object-centric inputs from OCAtari. The deep RL algorithms convert this information into numeric vectors, whereas the LLM takes the raw dictionary without additional annotation. Scores are normalized to map 0% to random performance on the task and 100% to human performance. See Appendix B.5 for details.
Figure 7: Validation Learning Curves Across Batch Sizes . Performance across optimization iterations for different batch sizes (3 trials, shaded area shows standard error). Larger batches often show faster initial learning but can plateau earlier.
| Batch | Dyck | Shapes | Linguini | Causal | Disambig | Bool Expr | Movie | Boardgame |
|---|---|---|---|---|---|---|---|---|
| 1 | 0.183 0.049 | 0.343 0.039 | 0.149 0.024 | 0.375 0.146 | 0.537 0.036 | 0.177 0.005 | 0.889 0.038 | 0.341 0.032 |
| 3 | 0.063 0.010 | 0.389 0.040 | 0.234 0.012 | 0.408 0.097 | 0.295 0.091 | 0.238 0.006 | 0.683 0.119 | 0.278 0.009 |
| 5 | 0.190 0.031 | 0.200 0.099 | 0.170 0.030 | 0.531 0.018 | 0.526 0.035 | 0.154 0.034 | 0.810 0.016 | 0.276 0.007 |
Table 1: Test Set Performance Across Batch Sizes . Best accuracy per task is bolded; standard error shown in gray. Different tasks have different optimal batch sizes, and larger batches do not always improve performance on the full test set (175+ holdout test examples). Base model is Claude Sonnet-3.5-v2.
| Framework | Batch | Boardgame | Causal | Movie |
|---|---|---|---|---|
| LangGraph | 1 | 0.208 0.152 | 0.425 0.082 | 0.680 0.031 |
| 3 | 0.278 0.012 | 0.430 0.109 | 0.703 0.035 | |
| 5 | 0.280 0.020 | 0.493 0.139 | 0.737 0.028 | |
| DSPy | 1 | 0.288 0.035 | 0.472 0.027 | 0.448 0.351 |
| 3 | 0.377 0.025 | 0.341 0.031 | 0.189 0.251 | |
| 5 | 0.299 0.035 | 0.278 0.015 | 0.495 0.173 |
Table 2: Cross-framework check. Test accuracy across minibatch sizes on the three shared tasks. Standard error obtained with 3 trials.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1 : Trace program for the two MLAgentBench initializations. The one-function design exposes the full pipeline through a single trainable function, whereas the many-function design exposes preprocessing, feature selection, model training, and prediction as separate trainable components.
Figure A2 : Feedback templates used for the learned ML pipeline on the Spaceship Titanic and Housing Price tasks.
Table A1 : Staged feedback templates for Spaceship Titanic and Housing Price. Highlighted phrases mark the parts that differ across feedback levels within each task.
Figure A3 : One Optimization Trajectory of LLM Optimizer on the ML Pipeline. We show a single optimization trajectory from the Spaceship Titanic dataset. The feedback string reports Accuracy, F1 score, Precision, and Recall, but both staged feedback and checkpoint selection are based on validation F1. Each dot (step) represents a full execution of the training pipeline code written by the LLM optimizer with a fully trained model. We see that LLM optimizer tends to overfit to the training data and exhibits classical overfitting behavior even without numerical gradient descent.
| Name | Repo |
|---|---|
| Deep RL Repo (CleanRL) | https://github.com/ameliakuang/cleanrl_obj_centric |
| LLM Code Repo | https://github.com/ameliakuang/LLM-Game-Playing-Agents |
| Wandb Log for DQN Training | https://wandb.ai/kuangzy-amelia-stanford-university/obj-dqn-5trials-new |
| Wandb Log for PPO Training | https://wandb.ai/kuangzy-amelia-stanford-university/obj-ppo-5trials-flatten |
Table B1: Repositories and experiment logs used for the Atari experiments.
| Game | One-Step | Multi-Step | Optimizer Steps | Evaluation Protocol |
|---|---|---|---|---|
| Pong | 1 step | 400 steps | 20 | 10 episodes 4000 steps |
| Breakout | 1 step | 300 steps | 30 | 1 episode 4000 steps |
| Space Invaders | 1 step | 25 steps | 20 | 1 episode 4000 steps |
| Freeway | 1 step | 100 steps | 30 | 3 episodes 2500 steps |
| Asterix | 1 step | 100 steps | 30 | 3 episodes 20000 steps |
| Enduro | 1 step | 100 steps | 30 | 1 episode 5000 steps |
Table B2: Per-game Atari credit-horizon configurations. All games use ALE {env}-NoFrameskip-v4 environments with action repeat 4, sticky action probability 0.0, optimizer memory size 5, and Claude Sonnet-3.5 via OptoPrime.
Figure B1 : OCAtari object annotations across eight Atari games. For each game, the left image shows the raw RGB frame and the right image shows the corresponding OCAtari ( Delfosse et al., 2024 ) annotated frame. These examples illustrate how pixel observations are translated into object-centric state representations across diverse Atari environments.
Figure B2 : Example of a returned observation, game information, and reward from Breakout. Note that this is the raw return from OCAtari, not further processed. There is no annotation of what “RB”, “OB”, “YB”, “GB”, “AB”, and “BB” represent. They are acronyms for the bounding boxes with colors.
Figure B3 : Representative initial systems for Atari games . We show the workflow design of different decision-making program components for three representative Atari game agents. The learned agent receives an object-centric dictionary of game state information and uses Python code to process the observation and output an action.
Table B3 : Staged feedback templates for Pong and Breakout. Highlighted phrases mark the parts that differ between the corresponding High, Medium, and Low feedback levels across the two games.
| Performance Level | Feedback |
|---|---|
| High (Reward 300) | “Great job! You’re performing well with an average score of 320 . Try to improve your shooting accuracy and dodging .” |
| Medium (100 Reward 300) | “Good progress! Your average score is 180 . Focus on better timing for shooting and avoiding enemy projectiles .” |
| Low (Reward 100) | “Your average score is 70 . Try to improve your strategy for shooting aliens and dodging projectiles .” |
Table B4: Staged feedback for the Space Invaders agent at different performance levels.
Table B5 : Left: concrete settings for the object-centric deep RL baselines. Right: median runtime and score statistics across games.
Figure B4 : Representative high-level strategies discovered by the optimizer. These short excerpts show that the learned revisions do more than tune local action thresholds: Pong adds distance- and momentum-aware interception, Breakout begins to target tunnels and higher-value bricks, and Space Invaders combines action constraints with threat-aware positioning.
| Parameter | Value |
|---|---|
| Training examples per task | First 15 examples |
| Validation examples per task | Next 10 examples |
| Held-out test examples per task | Remaining examples (typically 175+) |
| Batch sizes compared | 1, 3, 5 |
| Total optimizer update steps | 15 |
| Number of trials (seeds) | 3 |
Table C1 : BigBench Extra Hard experimental configurations.
Figure C1 : Agent design for BigBench Extra Hard . The agent consists of two optimizable components: call_llm and answer_extraction . During optimization, multiple traces can be concatenated through the batchify operator before being shown to the optimizer.
Figure D1 : An example workflow graph produced by a single execution of . The graph contains input node , intermediate node , output node , feedback , and the parameter nodes , that influence the output.
Figure D2 : Representing different learning problems through graph templates. Each dotted rectangle denotes a workflow graph produced by a single execution of the workflow. A learning template specifies how one or more workflow graphs are combined into the learning graph shown to the optimizer. In batch learning, independent workflow graphs are aggregated using . In episodic learning, workflow graphs are linked through temporal/environment transitions denoted by and the result is presented to the optimizer.
Figure E1 : Examples of LLM-based generative optimization applied to different kinds of systems: a prompted system with a search tool ( left ), a more complex prompting setup with retrieval code ( middle ), and an end-to-end machine learning pipeline ( right ). We use to mark an LLM API call and to mark text or code files.
Figure E2 : Infinite nested agentic system optimization . It is easy to construct an example in which one optimizer writes the instructions for another optimizer, creating an infinite regress of optimization problems. We do not study that setting in this paper and instead focus on commonly used, finite agentic systems.
More in this topic
Large Language Model-Guided Optimization
Large Language Model-Guided Optimization uses language models to propose, steer, and refine solutions to hard search and design problems.
See all 605 papers