Understanding the Challenges in Iterative Generative Optimization with LLMs

Allen Nie; Xavier Daull; Zhiyi Kuang; Abhinav Akkiraju; Anish Chaudhuri; Max Piasevoli; Ryan Rong; YuCheng Yuan

cs.LGMar 25, 2026

Understanding the Challenges in Iterative Generative Optimization with LLMs

Authors: Allen Nie, Xavier Daull, Zhiyi Kuang, Abhinav Akkiraju, Anish Chaudhuri, Max Piasevoli, Ryan Rong, YuCheng Yuan,

Organizations: Google DeepMind · CNRS · Stanford University · Carnegie Mellon University ·

Abstract

Generative optimization uses large language models (LLMs) to iteratively improve artifacts (such as code, workflows or prompts) using execution feedback. It is a promising approach to building self-improving agents, yet in practice remains brittle: despite active research, only 9% of surveyed agents used any automated optimization. We argue that this brittleness arises because, to set up a learning loop, an engineer must make ``hidden'' design choices: What can the optimizer edit and what is the "right" learning evidence to provide at each update? We investigate three factors that affect most applications: the starting artifact, the credit horizon for execution traces, and batching trials and errors into learning evidence. Through case studies in MLAgentBench, Atari, and BigBench Extra Hard, we find that these design decisions can determine whether generative optimization succeeds, yet they are rarely made explicit in prior work. Different starting artifacts determine which solutions are reachable in MLAgentBench, truncated traces can still improve Atari agents, and larger minibatches do not monotonically improve generalization on BBEH. We conclude that the lack of a simple, universal way to set up learning loops across domains is a major hurdle for productionization and adoption. We provide practical guidance for making these choices.

Figures & tables

Figure 1: The learning loop of generative optimization.

Figure 2: Three Key Decisions for Implementing a Learning Loop . To set up iterative generative optimization, an agent engineer must make three core decisions. For the initial system: What (1) starting artifacts ( instructions, files, specs) to provide? Different initializations can lead to different solution spaces. For the learning context ( ): What learning evidence to provide to the LLM optimizer – (2) how many steps to include per trial ( credit horizon ) and (3) how many trials to batch together ( experience batching ).

Figure 3: Starting artifacts. The one-function and many-function initializations provide equivalent docstring information but differ in pipeline modularization.

Panel of Figure 4 — Figure 4: Left: Kaggle leaderboard performance, reported as the percentile of the trained ML model submissions (higher is better). Right: MLAgentBench results. We run both systems 5 times and report the average and best Kaggle submission score across runs. One and Multi refer to a one-function and many-functions workflow design for the learned ML pipeline (Figure 3 ). Both systems use the same LLM API.

Figure 5: Credit Horizon Comparison Across Games . Performance of code agents optimized with one-step (immediate reward) vs multi-step (full rollout) credit horizons across 5 trials. Both setups use agents with the same starting artifacts. We see that observing full execution traces (multi-step) is only useful in discovering better code in 4 out of 8 games, suggesting credit horizon is a design choice and can be tuned to each task.

Figure 6: Training Efficiency Comparison . Deep RL training builds on CleanRL ( Huang et al., 2022 ) ; PPO uses ten parallel environments for sampling while DQN and LLM use a single environment. Both PPO and DQN use batch training on a single A100 GPU. All algorithms use object-centric inputs from OCAtari. The deep RL algorithms convert this information into numeric vectors, whereas the LLM takes the raw dictionary without additional annotation. Scores are normalized to map 0% to random performance on the task and 100% to human performance. See Appendix B.5 for details.

Figure 7: Validation Learning Curves Across Batch Sizes . Performance across optimization iterations for different batch sizes (3 trials, shaded area shows standard error). Larger batches often show faster initial learning but can plateau earlier.

Batch	Dyck	Shapes	Linguini	Causal	Disambig	Bool Expr	Movie	Boardgame
1	0.183 $\pm$ 0.049	0.343 $\pm$ 0.039	0.149 $\pm$ 0.024	0.375 $\pm$ 0.146	0.537 $\pm$ 0.036	0.177 $\pm$ 0.005	0.889 $\pm$ 0.038	0.341 $\pm$ 0.032
3	0.063 $\pm$ 0.010	0.389 $\pm$ 0.040	0.234 $\pm$ 0.012	0.408 $\pm$ 0.097	0.295 $\pm$ 0.091	0.238 $\pm$ 0.006	0.683 $\pm$ 0.119	0.278 $\pm$ 0.009
5	0.190 $\pm$ 0.031	0.200 $\pm$ 0.099	0.170 $\pm$ 0.030	0.531 $\pm$ 0.018	0.526 $\pm$ 0.035	0.154 $\pm$ 0.034	0.810 $\pm$ 0.016	0.276 $\pm$ 0.007

Table 1: Test Set Performance Across Batch Sizes . Best accuracy per task is bolded; standard error shown in gray. Different tasks have different optimal batch sizes, and larger batches do not always improve performance on the full test set (175+ holdout test examples). Base model is Claude Sonnet-3.5-v2.

Framework	Batch	Boardgame	Causal	Movie
LangGraph	1	0.208 $\pm$ 0.152	0.425 $\pm$ 0.082	0.680 $\pm$ 0.031
	3	0.278 $\pm$ 0.012	0.430 $\pm$ 0.109	0.703 $\pm$ 0.035
	5	0.280 $\pm$ 0.020	0.493 $\pm$ 0.139	0.737 $\pm$ 0.028
DSPy	1	0.288 $\pm$ 0.035	0.472 $\pm$ 0.027	0.448 $\pm$ 0.351
	3	0.377 $\pm$ 0.025	0.341 $\pm$ 0.031	0.189 $\pm$ 0.251
	5	0.299 $\pm$ 0.035	0.278 $\pm$ 0.015	0.495 $\pm$ 0.173

Table 2: Cross-framework check. Test accuracy across minibatch sizes on the three shared tasks. Standard error obtained with 3 trials.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1 : Trace program for the two MLAgentBench initializations. The one-function design exposes the full pipeline through a single trainable function, whereas the many-function design exposes preprocessing, feature selection, model training, and prediction as separate trainable components.

Panel of Figure 11 — Figure A2 : Feedback templates used for the learned ML pipeline on the Spaceship Titanic and Housing Price tasks.

Table A1 : Staged feedback templates for Spaceship Titanic and Housing Price. Highlighted phrases mark the parts that differ across feedback levels within each task.

Figure A3 : One Optimization Trajectory of LLM Optimizer on the ML Pipeline. We show a single optimization trajectory from the Spaceship Titanic dataset. The feedback string reports Accuracy, F1 score, Precision, and Recall, but both staged feedback and checkpoint selection are based on validation F1. Each dot (step) represents a full execution of the training pipeline code written by the LLM optimizer with a fully trained model. We see that LLM optimizer tends to overfit to the training data and exhibits classical overfitting behavior even without numerical gradient descent.

Name	Repo
Deep RL Repo (CleanRL)	https://github.com/ameliakuang/cleanrl_obj_centric
LLM Code Repo	https://github.com/ameliakuang/LLM-Game-Playing-Agents
Wandb Log for DQN Training	https://wandb.ai/kuangzy-amelia-stanford-university/obj-dqn-5trials-new
Wandb Log for PPO Training	https://wandb.ai/kuangzy-amelia-stanford-university/obj-ppo-5trials-flatten

Appendix

Table B1: Repositories and experiment logs used for the Atari experiments.

Game	One-Step	Multi-Step	Optimizer Steps	Evaluation Protocol
Pong	1 step	400 steps	20	10 episodes $\times$ 4000 steps
Breakout	1 step	300 steps	30	1 episode $\times$ 4000 steps
Space Invaders	1 step	25 steps	20	1 episode $\times$ 4000 steps
Freeway	1 step	100 steps	30	3 episodes $\times$ 2500 steps
Asterix	1 step	100 steps	30	3 episodes $\times$ 20000 steps
Enduro	1 step	100 steps	30	1 episode $\times$ 5000 steps

Appendix

Table B2: Per-game Atari credit-horizon configurations. All games use ALE {env}-NoFrameskip-v4 environments with action repeat 4, sticky action probability 0.0, optimizer memory size 5, and Claude Sonnet-3.5 via OptoPrime.

Panel of Figure 16 — Figure B1 : OCAtari object annotations across eight Atari games. For each game, the left image shows the raw RGB frame and the right image shows the corresponding OCAtari ( Delfosse et al., 2024 ) annotated frame. These examples illustrate how pixel observations are translated into object-centric state representations across diverse Atari environments.

Panel of Figure 17 — Figure B2 : Example of a returned observation, game information, and reward from Breakout. Note that this is the raw return from OCAtari, not further processed. There is no annotation of what “RB”, “OB”, “YB”, “GB”, “AB”, and “BB” represent. They are acronyms for the bounding boxes with colors.

Figure B3 : Representative initial systems for Atari games . We show the workflow design of different decision-making program components for three representative Atari game agents. The learned agent receives an object-centric dictionary of game state information and uses Python code to process the observation and output an action.

Table B3 : Staged feedback templates for Pong and Breakout. Highlighted phrases mark the parts that differ between the corresponding High, Medium, and Low feedback levels across the two games.

Performance Level	Feedback
High (Reward $\geq$ 300)	“Great job! You’re performing well with an average score of 320 . Try to improve your shooting accuracy and dodging .”
Medium (100 $\leq$ Reward $<$ 300)	“Good progress! Your average score is 180 . Focus on better timing for shooting and avoiding enemy projectiles .”
Low (Reward $<$ 100)	“Your average score is 70 . Try to improve your strategy for shooting aliens and dodging projectiles .”

Appendix

Table B4: Staged feedback for the Space Invaders agent at different performance levels.

Table B5 : Left: concrete settings for the object-centric deep RL baselines. Right: median runtime and score statistics across games.

Figure B4 : Representative high-level strategies discovered by the optimizer. These short excerpts show that the learned revisions do more than tune local action thresholds: Pong adds distance- and momentum-aware interception, Breakout begins to target tunnels and higher-value bricks, and Space Invaders combines action constraints with threat-aware positioning.

Parameter	Value
Training examples per task	First 15 examples
Validation examples per task	Next 10 examples
Held-out test examples per task	Remaining examples (typically 175+)
Batch sizes compared	1, 3, 5
Total optimizer update steps	15
Number of trials (seeds)	3

Appendix

Table C1 : BigBench Extra Hard experimental configurations.

Figure C1 : Agent design for BigBench Extra Hard . The agent consists of two optimizable components: call_llm and answer_extraction . During optimization, multiple traces can be concatenated through the batchify operator $\oplus$ before being shown to the optimizer.

Panel of Figure 25 — Figure D1 : An example workflow graph $g_{i}$ produced by a single execution of $W_{\theta}$ . The graph contains input node $x_{i}$ , intermediate node $o_{i}$ , output node $y_{i}$ , feedback $f_{i}$ , and the parameter nodes $\theta_{1}$ , $\theta_{2}$ that influence the output.

Figure D2 : Representing different learning problems through graph templates. Each dotted rectangle denotes a workflow graph produced by a single execution of the workflow. A learning template specifies how one or more workflow graphs are combined into the learning graph $G_{\mathrm{learn}}$ shown to the optimizer. In batch learning, independent workflow graphs are aggregated using $\oplus$ . In episodic learning, workflow graphs are linked through temporal/environment transitions denoted by $\Rightarrow$ and the result is presented to the optimizer.

Figure E1 : Examples of LLM-based generative optimization applied to different kinds of systems: a prompted system with a search tool ( left ), a more complex prompting setup with retrieval code ( middle ), and an end-to-end machine learning pipeline ( right ). We use to mark an LLM API call and to mark text or code files.

Figure E2 : Infinite nested agentic system optimization . It is easy to construct an example in which one optimizer writes the instructions for another optimizer, creating an infinite regress of optimization problems. We do not study that setting in this paper and instead focus on commonly used, finite agentic systems.

Large Language Model-Guided Optimization

Large Language Model-Guided Optimization uses language models to propose, steer, and refine solutions to hard search and design problems.

See all 605 papers

Apr 14, 2026

Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization

Current LLM agent benchmarks, which predominantly focus on binary pass/fail tasks such as code generation or search-based question answering, often neglect the value of real-world engineering that is often captured through the iterative optimization of feasible designs. To this end, we introduce Frontier-Eng, a human-verified benchmark for generative optimization -- an iterative propose-execute-evaluate loop in which an agent generates candidate artifacts, receives executable verifier feedback, and revises them under a fixed interaction budget -- spanning

47

tasks across five broad engineering categories. Unlike previous suites, Frontier-Eng tasks are grounded in industrial-grade simulators and verifiers that provide continuous reward signals and enforce hard feasibility constraints under constrained budgets. We evaluate eight frontier language models using representative search frameworks, finding that while GPT 5.4 achieves the most robust performance, the benchmark remains challenging for all models. Our analysis suggests a dual power-law decay in improvement frequency (

\sim

1/iteration) and magnitude (

\sim

1/improvement count). We further show that although width improves parallelism and diversity, depth remains crucial for hard-won improvements under a fixed budget. Frontier-Eng establishes a new standard for assessing the capacity of AI agents to integrate domain knowledge with executable feedback to solve complex, open-ended engineering problems.

Yizhe Chi, Deyao Hong, Dapeng Jiang +18

Navers Lab, Einsia.AI

AI Agent Evaluation AI Agent Benchmarks Large Language Model-Guided Optimization

Ask PDF

Dec 9, 2025

Evolving Excellence: Automated Optimization of LLM-based Agents

Agentic AI systems built on large language models (LLMs) offer significant potential for automating complex workflows, from software development to customer support. However, LLM agents often underperform due to suboptimal configurations; poorly tuned prompts, tool descriptions, and parameters that typically require weeks of manual refinement. Existing optimization methods either are too complex for general use or treat components in isolation, missing critical interdependencies. We present ARTEMIS, a no-code evolutionary optimization platform that jointly optimizes agent configurations through semantically-aware genetic operators. Given only a benchmark script and natural language goals, ARTEMIS automatically discovers configurable components, extracts performance signals from execution logs, and evolves configurations without requiring architectural modifications. We evaluate ARTEMIS on four representative agent systems: the ALE Agent for competitive programming on AtCoder Heuristic Contest, achieving a \textbf{

13.6\%

improvement} in acceptance rate; the Mini-SWE Agent for code optimization on SWE-Perf, with a statistically significant 10.1% performance gain; and the CrewAI Agent for cost and mathematical reasoning on Math Odyssey, achieving a statistically significant \textbf{

36.9\%

reduction} in the number of tokens required for evaluation. We also evaluate the MathTales-Teacher Agent powered by a smaller open-source model (Qwen2.5-7B) on GSM8K primary-level mathematics problems, achieving a 22% accuracy improvement and demonstrating that ARTEMIS can optimize agents based on both commercial and local models.

Paul Brookes, Vardan Voskanyan, Rafail Giavrimis +18

TurinTech AI, London, UK · University of Surrey, Guildford, United Kingdom · University of Leeds, Leeds, UK +4

Large Language Model-Guided Optimization LLM Agent Workflow Optimization LLM Agents Genetic Algorithms

Ask PDF

Mar 16, 2026

POLCA: Stochastic Generative Optimization with LLM

Optimizing complex systems, ranging from LLM prompts to multi-turn agents, traditionally requires labor-intensive manual iteration. We formalize this challenge as a stochastic generative optimization problem where a generative language model acts as the optimizer, guided by numerical rewards and text feedback to discover the best system. We introduce Prioritized Optimization with Local Contextual Aggregation (POLCA), a scalable framework designed to handle stochasticity in optimization -- such as noisy feedback, sampling minibatches, and stochastic system behaviors -- while effectively managing the unconstrained expansion of solution space. POLCA maintains a priority queue to manage the exploration-exploitation tradeoff, systematically tracking candidate solutions and their evaluation histories. To enhance efficiency, we integrate an

\varepsilon

-Net mechanism to maintain parameter diversity and an LLM Summarizer to perform meta-learning across historical trials. We theoretically prove that POLCA converges to near-optimal candidate solutions under stochasticity. We evaluate our framework on diverse benchmarks, including

τ

-bench, HotpotQA (agent optimization), VeriBench (code translation) and KernelBench (CUDA kernel generation). Experimental results demonstrate that POLCA achieves robust, sample and time-efficient performance, consistently outperforming state-of-the-art algorithms in both deterministic and stochastic problems. The codebase for this work is publicly available at https://github.com/rlx-lab/POLCA.

Xuanfei Ren, Allen Nie, Tengyang Xie +1

University of Wisconsin-Madison, Madison, WI · Google DeepMind, Mountain View, CA · Google Research, Kirkland, WA

Large Language Model-Guided Optimization Stochastic Optimization LLM Agent Workflow Optimization

Ask PDF

Abstract

Large Language Model-Guided Optimization

Explore similar work

Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization

Evolving Excellence: Automated Optimization of LLM-based Agents

POLCA: Stochastic Generative Optimization with LLM