Greedy Is a Strong Default: Agents as Iterative Optimizers

Yitao Li

cs.AIMar 28, 2026

Greedy Is a Strong Default: Agents as Iterative Optimizers

Authors: Yitao Li

Abstract

Classical optimization algorithms--hill climbing, simulated annealing, population-based methods--generate candidate solutions via random perturbations. We replace the random proposal generator with an LLM agent that reasons about evaluation diagnostics to propose informed candidates, and ask: does the classical optimization machinery still help when the proposer is no longer random? We evaluate on four tasks spanning discrete, mixed, and continuous search spaces (all replicated across 3 independent runs): rule-based classification on Breast Cancer (test accuracy 86.0% to 96.5%), mixed hyperparameter optimization for MobileNetV3-Small on STL-10 (84.5% to 85.8%, zero catastrophic failures vs. 60% for random search), LoRA fine-tuning of Qwen2.5-0.5B on SST-2 (89.5% to 92.7%, matching Optuna TPE with 2x efficiency), and XGBoost on Adult Census (AUC 0.9297 to 0.9317, tying CMA-ES with 3x fewer evaluations). Empirically, on these tasks: a cross-task ablation shows that simulated annealing, parallel investigators, and even a second LLM model (OpenAI Codex) provide no benefit over greedy hill climbing while requiring 2-3x more evaluations. In our setting, the LLM's learned prior appears strong enough that acceptance-rule sophistication has limited impact--round 1 alone delivers the majority of improvement, and variants converge to similar configurations across strategies. The practical implication is surprising simplicity: greedy hill climbing with early stopping is a strong default. Beyond accuracy, the framework produces human-interpretable artifacts--the discovered cancer classification rules independently recapitulate established cytopathology principles.

Figures & tables

Task	Artifact Type	Method	Metric	Seed	Best
Cancer rules	Discrete (YAML)	Greedy ( $n\!=\!3$ )	Test accuracy	0.860	0.965
STL-10 pipeline	Mixed (16 params)	Greedy ( $n\!=\!3$ )	Test accuracy	0.845	0.858
LoRA SST-2	Mixed (8 params)	Greedy ( $n\!=\!3$ )	Test accuracy	0.895	0.927
XGBoost Adult	Continuous (12 params)	Full ablation	Test AUC	0.9297	0.9317

Table 1: Summary of results across all four experiments (best of 3 independent runs per task).

Method	Runs	Mean Test	Std	Best Test	Mean Rounds
Seed	—	0.860	—	—	0
Greedy	3	0.959	0.005	0.965	3.3
SA ( $T_{0}\!=\!0.054$ )	3	0.941	0.018	0.956	6.7
sklearn DT (reference)	—	0.960	—	—	—
sklearn RF (reference)	—	0.960	—	—	—

Table 2: Breast Cancer rule optimization: 6 independent runs (3 greedy, 3 SA). Greedy achieves higher accuracy with lower variance and fewer rounds. SA’s stochastic acceptance is counterproductive for this task.

Figure 1: Agentic rule-based classifier for breast cancer diagnosis. Rules are evaluated sequentially (top to bottom); the first matching rule determines the classification. The system discovered 5 rules using 4 features from FNA biopsy measurements, achieving 95.6% test accuracy with 341 training samples—comparable to a pruned decision tree (96%) trained on 455 samples. Italic annotations are clinical interpretations based on the FNA cytopathology literature; they were not provided to the agent—the rules were discovered from data alone.

Method	Test Acc.	Trials	Failure Rate
Default (seed)	0.845	1	0%
SMAC	0.840	30	33%
Random Search	0.855	30	60%
Agentic (mean, $n\!=\!3$ )	0.852	${\sim}$ 8	0%
Agentic (best)	0.858	${\sim}$ 4	0%

Table 3: STL-10: comparison against standard HPO baselines (30 trials each).

Run	Rounds	Accepted	Val Acc.	Test Acc.
Run 1	5	3	0.862	0.858
Run 2	3	1	0.853	0.841
Run 3	3	1	0.862	0.858
Mean	—	—	0.859	0.852

Table 4: STL-10: three independent greedy runs. Runs 1 and 3 converged to identical 16-parameter configurations; Run 2 diverged to a different optimizer family. The best agentic result (0.858) beats random search (0.855) with zero catastrophic failures.

Method	Val Accuracy	Test Accuracy	Trials
Seed config	0.866	0.895	1
Random Search	0.905	0.921	20
Optuna TPE	0.911	0.922	20
Agentic (mean, $n\!=\!3$ )	0.910	0.924	${\sim}$ 9
Agentic (best)	0.910	0.927	${\sim}$ 12

Table 5: LoRA SST-2: comparison against standard HPO baselines (20 trials each for Random and TPE).

Run	Accepted	Best Val	Test	Training Runs
Run 1	2	0.910	0.927	${\sim}$ 12
Run 2	2	0.911	0.922	${\sim}$ 8
Run 3	2	0.909	0.923	${\sim}$ 8
Mean	2.0	0.910	0.924	${\sim}$ 9

Table 6: LoRA SST-2: three independent greedy runs. All runs match or exceed Optuna TPE (0.922 test) with

{\sim}

half the compute. The two-phase optimization pattern is highly reproducible.

Parameter	Seed	Phase 1	Phase 2 (Final)	Optuna TPE
LoRA rank	8	16	8	8
LoRA alpha	16	32	64	64
Alpha/rank	2	2	8	8
Learning rate	2 $\times 10^{-5}$	$10^{-4}$	2–3 $\times 10^{-4}$	3.3 $\times 10^{-4}$
Dropout	0.05	0.05	0.20–0.25	0.28
Weight decay	0.01	0.01	0.10–0.15	0.18

Table 7: LoRA SST-2: hyperparameter evolution showing the two-phase optimization pattern. All 3 runs independently converged to the same alpha/rank=8 regime that Optuna TPE also discovered.

Method	Test AUC-ROC	Evals
Default	0.9292	1
Random Search	0.9299	50
Optuna TPE	0.9301	50
OPRO-lite ( $n\!=\!3$ )	0.9303	11
Grid Search	0.9307	50
Optuna CMA-ES	0.9317	50

Table 8: XGBoost Adult: standard HPO baselines plus OPRO-lite (ablation comparator for agentic runs).

Single-model (Opus 4.6)
Variant	Mean Test	Best Test	CV Evals	vs. CMA-ES
Greedy ( $n\!=\!3$ )	0.9314	0.9317	${\sim}$ 13	$-0.0003$
SA ( $n\!=\!3$ )	0.9313	0.9316	${\sim}$ 28	$-0.0004$
Parallel ( $K\!=\!3$ , $n\!=\!3$ )	0.9313	0.9314	${\sim}$ 33	$-0.0004$
Multi-model (Opus 4.6 + Codex)
Codex Advisory ( $n\!=\!3$ )	0.9312	0.9313	${\sim}$ 15	$-0.0005$

Table 9: XGBoost Adult: full ablation across acceptance rules, parallelism, and multi-model variants (3 runs each). Greedy (single model) achieves the best mean test AUC with the fewest evaluations.

Figure 2: Best validation metric vs. optimization round across all four tasks (3 independent runs each). Round 1 delivers the majority of improvement in every case. Subsequent rounds add diminishing gains before early stopping. Cancer panel shows greedy (blue) vs. SA (orange); SA Run 2 achieved high validation but lower test accuracy (0.956 vs. greedy best 0.965), illustrating that validation-level advantage does not transfer to test.

Task	Greedy Test	SA Test	SA Helps?
Cancer (mean, $n\!=\!3$ )	0.959	0.941	No ( $-$ 0.018)
XGBoost (mean, $n\!=\!3$ )	0.9314	0.9313	No ( $-$ 0.0001)

Table 10: Cross-task comparison of greedy vs. SA on the two tasks where both were tested. Greedy matches or beats SA with 2–3

\times

fewer evaluations.

Figure 3: Per-trial test accuracy on STL-10 for random search, SMAC, and agentic greedy. Random search and SMAC produce catastrophic failures (test accuracy

<0.15

) in 60% and 33% of trials. The agentic method produces zero failures across all evaluations, with tight clustering near the optimum.

Task	Agentic Evals	Baseline Evals	Agentic Result	Best Baseline
Cancer	${\sim}$ 3	N/A	0.965	0.960 (DT/RF)
STL-10	${\sim}$ 4–12	30	0.858	0.855 (random)
LoRA SST-2	${\sim}$ 9	20	0.927	0.922 (TPE)
XGBoost (greedy)	${\sim}$ 13	50	0.9317	0.9317 (CMA-ES)
XGBoost (OPRO-lite ablation)	11	—	0.9303	0.9317 (agentic greedy)

Table 11: Evaluation efficiency: agentic greedy achieves competitive or superior results with fewer evaluations. OPRO-lite is shown as an ablation comparator for XGBoost and falls short of the agentic method.

Large Language Model-Guided Optimization

Large Language Model-Guided Optimization uses language models to propose, steer, and refine solutions to hard search and design problems.

See all 608 papers

Mar 25, 2026

Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearch

The autoresearch repository enables an LLM agent to optimize hyperparameters by editing training code directly. We use it as a testbed to compare classical HPO algorithms against LLM-based methods on tuning the hyperparameters of a small language model under a fixed compute budget. When defining a fixed search space over autoresearch, classical methods such as CMA-ES and TPE consistently outperform LLM-based agents, where avoiding out-of-memory failures matters more than search diversity. Allowing the LLM to directly edit source code narrows the gap to the classical methods but does not close it, even with frontier models available at the time of writing such as Claude Opus 4.6 and Gemini 3.1 Pro Preview. We observe that LLMs struggle to track optimization state across trials. In contrast, classical methods lack the domain knowledge of LLMs. To combine the strengths of both, we introduce Centaur, a hybrid that shares CMA-ES's interpretable internal state, including mean vector, step-size, and covariance matrix, with an LLM. Centaur achieves the best result in our experiments, and a 0.8B LLM already suffices to outperform all classical and pure LLM methods. Unconstrained code editing requires larger models to be competitive with classical methods. We further analyze search diversity, model scaling from 0.8B to frontier models, and ablate the fraction of LLM-proposed trials in Centaur. All in all, our results suggest that LLMs are most effective as a complement to classical optimizers, not as a replacement. Code is available at https://github.com/ferreirafabio/autoresearch-automl & interactive demo at https://ferreirafabio.github.io/autoresearch-automl.

Fabio Ferreira, Lucca Wobbe, Arjun Krishnakumar +2

ELLIS Institute Tübingen · University of Freiburg · Karlsruhe Institute of Technology +1

Hyperparameter Optimization Large Language Model-Guided Optimization Covariance Matrix Adaptation Evolution Strategy

Ask PDF

Mar 2, 2026

Reasoning as Gradient: Scaling MLE Agents Beyond Tree Search

LLM-based agents for machine learning engineering (MLE) predominantly rely on tree search, a form of gradient-free optimization that uses scalar validation scores to rank candidates. As LLM reasoning capabilities improve, exhaustive enumeration becomes increasingly inefficient compared to directed updates, analogous to how accurate gradients enable efficient descent over random search. We introduce Gome, an MLE agent that operationalizes gradient-based optimization. Gome maps structured diagnostic reasoning to gradient computation, success memory to momentum, and multi-trace execution to distributed optimization. Under a closed-world protocol that isolates architectural effects from external knowledge, Gome achieves a state-of-the-art 35.1% any-medal rate on MLE-Bench with a restricted 12-hour budget on a single V100 GPU. Scaling experiments across 10 models reveal a critical crossover: with weaker models, tree search retains advantages by compensating for unreliable reasoning through exhaustive exploration; as reasoning capability strengthens, gradient-based optimization progressively outperforms, with the gap widening at frontier-tier models. Given the rapid advancement of reasoning-oriented LLMs, this positions gradient-based optimization as an increasingly favorable paradigm. We release our codebase and GPT-5 traces at https://github.com/microsoft/RD-Agent.

Yifei Zhang, Xu Yang, Xiao Yang +8

Nanjing University · Microsoft Research Asia · Peking University +3

Large Language Model-Guided Optimization

Ask PDF

Sep 30, 2026

Initialization Improves LLM-Driven Discovery

Large Language Models (LLMs) have been used for novel discovery of algorithms, theorems, drugs, and other tasks through the use of harnesses that prompt an LLM to iteratively optimize an objective. In this work, we study the relationship between the population of previous iterates and eventual discovery success. We generalize past work on harness design to develop a suite of 12 harnesses called 'Modular' and characterize their performance across 5 diverse discovery tasks, finding that discovery success is brittle and sensitive to harness design. We uncover mode collapse, characterized by a dramatic drop in the diversity of iterates, as a common failure mode. We find that popular state-of-the-art harnesses and diversity-inducing harness interventions, which aim to prolong this collapse, yield inconsistent gains. Our results instead uncover that the performance of early discoveries is predictive of eventual success. We therefore propose a universally applicable intervention that performs an initial stage of parallel exploration in order to initialize subsequent iterative optimization. Our method provides consistent gains across many harnesses and target applications, confirming the importance of initialization in LLM-driven discovery.

Mansi Sakarvadia, Marco Ciccone, Colin Raffel

University of Chicago · Vector Institute · University of Toronto

Large Language Model-Guided Optimization Large Language Model-Guided Scientific Discovery

Ask PDF

Abstract

Large Language Model-Guided Optimization

Explore similar work

Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearch

Reasoning as Gradient: Scaling MLE Agents Beyond Tree Search

Initialization Improves LLM-Driven Discovery