The autoresearch repository enables an LLM agent to optimize hyperparameters by editing training code directly. We use it as a testbed to compare classical HPO algorithms against LLM-based methods on tuning the hyperparameters of a small language model under a fixed compute budget. When defining a fixed search space over autoresearch, classical methods such as CMA-ES and TPE consistently outperform LLM-based agents, where avoiding out-of-memory failures matters more than search diversity. Allowing the LLM to directly edit source code narrows the gap to the classical methods but does not close it, even with frontier models available at the time of writing such as Claude Opus 4.6 and Gemini 3.1 Pro Preview. We observe that LLMs struggle to track optimization state across trials. In contrast, classical methods lack the domain knowledge of LLMs. To combine the strengths of both, we introduce Centaur, a hybrid that shares CMA-ES's interpretable internal state, including mean vector, step-size, and covariance matrix, with an LLM. Centaur achieves the best result in our experiments, and a 0.8B LLM already suffices to outperform all classical and pure LLM methods. Unconstrained code editing requires larger models to be competitive with classical methods. We further analyze search diversity, model scaling from 0.8B to frontier models, and ablate the fraction of LLM-proposed trials in Centaur. All in all, our results suggest that LLMs are most effective as a complement to classical optimizers, not as a replacement. Code is available at https://github.com/ferreirafabio/autoresearch-automl & interactive demo at https://ferreirafabio.github.io/autoresearch-automl.
Figures & tables
Figure 1: Best val_bpb (mean ± std, 3 seeds) against cumulative training time. All methods receive the same 24-hour GPU budget; LLM inference overhead is excluded. All LLM-based methods use Qwen3.5-27B. Linestyle and markers encode method category: dashed with circles = classical, solid with stars = pure LLM, dash-dot with diamonds = hybrid (Centaur). Marker positions are offset per line for visual clarity. Bands shown for top 5 methods (see Figure 14 for all).
HP
Type
Range
Log
Default
DEPTH
int
4–24
8
ASPECT_RATIO
int
32–128
64
HEAD_DIM
int
64–256
✓
128
DEVICE_BATCH_SIZE
int
32–256
✓
128
TOTAL_BATCH_SIZE
int
65 536–2 097 152
✓
524 288
EMBEDDING_LR
float
0.01–2.0
✓
0.6
Table 1: Search space: 14 HPs auto-extracted via AST parsing. Ranges are set manually. Defaults are Karpathy’s starting config. WINDOW_PATTERN controls the per-layer attention window: S = short (local) attention, L = long (full) attention.
Method
Search Space
Description
TPE
Fixed
Tree-structured Parzen Estimator via Optuna ( Bergstra et al., ; Akiba et al., )
OptunaHub ( Ozaki et al., 2025 ) port: binary surrogate labels, random categorical sampling
LLAMBO (Paper)
Fixed
Reimplementation faithful to Liu et al. (2024b) : continuous labels, all HPs visible
Table 2: Methods evaluated. “Fixed” indicates the shared 14-HP search space; “Unconstrained” indicates direct source code editing.
Method
Trials
Best val_bpb
OOM%
Spread
Step
Centaur [Qwen 0.8B]
333 ± 4
0.9766 ± 0.0008
13%
0.131
0.368
Centaur [Qwen 27B]
334 ± 5
0.9763 ± 0.0005
15%
0.115
0.341
Centaur [Gemini 3.1 Pro]
344 ± 10
0.9767 ± 0.0013
20%
0.135
0.496
Centaur [Opus 4.6]
340 ± 13
0.9739 ± 0.0012
17%
0.138
0.499
CMA-ES
336 ± 13
0.9785 ± 0.0036
16%
0.158
0.580
KA (14 HPs) [Qwen 27B]
289 ± 2
0.9904 ± 0.0002
1%
0.035
0.057
Table 3: Search diversity analysis (3 seeds, alphabetical). “KA” abbreviates “Karpathy Agent”; LLM-based methods are labeled with their LLM optimizer in brackets as in Figures 1 and 2 . Best within each category is bold and colored: hybrid , classical , pure LLM . Trials : mean ± std total per seed. Best : mean ± std best val_bpb. Spread : mean per-HP std. Step : mean L2 between consecutive trials. Pairwise distance and unique-cell coverage metrics are defined in Section A.7 and reported for 0.8B variants in Table 5 .
Figure 2: Scaling the LLM optimizer: frontier models vs Qwen3.5-27B for Karpathy Agent (Code) and Centaur (mean ± std across 3 seeds). Linestyle encodes method category as in Figure 1 . Centaur [Opus 4.6] achieves the best result (0.9739), extending the hybrid method’s lead.
Figure 3: 0.8B vs 27B LLM optimizer comparison (wall-time). Solid: 27B, dashed: 0.8B. TPE and Random shown as classical references. The 0.8B model appears insufficient for unconstrained code editing but sufficient for hybrid optimization.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Convergence by trial number (mean ± std across 3 seeds). Same methods as Figure 1 . Trial-number view shows sample efficiency rather than wall-clock cost.
Figure 5: 0.8B vs 27B by trial number (mean ± std across 3 seeds). Same methods as Figure 3 .
Figure 6: Per-seed convergence for all 27B methods (wall-time). Thin lines: individual seeds. Thick lines: mean across 3 seeds.
Figure 7: Per-seed convergence for 0.8B vs 27B comparison (wall-time).
Figure 8: Per-seed convergence for Gemini 3.1 Pro Preview vs Qwen3.5-27B (wall-time).
Figure 9: Gemini 2.5 Flash vs Qwen3.5-27B as LLM optimizer (cumulative training time). Solid: Gemini, dashed: Qwen3.5-27B. Same color per method.
Figure 10: Gemini 3.1 Flash-Lite vs Qwen3.5-27B as LLM optimizer (cumulative training time). Solid: Gemini 3.1, dashed: Qwen3.5-27B. Same color per method.
LLM Ratio r
Centaur [0.8B]
Centaur [27B]
0.1
0.9744 ± 0.0003
0.9758 ± 0.0005
0.2
0.9748 ± 0.0013
0.9748 ± 0.0005
0.3 (default)
0.9766 ± 0.0008
0.9763 ± 0.0005
0.5
0.9768 ± 0.0019
0.9746 ± 0.0000
0.8
0.9849 ± 0.0060
0.9902 ± 0.0000
Appendix
Table 4: Centaur LLM ratio ablation. Best val_bpb (mean ± std). 2 seeds per ratio, 3 seeds for default r=0.3 . Bold = best per column.
Figure 11: Effect of LLM ratio on Centaur performance (2 seeds, 3 for default). Higher ratios give the LLM more trials. CMA-ES alone (dotted) shown as reference. Too much LLM control degrades performance, especially for the 27B model at r=0.8 .
Figure 12: Incumbent traces for classical and hybrid methods (wall-time, best seed). Grey dots: all trials. Colored dots: new incumbents. Staircase: best-so-far.
Figure 13: Incumbent traces for LLM-based methods (wall-time, best seed).
Method
Trials
Best val_bpb
OOM%
Spread
Step
Karpathy Agent (14 HPs)
338 ± 3
0.9908 ± 0.0002
16%
0.037
0.062
LLAMBO (Optuna)
668 ± 29
0.9873 ± 0.0003
64%
0.297
1.311
LLAMBO (Paper)
605 ± 6
0.9894 ± 0.0007
59%
0.272
1.330
Appendix
Table 5: Diversity analysis for 0.8B LLM variants of fixed-search-space pure LLM methods (3 seeds each). Same columns as Table 3 .
Aspect
Original Paper
OptunaHub Port
Surrogate labels
Continuous metric values
Binary 0/1 (top 20% threshold)
Categorical HPs
All HPs in LLM prompts
Delegated to random sampling
Failed trials
Visible to surrogate
Hidden ( TrialState.FAIL )
Appendix
Table 6: Key differences between LLAMBO (Optuna) and LLAMBO (Paper).
Figure 14: All 9 methods with ± std bands, split into top 5 (Centaur, TPE, SMAC, CMA-ES, Karpathy Agent Code) and remaining 4 (Karpathy Agent 14 HPs, LLAMBO Paper, LLAMBO Optuna, Random).
Large language models (LLMs) have been proposed as hyperparameter-optimization (HPO) advisors that "warm-start" search from prior knowledge, proposing strong configurations in very few evaluations. We test that claim under a budget-matched, multi-seed protocol on eight PMLB tabular benchmarks, comparing an LLM advisor (LLM-OptFlow) against four classical baselines (random search, Optuna-TPE, Gaussian-process Bayesian optimization, and successive halving) over one shared search space, with paired tests and bootstrap 95% CIs across 8 x 5 = 40 (task, seed) units. The finding is cautionary. The advisor's strong first point is not an LLM output at all: like prior LLM-HPO systems the loop is seeded with a fixed default configuration, evaluated before any model call, which alone reaches 88.7% mean best-CV, identical to within 0.01 pp across all seven advisor models tested. The LLM's own proposals add only +0.40 pp of cross-validation accuracy over that seed and nothing on held-out test (LLM-Default = -0.01 pp, p = 0.92). When the same seed is granted to classical search, the apparent lead collapses: against seeded random search it leads by +0.20 pp at 2 evaluations, is tied by 5, and is behind by 12 (-0.37 pp). Without the seed, classical search ties the advisor by 12 evaluations and beats it by 40 (+0.6 to +0.8 pp, p <= 1e-4). Two LLM-specific behaviors survive: a single-task exploration failure (vehicle), and a rule-based confidence filter that removes ~33% of wasted compute without changing accuracy. The recommendation is deflationary: on tabular HPO, seed classical search with a sensible default; an LLM advisor adds no measurable generalization benefit and is overtaken within a handful of evaluations. We release the harness and a script that reproduces every statistic.
Carson Rodrigues, Oysturn Vas, Isaiah Abner DCosta +1
Celabe · University of Waterloo · University of Queensland (EAIT)
Large Language Model (LLM) systems have been the frontier of AI in many application domains, leading to new challenges and opportunities for hyperparameter optimization (HPO) for the AutoML community. However, this type of system exhibits an unprecedented compound space of hyperparameter configuration from both the AI and non-AI components; rich and nonlinear implications from the fidelity factors; and diverse costs of measuring hyperparameter configurations, none of which have been fully captured in existing benchmarks. This paper presents the first (live) benchmark suite and datasets for HPO of real-world LLM systems, dubbed LLMSYS-HPOBench, covering data related to the inference objective values of hyperparameter configurations profiled from running the LLM systems. Currently, LLMSYS-HPOBench contains 364,450 hyperparameter configurations with a dimensionality of 12-23, 3-5 dimensions of fidelity factor leading to 932 settings, 3-9 inference objective metrics, and 2-10 cost metrics, together with generated logs from measuring the LLM systems. What we seek to advocate is not only a revalidation of the existing HPO algorithms over the frontier LLM systems, but also to provide an evolving platform for the AutoML community to explore new directions of research in this regard. The benchmark suite has been made available at: https://github.com/ideas-labo/llmsys-hpobench
Siyu Wu, Yulong Ye, Zezhen Xiang +3
UESTC China · IDEAS Lab University of Birmingham, UK
In evolutionary algorithms powered by language models, the LLM acts as a single operator that simultaneously updates structural components (like control flow) and continuous parameters. While LLMs can be good at the first, they are not efficient at the second, wasting tokens taking discrete jumps inside a trial and error loop. We resolve this by formalizing a hybrid nested search, in which an outer loop has the LLM propose a structural sketch, with numeric gaps, and an inner numerical optimizer tunes the sketch. Both the outer and inner solvers are pluggable: any text-based optimizer can be combined with a zero-order optimizer (CMA-ES), gradient-based routines, or MCMC samplers. We validate our framework across three scientific domains: (i) meta-optimizers on closed-form test functions, (ii) code-based policies for systems research and social dilemmas; and (iii) approximate Bayesian inference tasks. Across all three, the hybrid optimizer is superior to both vanilla LLM-driven search and pure numerical optimization baselines. Code at: https://github.com/vicgalle/hybrid-nested-search