How Low Can You Go: Fine-Tuning LLMs with Bayesian Optimization (LlMBO)
Authors: Aditya Kulkarni, Davide Sartor, Carlos Soto, Nathan Wycoff
Organizations: Department of Physics, University of Massachusetts Amherst · Department of Mathematics and Statistics, University of Massachusetts Amherst
Recently, Morris et al. introduced TinyLoRA, a strategy for LLM fine-tuning which introduces arbitrarily few parameters, and demonstrated accuracy improvements using less than 10 parameters rather than the ≳10,000 typically demanded by LoRA-based methods. However, that study continued to use gradient-based local search methods of the kind deployed for training multi-billion parameter LLMs in the first place. This article is predicated on the observation that, in the low dimensional setting, global optimization methods become tractable, unlocking Large language Model Bayesian Optimization (LlMBO). LlMBO is sample-efficient, gradient-free, memory-light, parallelizes across devices without synchronization, and, unlike policy-gradient methods, can optimize greedy-decoding accuracy directly. We describe numerical experiments comparing LlMBO against GRPO on fine-tuning tasks for a variety of models on a math benchmark suite which show that LlMBO can be a suitable alternative in this regime, matching or exceeding GRPO's fine tuning gains, while using a fraction of the compute and memory.
Figures & tables
Figure 1: LlMBO against GRPO, evaluated both in terms of mean performance at temperature 1 on 4 trials (avg@4, open points) and via greedy decoding (filled). Gain over the base model (mean ± s.e. over seeds) for runs with u≤8 trainable parameters (marker) and model family (color).
Figure 2: TinyLoRA scaling laws under GRPO. Greedy (solid) and avg@4 (dashed) accuracy at the end of training under GRPO (mean ± s.e. over seeds) for TinyLoRA, LoRA-XS and LoRA as a function of trainable parameters. Results shown for Qwen2.5-7B, scaling for other model families can be found in Figure 15 .
Figure 3: Gain over the base model of LlMBO against that of GRPO on the in-distribution benchmarks, for the four vanilla base models; avg@4 (open) and greedy accuracy (filled), one point per u≤4 (marker), mean ± s.e. over seeds. LlMBO ( b=1 ) is trained on the metric of each point; points in the shaded half favor LlMBO. Every model and benchmark in Figure 5 .
in-distribution
out-of-distribution
GSM8K
MATH-500
AIME24
AMC23
Minerva
Olympiad
u
Model
avg@4
greedy
avg@4
greedy
avg@4
greedy
avg@4
greedy
avg@4
greedy
avg@4
greedy
1
Llama3.1-8B
+0.4
+1.1
−0.1
+0.1
−0.1
+0.0
+0.6
+2.5
−0.2
+0.5
+0.0
+0.2
Mistral-7B
+0.0
+0.7
+0.3
−0.1
−0.1
+0.0
−0.1
−0.8
+0.1
−0.1
+0.1
+0.0
DeepSeek-7B
+0.4
+1.2
+0.0
+0.6
+0.0
+0.0
+1.5
+0.0
+0.0
+0.2
+0.4
+0.1
Qwen2.5-7B
+1.5
+0.8
+1.0
+0.1
+0.5
−4.4
+1.2
+5.8
+0.9
+0.6
−0.1
+1.0
Table 1: Paired gap LlMBO − GRPO (points) per model (rows) at u=1,2,4 and benchmark (columns), mean over seed pairs; green: LlMBO ahead. avg@4: accuracy averaged over four completions at T=1 ; greedy: accuracy of the greedy completion (Section 5.3 ); LlMBO is trained on the metric of each column. One row block per u , each listing the four vanilla base models and the two math-specialized ones, each group closed by its Avg row. Standard errors and u≤16 in Table 4 ; the Qwen2.5 1.5B and Instruct models in Appendices C and D .
Figure 4: LlMBO against GRPO, gain over the base model in greedy accuracy as a function of training completions generated, on GSM8K (solid) and MATH-500 (dashed, open markers). Mean ± s.e. over seeds, averaged over u≤4 and the four 7-8B model families; circles mark where LlMBO passes GRPO’s final gain (dotted) for good. Each model and u separately in Figure 11 .
Table 2: Single-GPU memory feasibility per architecture. Weights in bf16. Each cell reports the number of simultaneous full-length sequences that fit on each GPU, in LlMBO/GRPO order. Green: LlMBO fits more. ✗ : the method does not fit.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Table 3: Paired gaps (points) per u≤4 , pooled over the six 7-8B base and math models and over the two Qwen2.5 1.5B models (row blocks; each model counted once), mean ± s.e. over seed pairs, in avg@4 and greedy accuracy (sub-columns), LlMBO trained on the metric of each column. Left: LlMBO − GRPO; train tier : greedy accuracy on up to 1024 problems of each model’s training tier, the quantity LlMBO maximizes there (evaluated in greedy only). Right: the b=4 ablation of LlMBO against b=1 (Appendix F ). Tint: sign and size of the gap (green: the first term ahead).
Figure 5: Gain over the base model of LlMBO against GRPO’s, in avg@4 and in greedy accuracy (color; greedy filled, avg@4 open), LlMBO trained on the metric of each point, for the one 7-8B base model per family and the two math-specialized 7B models (rows, in two blocks) on each benchmark (columns), each panel on its own axes. Training uses GSM8K and MATH: the left two columns are in-distribution, the four benchmarks right of the dotted line out-of-distribution. One point per u≤4 (marker), mean ± s.e. over seeds. Points in the shaded half above the diagonal favor LlMBO; dashed: no gain.
Figure 6: Paired gap LlMBO − GRPO in avg@4 and in greedy accuracy (color), LlMBO trained on each metric, against u∈{1,…,16} , mean ± s.e. over seed pairs, for the one 7-8B base model per family and the two math-specialized 7B models (rows, in two blocks), one column per benchmark (in-distribution GSM8K and MATH-500 first, then the four out-of-distribution benchmarks), each panel on its own gap axis. Solid, filled: b=1 candidate per batch (the default); dashed, open: the b=4 ablation.
Table 5: Avg@4 (a) and greedy (b) accuracy at the end of training, for the 2 Qwen2.5 7B models (row blocks), per u∈{64,256} (rows) and benchmark (columns: in-distribution GSM8K and MATH-500, then the four out-of-distribution benchmarks); LlMBO is trained on the metric of each subtable (greedy: Section 5.3 ). base : accuracy of the base model in that metric (%). GRPO , LlMBO : gain over the base model (points), mean over seeds (3 per cell); bold: the larger, unless tied. Δ : paired gap LlMBO − GRPO, mean ± s.e. over the seed pairs both have, so it can differ from the difference of the two gains; tint: its sign and size (green: LlMBO ahead).
Table 6: Avg@4 (a) and greedy (b) accuracy at the end of training, for Qwen2.5-1.5B and Qwen2.5-Math-1.5B (row blocks), per u≤16 (rows) and benchmark (columns: in-distribution GSM8K and MATH-500, then the four out-of-distribution benchmarks); LlMBO is trained on the metric of each subtable (greedy: Section 5.3 ). base : accuracy of the base model in that metric (%). GRPO , LlMBO : gain over the base model (points), mean over seeds (GRPO and avg@4-trained LlMBO 8 per cell at u≤4 and 3 at u=8,16 , greedy-trained LlMBO 3 per cell); bold: the larger, unless tied. Δ : paired gap LlMBO − GRPO, mean ± s.e. over the seed pairs both have, so it can differ from the difference of the two gains; tint: its sign and size (green: LlMBO ahead).
Figure 7: Gain over the base model of LlMBO against GRPO’s, in avg@4 and in greedy accuracy (color; greedy filled, avg@4 open), LlMBO trained on the metric of each point, for Qwen2.5-1.5B and Qwen2.5-Math-1.5B (rows) on each benchmark (columns), each panel on its own axes. Training uses GSM8K and MATH: the left two columns are in-distribution, the four benchmarks right of the dotted line out-of-distribution. One point per u≤4 (marker), mean ± s.e. over seeds. Points in the shaded half above the diagonal favor LlMBO; dashed: no gain.
Figure 8: Paired gap LlMBO − GRPO in avg@4 and in greedy accuracy (color), LlMBO trained on each metric, against u∈{1,…,16} , mean ± s.e. over seed pairs, for Qwen2.5-1.5B and Qwen2.5-Math-1.5B (rows), one column per benchmark (in-distribution GSM8K and MATH-500 first, then the four out-of-distribution benchmarks), each panel on its own gap axis. Solid, filled: b=1 candidate per batch (the default); dashed, open: the b=4 ablation.
Table 7: Avg@4 (a) and greedy (b) accuracy at the end of training, for Qwen2.5-7B-Instruct and Qwen2.5-1.5B-Instruct (row blocks), per u≤16 (rows) and benchmark (columns: in-distribution GSM8K and MATH-500, then the four out-of-distribution benchmarks); LlMBO is trained on the metric of each subtable (greedy: Section 5.3 ). base : accuracy of the base model in that metric (%). GRPO , LlMBO : gain over the base model (points), mean over seeds (GRPO and avg@4-trained LlMBO 8 per cell at u≤4 and 3 at u=8,16 , greedy-trained LlMBO 3 per cell); bold: the larger, unless tied. Δ : paired gap LlMBO − GRPO, mean ± s.e. over the seed pairs both have, so it can differ from the difference of the two gains; tint: its sign and size (green: LlMBO ahead).
Figure 9: Gain over the base model of LlMBO against GRPO’s, in avg@4 and in greedy accuracy (color; greedy filled, avg@4 open), LlMBO trained on the metric of each point, for Qwen2.5-7B-Instruct and Qwen2.5-1.5B-Instruct (rows) on each benchmark (columns), each panel on its own axes. Training uses GSM8K and MATH: the left two columns are in-distribution, the four benchmarks right of the dotted line out-of-distribution. One point per u≤4 (marker), mean ± s.e. over seeds. Points in the shaded half above the diagonal favor LlMBO; dashed: no gain.
Figure 10: Paired gap LlMBO − GRPO in avg@4 and in greedy accuracy (color), LlMBO trained on each metric, against u∈{1,…,16} , mean ± s.e. over seed pairs, for Qwen2.5-7B-Instruct and Qwen2.5-1.5B-Instruct (rows), one column per benchmark (in-distribution GSM8K and MATH-500 first, then the four out-of-distribution benchmarks), each panel on its own gap axis. Solid, filled: b=1 candidate per batch (the default); dashed, open: the b=4 ablation.
training distribution
held-out gain
Model
u
base (%)
s.e./eval.
GP gain
∥θ∥∞/R
GSM8K
MATH-500
Qwen2.5-7B-Inst.
1
65.1± 0.2
3.0
+0.8± 0.2
0.11± 0.02
+0.1± 0.1
+0.1± 0.5
2
64.9± 0.3
3.0
+1.6± 0.2
0.20± 0.05
+0.0± 0.2
+0.0± 0.2
4
65.2± 0.3
2.9
+2.3± 0.3
0.45± 0.09
−0.4± 0.2
−1.0± 0.4
Qwen2.5-1.5B-Inst.
1
77.4± 0.2
2.6
+0.5± 0.2
0.16± 0.03
+1.3± 0.3
−1.0± 0.3
2
77.2± 0.2
2.6
+1.2± 0.3
0.18± 0.03
+0.8± 0.1
−0.4± 0.3
Appendix
Table 8: What the search sees and what it delivers, per LlMBO run at u≤4 ( b=1 , trained on the pass rate at T=1 ), mean ± s.e. over seeds; Qwen2.5-7B, which TinyLoRA moves, for contrast. Training distribution : the base model’s sampled pass rate on the training tier, estimated from the eight θ=0 replicates of the search design; the standard error of one candidate’s evaluation (one completion on each of n=256 prompts), in points; GP gain : the final GP’s posterior mean at the returned adapter minus its posterior mean at θ=0 , in pass-rate points, the improvement the search believes it found; ∥θ∥∞/R : how far out in the search box [−R,R]u the returned adapter lies. Held-out gain : avg@4 accuracy on GSM8K and MATH-500 minus the base model’s (points).
in-distribution
out-of-distribution
GSM8K
MATH-500
AIME24
AMC23
Minerva
Olympiad
u
avg@4
greedy
Δ
avg@4
greedy
Δ
avg@4
greedy
Δ
avg@4
greedy
Δ
avg@4
greedy
Δ
avg@4
greedy
Δ
Llama3.1 8B
base
43.9
16.4
0.0
5.0
7.4
2.8
1
+1.1
+2.2
+0.2± 0.6
−0.3
+0.1
−0.2± 0.4
+0.0
+0.0
+0.0± 0.0
−1.6
+0.0
+0.8± 1.7
−1.3
−0.7
+0.2± 0.3
−0.2
+0.0
+0.0± 0.2
2
−0.3
+1.6
+3.8± 0.3
−1.7
−0.3
+2.1± 0.6
+0.0
+0.0
+0.0± 0.0
−2.5
+0.0
+3.3± 1.7
−2.6
−2.0
+0.4± 1.0
−0.3
−0.5
+0.1± 0.7
4
−1.6
+2.7
+4.3± 3.9
−2.1
+1.2
+3.5± 1.3
+0.0
+0.0
+0.0± 0.0
−1.6
−1.7
+0.0± 1.4
−2.3
−2.0
+1.1± 1.2
−0.4
−0.4
+0.0± 0.3
Appendix
Table 9: Greedy accuracy at the end of training of LlMBO trained on the pass rate at T=1 ( avg@4 ) and on greedy accuracy ( greedy ), the same 3 seeds per cell searching the same subspaces, for the four vanilla base models, the two math-specialized 7B models and the two Qwen2.5 1.5B models (row blocks), per u≤16 (rows) and benchmark (columns: in-distribution GSM8K and MATH-500, then the four out-of-distribution benchmarks). base : greedy accuracy of the base model (%). avg@4 , greedy : gain over the base model (points), mean over seeds; bold: the larger, unless tied. Δ : paired gap of the greedy objective over the avg@4 one, mean ± s.e. over seed pairs; tint: its sign and size (green: the greedy objective ahead).
Figure 11: Gain of the current adapter over the base model, in greedy accuracy against training completions generated so far, on the two in-distribution tasks at u∈{1,2,4} (columns), for every model (rows) trained on the SimpleRL-Zoo tiers, mean ± s.e. LlMBO ( b=1 ) is trained on greedy accuracy (Section 5.3 ); its adapter at each point is its evaluated trial of highest posterior mean; its completions are counted from the end of its initial design. Dotted: GRPO at the end of training; gray dashed: the base model.
equal-reward groups (%)
Model
pass rate (%)
observed
independent
ρ
1+3ρ
keff
Llama3.1-8B
3
90
88
0.09
1.27
3.2
Mistral-7B
2
93
92
0.06
1.19
3.4
DeepSeek-7B
1
97
97
0.03
1.09
3.7
Qwen2.5-7B
42
43
14
0.35
2.04
2.0
Qwen2.5-Math-7B
46
35
13
0.27
1.81
2.2
Appendix
Table 10: Correlation between completions of the same prompt, per model, from GRPO’s first 20 steps at u≤4 , close to the base model, at T=1 on the training tier. Equal-reward groups : groups of 4 completions of one prompt that all get the same reward, as observed and as expected if the completions were independent at the same pass rate; such groups give GRPO no gradient. ρ : intraclass correlation of the beta-binomial matching pass rate and observed equal-reward groups; 1+3ρ : variance inflation of a pass-rate estimate from groups of 4 over one completion per prompt; keff=4/(1+3ρ) : independent completions a group is worth.
beyond vLLM
sequences in flight
layers ×
weights
KV/seq.
(GiB)
L4 24 GB
A100 40 GB
L40S 48 GB
H100 80 GB
Architecture
KV heads
(GiB)
(MiB)
GRPO
LlMBO
GRPO
LlMBO
GRPO
LlMBO
GRPO
LlMBO
GRPO
LlMBO
Qwen2.5-1.5B
28 × 2
2.9
42
1.4
0.0
45
418
90
841
103
963
195
1812
Qwen2.5-7B
28 × 4
14.2
84
8.9
0.0
✗
71
10
282
12
343
27
767
Llama3.1-8B
32 × 8
15.0
192
8.0
0.0
✗
27
10
119
12
146
28
331
Mistral-7B
32 × 8
13.5
192
10.4
0.0
✗
34
8
127
10
154
22
339
Appendix
Table 11: Memory per architecture, pooling its models (DeepSeek-7B: the DeepSeek-Math-7B runs, same architecture); weights in bf16. Layers × KV heads : the attention shape that sets the KV cache (head dimension 128 throughout); KV/seq. : KV cache of one full-length sequence, 512 prompt + 1024 completion tokens. Beyond vLLM : the working set M a method allocates beyond the vLLM engine’s reserve, median over runs (for GRPO, one micro-batch of 4 sequences). Sequences in flight : estimated full-length sequences a method keeps in flight on each card (22.0, 39.4, 44.4 and 79.2 GiB usable). Bold: the better method; ✗: fewer than 4 sequences fit. Derivation and a worked example in the text.
Figure 12: Gain of the current adapter over the base model, in greedy accuracy against training completions generated so far, on the two in-distribution tasks at u∈{1,2,4} (columns), pooled over the four 7-8B families and over the two Qwen2.5 1.5B models (rows; each model counted once), trained on the SimpleRL-Zoo tiers, mean ± s.e.: GRPO and LlMBO trained on greedy accuracy (Section 5.3 ) with b=1 (solid, filled) and b=4 (dashed, open) candidates per batch. LlMBO’s adapter at each point is its evaluated trial of highest posterior mean, its completions counted from the end of its initial design. Dotted: GRPO at the end of training; gray dashed: the base model.
Figure 13: Gain over the base model in avg@4 (dashed) and in greedy accuracy (solid) against u≤4 , training on the SimpleRL-Zoo tiers (filled; hard tier at 7B, easy tier at 1.5B) and on MATH alone (open), the four Qwen2.5 base and Math models (rows), one column per benchmark (the two in-distribution tasks, then the four OOD benchmarks), each panel on its own scale; GRPO and LlMBO trained on the metric of each curve, mean ± s.e. over seeds.
Figure 14: Gain of the current adapter over the base model, in greedy accuracy against training completions generated so far, on the two in-distribution tasks at u∈{1,2,4} (columns), for the four Qwen2.5 base and Math models (rows) trained on the SimpleRL-Zoo tiers (solid, filled) and on MATH alone (dashed, open), mean ± s.e., for GRPO and for LlMBO ( b=1 ) trained on greedy accuracy (Section 5.3 ). LlMBO’s adapter at each point is its evaluated trial of highest posterior mean, its completions counted from the end of its initial design. Gray dashed: the base model.
Figure 15: Greedy (solid) and avg@4 (dashed) accuracy at the end of training against trainable parameters under GRPO, for every model (rows, in three blocks: the one 7-8B base model per family; the two math-specialized 7B models; Qwen2.5-7B-Instruct and the Qwen2.5 1.5B models, which train on the easy tier), on each benchmark (columns: in-distribution GSM8K and MATH-500, then the four out-of-distribution benchmarks right of the dotted line), each panel on its own accuracy axis: LoRA and LoRA-XS at r∈{1,2,8,32} , TinyLoRA at u∈{1,…,256} . Gray: the base model, in the same line style. Mean ± s.e. over seeds.
Shared
Adapter
TinyLoRA, r=2 , one v∈Ru tied across all M modules (Appendix H.1 )
Prompt / completion length
512 / 1024 tokens
Completions per step or batch
256
Reward
1 if math-verify finds the final answer equivalent to the reference, else 0
GRPO
Prompts × rollouts per step
64×4 , at T=1
Appendix
Table 13: Settings of both optimizers. N : prompts in the training set; S : GRPO’s optimizer steps; s of n : a candidate’s solved prompts.
Figure 16: The Gaussian observation a GP receives for a candidate that solves s of n=256 prompts, against the Beta(1+s,1+n−s) posterior on its pass rate. Top : a Gaussian on the rate with the plug-in binomial variance, each in pass-rate space. Bottom : LlMBO’s Gaussian with the exact mean y and variance σ2 of the posterior’s logit (Section 4 ), in logit space, the space the GP models.
Optimization of LLM training and inference configurations, such as hyperparameters, data mixtures, and prompts, is critical to performance, but it is often approached heuristically in practice, leading to potentially suboptimal outcomes. By framing them as noisy, expensive, and derivative-free optimization problems, Bayesian optimization (BO) and other black-box optimization (BBO) methods offer a promising yet underexplored direction for principled, sample-efficient methods. However, LLM training and inference costs are prohibitively high for most of the BBO research community, and new methods are often only evaluated on synthetic test functions and small-scale datasets that fail to capture the challenges of modern LLM optimization problems. This impedes the development of BBO methods and makes it difficult to assess their effectiveness on modern LLM tasks. We introduce BoLT, the first LLM-centric benchmark that democratizes LLM research for the BBO community. BoLT is released at https://github.com/chewwt/bolt. BoLT covers broad and well-motivated LLM optimization problems, involving multi-fidelity, multi-objective, heteroscedastic noise, and high-dimensional search spaces. Each problem in BoLT is grounded in real experimental data and made fully reproducible and accessible through lightweight surrogate models fitted to the results of thousands of real LLM experiments. We benchmark BoLT against an extensive range of BO and BBO methods, showing that selected BO methods consistently outperform others across tasks and highlighting gaps in existing BBO methods on LLM tasks, underscoring the need to modernize benchmarks for the BBO community.
Ruth Wan Theng Chew, Zhiliang Chen, Apivich Hemachandra +1
Gradient-free post-training has emerged as a compelling alternative to gradient-based optimization for large language models (LLMs), but existing approaches remain costly. We ask whether structured search can identify a strong single expert under a modest evaluation budget. Motivated by evidence that useful weight updates lie in low-dimensional subspaces, we apply Bayesian optimization within a random linear embedding of weight space. Our method requires no backpropagation and uses a Gaussian process surrogate to guide candidate evaluations efficiently. Across several reasoning benchmarks with Qwen2.5-Instruct models from 0.5B to 3B parameters, Bayesian optimization using five times less candidate evaluations matches or exceeds RandOpt. These results show that surrogate-guided search can substantially reduce the evaluation cost of gradient-free post-training while producing stronger deployable single experts.
Nigel Bastian Cendra, Abdelhamid Ezzerg, Fernando Julio Cendra +2
University College London · Institut Polytechnique de Paris · University of Oxford
Large Language Models usually put more emphasis on accuracy and therefore, will guess even when not certain about the prediction, which is especially severe when fine-tuned on small datasets due to the inherent tendency toward miscalibration. In this work, we introduce Bayesian-LoRA, which reformulates the deterministic LoRA update as a probabilistic low-rank representation inspired by Sparse Gaussian Processes. We identify a structural isomorphism between LoRA's factorization and Kronecker-factored SGP posteriors, and show that LoRA emerges as a limiting case when posterior uncertainty collapses. We conduct extensive experiments on various LLM architectures across commonsense reasoning benchmarks. With only approximately 0.42M additional parameters and ≈1.2× training cost relative to standard LoRA, Bayesian-LoRA significantly improves calibration across models up to 30B, achieving up to 84% ECE reduction and 76% NLL reduction while maintaining competitive accuracy for both in-distribution and out-of-distribution (OoD) evaluations.
Moule Lin, Shuhao Guan, Andrea Patane +2
School of Computer Science and Statistics, Trinity College Dublin, Dublin, Ireland and Lero the Research Ireland Centre for Software, Ireland · School of Computer Science, University College Dublin, Dublin, Ireland