stat.MLOct 13, 2026
SaveHow Low Can You Go: Fine-Tuning LLMs with Bayesian Optimization (LlMBO)
Organizations: Department of Physics, University of Massachusetts Amherst · Department of Mathematics and Statistics, University of Massachusetts Amherst
Abstract
Recently, Morris et al. introduced TinyLoRA, a strategy for LLM fine-tuning which introduces arbitrarily few parameters, and demonstrated accuracy improvements using less than parameters rather than the typically demanded by LoRA-based methods. However, that study continued to use gradient-based local search methods of the kind deployed for training multi-billion parameter LLMs in the first place. This article is predicated on the observation that, in the low dimensional setting, global optimization methods become tractable, unlocking Large language Model Bayesian Optimization (LlMBO). LlMBO is sample-efficient, gradient-free, memory-light, parallelizes across devices without synchronization, and, unlike policy-gradient methods, can optimize greedy-decoding accuracy directly. We describe numerical experiments comparing LlMBO against GRPO on fine-tuning tasks for a variety of models on a math benchmark suite which show that LlMBO can be a suitable alternative in this regime, matching or exceeding GRPO's fine tuning gains, while using a fraction of the compute and memory.
Figures & tables
Figure 1: LlMBO against GRPO, evaluated both in terms of mean performance at temperature 1 on 4 trials (avg@4, open points) and via greedy decoding (filled). Gain over the base model (mean s.e. over seeds) for runs with trainable parameters (marker) and model family (color).
Figure 2: TinyLoRA scaling laws under GRPO. Greedy (solid) and avg@4 (dashed) accuracy at the end of training under GRPO (mean s.e. over seeds) for TinyLoRA, LoRA-XS and LoRA as a function of trainable parameters. Results shown for Qwen2.5-7B, scaling for other model families can be found in Figure 15 .
Figure 3: Gain over the base model of LlMBO against that of GRPO on the in-distribution benchmarks, for the four vanilla base models; avg@4 (open) and greedy accuracy (filled), one point per (marker), mean s.e. over seeds. LlMBO ( ) is trained on the metric of each point; points in the shaded half favor LlMBO. Every model and benchmark in Figure 5 .
| in-distribution | out-of-distribution | ||||||||||||
| GSM8K | MATH-500 | AIME24 | AMC23 | Minerva | Olympiad | ||||||||
| Model | avg@4 | greedy | avg@4 | greedy | avg@4 | greedy | avg@4 | greedy | avg@4 | greedy | avg@4 | greedy | |
| 1 | Llama3.1-8B | ||||||||||||
| Mistral-7B | |||||||||||||
| DeepSeek-7B | |||||||||||||
| Qwen2.5-7B | |||||||||||||
Table 1: Paired gap LlMBO GRPO (points) per model (rows) at and benchmark (columns), mean over seed pairs; green: LlMBO ahead. avg@4: accuracy averaged over four completions at ; greedy: accuracy of the greedy completion (Section 5.3 ); LlMBO is trained on the metric of each column. One row block per , each listing the four vanilla base models and the two math-specialized ones, each group closed by its Avg row. Standard errors and in Table 4 ; the Qwen2.5 1.5B and Instruct models in Appendices C and D .
Figure 4: LlMBO against GRPO, gain over the base model in greedy accuracy as a function of training completions generated, on GSM8K (solid) and MATH-500 (dashed, open markers). Mean s.e. over seeds, averaged over and the four 7-8B model families; circles mark where LlMBO passes GRPO’s final gain (dotted) for good. Each model and separately in Figure 11 .
Table 2: Single-GPU memory feasibility per architecture. Weights in bf16. Each cell reports the number of simultaneous full-length sequences that fit on each GPU, in LlMBO/GRPO order. Green: LlMBO fits more. ✗ : the method does not fit.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Table 3: Paired gaps (points) per , pooled over the six 7-8B base and math models and over the two Qwen2.5 1.5B models (row blocks; each model counted once), mean s.e. over seed pairs, in avg@4 and greedy accuracy (sub-columns), LlMBO trained on the metric of each column. Left: LlMBO GRPO; train tier : greedy accuracy on up to 1024 problems of each model’s training tier, the quantity LlMBO maximizes there (evaluated in greedy only). Right: the ablation of LlMBO against (Appendix F ). Tint: sign and size of the gap (green: the first term ahead).
Figure 5: Gain over the base model of LlMBO against GRPO’s, in avg@4 and in greedy accuracy (color; greedy filled, avg@4 open), LlMBO trained on the metric of each point, for the one 7-8B base model per family and the two math-specialized 7B models (rows, in two blocks) on each benchmark (columns), each panel on its own axes. Training uses GSM8K and MATH: the left two columns are in-distribution, the four benchmarks right of the dotted line out-of-distribution. One point per (marker), mean s.e. over seeds. Points in the shaded half above the diagonal favor LlMBO; dashed: no gain.
Figure 6: Paired gap LlMBO GRPO in avg@4 and in greedy accuracy (color), LlMBO trained on each metric, against , mean s.e. over seed pairs, for the one 7-8B base model per family and the two math-specialized 7B models (rows, in two blocks), one column per benchmark (in-distribution GSM8K and MATH-500 first, then the four out-of-distribution benchmarks), each panel on its own gap axis. Solid, filled: candidate per batch (the default); dashed, open: the ablation.
Table 5: Avg@4 (a) and greedy (b) accuracy at the end of training, for the 2 Qwen2.5 7B models (row blocks), per (rows) and benchmark (columns: in-distribution GSM8K and MATH-500, then the four out-of-distribution benchmarks); LlMBO is trained on the metric of each subtable (greedy: Section 5.3 ). base : accuracy of the base model in that metric (%). GRPO , LlMBO : gain over the base model (points), mean over seeds (3 per cell); bold: the larger, unless tied. : paired gap LlMBO GRPO, mean s.e. over the seed pairs both have, so it can differ from the difference of the two gains; tint: its sign and size (green: LlMBO ahead).
Table 6: Avg@4 (a) and greedy (b) accuracy at the end of training, for Qwen2.5-1.5B and Qwen2.5-Math-1.5B (row blocks), per (rows) and benchmark (columns: in-distribution GSM8K and MATH-500, then the four out-of-distribution benchmarks); LlMBO is trained on the metric of each subtable (greedy: Section 5.3 ). base : accuracy of the base model in that metric (%). GRPO , LlMBO : gain over the base model (points), mean over seeds (GRPO and avg@4-trained LlMBO 8 per cell at and 3 at , greedy-trained LlMBO 3 per cell); bold: the larger, unless tied. : paired gap LlMBO GRPO, mean s.e. over the seed pairs both have, so it can differ from the difference of the two gains; tint: its sign and size (green: LlMBO ahead).
Figure 7: Gain over the base model of LlMBO against GRPO’s, in avg@4 and in greedy accuracy (color; greedy filled, avg@4 open), LlMBO trained on the metric of each point, for Qwen2.5-1.5B and Qwen2.5-Math-1.5B (rows) on each benchmark (columns), each panel on its own axes. Training uses GSM8K and MATH: the left two columns are in-distribution, the four benchmarks right of the dotted line out-of-distribution. One point per (marker), mean s.e. over seeds. Points in the shaded half above the diagonal favor LlMBO; dashed: no gain.
Figure 8: Paired gap LlMBO GRPO in avg@4 and in greedy accuracy (color), LlMBO trained on each metric, against , mean s.e. over seed pairs, for Qwen2.5-1.5B and Qwen2.5-Math-1.5B (rows), one column per benchmark (in-distribution GSM8K and MATH-500 first, then the four out-of-distribution benchmarks), each panel on its own gap axis. Solid, filled: candidate per batch (the default); dashed, open: the ablation.
Table 7: Avg@4 (a) and greedy (b) accuracy at the end of training, for Qwen2.5-7B-Instruct and Qwen2.5-1.5B-Instruct (row blocks), per (rows) and benchmark (columns: in-distribution GSM8K and MATH-500, then the four out-of-distribution benchmarks); LlMBO is trained on the metric of each subtable (greedy: Section 5.3 ). base : accuracy of the base model in that metric (%). GRPO , LlMBO : gain over the base model (points), mean over seeds (GRPO and avg@4-trained LlMBO 8 per cell at and 3 at , greedy-trained LlMBO 3 per cell); bold: the larger, unless tied. : paired gap LlMBO GRPO, mean s.e. over the seed pairs both have, so it can differ from the difference of the two gains; tint: its sign and size (green: LlMBO ahead).
Figure 9: Gain over the base model of LlMBO against GRPO’s, in avg@4 and in greedy accuracy (color; greedy filled, avg@4 open), LlMBO trained on the metric of each point, for Qwen2.5-7B-Instruct and Qwen2.5-1.5B-Instruct (rows) on each benchmark (columns), each panel on its own axes. Training uses GSM8K and MATH: the left two columns are in-distribution, the four benchmarks right of the dotted line out-of-distribution. One point per (marker), mean s.e. over seeds. Points in the shaded half above the diagonal favor LlMBO; dashed: no gain.
Figure 10: Paired gap LlMBO GRPO in avg@4 and in greedy accuracy (color), LlMBO trained on each metric, against , mean s.e. over seed pairs, for Qwen2.5-7B-Instruct and Qwen2.5-1.5B-Instruct (rows), one column per benchmark (in-distribution GSM8K and MATH-500 first, then the four out-of-distribution benchmarks), each panel on its own gap axis. Solid, filled: candidate per batch (the default); dashed, open: the ablation.
| training distribution | held-out gain | ||||||
|---|---|---|---|---|---|---|---|
| Model | base (%) | s.e./eval. | GP gain | GSM8K | MATH-500 | ||
| Qwen2.5-7B-Inst. | 1 | 0.2 | 0.2 | 0.02 | 0.1 | 0.5 | |
| 2 | 0.3 | 0.2 | 0.05 | 0.2 | 0.2 | ||
| 4 | 0.3 | 0.3 | 0.09 | 0.2 | 0.4 | ||
| Qwen2.5-1.5B-Inst. | 1 | 0.2 | 0.2 | 0.03 | 0.3 | 0.3 | |
| 2 | 0.2 | 0.3 | 0.03 | 0.1 | 0.3 | ||
Table 8: What the search sees and what it delivers, per LlMBO run at ( , trained on the pass rate at ), mean s.e. over seeds; Qwen2.5-7B, which TinyLoRA moves, for contrast. Training distribution : the base model’s sampled pass rate on the training tier, estimated from the eight replicates of the search design; the standard error of one candidate’s evaluation (one completion on each of prompts), in points; GP gain : the final GP’s posterior mean at the returned adapter minus its posterior mean at , in pass-rate points, the improvement the search believes it found; : how far out in the search box the returned adapter lies. Held-out gain : avg@4 accuracy on GSM8K and MATH-500 minus the base model’s (points).
| in-distribution | out-of-distribution | ||||||||||||||||||
| GSM8K | MATH-500 | AIME24 | AMC23 | Minerva | Olympiad | ||||||||||||||
| avg@4 | greedy | avg@4 | greedy | avg@4 | greedy | avg@4 | greedy | avg@4 | greedy | avg@4 | greedy | ||||||||
| Llama3.1 8B | base | 43.9 | 16.4 | 0.0 | 5.0 | 7.4 | 2.8 | ||||||||||||
| 1 | 0.6 | 0.4 | 0.0 | 1.7 | 0.3 | 0.2 | |||||||||||||
| 2 | 0.3 | 0.6 | 0.0 | 1.7 | 1.0 | 0.7 | |||||||||||||
| 4 | 3.9 | 1.3 | 0.0 | 1.4 | 1.2 | 0.3 | |||||||||||||
Table 9: Greedy accuracy at the end of training of LlMBO trained on the pass rate at ( avg@4 ) and on greedy accuracy ( greedy ), the same 3 seeds per cell searching the same subspaces, for the four vanilla base models, the two math-specialized 7B models and the two Qwen2.5 1.5B models (row blocks), per (rows) and benchmark (columns: in-distribution GSM8K and MATH-500, then the four out-of-distribution benchmarks). base : greedy accuracy of the base model (%). avg@4 , greedy : gain over the base model (points), mean over seeds; bold: the larger, unless tied. : paired gap of the greedy objective over the avg@4 one, mean s.e. over seed pairs; tint: its sign and size (green: the greedy objective ahead).
Figure 11: Gain of the current adapter over the base model, in greedy accuracy against training completions generated so far, on the two in-distribution tasks at (columns), for every model (rows) trained on the SimpleRL-Zoo tiers, mean s.e. LlMBO ( ) is trained on greedy accuracy (Section 5.3 ); its adapter at each point is its evaluated trial of highest posterior mean; its completions are counted from the end of its initial design. Dotted: GRPO at the end of training; gray dashed: the base model.
| equal-reward groups (%) | ||||||
| Model | pass rate (%) | observed | independent | |||
| Llama3.1-8B | 3 | 90 | 88 | 0.09 | 1.27 | 3.2 |
| Mistral-7B | 2 | 93 | 92 | 0.06 | 1.19 | 3.4 |
| DeepSeek-7B | 1 | 97 | 97 | 0.03 | 1.09 | 3.7 |
| Qwen2.5-7B | 42 | 43 | 14 | 0.35 | 2.04 | 2.0 |
| Qwen2.5-Math-7B | 46 | 35 | 13 | 0.27 | 1.81 | 2.2 |
Table 10: Correlation between completions of the same prompt, per model, from GRPO’s first 20 steps at , close to the base model, at on the training tier. Equal-reward groups : groups of 4 completions of one prompt that all get the same reward, as observed and as expected if the completions were independent at the same pass rate; such groups give GRPO no gradient. : intraclass correlation of the beta-binomial matching pass rate and observed equal-reward groups; : variance inflation of a pass-rate estimate from groups of 4 over one completion per prompt; : independent completions a group is worth.
| beyond vLLM | sequences in flight | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| layers | weights | KV/seq. | (GiB) | L4 24 GB | A100 40 GB | L40S 48 GB | H100 80 GB | ||||||
| Architecture | KV heads | (GiB) | (MiB) | GRPO | LlMBO | GRPO | LlMBO | GRPO | LlMBO | GRPO | LlMBO | GRPO | LlMBO |
| Qwen2.5-1.5B | 28 2 | 2.9 | 42 | 1.4 | 0.0 | 45 | 418 | 90 | 841 | 103 | 963 | 195 | 1812 |
| Qwen2.5-7B | 28 4 | 14.2 | 84 | 8.9 | 0.0 | ✗ | 71 | 10 | 282 | 12 | 343 | 27 | 767 |
| Llama3.1-8B | 32 8 | 15.0 | 192 | 8.0 | 0.0 | ✗ | 27 | 10 | 119 | 12 | 146 | 28 | 331 |
| Mistral-7B | 32 8 | 13.5 | 192 | 10.4 | 0.0 | ✗ | 34 | 8 | 127 | 10 | 154 | 22 | 339 |
Table 11: Memory per architecture, pooling its models (DeepSeek-7B: the DeepSeek-Math-7B runs, same architecture); weights in bf16. Layers KV heads : the attention shape that sets the KV cache (head dimension 128 throughout); KV/seq. : KV cache of one full-length sequence, 512 prompt + 1024 completion tokens. Beyond vLLM : the working set a method allocates beyond the vLLM engine’s reserve, median over runs (for GRPO, one micro-batch of 4 sequences). Sequences in flight : estimated full-length sequences a method keeps in flight on each card (22.0, 39.4, 44.4 and 79.2 GiB usable). Bold: the better method; ✗: fewer than 4 sequences fit. Derivation and a worked example in the text.
Figure 12: Gain of the current adapter over the base model, in greedy accuracy against training completions generated so far, on the two in-distribution tasks at (columns), pooled over the four 7-8B families and over the two Qwen2.5 1.5B models (rows; each model counted once), trained on the SimpleRL-Zoo tiers, mean s.e.: GRPO and LlMBO trained on greedy accuracy (Section 5.3 ) with (solid, filled) and (dashed, open) candidates per batch. LlMBO’s adapter at each point is its evaluated trial of highest posterior mean, its completions counted from the end of its initial design. Dotted: GRPO at the end of training; gray dashed: the base model.
Figure 13: Gain over the base model in avg@4 (dashed) and in greedy accuracy (solid) against , training on the SimpleRL-Zoo tiers (filled; hard tier at 7B, easy tier at 1.5B) and on MATH alone (open), the four Qwen2.5 base and Math models (rows), one column per benchmark (the two in-distribution tasks, then the four OOD benchmarks), each panel on its own scale; GRPO and LlMBO trained on the metric of each curve, mean s.e. over seeds.
Figure 14: Gain of the current adapter over the base model, in greedy accuracy against training completions generated so far, on the two in-distribution tasks at (columns), for the four Qwen2.5 base and Math models (rows) trained on the SimpleRL-Zoo tiers (solid, filled) and on MATH alone (dashed, open), mean s.e., for GRPO and for LlMBO ( ) trained on greedy accuracy (Section 5.3 ). LlMBO’s adapter at each point is its evaluated trial of highest posterior mean, its completions counted from the end of its initial design. Gray dashed: the base model.
Figure 15: Greedy (solid) and avg@4 (dashed) accuracy at the end of training against trainable parameters under GRPO, for every model (rows, in three blocks: the one 7-8B base model per family; the two math-specialized 7B models; Qwen2.5-7B-Instruct and the Qwen2.5 1.5B models, which train on the easy tier), on each benchmark (columns: in-distribution GSM8K and MATH-500, then the four out-of-distribution benchmarks right of the dotted line), each panel on its own accuracy axis: LoRA and LoRA-XS at , TinyLoRA at . Gray: the base model, in the same line style. Mean s.e. over seeds.
| Shared | |
|---|---|
| Adapter | TinyLoRA, , one tied across all modules (Appendix H.1 ) |
| Prompt / completion length | 512 / 1024 tokens |
| Completions per step or batch | 256 |
| Reward | 1 if math-verify finds the final answer equivalent to the reference, else 0 |
| GRPO | |
| Prompts rollouts per step | , at |
Table 13: Settings of both optimizers. : prompts in the training set; : GRPO’s optimizer steps; of : a candidate’s solved prompts.
Figure 16: The Gaussian observation a GP receives for a candidate that solves of prompts, against the posterior on its pass rate. Top : a Gaussian on the rate with the plug-in binomial variance, each in pass-rate space. Bottom : LlMBO’s Gaussian with the exact mean and variance of the posterior’s logit (Section 4 ), in logit space, the space the GP models.