BBOWP-Bench: Evaluating LLMs on Black-Box Optimization Word Problems
Organizations: Yokohama National University
Abstract
Formulating an optimization problem strongly affects final solution quality, yet good formulations usually require substantial expertise. Recent studies have explored automatically deriving optimization problems from natural-language descriptions, but existing benchmarks focus on settings where objectives and constraints can be written explicitly as mathematical expressions. Many practically important problems are naturally treated as black-box optimization (BBO) problems, in which only objective values are observable, and the functional form is unavailable. In BBO, search space design (as part of problem formulation) and optimization algorithm selection are crucial for problem-solving. Automating these processes with large language models (LLMs) remains challenging. We introduce Black-Box Optimization Word Problems (BBOWP), a novel problem setting in which a system must infer both a search space and an optimization algorithm from a natural-language description of a black-box optimization task. To support research on this setting, we establish the BBOWP Benchmark Suite (BBOWP-Bench), a dataset and evaluation framework for BBOWP. Each instance combines a natural-language problem description, an executable evaluation environment, and a human-designed reference formulation, allowing evaluation of both search-space design and algorithm selection. Using this benchmark, we evaluate LLMs on the joint formulation of search spaces and optimizers from natural-language descriptions across multiple BBO domains. We show that current LLMs vary their optimizer choice with the evaluation budget, though they sometimes struggle with search space design, particularly in identifying important variables and balancing their ranges when the problem description is less informative or the search space is highly problem-specific. Our code and dataset are available at https://github.com/shiralab/bbowp-bench.
Figures & tables
| Source | Task | Scenario | Dim | Float | Int | Cat |
|---|---|---|---|---|---|---|
| YAHPO | 3 | HPO of Ranger | 8 | |||
| 5 | HPO of SVM | 6 | ||||
| Olympus | 14 | S N Ar reaction | 4 | |||
| 15 | HPLC system | 6 | ||||
| MECHBench | 17 | Star-shaped crash box | 5 | |||
| MuJoCo | 19 | Inverted Pendulum | 5 |
| YAHPO | Olympus | MECHBench | ||||
| Difficulty | Model | Task 3 ( ) | Task 5 ( ) | Task 14 ( ) | Task 15 ( ) | Task 17 ( ) |
| Easy | Gemini 3 Flash | 0.94 0.02 | 0.98 0.00 | 0.19 0.00 | 2521.76 155.43 | -12044.69 548.96 |
| Gemini 3.1 Pro | 0.86 0.00 | 0.95 0.01 | 0.20 0.01 | 2436.03 64.52 | -12044.69 548.96 | |
| GPT-5 mini | 0.94 0.00 | 0.97 0.01 | 0.32 0.02 | 2282.10 32.82 | -10823.81 682.55 | |
| GPT-5.2 | 0.95 0.01 | 0.95 0.01 | 0.19 0.01 | 2277.24 33.38 | -10804.95 709.63 | |
| Hard | Gemini 3 Flash | 0.88 0.00 | 0.95 0.00 | 0.91 0.01 | -8528.06 6.86 | |
| Difficulty | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
|---|---|---|---|---|
| Easy | 0.940 0.003 | 0.871 0.038 | 0.849 0.011 | 0.903 0.018 |
| Intermediate | 0.420 0.019 | 0.446 0.024 | 0.354 0.004 | 0.439 0.002 |
| Hard | 0.491 0.010 | 0.443 0.018 | 0.426 0.013 | 0.446 0.008 |
| Average | 0.617 0.004 | 0.587 0.006 | 0.543 0.002 | 0.596 0.008 |
Appendix figures & tables36 assets
Supplementary material from the paper’s appendix.
Appendix
| Source | Task | Scenario | Dim | Float | Int | Cat |
| YAHPO | 1 | HPO of MLP | 7 | |||
| 2 | HPO of RcppHNSW | 6 | ||||
| 3 | HPO of Ranger | 8 | ||||
| 4 | HPO of RPART | 5 | ||||
| 5 | HPO of SVM | 6 | ||||
| 6 | HPO of XGBoost | 14 |
| Domain | # Task | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
|---|---|---|---|---|---|
| YAHPO | 1 ( ) | 99.59 0.04 | 99.58 0.01 | 99.55 0.10 | 99.25 0.06 |
| 2 ( ) | 0.96 0.01 | 0.96 0.01 | 0.96 0.01 | 0.96 0.01 | |
| 3 ( ) | 0.94 0.02 | 0.86 0.00 | 0.94 0.00 | 0.95 0.01 | |
| 4 ( ) | 0.92 0.00 | 0.91 0.00 | 0.91 0.00 | 0.92 0.00 | |
| 5 ( ) | 0.98 0.00 | 0.95 0.01 | 0.97 0.01 | 0.95 0.01 | |
| 6 ( ) | 1.00 0.00 | 0.99 0.00 | 1.00 0.00 | 0.99 0.00 |
| Domain | # Task | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
|---|---|---|---|---|---|
| YAHPO | 1 ( ) | 99.42 0.09 | 97.34 0.10 | 97.80 0.96 | 99.07 0.19 |
| 2 ( ) | 0.95 0.00 | 0.96 0.00 | 0.94 0.00 | 0.92 0.00 | |
| 3 ( ) | 0.86 0.03 | 0.88 0.00 | 0.88 0.00 | 0.80 0.01 | |
| 4 ( ) | 0.91 0.00 | 0.92 0.00 | 0.91 0.00 | 0.91 0.00 | |
| 5 ( ) | 0.95 0.00 | 0.95 0.00 | 0.94 0.00 | 0.94 0.00 | |
| 6 ( ) | 0.99 0.00 | 0.99 0.00 | 0.99 0.00 | 0.99 0.00 |
| Domain | # Task | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
|---|---|---|---|---|---|
| YAHPO | 1 ( ) | 98.31 0.29 | 99.61 0.00 | 97.67 0.58 | |
| 2 ( ) | 0.93 0.00 | 0.94 0.00 | 0.94 0.00 | 0.93 0.00 | |
| 3 ( ) | 0.88 0.00 | 0.82 0.00 | 0.84 0.00 | 0.82 0.00 | |
| 4 ( ) | 0.91 0.00 | 0.92 0.00 | 0.92 0.00 | 0.92 0.00 | |
| 5 ( ) | 0.95 0.00 | 0.95 0.01 | 0.93 0.01 | 0.95 0.00 | |
| 6 ( ) | 0.99 0.00 | 0.99 0.00 | 0.99 0.00 | 0.99 0.00 |
| Domain | # Task | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
|---|---|---|---|---|---|
| YAHPO | 1 ( ) | 99.66 0.00 | 99.66 0.00 | 99.66 0.00 | 99.57 0.00 |
| 2 ( ) | 0.97 0.00 | 0.96 0.00 | 0.97 0.00 | 0.97 0.00 | |
| 3 ( ) | 0.96 0.00 | 0.97 0.00 | 0.96 0.00 | 0.96 0.00 | |
| 4 ( ) | 0.92 0.00 | 0.92 0.00 | 0.92 0.00 | 0.92 0.00 | |
| 5 ( ) | 0.98 0.00 | 0.98 0.00 | 0.98 0.00 | 0.98 0.00 | |
| 6 ( ) | 1.00 0.00 | 0.99 0.00 | 0.99 0.00 | – |
| Domain | # Task | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
|---|---|---|---|---|---|
| YAHPO | 1 ( ) | – | 99.66 0.00 | 99.63 0.00 | 99.57 0.00 |
| 2 ( ) | – | 0.96 0.00 | 0.96 0.00 | – | |
| 3 ( ) | – | 0.91 0.00 | – | – | |
| 4 ( ) | – | 0.92 0.00 | 0.92 0.00 | 0.92 0.00 | |
| 5 ( ) | 0.95 0.00 | 0.96 0.00 | – | – | |
| 6 ( ) | – | 0.99 0.00 | – | – |
| Domain | # Task | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
|---|---|---|---|---|---|
| YAHPO | 1 ( ) | 99.50 0.00 | 98.45 0.00 | 99.50 0.00 | |
| 2 ( ) | – | 0.96 0.00 | – | – | |
| 3 ( ) | 0.91 † | 0.91 0.00 | 0.89 0.00 | 0.86 † | |
| 4 ( ) | – | 0.92 0.00 | – | – | |
| 5 ( ) | – | 0.96 0.00 | 0.96 0.00 | 0.95 0.00 | |
| 6 ( ) | – | 0.99 0.00 | 0.99 0.00 |
| Domain | Task | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
|---|---|---|---|---|---|
| YAHPO | Task 1 ( ) | 97.01 | 96.12 | 99.55 | 98.11 |
| Task 2 ( ) | 0.95 | 0.96 | 0.96 | 0.96 | |
| Task 3 ( ) | 0.88 | 0.86 | 0.91 | 0.94 | |
| Task 4 ( ) | 0.92 | 0.91 | 0.92 | 0.92 | |
| Task 5 ( ) | 0.95 | 0.95 | 0.98 | 0.95 | |
| Task 6 ( ) | 0.99 | 0.99 | 0.99 | 0.99 |
| Domain | Task | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
|---|---|---|---|---|---|
| YAHPO | Task 1 ( ) | 98.33 | 95.25 | 99.36 | 99.45 |
| Task 2 ( ) | 0.95 | 0.96 | 0.94 | 0.92 | |
| Task 3 ( ) | 0.84 | 0.79 | 0.89 | 0.81 | |
| Task 4 ( ) | 0.90 | 0.92 | 0.92 | 0.91 | |
| Task 5 ( ) | 0.94 | 0.95 | 0.96 | 0.95 | |
| Task 6 ( ) | 0.99 | 0.99 | 0.99 | 0.99 |
| Domain | Task | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
|---|---|---|---|---|---|
| YAHPO | Task 1 ( ) | 97.78 | 96.52 | ||
| Task 2 ( ) | 0.94 | 0.93 | 0.93 | 0.92 | |
| Task 3 ( ) | 0.86 | 0.76 | 0.84 | 0.83 | |
| Task 4 ( ) | 0.92 | 0.92 | 0.92 | 0.91 | |
| Task 5 ( ) | 0.95 | 0.95 | 0.94 | 0.95 | |
| Task 6 ( ) | 0.99 | 0.99 | 0.99 |
| Domain | Task | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
|---|---|---|---|---|---|
| YAHPO | Task 1 ( ) | – | 99.66 | 99.57 | |
| Task 2 ( ) | 0.96 | 0.96 | 0.96 | 0.96 | |
| Task 3 ( ) | 0.96 | 0.91 | 0.96 | 0.96 | |
| Task 4 ( ) | 0.92 | 0.92 | 0.92 | 0.92 | |
| Task 5 ( ) | 0.96 | 0.98 | 0.98 | 0.96 | |
| Task 6 ( ) | – | 0.99 | 1.00 | 0.99 |
| Domain | Task | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
|---|---|---|---|---|---|
| YAHPO | Task 1 ( ) | 98.53 | 99.57 | 99.50 | 99.57 |
| Task 2 ( ) | – | 0.96 | – | – | |
| Task 3 ( ) | – | 0.91 | – | 0.89 | |
| Task 4 ( ) | – | 0.92 | 0.92 | – | |
| Task 5 ( ) | – | – | – | 0.95 | |
| Task 6 ( ) | 0.99 | 0.99 | – | 0.99 |
| Domain | Task | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
|---|---|---|---|---|---|
| YAHPO | Task 1 ( ) | – | 99.57 | ||
| Task 2 ( ) | – | 0.93 | – | 0.96 | |
| Task 3 ( ) | 0.86 | 0.91 | 0.87 | – | |
| Task 4 ( ) | – | 0.92 | 0.92 | – | |
| Task 5 ( ) | – | 0.95 | 0.95 | 0.95 | |
| Task 6 ( ) | – | 0.99 | – | 0.99 |
| Variables | Canonical Domain | Human Reference | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
| cost | [4.54e-5, 22026.47] | [4.54e-5, 22026.47], log | [1e-3, 1e3], log | [1e-3, 1e3], log | [1e-4, 1.0], log | [1e-3, 1e3], log |
| kernel | {linear, radial, polynomial} | {linear, radial, polynomial} | {linear, radial, polynomial} | {linear, radial} | {linear, radial, polynomial} | {linear, radial} |
| tolerance | [4.54e-5, 2.0] | [4.54e-5, 2.0], log | [1e-5, 1e-2], log | |||
| degree | [2, 5], step=1 | [2, 5], step=1 | [2, 5], step=2 | |||
| gamma | [4.54e-5, 22026.47], log | [4.54e-5, 22026.47], log | [1e-4, 10.0], log | [1e-4, 10.0], log | [1e-4, 1.0], log | [1e-4, 10.0], log |
| num.impute.selected.cpo | {mean, hist, median} | hist |
| Variables | Canonical Domain | Human Reference | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
| residence_time | [0.5, 2.0] | [0.5, 2.0] | [0.0, 1440] | [1.0, 24.0] | ||
| ratio | [1.0, 5.0] | [1.0, 5.0] | ||||
| concentration | [0.1, 0.5] | [0.1, 0.5] | [0.01, 2.0], log | [0.01, 1.0], log | [0.05, 1.0], log | |
| temperature | [60.0, 140.0] | [60.0, 140.0] | [0.0, 250.0] | [20.0, 140.0] | [20, 150], step=1 | [20.0, 140.0] |
| Additional variables | ||||||
| solvent | {dmf, dmso, nmp, thf, acetonitrile} | {dmf, dmso, nmp} | {dimethylformamide, dimethylsulfoxide, acetonitrile} | {dmf, dmso, acetonitrile, nmp} | ||
| Variables | Canonical Domain | Human Reference | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
| width | [60, 120] | [60, 120] | ||||
| depth | [60, 120] | [60, 120] | ||||
| notch_width | [0, 30] | 15 | ||||
| notch_depth | [0, 30] | 15 | ||||
| thickness | [0.7, 3.0] | [1.1, 2.4] | [1.0, 3.0] | [1.0, 5.0] | [5e-4, 2e-2] | [0.8, 4.0], log |
| Additional variables | ||||||
| Variables | Canonical Domain | Human Reference | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
|---|---|---|---|---|---|---|
| position_to_force | [ 1000, 1000] | 0.0 | [ 2.0, 2.0] (x_weight) | [ 10.0, 10.0] (w_1) | [ 5.0, 5.0] (w_position) | |
| angle_to_force | [ 1000, 1000] | [ 150.0, 150.0] | [ 2.0, 2.0] (theta_weight) | [ 10.0, 10.0] (w_2) | [ 10.0, 10.0] (w_angle) | |
| velocity_to_force | [ 1000, 1000] | 0.0 | [ 2.0, 2.0] (x_dot_weight) | [ 10.0, 10.0] (w_3) | [ 5.0, 5.0] (w_velocity) | |
| angular_velocity_ to_force | [ 1000, 1000] | [ 30.0, 30.0] | [ 5.0, 5.0] (w_angular_velocity) | |||
| bias_to_force | [ 1000, 1000] | 0.0 | [ 10.0, 10.0] (w_0) [ 10.0, 10.0] (bias) | [ 2.0, 2.0] (bias) | ||
| Additional variables | ||||||
| Variables | Canonical Domain | Human Reference | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
|---|---|---|---|---|---|---|
| position_to_force | [ 1000, 1000] | 0.0 | [ 50.0, 50.0] | [ 100.0, 100.0] | [ 50.0, 50.0] | [ 300.0, 300.0] |
| angle_to_force | [ 1000, 1000] | [ 150.0, 150.0] | [0.0, 500.0] | [ 100.0, 100.0] | [ 50.0, 50.0] | [ 300.0, 300.0] |
| velocity_to_force | [ 1000, 1000] | 0.0 | [ 50.0, 50.0] | [ 100.0, 100.0] | [ 50.0, 50.0] | [ 300.0, 300.0] |
| angular_velocity_ to_force | [ 1000, 1000] | [ 30.0, 30.0] | [0.0, 100.0] | [ 100.0, 100.0] | [ 50.0, 50.0] | [ 300.0, 300.0] |
| bias_to_force | [ 1000, 1000] | 0.0 | [ 10.0, 10.0] | 0.0 | [ 10.0, 10.0] | [ 3.0, 3.0] |
| Best obj. ( ) | 577.4 370.1 | 1000.0 0.0 | 427.4 511.6 | 421.9 501.2 | 180.6 7.7 |
| Variables | Canonical Domain | Human Reference | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
| position_to_force | [-1e3, 1e3] | 0 | [-50, 0] | [0, 10] | ||
| sin_pole1_to_force | [-1e3, 1e3] | [-10, 10] | ||||
| sin_pole2_to_force | [-1e3, 1e3] | [-10, 10] | ||||
| cos_pole1_to_force | [-1e3, 1e3] | 0 | ||||
| cos_pole2_to_force | [-1e3, 1e3] | 0 | ||||
| velocity_to_force | [-1e3, 1e3] | 0 | [-20, 0] |
| Variables | Canonical Domain | Human Reference | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
|---|---|---|---|---|---|---|
| position_to_force | [-1e3, 1e3] | 0 | [-5e2, 5e2] | [-50, 50] | [-2e2, 2e2] | [-1e2, 1e2] |
| sin_pole1_to_force | [-1e3, 1e3] | [-10, 10] | [-1e3, 1e3] | [-50, 50] | [-2e2, 2e2] | [-1e2, 1e2] |
| sin_pole2_to_force | [-1e3, 1e3] | [-10, 10] | [-1e3, 1e3] | [-50, 50] | [-2e2, 2e2] | [-1e2, 1e2] |
| cos_pole1_to_force | [-1e3, 1e3] | 0 | [-5e2, 5e2] | 0 | [-2e2, 2e2] | [-1e2, 1e2] |
| cos_pole2_to_force | [-1e3, 1e3] | 0 | [-5e2, 5e2] | 0 | [-2e2, 2e2] | [-1e2, 1e2] |
| velocity_to_force | [-1e3, 1e3] | 0 | [-2e2, 2e2] | [-50, 50] | [-2e2, 2e2] | [-1e2, 1e2] |
| Domain | # Task | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
|---|---|---|---|---|---|
| YAHPO | 1 | 0.984 0.022 | 0.985 0.006 | 0.955 0.070 | 0.787 0.019 |
| 2 | 0.969 0.030 | 0.969 0.030 | 0.969 0.030 | 0.981 0.020 | |
| 3 | 0.998 0.003 | 0.815 0.032 | 1.000 0.000 | 1.000 0.000 | |
| 4 | 0.999 0.001 | 0.997 0.005 | 0.998 0.002 | 1.000 0.001 | |
| 5 | 0.988 0.000 | 0.623 0.146 | 0.962 0.026 | 0.784 0.147 | |
| 6 | 0.989 0.019 | 0.968 0.045 | 1.000 0.000 | 0.986 0.014 |
| Domain | # Task | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
|---|---|---|---|---|---|
| YAHPO | 1 | 0.867 0.057 | 0.417 0.014 | 0.503 0.139 | 0.742 0.049 |
| 2 | 0.919 0.045 | 0.996 0.004 | 0.646 0.051 | 0.272 0.000 | |
| 3 | 0.740 0.206 | 0.929 0.008 | 0.916 0.009 | 0.249 0.082 | |
| 4 | 0.966 0.017 | 1.000 0.000 | 0.990 0.016 | 0.996 0.001 | |
| 5 | 0.636 0.091 | 0.679 0.020 | 0.577 0.090 | 0.578 0.090 | |
| 6 | 0.638 0.004 | 0.623 0.009 | 0.619 0.007 | 0.721 0.079 |
| Domain | # Task | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
|---|---|---|---|---|---|
| YAHPO | 1 | 0.578 0.059 | 0.994 0.001 | 0.000 0.000 | 0.471 0.096 |
| 2 | 0.555 0.021 | 0.705 0.006 | 0.618 0.012 | 0.301 0.038 | |
| 3 | 0.910 0.018 | 0.375 0.016 | 0.553 0.002 | 0.344 0.032 | |
| 4 | 0.990 0.010 | 0.998 0.003 | 1.000 0.000 | 1.000 0.000 | |
| 5 | 0.697 0.004 | 0.661 0.107 | 0.353 0.219 | 0.703 0.003 | |
| 6 | 0.913 0.009 | 0.906 0.117 | 0.919 0.011 | 0.927 0.066 |
| Model | Task | Diff. | ||||
|---|---|---|---|---|---|---|
| Gemini 3 Flash | Task 3 | Easy | 0.998 | 0.007 | 0.002 | 0.002 |
| Hard | 0.910 | 0.082 | 0.013 | 0.090 | ||
| Task 14 | Easy | 1.000 | 0.000 | 0.000 | 0.000 | |
| Hard | 0.469 | 0.531 | 0.105 | 0.531 | ||
| Task 20 | Easy | 0.898 | 0.386 | 0.134 | 0.447 | |
| Hard | 0.719 | 0.206 | 0.266 | 0.267 |
| Difficulty | Task | Gemini 3 Flash | Gemini 3.1 Pro | GPT-5 mini | GPT-5.2 |
|---|---|---|---|---|---|
| Easy | Task 3 (YAHPO) | 0.999 0.001 | 0.938 0.107 | 1.000 0.000 | 1.000 0.000 |
| Task 14 (Olympus) | 1.000 0.000 | 1.000 0.000 | 0.993 0.008 | 1.000 0.000 | |
| Task 20 (MuJoCo) | 0.913 0.025 | 0.830 0.060 | 0.586 0.118 | 0.536 0.029 | |
| Intermediate | Task 3 (YAHPO) | 0.786 0.046 | 0.718 0.240 | 0.742 0.177 | 0.564 0.287 |
| Task 14 (Olympus) | 0.258 0.156 | 0.000 0.000 | 0.525 0.046 | 0.335 0.201 | |
| Task 20 (MuJoCo) | 0.755 0.117 | 0.672 0.064 | 0.497 0.170 | 0.529 0.179 |
| Difficulty | Out of bounds | Ambiguous mapping | Type mismatch | Total |
|---|---|---|---|---|
| Easy | 0.6 | 0.0 | 0.0 | 0.6 |
| Intermediate | 11.5 | 1.9 | 0.6 | 14.0 |
| Hard | 5.4 | 3.9 | 7.8 | 17.2 |
| Domain | Out of bounds | Ambiguous mapping | Type mismatch | Total |
|---|---|---|---|---|
| YAHPO | 1.5 | 2.2 | 0.0 | 3.7 |
| Olympus | 11.9 | 0.5 | 6.3 | 18.7 |
| MECHBench | 4.2 | 0.0 | 0.0 | 4.2 |
| MuJoCo | 0.0 | 4.2 | 0.0 | 4.2 |
| Difficulty | YAHPO | Olympus | MECHBench | MuJoCo | Total |
| Easy | 0 | 0 | 0 | 18 | 18 (1.1%) |
| Intermediate | 37 | 67 | 1 | 204 | 309 (21.3%) |
| Hard | 19 | 63 | 2 | 46 | 130 (9.8%) |
| Total | 56 (3.7%) | 130 (12.8%) | 3 (1.0%) | 268 (16.3%) | 457 (10.2%) |
| Task | Canonical variable | Designed variable | Setting | Result |
|---|---|---|---|---|
| 1 | max_units | hidden_units | Hard / Large / GPT-5.2 | Mapped |
| 3 | mtry.power | mtry_fraction | Hard / Large / GPT-5.2 | Unmapped |
| 9 | orange | orange_ratio | Hard / Large / GPT-5 mini | Mapped |
| Hard / Large / GPT-5.2 | ||||
| Hard / Small / Gemini 3 Flash | ||||
| dye_orange | Hard / Small / GPT-5 mini | Mapped |
Black-Box Optimization
Black-box optimization finds high-quality solutions when objectives can only be queried, not directly analyzed or differentiated.
See all 194 papers