FoMEMO: Towards Foundation Models for Expensive Multi-objective Optimization
Organizations: City University of Hong Kong · Xi’an Jiaotong University
Abstract
Expensive multi-objective optimization is a prevalent and crucial concern in many real-world scenarios, where sample-efficiency is vital due to the limited evaluations to recover the true Pareto front for decision making. Existing works either involve repeatedly fitting Gaussian process surrogates from scratch for each newly encountered problem, or rely on computationally demanding hypervolume-oriented policy learning for amortized optimization, making it challenging to achieve scalable pre-training and efficient yet robust adaptation for diverse emerging real-world applications. To address these challenges, we propose FoMEMO (Foundation Models for Expensive Multi-objective Optimization), which adopts a decomposition-based training paradigm that converts multi-objective optimization into preference-wise aggregated posterior learning tasks, enabling scalable pre-training on hundreds of millions of diverse synthetic datasets without relying on extensive real-world domain experiments. At test time, given observed trajectories from unseen problems and user preferences, the foundation model performs in-context posterior prediction and optimizes the derived acquisition functions for efficient candidate generation, achieving strong generalization and optimization performance without any subsequent model training or updates.
Figures & tables
| Algorithm | BC | ABD | Four Bar Truss | Gear Train | Materials | Mic |
| MOEA/D-EGO | 7.015e-01(7.5e-03) | 6.934e-01(1.1e-02) | 7.462e-01(1.4e-02) | 5.623e-01(1.3e-01) | 1.166e+00(2.2e-03) | 1.257e+00(4.9e-03) |
| TSEMO | 8.056e-01(9.1e-04) | 7.114e-01(9.4e-03) | 8.506e-01(4.7e-04) | 8.092e-01(2.3e-02) | 1.160e+00(5.1e-03) | 1.251e+00(4.8e-03) |
| USEMO-EI | 7.750e-01(1.2e-02) | 7.298e-01(1.1e-02) | 8.232e-01(4.3e-03) | 6.433e-01(4.0e-02) | 1.167e+00(2.8e-03) | 1.255e+00(5.4e-03) |
| DGEMO | 8.087e-01(1.3e-03) | 5.918e-01(6.8e-02) | 8.526e-01(9.8e-04) | 8.859e-01(3.2e-02) | 1.172e+00(3.2e-03) | 1.273e+00(2.1e-02) |
| qNEHVI | 7.962e-01(1.7e-03) | 7.181e-01(1.0e-02) | 8.244e-01(3.8e-03) | 8.387e-01(2.6e-02) | 1.144e+00(1.1e-02) | 1.198e+00(1.9e-02) |
| qParEGO | 7.843e-01(4.2e-03) | 7.032e-01(9.1e-03) | 7.917e-01(1.7e-02) | 7.813e-01(3.1e-02) | 1.150e+00(3.3e-03) | 1.191e+00(2.0e-02) |
| Problem | # Objectives | Tchebycheff (s) | Hypervolume (s) | Hypervolume / Tchebycheff |
|---|---|---|---|---|
| Four Bar Truss | 2 | 6.10e-05 | 4.14e-04 | 6.79 |
| Speed Reducer | 3 | 6.45e-05 | 2.55e-03 | 39.53 |
| Car Side Impact | 4 | 6.62e-05 | 1.23e-01 | 1853.56 |
| Water Resource Planning | 6 | 8.56e-05 | 3.97e+01 | 463594.84 |
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
| Type | Name | Value |
| Model Architecture | Embedding Size | 512 |
| Hidden Size | 1024 | |
| Attention Heads | 4 | |
| Transformer Layers | 12 | |
| Output Logits Size | 1000 | |
| Data Generation | GP Mean Prior | 0.0 |
| Type | Name | Feature Dimension ( ) | Objective Dimension ( ) |
| Synthetic Function | BC (Branin-Currin) | 2 | 2 |
| AR (Ackley-Rosenbrock) | 2 | 2 | |
| ABC (Ackley-Branin-Currin) | 2 | 3 | |
| ABD (Ackley-Branin-Dixon) | 2 | 3 | |
| ACD (Ackley-Currin-Dixon) | 2 | 3 | |
| BCD (Branin-Currin-Dixon) | 2 | 3 |
| Algorithm | BC | AR | ABC | ABD | ACD | BCD |
| MOEA/D-EGO | 7.015e-01(7.5e-03) | 1.203e+00(2.4e-03) | 4.826e-01(8.1e-03) | 6.934e-01(1.1e-02) | 9.320e-01(1.4e-02) | 5.735e-01(1.9e-02) |
| TSEMO | 8.056e-01(9.1e-04) | 1.172e+00(1.4e-02) | 4.853e-01(4.5e-03) | 7.114e-01(9.4e-03) | 9.117e-01(2.6e-02) | 6.938e-01(1.1e-02) |
| USEMO-EI | 7.750e-01(1.2e-02) | 1.118e+00(2.3e-02) | 4.648e-01(8.2e-03) | 7.298e-01(1.1e-02) | 8.991e-01(2.5e-02) | 6.766e-01(9.5e-03) |
| DGEMO | 8.087e-01(1.3e-03) | 1.158e+00(6.8e-02) | 4.477e-01(3.8e-02) | 5.918e-01(6.8e-02) | 7.266e-01(9.1e-02) | 7.054e-01(3.9e-03) |
| qNEHVI | 7.962e-01(1.7e-03) | 1.159e+00(1.8e-02) | 4.929e-01(6.5e-03) | 7.181e-01(1.0e-02) | 9.167e-01(1.3e-02) | 6.858e-01(1.5e-03) |
| qParEGO | 7.843e-01(4.2e-03) | 1.194e+00(4.4e-03) | 4.782e-01(7.5e-03) | 7.032e-01(9.1e-03) | 9.285e-01(1.3e-02) | 7.041e-01(4.5e-03) |
| Algorithm | Four Bar Truss | Pressure Vessel | Hatch Cover | Disc Brake | Speed Reducer | Gear Train | Rocket Injector |
| MOEA/D-EGO | 7.462e-01(1.4e-02) | 1.129e+00(8.8e-03) | 1.119e+00(2.3e-02) | 1.280e+00(1.4e-02) | 1.261e+00(1.3e-02) | 5.623e-01(1.3e-01) | 7.539e-01(1.1e-02) |
| TSEMO | 8.506e-01(4.7e-04) | 1.099e+00(1.6e-02) | 1.022e+00(3.5e-02) | 1.285e+00(3.0e-02) | 1.297e+00(1.3e-03) | 8.092e-01(2.3e-02) | 8.618e-01(7.5e-04) |
| USEMO-EI | 8.232e-01(4.3e-03) | 1.153e+00(1.6e-03) | 1.145e+00(6.2e-03) | 1.305e+00(2.3e-03) | 1.286e+00(3.7e-03) | 6.433e-01(4.0e-02) | 8.316e-01(3.7e-03) |
| DGEMO | 8.526e-01(9.8e-04) | 1.132e+00(6.3e-03) | 1.003e+00(7.9e-02) | 1.286e+00(2.9e-02) | 1.304e+00(1.6e-04) | 8.859e-01(3.2e-02) | 8.605e-01(1.9e-03) |
| qNEHVI | 8.244e-01(3.8e-03) | 1.069e+00(1.6e-02) | 1.119e+00(1.3e-02) | 1.192e+00(3.1e-02) | 1.263e+00(7.9e-03) | 8.387e-01(2.6e-02) | 8.182e-01(5.0e-03) |
| qParEGO | 7.917e-01(1.7e-02) | 1.099e+00(1.5e-02) | 1.144e+00(6.7e-03) | 1.199e+00(7.4e-02) | 1.215e+00(1.4e-02) | 7.813e-01(3.1e-02) | 6.700e-01(2.8e-02) |
| Algorithm | Lego | Materials | Mic | Ship |
| MOEA/D-EGO | 1.184e+00(1.1e-03) | 1.166e+00(2.2e-03) | 1.257e+00(4.9e-03) | 1.321e+00(1.5e-03) |
| TSEMO | 1.174e+00(3.3e-03) | 1.160e+00(5.1e-03) | 1.251e+00(4.8e-03) | 1.315e+00(2.1e-03) |
| USEMO-EI | 1.179e+00(1.6e-03) | 1.167e+00(2.8e-03) | 1.255e+00(5.4e-03) | 1.315e+00(3.6e-03) |
| DGEMO | 1.171e+00(5.0e-03) | 1.172e+00(3.2e-03) | 1.273e+00(2.1e-02) | 1.313e+00(3.6e-03) |
| qNEHVI | 1.147e+00(9.8e-03) | 1.144e+00(1.1e-02) | 1.198e+00(1.9e-02) | 1.293e+00(6.6e-03) |
| qParEGO | 1.137e+00(1.2e-02) | 1.150e+00(3.3e-03) | 1.191e+00(2.0e-02) | 1.291e+00(7.6e-03) |
| Algorithm | 2 Objectives | 3 Objectives | 4 Objectives | 6 Objectives |
| MOEA/D-EGO | 38.6409 | 46.8428 | 49.9341 | 54.1983 |
| TSEMO | 5.6616 | 6.1670 | 5.8793 | 6.3524 |
| USEMO-EI | 8.1582 | 6.6809 | 7.2214 | 6.7517 |
| DGEMO | 36.2580 | 95.3770 | N/A | N/A |
| qParEGO | 11.9262 | 19.0253 | 85.5916 | 185.4954 |
| qNEHVI | 10.1029 | 22.0746 | 62.9022 | 78.2812 |
| Model Configuration | Data Time | Step Time | Forward Time |
|---|---|---|---|
| Base | 0.0474 | 0.0323 | 0.0065 |
| Matérn-3/2 Kernel | 0.0491 | 0.0325 | 0.0066 |
| Matérn-5/2 Kernel | 0.0489 | 0.0326 | 0.0067 |
| Objective-Agnostic Head | 0.0518 | 0.0325 | 0.0100 |
| Smoothed Aggregation | 0.0588 | 0.0382 | 0.0067 |
| Model Configuration | Algorithm | BC | ABD | Four Bar Truss | Gear Train | Materials | Mic |
|---|---|---|---|---|---|---|---|
| Base | FoMEMO-EI | 8.066e-01 | 7.308e-01 | 8.667e-01 | 9.107e-01 | 1.173e+00 | 1.261e+00 |
| FoMEMO-UCB | 8.075e-01 | 7.369e-01 | 8.722e-01 | 9.270e-01 | 1.175e+00 | 1.262e+00 | |
| FoMEMO-UHVI | 8.052e-01 | 7.430e-01 | 8.609e-01 | 8.573e-01 | 1.176e+00 | 1.271e+00 | |
| FoMEMO-UR2I | 8.016e-01 | 7.126e-01 | 8.741e-01 | 8.961e-01 | 1.179e+00 | 1.274e+00 | |
| Matérn-3/2 Kernel | FoMEMO-EI | 8.071e-01 (+5.0e-04) | 7.354e-01 (+4.6e-03) | 8.676e-01 (+9.0e-04) | 9.134e-01 (+2.7e-03) | 1.175e+00 (+2.0e-03) | 1.263e+00 (+2.0e-03) |
| FoMEMO-UCB | 8.063e-01 (-1.2e-03) | 7.501e-01 (+1.3e-02) | 8.693e-01 (-2.9e-03) | 9.145e-01 (-1.3e-02) | 1.175e+00 (+0.0e+00) | 1.235e+00 (-2.7e-02) |
| Problem | Method | RMSE | MAE | NLL | Coverage@95% | MPIW | |
|---|---|---|---|---|---|---|---|
| Four Bar Truss (2 Objectives) | FoMEMO | 0.0085 | 0.0057 | 0.9925 | -3.5560 | 0.9486 | 0.0316 |
| PFNs | 0.0334 | 0.0262 | 0.7973 | -1.5384 | 1.0000 | 0.3174 | |
| GP-RBF | 0.0114 | 0.0076 | 0.9709 | -3.0026 | 0.7957 | 0.0241 | |
| GP-Matérn-3/2 | 0.0103 | 0.0065 | 0.9854 | -3.1493 | 0.9700 | 0.0385 | |
| GP-Matérn-5/2 | 0.0091 | 0.0059 | 0.9857 | -3.4093 | 0.9257 | 0.0277 | |
| Speed Reducer (3 Objectives) | FoMEMO | 0.0287 | 0.0202 | 0.9214 | -2.4440 | 0.9643 | 0.1115 |
| Method | BC | ABD | Four Bar Truss | Gear Train | Materials | Mic |
|---|---|---|---|---|---|---|
| PFNs4BO-UCB | 8.049e-1(1.4e-3) | 6.993e-1(1.2e-2) | 8.465e-1(4.3e-3) | 8.163e-1(4.8e-2) | 1.163e+0(7.0e-3) | 1.244e+0(1.3e-2) |
| PFNs4BO-UHVI | 7.734e-1(6.5e-3) | 5.644e-1(4.1e-2) | 8.023e-1(7.8e-3) | 6.612e-1(1.3e-1) | 1.149e+0(1.4e-2) | 1.217e+0(1.7e-2) |
| FoMEMO-UCB | 8.075e-1(5.8e-4) | 7.369e-1(8.5e-3) | 8.722e-1(1.2e-3) | 9.270e-1(4.5e-3) | 1.175e+0(1.9e-3) | 1.262e+0(1.8e-2) |
| FoMEMO-UHVI | 8.052e-1(4.9e-4) | 7.430e-1(8.4e-3) | 8.609e-1(1.3e-3) | 8.573e-1(9.6e-3) | 1.176e+0(2.4e-3) | 1.271e+0(1.6e-2) |
| Method | 2 Objectives | 3 Objectives | 4 Objectives | 6 Objectives |
|---|---|---|---|---|
| PFNs4BO-UCB | 3.2040 | 0.9760 | 3.0006 | 1.5506 |
| PFNs4BO-UHVI | 0.8773 | 1.9742 | 1.3186 | 0.8821 |
| FoMEMO-UCB | 1.7201 | 1.1666 | 1.1155 | 2.0801 |
| FoMEMO-UHVI | 0.6189 | 0.6094 | 0.6197 | 0.7432 |
| Algorithm | Car Side Impact (7d4m) | Conceptual Marine (6d4m) | Water Resource (3d6m) | WFG4 (30d3m) |
| MOEA/D-EGO | 7.039e-01(5.7e-03) | 5.915e-01(2.4e-02) | 1.412e+00(1.3e-02) | 2.754e-01(5.3e-03) |
| TSEMO | 6.942e-01(7.9e-03) | 7.358e-01(7.9e-03) | 1.198e+00(5.7e-02) | 2.869e-01(4.3e-03) |
| USEMO-EI | 7.151e-01(7.2e-03) | 6.750e-01(1.1e-02) | 1.433e+00(9.4e-03) | 2.842e-01(5.3e-03) |
| DGEMO | N/A | N/A | N/A | 2.792e-01(4.6e-03) |
| qNEHVI | 5.900e-01(7.0e-03) | 6.472e-01(2.2e-02) | 1.342e+00(4.8e-02) | 3.276e-01(2.2e-02) |
| qParEGO | 5.740e-01(2.3e-02) | 3.887e-01(2.6e-02) | 1.404e+00(3.7e-02) | 3.213e-01(1.5e-02) |
| Method | WFG4 (40d3m) | Car Cab (7d9m) | Gear Train (310 evals.) |
| MOEA/D-EGO | 2.879e-1(5.0e-3) | 1.575e-1(3.4e-3) | 6.162e-1(1.2e-1) |
| TSEMO | 2.959e-1(6.0e-3) | N/A | 8.619e-1(2.7e-2) |
| USeMO-EI | 2.984e-1(2.0e-3) | 1.466e-1(2.8e-3) | 7.962e-1(2.7e-2) |
| qParEGO | 3.164e-1(1.3e-2) | N/A | 8.488e-1(2.0e-2) |
| qNEHVI | 3.076e-1(7.1e-3) | N/A | 8.915e-1(1.1e-2) |
| JES | 3.155e-1(8.7e-3) | N/A | 6.878e-1(8.1e-2) |
| Method | WFG4 (40d3m) | Car Cab (7d9m) | Gear Train (310 evals.) |
| MOEA/D-EGO | 37.2543 | 51.6466 | 31.8981 |
| TSEMO | 7.1097 | N/A | 2.6140 |
| USeMO-EI | 7.5294 | 7.9049 | 3.0156 |
| qParEGO | 27.5029 | N/A | 721.5189 |
| qNEHVI | 15.0819 | N/A | 414.0605 |
| JES | 8.8853 | N/A | 212.3084 |
| Algorithm | BC | ABD | Four Bar Truss | Gear Train | Materials | Mic |
| MOEA/D-EGO | 7.052e-01(1.5e-02) | 6.227e-01(5.3e-02) | 7.658e-01(1.7e-02) | 5.365e-01(1.2e-01) | 1.164e+00(6.2e-03) | 1.247e+00(9.0e-03) |
| TSEMO | 8.081e-01(5.8e-04) | 7.117e-01(1.5e-02) | 8.520e-01(1.9e-03) | 7.811e-01(3.8e-02) | 1.155e+00(5.9e-03) | 1.238e+00(7.5e-03) |
| USEMO-EI | 7.882e-01(5.6e-03) | 7.057e-01(1.8e-02) | 8.287e-01(6.3e-03) | 6.742e-01(4.9e-02) | 1.150e+00(5.3e-03) | 1.243e+00(1.5e-02) |
| DGEMO | 8.068e-01(8.8e-04) | 5.838e-01(5.9e-02) | 8.549e-01(3.0e-03) | 8.413e-01(6.8e-02) | 1.162e+00(7.8e-03) | 1.272e+00(1.8e-02) |
| qNEHVI | 7.990e-01(1.5e-03) | 7.105e-01(1.1e-02) | 8.240e-01(5.8e-03) | 7.810e-01(7.5e-02) | 1.150e+00(7.9e-03) | 1.232e+00(2.1e-02) |
| qParEGO | 7.744e-01(3.3e-03) | 6.852e-01(8.5e-03) | 7.651e-01(1.8e-02) | 6.832e-01(7.2e-02) | 1.142e+00(1.6e-02) | 1.176e+00(2.1e-02) |
| Algorithm | BC | ABD | Four Bar Truss | Gear Train | Materials | Mic |
| MOEA/D-EGO | 7.221e-01(2.1e-02) | 5.363e-01(6.9e-02) | 7.872e-01(2.0e-02) | 3.939e-01(1.3e-01) | 1.154e+00(5.7e-03) | 1.238e+00(1.1e-02) |
| TSEMO | 8.074e-01(7.6e-04) | 6.512e-01(3.3e-02) | 8.521e-01(2.1e-03) | 7.060e-01(6.9e-02) | 1.145e+00(9.6e-03) | 1.242e+00(9.8e-03) |
| USEMO-EI | 7.708e-01(1.4e-02) | 6.633e-01(2.2e-02) | 8.267e-01(1.2e-02) | 6.155e-01(9.7e-02) | 1.143e+00(7.9e-03) | 1.231e+00(1.4e-02) |
| DGEMO | 8.063e-01(1.3e-03) | 5.228e-01(5.8e-02) | 8.713e-01(3.0e-03) | 8.221e-01(5.7e-02) | 1.149e+00(7.3e-03) | 1.242e+00(1.3e-02) |
| qNEHVI | 7.975e-01(2.4e-03) | 7.222e-01(7.5e-03) | 8.194e-01(3.9e-03) | 7.600e-01(8.7e-02) | 1.145e+00(9.2e-03) | 1.207e+00(2.1e-02) |
| qParEGO | 7.722e-01(7.8e-03) | 6.892e-01(1.2e-02) | 7.286e-01(2.3e-02) | 6.207e-01(1.0e-01) | 1.150e+00(1.1e-02) | 1.175e+00(1.4e-02) |
| Model Configuration | Algorithm | BC | ABD | Four Bar Truss | Gear Train | Materials | Mic |
|---|---|---|---|---|---|---|---|
| Base | FoMEMO-EI | 8.066e-01 | 7.308e-01 | 8.667e-01 | 9.107e-01 | 1.173e+00 | 1.261e+00 |
| FoMEMO-UCB | 8.075e-01 | 7.369e-01 | 8.722e-01 | 9.270e-01 | 1.175e+00 | 1.262e+00 | |
| FoMEMO-UHVI | 8.052e-01 | 7.430e-01 | 8.609e-01 | 8.573e-01 | 1.176e+00 | 1.271e+00 | |
| FoMEMO-UR2I | 8.016e-01 | 7.126e-01 | 8.741e-01 | 8.961e-01 | 1.179e+00 | 1.274e+00 | |
| Matérn-3/2 Kernel | FoMEMO-EI | 8.071e-01 (+5.0e-04) | 7.354e-01 (+4.6e-03) | 8.676e-01 (+9.0e-04) | 9.134e-01 (+2.7e-03) | 1.175e+00 (+2.0e-03) | 1.263e+00 (+2.0e-03) |
| FoMEMO-UCB | 8.063e-01 (-1.2e-03) | 7.501e-01 (+1.3e-02) | 8.693e-01 (-2.9e-03) | 9.145e-01 (-1.3e-02) | 1.175e+00 (+0.0e+00) | 1.235e+00 (-2.7e-02) |
| Function / Kernel | Weierstrass | Takagi | Matérn-3/2 | Matérn-5/2 | RBF |
|---|---|---|---|---|---|
| 0.325 | 1.0 | 1.5 | 2.5 |
| Kernel | RMSE ( ) | MAE ( ) | R2 ( ) |
|---|---|---|---|
| RBF | 1.237e-02 | 9.588e-03 | 9.792e-01 |
| Matérn-3/2 | 6.864e-03 | 5.125e-03 | 9.923e-01 |
| Matérn-5/2 | 8.469e-03 | 6.334e-03 | 9.907e-01 |
| Mixed | 7.265e-03 | 5.521e-03 | 9.917e-01 |
| Kernel | FoMEMO-EI | FoMEMO-UCB | FoMEMO-UHVI | FoMEMO-UR2I |
|---|---|---|---|---|
| RBF | 6.573e-01 | 6.469e-01 | 6.588e-01 | 6.323e-01 |
| Matérn-3/2 | 6.674e-01 (+1.0e-02) | 6.548e-01 (+7.9e-03) | 6.768e-01 (+1.8e-02) | 6.599e-01 (+2.8e-02) |
| Matérn-5/2 | 6.633e-01 (+6.0e-03) | 6.617e-01 (+1.5e-02) | 6.787e-01 (+2.0e-02) | 6.723e-01 (+4.0e-02) |
| Mixed | 6.564e-01 (-9.0e-04) | 6.570e-01 (+1.0e-02) | 6.654e-01 (+6.6e-03) | 6.551e-01 (+2.3e-02) |
Multi-Objective Optimization
Multi-objective optimization studies how to find and choose solutions that balance competing goals, constraints, and preferences.
See all 283 papers