ParetoM3: Learning on the Pareto Set under Preference Guidance via Min-Max-Min Optimization
Authors: Pei Tang, Songtao Lu
Organizations: Department of Computer Science and Engineering, Shun Hing Institute of Advanced Engineering, The Chinese University of Hong Kong, Hong Kong SAR
Learning across multiple inherently conflicting objectives requires selecting desirable trade-offs from the Pareto set. In this work, we study preference-guided multi-objective optimization, where a predefined preference function is minimized over the weak Pareto set of multiple nonconvex objectives. Leveraging a merit function that vanishes on the weak Pareto set, we propose a penalty-based min-max-min formulation, termed the M3 problem, which unifies preference optimization and the enforcement of weak Pareto optimality in a single objective. We further develop ParetoM3, a single-loop first-order algorithm for computing stationary solutions of the resulting M3 problem. Under a local Kurdyka-Lojasiewicz assumption, we prove that ParetoM3 finds an ε-stationary point of the M3 problem that is also approximately weakly Pareto optimal for the original multi-objective problem when the penalty parameter is sufficiently large, with explicit non-asymptotic convergence rates. We also show that this stationarity notion recovers existing optimality criteria, including approximate preference stationarity and approximate Karush-Kuhn-Tucker conditions, under the same assumptions used in prior works. Experiments on a synthetic benchmark, image classification, regression, and math reasoning demonstrate that ParetoM3 reliably attains preferred weak Pareto optimal solutions while achieving strong practical performance. Code is available at https://github.com/tpgoodboy/ParetoM3.
Figures & tables
Methods
# of Loops
Oracle
LL Assump.
UL Optim.
Convergence Rate
EPO [ 50 ]
double
1st-order
NC
✗
-
PNG [ 75 ]
double
1st-order
NC
✗
O(ϵ−2)
FERERO [ 8 ]
single
1st-order
PŁ
✗
O(ϵ−2)
PMM [ 60 ]
double
2nd-order
SC
✓
O(ϵ−1)
FOOPS [ 9 ]
double
1st-order
KŁ
✓
-
ParetoM 3
single
1st-order
KŁ
✓
O((ρϵ−1)max{4θ,2})
Table 1: Comparison of representative preference-guided MOO algorithms. “Oracle” indicates the required derivative information of the loss functions; “LL Assump.” denotes the structural assumption on the LL objectives; “UL Optim.” indicates whether the obtained point satisfies the necessary optimality conditions for the UL objective; and “Convergence Rate” reports the iteration complexity for reaching the stated solution of problem ( 1 ). Here, “NC” and “SC” stand for nonconvex and strongly convex, respectively. Moreover, “-” indicates that no convergence rate is provided, θ∈(0,1) is the exponent in the KŁ assumption, and ρ is the penalty parameter.
Figure 1: Comparison of preference-guided algorithms on the synthetic benchmark. The top row shows the trajectories in the objective space, and the bottom row reports the norm-normalized distance to the preference-optimal objective values. Different colors correspond to different preference vectors, and each column corresponds to one algorithm.
Figure 2: Comparison on three two-task image-classification benchmarks: MultiMNIST (left), MultiFashion (middle), and Multi-Fashion+MNIST (right). The top row shows the final training losses of the two tasks, and the bottom row reports the average accuracy curves. Here, colors indicate different algorithms, while line styles and markers indicate different preference vectors. The gray lines in the top row mark the single-task training losses.
Figure 3: Comparison of RLP performance achieved by the compared algorithms on the River Flow dataset. Error bars denote standard deviations over different preferences and runs.
GSM8K
MathQA
Average
Baseline
60.24±0.62
37.63±1.10
-
LS
58.85±1.24
34.89±0.50
46.87±0.62
EPO
58.05±1.10
35.50±0.51
46.78±0.57
FERERO
58.60±1.36
34.53±1.01
46.57±0.61
ParetoM 3
60.42±0.93
36.23±1.06
48.33±0.92
Table 2: Accuracy (%) comparison on math reasoning datasets. The “Baseline” column reports single-dataset fine-tuning results, while the other methods jointly fine-tune on both datasets.
Table 5: Summary of constants in the error bounds w.r.t. variables.
Dataset
Metric
LS
EPO
FERERO
PNG
FOOPS
ParetoM 3
Forward/backward per batch
1/1
2/2
2/2
2/2
2K+2/K+1
2/2
Synthetic
Total running time (seconds)
0.07
0.71
1.46
0.65
1.53(K=50)
0.97
MNIST
Epoch time (seconds)
1.10
2.57
4.09
-
-
2.60
GPU memory (MB)
784
786
808
-
-
824
Math
Epoch time (minutes)
2.14
3.09
4.64
-
-
3.57
GPU memory (MB)
74360
75116
75384
-
-
89092
Appendix
Table 6: Summary of computation cost. ( K is the number of inner loops.)
m
4
8
12
16
20
24
28
32
36
40
LS
0.66
1.48
1.60
1.61
1.57
1.76
1.66
1.68
1.91
1.86
EPO
0.48
1.17
1.34
1.29
1.30
1.48
1.42
1.44
1.69
1.67
FERERO
1.42
1.98
2.00
1.90
1.77
1.95
1.81
1.81
2.04
1.94
ParetoM 3
0.57
1.23
1.20
1.07
1.09
1.27
1.22
1.22
1.47
1.45
Appendix
Table 7: Non-uniformity ↓(×10−1) of different algorithms.
Dataset
Objective
LS
EPO
FERERO
ParetoM 3
Mt-M
Loss
0.96
1.92
1.01
2.52
Mt-M
Accuracy
0.06
0.07
0.07
0.13
Mt-F
Loss
8.14
8.87
7.78
11.13
Mt-F
Accuracy
0.90
0.88
0.82
1.13
Mt-F+M
Loss
3.42
5.42
3.65
6.34
Mt-F+M
Accuracy
0.33
0.41
0.36
0.54
Appendix
Table 8: Hypervolumes ↑(×10−2) of different algorithms on multi-class classification with respect to Nadir points in Table 9 .
Dataset
Objective
Nadir points
Mt-M
Loss
[0.396,0.390]
Mt-M
Accuracy
[0.865,0.865]
Mt-F
Loss
[0.756,0.780]
Mt-F
Accuracy
[0.705,0.704]
Mt-F+M
Loss
[0.555,0.519]
Mt-F+M
Accuracy
[0.817,0.796]
Appendix
Table 9: Nadir points for computing hypervolumes.
Parameter value
Mt-M
Mt-F
Mt-F+M
Loss
Accuracy
Loss
Accuracy
Loss
Accuracy
ρ=0.5
2.68
0.08
12.37
1.14
6.96
0.42
ρ=2.0
2.52
0.13
11.13
1.13
6.34
0.54
ρ=5.0
1.87
0.13
9.90
1.12
5.21
0.50
ρ=10.0
1.53
0.13
9.04
1.03
4.44
0.46
px,py=10−5
2.53
0.12
10.99
1.14
6.30
0.52
Appendix
Table 10: Hypervolumes ↑(×10−2) under different hyperparameter settings.
Optimizer
Mt-M
Mt-F
Mt-F+M
Loss
Accuracy
Loss
Accuracy
Loss
Accuracy
AdamW
3.94
0.09
17.79
1.14
9.42
0.39
Smoothing
2.52
0.13
11.13
1.13
6.34
0.54
Appendix
Table 11: Hypervolumes ↑(×10−2) under different optimizers.
Dataset
Objective
LS
EPO
FERERO
FOOPS ⋆
ParetoM 3
Mt-M
Loss
2.97±0.37
4.60±0.70
2.84±0.53
2.62±0.21
6.26±0.39
Mt-M
Accuracy
0.24±0.03
0.30±0.03
0.23±0.04
0.33±0.02
0.41±0.02
Mt-F
Loss
9.85±1.04
10.88±1.09
9.21±1.30
8.32±0.37
15.01±0.92
Mt-F
Accuracy
0.99±0.09
0.99±0.09
0.90±0.13
1.22±0.07
1.44±0.06
Mt-F+M
Loss
5.72±0.71
7.80±1.19
5.47±0.96
4.80±0.45
10.31±0.39
Mt-F+M
Accuracy
0.50±0.06
0.60±0.08
0.48±0.10
0.72±0.06
0.85±0.07
Appendix
Table 12: Hypervolumes ↑(×10−2) of different algorithms on multi-class classification with respect to Nadir points from [ 9 ] (results marked with ⋆ are directly taken from [ 9 ] ).
Epoch
Method
GSM8K
MathQA
Average
1
LS
59.67
28.94
44.31
EPO
34.42
29.58
32.00
FERERO
58.98
29.48
44.23
ParetoM 3
60.35
29.48
44.92
5
LS
60.35
32.36
46.36
EPO
51.93
34.10
43.02
Appendix
Table 13: Test accuracy (%) on reasoning datasets during training.
rMathQA/rGSM8K
GSM8K
MathQA
Average
1
61.11
36.92
49.02
1.2
60.80
37.32
49.06
Appendix
Table 14: Test accuracy (%) on reasoning datasets with different preferences.
International Research Center for Neurointelligence, The University of Tokyo, Tokyo, Japan · ATR Neural Information Analysis Laboratories, Kyoto, Japan · Department of Information Science, Kyoto University, Kyoto, Japan