ParetoM3: Learning on the Pareto Set under Preference Guidance via Min-Max-Min Optimization
Authors: Pei Tang, Songtao Lu
Organizations: Department of Computer Science and Engineering, Shun Hing Institute of Advanced Engineering, The Chinese University of Hong Kong, Hong Kong SAR
Learning across multiple inherently conflicting objectives requires selecting desirable trade-offs from the Pareto set. In this work, we study preference-guided multi-objective optimization, where a predefined preference function is minimized over the weak Pareto set of multiple nonconvex objectives. Leveraging a merit function that vanishes on the weak Pareto set, we propose a penalty-based min-max-min formulation, termed the M3 problem, which unifies preference optimization and the enforcement of weak Pareto optimality in a single objective. We further develop ParetoM3, a single-loop first-order algorithm for computing stationary solutions of the resulting M3 problem. Under a local Kurdyka-Lojasiewicz assumption, we prove that ParetoM3 finds an ε-stationary point of the M3 problem that is also approximately weakly Pareto optimal for the original multi-objective problem when the penalty parameter is sufficiently large, with explicit non-asymptotic convergence rates. We also show that this stationarity notion recovers existing optimality criteria, including approximate preference stationarity and approximate Karush-Kuhn-Tucker conditions, under the same assumptions used in prior works. Experiments on a synthetic benchmark, image classification, regression, and math reasoning demonstrate that ParetoM3 reliably attains preferred weak Pareto optimal solutions while achieving strong practical performance. Code is available at https://github.com/tpgoodboy/ParetoM3.
Figures & tables
Methods
# of Loops
Oracle
LL Assump.
UL Optim.
Convergence Rate
EPO [ 50 ]
double
1st-order
NC
✗
-
PNG [ 75 ]
double
1st-order
NC
✗
O(ϵ−2)
FERERO [ 8 ]
single
1st-order
PŁ
✗
O(ϵ−2)
PMM [ 60 ]
double
2nd-order
SC
✓
O(ϵ−1)
FOOPS [ 9 ]
double
1st-order
KŁ
✓
-
ParetoM 3
single
1st-order
KŁ
✓
O((ρϵ−1)max{4θ,2})
Table 1: Comparison of representative preference-guided MOO algorithms. “Oracle” indicates the required derivative information of the loss functions; “LL Assump.” denotes the structural assumption on the LL objectives; “UL Optim.” indicates whether the obtained point satisfies the necessary optimality conditions for the UL objective; and “Convergence Rate” reports the iteration complexity for reaching the stated solution of problem ( 1 ). Here, “NC” and “SC” stand for nonconvex and strongly convex, respectively. Moreover, “-” indicates that no convergence rate is provided, θ∈(0,1) is the exponent in the KŁ assumption, and ρ is the penalty parameter.
Figure 1: Comparison of preference-guided algorithms on the synthetic benchmark. The top row shows the trajectories in the objective space, and the bottom row reports the norm-normalized distance to the preference-optimal objective values. Different colors correspond to different preference vectors, and each column corresponds to one algorithm.
Figure 2: Comparison on three two-task image-classification benchmarks: MultiMNIST (left), MultiFashion (middle), and Multi-Fashion+MNIST (right). The top row shows the final training losses of the two tasks, and the bottom row reports the average accuracy curves. Here, colors indicate different algorithms, while line styles and markers indicate different preference vectors. The gray lines in the top row mark the single-task training losses.
Figure 3: Comparison of RLP performance achieved by the compared algorithms on the River Flow dataset. Error bars denote standard deviations over different preferences and runs.
GSM8K
MathQA
Average
Baseline
60.24±0.62
37.63±1.10
-
LS
58.85±1.24
34.89±0.50
46.87±0.62
EPO
58.05±1.10
35.50±0.51
46.78±0.57
FERERO
58.60±1.36
34.53±1.01
46.57±0.61
ParetoM 3
60.42±0.93
36.23±1.06
48.33±0.92
Table 2: Accuracy (%) comparison on math reasoning datasets. The “Baseline” column reports single-dataset fine-tuning results, while the other methods jointly fine-tune on both datasets.
Table 5: Summary of constants in the error bounds w.r.t. variables.
Dataset
Metric
LS
EPO
FERERO
PNG
FOOPS
ParetoM 3
Forward/backward per batch
1/1
2/2
2/2
2/2
2K+2/K+1
2/2
Synthetic
Total running time (seconds)
0.07
0.71
1.46
0.65
1.53(K=50)
0.97
MNIST
Epoch time (seconds)
1.10
2.57
4.09
-
-
2.60
GPU memory (MB)
784
786
808
-
-
824
Math
Epoch time (minutes)
2.14
3.09
4.64
-
-
3.57
GPU memory (MB)
74360
75116
75384
-
-
89092
Appendix
Table 6: Summary of computation cost. ( K is the number of inner loops.)
m
4
8
12
16
20
24
28
32
36
40
LS
0.66
1.48
1.60
1.61
1.57
1.76
1.66
1.68
1.91
1.86
EPO
0.48
1.17
1.34
1.29
1.30
1.48
1.42
1.44
1.69
1.67
FERERO
1.42
1.98
2.00
1.90
1.77
1.95
1.81
1.81
2.04
1.94
ParetoM 3
0.57
1.23
1.20
1.07
1.09
1.27
1.22
1.22
1.47
1.45
Appendix
Table 7: Non-uniformity ↓(×10−1) of different algorithms.
Dataset
Objective
LS
EPO
FERERO
ParetoM 3
Mt-M
Loss
0.96
1.92
1.01
2.52
Mt-M
Accuracy
0.06
0.07
0.07
0.13
Mt-F
Loss
8.14
8.87
7.78
11.13
Mt-F
Accuracy
0.90
0.88
0.82
1.13
Mt-F+M
Loss
3.42
5.42
3.65
6.34
Mt-F+M
Accuracy
0.33
0.41
0.36
0.54
Appendix
Table 8: Hypervolumes ↑(×10−2) of different algorithms on multi-class classification with respect to Nadir points in Table 9 .
Dataset
Objective
Nadir points
Mt-M
Loss
[0.396,0.390]
Mt-M
Accuracy
[0.865,0.865]
Mt-F
Loss
[0.756,0.780]
Mt-F
Accuracy
[0.705,0.704]
Mt-F+M
Loss
[0.555,0.519]
Mt-F+M
Accuracy
[0.817,0.796]
Appendix
Table 9: Nadir points for computing hypervolumes.
Parameter value
Mt-M
Mt-F
Mt-F+M
Loss
Accuracy
Loss
Accuracy
Loss
Accuracy
ρ=0.5
2.68
0.08
12.37
1.14
6.96
0.42
ρ=2.0
2.52
0.13
11.13
1.13
6.34
0.54
ρ=5.0
1.87
0.13
9.90
1.12
5.21
0.50
ρ=10.0
1.53
0.13
9.04
1.03
4.44
0.46
px,py=10−5
2.53
0.12
10.99
1.14
6.30
0.52
Appendix
Table 10: Hypervolumes ↑(×10−2) under different hyperparameter settings.
Optimizer
Mt-M
Mt-F
Mt-F+M
Loss
Accuracy
Loss
Accuracy
Loss
Accuracy
AdamW
3.94
0.09
17.79
1.14
9.42
0.39
Smoothing
2.52
0.13
11.13
1.13
6.34
0.54
Appendix
Table 11: Hypervolumes ↑(×10−2) under different optimizers.
Dataset
Objective
LS
EPO
FERERO
FOOPS ⋆
ParetoM 3
Mt-M
Loss
2.97±0.37
4.60±0.70
2.84±0.53
2.62±0.21
6.26±0.39
Mt-M
Accuracy
0.24±0.03
0.30±0.03
0.23±0.04
0.33±0.02
0.41±0.02
Mt-F
Loss
9.85±1.04
10.88±1.09
9.21±1.30
8.32±0.37
15.01±0.92
Mt-F
Accuracy
0.99±0.09
0.99±0.09
0.90±0.13
1.22±0.07
1.44±0.06
Mt-F+M
Loss
5.72±0.71
7.80±1.19
5.47±0.96
4.80±0.45
10.31±0.39
Mt-F+M
Accuracy
0.50±0.06
0.60±0.08
0.48±0.10
0.72±0.06
0.85±0.07
Appendix
Table 12: Hypervolumes ↑(×10−2) of different algorithms on multi-class classification with respect to Nadir points from [ 9 ] (results marked with ⋆ are directly taken from [ 9 ] ).
Epoch
Method
GSM8K
MathQA
Average
1
LS
59.67
28.94
44.31
EPO
34.42
29.58
32.00
FERERO
58.98
29.48
44.23
ParetoM 3
60.35
29.48
44.92
5
LS
60.35
32.36
46.36
EPO
51.93
34.10
43.02
Appendix
Table 13: Test accuracy (%) on reasoning datasets during training.
rMathQA/rGSM8K
GSM8K
MathQA
Average
1
61.11
36.92
49.02
1.2
60.80
37.32
49.06
Appendix
Table 14: Test accuracy (%) on reasoning datasets with different preferences.
Multi-objective optimization involves diverse preferences over conflicting objectives. Rather than returning a finite set of trade-off solutions, preference-conditioned Pareto set learning seeks to map a specified preference directly to a corresponding Pareto optimal solution. Yet in many real-world scenarios, optimization must proceed offline without further objective evaluations, leaving only a limited static dataset that sparsely covers the decision and objective spaces. Existing methods either optimize separately for each preference or train a problem-specific Pareto solution model, requiring repeated search across preferences or repeated training across problems. We introduce in-context preference-conditioned Pareto set learning (ICPSL), a framework that amortizes both preference-wise solution search and problem-specific model construction. ICPSL pretrains a foundation-style hypernetwork on a large and diverse set of synthetic prior data. Given the offline observations of an unseen problem, the pretrained hypernetwork learns to generate a problem-specific, preference-conditioned Pareto solution model in a single forward pass. The generated model can then be reused to predict solutions for arbitrary preferences without problem-specific training or preference-wise search. Experiments on synthetic and real-world problems show that ICPSL produces high-quality Pareto solutions while providing substantial speedups in problem-specific solution-model construction.
Preference-conditioned multi-objective reinforcement learning aims to learn a single policy that captures trade-offs across preferences, but under nonlinear scalarization the uniqueness and continuity of the preference-to-solution correspondence remain unclear. We study this problem in tabular multi-objective Markov decision processes (MDPs) using smooth Tchebycheff scalarization as a monotone utility. Under mild interior conditions on the preference set, we prove that each preference induces a unique Pareto-optimal return vector and that this vector depends Lipschitz-continuously on the preference, providing a principled foundation for preference sweeping toward dense Pareto-front coverage. To compute these targets, we formulate the problem over occupancy measures and derive Concave Mirror Descent Policy Iteration (CMDPI), which achieves an O(1/k) objective-suboptimality rate. We further show that each update is equivalent to solving a Kullback-Leibler-regularized MDP with the previous policy as reference, yielding a policy-iteration interpretation and finite-iterate policy continuity across preferences. We instantiate the update as a deep actor-critic algorithm preserving previous-policy regularization. On eight MO-Gymnasium tasks, it achieves the best average hypervolume rank among recent baselines and strong expected-utility performance. Continuous-control experiments indicate gains beyond the discrete-action setting.
Akihiro Kubo, Kosuke Nakanishi, Shin Ishii
International Research Center for Neurointelligence, The University of Tokyo, Tokyo, Japan · ATR Neural Information Analysis Laboratories, Kyoto, Japan · Department of Information Science, Kyoto University, Kyoto, Japan
Constructing a finite Pareto representation requires allocating a limited number of solutions across the Pareto front (PF). A desired preference distribution specifies this allocation by assigning more solutions to some PF regions and fewer to others. Scalarization generates Pareto solutions by varying objective weights and solving single-objective problems. However, a distribution of scalarization weights generally does not translate into the same distribution of Pareto solutions along the PF, because the weight-to-solution map is nonlinear and unknown in advance.To account for this distortion, we introduce SURFACE (Scalarization Updates for Redistributing Finite Approximations with Chart Estimation}), which adaptively updates scalarization weights to match a desired PF distribution. SURFACE estimates how each region of the weight grid corresponds to an area of the PF from the current scalarized solutions, then redistributes the weights accordingly. The method supports parallel, warm-started, inexact optimization with standard single-objective solvers. Theoretically, we establish that, under the stated regularity conditions, SURFACE achieves linear convergence of the preference distribution mismatch to a finite-resolution error floor of O(N−2/(M−1)), where N is the number of adaptive weights and M is the number of objectives. Empirically, experiments across multiple benchmarks demonstrate improved target-mass balance and PF coverage compared with established baselines.