Organizations: Laboratory for Applied Mechanical Design Ecole Polytechnique Fédérale de Lausanne (EPFL) CH-1015 Lausanne, Switzerland · Department of Mechanical and Process Engineering ETH Zürich 8092 Zurich, Switzerland
Many engineering design problems involve multiple conflicting objectives and strict feasibility constraints. They are typically solved with population-based optimizers that require many evaluations and offer limited online control over where the search is focused. We present MOON-RL, a reinforcement-learning-based optimization framework that formulates constrained multi-objective search as preference-directed navigation in the objective space. In MOON-RL, a lightweight PPO policy receives objective-space and constraint-related observations and acts as an instance-specific adaptive search process. It learns how to update the design so that the current solution advances along a user-specified trade-off direction, while a simple meta-algorithm periodically resets the policy, resamples starting designs from a continuously updated archive of non-dominated solutions, and adjusts step-size and penalty schedules. We evaluate MOON-RL on real-world actuator-design problems and additional convergence-hard constrained benchmarks, comparing against established evolutionary algorithms under matched evaluation budgets. In focused search scenarios, MOON-RL achieves competitive performance in the region of interest, and ablation studies show that both the RL-driven adaptive sampling process and the meta-algorithm contribute to this behaviour, suggesting that repurposing RL training as objective-space navigation is a viable building block for interactive, preference-guided design tools. Additional experiments show that the framework also accommodates adaptive non-RL samplers, while PPO remains a strong choice for fixed-target, preference-directed objective-space navigation.
Figures & tables
Figure 1: T -step rollouts for an untrained (a) and trained (b) agent in a 2-objective max-max problem. The trained agent drives the solution efficiently towards the Pareto Front along a reference direction defined by a target point in the objective space
Figure 2: (a) monitored geometric metrics in the objective space, and (b) visualization of the reward function (without constraint penalty) for a two-objective optimization.
Figure 3: Overview of the MOON-RL framework and data flow. The framework starts from the user-defined inputs, including the runtime parameters, initial design x0 , and target point ztarget . The policy πθ produces an action at , which is used to generate a new candidate design xt+1 . The candidate design is evaluated through the multi-objective evaluation functions, producing the objective values ft+1 and constraint violations CVt+1 . These evaluated quantities are used to update the non-dominated archive P , construct the observation and reward, and analyze the position of the candidate design in objective space relative to the target direction. The meta-algorithm updates runtime parameters, such as scaling factors and constraint violation coefficients, and selects new starting designs during restarts. Arrow colors distinguish initial inputs, MOON-RL internal data, problem-specific data, updated parameters, and archive output.
Figure 4: Analysis region (shaded red) example for CS1: only points B, C, and D are chosen for analysis.
Figure 5: “CS1” problem: EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles.
Figure 6: “CT1” problem: EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles.
Figure 7: “CTS1” problem: Comparison of median EAFs for MOON-RL against (a) NSGA-II (pop. 200), (b) R-NSGA-II ( ε=0.01 ), (c) APSEA, and (d) CMODRL. The target point is shown as a yellow cross. Grey points depict the baseline algorithm EAF, while colored points correspond to the MOON-RL EAF. The color encodes the signed additive ε -indicator: red regions indicate that MOON-RL outperforms the baseline, blue regions that the baseline outperforms MOON-RL, and white denotes parity. (e) HV progression including 5th and 95th percentiles.
Figure 8: “DAS-CMOP 3-11” problem: EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles.
Figure 9: “CS1” problem (global search): EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles.
Algorithm
HV ↑
GD ↓
Eval @ 50% HV ↓
Eval @ 75% HV ↓
CS1
MOON-RL
52.0
0.1
4880(30/30)
16440(30/30)
APSEA
40.2
1.1
123194(30/30)
135346(12/30)
LCMEA
54.6
0.05
7800(29/30)
30800(27/30)
CMODRL
52.5
0.16
158325(29/30)
183600(29/30)
NSGA-II (pop. 200)
41.0
1.1
7600(30/30)
159800(13/30)
Table 1: Results summary for main results. Success rate in parentheses shows how many runs reached the corresponding HV target. ↑ means higher is better, ↓ means lower is better. Only median results for the best performing algorithms are shown. Full tables with more algorithms and statistical results are available in Appendices J , K , L , M , and N .
Figure 10: EAF and HV progression of MOON-RL run with and without meta-algorithm and with and without training the policy network. Panels (a, b) show runs with the meta-algorithm active, panels (c, d) runs without it. Focused search scenario.
Figure 11: EAF and HV progression of MOON-RL run with and without meta-algorithm and with and without training the policy network. Panels (a, b) show runs with the meta-algorithm active, panels (c, d) runs without it. Global search scenario.
Figure 12: “CS1” problem: EAF (a) and HV progression (b) of MOON-RL with different adaptive samplers, including 5th and 95th percentiles.
Figure 13: “DAS-CMOP 3-11” problem: EAF (a) and HV progression (b) of MOON-RL with different adaptive samplers, including 5th and 95th percentiles.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
0
9
55
159
248
921
1118
217
2674
2957
2968
3008
3376
3750
4352
4779
4844
5319
5520
5806
5859
6215
7106
8523
8642
8694
9213
9256
9348
9766
Appendix
Table 2: Random seeds used in all experiments
Problem
Objective 1
Objective 2
Objective 3
CS1
0.1
20
–
CT1
0.1
0.2
–
CTS1
0.1
0.2
20
DAS-CMOP3-11
0.1
0.1
–
Appendix
Table 3: Objective scaling factors used for MOON-RL and for hypervolume computation. A dash indicates that no third objective is defined for the problem.
Parameter
Value / Description
Policy architecture
Separate networks for actor and critic
Actor architecture
Separate networks for means and standard deviation
Action distribution
Independent Gaussian per action dimension
Hidden layers
2 per network (all networks)
Hidden layer sizes
64 units
Hidden activation
ReLU
Appendix
Table 4: PPO agent hyperparameters used in all experiments.
Parameter
Value / Description
Trajectory length T
8 steps
Parallel trajectories per episode N
10
Candidate evaluations per episode
N×T=80
Policy update interval
After every episode
Number of episodes
3750
Candidate evaluations from rollouts
300,000
Appendix
Table 5: PPO rollout and update settings used in the experiments.
Parameter
Schedule Type
Value
Initial scaling factor κ0
Linear decay
7% of design-variable ranges
Target scaling factor κmin
1%
Decay interval Sκ
Every 100 episodes
Decay rate δκ
1% of design-variable ranges (subtracted per interval)
Minimum categorical-variable scaling
Fixed floor
0.6 for MODAct variables
Initial constraint penalty λ0
Linear increase
2
Appendix
Table 6: Runtime scheduler parameters used to adjust scaling factors κ and constraint penalties λ .
Algorithm
Parameters
NSGA-II
Population size: 200 or 100
R-NSGA-II
Population size: 200 Reference point(s): Same as for MOON-RL, depending global or local search ε : 0.01 or 0.001
MOEA/D
Reference directions: “Das-Dennis”, 2 objectives, 199 partitions Number of neighbors: 200 Probability of neighbor mating: 0.7
NSGA-III
Population size: 200 Reference directions: “Das-Dennis”, 2 objectives, 14 or 199 partitions
R-NSGA-III
Population size: 200 Reference point(s): Same as for MOON-RL, depending global or local search μ : 0.1 or 0.01
C-TAEA
Reference directions: “Das-Dennis”, 2 objectives, 14 or 199 partitions
Appendix
Table 7: Benchmark Algorithm Parameters Used in Experiments
Figure 14: EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles. Target point: (0.22,51.2) .
Figure 15: EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles. Target point: (0.39,138.) .
Figure 16: EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles. Target point: (0.58,172.) .
Figure 17: EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles. Target point: (0.85,182.8) .
Figure 18: EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles. Target point: (0.4,1)
Figure 19: EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles. Target point: (0.25,0.4) .
Figure 20: EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles. Target point: (0.53,1.5) .
Figure 21: EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles. Target point: (0.8,1.8)
Figure 22: Pairwise projections of median EAF for different algorithms: (a) NSGA-II (pop. 200 ), (b) R-NSGA-II ( ε=0.01 ), (c) NSGA-III (200 ref. dir.) and (d) R-NSGA-III ( μ=0.01 ). Target point is shown as yellow cross. Target point: (0.3,0.0,100.) .
Figure 23: Pairwise projections of median EAF for different algorithms: (a) NSGA-II (pop. 200 ), (b) R-NSGA-II ( ε=0.01 ), (c) NSGA-III (200 ref. dir.) and (d) R-NSGA-III ( μ=0.01 ). Target point is shown as yellow cross. Target point: (0.9,1.1,80.) .
Figure 24: Pairwise projections of median EAF for different algorithms: (a) NSGA-II (pop. 200 ), (b) R-NSGA-II ( ε=0.01 ), (c) NSGA-III (200 ref. dir.) and (d) R-NSGA-III ( μ=0.01 ). Target point is shown as yellow cross. Target point: (0.95,−0.35,480.) .
Figure 25: Pairwise projections of median EAF for different algorithms: (a) NSGA-II (pop. 200 ), (b) R-NSGA-II ( ε=0.01 ), (c) NSGA-III (200 ref. dir.) and (d) R-NSGA-III ( μ=0.01 ). Target point is shown as yellow cross. Target point: (1.45,1.58,65.)