Organizations: Laboratory for Applied Mechanical Design Ecole Polytechnique Fédérale de Lausanne (EPFL) CH-1015 Lausanne, Switzerland · Department of Mechanical and Process Engineering ETH Zürich 8092 Zurich, Switzerland
Many engineering design problems involve multiple conflicting objectives and strict feasibility constraints. They are typically solved with population-based optimizers that require many evaluations and offer limited online control over where the search is focused. We present MOON-RL, a reinforcement-learning-based optimization framework that formulates constrained multi-objective search as preference-directed navigation in the objective space. In MOON-RL, a lightweight PPO policy receives objective-space and constraint-related observations and acts as an instance-specific adaptive search process. It learns how to update the design so that the current solution advances along a user-specified trade-off direction, while a simple meta-algorithm periodically resets the policy, resamples starting designs from a continuously updated archive of non-dominated solutions, and adjusts step-size and penalty schedules. We evaluate MOON-RL on real-world actuator-design problems and additional convergence-hard constrained benchmarks, comparing against established evolutionary algorithms under matched evaluation budgets. In focused search scenarios, MOON-RL achieves competitive performance in the region of interest, and ablation studies show that both the RL-driven adaptive sampling process and the meta-algorithm contribute to this behaviour, suggesting that repurposing RL training as objective-space navigation is a viable building block for interactive, preference-guided design tools. Additional experiments show that the framework also accommodates adaptive non-RL samplers, while PPO remains a strong choice for fixed-target, preference-directed objective-space navigation.
Figures & tables
Figure 1: T -step rollouts for an untrained (a) and trained (b) agent in a 2-objective max-max problem. The trained agent drives the solution efficiently towards the Pareto Front along a reference direction defined by a target point in the objective space
Figure 2: (a) monitored geometric metrics in the objective space, and (b) visualization of the reward function (without constraint penalty) for a two-objective optimization.
Figure 3: Overview of the MOON-RL framework and data flow. The framework starts from the user-defined inputs, including the runtime parameters, initial design x0 , and target point ztarget . The policy πθ produces an action at , which is used to generate a new candidate design xt+1 . The candidate design is evaluated through the multi-objective evaluation functions, producing the objective values ft+1 and constraint violations CVt+1 . These evaluated quantities are used to update the non-dominated archive P , construct the observation and reward, and analyze the position of the candidate design in objective space relative to the target direction. The meta-algorithm updates runtime parameters, such as scaling factors and constraint violation coefficients, and selects new starting designs during restarts. Arrow colors distinguish initial inputs, MOON-RL internal data, problem-specific data, updated parameters, and archive output.
Figure 4: Analysis region (shaded red) example for CS1: only points B, C, and D are chosen for analysis.
Figure 5: “CS1” problem: EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles.
Figure 6: “CT1” problem: EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles.
Figure 7: “CTS1” problem: Comparison of median EAFs for MOON-RL against (a) NSGA-II (pop. 200), (b) R-NSGA-II ( ε=0.01 ), (c) APSEA, and (d) CMODRL. The target point is shown as a yellow cross. Grey points depict the baseline algorithm EAF, while colored points correspond to the MOON-RL EAF. The color encodes the signed additive ε -indicator: red regions indicate that MOON-RL outperforms the baseline, blue regions that the baseline outperforms MOON-RL, and white denotes parity. (e) HV progression including 5th and 95th percentiles.
Figure 8: “DAS-CMOP 3-11” problem: EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles.
Figure 9: “CS1” problem (global search): EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles.
Algorithm
HV ↑
GD ↓
Eval @ 50% HV ↓
Eval @ 75% HV ↓
CS1
MOON-RL
52.0
0.1
4880(30/30)
16440(30/30)
APSEA
40.2
1.1
123194(30/30)
135346(12/30)
LCMEA
54.6
0.05
7800(29/30)
30800(27/30)
CMODRL
52.5
0.16
158325(29/30)
183600(29/30)
NSGA-II (pop. 200)
41.0
1.1
7600(30/30)
159800(13/30)
Table 1: Results summary for main results. Success rate in parentheses shows how many runs reached the corresponding HV target. ↑ means higher is better, ↓ means lower is better. Only median results for the best performing algorithms are shown. Full tables with more algorithms and statistical results are available in Appendices J , K , L , M , and N .
Figure 10: EAF and HV progression of MOON-RL run with and without meta-algorithm and with and without training the policy network. Panels (a, b) show runs with the meta-algorithm active, panels (c, d) runs without it. Focused search scenario.
Figure 11: EAF and HV progression of MOON-RL run with and without meta-algorithm and with and without training the policy network. Panels (a, b) show runs with the meta-algorithm active, panels (c, d) runs without it. Global search scenario.
Figure 12: “CS1” problem: EAF (a) and HV progression (b) of MOON-RL with different adaptive samplers, including 5th and 95th percentiles.
Figure 13: “DAS-CMOP 3-11” problem: EAF (a) and HV progression (b) of MOON-RL with different adaptive samplers, including 5th and 95th percentiles.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
0
9
55
159
248
921
1118
217
2674
2957
2968
3008
3376
3750
4352
4779
4844
5319
5520
5806
5859
6215
7106
8523
8642
8694
9213
9256
9348
9766
Appendix
Table 2: Random seeds used in all experiments
Problem
Objective 1
Objective 2
Objective 3
CS1
0.1
20
–
CT1
0.1
0.2
–
CTS1
0.1
0.2
20
DAS-CMOP3-11
0.1
0.1
–
Appendix
Table 3: Objective scaling factors used for MOON-RL and for hypervolume computation. A dash indicates that no third objective is defined for the problem.
Parameter
Value / Description
Policy architecture
Separate networks for actor and critic
Actor architecture
Separate networks for means and standard deviation
Action distribution
Independent Gaussian per action dimension
Hidden layers
2 per network (all networks)
Hidden layer sizes
64 units
Hidden activation
ReLU
Appendix
Table 4: PPO agent hyperparameters used in all experiments.
Parameter
Value / Description
Trajectory length T
8 steps
Parallel trajectories per episode N
10
Candidate evaluations per episode
N×T=80
Policy update interval
After every episode
Number of episodes
3750
Candidate evaluations from rollouts
300,000
Appendix
Table 5: PPO rollout and update settings used in the experiments.
Parameter
Schedule Type
Value
Initial scaling factor κ0
Linear decay
7% of design-variable ranges
Target scaling factor κmin
1%
Decay interval Sκ
Every 100 episodes
Decay rate δκ
1% of design-variable ranges (subtracted per interval)
Minimum categorical-variable scaling
Fixed floor
0.6 for MODAct variables
Initial constraint penalty λ0
Linear increase
2
Appendix
Table 6: Runtime scheduler parameters used to adjust scaling factors κ and constraint penalties λ .
Algorithm
Parameters
NSGA-II
Population size: 200 or 100
R-NSGA-II
Population size: 200 Reference point(s): Same as for MOON-RL, depending global or local search ε : 0.01 or 0.001
MOEA/D
Reference directions: “Das-Dennis”, 2 objectives, 199 partitions Number of neighbors: 200 Probability of neighbor mating: 0.7
NSGA-III
Population size: 200 Reference directions: “Das-Dennis”, 2 objectives, 14 or 199 partitions
R-NSGA-III
Population size: 200 Reference point(s): Same as for MOON-RL, depending global or local search μ : 0.1 or 0.01
C-TAEA
Reference directions: “Das-Dennis”, 2 objectives, 14 or 199 partitions
Appendix
Table 7: Benchmark Algorithm Parameters Used in Experiments
Figure 14: EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles. Target point: (0.22,51.2) .
Figure 15: EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles. Target point: (0.39,138.) .
Figure 16: EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles. Target point: (0.58,172.) .
Figure 17: EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles. Target point: (0.85,182.8) .
Figure 18: EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles. Target point: (0.4,1)
Figure 19: EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles. Target point: (0.25,0.4) .
Figure 20: EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles. Target point: (0.53,1.5) .
Figure 21: EAF (a) and HV progression (b) of different algorithms, including 5th and 95th percentiles. Target point: (0.8,1.8)
Figure 22: Pairwise projections of median EAF for different algorithms: (a) NSGA-II (pop. 200 ), (b) R-NSGA-II ( ε=0.01 ), (c) NSGA-III (200 ref. dir.) and (d) R-NSGA-III ( μ=0.01 ). Target point is shown as yellow cross. Target point: (0.3,0.0,100.) .
Figure 23: Pairwise projections of median EAF for different algorithms: (a) NSGA-II (pop. 200 ), (b) R-NSGA-II ( ε=0.01 ), (c) NSGA-III (200 ref. dir.) and (d) R-NSGA-III ( μ=0.01 ). Target point is shown as yellow cross. Target point: (0.9,1.1,80.) .
Figure 24: Pairwise projections of median EAF for different algorithms: (a) NSGA-II (pop. 200 ), (b) R-NSGA-II ( ε=0.01 ), (c) NSGA-III (200 ref. dir.) and (d) R-NSGA-III ( μ=0.01 ). Target point is shown as yellow cross. Target point: (0.95,−0.35,480.) .
Figure 25: Pairwise projections of median EAF for different algorithms: (a) NSGA-II (pop. 200 ), (b) R-NSGA-II ( ε=0.01 ), (c) NSGA-III (200 ref. dir.) and (d) R-NSGA-III ( μ=0.01 ). Target point is shown as yellow cross. Target point: (1.45,1.58,65.)
Many decision-making problems in computing and networking systems can be naturally formulated as cost-minimization problems under performance constraints. In dynamic environments, reinforcement learning (RL) is often used to solve such problems at runtime by embedding both costs and constraint violations into a single scalar reward through weighted penalty terms, following a Lagrangian-inspired formulation. However, in this context the behavior of the learned policy critically depends on the choice of these weights, which are typically selected manually. This makes it difficult to identify an appropriate trade-off between optimizing the primary objective and effectively avoiding constraint violations, particularly in non-stationary environments where their relative importance may change. This paper presents MAMO (Multi-Agent system for Multi-Objective constrained optimization), an approach to tackle this balancing problem through multi-agent RL. MAMO decouples task execution from objective design by formulating the selection of reward weights as a learning problem, providing a !rst step towards more autonomous and robust RL-based solutions for constrained optimization problems in dynamic environments.
In this work, we present MOCHA, the first, to our knowledge, reinforcement learning based approach to computing a family of Pareto-optimal policies across the design space of a robot using a single network. Specifically, MOCHA leverages the hypernetwork architecture to learn a network that produces specialized network parameters optimized for a given objective and parameterized robot design; we term this a multi-objective design hypernetwork (MDH). We demonstrate the capabilities of MDHs to represent a complex family of design-dependent strategies on two distinct robot morphologies, each with six design dimensions and across 2-3 objectives. Moreover, we propose an approach for efficiently producing a Design Pareto set using evolutionary search of the learned policy network, generating the optimal design-policy combination for each objective prioritization. Lastly, we provide an efficient method for computing generalist robot designs which achieve the best cumulative performance across the entire set of objectives.
Varun Madabushi, Neil Janwani, Maegan Tucker
Dynamic Mobility Lab at Georgia Tech, Atlanta, U.S.
Analog circuit design automation using reinforcement learning (RL) has emerged as a promising approach for reducing manual effort. However, many existing RL-based methods focus on single-objective optimization. Even methods designed for multi-objective (MO) problems often reduce multiple design specifications to a single scalar reward. This simplification limits the ability to capture the true Pareto trade-off among competing objectives and often leads to suboptimal designs. Moreover, requiring the model to be retrained from scratch whenever the desired MO specifications change remains a key limitation. To address these challenges, we present ORACLE, an open-source RL-based framework for MO analog circuit design optimization that replaces scalar reward optimization with vector-valued learning and preference-aware conditioning. ORACLE represents a true MO analog circuit design optimizer that uses a preference vector to specify the relative weights of multiple objectives, enabling a single trained model to generate designs across diverse trade-off settings without retraining. We further propose two preference-guidance strategies, namely normalized-weight guidance and cosine-aligned guidance, to improve convergence. In addition, we incorporate a large language model (LLM)-guided action selection mechanism to filter actions that are likely to lead to suboptimal designs or increased runtime. Our results show that, on multiple circuit topologies with 2,000 test cases, ORACLE reduces runtime by 20.4x - 104.4x compared to state-of-the-art approaches. It also meets 99.9% of the 2,000 target specifications, and achieves 5.1x - 318.6x better figure of merit in the resulting output specs.
Osei Brempong, Mohammed Ayman Habib, Vivan Poddar +1