In this research, we investigate deep reinforcement learning (DRL) as a practical tool for constrained, multi-objective optimization for tokamak fusion reactor designs. The task is to optimize the design configurations that minimize building cost while achieving high energy gain and long energy confinement time, subject to the four operational limits required for stable operation. Due to the coupled physics-engineering evaluation, the resulting objectives and constraints are black-box, making it challenging to find optimal solutions with conventional search strategies. To address this, we implement Reward-Constrained Policy Optimization (RCPO) for tokamak fusion reactor design. This method integrates multiple constraints into the reward functions, guiding the policy towards feasible solutions and simplifying the design optimization process. Our numerical results show that the proposed DRL-based approach finds more feasible designs that satisfy operational constraints more efficiently than other data-driven optimization methods, demonstrating the potential of DRL for efficient future reactor design.
Figures & tables
Figure 1: A simplified process of a tokamak fusion reactor design computation
Quantity
Description
βN
Normalized ratio of plasma thermal and magnetic pressure
κ
Elongation
A
Aspect ratio
Tˉ
Average electron temperature
Bmax
Maximum allowable magnetic field
H
Confinement enhancement factor
Table 1: A list of input design variables required for reactor design computation
Quantity
Description
R
Major radius
a
Minor radius
ΔBlanket
Blanket thickness
ΔTFcoil
TF coil thickness
τE
Energy confinement time
β
Plasma beta
Table 2: A list of output design variables determined by reactor design computation
Figure 2: The proposed framework for reactor design optimization
Input parameter
Search space
βN
[2.0, 4.0]
κ
[1.5, 1.9]
A
[3.5, 4.5]
Tˉ (KeV)
[10, 15]
Bmax (T)
[10, 16]
H
[1.0, 1.3]
Table 3: A list of the search space for each input parameter corresponding to Table 2.1 .
Figure 3: Temporally averaged total reward (a) and penalty term (b) during the optimization process for different methods.
Figure 4: Temporal averaged reward corresponding to energy confinement time τE , building cost C , and energy gain Q during the optimization process.
Figure 5: Temporal averaged penalty term corresponding to β/βTroyon , qkink/q , nˉ/nG , and fop/fbs during the optimization process.
Figure 6: Distribution of the cost C on β/βTroyon , qkink/q , nˉ/nG , and fop/fbs .
Figure 7: Distribution of the energy confinement time τE on β/βTroyon , qkink/q , nˉ/nG , and fop/fbs .
Figure 8: Distribution of the energy gain Q on β/βTroyon , qkink/q , nˉ/nG , and fop/fbs .
Figure 9: Comparison of the poloidal wall design configurations with different methods.
Figure 10: (Reference) Operational regime on the design parameters a , H , Bmax , and PE , with the curves of nGnˉ , βTroyonβ , qqkink , and fbsfop . The unshaded region represents the successful operational regime.
Figure 11: Operational regime on the design parameters a , H , Bmax , and PE , with the curves of nGnˉ , βTroyonβ , qqkink , and fbsfop , optimized by DRL.
Many engineering design problems involve multiple conflicting objectives and strict feasibility constraints. They are typically solved with population-based optimizers that require many evaluations and offer limited online control over where the search is focused. We present MOON-RL, a reinforcement-learning-based optimization framework that formulates constrained multi-objective search as preference-directed navigation in the objective space. In MOON-RL, a lightweight PPO policy receives objective-space and constraint-related observations and acts as an instance-specific adaptive search process. It learns how to update the design so that the current solution advances along a user-specified trade-off direction, while a simple meta-algorithm periodically resets the policy, resamples starting designs from a continuously updated archive of non-dominated solutions, and adjusts step-size and penalty schedules. We evaluate MOON-RL on real-world actuator-design problems and additional convergence-hard constrained benchmarks, comparing against established evolutionary algorithms under matched evaluation budgets. In focused search scenarios, MOON-RL achieves competitive performance in the region of interest, and ablation studies show that both the RL-driven adaptive sampling process and the meta-algorithm contribute to this behaviour, suggesting that repurposing RL training as objective-space navigation is a viable building block for interactive, preference-guided design tools. Additional experiments show that the framework also accommodates adaptive non-RL samplers, while PPO remains a strong choice for fixed-target, preference-directed objective-space navigation.
Andrii Artomov, Soheyl Massoudi, Jürg Schiffmann
Laboratory for Applied Mechanical Design Ecole Polytechnique Fédérale de Lausanne (EPFL) CH-1015 Lausanne, Switzerland · Department of Mechanical and Process Engineering ETH Zürich 8092 Zurich, Switzerland
Constrained multi-objective optimization problems (CMOPs) are of great significance in the context of practical applications, ranging from scientific to engineering domains. Most existing constrained multi-objective evolutionary algorithms (CMOEAs) usually employ fixed operators all the time, which exhibit poor versatility in handling various CMOPs. Therefore, some recent studies have focused on adaptively selecting the best operators for the current population states during the search process. The evolutionary algorithms proposed in these studies learn the value of each operator and recommend the operator with the highest value for the current population, resulting in only a single operator being recommended at each generation, which can potentially lead to local optima and inefficient utilization of function evaluations. To address the dilemma in operator adaptation, this paper proposes a reinforcement learning-based automated operator portfolio approach to learn an allocation scheme of operators at each generation. This approach considers the optimization-related and constraint-related features of the current population as states, the overall improvement in population convergence and diversity as rewards, and different operator portfolios as actions. By utilizing deep neural networks to establish a mapping model between the population states and the expected cumulative rewards, the proposed approach determines the optimal operator portfolio during the evolutionary process. By embedding the proposed approach into existing CMOEAs, a deep reinforcement learning-assisted automated operator portfolio based evolutionary algorithm for solving CMOPs, abbreviated as CMOEA-AOP, is developed. Empirical studies on 33 benchmark problems demonstrate that the proposed algorithm significantly enhances the performance of CMOEAs and exhibits more stable performance across different CMOPs.
Shuai Shao, Ye Tian, Shangshang Yang +1
the School of Computer Science and Technology, Anhui University, Hefei 230601, China
Many decision-making problems in computing and networking systems can be naturally formulated as cost-minimization problems under performance constraints. In dynamic environments, reinforcement learning (RL) is often used to solve such problems at runtime by embedding both costs and constraint violations into a single scalar reward through weighted penalty terms, following a Lagrangian-inspired formulation. However, in this context the behavior of the learned policy critically depends on the choice of these weights, which are typically selected manually. This makes it difficult to identify an appropriate trade-off between optimizing the primary objective and effectively avoiding constraint violations, particularly in non-stationary environments where their relative importance may change. This paper presents MAMO (Multi-Agent system for Multi-Objective constrained optimization), an approach to tackle this balancing problem through multi-agent RL. MAMO decouples task execution from objective design by formulating the selection of reward weights as a learning problem, providing a !rst step towards more autonomous and robust RL-based solutions for constrained optimization problems in dynamic environments.