cs.NEMar 17, 2026

Deep Reinforcement Learning-Assisted Automated Operator Portfolio for Constrained Multi-objective Optimization

Authors: Shuai Shao, Ye Tian, Shangshang Yang, Xingyi Zhang

Organizations: the School of Computer Science and Technology, Anhui University, Hefei 230601, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Constrained multi-objective optimization problems (CMOPs) are of great significance in the context of practical applications, ranging from scientific to engineering domains. Most existing constrained multi-objective evolutionary algorithms (CMOEAs) usually employ fixed operators all the time, which exhibit poor versatility in handling various CMOPs. Therefore, some recent studies have focused on adaptively selecting the best operators for the current population states during the search process. The evolutionary algorithms proposed in these studies learn the value of each operator and recommend the operator with the highest value for the current population, resulting in only a single operator being recommended at each generation, which can potentially lead to local optima and inefficient utilization of function evaluations. To address the dilemma in operator adaptation, this paper proposes a reinforcement learning-based automated operator portfolio approach to learn an allocation scheme of operators at each generation. This approach considers the optimization-related and constraint-related features of the current population as states, the overall improvement in population convergence and diversity as rewards, and different operator portfolios as actions. By utilizing deep neural networks to establish a mapping model between the population states and the expected cumulative rewards, the proposed approach determines the optimal operator portfolio during the evolutionary process. By embedding the proposed approach into existing CMOEAs, a deep reinforcement learning-assisted automated operator portfolio based evolutionary algorithm for solving CMOPs, abbreviated as CMOEA-AOP, is developed. Empirical studies on 33 benchmark problems demonstrate that the proposed algorithm significantly enhances the performance of CMOEAs and exhibits more stable performance across different CMOPs.

Figures & tables

Fig. 1: Convergence profiles of IGD values obtained by EMCMO1 (using only genetic operators), EMCMO2 (using only differential evolution operators), EMCMO3 (using both genetic and differential evolution operators) on CF2, CF6, and CF9.
Fig. 2: Illustration of the proposed CMOEA-AOP with deep reinforcement learning based automated operator portfolio, where a DDPG agent consisting of an actor network and a critic network is employed to recommend the optimal operator portfolio scheme for CMOEAs. Here “States” represent the optimization-related and constraint-related features of the current population, and “Action” represents the optimal operator portfolio scheme recommended by the deep reinforcement learning agent based on the current evolutionary states.
ProblemEMCMOBiCoAGEMOEA-IITSTIDRLOSCMOEA-AOP
CF15.8878e-3 (6.15e-4) −-1.6353e-2 (2.05e-3) −-2.1244e-2 (3.06e-3) −-2.6988e-2 (3.28e-3) −-2.3242e-3 (4.21e-4) −-1.8565e-3 (3.43e-4)
CF23.5223e-2 (1.33e-2) −-4.9712e-2 (1.55e-2) −-5.0383e-2 (1.93e-2) −-7.0321e-2 (2.01e-2) −-7.5237e-3 (3.09e-3) ≈\approx8.1487e-3 (3.15e-3)
CF32.1806e-1 (7.60e-2) ≈\approx2.3767e-1 (7.24e-2) −-3.0784e-1 (1.89e-1) −-3.3490e-1 (1.08e-1) −-2.7928e-1 (1.32e-1) −-1.8946e-1 (1.33e-1)
CF49.9130e-2 (3.04e-2) −-9.8463e-2 (3.58e-2) −-1.0766e-1 (3.35e-2) −-1.4982e-1 (3.89e-2) −-4.5235e-2 (8.80e-3) ++5.2360e-2 (6.51e-3)
CF52.7146e-1 (9.80e-2) −-3.0442e-1 (1.36e-1) −-3.2254e-1 (1.16e-1) −-4.0078e-1 (1.25e-1) −-2.2847e-1 (1.09e-1) −-1.8643e-1 (1.41e-1)
CF65.6069e-2 (2.43e-2) −-9.4654e-2 (4.28e-2) −-9.4154e-2 (3.35e-2) −-1.0310e-1 (2.49e-2) −-4.3410e-2 (2.53e-2) ≈\approx3.5271e-2 (6.68e-3)
TABLE I: IGD Values Obtained by EMCMO, BICO, AGEMOEA-II, DRLOS, and the Proposed CMOEA-AOP on the CF Benchmark Suite, LIR-CMOP Benchmark Suite, and DAS-CMOP Benchmark Suite. Best Result in Each Row Is Highlighted
Fig. 3: Populations with the median IGD obtained by EMCMO, Bico, AGEMOEA-II, DRLOS, and the proposed CMOEA-AOP on LIR-CMOP12.
Fig. 4: Populations with the median IGD obtained by EMCMO, Bico, AGEMOEA-II, DRLOS, and the proposed CMOEA-AOP on DAS-CMOP8.
Fig. 5: Convergence profiles of mean IGD values by EMCMO, Bico, AGEMOEA-II, DRLOS, and the proposed CMOEA-AOP on CF6, LIR-CMOP3 and LIR-CMOP4.
ProblemCMOEA-AOP1CMOEA-AOP2CMOEA-AOP3CMOEA-AOP
CF13.9110e-3 −-1.6821e-3 ≈\approx4.9569e-3 −-1.8565e-3
CF27.1605e-2 −-9.0358e-3 ≈\approx1.5847e-2 −-8.1487e-3
CF32.4892e-1 −-2.5020e-1 −-1.3385e-1 ≈\approx1.8946e-1
CF41.4285e-1 −-5.3927e-2 ≈\approx5.9382e-2 −-5.2360e-2
CF52.8200e-1 −-2.8397e-1 −-2.0711e-1 ≈\approx1.8643e-1
CF66.9170e-2 −-3.8727e-2 −-4.6500e-2 −-3.5271e-2
TABLE II: IGD Values Obtained by CMOEA-AOP and Its Three Variants on the CF Benchmark Suite, LIR-CMOP Benchmark Suite, and DAS-CMOP Benchmark Suite. Best Result in Each Row Is Highlighted
Figure 8
Figure 9
Figure 10
Figure 11
More in this topic

Multi-Objective Optimization

Multi-objective optimization studies how to find and choose solutions that balance competing goals, constraints, and preferences.

See all 266 papers

Explore similar work

CardsList
Apr 6, 2026

Ranking Constraints via Topological Dual-Directional Search in Evolutionary Multi-Objective Optimization

Existing evolutionary algorithms for Constrained Multi-objective Optimization Problems (CMOPs) typically treat all constraints uniformly, overlooking their distinct geometric relationships with the true Constrained Pareto Front (CPF). In reality, constraints play different roles: some directly shape the final CPF, some create infeasible obstacles, while others are irrelevant. To exploit this insight, we propose a novel algorithm named RCCMO, which sequentially performs unconstrained exploration, single-constraint exploitation, and full-constraint refinement. The core innovation of RCCMO lies in a constraint prioritization method derived from these geometric insights, seamlessly coupled with a unique dual-directional search mechanism. Specifically, RCCMO first prioritizes constraints that constitute the final CPF, approaching them from the evolutionary direction (optimizing objectives) to locate the CPF directly shaped by single-constraint boundaries. Subsequently, for constraints that merely hinder the population's progress, RCCMO searches from the anti-evolutionary direction (targeting the infeasible boundaries where hindering constraints intersect with the CPF) to effectively discover how these constraints obstruct and form the final CPF. Meanwhile, irrelevant constraints are intentionally bypassed. Furthermore, a series of specialized mechanisms are proposed to accelerate the algorithm's execution, reduce heuristic misjudgments, and dynamically adjust search directions in real time. Extensive experiments on 5 benchmark test suites and 29 real-world CMOPs demonstrate that RCCMO significantly outperforms seven state-of-the-art algorithms.

Ruiqing Sun, Dawei Feng, Sheng Qi +6

National Key Laboratory of Parallel and Distributed Computing, College of Computer Science and Technology, National University of Defense Technology, Changsha 410000, P.R. China · College of System Engineering, National University of Defense Technology, Changsha 410000, P.R. China · College of Intelligence Science and Technology, National University of Defense Technology, Changsha 410073, P.R. China +1

Multi-Objective Evolutionary OptimizationEvolutionary OptimizationMulti-Objective Optimization
AskPDF
Jun 18, 2026

A Multi-Agent system for Multi-Objective constrained optimization

Many decision-making problems in computing and networking systems can be naturally formulated as cost-minimization problems under performance constraints. In dynamic environments, reinforcement learning (RL) is often used to solve such problems at runtime by embedding both costs and constraint violations into a single scalar reward through weighted penalty terms, following a Lagrangian-inspired formulation. However, in this context the behavior of the learned policy critically depends on the choice of these weights, which are typically selected manually. This makes it difficult to identify an appropriate trade-off between optimizing the primary objective and effectively avoiding constraint violations, particularly in non-stationary environments where their relative importance may change. This paper presents MAMO (Multi-Agent system for Multi-Objective constrained optimization), an approach to tackle this balancing problem through multi-agent RL. MAMO decouples task execution from objective design by formulating the selection of reward weights as a learning problem, providing a !rst step towards more autonomous and robust RL-based solutions for constrained optimization problems in dynamic environments.

Federica Filippini

University of Milano-Bicocca Milan, Italy

Reward ShapingMulti-Agent System OptimizationReinforcement LearningMulti-Objective Reinforcement LearningMulti-Agent Reinforcement LearningConstrained RL+1
AskPDF
Apr 12, 2026

Reinforcement Learning-Guided NSGA-II Enhanced with Gray Relational Coefficient for Multi-Objective Optimization: Application to NASDAQ Portfolio Optimization

In modern financial markets, decision-makers increasingly rely on quantitative methods to navigate complex trade-offs among multiple, often conflicting objectives. This paper addresses constrained multi-objective optimization (MOO) with an application to portfolio optimization for minimizing risk and maximizing return. To address existing gaps, we propose a novel reinforcement learning (RL)-guided non-dominated sorting genetic algorithm II (NSGA-II) enhanced with gray relational coefficients (GRC), termed RL-NSGA-II-GRC, which combines an RL agent controller and GRC-based selection to improve convergence and diversity of Pareto fronts. The agent adapts evolutionary parameters online using metrics of hypervolume, feasibility, and diversity, while the GRC tournament operator ranks parents via a unified score considering dominance rank, crowding distance, and proximity to ideal reference. We evaluate the framework on the Kursawe and CONSTR benchmarks and a NASDAQ portfolio application. On the benchmarks, RL-NSGA-II-GRC achieves convergence improvements of about 5.8% and 4.4% over NSGA-II, while preserving well-distributed non-dominated solutions. In the portfolio application, it produces a smooth, densely populated efficient frontier supporting identification of the maximum Sharpe ratio portfolio (annualized Sharpe =1.92) and utility-optimal portfolios for different risk-aversion levels. The main contributions are three-fold: 1) we propose an RL-NSGA-II-GRC method integrating an RL agent into the evolutionary framework to adaptively control parameters via generational feedback; 2) we design a GRC-enhanced binary tournament operator providing a comprehensive indicator to guide the search toward the Pareto front; 3) we demonstrate, on benchmark MOO and a NASDAQ case study, that the method delivers improved convergence and well-populated frontiers supporting actionable insights.

Zhiyuan Wang, Qinxu Ding, Ding Ding +4

School of Business, Singapore University of Social Sciences, Singapore 599494, Singapore

Multi-Objective Evolutionary OptimizationMulti-Objective OptimizationPortfolio Optimization
AskPDF