Evolutionary Warm-Starts for Reinforcement Learning in Industrial Continuous Control

Tom Maus; Stephan Frank; Tobias Glasmachers

cs.NEMar 23, 2026

Evolutionary Warm-Starts for Reinforcement Learning in Industrial Continuous Control

Authors: Tom Maus, Stephan Frank, Tobias Glasmachers

Organizations: Inst. for Neural Computation, Ruhr University Bochum, Bochum, Germany

Abstract

Reinforcement learning (RL) is still rarely applied in industrial control, partly due to the difficulty of training reliable agents for real-world conditions. This work investigates how evolution strategies can support RL in such settings by introducing a continuous-control adaptation of an industrial sorting benchmark. The CMA-ES algorithm is used to generate high-quality demonstrations that warm-start RL agents. Results show that CMA-ES-guided initialization significantly improves stability and performance. Furthermore, the demonstration trajectories generated with the CMA-ES provide a strong oracle reference performance level, which is of interest in its own right. The study delivers a focused proof of concept for hybrid evolutionary-RL approaches and a basis for future, more complex industrial applications.

Figures & tables

Fig. 1: Simplified schematic of the continuous sorting process, based on [ 3 , 8 ] . The agent regulates the input stream into the sorting machine for separation of materials. Output quantities and purities define the performance targets.

Fig. 2: Performance comparison across 20 test seeds with mean cumulative reward, standard deviation (top) and reward distributions (bottom).

PPO Parameter	Value	CMA-ES / BC	Value
Learning Rate	$3\cdot 10^{-4}$	Population Size ( $\lambda$ )	$16$
Policy Arch. (MLP)	$[32,32]$	Max. Generations	$30$
Value Arch. (MLP)	$[32,32]$	Initial Sigma ( $\sigma$ )	$0.1$
Activation Func.	Tanh	BC Training Epochs	$10$
Total Timesteps	$100,000$	Demo Trajectories	$100$
Eval. Frequency	$5,000$	Episode Length	$100$

TABLE I: Hyperparameters for PPO, CMA-ES, and BC

Evolutionary RL

See all 13 papers

Continual Reinforcement Learning with Neuroevolution
Oct 1, 2026Eleni Nisioti, Andrea Cossu, Kathrin Korte +1Neuroevolution Loss of Plasticity
IT University of Copenhagen · University of Pisa · Sakana AI
Position: Deployed Reinforcement Learning should be Continual
Jun 1, 2026Parnian Behdin, Kevin Roice, Golnaz MesbahiNon-Stationary RL Reinforcement Learning
Alberta Machine Intelligence Institute (Amii), Edmonton, Canada.
The Challenges of Using Reinforcement Learning for Controlling Industrial Energy Systems
May 29, 2026Tobias Lademann, Théo Vincent, Jan Peters +1RL Control Partially Observable RL
Institute for Production Management, Technology and Machine Tools (PTW), Technical University of Darmstadt · DFKI GmbH, SAIROL · Department of Computer Science, Technical University of Darmstadt +1

Oct 1, 2026J/K move · Enter open · S save

Continual Reinforcement Learning with Neuroevolution

Eleni Nisioti, Andrea Cossu, Kathrin Korte, Sebastian Risi

IT University of Copenhagen · University of Pisa · Sakana AI

Neuroevolution Loss of Plasticity Evolution Strategies Stability-Plasticity Dilemma Continual RL Genetic Algorithms+1

Open paper AskPDF arXiv

Despite many studies about causes and remedies of plasticity loss in Reinforcement Learning (RL) under continual task changes, no RL method has yet consistently achieved a good balance between adaptation and forgetting. Here we turn to an alternative optimization paradigm, neuroevolution (NE): algorithms that search directly in weight space through mutation and selection over a population of neural networks. Across a wide array of environments and environmental changes, with policies ranging from a few hundred parameters to million-parameter networks, we compare evolution strategies (ES) and genetic algorithms (GAs) against state-of-the-art continual RL variants and population-based RL. ES most consistently achieves a good stability-plasticity trade-off, while the GA is the most plastic method but forgets more than ES. To explain this, we study the return landscape around each method's solutions. ES finds the widest neighborhoods, i.e.\ regions of weight space in which perturbed policies still solve the task, and the size of the overlap between the neighborhoods of consecutive tasks correlates with a method's stability-plasticity trade-off. Rewarding behavioral diversity in a GA through novelty search makes the population even more plastic, at the cost of forgetting. Finally, symptoms of plasticity loss commonly reported in RL do not transfer to NE. Overall, these results establish NE as a competitive alternative to RL under continual task changes, and suggest that training under perturbations in weight space may be a useful mechanism for continual learning more broadly.

Abstract

Evolutionary RL

Explore similar work

Continual Reinforcement Learning with Neuroevolution

Position: Deployed Reinforcement Learning should be Continual

The Challenges of Using Reinforcement Learning for Controlling Industrial Energy Systems