Reinforcement Learning Controllers for Fusion Scenario Development

Evgeny Adishchev; Maxim Nurgaliev; Dmitry Sorokin; Alexander Granovsky; Maksim Stokolesov; Igor Prokofyev; Eduard Khairutdinov; Georgy Subbotin

physics.plasm-phOct 19, 2026

Reinforcement Learning Controllers for Fusion Scenario Development

Authors: Evgeny Adishchev, Maxim Nurgaliev, Dmitry Sorokin, Alexander Granovsky, Maksim Stokolesov, Igor Prokofyev, Eduard Khairutdinov, Georgy Subbotin

Organizations: Next Step Fusion s.a.r.l.

Abstract

Scenario development, the design of the waveforms that take a tokamak discharge from breakdown to flat-top, is still an iterative, labour-intensive process. Every candidate scenario has to be simulated with a controller in the loop, and the classical shape and current controllers need low-level actuator references. So the process is iterative and includes a number of trial-and-error steps. We propose to replace the per-scenario controller tuning by a single goal-conditioned reinforcement-learning controller trained once, in advance, against a free-boundary equilibrium and transport simulator. A scenario is then specified only by the target waveforms of the physical quantities of interest (plasma current, line-averaged electron density, LCFS shape, x-point positions), and the controller produces the coil-current and fueling commands that realize them. We demonstrate the approach on an EAST plasma-current ramp-up: the controller takes over a small limited plasma, ramps the current and the density, grows the cross-section, forms two x-points and holds a double-null diverted configuration against sampled shape targets for the rest of a 1.2 s episode. The neural network controller is trained using TQC algorithm. On 1024 test scenarios sampled from a low-discrepancy scrambled Sobol sequence in 11-dimensional target space, the controller forms the divertor in 86% of the cases, holds it for more than 0.5 s in 67% and reaches the end of the episode in a diverted state in 58%, with flat-top tracking errors of a few kA in plasma current, $\sim 0.15\times10^{19}$ m $^{-3}$ in density, $\sim 2$ cm in position and $\sim 8$ cm in x-point location; the failures concentrate at the high-current density ( $J$ ) region which is believed to be physically or from engineering perspective not feasible within ohmic mode.

Figures & tables

Quantity	Symbol	Final value	Ramp end [s]
Plasma current	$I_{p}$	200–350 kA	0.4–1.0
Line-averaged density	$\bar{n}_{e}$	1.5–3.0 $\times 10^{19}$ m -3	0.4–1.0
Inner-wall boundary radius	$R_{\rm in}$	1.40–1.53 m	0.4–1.0
Minor radius	$a$	0.24–0.36 m	0.4–1.0
Elongation	$\varkappa$	1.7–2.0	0.4–1.0
Squareness (4 quadrants)	$\zeta$	0.2–0.4	0.4–1.0

Table 1: Scenario target ranges, sampled uniformly and independently at every training episode. Each target is a ramp from the value measured at hand-over to the sampled final value, reached at the sampled time.

Figure 1: Test episode with a low-current target:

I_{p}^{\rm t}=217

kA,

\bar{n}_{e}^{\rm t}=2.2\times 10^{19}

m -3 ,

a=0.27

\varkappa=1.95

, targets reached at 0.6 s. The plasma is diverted at 0.21 s and held to the end of the episode.

Figure 2: Test episode with a high-current, low-density target:

I_{p}^{\rm t}=323

kA,

\bar{n}_{e}^{\rm t}=1.7\times 10^{19}

m -3 ,

a=0.34

\varkappa=1.92

. The plasma is diverted at 0.20 s and held to the end of the episode; the current settles 13 kA above its target.

Figure 3: Test episode with a high-current, high-density target, the target relevant to the long-pulse targets:

I_{p}^{\rm t}=297

kA,

\bar{n}_{e}^{\rm t}=2.8\times 10^{19}

m -3 ,

a=0.33

\varkappa=1.83

. The plasma is diverted at 0.20 s and held to the end of the episode at a Greenwald fraction of 0.36.

Figure 4: The evolution of plasma around limited to diverted mode transition

	$n$	divertor formed	held $\geq$ 0.1 s	held $\geq$ 0.5 s	held to 1.2 s
$I_{p}^{\rm t}$ 200–250 kA	341	1.00	0.98	0.85	0.73
$I_{p}^{\rm t}$ 250–300 kA	341	0.98	0.95	0.84	0.74
$I_{p}^{\rm t}$ 300–350 kA	342	0.59	0.38	0.33	0.28
all	1024	0.86	0.77	0.67	0.58

Table 2: Test on 1024 Sobol scenarios, by target plasma current.

Figure 5: Test on 1024 Sobol scenarios. Color = plasma successfully changed mode to diverted, size = episode length (the majority of simulations lasted full 1200 steps)

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Input	Size	Form
Goal: target centre, 8 boundary points, $I_{p}$ , $\bar{n}_{e}$	20	raw
Plasma current	1	raw, derivative
Coil currents (13 coils)	13	raw, derivative
Boundary points (8, $R$ and $Z$ )	16	offset, derivative
Limiter contact point	2	raw
Lower and upper x-point	4	offset

Appendix

Table 3: Inputs of the policy network (109 values). “offset” means the quantity is given as achieved minus target; “derivative” means its one-step finite difference is appended. The critic receives the same inputs plus actions.

Component	Error $d$	$\gamma$	Phase
plasma current	$\|I_{p}-I_{p}^{\rm t}\|$	50 kA	L, D
density	$\|\bar{n}_{e}-\bar{n}_{e}^{\rm t}\|$	$2.0\times 10^{19}$ m -3	L, D
centre	$\\|C-C^{\rm t}\\|$	0.15 m	L, D
boundary shape	softmax ${}_{i}\\|p_{i}-p_{i}^{\rm t}\\|$ (8 points, relative to the centre)	0.6 m	L, D
flux at target points	softmax ${}_{i}\|\psi_{N}(p_{i}^{\rm t})-1\|$	0.5	L, D
limiter contact height	$\|Z_{\rm lim}-Z^{\rm t}\|$	0.3 m	L

Appendix

Table 4: Reward components of the EAST ramp-up task with the margin

\gamma

of their Lorentzian transform (the error at which the component is worth one half). All components have unit weight in the smooth minimum of equation ( 2 ). Phase membership: L = limited, D = diverted.

Condition	Threshold	Phase
current tracking	$\|I_{p}-I_{p}^{\rm t}\|>50$ kA	L, D
current collapse	$I_{p}<20$ kA	L
minor radius	$a<0.2$ m	L, D
edge safety factor	$q_{\rm edge}<2$	L, D
CS current spread	growth $>9.5$ kA	L, D
boundary shape	$\max_{i}\\|p_{i}-p_{i}^{\rm t}\\|>0.2$ m	L, D

Appendix

Table 5: Termination conditions by phase. The episode ends on the first condition that holds; its name is recorded as the termination reason.

Algorithm	SAC objective, TQC critic, automatic entropy tuning
Critics / quantiles per critic / atoms dropped	3 / 25 / 3 of the pooled 75
Networks	MLP $256\times 256$ , SiLU
Input normalisation	running Welford statistics
Actor input / output	109 / 10 (9 coil-current changes, 1 fueling change)
Critic input	109 observations + 10 actions
Auxiliary head	6 shape scalars, mean-squared loss, weight 0.1

Appendix

Table 6: Agent and training hyperparameters.

Feedback Control

Feedback control uses ongoing measurements to adjust actions, stabilize dynamical systems, track goals, and respond to disturbances and constraints.

See all 255 papers

Dynamic Plasma Shape Control with Arbitrary Sensor Subsets
May 15, 2026D. Sorokin, M. Stokolesov, A. Granovskiy +7RL Control Partially Observable RL
Next Step Fusion, Bertrange, L-8070, Luxembourg · Center for Energy Research, University of California San Diego, CA 92093, USA
Advantage-level Aggregation Reinforcement Learning for X-point Target Magnetic Configuration Control in an EXL-50U Experiment-Calibrated Simulation Environment
Aug 21, 2026Siqi Ding, Xuanhe Wang, Pei Guo +10RL Control Feedback Control
Beijing ENN Fusion Energy Science and Technology Co., Ltd., Beijing 101111, China · Beijing Key Laboratory of High Magnetic Field Spherical Torus Fusion Energy, Beijing 101111, China · Hebei Key Laboratory of Compact Fusion, Langfang 065000, China +3
Offline Reinforcement Learning for Plasma Control in Nuclear Fusion: Codebase and Benchmark
May 19, 2026Yang Fu, Haomin Bao, Rohit Sonker +4RL Benchmarks Offline RL
Central South University · Chongqing University · Carnegie Mellon University +1

May 15, 2026J/K move · Enter open · S save

Dynamic Plasma Shape Control with Arbitrary Sensor Subsets

D. Sorokin, M. Stokolesov, A. Granovskiy, I. Prokofyev+6

Next Step Fusion, Bertrange, L-8070, Luxembourg · Center for Energy Research, University of California San Diego, CA 92093, USA

RL Control Partially Observable RL Robust RL

Open paper AskPDF arXiv

Plasma shape control in tokamaks requires a real-time controller that tracks dynamically changing shape targets while tolerating diagnostic failures. Classical approaches decompose the problem into equilibrium reconstruction followed by a linear controller, and assume a fixed, fully operational sensor set. We present a reinforcement learning agent that addresses both limitations simultaneously. The agent is trained in NSFsim, a high-fidelity tokamak simulator configured for DIII-D, on a curated dataset of 120 experimental plasma shapes. The shape targets are resampled as random step changes every 0.25 s, exposing the agent to diverse transitions across the full shape envelope. At test time the agent zero-shot tracks dynamic shape sequences; on a held-out static configuration in simulation it achieves a mean shape error of 2.01 cm, and dynamic trajectory following is demonstrated qualitatively in simulation and on the physical device. Diagnostic dropout randomly masks 30% of magnetic sensors per episode, yielding a single policy robust to arbitrary sensor subsets without backup controllers or mode-switching logic. An asymmetric actor-critic architecture with privileged equilibrium information improves value estimation under partial observability; an auxiliary shape reconstruction head on the actor enables end-to-end shape reconstruction from raw diagnostics and serves as an interpretability tool for policy analysis. The policy transfers to experimental DIII-D shots, where it directly commands the coil actuators on two dynamic shape maneuvers, and to the independent GSevolve simulator.

Abstract

Feedback Control

Explore similar work

Dynamic Plasma Shape Control with Arbitrary Sensor Subsets

Advantage-level Aggregation Reinforcement Learning for X-point Target Magnetic Configuration Control in an EXL-50U Experiment-Calibrated Simulation Environment

Offline Reinforcement Learning for Plasma Control in Nuclear Fusion: Codebase and Benchmark