Reinforcement Learning Controllers for Fusion Scenario Development

Evgeny Adishchev; Maxim Nurgaliev; Dmitry Sorokin; Alexander Granovsky; Maksim Stokolesov; Igor Prokofyev; Eduard Khairutdinov; Georgy Subbotin

physics.plasm-phOct 19, 2026

Reinforcement Learning Controllers for Fusion Scenario Development

Authors: Evgeny Adishchev, Maxim Nurgaliev, Dmitry Sorokin, Alexander Granovsky, Maksim Stokolesov, Igor Prokofyev, Eduard Khairutdinov, Georgy Subbotin

Organizations: Next Step Fusion s.a.r.l.

Abstract

Scenario development, the design of the waveforms that take a tokamak discharge from breakdown to flat-top, is still an iterative, labour-intensive process. Every candidate scenario has to be simulated with a controller in the loop, and the classical shape and current controllers need low-level actuator references. So the process is iterative and includes a number of trial-and-error steps. We propose to replace the per-scenario controller tuning by a single goal-conditioned reinforcement-learning controller trained once, in advance, against a free-boundary equilibrium and transport simulator. A scenario is then specified only by the target waveforms of the physical quantities of interest (plasma current, line-averaged electron density, LCFS shape, x-point positions), and the controller produces the coil-current and fueling commands that realize them. We demonstrate the approach on an EAST plasma-current ramp-up: the controller takes over a small limited plasma, ramps the current and the density, grows the cross-section, forms two x-points and holds a double-null diverted configuration against sampled shape targets for the rest of a 1.2 s episode. The neural network controller is trained using TQC algorithm. On 1024 test scenarios sampled from a low-discrepancy scrambled Sobol sequence in 11-dimensional target space, the controller forms the divertor in 86% of the cases, holds it for more than 0.5 s in 67% and reaches the end of the episode in a diverted state in 58%, with flat-top tracking errors of a few kA in plasma current, $\sim 0.15\times10^{19}$ m $^{-3}$ in density, $\sim 2$ cm in position and $\sim 8$ cm in x-point location; the failures concentrate at the high-current density ( $J$ ) region which is believed to be physically or from engineering perspective not feasible within ohmic mode.

Figures & tables

Quantity	Symbol	Final value	Ramp end [s]
Plasma current	$I_{p}$	200–350 kA	0.4–1.0
Line-averaged density	$\bar{n}_{e}$	1.5–3.0 $\times 10^{19}$ m -3	0.4–1.0
Inner-wall boundary radius	$R_{\rm in}$	1.40–1.53 m	0.4–1.0
Minor radius	$a$	0.24–0.36 m	0.4–1.0
Elongation	$\varkappa$	1.7–2.0	0.4–1.0
Squareness (4 quadrants)	$\zeta$	0.2–0.4	0.4–1.0

Table 1: Scenario target ranges, sampled uniformly and independently at every training episode. Each target is a ramp from the value measured at hand-over to the sampled final value, reached at the sampled time.

Figure 1: Test episode with a low-current target:

I_{p}^{\rm t}=217

kA,

\bar{n}_{e}^{\rm t}=2.2\times 10^{19}

m -3 ,

a=0.27

\varkappa=1.95

, targets reached at 0.6 s. The plasma is diverted at 0.21 s and held to the end of the episode.

Figure 2: Test episode with a high-current, low-density target:

I_{p}^{\rm t}=323

kA,

\bar{n}_{e}^{\rm t}=1.7\times 10^{19}

m -3 ,

a=0.34

\varkappa=1.92

. The plasma is diverted at 0.20 s and held to the end of the episode; the current settles 13 kA above its target.

Figure 3: Test episode with a high-current, high-density target, the target relevant to the long-pulse targets:

I_{p}^{\rm t}=297

kA,

\bar{n}_{e}^{\rm t}=2.8\times 10^{19}

m -3 ,

a=0.33

\varkappa=1.83

. The plasma is diverted at 0.20 s and held to the end of the episode at a Greenwald fraction of 0.36.

Figure 4: The evolution of plasma around limited to diverted mode transition

	$n$	divertor formed	held $\geq$ 0.1 s	held $\geq$ 0.5 s	held to 1.2 s
$I_{p}^{\rm t}$ 200–250 kA	341	1.00	0.98	0.85	0.73
$I_{p}^{\rm t}$ 250–300 kA	341	0.98	0.95	0.84	0.74
$I_{p}^{\rm t}$ 300–350 kA	342	0.59	0.38	0.33	0.28
all	1024	0.86	0.77	0.67	0.58

Table 2: Test on 1024 Sobol scenarios, by target plasma current.

Figure 5: Test on 1024 Sobol scenarios. Color = plasma successfully changed mode to diverted, size = episode length (the majority of simulations lasted full 1200 steps)

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Input	Size	Form
Goal: target centre, 8 boundary points, $I_{p}$ , $\bar{n}_{e}$	20	raw
Plasma current	1	raw, derivative
Coil currents (13 coils)	13	raw, derivative
Boundary points (8, $R$ and $Z$ )	16	offset, derivative
Limiter contact point	2	raw
Lower and upper x-point	4	offset

Appendix

Table 3: Inputs of the policy network (109 values). “offset” means the quantity is given as achieved minus target; “derivative” means its one-step finite difference is appended. The critic receives the same inputs plus actions.

Component	Error $d$	$\gamma$	Phase
plasma current	$\|I_{p}-I_{p}^{\rm t}\|$	50 kA	L, D
density	$\|\bar{n}_{e}-\bar{n}_{e}^{\rm t}\|$	$2.0\times 10^{19}$ m -3	L, D
centre	$\\|C-C^{\rm t}\\|$	0.15 m	L, D
boundary shape	softmax ${}_{i}\\|p_{i}-p_{i}^{\rm t}\\|$ (8 points, relative to the centre)	0.6 m	L, D
flux at target points	softmax ${}_{i}\|\psi_{N}(p_{i}^{\rm t})-1\|$	0.5	L, D
limiter contact height	$\|Z_{\rm lim}-Z^{\rm t}\|$	0.3 m	L

Appendix

Table 4: Reward components of the EAST ramp-up task with the margin

\gamma

of their Lorentzian transform (the error at which the component is worth one half). All components have unit weight in the smooth minimum of equation ( 2 ). Phase membership: L = limited, D = diverted.

Condition	Threshold	Phase
current tracking	$\|I_{p}-I_{p}^{\rm t}\|>50$ kA	L, D
current collapse	$I_{p}<20$ kA	L
minor radius	$a<0.2$ m	L, D
edge safety factor	$q_{\rm edge}<2$	L, D
CS current spread	growth $>9.5$ kA	L, D
boundary shape	$\max_{i}\\|p_{i}-p_{i}^{\rm t}\\|>0.2$ m	L, D

Appendix

Table 5: Termination conditions by phase. The episode ends on the first condition that holds; its name is recorded as the termination reason.

Algorithm	SAC objective, TQC critic, automatic entropy tuning
Critics / quantiles per critic / atoms dropped	3 / 25 / 3 of the pooled 75
Networks	MLP $256\times 256$ , SiLU
Input normalisation	running Welford statistics
Actor input / output	109 / 10 (9 coil-current changes, 1 fueling change)
Critic input	109 observations + 10 actions
Auxiliary head	6 shape scalars, mean-squared loss, weight 0.1

Appendix

Table 6: Agent and training hyperparameters.

Feedback Control

Feedback control uses ongoing measurements to adjust actions, stabilize dynamical systems, track goals, and respond to disturbances and constraints.

See all 255 papers

May 15, 2026

Dynamic Plasma Shape Control with Arbitrary Sensor Subsets

Plasma shape control in tokamaks requires a real-time controller that tracks dynamically changing shape targets while tolerating diagnostic failures. Classical approaches decompose the problem into equilibrium reconstruction followed by a linear controller, and assume a fixed, fully operational sensor set. We present a reinforcement learning agent that addresses both limitations simultaneously. The agent is trained in NSFsim, a high-fidelity tokamak simulator configured for DIII-D, on a curated dataset of 120 experimental plasma shapes. The shape targets are resampled as random step changes every 0.25 s, exposing the agent to diverse transitions across the full shape envelope. At test time the agent zero-shot tracks dynamic shape sequences; on a held-out static configuration in simulation it achieves a mean shape error of 2.01 cm, and dynamic trajectory following is demonstrated qualitatively in simulation and on the physical device. Diagnostic dropout randomly masks 30% of magnetic sensors per episode, yielding a single policy robust to arbitrary sensor subsets without backup controllers or mode-switching logic. An asymmetric actor-critic architecture with privileged equilibrium information improves value estimation under partial observability; an auxiliary shape reconstruction head on the actor enables end-to-end shape reconstruction from raw diagnostics and serves as an interpretability tool for policy analysis. The policy transfers to experimental DIII-D shots, where it directly commands the coil actuators on two dynamic shape maneuvers, and to the independent GSevolve simulator.

D. Sorokin, M. Stokolesov, A. Granovskiy +7

Next Step Fusion, Bertrange, L-8070, Luxembourg · Center for Energy Research, University of California San Diego, CA 92093, USA

RL Control Partially Observable RL Robust RL

Ask PDF

Aug 21, 2026

Advantage-level Aggregation Reinforcement Learning for X-point Target Magnetic Configuration Control in an EXL-50U Experiment-Calibrated Simulation Environment

Managing divertor heat loads is a central challenge for compact, high-power tokamaks. To increase local flux expansion and decouple the dissipation volume from the core, EHL-2 adopts the X-point target (XPT) divertor. This requires the secondary X-point to remain on the divertor leg; displacement degrades the topology and exhaust geometry. Current experiments, including EXL-50U discharges, rely on precomputed feedforward waveforms with PID loops on global quantities. Lacking dedicated closed-loop feedback for the secondary null, XPT operation is repeatable but not routine. We formulate XPT feedback as a multi-objective reinforcement learning (RL) control problem in a free-boundary environment calibrated to EXL-50U discharge #13906. To address strong coupling among plasma current, shape, and null constraints - where reward scalarisation collapses objective-specific temporal credit - we develop Advantage Aggregation (AdvA). AdvA preserves objective-wise temporal credit before worst-objective-aware nonlinear scalarisation and introduces a residual correction to policy updates. AdvA-PPO is evaluated against Reward-PPO and a feedforward-plus-PID baseline under nominal operation, measurement uncertainties, and unseen initial equilibria. On a 500 ms rollout, AdvA-PPO raises the mean worst-channel score from 0.23 to 0.81 over Reward-PPO, reducing X-point flux RMSE by ~20x. Under combined measurement uncertainties, it is the only learned controller completing the horizon while retaining a usable XPT shape. Multi-initialization fine-tuning enables a single AdvA-PPO policy to complete full-horizon operation across divertor and limiter initial equilibria. These results provide a simulation-based foundation for future real-time XPT validation on EXL-50U.

Siqi Ding, Xuanhe Wang, Pei Guo +10

Beijing ENN Fusion Energy Science and Technology Co., Ltd., Beijing 101111, China · Beijing Key Laboratory of High Magnetic Field Spherical Torus Fusion Energy, Beijing 101111, China · Hebei Key Laboratory of Compact Fusion, Langfang 065000, China +3

RL Control Feedback Control Multi-Objective Reinforcement Learning Proximal Policy Optimization

Ask PDF

May 19, 2026

Offline Reinforcement Learning for Plasma Control in Nuclear Fusion: Codebase and Benchmark

Offline reinforcement learning (RL) offers a promising route for developing plasma controllers from historical tokamak data, since online trial-and-error on real devices is costly and risky. However, progress in this direction remains difficult to measure due to the lack of a standardized offline RL benchmark for realistic multi-actuator, long-horizon plasma control problems in nuclear fusion. We introduce RL4F, an Offline Reinforcement Learning Benchmark for Plasma Control in Nuclear Fusion, providing closed-loop evaluation environments and baseline comparisons across four full-profile tracking tasks: rotation, density, temperature, and pressure. The dynamics function underlying the evaluation environment is built from historical discharge data from DIII-D, a real-world Tokamak. We evaluate a broad set of imitation learning and offline RL baselines under a unified protocol. We find that offline model-based RL methods obtain the best average performance on most objectives, although no single method dominates all tasks, highlighting the importance of dynamics modeling in complex, long-horizon plasma control tasks. To foster further research, we open-source the codebase, datasets, and evaluation framework, providing a benchmark not only for the fusion community but also for algorithm development in offline RL.

Yang Fu, Haomin Bao, Rohit Sonker +4

Central South University · Chongqing University · Carnegie Mellon University +1

RL Benchmarks Offline RL Model-Based RL

Ask PDF

Abstract

Feedback Control

Explore similar work

Dynamic Plasma Shape Control with Arbitrary Sensor Subsets

Advantage-level Aggregation Reinforcement Learning for X-point Target Magnetic Configuration Control in an EXL-50U Experiment-Calibrated Simulation Environment

Offline Reinforcement Learning for Plasma Control in Nuclear Fusion: Codebase and Benchmark