Advantage-level Aggregation Reinforcement Learning for X-point Target Magnetic Configuration Control in an EXL-50U Experiment-Calibrated Simulation Environment

Siqi Ding; Xuanhe Wang; Pei Guo; Guoyang Shi; Changquan Yu; Yiting Wang; Xianming Song; Xiang Gu

physics.plasm-phAug 21, 2026

Advantage-level Aggregation Reinforcement Learning for X-point Target Magnetic Configuration Control in an EXL-50U Experiment-Calibrated Simulation Environment

Authors: Siqi Ding, Xuanhe Wang, Pei Guo, Guoyang Shi, Changquan Yu, Yiting Wang, Xianming Song, Xiang Gu,

Organizations: Beijing ENN Fusion Energy Science and Technology Co., Ltd., Beijing 101111, China · Beijing Key Laboratory of High Magnetic Field Spherical Torus Fusion Energy, Beijing 101111, China · Hebei Key Laboratory of Compact Fusion, Langfang 065000, China · Key Laboratory of Materials Modification by Beams of the Ministry of Education, School of Physics, Dalian University of Technology, Dalian 116024, China ·

Abstract

Managing divertor heat loads is a central challenge for compact, high-power tokamaks. To increase local flux expansion and decouple the dissipation volume from the core, EHL-2 adopts the X-point target (XPT) divertor. This requires the secondary X-point to remain on the divertor leg; displacement degrades the topology and exhaust geometry. Current experiments, including EXL-50U discharges, rely on precomputed feedforward waveforms with PID loops on global quantities. Lacking dedicated closed-loop feedback for the secondary null, XPT operation is repeatable but not routine. We formulate XPT feedback as a multi-objective reinforcement learning (RL) control problem in a free-boundary environment calibrated to EXL-50U discharge #13906. To address strong coupling among plasma current, shape, and null constraints - where reward scalarisation collapses objective-specific temporal credit - we develop Advantage Aggregation (AdvA). AdvA preserves objective-wise temporal credit before worst-objective-aware nonlinear scalarisation and introduces a residual correction to policy updates. AdvA-PPO is evaluated against Reward-PPO and a feedforward-plus-PID baseline under nominal operation, measurement uncertainties, and unseen initial equilibria. On a 500 ms rollout, AdvA-PPO raises the mean worst-channel score from 0.23 to 0.81 over Reward-PPO, reducing X-point flux RMSE by ~20x. Under combined measurement uncertainties, it is the only learned controller completing the horizon while retaining a usable XPT shape. Multi-initialization fine-tuning enables a single AdvA-PPO policy to complete full-horizon operation across divertor and limiter initial equilibria. These results provide a simulation-based foundation for future real-time XPT validation on EXL-50U.

Figures & tables

Figure 1: Representative experimental XPT equilibrium in EXL-50U discharge #13906 at $t=500\,\mathrm{ms}$ . The reconstructed magnetic topology demonstrates experimental access to XPT operation.

Device	Method	FB targets	Real-time secondary null in FB
TCV [ 14 , 7 , 8 , 10 ]	Divertor shape control	$I_{p}$ , boundary / shape	No
MAST-U [ 13 , 9 ]	LEMUR + FF $B_{r,z}\!\to\!0$	$R_{\mathrm{OUT}}$ , $R_{\mathrm{IN}}$ , $Z_{X}$	No
EXL-50U (this exp.)	FF+PID	$I_{p}$ , $R$ , $Z$	No

Table 1: Experimental XPT magnetic control to date. “Real-time secondary null in FB” asks whether a real-time estimate of the secondary null enters the feedback law (as opposed to feedforward coil programming alone).

Figure 2: End-to-end architecture for reconstructed-null XPT control. Simulation closes the loop through FGE; deployment swaps FGE observations for real-time reconstruction while keeping the same state and coil-command interfaces (11 policy coils; VS under PID).

Figure 3: Free-boundary equilibria from the FGE validation run of EXL-50U discharge #13906 at

t=0.50

0.55

0.63

, and

0.69\,\mathrm{s}

(left to right). Each panel overlays the simulated separatrix (solid blue) and X points (blue crosses) on the experimental LIUQE-reconstructed separatrix (dashed black) and X points (open circles); thin gray curves show simulated flux surfaces and the dark gray contour marks the limiter.

Figure 4: Time traces from the FGE validation run of EXL-50U discharge #13906 (solid blue) compared with the experimental LIUQE reconstruction (dashed black): (a) plasma current

I_{p}

; (b) plasma radial position

R

; (c) outer radial extent

R_{\max}

of the LCFS; (d) LCFS elongation

\kappa

Quantity	Bias	RMSE	Maximum
$I_{p}$ (kA)	$+1.4$	3.1	6.5
$R$ (mm)	$+4.7$	5.3	11.4
$R_{\max}$ (mm)	$+7.6$	8.7	16.7
$\kappa$	$+0.045$	0.051	0.101
LCFS distance (mm)	—	13.8	30.5
Primary X points $X_{2}$ , $X_{3}$ (mm)	—	18.3	38.0

Table 2: Deviations between the FGE validation run and the LIUQE reconstruction of discharge #13906 over

0.5

–

0.7\,\mathrm{s}

: time-averaged signed deviation (bias, simulation minus reconstruction), root-mean-square error (RMSE), and maximum absolute deviation. The LCFS row is based on the symmetrized mean distance between the simulated and reconstructed boundary contours, and the X-point rows on the Euclidean position offsets averaged over the upper and lower X point of each type; these distances are nonnegative by construction, so no bias is reported.

Figure 5: Baseline FF+PID control structure.

Figure 6: Policy update and environment interaction for AdvA-PPO. The FGE environment provides the reconstructed magnetic observation and component rewards. Per-objective value heads and GAE preserve temporal credit assignment before SmoothMax aggregation; the PPO anchor is then corrected by the norm-capped residual of Eqs. ( 15 )–( 16 ). Reliability gating and gradient projection are ablation options and are off in AdvA-PPO.

Name	Locus	$\alpha$	Gate	$\beta$	Proj.	$K$	$d_{a}/d_{o}$
Reward-PPO	reward	$-3$	off	0	off	10	$11/45$
Adv-mean	advantage	$0$	off	0	off	10	$11/45$
Adv-SmoothMax	advantage	$-3$	off	0	off	10	$11/45$
$+$ gate	advantage	$-3$	on	0	off	10	$11/45$
$+$ resid (AdvA-PPO)	advantage	$-3$	off	$0.2$	off	10	$11/45$
$+$ proj	advantage	$-3$	off	$0$	on	10	$11/45$

Table 3: Controllers in the component ablation, in the same order as the Results ablation (Sec. 3.2 ). “Locus” is where nonlinear aggregation occurs relative to objective-wise temporal credit. For AdvA rows, Gate/

\beta

/Proj. act on advantage weights or the residual of Eqs. ( 15 )–( 16 ). All rows share the same

d_{a}{=}11

d_{o}{=}45

interface (Sec. 2.3.2 ; VS under PID). Reward-PPO is the reward-level SmoothMax baseline of Sec. 2.3.4 .

ID	Shot	$t_{0}$	Topology	$I_{p}^{\star}$
I1	#13906	$\approx 500\,\mathrm{ms}$	XPT (training / test)	$\approx 400\,\mathrm{kA}$
I2	#13705	$\approx 500\,\mathrm{ms}$	divertor (test)	$\approx 500\,\mathrm{kA}$
I3	#13844	$\approx 500\,\mathrm{ms}$	divertor (test)	$\approx 500\,\mathrm{kA}$
I4	#15892	$\approx 500\,\mathrm{ms}$	limiter (test)	$\approx 500\,\mathrm{kA}$

Table 4: Evaluation initials I1–I4. All windows start at

t_{0}\approx 500\,\mathrm{ms}

(CSV start

0.502\,\mathrm{s}

) and use the frozen #13906 XPT geometric target;

I_{p}^{\star}

is the current reference for that cell.

Figure 7: Closed-loop evolution on #13906 (

500\,\mathrm{ms}

). Grey: FF+PID; blue: Reward-PPO; red: AdvA-PPO. (a)

I_{p}

; (b) LCFS error; (c)–(d) primary

X

distance and flux (

X_{2}

X_{3}

); (e)–(f) secondary

X

distance and flux (

X_{1}

X_{4}

Figure 8: Flux snapshots on #13906 at

t=500

–

900\,\mathrm{ms}

(shot clock; control from

t_{0}=500\,\mathrm{ms}

). Rows: FF+PID, Reward-PPO, AdvA-PPO. Blue solid: current LCFS; black dashed: target; open circles: target

X

; blue crosses: actual

X

Quantity	FF+PID	Reward-PPO	AdvA-PPO (ours)
$I_{p}$ (A)	454 / 1488	735 / 1310	260 / 624
Primary $X$ dist. (mm)	39.0 / 58.3	22.8 / 39.2	6.3 / 9.5
Secondary $X$ dist. (mm)	30.1 / 38.3	19.0 / 31.8	28.2 / 44.2
Mean $X$ flux ( $10^{-4}\,\mathrm{Wb}$ )	77.38 / 117.02	13.06 / 18.89	0.66 / 1.93
LCFS ( $\times 10^{-3}$ )	120.1 / 430.3	14.7 / 21.9	12.2 / 17.2
SmoothMax $\bar{u}$ $\uparrow$	0.093	0.563	0.925

Table 5: Nominal tracking on I1 (#13906,

500\,\mathrm{ms}

); metrics as in Sec. 2.4.2 . FF+PID X-point and LCFS rows are open-loop observations.

Variant	$I_{p}$ (A)	mean $d_{X}$ (mm)	mean flux ( $10^{-4}\,\mathrm{Wb}$ )	LCFS ( $\times 10^{-3}$ )	$\bar{u}$ $\uparrow$	$\overline{r}$ $\uparrow$
Reward-PPO	735 / 1310	20.9 / 35.5	13.06 / 18.89	14.7 / 21.9	0.563	0.232
Adv-mean	1615 / 2149	34.2 / 45.8	1.77 / 2.70	8.7 / 14.9	0.830	0.665
Adv-SmoothMax	1630 / 2545	17.6 / 27.3	2.44 / 5.22	13.4 / 18.0	0.879	0.760
+ gate	2779 / 3384	18.5 / 29.0	1.30 / 2.76	15.3 / 21.7	0.899	0.760
+ resid $=$ AdvA-PPO	260 / 624	17.3 / 26.9	0.66 / 1.93	12.2 / 17.2	0.925	0.810
+ proj	1601 / 1916	18.9 / 29.7	1.27 / 2.10	8.5 / 11.3	0.894	0.765

Table 6: Component ablation on I1 (#13906,

500\,\mathrm{ms}

). Names as in Table 3 ; metrics as in Sec. 2.4.2 .

	Method	Surv. (ms)	$I_{p}$ (A)	Prim. $X$ (mm)	Prim. flux ( $10^{-4}\,\mathrm{Wb}$ )	Sec. $X$ (mm)	Sec. flux ( $10^{-4}\,\mathrm{Wb}$ )	LCFS ( $\times 10^{-3}$ )
R1: delay $1$ – $3\,\mathrm{ms}$
	FF+PID	500	637	38.1	2.16	29.6	149.81	160.8
	Reward-PPO	500	11997	23.0	28.38	41.1	36.88	229.6
	AdvA-PPO	500	4521	26.6	13.04	24.8	17.85	18.8
R2: full diagnostic noise
	FF+PID	500	74817	46.1	23.02	22.3	55.54	222.0

Table 7: Zero-shot noise/delay on I1 (#13906); RMSE over the survived horizon (Sec. 2.4.2 ).

Figure 9: R3: delay + noise on I1 (#13906). Panel layout as Figure 7 .

Figure 10: Zero-shot cross-init flux under frozen #13906 XPT. Rows: #13705, #13844, #15892. Columns: initial equilibrium and FF+PID / Reward-PPO / AdvA-PPO at episode end. Grey: initial LCFS/

X

; blue: final; black dashed/circles: target.

Init.	Controller	Surv.	$I_{p}$ (kA)	mean $d_{X}$ (mm)	mean flux ( $10^{-4}\,\mathrm{Wb}$ )	LCFS ( $\times 10^{-3}$ )	$\bar{u}$ $\uparrow$	$\overline{r}$ $\uparrow$
I2 #13705 divertor	FF+PID	full	9.4 / 42.3	125 / 178	1315 / 5054	232 / 596	0.02	0.00
	Reward-PPO	full	32.8 / 79.3	85 / 179	889 / 5013	46 / 70	0.07	0.00
	AdvA-PPO	full	31.8 / 75.6	69 / 179	909 / 5187	63 / 129	0.11	0.01
I3 #13844 divertor	FF+PID	full	2.8 / 6.1	126 / 174	1305 / 5009	97 / 119	0.02	0.00
	Reward-PPO	full	19.6 / 56.8	72 / 168	990 / 5333	50 / 110	0.10	0.00
	AdvA-PPO	full	17.6 / 57.3	60 / 176	1061 / 5781	79 / 241	0.14	0.00

Table 8: Zero-shot cross-init to frozen #13906 XPT (I2–I4); metrics as in Sec. 2.4.2 (full survival

=500\,\mathrm{ms}

). FF+PID feedforward is re-anchored as described in the text.

Figure 11: AdvA-PPO multi-initialization fine-tune (

100

iterations, frozen #13906 XPT). Grey dashed: zero-shot; red: fine-tuned. Top:

\bar{u}

; bottom: mean

d_{X}

. Early stop marked by a dotted vertical line.

Init.	Controller	Surv.	$I_{p}$ (kA)	mean $d_{X}$ (mm)	LCFS ( $\times 10^{-3}$ )	$\bar{u}$ $\uparrow$	$\overline{r}$ $\uparrow$
I1 #13906 (XPT / training)	zero-shot	full	0.3	17	12	0.92	0.81
	fine-tuned	full	0.7	24	17	0.77	0.58
I2 #13705 (divertor)	zero-shot	full	31.8	69	63	0.11	0.01
	fine-tuned	full	26.2	67	64	0.11	0.00
I3 #13844 (divertor)	zero-shot	full	17.6	60	79	0.14	0.00
	fine-tuned	full	13.7	58	66	0.13	0.00

Table 9: AdvA-PPO before/after multi-initialization fine-tuning (

100

iterations, frozen #13906 XPT). Errors and

\bar{u}

\overline{r}

on the common window of each pair; survival separate. On #15892 the common window is the

96

zero-shot steps; over its own

500\,\mathrm{ms}

the fine-tuned run reaches

\bar{u}=0.08

I_{p}

RMSE

20.8\,\mathrm{kA}

, mean

d_{X}=74\,\mathrm{mm}

RL Control

Reinforcement learning control uses reward-driven learning to make decisions in dynamic systems, from robots and vehicles to energy and infrastructure.

See all 274 papers

Reinforcement Learning Controllers for Fusion Scenario Development
Oct 19, 2026Evgeny Adishchev, Maxim Nurgaliev, Dmitry Sorokin +5Feedback Control RL Control
Next Step Fusion s.a.r.l.
Dynamic Plasma Shape Control with Arbitrary Sensor Subsets
May 15, 2026D. Sorokin, M. Stokolesov, A. Granovskiy +7RL Control Partially Observable RL
Next Step Fusion, Bertrange, L-8070, Luxembourg · Center for Energy Research, University of California San Diego, CA 92093, USA
Offline Reinforcement Learning for Rotation Profile Control in Tokamaks
May 7, 2026Rohit Sonker, Hiro Josep Farre Kaga, Jiayu Chen +5RL Control Offline RL
1Robotics Institute, Carnegie Mellon University · 2Princeton University · 4The University of Hong Kong +2

Oct 19, 2026J/K move · Enter open · S save

Reinforcement Learning Controllers for Fusion Scenario Development

Evgeny Adishchev, Maxim Nurgaliev, Dmitry Sorokin, Alexander Granovsky+4

Next Step Fusion s.a.r.l.

Feedback Control RL Control Goal-Conditioned RL

Open paper AskPDF arXiv

Scenario development, the design of the waveforms that take a tokamak discharge from breakdown to flat-top, is still an iterative, labour-intensive process. Every candidate scenario has to be simulated with a controller in the loop, and the classical shape and current controllers need low-level actuator references. So the process is iterative and includes a number of trial-and-error steps. We propose to replace the per-scenario controller tuning by a single goal-conditioned reinforcement-learning controller trained once, in advance, against a free-boundary equilibrium and transport simulator. A scenario is then specified only by the target waveforms of the physical quantities of interest (plasma current, line-averaged electron density, LCFS shape, x-point positions), and the controller produces the coil-current and fueling commands that realize them. We demonstrate the approach on an EAST plasma-current ramp-up: the controller takes over a small limited plasma, ramps the current and the density, grows the cross-section, forms two x-points and holds a double-null diverted configuration against sampled shape targets for the rest of a 1.2 s episode. The neural network controller is trained using TQC algorithm. On 1024 test scenarios sampled from a low-discrepancy scrambled Sobol sequence in 11-dimensional target space, the controller forms the divertor in 86% of the cases, holds it for more than 0.5 s in 67% and reaches the end of the episode in a diverted state in 58%, with flat-top tracking errors of a few kA in plasma current,

\sim 0.15\times10^{19}

^{-3}

in density,

\sim 2

cm in position and

\sim 8

cm in x-point location; the failures concentrate at the high-current density (

J

) region which is believed to be physically or from engineering perspective not feasible within ohmic mode.

Abstract

RL Control

Explore similar work

Reinforcement Learning Controllers for Fusion Scenario Development

Dynamic Plasma Shape Control with Arbitrary Sensor Subsets

Offline Reinforcement Learning for Rotation Profile Control in Tokamaks