We study the problem of controlling instability in the Vlasov--Poisson plasma systems, a fundamental challenge for nuclear fusion control. Motivated by the partial observability of plasma states in practice, we investigate imitation learning algorithms that learn from a fully-observable expert controller, but operate under macroscopic measurements. We theoretically establish a separation between online and offline imitation learning --- even a small behavior cloning error leads to error compounding that exponentially amplifies instability over time. By way of contrast, the online imitation learning loss can polynomially control the stability of rollout trajectories. We then propose a DAgger-style algorithm that learns the stabilizing controller under partial observability. Simulation results on a 1D Vlasov--Poisson system demonstrate that our algorithm can effectively stabilize plasma instabilities over longer time horizons than behavior cloning.
Figures & tables
Figure 1: Evolution of the electric energy E(t)=21∥Et∥Lx22 for the uncontrolled dynamics, the analytic expert, BC policies trained on 400 and 1200 offline trajectories, and DAgger-1. Curves show pointwise empirical means over 10 rollout trajectories with independently sampled random initial phases, and shaded regions denote one pointwise standard deviation across those trajectories. Both BC policies eventually exhibit unstable energy growth, while one round of online aggregation substantially delays the instability and keeps the rollout closer to the expert trajectory over long horizons.
Figure 2: Phase-space density snapshots ft(x,v) at times t∈{0,15,30,45,60,75} for a representative rollout trajectory under five controllers: uncontrolled, expert, BC-400, BC-1200, and DAgger-1. The representative case is chosen so that its final-time electric-energy value is closest to the median across the sampled rollout cases. The uncontrolled dynamics quickly develop large-scale deformation and filamentation. BC-400 eventually reproduces this unstable behavior, while BC-1200 delays but does not remove the onset of kinetic distortion. DAgger-1 remains visibly closer to the expert evolution throughout the rollout, with substantially less phase-space twisting and mode excitation at late times.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Additional rollout visualization for the first sampled initial phase pair (ϕ1,ϕ2) . The format matches Figure 2 .
Figure 4: Additional rollout visualization for the second sampled initial phase pair (ϕ1,ϕ2) . The format matches Figure 2 .
Figure 5: Additional rollout visualization for the third sampled initial phase pair (ϕ1,ϕ2) . The format matches Figure 2 .
Figure 6: Additional rollout visualization for the fourth sampled initial phase pair (ϕ1,ϕ2) . The format matches Figure 2 .
Figure 7: Additional rollout visualization for the fifth sampled initial phase pair (ϕ1,ϕ2) . The format matches Figure 2 .
Method
Median E(60)↓
Mean τ1↑
Expert
3.94×10−7 [ 3.93×10−7 , 3.95×10−7 ]
80.00 [80.00, 80.00]
Instantaneous reconstruction
2.472 [1.143, 4.013]
40.24 [38.67, 42.04]
BC-1200
0.750 [0.648, 1.117]
61.33 [59.48, 63.16]
Noisy-expert BC-1200
0.624 [0.405, 0.864]
61.39 [58.97, 63.78]
DAgger-1
0.0340 [0.0151, 0.0699]
74.78 [73.31, 76.20]
Appendix
Table 1: Expanded paired evaluation in the two-stream setting over 60 held-out initial conditions. The two metrics consistently distinguish targeted learner-induced aggregation from increasing or generically broadening the offline dataset.
Instability setting
BC-1200
DAgger-1
Two-stream
0.750 [0.648, 1.117]
0.0340 [0.0151, 0.0699]
Bump-on-tail
9.79 [8.40, 12.43]
0.381 [0.145, 0.664]
Appendix
Table 2: Median electric energy at t=60 in two distinct kinetic-instability settings; brackets denote 95% case-level bootstrap confidence intervals from 10,000 resamples. In the bump-on-tail experiment, the last stored sample is at t=59.98 and is reported as t=60 .
Figure 8: Electric-energy evolution for the bump-on-tail instability over 60 paired initial conditions. Curves are pointwise medians and shaded bands are interquartile ranges. DAgger-1 remains much closer to the expert than either offline BC policy over the evaluation horizon, although a gap to the exact expert remains at late times.
Figure 9: Full phase-space density f(t,x,v) for a representative paired bump-on-tail rollout at t∈{0,15,30,45,60} . All panels share one color scale. The representative case is selected before inspecting the phase-space images by minimizing the Euclidean distance to (0.5,0.5) in the empirical percentile-rank plane of the final BC-1200 and DAgger-1 energies; this selects case 28. The offline policies develop pronounced deformation at late times, whereas DAgger-1 remains visibly closer to the expert evolution.
We consider the stabilization of Vlasov--Poisson plasma dynamics, a central control problem in nuclear fusion. Our focus is the gap between what an ideal controller would use and what experiments can actually observe: while optimal policy may rely on the full phase-space state, practical feedback is typically limited to sparse macroscopic diagnostics. We therefore study imitation learning methods that distill a fully observed expert policy into controllers operating only on macroscopic measurements. We show the stability guarantees of the learned policy, where the error floor depends on the minimal behavior cloning loss achievable under the observation constraints. We further characterize this minimal loss in terms of a notion of entropy that quantifies the complexity of the initial distribution. Our results demonstrates the theoretical feasibility of learning stabilizing feedback policies for kinetic plasma dynamics from macroscopic observations, and exhibits the adaptivity of the learning approach to low-complexity structures. Through extensive numerical experiments, we validate our theory and show that the learned policies can stabilize the system using only macroscopic observations, within a significantly longer time horizon than non-adaptive baseline controllers.
Xiaofan Xia, Qin Li, Wenlong Mou
Department of Statistical Sciences, University of Toronto† · Department of Mathematics, University of Wisconsin–Madison⋄
Offline reinforcement learning (RL) offers a promising route for developing plasma controllers from historical tokamak data, since online trial-and-error on real devices is costly and risky. However, progress in this direction remains difficult to measure due to the lack of a standardized offline RL benchmark for realistic multi-actuator, long-horizon plasma control problems in nuclear fusion. We introduce RL4F, an Offline Reinforcement Learning Benchmark for Plasma Control in Nuclear Fusion, providing closed-loop evaluation environments and baseline comparisons across four full-profile tracking tasks: rotation, density, temperature, and pressure. The dynamics function underlying the evaluation environment is built from historical discharge data from DIII-D, a real-world Tokamak. We evaluate a broad set of imitation learning and offline RL baselines under a unified protocol. We find that offline model-based RL methods obtain the best average performance on most objectives, although no single method dominates all tasks, highlighting the importance of dynamics modeling in complex, long-horizon plasma control tasks. To foster further research, we open-source the codebase, datasets, and evaluation framework, providing a benchmark not only for the fusion community but also for algorithm development in offline RL.
Yang Fu, Haomin Bao, Rohit Sonker +4
Central South University · Chongqing University · Carnegie Mellon University +1
Plasma shape control in tokamaks requires a real-time controller that tracks dynamically changing shape targets while tolerating diagnostic failures. Classical approaches decompose the problem into equilibrium reconstruction followed by a linear controller, and assume a fixed, fully operational sensor set. We present a reinforcement learning agent that addresses both limitations simultaneously. The agent is trained in NSFsim, a high-fidelity tokamak simulator configured for DIII-D, on a curated dataset of 120 experimental plasma shapes. The shape targets are resampled as random step changes every 0.25 s, exposing the agent to diverse transitions across the full shape envelope. At test time the agent zero-shot tracks dynamic shape sequences; on a held-out static configuration in simulation it achieves a mean shape error of 2.01 cm, and dynamic trajectory following is demonstrated qualitatively in simulation and on the physical device. Diagnostic dropout randomly masks 30% of magnetic sensors per episode, yielding a single policy robust to arbitrary sensor subsets without backup controllers or mode-switching logic. An asymmetric actor-critic architecture with privileged equilibrium information improves value estimation under partial observability; an auxiliary shape reconstruction head on the actor enables end-to-end shape reconstruction from raw diagnostics and serves as an interpretability tool for policy analysis. The policy transfers to experimental DIII-D shots, where it directly commands the coil actuators on two dynamic shape maneuvers, and to the independent GSevolve simulator.
D. Sorokin, M. Stokolesov, A. Granovskiy +7
Next Step Fusion, Bertrange, L-8070, Luxembourg · Center for Energy Research, University of California San Diego, CA 92093, USA