Reinforcement Learning Techniques for the Optimization of Target Polarization in Nuclear Physics Scattering Experiments

Armen Kasparian; Torri Jeske; Monibor Rahman; Chris Keith; James Maxwell; Thomas Britton; Malachi Schram; David Lawrence

cs.AIOct 1, 2026

Reinforcement Learning Techniques for the Optimization of Target Polarization in Nuclear Physics Scattering Experiments

Authors: Armen Kasparian, Torri Jeske, Monibor Rahman, Chris Keith, James Maxwell, Thomas Britton, Malachi Schram, David Lawrence

Organizations: Data Science Department, Thomas Jefferson National Accelerator Facility, Newport News, VA 23606 USA · EPSCI, Thomas Jefferson National Accelerator Facility, Newport News, VA 23606 USA · Target Group, Thomas Jefferson National Accelerator Facility, Newport News, VA 23606 USA · Advanced Computing, Mathematics, and Data (ACMD) Division, Pacific Northwest National Lab, Richland, WA 99354 USA

Abstract

The operation of dynamically polarized targets in nuclear physics experiments relies on continuous tuning of the microwave frequency to compensate for radiation damage and evolving material properties, a task that is traditionally performed through manual trial-and-error by expert operators. This work presents a data-driven control framework that combines surrogate modeling with reinforcement learning to optimize the target polarization. Using operational data from the APOLLO cryogenic target system, we train and evaluate multilayer perceptron and Gaussian process regression models to predict polarization as a function of microwave frequency, beam current, and accumulated radiation dose. We show that Gaussian process-based models provide calibrated uncertainty estimates and reliably identify regions outside the training distribution, while MLPs exhibit limited sensitivity to distributional shift. To enable learning and control across multiple target samples, we introduce a Gaussian process approximation and embed the surrogate model within a standardized simulation environment. A reinforcement learning agent is trained using a lower-confidence-bound reward formulation that balances performance maximization against uncertainty. We are able to show an almost 2x improvement on the operators actions utilizing our RL agent.

Figures & tables

Figure 1: Proton polarization as a function of cumulative radiation dose for NH 3 sample #20-3, used during data taking periods P08 through P11. The blue points correspond to positive polarization, and the red points correspond to negative polarization. In general, the rate at which the polarization degrades increases with the cumulative dose the material has received. The spikes in polarization can be due to either sudden changes in beam current or microwave frequency adjustments.

Figure 2: Example polarization extraction for protons in NH 3 . The normalized phase, baseline, baseline-subtracted, and fit-subtracted signals versus frequency are shown. The polarization is proportional to the area under the fit-subtracted curve (shaded region).

Figure 3: Sample response to changes in microwave frequency given the current polarization and dosage.

Figure 4: Schematic representation of the neural network architecture consisting of two dense hidden layers with ReLU activation and dropout regularization. The output layer predicts both the mean ( $\mu$ ) and standard deviation ( $\sigma$ ) of the target distribution for probabilistic regression.

Figure 5: The agent-environment interaction in a reinforcement learning workflow [ 17 ] . The agent executes a policy that selects an action $A_{t}$ given the current $S_{t}$ , which results in a reward $R_{t+1}$ and a new state $S_{t+1}$ of the environment.

Figure 6: Learning curves showing the training and validation loss for the MLP model. The validation loss decreases as the number of epochs increases, and training stops when the minimum validation loss is reached.

Figure 7: Observed and predicted polarization for the MLP model. The red dotted line indicates the ideal prediction, and the blue markers represent the model’s predictions.

Figure 8: Sample-wise observed values, predictions, and prediction intervals. The red dotted line represents the observed values, and the blue markers represent the model predictions. The predictions follow patterns similar to the true values, with associated uncertainty.

Figure 9: Average calibration of predicted vs. observed values for the MLP model.

Figure 10: Optimized NLL loss values for Laplace, RBF, and Cauchy kernels using Powell, Nelder–Mead, L-BFGS-B, CG, and COBYQA optimization methods.

Figure 11: Observed and predicted polarization for the GP model. The red dotted line indicates the ideal prediction, and the blue markers represent the model’s predictions.

Figure 12: Sample-wise observed values, predictions, and prediction intervals. The red dotted line represents the observed values, and the blue markers represent the model predictions. The predictions follow patterns similar to the true values, with associated uncertainty.

Figure 13: Average calibration of predicted vs. observed values for the MLP model.

Figure 14: Sample-wise comparison of observed polarization values and GPA model predictions for the P07, P11, and P14 test datasets. Solid lines indicate the observed values, while markers denote the model predictions. The shaded regions represent the 95% predictive confidence intervals for each individual sample ID.

Figure 15: Average calibration of predicted vs. observed values for the GPA model for P07, P11, and P14.

Panel of Figure 16 — Figure 16: Comparison of MLP, GP, and GPA responses during controlled ID-to-OOD transitions. The MLP and GP models are trained using single-sample P07 data, while the GPA model is trained using multisample data from samples P07, P11, and P14. All models are evaluated on P07 test data, with microwave frequency intentionally varied from ID (solid red line) to OOD region (dashed red line). Shaded regions represent prediction uncertainty. In (a) and (c), the frequency is increased beyond the upper training limit of $140.1822$ GHz, while in (b) and (d), it is decreased below the lower limit of $140.0979$ GHz.

Num	Name	$\Delta$ UWave
0	No action	0
1	Decrease UWave Frequency	$-0.001$
2	Increase UWave Frequency	$+0.001$

Table 1: Discrete Action Space for P07 Environment

Num	Name	$\Delta$ UWave
0	UWave Frequency Adjustment	$[-0.01,+0.01]$

Table 2: Continuous Action Space for P07 Environment

Num	Name	Min	Max
0	Normalized Frequency	$0.0$	$1.0$
1	Scaled Target Dose	$0.0$	$\approx 4.0$
2	Beam Current	$0.0$	$\approx 5.0$
3	Predicted Mean Polarization	$0.0$	$1.0$
4	Predicted Std. Deviation	$0.0$	$0.2$
5	Lower Confidence Bound	$0.0$	$1.0$

Table 3: Observation Space for Proton Polarization Environment

Figure 17: Episode reward curves for DDPG, TD3, and SAC. All three algorithms converge to comparable reward values, indicating that final performance is largely independent of the specific off policy actor-critic algorithm used.

\begin{array}[]{|l|c|}\hline\cr\textbf{Hyperparameter}&\textbf{Value}\\ \hline\cr\text{Warmup Size}&2500\\ \text{Batch Size}&256\\ \text{Critic Learning Rate}&0.0003\\ \text{Actor Learning Rate}&0.0003\\ \text{Tau (Soft Update)}&0.005\\ \text{Discount Factor }(\gamma)&0.99\\ \text{Buffer Capacity}&1{,}000{,}000\\ \text{Buffer Type}&\text{ER}\\ \text{Actor Model}&\text{FCNN}\\ \text{Critic Model}&\text{FCNN}\\ \text{Exploration Noise Fraction}&0.1\\ \hline\cr\textbf{Network Architecture}&\textbf{A\&C}\\ \hline\cr\text{Hidden Layers}&2\\ \text{Nodes per Layer}&[256,256]\\ \text{Activation Functions}&[\text{ReLU},\text{ReLU}]\\ \text{Final Activation}&[\text{Tanh}]\\ \hline\cr\end{array}

Table 4: Default hyperparameters for the TD3 algorithm.

Figure 18: Human operator vs RL learned policy. Human operators actions are matched from the dataset while the RL agent is generating the actions through the policy model. These action are passed through the environment and show that that the total polarization over the episode is about equivalent

Method	Prediction Time (ms)	Slowdown
Single Sample GP	32.94	1.00 $\times$
Multi Sample GPA	1103.38	33.49 $\times$

Table 5: Single-Sample GP vs. Multi-Sample GPA Prediction Time Comparison

Parameter	Value
Temporal Steps	950
Frequency Range	[139.50, 140.25] GHz
Frequency Granularity	0.0001 GHz
Frequency States per Step	7,500
Total State Space	7,125,000

Table 6: Lookup Table Discretization Parameters

Figure 19: Multi sample GPA RL learned policy.

RL Control

Reinforcement learning control uses reward-driven learning to make decisions in dynamic systems, from robots and vehicles to energy and infrastructure.

See all 276 papers

RL-ABC: Reinforcement Learning for Accelerator Beamline Control
Apr 21, 2026Anwar Ibrahim, Fedor Ratnikov, Maxim Kaledin +2RL Control High-Energy Physics
Learning to Trigger: Reinforcement Learning at the Large Hadron Collider
Jun 22, 2026Zixin Ding, Shaghayegh Emami, Giovanna Salvi +7RL Control Non-Stationary RL
University of Chicago · University of Michigan, Ann Arbor · Fermi National Accelerator Laboratory
Advantage-level Aggregation Reinforcement Learning for X-point Target Magnetic Configuration Control in an EXL-50U Experiment-Calibrated Simulation Environment
Aug 21, 2026Siqi Ding, Xuanhe Wang, Pei Guo +10RL Control Feedback Control
Beijing ENN Fusion Energy Science and Technology Co., Ltd., Beijing 101111, China · Beijing Key Laboratory of High Magnetic Field Spherical Torus Fusion Energy, Beijing 101111, China · Hebei Key Laboratory of Compact Fusion, Langfang 065000, China +3

Apr 21, 2026J/K move · Enter open · S save

RL-ABC: Reinforcement Learning for Accelerator Beamline Control

Anwar Ibrahim, Fedor Ratnikov, Maxim Kaledin, Alexey Petrenko+1

RL Control High-Energy Physics

Open paper AskPDF arXiv

Particle accelerator beamline optimization is a high-dimensional control problem traditionally requiring significant expert intervention. We present RLABC (Reinforcement Learning for Accelerator Beamline Control), an open-source Python framework that automatically transforms standard Elegant beamline configurations into reinforcement learning environments. RLABC integrates with the widely-used Elegant beam dynamics simulation code via SDDS-based interfaces, enabling researchers to apply modern RL algorithms to beamline optimization with minimal RL-specific development. The main contribution is a general methodology for formulating beamline tuning as a Markov decision process: RLABC automatically preprocesses lattice files to insert diagnostic watch points before each tunable element, constructs a 57-dimensional state representation from beam statistics, covariance information, and aperture constraints, and provides a configurable reward function for transmission optimization. The framework supports multiple RL algorithms through Stable-Baselines3 compatibility and implements stage learning strategies for improved training efficiency. Validation on a test beamline derived from the VEPP-5 injection complex (37 control parameters across 11 quadrupoles and 4 dipoles) demonstrates that the framework successfully enables RL-based optimization, with a Deep Deterministic Policy Gradient agent achieving 70.3% particle transmission -- performance matching established methods such as differential evolution. The framework's stage learning capability allows decomposition of complex optimization problems into manageable subproblems, improving training efficiency. The complete framework, including configuration files and example notebooks, is available as open-source software to facilitate adoption and further research.

Abstract

RL Control

Explore similar work

RL-ABC: Reinforcement Learning for Accelerator Beamline Control

Learning to Trigger: Reinforcement Learning at the Large Hadron Collider

Advantage-level Aggregation Reinforcement Learning for X-point Target Magnetic Configuration Control in an EXL-50U Experiment-Calibrated Simulation Environment