hep-exNov 1, 2026

Reinforcement Learning in High-Energy Physics: A Survey from Accelerator Control to Theory Exploration

Authors: Luca Anzalone

Organizations: Independent researcher

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reinforcement learning (RL) is a machine learning paradigm in which an agent learns a decision-making policy by interacting with an environment, guided by a scalar reward rather than explicit supervision. Supervised and unsupervised learning have become indispensable tools in high-energy physics (HEP), but they learn from fixed datasets, whereas many HEP tasks are closed-loop decisions: the data an algorithm observes depend on its choices, the feedback is sparse or delayed, and the objective is often measured on a real machine or in a stochastic simulation, not by a differentiable model. This survey reviews the application of RL to such tasks: accelerator control and feedback, data quality monitoring, jet grooming, clustering, and representation, charged particle tracking, beyond-the-Standard-Model (BSM) and string model building, detector design, trigger control, analysis, and formal quantum field theory computation. It provides a self-contained introduction to the underlying RL methods, from model-free and model-based to multi-agent, offline, inverse, and bandit approaches, reviews each domain against its traditional methods, and summarizes the head-to-head comparisons. The applications range from policies deployed on real accelerators and evaluated on recorded collision data to exploratory studies in formal theory. RL is most advantageous for repeated tasks with a fast simulator, decisions inside fast control loops, operation under faults and corrupted measurements, physics-based rewards in place of labels, and sequential discrete constructions with a reusable policy, while Bayesian optimization, genetic algorithms, and differentiable methods remain competitive elsewhere. We conclude with the open challenges that recur across domains, including reward design, sim-to-real transfer, safety, the lack of shared benchmarks, and the statistical validity of policy-driven decisions.

Figures & tables

Figure 1: RL decision points along the HEP workflow, each labeled with the decision taken by the agent and the section that reviews it. Run-time applications act on the accelerator and on the data stream, offline applications act on recorded data and on theory, and detector design precedes data taking. Dashed arrows denote indirect influence: the designed detector determines the data that reach the trigger, and theory predictions inform the analyses that constrain them.
Figure 2: Modular view of an RL system, labeled with the subsections that cover each component. The policy of the agent acts on the environment through an optional safety layer and receives observations; an optional learned model supports planning. The reward is either designed or learned from demonstrations by inverse RL, while imitation learning maps demonstrations to the policy directly. The environment is a simulator or the real system, bridged by sim-to-real transfer, and experience comes either from online interaction through a replay buffer or from a fixed dataset in offline RL. The R badges depict that the reward is computed from environment transitions; the P badges indicate where physics knowledge enters in physics-informed RL.
Figure 3: Generalized RL control loop for accelerators and physics equipment. Solid arrows: online control loop. The encoder maps the diagnostics to an observation oto_{t} , the policy πθ\pi_{\theta} (deployed on a CPU or FPGA) selects actuator changes ata_{t} given the operator target gg , and a safety layer clips actions or resets the episode. The human operator monitors the machine, sets the target, and can override the agent. Dashed arrows: training. The learner updates θ\theta from rollouts of a dynamics model (a physics simulator, or a surrogate fitted to logged data), from replayed real transitions, or both, under a reward shaped from beam quality.
Facility; task; algorithmObservation ooAction aaReward rrTraining and transfer
CERN AWAKE & LINAC4; trajectory correction; PER-NAF [ 126 ]BPM readings (orbit vector)Corrector current increments−RMSE(xmeas, xref)-\mathrm{RMSE}(\mathbf{x}_{\mathrm{meas}},\,\mathbf{x}_{\mathrm{ref}})Online on the real machine; separate sim-trained agent deployed zero-shot
Fermilab Booster; magnet regulation; DQN [ 19 ]LSTM surrogate outputDiscretized 1-D current−∣et∣-|e_{t}| (regulation error)LSTM surrogate learned from real logs
KARA; betatron oscillation damping; PPO [ 20 ]Last 8 BPM samplesStripline kicker signalL2 penalty on beam positionOnline on the real machine; 2.8 \SIUnitSymbolMicros2.8\text{\,}\mathrm{\SIUnitSymbolMicro s} inference
LHC; tune feedback; NAF, PPO [ 21 ]Horizontal & vertical tune errors16 quadrupole current changesNegative mean squared tune errorGym sim of the feedback loop; sim only
CAFe II; orbit correction; trend-based SAC [ 127 ]BPM readings ++ two-step trendCorrector increments−RMS-\mathrm{RMS} orbit ++ per-BPM trend termPhysics sim; zero-shot
DESY ARES; beam focusing under partial observability; TD3 (RLO) [ 107 ]Actuator values ++ beam params ++ target b′b^{\prime}Δ\Delta actuator setpointsSigned log-MAE improvement; asymmetric penaltyCheetah; DR over II , MM ; zero-shot
Table 1: MDP formulations of representative reinforcement learning approaches reviewed in Section 3 ; baselines and outcomes are reported in Table 3 . Symbols follow Figure 3 : observation oo , action aa , and reward rr ; all action spaces are continuous except for the Fermilab Booster, where the current is discretized. Training and transfer : environment used for policy training (physics simulator, learned or empirical surrogate, or real machine) and strategy used to bridge it to the real machine. DR: domain randomization; FT: fine-tuning on the target lattice; RMS: root mean square; RMSE: root mean squared error; MAE: mean absolute error; Δf\Delta f : step-wise improvement in beam quality metric; RLO: reinforcement learning-trained optimization; DDRL: deep differentiable reinforcement learning; TES: transition-edge sensor.
Figure 4: Human-in-the-loop and bandit formulations of data quality monitoring. Top: the multi-agent PPO system of Jullian Parra et al. [ 171 ] , in which a predictor flags each histogram as good or bad and a checker decides whether to query the human shifter, whose label updates the predictor. Bottom: the adversarial thresholding semi-bandit of Bower [ 172 ] , which selects and queries channels of the ALICE Transition Radiation Detector (TRD), receives feedback only above the threshold τ\tau , and raises early alarms to the operator. In the first case the human supplies the learning signal; in the second, the agent supplies alerts to the human.
SystemMethodSupervisionOnline adaptationValidation
Threshold / χ2\chi^{2} / KS testsRule-basedNoneNoReal data; all LHC experiments
Pol et al. [ 166 ]Supervised ++ autoencoderPassive; full labels (known faults)NoReal data; CMS muon detectors
AutoDQM [ 167 ]Unsupervised autoencoderNoneNoReal data; CMS Run 3
Asres et al. [ 168 ]Transfer-learning autoencoderPassive; weak labels (source domain)NoReal data; CMS HCAL
MEDIC [ 169 ]Supervised neural network, simulation-drivenPassive; full labels (simulated)NoSimulation; generic collider detector
Bower [ 172 ]Adversarial semi-banditActive; per-channel reward, no labelsYes, with regret boundReal data; ALICE TRD
Table 2: Comparison of data quality monitoring approaches discussed in Section 4 , spanning traditional automated methods and reinforcement learning-based systems; the outcomes of the RL-based systems are also reported in Table 3 . Supervision : source of the supervisory signal during operation (passive labels, active queries or rewards, or none) and the labels required for deployment (full: fully labeled dataset; weak: weakly labeled or reference histograms only). Online adaptation : whether the system updates its model during operation without full retraining, and whether formal guarantees are provided. Validation : data on which the method was evaluated, and the experiment or subsystem.
Figure 5: Emergent construction pattern in RL for HEP: an agent builds a physics object edit by edit, as in several of the applications of Sections 5 to 9 . The agent, a policy followed by a feasibility mask, applies an edit aa to the partial object ss ; the edited object becomes the next state s′s^{\prime} and is scored by an evaluator, whose reward trains the policy. An optional search procedure or prior guides the policy, and the complete object is returned as output. Subtitles give examples from the reviewed applications.
WorkTaskRL methodBaselineEnv.Outcome
Accelerator control (Section 3 )
Kain et al. [ 126 ]AWAKE, LINAC4 trajectory correctionPER-NAFnoneReal; Sim →\to Realn/a
O’Shea et al. [ 131 ]FERMI FEL intensity; THz sourcePolicy gradientnoneRealn/a
Bruchon et al. [ 132 ]FERMI seed laser alignmentQ-learning; NPG REINFORCEnoneRealn/a
Kaiser et al. [ 107 ]ARES transverse tuningTD3classical optimizers incl. Nelder–Mead; human operatorsSim →\to Real++ (Nelder–Mead equal final error, more steps); == vs. operators, faster
Kaiser et al. [ 18 ]ARES focusing and steeringTD3 (RLO)Bayesian optimizationSim →\to Real++ (lower final error in most real trials; much faster inference)
Table 3: Summary of the RL works reviewed in Sections 3 to 9 that report empirical results. Baseline : the method the RL approach is compared against in the reviewed work or, where cited, in a closely related study. Env. : environment in which the reported results are obtained. Sim: simulation; Surr: learned or empirical surrogate of the system; Real: real machine or detector; Data: recorded experimental data; Formal: exact theoretical evaluation; Data fit: likelihood against experimental measurements; →\to : training in the first, evaluation in the second; FT: fine-tuning. Outcome of RL relative to the baseline: ++ better; == comparable or complementary; −- baseline better; R: validated only against a reference configuration or a known solution; n/a: no comparison reported. Position papers and studies validated only on standard RL benchmarks are omitted.
More in this topic

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,767 papers

Explore similar work

CardsList
Jun 22, 2026

Learning to Trigger: Reinforcement Learning at the Large Hadron Collider

High-throughput scientific facilities such as the Large Hadron Collider depend on real-time event filtering (triggering) under tight constraints on bandwidth, latency, and storage. In practice, trigger menus are largely static and hand-tuned and can become suboptimal as detector conditions, pileup, and background composition drift over time. We cast online threshold tuning as a sequential decision-making problem: a reinforcement learning agent ingests streaming summaries of recent rates and signal-sensitive features and updates trigger thresholds to maximize signal efficiency while tracking a target background rate within a tolerance band. We adapt Group-Filtered Policy Optimization (GFPO) to streaming control and introduce two variants (GFPO-F, GFPO-FR) that enforce background rate feasibility during training. On a benchmark that emulates realistic collider operation, we study two representative triggers: a total transverse energy (HTH_{T}) trigger sensitive to pileup variation, and an anomaly-detection (AD) trigger based on reconstruction loss for rare or non-standard signatures. On Monte Carlo streams, our agent increases the fraction of in-tolerance time intervals by 48% (HTH_T) and 28% (AD), with a cumulative gain of up to 2% in signal efficiency on those in-tolerance intervals. Transferring from simulation to real collision data (CMS Run 283408), the same agent, without fine-tuning, achieves a 56% (HTH_T) and 28% (AD) in-tolerance improvement over baselines, with further signal-efficiency gain on both triggers. To our knowledge, this is the first demonstration of RL-based trigger control on real Large Hadron Collider collision data. Code is available at https://github.com/Zixind/GFPO_LHC (see repo for details).

Zixin Ding, Shaghayegh Emami, Giovanna Salvi +7

University of Chicago · University of Michigan, Ann Arbor · Fermi National Accelerator Laboratory

RL ControlNon-Stationary RLConstrained RLHigh-Energy Physics
AskPDF
Apr 21, 2026

RL-ABC: Reinforcement Learning for Accelerator Beamline Control

Particle accelerator beamline optimization is a high-dimensional control problem traditionally requiring significant expert intervention. We present RLABC (Reinforcement Learning for Accelerator Beamline Control), an open-source Python framework that automatically transforms standard Elegant beamline configurations into reinforcement learning environments. RLABC integrates with the widely-used Elegant beam dynamics simulation code via SDDS-based interfaces, enabling researchers to apply modern RL algorithms to beamline optimization with minimal RL-specific development. The main contribution is a general methodology for formulating beamline tuning as a Markov decision process: RLABC automatically preprocesses lattice files to insert diagnostic watch points before each tunable element, constructs a 57-dimensional state representation from beam statistics, covariance information, and aperture constraints, and provides a configurable reward function for transmission optimization. The framework supports multiple RL algorithms through Stable-Baselines3 compatibility and implements stage learning strategies for improved training efficiency. Validation on a test beamline derived from the VEPP-5 injection complex (37 control parameters across 11 quadrupoles and 4 dipoles) demonstrates that the framework successfully enables RL-based optimization, with a Deep Deterministic Policy Gradient agent achieving 70.3% particle transmission -- performance matching established methods such as differential evolution. The framework's stage learning capability allows decomposition of complex optimization problems into manageable subproblems, improving training efficiency. The complete framework, including configuration files and example notebooks, is available as open-source software to facilitate adoption and further research.

Anwar Ibrahim, Fedor Ratnikov, Maxim Kaledin +2

RL ControlHigh-Energy Physics
AskPDF
Sep 9, 2026

Searching for New Physics with Reinforcement Learning

Finding new physics (NP) is the most important problem in particle physics today. Studying ``anomalies'', i.e., measurements of low-energy observables whose values disagree with the predictions of the Standard Model (SM), is a powerful search strategy. The SM Effective Field Theory (SMEFT) provides a general model-independent framework for parameterizing NP; it is natural to try to find the SMEFT operator(s) that can explain such anomalies. This is a challenging task because (i) the number of SMEFT operators is enormous, and (ii) at loop level there are very complicated correlations among the operators. Analyses by humans typically rely on phenomenological intuition to decide which operators are relevant. This is often biased and does not explore the complete SMEFT operator space. Interestingly, reinforcement learning (RL) techniques excel at tasks that require decision making to achieve their goals. In this paper, we introduce an RL method that can be used to find the SMEFT operators that explain any anomalies. We test it on the CDF WW-mass anomaly, and show that it reproduces (and improves upon) known results. We then consider a far more complicated situation with multiple anomalies and show that, even here, this method is able to find the SMEFT operators that explain the data. Our RL method can therefore be used to efficiently search for NP at the level of SMEFT.

Jacky Kumar, Marianne Bouchard, David London

Physique des Particules, Universit´e de Montr´eal, 1375 Avenue Th´er`ese-Lavoie-Roux, Montr´eal, QC H2V 0B3, Canada

High-Energy Physics
AskPDF