Reinforcement Learning in High-Energy Physics: A Survey from Accelerator Control to Theory Exploration

Luca Anzalone

hep-exNov 1, 2026

Reinforcement Learning in High-Energy Physics: A Survey from Accelerator Control to Theory Exploration

Authors: Luca Anzalone

Organizations: Independent researcher

Abstract

Reinforcement learning (RL) is a machine learning paradigm in which an agent learns a decision-making policy by interacting with an environment, guided by a scalar reward rather than explicit supervision. Supervised and unsupervised learning have become indispensable tools in high-energy physics (HEP), but they learn from fixed datasets, whereas many HEP tasks are closed-loop decisions: the data an algorithm observes depend on its choices, the feedback is sparse or delayed, and the objective is often measured on a real machine or in a stochastic simulation, not by a differentiable model. This survey reviews the application of RL to such tasks: accelerator control and feedback, data quality monitoring, jet grooming, clustering, and representation, charged particle tracking, beyond-the-Standard-Model (BSM) and string model building, detector design, trigger control, analysis, and formal quantum field theory computation. It provides a self-contained introduction to the underlying RL methods, from model-free and model-based to multi-agent, offline, inverse, and bandit approaches, reviews each domain against its traditional methods, and summarizes the head-to-head comparisons. The applications range from policies deployed on real accelerators and evaluated on recorded collision data to exploratory studies in formal theory. RL is most advantageous for repeated tasks with a fast simulator, decisions inside fast control loops, operation under faults and corrupted measurements, physics-based rewards in place of labels, and sequential discrete constructions with a reusable policy, while Bayesian optimization, genetic algorithms, and differentiable methods remain competitive elsewhere. We conclude with the open challenges that recur across domains, including reward design, sim-to-real transfer, safety, the lack of shared benchmarks, and the statistical validity of policy-driven decisions.

Figures & tables

Panel of Figure 1 — Figure 1: RL decision points along the HEP workflow, each labeled with the decision taken by the agent and the section that reviews it. Run-time applications act on the accelerator and on the data stream, offline applications act on recorded data and on theory, and detector design precedes data taking. Dashed arrows denote indirect influence: the designed detector determines the data that reach the trigger, and theory predictions inform the analyses that constrain them.

Panel of Figure 2 — Figure 2: Modular view of an RL system, labeled with the subsections that cover each component. The policy of the agent acts on the environment through an optional safety layer and receives observations; an optional learned model supports planning. The reward is either designed or learned from demonstrations by inverse RL, while imitation learning maps demonstrations to the policy directly. The environment is a simulator or the real system, bridged by sim-to-real transfer, and experience comes either from online interaction through a replay buffer or from a fixed dataset in offline RL. The R badges depict that the reward is computed from environment transitions; the P badges indicate where physics knowledge enters in physics-informed RL.

Panel of Figure 3 — Figure 3: Generalized RL control loop for accelerators and physics equipment. Solid arrows: online control loop. The encoder maps the diagnostics to an observation $o_{t}$ , the policy $\pi_{\theta}$ (deployed on a CPU or FPGA) selects actuator changes $a_{t}$ given the operator target $g$ , and a safety layer clips actions or resets the episode. The human operator monitors the machine, sets the target, and can override the agent. Dashed arrows: training. The learner updates $\theta$ from rollouts of a dynamics model (a physics simulator, or a surrogate fitted to logged data), from replayed real transitions, or both, under a reward shaped from beam quality.

Facility; task; algorithm	Observation $o$	Action $a$	Reward $r$	Training and transfer
CERN AWAKE & LINAC4; trajectory correction; PER-NAF [ 126 ]	BPM readings (orbit vector)	Corrector current increments	$-\mathrm{RMSE}(\mathbf{x}_{\mathrm{meas}},\,\mathbf{x}_{\mathrm{ref}})$	Online on the real machine; separate sim-trained agent deployed zero-shot
Fermilab Booster; magnet regulation; DQN [ 19 ]	LSTM surrogate output	Discretized 1-D current	$-\|e_{t}\|$ (regulation error)	LSTM surrogate learned from real logs
KARA; betatron oscillation damping; PPO [ 20 ]	Last 8 BPM samples	Stripline kicker signal	L2 penalty on beam position	Online on the real machine; $2.8\text{\,}\mathrm{\SIUnitSymbolMicro s}$ inference
LHC; tune feedback; NAF, PPO [ 21 ]	Horizontal & vertical tune errors	16 quadrupole current changes	Negative mean squared tune error	Gym sim of the feedback loop; sim only
CAFe II; orbit correction; trend-based SAC [ 127 ]	BPM readings $+$ two-step trend	Corrector increments	$-\mathrm{RMS}$ orbit $+$ per-BPM trend term	Physics sim; zero-shot
DESY ARES; beam focusing under partial observability; TD3 (RLO) [ 107 ]	Actuator values $+$ beam params $+$ target $b^{\prime}$	$\Delta$ actuator setpoints	Signed log-MAE improvement; asymmetric penalty	Cheetah; DR over $I$ , $M$ ; zero-shot

Table 1: MDP formulations of representative reinforcement learning approaches reviewed in Section 3 ; baselines and outcomes are reported in Table 3 . Symbols follow Figure 3 : observation

o

, action

a

, and reward

r

; all action spaces are continuous except for the Fermilab Booster, where the current is discretized. Training and transfer : environment used for policy training (physics simulator, learned or empirical surrogate, or real machine) and strategy used to bridge it to the real machine. DR: domain randomization; FT: fine-tuning on the target lattice; RMS: root mean square; RMSE: root mean squared error; MAE: mean absolute error;

\Delta f

: step-wise improvement in beam quality metric; RLO: reinforcement learning-trained optimization; DDRL: deep differentiable reinforcement learning; TES: transition-edge sensor.

Panel of Figure 4 — Figure 4: Human-in-the-loop and bandit formulations of data quality monitoring. Top: the multi-agent PPO system of Jullian Parra et al. [ 171 ] , in which a predictor flags each histogram as good or bad and a checker decides whether to query the human shifter, whose label updates the predictor. Bottom: the adversarial thresholding semi-bandit of Bower [ 172 ] , which selects and queries channels of the ALICE Transition Radiation Detector (TRD), receives feedback only above the threshold $\tau$ , and raises early alarms to the operator. In the first case the human supplies the learning signal; in the second, the agent supplies alerts to the human.

System	Method	Supervision	Online adaptation	Validation
Threshold / $\chi^{2}$ / KS tests	Rule-based	None	No	Real data; all LHC experiments
Pol et al. [ 166 ]	Supervised $+$ autoencoder	Passive; full labels (known faults)	No	Real data; CMS muon detectors
AutoDQM [ 167 ]	Unsupervised autoencoder	None	No	Real data; CMS Run 3
Asres et al. [ 168 ]	Transfer-learning autoencoder	Passive; weak labels (source domain)	No	Real data; CMS HCAL
MEDIC [ 169 ]	Supervised neural network, simulation-driven	Passive; full labels (simulated)	No	Simulation; generic collider detector
Bower [ 172 ]	Adversarial semi-bandit	Active; per-channel reward, no labels	Yes, with regret bound	Real data; ALICE TRD

Table 2: Comparison of data quality monitoring approaches discussed in Section 4 , spanning traditional automated methods and reinforcement learning-based systems; the outcomes of the RL-based systems are also reported in Table 3 . Supervision : source of the supervisory signal during operation (passive labels, active queries or rewards, or none) and the labels required for deployment (full: fully labeled dataset; weak: weakly labeled or reference histograms only). Online adaptation : whether the system updates its model during operation without full retraining, and whether formal guarantees are provided. Validation : data on which the method was evaluated, and the experiment or subsystem.

Panel of Figure 5 — Figure 5: Emergent construction pattern in RL for HEP: an agent builds a physics object edit by edit, as in several of the applications of Sections 5 to 9 . The agent, a policy followed by a feasibility mask, applies an edit $a$ to the partial object $s$ ; the edited object becomes the next state $s^{\prime}$ and is scored by an evaluator, whose reward trains the policy. An optional search procedure or prior guides the policy, and the complete object is returned as output. Subtitles give examples from the reviewed applications.

Work	Task	RL method	Baseline	Env.	Outcome
Accelerator control (Section 3 )
Kain et al. [ 126 ]	AWAKE, LINAC4 trajectory correction	PER-NAF	none	Real; Sim $\to$ Real	n/a
O’Shea et al. [ 131 ]	FERMI FEL intensity; THz source	Policy gradient	none	Real	n/a
Bruchon et al. [ 132 ]	FERMI seed laser alignment	Q-learning; NPG REINFORCE	none	Real	n/a
Kaiser et al. [ 107 ]	ARES transverse tuning	TD3	classical optimizers incl. Nelder–Mead; human operators	Sim $\to$ Real	$+$ (Nelder–Mead equal final error, more steps); $=$ vs. operators, faster
Kaiser et al. [ 18 ]	ARES focusing and steering	TD3 (RLO)	Bayesian optimization	Sim $\to$ Real	$+$ (lower final error in most real trials; much faster inference)

Table 3: Summary of the RL works reviewed in Sections 3 to 9 that report empirical results. Baseline : the method the RL approach is compared against in the reviewed work or, where cited, in a closely related study. Env. : environment in which the reported results are obtained. Sim: simulation; Surr: learned or empirical surrogate of the system; Real: real machine or detector; Data: recorded experimental data; Formal: exact theoretical evaluation; Data fit: likelihood against experimental measurements;

\to

: training in the first, evaluation in the second; FT: fine-tuning. Outcome of RL relative to the baseline:

+

better;

=

comparable or complementary;

-

baseline better; R: validated only against a reference configuration or a known solution; n/a: no comparison reported. Position papers and studies validated only on standard RL benchmarks are omitted.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,767 papers

Learning to Trigger: Reinforcement Learning at the Large Hadron Collider
Jun 22, 2026Zixin Ding, Shaghayegh Emami, Giovanna Salvi +7RL Control Non-Stationary RL
University of Chicago · University of Michigan, Ann Arbor · Fermi National Accelerator Laboratory
RL-ABC: Reinforcement Learning for Accelerator Beamline Control
Apr 21, 2026Anwar Ibrahim, Fedor Ratnikov, Maxim Kaledin +2RL Control High-Energy Physics
Searching for New Physics with Reinforcement Learning
Sep 9, 2026Jacky Kumar, Marianne Bouchard, David LondonHigh-Energy Physics
Physique des Particules, Universit´e de Montr´eal, 1375 Avenue Th´er`ese-Lavoie-Roux, Montr´eal, QC H2V 0B3, Canada

Jun 22, 2026J/K move · Enter open · S save

Learning to Trigger: Reinforcement Learning at the Large Hadron Collider

Zixin Ding, Shaghayegh Emami, Giovanna Salvi, Cecilia Tosciri+6

University of Chicago · University of Michigan, Ann Arbor · Fermi National Accelerator Laboratory

RL Control Non-Stationary RL Constrained RL High-Energy Physics

Open paper AskPDF arXiv

High-throughput scientific facilities such as the Large Hadron Collider depend on real-time event filtering (triggering) under tight constraints on bandwidth, latency, and storage. In practice, trigger menus are largely static and hand-tuned and can become suboptimal as detector conditions, pileup, and background composition drift over time. We cast online threshold tuning as a sequential decision-making problem: a reinforcement learning agent ingests streaming summaries of recent rates and signal-sensitive features and updates trigger thresholds to maximize signal efficiency while tracking a target background rate within a tolerance band. We adapt Group-Filtered Policy Optimization (GFPO) to streaming control and introduce two variants (GFPO-F, GFPO-FR) that enforce background rate feasibility during training. On a benchmark that emulates realistic collider operation, we study two representative triggers: a total transverse energy (

H_{T}

) trigger sensitive to pileup variation, and an anomaly-detection (AD) trigger based on reconstruction loss for rare or non-standard signatures. On Monte Carlo streams, our agent increases the fraction of in-tolerance time intervals by 48% (

H_T

) and 28% (AD), with a cumulative gain of up to 2% in signal efficiency on those in-tolerance intervals. Transferring from simulation to real collision data (CMS Run 283408), the same agent, without fine-tuning, achieves a 56% (

H_T

) and 28% (AD) in-tolerance improvement over baselines, with further signal-efficiency gain on both triggers. To our knowledge, this is the first demonstration of RL-based trigger control on real Large Hadron Collider collision data. Code is available at https://github.com/Zixind/GFPO_LHC (see repo for details).

Abstract

Reinforcement Learning

Explore similar work

Learning to Trigger: Reinforcement Learning at the Large Hadron Collider

RL-ABC: Reinforcement Learning for Accelerator Beamline Control

Searching for New Physics with Reinforcement Learning