Reward-Driven Collider Physics Challenges for Self-Improving Agentic AI via Reinforcement Learning
Organizations: Stanford University 450 Jane Stanford Way, Stanford, CA 94305 · SLAC National Accelerator Laboratory Menlo Park, CA 94025
Abstract
Recent advances in agentic AI have demonstrated the potential to automate complex high-energy physics (HEP) workflows. We focus on the regime of HEP tasks where frontier models still struggle to provide correct and complete solutions, and introduce an open-ended optimization task format with continuous, automatically evaluated rewards. Unlike conventional benchmarks focused on task completion, these tasks target reinforcement learning (RL) environments to generate agentic improvement on state- of-the-art HEP challenges. We benchmark frontier reasoning models on several HEP tasks and demonstrate an initial proof of concept in which a small language model improves through RL using automated feedback without human intervention. By connecting challenging HEP problems with verifiable scientific rewards and RL, we aim to advance agentic AI from reproducing established workflows to autonomously discovering better scientific solutions, paving the way toward self-improving scientific agents.
Figures & tables
| Task | Models | Reward range | Reference |
|---|---|---|---|
| HiggsML event selection | 4 | 0.708–0.732 | HiggsML winner Melis (2015) (0.729, rescored); in-house HistGB (0.651) |
| Unfolding (OmniFold +jets) | 5 | 0.793–0.841 | OmniFold Andreassen et al. (2020) ; in-house one-step baseline (0.700); truth-informed oracle (0.988) |
| Anomaly detection (LHCO 2020) | 8 | 0.203–0.911 | LHCO Kasieczka et al. (2021b) ; in-house CWoLa-lite Collins et al. (2018a) (0.436) |
| CaloChallenge photon showers | 8 | 0.000–0.739 | CaloDiffusion Amram and Pedro (2023) (0.72); CaloINN Ernst et al. (2025) (0.64); both rescored using our metric Krause and others (2025) |
| TrackML track reconstruction | 8 | 0.000–0.770 | TrackML Kiehn and others (2019) ; Kaggle winner (0.922), fifth place (0.864) CERN (2018) |
| Diphoton resonance search) | 8 | 0.487–0.782 | In-house reference program (0.689) |
| Stage | Reward | Reward by resonance count | |||
|---|---|---|---|---|---|
| none | one | two | |||
| Qwen3.8-27B | |||||
| 30-min session, execution feedback | 1 | 0.516 | 1.000 | 0.122 | 0.426 |
| Base, single shot | 32 | 0.116 (0.578 ∗ ) | 0.281 (1.000) | 0.029 (0.329) | 0.037 (0.404) |
| Evolution, generation 0 | 32 | 0.382 | 0.805 | 0.163 | 0.177 |
| Evolution, generation 12 | 32 | 0.626 | 0.898 | 0.584 | 0.397 |
| Stage | Setting |
|---|---|
| Models | Qwen3-4B-Instruct-2507 and Qwen3.8-27B, base weights, no thinking. |
| Out of the box | 8 prompt variants 4 samples; turn 1 = initial program, turn 2 = its revision after execution feedback; the paper reports turn 1. |
| Self-sampling pool | 256 chains up to 4 turns with execution feedback; synthetic signals 4–8 the template; kept if reward : 109 programs (27B), 46 (4B). |
| Evolution | population 32, 64 children per generation, 20 generations; 60 % mutation / 40 % crossover; fitness reward on two synthetic variants at real signal strength; 24 elites 8 random survivors. |
| SFT | LoRA rank 32, lr , 2 epochs; targets the model’s own evolved programs with fitness (27B) / (4B): 149 programs 2 prompts (27B), 76 2 (4B). |
| GRPO | from the SFT adapter, lr , 6 prompts 8 rollouts per step, 30 steps (27B) / 50 steps (4B); reward score on the synthetic datasets of the prompt variant. |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Step | What it does | Key numbers |
|---|---|---|
| 1 Inputs and selection | Read photon , , ; compute ; apply the pre-selection declared in data.json . | , crack – vetoed, ; window 105–145 GeV. |
| 2 Signal-width prior | Try signal_template.root in the dataset directory; fall back to a fixed width if absent. The template is never in the dataset directory, so the fallback is always used. The peak widths of the hidden datasets are 1.35–1.50 GeV, so the prior is too narrow by about 40 %. | GeV (always the fallback). |
| 3 Binning | Fine binning tied to the assumed width, far below the detector resolution. | bin GeV, about 600 bins. |
| 4 Global background | Fit to the sidebands by least squares on (not a likelihood fit). A standard smooth continuum, no spurious-signal study. | Polynomial degree 2; degrees 3 and 4 are tried only if the degree-2 fit fails. Sidebands: more than 2 GeV outside the signal region, i.e. or GeV on the hidden datasets (the margin is GeV). |
| 5 Candidate seeding | Residual = data background, boxcar-smoothed; robust noise estimate from the MAD; find_peaks on a descending threshold ladder, the first rung that returns candidates wins. If no rung does, the single highest point is used when it exceeds . | 18-bin (1.2 GeV) boxcar; over the whole window; ladder ; prominence threshold; distance 9 bins (0.6 GeV); search the signal region widened by 2 GeV on each side (116.5–134.5 GeV on the hidden datasets); keep the 10 highest. |
| 6 Local background | For each candidate, refit over the whole window with a region around the candidate removed, so the bump does not inflate its own background; a second signal farther away stays inside the fit. | Exclude GeV ( assumed ); first clipped into the SR. |
AI Agents for Scientific Discovery
AI agents for scientific discovery use language models, tools, and experiments to generate, test, and refine scientific knowledge.
See all 323 papers