hep-exApr 14, 2026

Cross-Domain Transfer with Particle Physics Foundation Models: From Jets to Neutrino Interactions

Authors: Gregor Krzmanc, Vinicius Mikuni, Benjamin Nachman, Callum Wilkinson

Organizations: Department of Physics, Stanford University, Stanford, CA 94305, USA · Nagoya University, Kobayashi-Maskawa Institute, Aichi 464-8602, Japan · Fundamental Physics Directorate, SLAC National Accelerator Laboratory, Menlo Park, CA 94025, USA · Lawrence Berkeley National Laboratory, Berkeley, CA 94720, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Future AI-based studies in particle physics will likely start from a foundation model to accelerate training and enhance sensitivity. As a step toward a general-purpose foundation model for particle physics, we investigate whether the OmniLearned and ParticleViT foundation models pretrained on diverse high-Q2Q^2 simulated and real pppp and epep collisions retain useful knowledge to a few-GeV fixed-target neutrino experiment. We process MINERvA neutrino--nucleus scattering events and evaluate pretrained models on two types of tasks: regression of available energy and binary classification of charged-current pion final states (CC1π±\mathrm{CC1π^{\pm}}, CCNπ±\mathrm{CCNπ^{\pm}}, and CC1π0\mathrm{CC1π^{0}}). Pretrained OmniLearned and ParticleViT models outperform similarly sized models trained from scratch at the same compute budget, with the largest gains for OmniLearned on regression and for ParticleViT on classification. When the same transformer architecture is instead initialized from unrelated text pretraining (BERT), this advantage appears only marginally for classification in terms of compute efficiency and not in any way for regression. These results suggest that particle-level foundation models acquire inductive biases that generalize across large differences in energy scale, detector technology, and underlying physics processes, pointing toward detector-agnostic inference in particle physics.

Figures & tables

Figure 1: Validation loss vs. number of cumulative training FLOPs for classification tasks (left) and the regression task (right), using the MINERvA Open Data playlist 1A. The top row compares BERT-small, ParticleViT-small and the MLP baseline, while the bottom row compares all the models pretrained on jets. Pretrained models on jet data (OmniLearned-small and ParticleViT models) achieve lower loss at equivalent compute budgets compared to the same scratch-trained models (OmniLearned-small-rw, ParticleViT-small-rw and ParticleViT-medium-rw models). Pre-trained ParticleViT-small model achieves its lowest validation loss in around a third of the compute required for OmniLearned-small both for classification and regression. Validation loss is reported every 1000 steps. For some models, the curves are shown only from a later point on, as the validation loss has a high variance early in training.
Figure 2: Discrimination metrics for the CC1π±\mathrm{CC1\pi^{\pm}} tagging task as a function of true pion kinematics: area under the precision–recall curve (AUPRC, left column), area under the ROC curve (AUROC, center), and true positive rate at a fixed false positive rate (right column). The top row uses bins in true pion energy; the bottom row uses bins in true pion polar angle. In each bin, positives are true signal events falling in that bin, while the negative sample is the full background, held fixed across bins so that only the in-bin signal population changes. TPR is computed at the baseline false-positive rate matched globally on the full test set; see Fig. 5 for the per-bin matched comparison.
Figure 3: Discrimination metrics for the CC1π0\mathrm{CC1\pi^{0}} tagging task as a function of true pion kinematics: area under the precision–recall curve (AUPRC, left column), area under the ROC curve (AUROC, center), and true positive rate at a fixed false positive rate (right column). The top row uses bins in true pion energy; the bottom row uses bins in true pion polar angle. In each bin, positives are true signal events falling in that bin, while the negative sample is the full background, held fixed across bins so that only the in-bin signal population changes. TPR is computed at the baseline false-positive rate matched globally on the full test set; see Fig. 5 for the per-bin matched comparison.
Figure 4: Discrimination metrics for the CCNπ±\mathrm{CCN\pi^{\pm}} tagging task as a function of invariant mass of the hadronic system WW : area under the precision–recall curve (AUPRC, left column), area under the ROC curve (AUROC, center), and true positive rate at a fixed false positive rate (right column). Efficiency is computed using the baseline’s false-positive rate matched globally on the full test set; see Fig. 5 for the per-bin matched comparison.
Figure 5: Event classification efficiency at the cut-based baseline’s false-positive rate, matched independently in each kinematic bin, for CC1π±\mathrm{CC1\pi^{\pm}} and CC1π0\mathrm{CC1\pi^{0}} (vs. true pion energy EπE_{\pi} ) and CCNπ±\mathrm{CCN\pi^{\pm}} with N≥1N\geq 1 (vs. true hadronic invariant mass WW ). Shaded bands show the spread over four independent training runs. Unlike the globally matched TPR panels in Figs. 2 – 4 , every neural model remains above the cut-based baseline in all bins.
Figure 6: Interquartile range (top) and most probable value (bottom) of the available hadronic energy residual distribution vs. q3q_{3} , for playlist 1A. Smaller IQR indicates better energy resolution; MPV near unity indicates no systematic bias. Events with Eavailabletrue<0.05E^{\mathrm{true}}_{\mathrm{available}}<0.05 GeV are excluded.
Figure 7: Available hadronic energy resolution summary for playlists 1A and 1B. The relative ordering of models is consistent with the single-playlist result in Fig. 6 .
Figure 8: Available hadronic energy residual distributions in four bins of q3q_{3} for muon-selected events (top row: signed residual Eavailablereco−EavailabletrueE^{\mathrm{reco}}_{\mathrm{available}}-E^{\mathrm{true}}_{\mathrm{available}} ; bottom row: ratio Eavailablereco/EavailabletrueE^{\mathrm{reco}}_{\mathrm{available}}/E^{\mathrm{true}}_{\mathrm{available}} ). Pretrained OmniLearned models (OmniLearned-small) produce narrower, more symmetric distributions compared to scratch-trained models and ParticleViT models, especially at low q3q_{3} . Events with Eavailabletrue<0.05E^{\mathrm{true}}_{\mathrm{available}}<0.05 GeV are excluded.
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Event composition in bins of true hadronic invariant mass WW for playlist 1A. Each row corresponds to a different signal definition. Left: signal counts by interaction category. Center: background counts by interaction category. Right: signal-to-background ratio.
Figure 10: Same as Fig. 9 , but reported in bins of the true three-momentum transfer q3q_{3} .
Figure 11: Composition of the single-pion signals versus true pion energy EπE_{\pi} and polar angle θπ\theta_{\pi} , together with the corresponding background composition, for playlist 1A.
Model# paramsArchitecture summary
OmniLearned-small3MParticle-encoder transformer (PET2), as described in [ 18 ] .
OmniLearned-medium (OL-medium-frozen)52MParticle-encoder transformer (PET2) [ 18 ] with frozen backbone ( ∼\sim 21M trainable parameters).
ParticleViT-small9.7MParticleViT (embedding variant): width 448, 4 transformer layers, 7 attention heads; SwiGLU MLP ratio 8/38/3 , RMSNorm, QK-Norm; CLS readout. Pretrained vs. random init ( -rw ).
ParticleViT-medium16MParticleViT: width 512, 5 transformer layers, 8 attention heads; SwiGLU MLP ratio 8/38/3 , RMSNorm, QK-Norm; CLS readout. Pretrained vs. random init ( -rw ). Corresponds to the ParticleViT model from [ 60 ] .
BERT-small4.4MHugging Face prajjwal1/bert-tiny encoder without tokenization [ 17 ] : per-particle features and global features are linearly projected in the model’s embedding space and outputs undergo mean-pooling.
MLP (global features)218kUsing global features only; hidden width 128, 3 residual blocks, sigmoid linear unit activation.
Appendix
Table 1: Model configurations used in the experiments. Randomly-initialized models are marked with -rw .
Figure 12: Efficiency of the event classifier for discriminating signal from background at matched false-positive rate of the cut-based baseline, for ParticleViT-medium and BERT-small models as well as their randomly-initialized counterparts ( -rw ). Starting training from a pre-trained BERT model seems to offer no advantage to final performance of the model.
Figure 13: Validation loss vs. training step for classification tasks (left) and regression tasks (right). The top row compares BERT-small, ParticleViT-small and the MLP baseline, while the bottom row compares the models pretrained on jets. The pretrained model ParticleViT-small reaches its lowest validation loss in approx. 40-50% fewer steps than pretrained OmniLearned-small model both for classification and regression. For some models, the curves are shown only from a later point on, as the validation loss has a high variance early in training.

Explore similar work

CardsList

  1. Panda Diplomacy: Foundation Model Pre-training across Particle Imaging Detectors for High Energy and Nuclear Physics

    Sep 1, 2026Samuel Young, César Jesús-Valls, Kazuhiro TeraoSelf-Supervised Pre-TrainingPoint Cloud Learning

    Stanford University · Stanford, CA 94305 · European Organization for Nuclear Research (CERN) +3

  2. Predict before you train: Scaling Laws for particle physics foundation models

    Jul 25, 2026Jan-Lucas Uslu, Benjamin Nachman, Christopher ReHigh-Energy PhysicsAI Scaling Laws

    Department of Applied Physics, Stanford University, Stanford, CA 94305, USA · Fundamental Physics Directorate, SLAC National Accelerator Laboratory, Menlo Park, CA 94025, USA · Department of Physics, Stanford University, Stanford, CA 94305, USA +1

  3. A Lightweight Foundation Model for Collider Physics with Multi-Domain Adaptation

    Jul 29, 2026Liangyu Wu, Qibin Liu, Alexander Yue +1Self-Supervised Pre-TrainingHigh-Energy Physics

    Department of Physics, Stanford University, 450 Jane Stanford Way, Stanford, 94305, CA, USA. · SLAC National Accelerator Laboratory, 2575 Sand Hill Road, Menlo Park, 94025, CA, USA.