Cross-Domain Transfer with Particle Physics Foundation Models: From Jets to Neutrino Interactions

Gregor Krzmanc; Vinicius Mikuni; Benjamin Nachman; Callum Wilkinson

hep-exApr 14, 2026

Cross-Domain Transfer with Particle Physics Foundation Models: From Jets to Neutrino Interactions

Authors: Gregor Krzmanc, Vinicius Mikuni, Benjamin Nachman, Callum Wilkinson

Organizations: Department of Physics, Stanford University, Stanford, CA 94305, USA · Nagoya University, Kobayashi-Maskawa Institute, Aichi 464-8602, Japan · Fundamental Physics Directorate, SLAC National Accelerator Laboratory, Menlo Park, CA 94025, USA · Lawrence Berkeley National Laboratory, Berkeley, CA 94720, USA

Abstract

Future AI-based studies in particle physics will likely start from a foundation model to accelerate training and enhance sensitivity. As a step toward a general-purpose foundation model for particle physics, we investigate whether the OmniLearned and ParticleViT foundation models pretrained on diverse high- $Q^2$ simulated and real $pp$ and $ep$ collisions retain useful knowledge to a few-GeV fixed-target neutrino experiment. We process MINERvA neutrino--nucleus scattering events and evaluate pretrained models on two types of tasks: regression of available energy and binary classification of charged-current pion final states ( $\mathrm{CC1π^{\pm}}$ , $\mathrm{CCNπ^{\pm}}$ , and $\mathrm{CC1π^{0}}$ ). Pretrained OmniLearned and ParticleViT models outperform similarly sized models trained from scratch at the same compute budget, with the largest gains for OmniLearned on regression and for ParticleViT on classification. When the same transformer architecture is instead initialized from unrelated text pretraining (BERT), this advantage appears only marginally for classification in terms of compute efficiency and not in any way for regression. These results suggest that particle-level foundation models acquire inductive biases that generalize across large differences in energy scale, detector technology, and underlying physics processes, pointing toward detector-agnostic inference in particle physics.

Figures & tables

Figure 1: Validation loss vs. number of cumulative training FLOPs for classification tasks (left) and the regression task (right), using the MINERvA Open Data playlist 1A. The top row compares BERT-small, ParticleViT-small and the MLP baseline, while the bottom row compares all the models pretrained on jets. Pretrained models on jet data (OmniLearned-small and ParticleViT models) achieve lower loss at equivalent compute budgets compared to the same scratch-trained models (OmniLearned-small-rw, ParticleViT-small-rw and ParticleViT-medium-rw models). Pre-trained ParticleViT-small model achieves its lowest validation loss in around a third of the compute required for OmniLearned-small both for classification and regression. Validation loss is reported every 1000 steps. For some models, the curves are shown only from a later point on, as the validation loss has a high variance early in training.

Figure 2: Discrimination metrics for the

\mathrm{CC1\pi^{\pm}}

tagging task as a function of true pion kinematics: area under the precision–recall curve (AUPRC, left column), area under the ROC curve (AUROC, center), and true positive rate at a fixed false positive rate (right column). The top row uses bins in true pion energy; the bottom row uses bins in true pion polar angle. In each bin, positives are true signal events falling in that bin, while the negative sample is the full background, held fixed across bins so that only the in-bin signal population changes. TPR is computed at the baseline false-positive rate matched globally on the full test set; see Fig. 5 for the per-bin matched comparison.

Figure 3: Discrimination metrics for the

\mathrm{CC1\pi^{0}}

Figure 4: Discrimination metrics for the

\mathrm{CCN\pi^{\pm}}

tagging task as a function of invariant mass of the hadronic system

W

: area under the precision–recall curve (AUPRC, left column), area under the ROC curve (AUROC, center), and true positive rate at a fixed false positive rate (right column). Efficiency is computed using the baseline’s false-positive rate matched globally on the full test set; see Fig. 5 for the per-bin matched comparison.

Figure 5: Event classification efficiency at the cut-based baseline’s false-positive rate, matched independently in each kinematic bin, for

\mathrm{CC1\pi^{\pm}}

and

\mathrm{CC1\pi^{0}}

(vs. true pion energy

E_{\pi}

) and

\mathrm{CCN\pi^{\pm}}

with

N\geq 1

(vs. true hadronic invariant mass

W

). Shaded bands show the spread over four independent training runs. Unlike the globally matched TPR panels in Figs. 2 – 4 , every neural model remains above the cut-based baseline in all bins.

Figure 6: Interquartile range (top) and most probable value (bottom) of the available hadronic energy residual distribution vs.

q_{3}

, for playlist 1A. Smaller IQR indicates better energy resolution; MPV near unity indicates no systematic bias. Events with

E^{\mathrm{true}}_{\mathrm{available}}<0.05

GeV are excluded.

Figure 7: Available hadronic energy resolution summary for playlists 1A and 1B. The relative ordering of models is consistent with the single-playlist result in Fig. 6 .

Figure 8: Available hadronic energy residual distributions in four bins of

q_{3}

for muon-selected events (top row: signed residual

E^{\mathrm{reco}}_{\mathrm{available}}-E^{\mathrm{true}}_{\mathrm{available}}

; bottom row: ratio

E^{\mathrm{reco}}_{\mathrm{available}}/E^{\mathrm{true}}_{\mathrm{available}}

). Pretrained OmniLearned models (OmniLearned-small) produce narrower, more symmetric distributions compared to scratch-trained models and ParticleViT models, especially at low

q_{3}

. Events with

E^{\mathrm{true}}_{\mathrm{available}}<0.05

GeV are excluded.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Event composition in bins of true hadronic invariant mass

W

for playlist 1A. Each row corresponds to a different signal definition. Left: signal counts by interaction category. Center: background counts by interaction category. Right: signal-to-background ratio.

Figure 10: Same as Fig. 9 , but reported in bins of the true three-momentum transfer

q_{3}

Figure 11: Composition of the single-pion signals versus true pion energy

E_{\pi}

and polar angle

\theta_{\pi}

, together with the corresponding background composition, for playlist 1A.

Model	# params	Architecture summary
OmniLearned-small	3M	Particle-encoder transformer (PET2), as described in [ 18 ] .
OmniLearned-medium (OL-medium-frozen)	52M	Particle-encoder transformer (PET2) [ 18 ] with frozen backbone ( $\sim$ 21M trainable parameters).
ParticleViT-small	9.7M	ParticleViT (embedding variant): width 448, 4 transformer layers, 7 attention heads; SwiGLU MLP ratio $8/3$ , RMSNorm, QK-Norm; CLS readout. Pretrained vs. random init ( -rw ).
ParticleViT-medium	16M	ParticleViT: width 512, 5 transformer layers, 8 attention heads; SwiGLU MLP ratio $8/3$ , RMSNorm, QK-Norm; CLS readout. Pretrained vs. random init ( -rw ). Corresponds to the ParticleViT model from [ 60 ] .
BERT-small	4.4M	Hugging Face prajjwal1/bert-tiny encoder without tokenization [ 17 ] : per-particle features and global features are linearly projected in the model’s embedding space and outputs undergo mean-pooling.
MLP (global features)	218k	Using global features only; hidden width 128, 3 residual blocks, sigmoid linear unit activation.

Appendix

Table 1: Model configurations used in the experiments. Randomly-initialized models are marked with -rw .

Figure 12: Efficiency of the event classifier for discriminating signal from background at matched false-positive rate of the cut-based baseline, for ParticleViT-medium and BERT-small models as well as their randomly-initialized counterparts ( -rw ). Starting training from a pre-trained BERT model seems to offer no advantage to final performance of the model.

Figure 13: Validation loss vs. training step for classification tasks (left) and regression tasks (right). The top row compares BERT-small, ParticleViT-small and the MLP baseline, while the bottom row compares the models pretrained on jets. The pretrained model ParticleViT-small reaches its lowest validation loss in approx. 40-50% fewer steps than pretrained OmniLearned-small model both for classification and regression. For some models, the curves are shown only from a later point on, as the validation loss has a high variance early in training.

Explore similar work

Sep 1, 2026

Panda Diplomacy: Foundation Model Pre-training across Particle Imaging Detectors for High Energy and Nuclear Physics

Foundation models are increasingly being pursued in particle and nuclear physics, but existing approaches remain strongly tied to individual experiments through detector-specific architectures or pre-training objectives, limiting their reuse across sensing modalities. We show that a point cloud self-distillation framework yields a substantially more general sensor-level pre-training recipe. We show that the same refined architecture and objective can be independently pre-trained with minimal changes on three qualitatively different detector modalities: liquid argon time projection chamber (LArTPC), collider TPC, and water Cherenkov. Using 1,000 labeled images for downstream task adaptation, Panda V2 matches or exceeds specialized foundation-model baselines trained with orders of magnitude more supervision, matching state-of-the-art particle-clustering performance with 70x fewer labeled events on sPHENIX while substantially improving particle identification, and on LArTPC data matching Panda (arXiv:2512.01324) particle reconstruction with up to 1,000x fewer labels. Beyond reconstruction, simple linear probes reveal physically meaningful latent structure associated with particle causality and track curvature.

Samuel Young, César Jesús-Valls, Kazuhiro Terao

Stanford University · Stanford, CA 94305 · European Organization for Nuclear Research (CERN) +3

Self-Supervised Pre-Training Point Cloud Learning High-Energy Physics

Ask PDF

Jul 25, 2026

Predict before you train: Scaling Laws for particle physics foundation models

The largest machine learning models in particle physics are also the most expensive to train, yet the return on scaling a given architecture cannot be estimated before that compute is spent. Scaling laws have been fit for jets, but none has yet been shown to predict the performance of models it was not fit on. We show that, for a generic transformer pretrained on collider jets, it can be forecast. Fitting a joint model-and-data scaling law on small models alone, spanning three orders of magnitude of training compute, we predict the loss of models trained afterward with more than one hundred times more compute to within one percent. We then connect the forecast to downstream physics performance: across two standard tagging benchmarks, lower pretraining loss yields systematically lower fine-tuning loss and higher background rejection after fine-tuning. Within this model family and these tasks, a compute budget can therefore be translated into expected physics performance before any large model is trained. The final frontier model is consistent with the published numbers for current state-of-the-art physics-aware foundation models trained on the same corpus, on accuracy, AUC, and quark/gluon rejection, with a residual edge for the physics-aware model only in the high-purity tail of top tagging. We release five pretrained models spanning multiple sizes, together with the complete training recipe and code.

Jan-Lucas Uslu, Benjamin Nachman, Christopher Re

Department of Applied Physics, Stanford University, Stanford, CA 94305, USA · Fundamental Physics Directorate, SLAC National Accelerator Laboratory, Menlo Park, CA 94025, USA · Department of Physics, Stanford University, Stanford, CA 94305, USA +1

High-Energy Physics AI Scaling Laws

Ask PDF

Jul 29, 2026

A Lightweight Foundation Model for Collider Physics with Multi-Domain Adaptation

We present a lightweight approach to foundation modeling (NEXUS) that leverages pre-trained learning from collider physics data towards out-of-domain tasks in other scientific datasets, using a fully connected autoencoder model with approximately 3 million parameters. The model pre-trains with no supervision over a large-scale collision dataset from the Large Hadron Collider modeled by charged particle track features. Downstream tasks for collider analyses, such as kinematic regression and event classification, are developed on pre-trained model weights and achieve improved accuracy with only small labeled datasets when compared to equivalent architectures trained from scratch. The benefits of pre-training are additionally investigated through latent space interpretation and application to other domains, including gravitational waves, flood forecasting, and neural activity. Furthermore, the relative computational simplicity of NEXUS is demonstrated compared to transformer approaches at comparable scale, opening the door to power-efficient inference and real-time or edge applications of foundation models in scientific experiments.

Liangyu Wu, Qibin Liu, Alexander Yue +1

Department of Physics, Stanford University, 450 Jane Stanford Way, Stanford, 94305, CA, USA. · SLAC National Accelerator Laboratory, 2575 Sand Hill Road, Menlo Park, 94025, CA, USA.

Self-Supervised Pre-Training High-Energy Physics Cross-Domain Transfer Learning Scientific ML

Ask PDF