Towards foundation-style models for energy-frontier heterogeneous neutrino detectors via self-supervised pre-training

Saúl Alonso-Monsalve; Fabio Cufino; Umut Kose; Anna Mascellani; André Rubbia

hep-exApr 8, 2026

Towards foundation-style models for energy-frontier heterogeneous neutrino detectors via self-supervised pre-training

Authors: Saúl Alonso-Monsalve, Fabio Cufino, Umut Kose, Anna Mascellani, André Rubbia

Organizations: IPA, ETH Zürich, Otto Stern Weg 5, Zurich, 8093, Switzerland.

Abstract

Accelerator-based neutrino physics is entering an energy-frontier regime in which interactions reach the TeV scale and produce exceptionally dense, overlapping detector signatures. In this regime, event interpretation becomes impractical for conventional reconstruction approaches, particularly when labelled data are scarce and the analysis spans diverse downstream objectives. We present a sparse ViT framework for learning reusable representations from heterogeneous detector data. Self-supervised pre-training combines masked autoencoder reconstruction with relational voxel-level objectives for hierarchy, ghost and particle identification, and the resulting shared encoder is then jointly fine-tuned across classification and regression tasks. Evaluated on simulated events from the proposed FASERCal concept at the LHC, we find that pre-training consistently improves neutrino flavour and charm-quark identification, momentum regression, and vertex reconstruction over training from scratch, with the addition of relational objectives yielding further gains in the most topologically complex channels. Interpretability analyses further show that pre-training yields a more structured latent space, while detector-subsystem ablations recover physically plausible channel-dependent roles for the heterogeneous inputs. A data-efficiency study shows that, with roughly $10^3$ labelled events, the pre-trained encoder already matches the flavour-classification performance of a randomly initialised model trained on an order of magnitude more data. The learned representations also transfer effectively to publicly available benchmarks spanning different detector technologies and energy scales, matching or exceeding published baselines. These results support self-supervised pre-training on multimodal detector data as a scalable route towards reusable representations for neutrino and particle-detector analysis.

Figures & tables

Panel of Figure 1 — Figure 1: Example events from the test set illustrating masked reconstruction and relational tasks in the 3DCal detector. For each event, the top panel displays the ground-truth detector readout. The second row presents the visible voxels in the masked input (25% kept ) and the reconstructed output combined with the visible voxels ( reconstructed + kept ). The third row compares ghost-hit removal using ground truth and model predictions. The fourth row illustrates interaction hierarchy labelling (background, primary, secondary). The bottom row presents particle identification (electromagnetic, muon, hadronic). For the hierarchy and particle-identification panels, each voxel is assigned to its dominant contributing class for visualisation.

Figure 2: Classification and regression performance. A One-vs-rest ROC curves for the six flavour categories, comparing Scratch, MAE and MAE+Rel. B Row-normalised flavour confusion matrices using per-class thresholds that maximise the figure of merit; events passing multiple thresholds are assigned to the highest score. C One-vs-rest ROC curves for charm categories. D All-class regression-error summary for the selected event sample, shown for $E_{\mathrm{vis}}$ , $p_{T}^{\mathrm{miss}}$ , $|p_{\ell}|$ , $|p_{\mathrm{jet}}|$ and the primary-vertex error. For the momentum and energy observables, errors are fractional residuals $(x^{\mathrm{true}}-x^{\mathrm{reco}})/x^{\mathrm{true}}$ ; for the primary vertex, the error is $d_{\mathrm{PV}}=\|\vec{x}_{\mathrm{PV}}^{\mathrm{true}}-\vec{x}_{\mathrm{PV}}^{\mathrm{reco}}\|$ in mm. Boxplots show the median, interquartile range and whiskers extending to approximately 95% of the selected sample. The corresponding purity, efficiency and figure-of-merit scans, together with class-resolved regression boxplots, are shown in Supplementary Information A . Definitions: ROC: receiver operating characteristic; AUC: area under the curve; NC: neutral current; $\mathrm{had}$ : hadronic; FOM: figure of merit ( $S/\sqrt{S+B}$ ); $S$ and $B$ denote expected signal and background yields.

Figure 3: Attribution and representation diagnostics for the learned representation. A Two example patch-level 3DCal saliency maps for test charged-current events from the fine-tuned MAE+Rel. B Uniform manifold approximation and projection (UMAP) of the pooled latent representation for Scratch, MAE, and MAE+Rel, coloured by true flavour (top) and visible energy, $E_{\mathrm{vis}}$ (bottom). C Detector-subsystem ablation for MAE+Rel, evaluated on the full test set after suppressing each detector branch at token fusion. Metrics are one-vs-rest flavour AUROC, charmed-quark macro-AUROC, mean three-dimensional vertex error, and $\sigma_{\mathrm{MAD}}$ for visible energy, missing transverse momentum, primary-lepton momentum and jet momentum. D Response of MAE+Rel to coherent global calorimeter energy-scale shifts applied at inference time in 3DCal , AHCAL and ECAL . The curves show flavour macro-AUROC, charm macro-AUROC, $d_{\mathrm{PV}}$ , and the absolute change in the median signed fractional residual, $|\Delta\,\mathrm{median}[(x^{\mathrm{true}}-x^{\mathrm{reco}})/x^{\mathrm{true}}]|$ , for $E_{\mathrm{vis}}$ , $p_{T}^{\mathrm{miss}}$ , $|p_{\ell}|$ and $|p_{\mathrm{jet}}|$ , always measured relative to the nominal scale point. Shaded bands show paired bootstrap percentile intervals obtained by resampling the same evaluation events across all scale points. The panels should be read as diagnostic probes of model reliance and latent organisation rather than exhaustive explanations of the learned physics.

Figure 4: Data-efficiency study for the three initialisation strategies. Performance of Scratch, MAE and MAE+Rel across labelled training budgets from 100 to 100,000 events. A,B , flavour and charm macro-AUROC. C , mean three-dimensional primary-vertex error. D–G , $\sigma_{\mathrm{MAD}}$ of the fractional residuals for visible energy, missing transverse momentum, primary-lepton momentum and jet momentum, respectively. Points are means across three random seeds and error bars show their standard deviation. Higher values are better for AUROC; lower values are better for regression and vertex errors.

Figure 5: Overview of the framework. A FASERCal case study, with the 3DCal as the primary interaction volume followed by the ECAL , AHCAL and muon spectrometer. B The sparse 3DCal and AHCAL voxel inputs, dense $5\times 5$ ECAL energy matrix and variable-length muon-spectrometer summaries are converted into detector-specific tokens. C A hierarchical encoder performs module-level self-attention before Perceiver-IO fusion of the heterogeneous detector streams. D Stage 1 pre-training combines masked reconstruction with a relational voxel-level pass. E The pre-trained encoder is then fine-tuned jointly on event-level classification and regression tasks. A representative event spanning the four input subsystems is shown in Supplementary Figure 3 .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Supplementary classification operating-point scans. A Purity (solid), efficiency (dashed) and figure of merit $S/\sqrt{S+B}$ (dotted) as functions of the classifier score threshold for the six flavour categories. B The corresponding scans for the four charm categories. Rows compare Scratch, MAE and MAE+Rel. These scans define the per-class thresholds used for the confusion matrices in Fig. 2 and provide the maximum-FOM values quoted in the Results.

Figure 2: Supplementary class-resolved regression boxplots. Regression-error boxplots for the selected event sample, resolved by true flavour category and target observable. Columns show the fractional residuals in

E_{\mathrm{vis}}

p_{T}^{\mathrm{miss}}

|p_{\ell}|

and

|p_{\mathrm{jet}}|

, defined as

(x^{\mathrm{true}}-x^{\mathrm{reco}})/x^{\mathrm{true}}

, together with the primary-vertex reconstruction error

d_{\mathrm{PV}}=\|\vec{x}_{\mathrm{PV}}^{\mathrm{true}}-\vec{x}_{\mathrm{PV}}^{\mathrm{reco}}\|

in mm. Within each panel, the three boxplots compare Scratch, MAE and MAE+Rel using the same convention as Fig. 2 : boxes indicate the interquartile range, horizontal bars the median, and whiskers the central 95% of the selected events. These class-resolved boxplots complement the all-class summary in Fig. 2 .

Figure 3: Event display of a simulated charged-current muon antineutrino interaction in the FASERCal detector. The event is a high-energy $\bar{\nu}_{\mu}$ CC interaction with a true visible energy of $E_{\mathrm{vis}}\simeq 1265~\mathrm{GeV}$ and a reconstructed primary-lepton momentum of $p_{\mu}\simeq 745~\mathrm{GeV}$ . The neutrino interaction occurs inside the 3DCal and produces a dense shower with both hadronic and electromagnetic components, shown respectively by the reconstructed HAD voxels in red and EM voxels in blue. Voxels associated with the outgoing charged-current muon are shown in green. The display also shows the ghost voxels in grey. A long penetrating track, compatible with the outgoing muon, traverses the downstream ECAL and AHCAL before entering the muon spectrometer. A secondary muon is also produced and traverses the AHCAL and the muon spectrometer.

Parameter	Phase 1 (MAE)	Phase 2 (MAE+Rel)	FT (pre-trained)	Scratch
Epochs	400	+100	20	40
Batch size (per GPU)	512	512	1,024	1,024
Base learning rate	$10^{-4}$	$10^{-4}$	$5{\times}10^{-4}$	$10^{-3}$
Warm-up epochs	40	10	5	5
Cosine-annealing epochs	360	90	15	35
Weight decay	0.05	0.05	0.05	0.05

Appendix

Table 1: Training hyperparameters. Pre-training proceeds in two phases: Phase 1 trains with masked reconstruction only (MAE), and Phase 2 continues from the Phase 1 checkpoint with an additional relational pass (MAE+Rel). Fine-tuning (FT) and Scratch use the same architecture and recipe except where noted.

Stage	Trainable model parameters	Optimisation steps	Hardware	Active wall time	GPU-hours
MAE pre-training	34,258,301	101,600	$8\times$ GH200	73.0 h	584
MAE+Rel continuation	34,258,301	25,400	$8\times$ GH200	53.4 h	427
Total pre-training	34,258,301	127,000	$8\times$ GH200	126.4 h	1,011
Joint fine-tuning	31,936,341	20,340	$1\times$ H100	4.1 h	4.1

Appendix

Table 2: Model size and active computational cost. The downstream count refers to the model used for fine-tuning and inference in the headline FASERCal experiments. The pre-training count includes the masked-autoencoder decoder and relational heads, which are discarded before downstream fine-tuning.

Self-Supervised Pre-Training

Self-supervised pre-training learns transferable representations from unlabeled data to reduce the labels needed for downstream tasks.

See all 236 papers

Cross-Domain Transfer with Particle Physics Foundation Models: From Jets to Neutrino Interactions
Apr 14, 2026Gregor Krzmanc, Vinicius Mikuni, Benjamin Nachman +1Cross-Domain Transfer Learning High-Energy Physics
Department of Physics, Stanford University, Stanford, CA 94305, USA · Nagoya University, Kobayashi-Maskawa Institute, Aichi 464-8602, Japan · Fundamental Physics Directorate, SLAC National Accelerator Laboratory, Menlo Park, CA 94025, USA +1
Finding and using interpretable latents in a neutrino foundation model with sparse autoencoders
Aug 26, 2026Raphaël Bonnet-Guerrini, Johann Ioannou-Nikolaides, Inar Timiryasov +1Mechanistic Interpretability Uncertainty Quantification
Computer Science Department, Università degli Studi di Milano, Milano, Italy · INFN Sezione di Milano, Milano, Italy · Niels Bohr Institute, University of Copenhagen, Copenhagen, Denmark
Panda Diplomacy: Foundation Model Pre-training across Particle Imaging Detectors for High Energy and Nuclear Physics
Sep 1, 2026Samuel Young, César Jesús-Valls, Kazuhiro TeraoSelf-Supervised Pre-Training Point Cloud Learning
Stanford University · Stanford, CA 94305 · European Organization for Nuclear Research (CERN) +3

Apr 14, 2026J/K move · Enter open · S save

Cross-Domain Transfer with Particle Physics Foundation Models: From Jets to Neutrino Interactions

Gregor Krzmanc, Vinicius Mikuni, Benjamin Nachman, Callum Wilkinson

Department of Physics, Stanford University, Stanford, CA 94305, USA · Nagoya University, Kobayashi-Maskawa Institute, Aichi 464-8602, Japan · Fundamental Physics Directorate, SLAC National Accelerator Laboratory, Menlo Park, CA 94025, USA+1

Cross-Domain Transfer Learning High-Energy Physics

Open paper AskPDF arXiv

Future AI-based studies in particle physics will likely start from a foundation model to accelerate training and enhance sensitivity. As a step toward a general-purpose foundation model for particle physics, we investigate whether the OmniLearned and ParticleViT foundation models pretrained on diverse high-

Q^2

simulated and real

pp

and

ep

collisions retain useful knowledge to a few-GeV fixed-target neutrino experiment. We process MINERvA neutrino--nucleus scattering events and evaluate pretrained models on two types of tasks: regression of available energy and binary classification of charged-current pion final states (

\mathrm{CC1π^{\pm}}

\mathrm{CCNπ^{\pm}}

, and

\mathrm{CC1π^{0}}

). Pretrained OmniLearned and ParticleViT models outperform similarly sized models trained from scratch at the same compute budget, with the largest gains for OmniLearned on regression and for ParticleViT on classification. When the same transformer architecture is instead initialized from unrelated text pretraining (BERT), this advantage appears only marginally for classification in terms of compute efficiency and not in any way for regression. These results suggest that particle-level foundation models acquire inductive biases that generalize across large differences in energy scale, detector technology, and underlying physics processes, pointing toward detector-agnostic inference in particle physics.

Abstract

Self-Supervised Pre-Training

Explore similar work

Cross-Domain Transfer with Particle Physics Foundation Models: From Jets to Neutrino Interactions

Finding and using interpretable latents in a neutrino foundation model with sparse autoencoders

Panda Diplomacy: Foundation Model Pre-training across Particle Imaging Detectors for High Energy and Nuclear Physics