Accelerator-based neutrino physics is entering an energy-frontier regime in which interactions reach the TeV scale and produce exceptionally dense, overlapping detector signatures. In this regime, event interpretation becomes impractical for conventional reconstruction approaches, particularly when labelled data are scarce and the analysis spans diverse downstream objectives. We present a sparse ViT framework for learning reusable representations from heterogeneous detector data. Self-supervised pre-training combines masked autoencoder reconstruction with relational voxel-level objectives for hierarchy, ghost and particle identification, and the resulting shared encoder is then jointly fine-tuned across classification and regression tasks. Evaluated on simulated events from the proposed FASERCal concept at the LHC, we find that pre-training consistently improves neutrino flavour and charm-quark identification, momentum regression, and vertex reconstruction over training from scratch, with the addition of relational objectives yielding further gains in the most topologically complex channels. Interpretability analyses further show that pre-training yields a more structured latent space, while detector-subsystem ablations recover physically plausible channel-dependent roles for the heterogeneous inputs. A data-efficiency study shows that, with roughly 103 labelled events, the pre-trained encoder already matches the flavour-classification performance of a randomly initialised model trained on an order of magnitude more data. The learned representations also transfer effectively to publicly available benchmarks spanning different detector technologies and energy scales, matching or exceeding published baselines. These results support self-supervised pre-training on multimodal detector data as a scalable route towards reusable representations for neutrino and particle-detector analysis.
Figures & tables
Figure 1: Example events from the test set illustrating masked reconstruction and relational tasks in the 3DCal detector. For each event, the top panel displays the ground-truth detector readout. The second row presents the visible voxels in the masked input (25% kept ) and the reconstructed output combined with the visible voxels ( reconstructed + kept ). The third row compares ghost-hit removal using ground truth and model predictions. The fourth row illustrates interaction hierarchy labelling (background, primary, secondary). The bottom row presents particle identification (electromagnetic, muon, hadronic). For the hierarchy and particle-identification panels, each voxel is assigned to its dominant contributing class for visualisation.
Figure 2: Classification and regression performance. A One-vs-rest ROC curves for the six flavour categories, comparing Scratch, MAE and MAE+Rel. B Row-normalised flavour confusion matrices using per-class thresholds that maximise the figure of merit; events passing multiple thresholds are assigned to the highest score. C One-vs-rest ROC curves for charm categories. D All-class regression-error summary for the selected event sample, shown for Evis , pTmiss , ∣pℓ∣ , ∣pjet∣ and the primary-vertex error. For the momentum and energy observables, errors are fractional residuals (xtrue−xreco)/xtrue ; for the primary vertex, the error is dPV=∥xPVtrue−xPVreco∥ in mm. Boxplots show the median, interquartile range and whiskers extending to approximately 95% of the selected sample. The corresponding purity, efficiency and figure-of-merit scans, together with class-resolved regression boxplots, are shown in Supplementary Information A . Definitions: ROC: receiver operating characteristic; AUC: area under the curve; NC: neutral current; had : hadronic; FOM: figure of merit ( S/S+B ); S and B denote expected signal and background yields.
Figure 3: Attribution and representation diagnostics for the learned representation. A Two example patch-level 3DCal saliency maps for test charged-current events from the fine-tuned MAE+Rel. B Uniform manifold approximation and projection (UMAP) of the pooled latent representation for Scratch, MAE, and MAE+Rel, coloured by true flavour (top) and visible energy, Evis (bottom). C Detector-subsystem ablation for MAE+Rel, evaluated on the full test set after suppressing each detector branch at token fusion. Metrics are one-vs-rest flavour AUROC, charmed-quark macro-AUROC, mean three-dimensional vertex error, and σMAD for visible energy, missing transverse momentum, primary-lepton momentum and jet momentum. D Response of MAE+Rel to coherent global calorimeter energy-scale shifts applied at inference time in 3DCal , AHCAL and ECAL . The curves show flavour macro-AUROC, charm macro-AUROC, dPV , and the absolute change in the median signed fractional residual, ∣Δmedian[(xtrue−xreco)/xtrue]∣ , for Evis , pTmiss , ∣pℓ∣ and ∣pjet∣ , always measured relative to the nominal scale point. Shaded bands show paired bootstrap percentile intervals obtained by resampling the same evaluation events across all scale points. The panels should be read as diagnostic probes of model reliance and latent organisation rather than exhaustive explanations of the learned physics.
Figure 4: Data-efficiency study for the three initialisation strategies. Performance of Scratch, MAE and MAE+Rel across labelled training budgets from 100 to 100,000 events. A,B , flavour and charm macro-AUROC. C , mean three-dimensional primary-vertex error. D–G , σMAD of the fractional residuals for visible energy, missing transverse momentum, primary-lepton momentum and jet momentum, respectively. Points are means across three random seeds and error bars show their standard deviation. Higher values are better for AUROC; lower values are better for regression and vertex errors.
Figure 5: Overview of the framework. A FASERCal case study, with the 3DCal as the primary interaction volume followed by the ECAL , AHCAL and muon spectrometer. B The sparse 3DCal and AHCAL voxel inputs, dense 5×5 ECAL energy matrix and variable-length muon-spectrometer summaries are converted into detector-specific tokens. C A hierarchical encoder performs module-level self-attention before Perceiver-IO fusion of the heterogeneous detector streams. D Stage 1 pre-training combines masked reconstruction with a relational voxel-level pass. E The pre-trained encoder is then fine-tuned jointly on event-level classification and regression tasks. A representative event spanning the four input subsystems is shown in Supplementary Figure 3 .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 1: Supplementary classification operating-point scans. A Purity (solid), efficiency (dashed) and figure of merit S/S+B (dotted) as functions of the classifier score threshold for the six flavour categories. B The corresponding scans for the four charm categories. Rows compare Scratch, MAE and MAE+Rel. These scans define the per-class thresholds used for the confusion matrices in Fig. 2 and provide the maximum-FOM values quoted in the Results.
Figure 2: Supplementary class-resolved regression boxplots. Regression-error boxplots for the selected event sample, resolved by true flavour category and target observable. Columns show the fractional residuals in Evis , pTmiss , ∣pℓ∣ and ∣pjet∣ , defined as (xtrue−xreco)/xtrue , together with the primary-vertex reconstruction error dPV=∥xPVtrue−xPVreco∥ in mm. Within each panel, the three boxplots compare Scratch, MAE and MAE+Rel using the same convention as Fig. 2 : boxes indicate the interquartile range, horizontal bars the median, and whiskers the central 95% of the selected events. These class-resolved boxplots complement the all-class summary in Fig. 2 .
Figure 3: Event display of a simulated charged-current muon antineutrino interaction in the FASERCal detector. The event is a high-energy νˉμ CC interaction with a true visible energy of Evis≃1265GeV and a reconstructed primary-lepton momentum of pμ≃745GeV . The neutrino interaction occurs inside the 3DCal and produces a dense shower with both hadronic and electromagnetic components, shown respectively by the reconstructed HAD voxels in red and EM voxels in blue. Voxels associated with the outgoing charged-current muon are shown in green. The display also shows the ghost voxels in grey. A long penetrating track, compatible with the outgoing muon, traverses the downstream ECAL and AHCAL before entering the muon spectrometer. A secondary muon is also produced and traverses the AHCAL and the muon spectrometer.
Parameter
Phase 1 (MAE)
Phase 2 (MAE+Rel)
FT (pre-trained)
Scratch
Epochs
400
+100
20
40
Batch size (per GPU)
512
512
1,024
1,024
Base learning rate
10−4
10−4
5×10−4
10−3
Warm-up epochs
40
10
5
5
Cosine-annealing epochs
360
90
15
35
Weight decay
0.05
0.05
0.05
0.05
Appendix
Table 1: Training hyperparameters. Pre-training proceeds in two phases: Phase 1 trains with masked reconstruction only (MAE), and Phase 2 continues from the Phase 1 checkpoint with an additional relational pass (MAE+Rel). Fine-tuning (FT) and Scratch use the same architecture and recipe except where noted.
Stage
Trainable model parameters
Optimisation steps
Hardware
Active wall time
GPU-hours
MAE pre-training
34,258,301
101,600
8× GH200
73.0 h
584
MAE+Rel continuation
34,258,301
25,400
8× GH200
53.4 h
427
Total pre-training
34,258,301
127,000
8× GH200
126.4 h
1,011
Joint fine-tuning
31,936,341
20,340
1× H100
4.1 h
4.1
Appendix
Table 2: Model size and active computational cost. The downstream count refers to the model used for fine-tuning and inference in the headline FASERCal experiments. The pre-training count includes the masked-autoencoder decoder and relational heads, which are discarded before downstream fine-tuning.
Future AI-based studies in particle physics will likely start from a foundation model to accelerate training and enhance sensitivity. As a step toward a general-purpose foundation model for particle physics, we investigate whether the OmniLearned and ParticleViT foundation models pretrained on diverse high-Q2 simulated and real pp and ep collisions retain useful knowledge to a few-GeV fixed-target neutrino experiment. We process MINERvA neutrino--nucleus scattering events and evaluate pretrained models on two types of tasks: regression of available energy and binary classification of charged-current pion final states (CC1π±, CCNπ±, and CC1π0). Pretrained OmniLearned and ParticleViT models outperform similarly sized models trained from scratch at the same compute budget, with the largest gains for OmniLearned on regression and for ParticleViT on classification. When the same transformer architecture is instead initialized from unrelated text pretraining (BERT), this advantage appears only marginally for classification in terms of compute efficiency and not in any way for regression. These results suggest that particle-level foundation models acquire inductive biases that generalize across large differences in energy scale, detector technology, and underlying physics processes, pointing toward detector-agnostic inference in particle physics.
Gregor Krzmanc, Vinicius Mikuni, Benjamin Nachman +1
Department of Physics, Stanford University, Stanford, CA 94305, USA · Nagoya University, Kobayashi-Maskawa Institute, Aichi 464-8602, Japan · Fundamental Physics Directorate, SLAC National Accelerator Laboratory, Menlo Park, CA 94025, USA +1
We present a first application of sparse-autoencoder-based mechanistic interpretability to particle physics. Studying a neutrino foundation model pretrained on IceCube data and fine-tuned for direction reconstruction, we identify a validated atlas of physical concepts in the model representation, using a strict validation protocol consisting of held-out tests, matched nuisance controls, and replication across independent dictionary trainings. Causal interventions show that the direction head barely draws on this atlas. Motivated by this underused information, we train an uncertainty head on the same event-level representation to predict the model's angular reconstruction error. Unlike the direction head, it depends causally on quality and brightness features from the atlas. At 20% selection efficiency, this interpretable estimator improves the median angular resolution from 20.2∘ to 3.2∘. These results suggest that mechanistic interpretability can reveal learned latent physics encoded within a model's internal representation and help design downstream tasks that exploit it.
Raphaël Bonnet-Guerrini, Johann Ioannou-Nikolaides, Inar Timiryasov +1
Computer Science Department, Università degli Studi di Milano, Milano, Italy · INFN Sezione di Milano, Milano, Italy · Niels Bohr Institute, University of Copenhagen, Copenhagen, Denmark
Foundation models are increasingly being pursued in particle and nuclear physics, but existing approaches remain strongly tied to individual experiments through detector-specific architectures or pre-training objectives, limiting their reuse across sensing modalities. We show that a point cloud self-distillation framework yields a substantially more general sensor-level pre-training recipe. We show that the same refined architecture and objective can be independently pre-trained with minimal changes on three qualitatively different detector modalities: liquid argon time projection chamber (LArTPC), collider TPC, and water Cherenkov. Using 1,000 labeled images for downstream task adaptation, Panda V2 matches or exceeds specialized foundation-model baselines trained with orders of magnitude more supervision, matching state-of-the-art particle-clustering performance with 70x fewer labeled events on sPHENIX while substantially improving particle identification, and on LArTPC data matching Panda (arXiv:2512.01324) particle reconstruction with up to 1,000x fewer labels. Beyond reconstruction, simple linear probes reveal physically meaningful latent structure associated with particle causality and track curvature.
Samuel Young, César Jesús-Valls, Kazuhiro Terao
Stanford University · Stanford, CA 94305 · European Organization for Nuclear Research (CERN) +3