SensOFormer: Foundation Models for Particle Accelerators
Organizations: Los Alamos National Laboratory, Los Alamos, NM, USA.
Abstract
Large-scale scientific facilities, such as particle accelerators, are complex systems composed of thousands of interacting components that must operate collectively to support diagnostics and control, and ensure safe, reliable operation. These facilities rely on extensive sensor networks that generate large volumes of heterogeneous, incomplete, and noisy telemetry, often sampled at different rates across subsystems. In this paper, we adopt the foundation-model paradigm to learn general-purpose sensor representations from historical accelerator data through self-supervised pretraining. We introduce SensOFormer, a denoising masked transformer with a Perceiver-style encoder-decoder architecture designed to handle variable sensor sets, accommodate heterogeneous sampling rates, and learn robust representations from noisy measurements. The model is pretrained on multiple particle-accelerator datasets and transferred to downstream tasks, including missing-data imputation, anomaly detection, cavity identification, and fault identification. Through extensive evaluations and comparisons, we demonstrate that a single self-supervised model can learn reusable representations for diverse operational tasks in large-scale scientific facilities. Project page: https://mahindrautela.github.io/sensoformer.
Figures & tables
| Dataset | Facility / subsystem | Dataset/Tensor shape | Downstream task |
| (N,T,C) | |||
| LANSCE (2019–2024) | LANSCE operational data | (4250, 10000, ) | MDI on 10% test set |
| LHC-BL (2017) | LHC beam-loss monitoring | (1, 542061, 66) | MDI on 10% test set |
| LHC-BL (2018) | (1, 524430, 66) | ||
| RFS-LCLS (2020) | LCLS RF station signals (AMM and beam) | (1269, 2196, 9) | MDI on 10% test set |
| LCLS RF station signals (AMPL and beam) | (3208, 2076, 9) |
| Dataset | Mask ratio | XTi | Ti | Mi | S | B |
| LANSCE 2019–2024 | 5% | 2.4928 | 1.9969 | 2.0296 | 1.8414 | 1.7130 |
| 15% | 2.4693 | 1.9984 | 1.9526 | 1.8916 | 1.6769 | |
| 25% | 2.5411 | 2.1002 | 1.9933 | 2.0337 | 1.7580 | |
| 50% | 2.6005 | 2.3238 | 2.1687 | 2.1835 | 2.0624 | |
| LHC-BL | 5% | 3.0867 | 2.8527 | 2.6608 | 2.4952 | 2.3110 |
| 15% | 3.1752 | 2.9858 | 2.8162 | 2.5671 | 2.5179 |
| Model | Passed tests | Success rate (%) |
| Hierarchical voting ensemble (VE) | 20 / 21 | 95.2 |
| AdaBoost (AB) | 11 / 21 | 52.4 |
| Extremely randomized trees (ET) | 10 / 21 | 47.6 |
| CNN | 9 / 21 | 42.9 |
| Bagging classifier (BC) | 9 / 21 | 42.9 |
| Random forest (RF) | 9 / 21 | 42.9 |
| Task | LSTM | CNN, direct mapping | CNN, scalogram | SensOFormer-XTi (Ours) |
| Cavity identification | 84.60 | 84.60 | 45.00 | 84.80 |
| Fault identification | 74.10 | 74.10 | 58.90 | 84.01 |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| SeFT | MOIRAI | SensOFormer (Ours) | |
| Input token | Observation event | Time–variate patch | Sensor-channel set at each time step |
| Channel encoder | Independent MLPs with Deep-Set aggregation | Any-variate full self-attention | Perceiver-style latent set encoder with cross-attention |
| Temporal modeling | No sequential processor; time is encoded independently for each event | Joint time–variate encoder Transformer | Separate latent temporal Transformer |
| Decoder / output | Attention pooling followed by a classification MLP | Linear patch projection to probabilistic mixture parameters | Permutation-equivariant set decoder with cross-attention |
| Self-supervised pretraining | ✗ | ✓ | ✓ |
| Training objective | Supervised classification | Self-supervised probabilistic forecasting | Self-supervised denoising masked reconstruction |
| Released MOIRAI (Uni2TS) | Accelerator benchmark requirement | |
| Task | Predicts a future time-series horizon. | Reconstructs missing values occurring anywhere in the sensor sequence. |
| Masking | Masks complete future patches. | Uses point-wise, time-wise, channel-wise, and block-wise masks. |
| Mask granularity | Prediction masks operate at the patch-token level. | Missingness is defined for individual sensor values. |
| Output head | Uses a probabilistic forecasting output head. | Requires anomaly, cavity, and fault-classification heads. |
| Number of variates | The default fine-tuning configuration supports at most 128 variates. | Accelerator samples contain as many as 1,754 sensor channels. |
| Attention | Flattens time–variate patches and applies full self-attention. | Requires efficient processing of very large and variable sensor sets. |
| Model | Attention dim. | Heads | Depth | MLP dim. | Parameters |
| XTi | 192 | 4 | 2 | 768 | 2.38M |
| Ti | 256 | 4 | 4 | 1024 | 5.80M |
| Mi | 384 | 6 | 4 | 1536 | 13.12M |
| S | 512 | 8 | 8 | 2048 | 35.87M |
| B | 768 | 12 | 8 | 3072 | 80.54M |
| Model/component | Parameters | Scope |
| CEBAF bidirectional LSTM | M | Specified recurrent layers |
| CEBAF direct-mapping CNN | M | Convolutions and batch norm |
| CEBAF scalogram CNN | M | Convolutions and batch norm |
| SensOFormer-XTi | M | Full pretrained model |
| SensOFormer-B | M | Full pretrained model |
| SensOFormer HVCM head | M | Trainable head only |
Self-Supervised Time Series Representation Learning
Self-supervised time series representation learning uses unlabeled sequences to learn features that transfer to forecasting, classification, and anomaly detection.
See all 164 papers