Self-Supervised Pretraining for PHM: What Transfers, What Does Not, and What Federating It Costs

José Sanz-Durán; David Solís-Martín

cs.AIOct 26, 2026

Self-Supervised Pretraining for PHM: What Transfers, What Does Not, and What Federating It Costs

Authors: José Sanz-Durán, David Solís-Martín

Organizations: Department of Computer Science and Artificial Intelligence, Universidad de Sevilla, Seville, Spain

Abstract

Self-supervised pretraining has produced capable time-series foundation models, but its value for Prognostics and Health Management (PHM) has not been measured under leakage-controlled conditions. We measure it on a corpus of 47 public PHM datasets, harmonized to a single tensor contract and partitioned so that 11 datasets are withheld from pretraining in full. Transfer turns out to be task-dependent: a compact channel-independent Transformer encoder (801,808 parameters) pretrained with masked patch prediction raises fault-classification macro-F1 from 0.350 to 0.812 on CWRU and from 0.569 to 0.951 on HSG18, yet on CMAPSS remaining-useful-life estimation no pretrained checkpoint reliably improves upon a from-scratch baseline. We pretrain under two regimes---centralized, and simulated cross-silo federated with ten clients grouped by physical domain---holding architecture, objective, source data and optimization budget fixed. Two further findings are negative. At matched budget the centralized encoder leads by 15.5--20.2 macro-F1 points in four of four comparisons, although the comparison is confounded by a learning-rate schedule that favours the centralized arm and rests on a single pretraining run per regime; on RUL the two regimes are indistinguishable. And the intrinsic SSL loss is not a valid selection criterion: the centralized encoder more than halves the federated reconstruction loss without a corresponding gap on RUL, and a server-momentum sweep moves downstream validation macro-F1 by up to 0.29 while the SSL loss varies by 0.021. Transfer also hinges on the backbone learning rate: the same checkpoint yields 0.169 or 0.829 macro-F1 on CWRU depending on a single order of magnitude. We release the corpus pipeline, the protocol and the code.

Figures & tables

Figure 1: From the PHMD catalogue to experimental roles. Of 55 candidate datasets, 47 satisfy the structural contract and are partitioned into 36 source datasets (pretraining only) and 11 target datasets (evaluation only). No dataset was dropped on the basis of downstream performance.

Quantity	Value
Candidate datasets in catalogue	55
Downloaded	53
Audited	51
Accepted into the protocol	47
Source (pretraining only)	36
Target (evaluation only)	11

Table 1: Harmonized corpus. Source totals are the quantities that enter pretraining; the target subset is reserved for downstream evaluation only.

Figure 2: Separation between source and target data. Pretraining, centralized or federated, sees only source datasets; the resulting encoder is then adapted and evaluated on target datasets that never entered pretraining. The greyed-out path is the one the protocol forbids.

Checkpoint	Regime	Budget (steps)	SSL val. loss
Central-100k	centralized	100,000	—
Central-25k	centralized	25,000	0.3467
FedAvg	federated	25,000	0.7844
FedAvgM $\beta{=}0.7$	federated	25,000	0.7636

Table 2: Intrinsic pretraining outcome. SSL validation loss is a diagnostic of the reconstruction objective, not evidence of downstream utility. Central-100k was not evaluated under this protocol, and we do not estimate it. All values are computed on ten batches drawn from five of the 36 source datasets (three of ten clients) and are comparable only among themselves.

Configuration	Backbone LR	CWRU	HSG18
From scratch (1 seed)	—	0.350	0.569
Linear probing, Central-100k (1 seed)	frozen	0.705	0.906
Central-100k	$3{\cdot}10^{-6}$ / $1{\cdot}10^{-5}$	$0.812\pm 0.033$	$0.951\pm 0.002$
Central-25k	$3{\cdot}10^{-6}$ / $1{\cdot}10^{-5}$	$0.734\pm 0.022$	$0.911\pm 0.008$
FedAvg	$3{\cdot}10^{-6}$ / $1{\cdot}10^{-5}$	$0.572\pm 0.119$	$0.742\pm 0.048$
FedAvgM $\beta{=}0.7$	$1{\cdot}10^{-6}$ / $1{\cdot}10^{-5}$	$0.532\pm 0.079$	$0.756\pm 0.068$

Table 3: Fault classification, test macro-F1 (higher is better). Pretrained checkpoints: mean

\pm

sample standard deviation over three seeds, full fine-tuning with the backbone learning rate calibrated on validation. From scratch and linear probing rows are single-seed and are reported as reference points, not as matched comparators.

Dataset	Comparison	$\Delta$	$t$	df	$p$
CWRU	Central-25k $-$ FedAvg	$+0.162$	2.31	2.14	0.139
CWRU	Central-25k $-$ FedAvgM	$+0.202$	4.25	2.30	0.040
HSG18	Central-25k $-$ FedAvg	$+0.169$	6.01	2.11	0.023
HSG18	Central-25k $-$ FedAvgM	$+0.155$	3.92	2.06	0.057

Table 4: Budget-matched comparisons (Central-25k versus federated), test macro-F1, Welch’s

t

-test. The direction favours the centralized encoder in all four cases. Two are nominally significant, but none survives a Bonferroni correction for four comparisons (

\alpha=0.0125

). The dispersion being tested comes from downstream adaptation seeds over a single pretraining run per arm (Section 8 ), so these

p

-values quantify adaptation noise, not the regime.

Checkpoint	Adaptation	Validation	Test
From scratch (1 seed)	linear head, full training	39.08	43.67
Central-100k (1 seed)	linear probing	43.21	53.86
Central-100k (1 seed)	full fine-tuning	39.54	44.33
Central-100k	MLP head, frozen	$38.84\pm 0.07$	$44.16\pm 0.21$
Central-25k	MLP head, frozen	$39.07\pm 0.06$	$44.44\pm 0.58$
FedAvg	MLP head, frozen	$40.39\pm 0.06$	$42.72\pm 0.15$

Table 5: CMAPSS RUL, RMSE in cycles (lower is better), mean

\pm

sample standard deviation over three seeds. The from-scratch reference is single-seed and uses a linear head, so it is not a matched comparator; it is shown because it bounds the practical value of pretraining on this task.

Mode	Test RMSE	Asymmetric score	$R^{2}$
From scratch	43.67	754,754	$-0.40$
Linear probing	53.86	1,002,485	$-1.13$
Full fine-tuning	44.33	645,389	$-0.44$

Table 6: CMAPSS, single-seed basic modes: the two metrics disagree. The asymmetric score is the benchmark’s official criterion; lower is better for both columns.

	$\beta{=}0.0$	$\beta{=}0.3$	$\beta{=}0.5$	$\beta{=}0.7$	$\beta{=}0.9$
SSL validation loss ( $\downarrow$ )	0.784	0.778	0.765	0.764	0.768
CWRU, linear probing ( $\uparrow$ )	0.163	0.160	0.217	0.371	0.273
HSG18, full fine-tuning ( $\uparrow$ )	0.713	0.464	0.750	0.724	0.462

Table 7: Server-momentum sweep for FedAvgM (single seed, selected on validation). The intrinsic reconstruction loss is nearly constant while downstream utility varies by an order of magnitude more.

Figure 3: The intrinsic metric is flat where the extrinsic one is not. Server-momentum sweep for FedAvgM (data of Table 7 ; single seed, screening). Both panels share the vertical axis. The SSL validation loss moves within 0.021 across the sweep, while downstream validation macro-F1 moves by 0.21 (CWRU, linear probing) and 0.29 (HSG18, full fine-tuning), including the collapse at

\beta{=}0.9

Figure 4: The same checkpoint, two outcomes. Adapting Central-100k to CWRU with a backbone learning rate of

1{\cdot}10^{-4}

overwrites the pretrained representation and lands below the from-scratch baseline;

1{\cdot}10^{-5}

preserves it. Both runs use a single seed; the rates selected on validation for each checkpoint are given in the text.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Status	Reason
CWRU	final, $n{=}3$	classification, body of the paper
HSG18	final, $n{=}3$	classification, body of the paper
CMAPSS_RUL	final, $n{=}3$	RUL, body of the paper
PBCP16	diagnostic, $n{=}1$	negative case, Appendix B
PHM18	excluded from body	run with uncapped batch; pending re-execution
PHMAP23	documented negative	extreme class imbalance, noise floor

Appendix

Table 8: Status of the eleven held-out target datasets.

Class	Support	Precision	Recall	F1
0	477	0.480	0.998	0.648
1	2378	0.997	0.799	0.887
2	476	0.998	0.910	0.952
3	0	—	—	undefined
Macro-F1 over the three supported classes				0.829

Appendix

Table 9: CWRU per-class performance, best configuration. Confusion matrix rows are true classes, columns predicted.

Parameter	Value
Window / stride / patch	512 / 256 / 16
Patches per window ( $N$ )	32
Mask ratio	0.30 of valid patches
$d_{\mathrm{model}}$ / layers / heads	128 / 4 / 4
Normalization	pre-norm; per-window, per-channel instance norm
Trainable parameters	801,808

Appendix

Table 10: Principal configuration. Identical across regimes unless stated.

Self-Supervised Pre-Training

Self-supervised pre-training learns transferable representations from unlabeled data to reduce the labels needed for downstream tasks.

See all 251 papers

Towards Unified and Data-Efficient Prognostics and Health Management with Tabular Foundation Models
Jun 3, 2026Raffael Theiler, Lev Telyatnikov, Leandro Von Krannichfeldt +1Tabular Foundation Models RUL Estimation
IMOS Lab, EPFL, Lausanne, Switzerland
FreqCondNorm: Towards Cross-domain Predictive Maintenance through a Frequency-Conditioned Transformer Foundation Model
Sep 17, 2026Zaynab Raounak, Camille LHermine, Zhiguo ZengMulti-Source Domain Adaptation Self-Supervised Time Series Representation Learning
Laboratoire Génie Industriel, CentraleSupélec, Université Paris-Saclay, France
Picid: A Modular Evaluation Infrastructure for Reproducible PHM Across Tasks and Domains
May 27, 2026Lev Telyatnikov, Raffael Theiler, Leandro Von Krannichfeldt +1Benchmark Design Predictive Maintenance
EPFL

Jun 3, 2026J/K move · Enter open · S save

Towards Unified and Data-Efficient Prognostics and Health Management with Tabular Foundation Models

Raffael Theiler, Lev Telyatnikov, Leandro Von Krannichfeldt, Olga Fink

IMOS Lab, EPFL, Lausanne, Switzerland

Tabular Foundation Models RUL Estimation Predictive Maintenance Tabular ICL Fault Diagnosis

Open paper AskPDF arXiv

Data-driven Prognostics and Health Management (PHM) uses time-varying condition-monitoring data to diagnose system states and estimate remaining useful life in engineered assets. These tasks are central to maintenance planning, but industrial PHM data are often fragmented, partially observed, and poorly labeled, which hinders supervised learning. Foundation models offer a route toward reusable predictive systems, yet most time-series foundation models are designed for forecasting and assume long, coherent, regularly sampled sequences. To address this gap, we propose a framework for applying Tabular Foundation Models to industrial time series using in-context learning, and we evaluate them on a variety of PHM tasks. By converting raw unit-level signals into tabular rows, we show that these models perform well across multiple tasks - including prognostics, and diagnostics - and are highly data efficient. We compare them directly with sequence models, transformer baselines, and gradient-boosted trees under a common evaluation protocol. The results indicate that tabular foundation models achieve the best average ranks across prognostic and diagnostic tasks. Our findings further show that PFN-based models are competitive in low-data regimes, that temporal context can be preserved in the tabular representation, and that performance depends on representative context construction under subsampling. These results demonstrate that tabular foundation models provide a practical and general interface for heterogeneous PHM problems.

Abstract

Self-Supervised Pre-Training

Explore similar work

Towards Unified and Data-Efficient Prognostics and Health Management with Tabular Foundation Models

FreqCondNorm: Towards Cross-domain Predictive Maintenance through a Frequency-Conditioned Transformer Foundation Model

Picid: A Modular Evaluation Infrastructure for Reproducible PHM Across Tasks and Domains