Training and Scaling Compute-Optimal Physiological Waveform Foundation Models

Pingzhi Li; Jie Peng; Shuqing Luo; Zachary Plotkin; Tianlong Chen

cs.LGOct 5, 2026

Training and Scaling Compute-Optimal Physiological Waveform Foundation Models

Authors: Pingzhi Li, Jie Peng, Shuqing Luo, Zachary Plotkin, Tianlong Chen

Organizations: University of North Carolina at Chapel Hill · Foci Labs

Abstract

We investigate the scaling laws and compute-optimal training of physiological waveform foundation models (FMs). We train Aether, a family of over one hundred FMs ranging from 20M to 2.1B parameters, on up to 36.3M hours of physiological waveforms. We construct eight clinical prediction tasks from MIMIC-III and evaluate the FMs through linear probing. The 720M FM outperforms all existing baseline FMs across all eight tasks. A scaling law of model size, pretraining hours, and labeled patients predicts downstream ranking error, i.e. $1-\mathrm{AUROC}$ , effectively with $0.5\%$ prediction MAE at held-out resource scales and $0.9\%$ MAE when extrapolating to 2.1B parameters. We present three findings: (1) Compute-optimal training scales both FM size and pretraining hours. Under the fitted law, a $10.0\times$ increase in compute FLOPs scales model size by $1.2\times$ and pretraining hours by $8.2\times$ . (2) Larger FMs use waveform data more efficiently, and greater pretraining exposure increases the benefit of model scaling. Starting from 25M parameters and 4.8M pretraining hours, doubling FM size reduces the predicted hours needed for the same performance by $51.8\%$ . (3) Pretraining and clinical supervision reinforce each other: more labeled patients increase the return to pretraining, while larger FMs and longer pretraining reduce labeling requirements. For the example of the 720M FM, extending pretraining from 120K to 36.3M hours reduces the predicted patient requirement by $61\%$ at a target ranking error. These findings provide a quantitative training recipe and a promising and durable scaling path for physiological waveform modeling and downstream clinical prediction.

Figures & tables

Figure 1: Scaling clinical prediction and label efficiency. (a) Measured eight-task mean ranking error against

S=P^{g}/(AN^{-a}+BD^{-b})

. (b) Predicted labeled patient requirements at

\mathcal{R}=0.32

. Extending 720M pretraining from 120K to 36.3M hours reduces required patients by

61\%

Figure 2: Overview of Aether architecture, pretraining, and downstream adaptation. (a) Bidirectional Gated DeltaNet (Bi-GDN) and grouped-query attention (GQA) blocks form a repeating $3{:}1$ hybrid pattern. (b) Multichannel waveforms are divided into 200 ms patches, and the FM learns to reconstruct randomly masked patches. (c) The pretrained FM is frozen, and its mean-pooled representations support linear probing for downstream clinical prediction.

Figure 3: Comparison with existing waveform FMs across eight clinical prediction tasks. Each panel shows macro AUROC against FM size using 100% of labeled patients. To ensure a fair comparison, we report each model’s best linear-probe AUROC across the same regularization sweep,

\lambda\in\{0.001,0.01,0.1,1,10\}

. Higher AUROC corresponds to lower ranking error.

Figure 4: Scaling of downstream average ranking error

\mathcal{R}

across eight tasks. (a) Fitted response to the resource score

S=P^{g}/(AN^{-a}+BD^{-b})

. (b) Pretraining scaling at

P=1{,}222

. (c) Patient scaling at 4.8M pretraining hours. (d) Predictions at held-out resource levels: we hold out all configurations at one value of

N

D

, or

P

, fit on the rest, and predict the held-out configurations. MAE averages absolute prediction errors equally across the three resource axes.

Figure 5: Scaling and prediction at larger resources. (a) Pretraining scaling at 2.1B for all four values of

P

. (b) Patient scaling at 2.1B for all five values of

D

. (c) Predictions for all 20 configurations at 2.1B. (d) Predictions for 20M, 60M, and 180M after 36.3M, 17.0M, and 5.9M pretraining waveform hours, respectively. Ranking errors and absolute patient counts are averaged across the eight tasks. Solid lines connect measurements in (a,b); dashed diagonals in (c,d) mark

\widehat{\mathcal{R}}=\mathcal{R}

Figure 6: Iso-error contours under a FLOPs constraint, from Equation 5.1 . (a–d) $P=408$ , $1{,}222$ , $2{,}443$ , and $4{,}071$ , respectively. Dashed lines mark iso-FLOPs slices; red curves show the efficient frontier over the observed FM-size range. Each panel shows all 57 configurations. Ranking errors and patient counts are eight-task means.

Figure 7: Iso-error contours for FM size and pretraining waveform hours. Each curve holds the predicted eight-task average ranking error fixed under Equation 4.1 . (a–d) $P=408$ , $1{,}222$ , $2{,}443$ , and $4{,}071$ , respectively. Points show measurements, with color indicating observed ranking error. Dashed lines mark iso-parameters slices. Patient counts are averaged across the eight tasks. Moving right along a curve reduces waveform hours at fixed ranking error $\mathcal{R}$ .

Figure 8: Iso-ranking-error curves showing predicted labeling requirements averaged across eight tasks. Target ranking error

\mathcal{R}_{\mathrm{t}}

is 0.32 in (a), 0.37 in (b).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Task	Prediction time	$W$	$G$	$H$	Targets	Eligibility at $T$
Acute deterioration	Sliding	6 h	0	6 h	3	Eligible for at least one component
Circulatory deterioration	Sliding	6 h	0	6 h	1	No active vasopressors
Dynamic mortality	Sliding	6 h	6 h	24 h	1	All monitored patients
Respiratory escalation	Sliding	6 h	0	6 h	1	No invasive ventilation
Sepsis onset	Sliding	6 h	1 h	5 h	1	No prior Sepsis-3 onset
Fluid response	Infusion start	6 h	30 min	30 min	2	Documented rapid crystalloid course

Appendix

Table 4: Prediction windows and eligibility for the eight downstream tasks.

W

denotes clinical history,

G

the forecast gap, and

H

the outcome horizon. Each FM input contains one hour of waveforms ending at

T

. “Targets” gives the number of binary outcomes evaluated per task.

Explore similar work

Sep 23, 2026cs.LG

A Scaling Study for fMRI Foundation Models

Scaling laws have guided large-model development in computer vision and natural language processing, but the relationships among data, model size, and compute remain unclear for functional magnetic resonance imaging (fMRI) foundation models. Here, we conduct a controlled empirical study using pretraining data from more than 200 source datasets and over 10,000 GPU-hours of experiments. Holding the pretraining framework and downstream protocol fixed, we vary pretraining data size, model size, and training duration. Downstream performance generally improves with compute, yet models using similar compute can perform substantially differently. Additional pretraining data bring larger gains at larger model sizes, suggesting that data and model size should be scaled together. At matched compute, increasing pretraining data benefits more tasks than increasing model size, although the pattern varies across tasks. We then use in-distribution (ID) downstream performance to select the combination of pretraining data size, model size, and training duration at two fixed compute budgets. The resulting models are locked before out-of-distribution (OOD) evaluation. They achieve the highest average performance across the evaluated OOD tasks among the compared fMRI foundation models while using less pretraining compute. Overall, our results show that compute alone does not characterize fMRI scaling: performance depends on how pretraining data, model size, and training duration are combined.

Wenhao Ye, Xuanye Pan, Junfeng Xia +3

Southern University of Science and Technology · Shenzhen University · SLAI +1

Model Size Functional Magnetic Resonance Imaging Pretraining Foundation Model Scaling Computer Vision

Ask PDF

May 12, 2026eess.SP

Pretraining Strategies and Scaling for ECG Foundation Models: A Systematic Study

Specialized foundation models are beginning to emerge in various medical subdomains, but pretraining methodologies and parametric scaling with the size of the pretraining dataset are rarely assessed systematically and in a like-for-like manner. This work focuses on foundation models for electrocardiography (ECG) data, one of the most widely captured physiological time series world-wide. We present a comprehensive assessment of pretraining methodologies, covering five different contrastive and non-contrastive self-supervised learning objectives for ECG foundation models, and investigate their scaling behavior with pretraining dataset sizes up to 11M input samples, exclusively from publicly available sources. Pretraining strategy has a meaningful and consistent impact on downstream performance, with contrastive predictive coding (slightly ahead of JEPA) yielding the most transferable representations across diverse clinical tasks. Scaling pretraining data continues to yield meaningful improvements up to 11M samples for most objectives. We also compare model architectures across all pretraining methodologies and find evidence for a clear superiority of structured state space models compared to transformers and CNN models. We hypothesize that the strong inductive biases of structured state space models, rather than pretraining scale alone, are the primary driver of effective ECG representation learning, with important implications for future foundation model development in this and potentially other physiological signal domains.

M A Al-Masud, Nils Strodthoff

AI4Health Division Carl von Ossietzky Universität Oldenburg Oldenburg, Lower Saxony, Germany

Electrocardiogram Foundation Models Electrocardiography Medical World Model Predictive Coding Scaling Systematic Study

Ask PDF

Apr 24, 2026cs.LG

A Nationwide Japanese Medical Claims Foundation Model: Balancing Model Scaling and Task-Specific Computational Efficiency

Clinical risk prediction using longitudinal medical data supports individualized care. Self-supervised foundation models have emerged as a promising approach for leveraging large-scale unlabeled healthcare records. In natural language processing, scaling laws suggest that larger models achieve predictably lower pretraining losses, supporting the foundation model paradigm. However, for structured medical data, characterized by a limited vocabulary and sparse observations, whether increasing model size consistently improves downstream predictions is unclear, as most studies evaluate only a single model scale. In this study, we evaluated the relationship between model scale and downstream task performance for structured medical foundation models. Using a random sample (2.3 million patients, 32 hospitals) from a nationwide 519-hospital Japanese claims database, we pretrained encoder-only Transformers at five scales (2.2M-101M parameters) for disease incidence and medication prediction. Downstream performance saturated at task-dependent thresholds: disease prediction benefited from larger models (32M-101M), whereas medication prediction saturated at 11M, reducing pretraining time by 178 h. Across all tasks, the best-performing model consistently outperformed a Light Gradient Boosting Machine baseline in the area under the precision-recall curve. These findings indicate that, unlike the monotonically decreasing pretraining loss, the optimal model size varied depending on task characteristics. This task-dependent saturation provides practical guidance for balancing predictive performance and computational cost in structured medical foundation models.

Nanae Aratake, Taisei Tosaki, Yuji Okamoto +5

1Graduate School of Medicine, Kyoto University, Kyoto, Japan. · 2Medical Data Vision Co., Ltd., Tokyo, Japan. · 3IQVIA Solutions Japan G.K., Tokyo, Japan. +1

Model Size Medical World Model Wireless Foundation Models Pretraining Patient Trajectories Downstream Reasoning

Ask PDF