eess.ASAug 26, 2026
SaveWhy ML-based cough models do not generalize: a systematic cross-dataset evaluation for tuberculosis screening
Organizations: Embedded Systems Laboratory, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland. · Basque Center for Applied Mathematics (BCAM), Bilbao, Spain. · University of the Basque Country (EHU), Leioa, Spain.
Abstract
Cough acoustics are promising for non-invasive tuberculosis (TB) screening, yet whether machine learning (ML) models capture disease-related acoustics or artifacts of data collection remains unresolved. We evaluated the cross-dataset generalizability of classical ML and deep learning (DL) cough-based TB classifiers across three independent datasets. Despite moderate within-dataset performance (ROC-AUC up to ), both pipelines fail to generalize, with external performance frequently below 0.6, indicating a possible limitation of the data. We further observed audio representations are organized by recording device and dataset rather than TB status, predicted TB probability tracks country-level prevalence in CODA, and device mismatch degrades transfer while device-diverse training improves it. Additionally, a clinical-variable baseline generalizes more consistently (ROC-AUC ), indicating acquisition-specific variability is a stronger driver of poor generalizability than population shift. High within-dataset performance is not enough. External validation is essential before cough-based TB models are clinically ready.
Figures & tables
Figure 1: Overview of the study design. The study evaluates cross-dataset generalization of cough-based TB classifiers across three public datasets using classical ML and DL pipelines. Four major analyses are conducted: cross-dataset generalization, acquisition-related bias in CODA, device generalization in Zambia, and a clinical-variable baseline. Key findings and implications for future study design are also presented.
Figure 2: Cross-dataset performance of the acoustic pipelines and the clinical-variable baseline. For each source dataset, the model configuration was selected using within-dataset validation and then evaluated on all target datasets. Values show subject-level ROC-AUC, with variability reported across outer folds. Across both acoustic pipelines, within-dataset performance did not translate into consistent external generalization, and the selected configuration differed by source dataset. In contrast, the clinical-variable baseline showed more stable external generalization across datasets.
Figure 3: MMD-based MDS visualization of subgroup structure in CLAP audio embeddings. Pairwise subgroup distances were computed using MMD and visualized using MDS. The axes do not have a physical meaning. They represent latent dimensions that preserve pairwise similarities, with points closer together indicating more similar data distributions. The different scatter points are colored by the dataset source, and they represent subgroups defined by recording device, location, or TB label. Subgroups defined by TB label were labeled in (c) to indicate positive (1) or negative (0) status. In CODA, location and device are overlapping factors because country source is associated with recording-device choice. Ideally, device/location should not cause much separation and subgroups by device/location of different dataset sources should overlap, whereas subgroups by TB status should cluster by their TB labels. However, it is the opposite in reality.
Figure 4: Predicted TB probabilities by country source and TB label in CODA. The blue line shows the country-level TB+ rate. (a) Under the challenge-replication setting, predicted TB probabilities increased strongly with country-level TB prevalence for both TB+ and TB- samples. (b) Under the conservative setting, predicted probabilities were less extreme and the association with country-level prevalence was reduced, although not fully removed.
Figure 5: Recording-device effects on generalization in the Zambia dataset. Values are subject-level ROC-AUC. (a) Single-device training. The left heatmap reports ROC-AUC for each training-device and testing-device pair, with diagonal entries showing within-device evaluation and off-diagonal entries showing cross-device evaluation; the right heatmap shows the change in ROC-AUC of each cross-device evaluation relative to the corresponding within-device value. Cross-device evaluation generally reduced performance. (b) Multi-device training. The left heatmap reports ROC-AUC for models trained on three devices and evaluated on either the training devices or the held-out device. The right plot compares held-out-device performance between single-device training and training on the other three devices, where the single-device value is the average over the three possible single-device choices. Training across multiple devices generally improved performance on the unseen device.
| Classical machine learning | Deep learning | |
|---|---|---|
| Filtering | Bandpass: lower cut-off 10–500 Hz, upper cut-off 4,000–8,000 Hz. | |
| Augmentation | Crop-and-pad, additive Gaussian noise, Brownian noise, tape-speed perturbation, amplitude scaling, pitch shifting, time stretching, circular time shifting Blankemeier et al. (2023) . | |
| • Additionally: SpecAugment (frequency and time masking) | ||
| Features | Feature selection via recursive backward elimination. • CLAP embeddings • Time-frequency domain features (Supplementary F) | • Spectrogram • Mel-spectrogram • Scalogram (continuous wavelet transform) |
| Models | • Random forest • Logistic regression • Gradient boosting | • ResNet-18, ResNet-34 • VGGish • OPERA • CLAP |
Table 1: Summary of explored configurations for the classical ML and deep learning pipelines. Hyperparameters of each augmentation method were determined by grid search.
Extended Data Fig. 1 : Classical ML performance across training datasets, testing datasets, classifiers, feature representations, and feature-selection settings. Each heatmap corresponds to one source training dataset. Rows show combinations of classifier, feature type, and RFE; columns show testing datasets. Performance is reported as subject-level ROC-AUC. Overall, performance varied more strongly by training and testing dataset than by classifier choice, feature representation, or use of feature selection.
Extended Data Fig. 2 : Within-dataset DL performance for the Zambia dataset, stratified by recording device and by location. Subject-level ROC-AUC for the DL pipeline evaluated within Zambia, shown separately for each recording device and for each recruitment site.
Extended Data Fig. 3 : Within-dataset performance analysis for TBscreen dataset stratified by device. Subject-level ROC-AUC for the DL pipeline evaluated within TBscreen passive cough, shown separately for each recording device. Performance is similar between Pixel and codec, but is lower when using yeti.
Extended Data Fig. 4 : Within-dataset performance analysis for CODA dataset stratified by device. Subject-level ROC-AUC for the DL pipeline evaluated within CODA, shown separately for each recording device, each of which also corresponds to a different country. Performance variability is too large to draw any conclusion.
Extended Data Fig. 5 : Feature importance of the clinical-variable baseline. The left panel shows mean logistic-regression coefficients across outer CV folds for each training dataset. Positive coefficients indicate an association with higher predicted TB probability, while negative coefficients indicate an association with lower predicted TB probability. The right panel shows permutation importance, measured as the mean decrease in ROC-AUC after randomly permuting each feature in the testing fold.
| Dataset | Subgroup | Device | Number of subjects | Number of cough events | TB+ rate (%) | Usage |
|---|---|---|---|---|---|---|
| CODA | Vietnam | OPPOA54 | 161 | 1791 | 40 | Train / Vali |
| CODA | Madagascar | Motorola G9 play | 157 | 1293 | 48 | Train / Vali |
| CODA | Tanzania | Nokia 3.4 Ta-1288 | 87 | 1004 | 16 | Train / Vali |
| TBscreen | Passive cough | Pixel | 118 | 3762 | 70 | Train / Vali |
| TBscreen | Passive cough | Boundary microphone | 114 | 3370 | 72 | Train / Vali |
| TBscreen | Passive cough | Condenser microphone | 106 | 2606 | 74 | Train / Vali |
Extended Data Table 1 : Summary of the study-specific dataset subsets analyzed in this work . Disease rate is reported at the subject level.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 1: Feature-selection consistency for CLAP embeddings with random forest classifiers. Jaccard similarity was used to compare selected feature subsets across source datasets and across CV folds. The selected CLAP dimensions showed greater overlap than random selection, suggesting that RFE identified some stable embedding dimensions in this setting. For between-fold consistency, the mean number of selected features and the mean Jaccard similarity between folds were plotted as scatter points and marked on the plot.
Figure 2: MMD-based MDS visualization of subgroup structure in handcrafted time-frequency features. As with CLAP embeddings, the time-frequency feature space showed stronger separation by recording device than by TB label. However, the separation by dataset domain is generally weaker than CLAP embeddings.
Figure 3: t-SNE visualization of sample-level feature structure in CLAP embedding by dataset and recording device. Individual cough samples were projected into two dimensions using t-SNE. Samples showed strong organization by dataset source and recording device. This acquisition-related clustering was more prominent than separation by TB label, supporting the conclusion that dataset and device structure dominate the feature spaces used by the classical ML models.
Figure 4: t-SNE visualization of sample-level feature structure in CLAP embedding by dataset and TB status. Similarly, samples showed strong organization by dataset source, while distribution by TB status seems random.
Figure 5: t-SNE visualization of sample-level feature structure in time/frequency-domain descriptors by dataset and recording device. Similar to CLAP embeddings, samples showed strong organization by dataset source and recording device.
Figure 6: t-SNE visualization of sample-level feature structure in time/frequency-domain descriptors by dataset and TB status. Similar to previous plot, organization by dataset is strong and organization by TB status is weak.
Figure 7: Illustration of the nested CV scheme. In each outer split, one fold is held out for testing, while the remaining four folds are used for model development. Within the development folds, inner CV is used for hyperparameter selection, feature selection, or early stopping.