Why ML-based cough models do not generalize: a systematic cross-dataset evaluation for tuberculosis screening
Authors: Wensi Zhang, Tomas Teijeiro, Jérôme Thevenot, David Atienza
Organizations: Embedded Systems Laboratory, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland. · Basque Center for Applied Mathematics (BCAM), Bilbao, Spain. · University of the Basque Country (EHU), Leioa, Spain.
Cough acoustics are promising for non-invasive tuberculosis (TB) screening, yet whether machine learning (ML) models capture disease-related acoustics or artifacts of data collection remains unresolved. We evaluated the cross-dataset generalizability of classical ML and deep learning (DL) cough-based TB classifiers across three independent datasets. Despite moderate within-dataset performance (ROC-AUC up to 0.755±0.056), both pipelines fail to generalize, with external performance frequently below 0.6, indicating a possible limitation of the data. We further observed audio representations are organized by recording device and dataset rather than TB status, predicted TB probability tracks country-level prevalence in CODA, and device mismatch degrades transfer while device-diverse training improves it. Additionally, a clinical-variable baseline generalizes more consistently (ROC-AUC 0.655−0.711), indicating acquisition-specific variability is a stronger driver of poor generalizability than population shift. High within-dataset performance is not enough. External validation is essential before cough-based TB models are clinically ready.
Figures & tables
Figure 1: Overview of the study design. The study evaluates cross-dataset generalization of cough-based TB classifiers across three public datasets using classical ML and DL pipelines. Four major analyses are conducted: cross-dataset generalization, acquisition-related bias in CODA, device generalization in Zambia, and a clinical-variable baseline. Key findings and implications for future study design are also presented.
Figure 2: Cross-dataset performance of the acoustic pipelines and the clinical-variable baseline. For each source dataset, the model configuration was selected using within-dataset validation and then evaluated on all target datasets. Values show subject-level ROC-AUC, with variability reported across outer folds. Across both acoustic pipelines, within-dataset performance did not translate into consistent external generalization, and the selected configuration differed by source dataset. In contrast, the clinical-variable baseline showed more stable external generalization across datasets.
Figure 3: MMD-based MDS visualization of subgroup structure in CLAP audio embeddings. Pairwise subgroup distances were computed using MMD and visualized using MDS. The axes do not have a physical meaning. They represent latent dimensions that preserve pairwise similarities, with points closer together indicating more similar data distributions. The different scatter points are colored by the dataset source, and they represent subgroups defined by recording device, location, or TB label. Subgroups defined by TB label were labeled in (c) to indicate positive (1) or negative (0) status. In CODA, location and device are overlapping factors because country source is associated with recording-device choice. Ideally, device/location should not cause much separation and subgroups by device/location of different dataset sources should overlap, whereas subgroups by TB status should cluster by their TB labels. However, it is the opposite in reality.
Figure 4: Predicted TB probabilities by country source and TB label in CODA. The blue line shows the country-level TB+ rate. (a) Under the challenge-replication setting, predicted TB probabilities increased strongly with country-level TB prevalence for both TB+ and TB- samples. (b) Under the conservative setting, predicted probabilities were less extreme and the association with country-level prevalence was reduced, although not fully removed.
Figure 5: Recording-device effects on generalization in the Zambia dataset. Values are subject-level ROC-AUC. (a) Single-device training. The left heatmap reports ROC-AUC for each training-device and testing-device pair, with diagonal entries showing within-device evaluation and off-diagonal entries showing cross-device evaluation; the right heatmap shows the change in ROC-AUC of each cross-device evaluation relative to the corresponding within-device value. Cross-device evaluation generally reduced performance. (b) Multi-device training. The left heatmap reports ROC-AUC for models trained on three devices and evaluated on either the training devices or the held-out device. The right plot compares held-out-device performance between single-device training and training on the other three devices, where the single-device value is the average over the three possible single-device choices. Training across multiple devices generally improved performance on the unseen device.
Classical machine learning
Deep learning
Filtering
Bandpass: lower cut-off 10–500 Hz, upper cut-off 4,000–8,000 Hz.
Augmentation
Crop-and-pad, additive Gaussian noise, Brownian noise, tape-speed perturbation, amplitude scaling, pitch shifting, time stretching, circular time shifting Blankemeier et al. (2023) .
• Additionally: SpecAugment (frequency and time masking)
Features
Feature selection via recursive backward elimination. • CLAP embeddings • Time-frequency domain features (Supplementary F)
• Random forest • Logistic regression • Gradient boosting
• ResNet-18, ResNet-34 • VGGish • OPERA • CLAP
Table 1: Summary of explored configurations for the classical ML and deep learning pipelines. Hyperparameters of each augmentation method were determined by grid search.
Extended Data Fig. 1 : Classical ML performance across training datasets, testing datasets, classifiers, feature representations, and feature-selection settings. Each heatmap corresponds to one source training dataset. Rows show combinations of classifier, feature type, and RFE; columns show testing datasets. Performance is reported as subject-level ROC-AUC. Overall, performance varied more strongly by training and testing dataset than by classifier choice, feature representation, or use of feature selection.
Extended Data Fig. 2 : Within-dataset DL performance for the Zambia dataset, stratified by recording device and by location. Subject-level ROC-AUC for the DL pipeline evaluated within Zambia, shown separately for each recording device and for each recruitment site.
Extended Data Fig. 3 : Within-dataset performance analysis for TBscreen dataset stratified by device. Subject-level ROC-AUC for the DL pipeline evaluated within TBscreen passive cough, shown separately for each recording device. Performance is similar between Pixel and codec, but is lower when using yeti.
Extended Data Fig. 4 : Within-dataset performance analysis for CODA dataset stratified by device. Subject-level ROC-AUC for the DL pipeline evaluated within CODA, shown separately for each recording device, each of which also corresponds to a different country. Performance variability is too large to draw any conclusion.
Extended Data Fig. 5 : Feature importance of the clinical-variable baseline. The left panel shows mean logistic-regression coefficients across outer CV folds for each training dataset. Positive coefficients indicate an association with higher predicted TB probability, while negative coefficients indicate an association with lower predicted TB probability. The right panel shows permutation importance, measured as the mean decrease in ROC-AUC after randomly permuting each feature in the testing fold.
Dataset
Subgroup
Device
Number of subjects
Number of cough events
TB+ rate (%)
Usage
CODA
Vietnam
OPPOA54
161
1791
40
Train / Vali
CODA
Madagascar
Motorola G9 play
157
1293
48
Train / Vali
CODA
Tanzania
Nokia 3.4 Ta-1288
87
1004
16
Train / Vali
TBscreen
Passive cough
Pixel
118
3762
70
Train / Vali
TBscreen
Passive cough
Boundary microphone
114
3370
72
Train / Vali
TBscreen
Passive cough
Condenser microphone
106
2606
74
Train / Vali
Extended Data Table 1 : Summary of the study-specific dataset subsets analyzed in this work . Disease rate is reported at the subject level.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 1: Feature-selection consistency for CLAP embeddings with random forest classifiers. Jaccard similarity was used to compare selected feature subsets across source datasets and across CV folds. The selected CLAP dimensions showed greater overlap than random selection, suggesting that RFE identified some stable embedding dimensions in this setting. For between-fold consistency, the mean number of selected features and the mean Jaccard similarity between folds were plotted as scatter points and marked on the plot.
Figure 2: MMD-based MDS visualization of subgroup structure in handcrafted time-frequency features. As with CLAP embeddings, the time-frequency feature space showed stronger separation by recording device than by TB label. However, the separation by dataset domain is generally weaker than CLAP embeddings.
Figure 3: t-SNE visualization of sample-level feature structure in CLAP embedding by dataset and recording device. Individual cough samples were projected into two dimensions using t-SNE. Samples showed strong organization by dataset source and recording device. This acquisition-related clustering was more prominent than separation by TB label, supporting the conclusion that dataset and device structure dominate the feature spaces used by the classical ML models.
Figure 4: t-SNE visualization of sample-level feature structure in CLAP embedding by dataset and TB status. Similarly, samples showed strong organization by dataset source, while distribution by TB status seems random.
Figure 5: t-SNE visualization of sample-level feature structure in time/frequency-domain descriptors by dataset and recording device. Similar to CLAP embeddings, samples showed strong organization by dataset source and recording device.
Figure 6: t-SNE visualization of sample-level feature structure in time/frequency-domain descriptors by dataset and TB status. Similar to previous plot, organization by dataset is strong and organization by TB status is weak.
Figure 7: Illustration of the nested CV scheme. In each outer split, one fold is held out for testing, while the remaining four folds are used for model development. Within the development folds, inner CV is used for hyperparameter selection, feature selection, or early stopping.
The automatic identification of cough segments in audio through the determination of start and end points is pivotal to building scalable screening tools in health technologies for pulmonary related diseases. We propose the application of two current pre-trained architectures to the task of cough activity detection. A dataset of recordings containing cough from patients symptomatic for tuberculosis (TB) who self-present at community-level care centres in South Africa and Uganda is employed. When automatic start and end points are determined using XLS-R, an average precision of 0.96 and an area under the receiver-operating characteristic of 0.99 are achieved for the test set. We show that best average precision is achieved by utilising only the first three layers of the network, which has the dual benefits of reduced computational and memory requirements, pivotal for smartphone-based applications. This XLS-R configuration is shown to outperform an audio spectrogram transformer (AST) as well as a logistic regression baseline by 9% and 27% absolute in test set average precision respectively. Furthermore, a downstream TB classification model trained using the coughs automatically isolated by XLS-R comfortably outperforms a model trained on the coughs isolated by AST, and is only narrowly outperformed by a classifier trained on the ground truth coughs. We conclude that the application of large pre-trained transformer models is an effective approach to identifying cough end-points and that the integration of such a model into a screening tool is feasible.
Joshua Jansen van Vüren, Devendra Singh Parihar, Daphne Naidoo +4
Department of Electrical and Electronic Engineering, Stellenbosch University, Stellenbosch, South Africa · DSI-NRF Centre of Excellence for Biomedical Tuberculosis Research, South African Medical Research Council Centre for Tuberculosis Research, Division of Molecular Biology and Human Genetics, Faculty of Medicine and Health Sciences, Stellenbosch University, Cape Town, South Africa · Faculty of Business and Economics, University of Göttingen, Göttingen, Germany +1
We present a first evaluation of machine learning applied to patient clinical and demographic data gathered in two different countries for the purpose of tuberculosis (TB) screening to identify people who would benefit from expensive molecular testing. Experiments are based on the recently-compiled CAGE-TB dataset, which includes sub-cohorts of people with presumptive TB presenting at community health care centres in South Africa and Uganda. Three neural network architectures (logistic regression (LR), multilayer perceptrons (MLP) and convolutional neural networks (CNN)) are considered in conjunction with greedy feature selection. For the convolutional neural network, a strategy that jointly optimises feature selection and feature ordering is proposed and shown to lead to consistent development and test set improvements. For all three models, development set area under the receiver operating characteristic (AUROC) curve is improved by 2-7% using feature selection. LR after feature selection achieves an AUROC of 0.8 [0.75,0.86] (95% CI) and 0.84 [0.78,0.9] when testing on the held-out Ugandan and South African data respectively. Although outperforming LR on the development cohort, the deeper networks (MLP, CNN) show inconsistent trends on the held-out cohorts, while LR achieves performance within 1-2% of the best achieved in terms of AUROC. LR narrowly misses the WHO minimum requirements by 4-9% in sensitivity even though the network is being evaluated on a completely held-out cohort. The development of neural-network based classifiers for TB screening therefore appears viable.
Joshua M. Jansen van Vüren, Devendra S. Parihar, Daphne Naidoo +8
Department of Electrical and Electronic Engineering, University of Stellenbosch, Stellenbosch, South Africa · South African Medical Research Council Centre for Tuberculosis Research (CTR) Division of Molecular Biology and Human Genetics, Faculty of Medicine and Health Sciences Stellenbosch University, Cape Town, South Africa · Department of Global and Amsterdam Institute for Global Health and Development Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, The Netherlands +2
We consider the fusion of lung ultrasound images with routinely-collected clinical and demographic data for the purpose of automated tuberculosis (TB) screening using deep-learning. Such deep-learning based screening tools for TB could meaningfully support the health care system in Africa, where the burden of disease is severe and resources are constrained. Beginning with an established ResNet baseline for classification of lung ultrasound images, which achieves an area under the receiver operating characteristic (AUROC) curve of 0.91 [0.86,0.96] (95% CI), we consider the incorporation of the clinical and demographic data using three fusion approaches. We find that a simple average-based fusion of the output scores of separately-trained image and clinical data classifiers consistently matches or outperforms a more complex approach where the data is fused earlier and a combined classifier is trained. Fusing the image and the clinical classifiers in this way leads to a classifier with an overall AUROC of 0.95 [0.91,0.99] (specificity of 0.76 at sensitivity 0.93) which is an improvement of 4% absolute over the image-only baseline. We also find that greedy feature selection can be used to reduce the number of clinical and demographic inputs without sacrificing classification performance. Finally, when we differentiate between clinical and demographic data that are self-reported, that require some basic measurement or calculation, and that require a point-of-care (POC) test, we find the inclusion of the POC tests included in this study to be of minimal benefit to classification performance. We conclude that the incorporation of routinely-collected clinical and demographic data is a promising way to improve the performance of lung ultrasound based automatic classification.
Joshua M. Jansen van Vüren, Christiaan M. Geldenhuys, Devendra S. Parihar +7
Department of Electrical and Electronic Engineering, University of Stellenbosch, Stellenbosch, South Africa · Laboratory for intelligent Global Health and Humanitarian Response Technologies (LiGHT) Swiss Federal Institute of Technology (EPFL), Lausanne, Switzerland · National Teaching Hospital for Tuberculosis and Pulmonary Diseases (CNHU-PPC), Cotonou, Benin +1