Effectiveness of Automatically Curated Dataset in Thyroid Nodules Classification Algorithms Using Deep Learning

Jichen Yang; Jikai Zhang; Benjamin Wildman-Tobriner; Maciej A. Mazurowski

cs.CVFeb 1, 2026

Effectiveness of Automatically Curated Dataset in Thyroid Nodules Classification Algorithms Using Deep Learning

Authors: Jichen Yang, Jikai Zhang, Benjamin Wildman-Tobriner, Maciej A. Mazurowski

Organizations: Department of Electrical & Computer Engineering, Duke University, Durham, NC, USA · Department of Radiology, Duke University Medical Center, Durham, NC, USA

Abstract

The diagnosis of thyroid nodule cancers commonly utilizes ultrasound images. Several studies showed that deep learning algorithms designed to classify benign and malignant thyroid nodules could match radiologists' performance. However, data availability for training deep learning models is often limited due to the significant effort required to curate such datasets. Our previous study proposed a method (MADLaP) to curate thyroid nodule datasets automatically. It was tested to have a 63% yield rate and 83% accuracy. However, the usefulness of the generated data for training deep learning models remains unknown. In this study, we conducted experiments to determine whether using an automatically-curated dataset improves deep learning algorithms' performance. The expert annotations of 378 patients either trained a model directly (803 images) or built MADLaP, which curated 5,228 images from 3,981 patients; the same network and training method were used throughout, and 10-model cross-validation ensembles were tested on 378 nodules from an earlier period. The model trained on the automatically-curated dataset had a significantly higher AUC than the model trained on the manually selected dataset (0.657 vs 0.607; difference 0.051, 95% CI: 0.020, 0.081; P = .001), an advantage seen in all 22 configuration pairs. On the same test set, this model was within the AUC range of four radiologists (0.63-0.66) and not significantly different from a benchmark trained on 1,278 expert-curated nodules (0.69; difference -0.036, 95% CI: -0.083, 0.011), whereas manually curated models fell significantly below it. The Stage-1 subset (86% accuracy) did not differ significantly from the full MADLaP dataset (difference 0.015, 95% CI: -0.006, 0.036). In conclusion, we showed that using an automatically-curated dataset can substantially increase the performance of deep learning algorithms.

Figures & tables

Figure 2: Study design. (a) Development cohort [ 60 ] and test set [ 50 ] , with their biopsy periods and every exclusion with its reason. (b) The three training sets: the Manual Set (images curated by a radiologist for 378 randomly selected patients, on whom MADLaP was developed), the MADLaP Set (the output of MADLaP run on all 3,981 patients) and the S1 Set (the output of MADLaP’s first stage). (c) The same network and procedure were used for every set; the ten fold models of each configuration were averaged into an ensemble and evaluated on the test set.

Characteristic	Development cohort	Manual Set	MADLaP Set	S1 Set	Test set
Institution; biopsy period	Duke; Oct 2014 – Oct 2020	Duke; development cohort	Duke; development cohort	Duke; development cohort	Duke; Jan 2004 – Oct 2014
Patients	3,981 (of 4,354 queried)	378 (random)	run on all 3,981	subset of MADLaP	320
Nodules	—	404	2,605	2,066	378
Images (benign / malignant)	—	803 (752 / 51)	5,228 (4,970 / 258)	4,150 (3,958 / 192)	756 (462 / 294)
Malignant nodules, n (%)	—	26 (6.4)	129 (5.0)	96 (4.6)	147 (38.9)
Image selection and label	—	radiologist; pathology report	MADLaP Stages 1–2	MADLaP Stage 1	radiologists; cytopathology

Table 1: Characteristics of the development cohort, the training sets and the test set.

Figure 3: ROC curves on the test set (378 nodules, 147 malignant; nodule level) of the primary MADLaP and Manual models (10-model ensembles; MADLaP 512/250, Manual 256/250) and of the benchmark network. Markers are the biopsy recommendations of four radiologists for the same nodules (filled) and the ACR TI-RADS recommendations of three of them (open; radiologist 3’s on 377 nodules) [ 50 , 51 ] ; they are decisions, not ROC curves. AUCs with 95% DeLong CIs in the legend; the benchmark AUC was recomputed from its predictions (Table 3 ).

Model (batch/epochs)	Training images (malignant)	AUC (95% CI)	Fold models, mean $\pm$ SD	Difference vs Manual 256/250 (95% CI)	P
Manual 256/250	803 (51)	0.607 (0.547, 0.666)	0.600 $\pm$ 0.015	reference	—
MADLaP 512/250 (primary)	5,228 (258)	0.657 (0.600, 0.714)	0.641 $\pm$ 0.016	0.051 (0.020, 0.081)	.001
MADLaP 256/300 (same batch size)	5,228 (258)	0.662 (0.605, 0.719)	0.638 $\pm$ 0.016	0.055 (0.022, 0.088)	.001
MADLaP 512/350	5,228 (258)	0.666 (0.608, 0.723)	0.646 $\pm$ 0.026	0.059 (0.026, 0.092)	< .001
MADLaP 1,024/450 (highest CV AUC)	5,228 (258)	0.651 (0.594, 0.708)	0.635 $\pm$ 0.024	0.045 (0.014, 0.076)	.005
S1 512/350 (highest CV AUC)	4,150 (192)	0.677 (0.621, 0.732)	0.659 $\pm$ 0.014	0.070 (0.033, 0.107)	< .001

Table 2: Test-set discrimination of the 10-model and pooled ensembles (378 nodules, 147 malignant).

Radiologists (5-point score)
		MADLaP 512/250 minus comparator		Manual 256/250 minus comparator
Comparator	AUC (95% CI)	Difference (95% CI)	P	Difference (95% CI)	P
Radiologist 1	0.630 (0.585, 0.674)	0.027 ( $-$ 0.039, 0.094)	.42	$-$ 0.023 ( $-$ 0.093, 0.046)	.51
Radiologist 2	0.661 (0.609, 0.713)	$-$ 0.004 ( $-$ 0.070, 0.062)	.91	$-$ 0.055 ( $-$ 0.121, 0.012)	.11
Radiologist 3	0.649 (0.601, 0.697)	0.008 ( $-$ 0.055, 0.071)	.80	$-$ 0.042 ( $-$ 0.109, 0.024)	.21
Radiologist 4	0.627 (0.582, 0.673)	0.030 ( $-$ 0.033, 0.093)	.35	$-$ 0.021 ( $-$ 0.088, 0.046)	.54

Table 3: Paired comparison with radiologists, ACR TI-RADS and the benchmark on the same 378 nodules.

Figure 4: Difference in test-set AUC (378 nodules) between every automatic configuration (MADLaP or S1; batch size/epochs) and each Manual configuration (open: 128/200; filled: 256/250), with 95% DeLong CIs. Diamonds: the pooled MADLaP (90 models) and pooled S1 (20 models) ensembles of the complete configurations minus the pooled Manual ensemble (20 models). All 22 differences are positive and each has P < .05; after Holm adjustment, all 22 remain significant with DeLong variance and 20 with patient-clustered variance (Table A4 ). The 22 differences share two Manual ensembles and the same nodules, so they are correlated. Thick line: the primary comparison (MADLaP 512/250 minus Manual 256/250).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Version 1	Version 2	Basis
Test-set AUC	For each configuration, the highest test AUC among its ten fold models (e.g., MADLaP 0.694, S1 0.689, the batch-size points)	10-model ensemble for every configuration; fold-model mean $\pm$ SD; pooled ensembles	On the 383 nodules v1 used, the fold models whose test AUCs v1 reported exceeded their ensembles by 0.012 to 0.035
Manual result at batch size 512 (0.643)	Reported	Not reported; the Manual models compared have batch sizes 128 and 256	Model not archived
Confidence intervals and P values	About $\pm$ 0.03, bootstrap with unstated unit; “P > .43”	Nodule-level DeLong ( $\pm$ 0.056 to 0.059 for the ensembles); paired DeLong; patient-clustered sensitivity analyses; Holm adjustment	v1 intervals could not be reproduced with a standard resampling unit; “.43” is close to a one-sided bootstrap share (0.41 to 0.46), not a two-sided P value (0.81 to 0.84)
Section 3.1 cross-validation	0.78 (0.74, 0.82) for both sets, which is the cross-validation result of the benchmark study [ 4 ]	This study’s cross-validation for every configuration	—
Epoch selection	“selected based on ROCAUC in 10-fold cross validation”	Epochs fixed per configuration; epoch variants had been compared on the test set, so all archived configurations are reported	Training records
S1 vs MADLaP	S1 slightly lower; “use all the data”	No significant difference; S1 numerically higher	Ensemble estimates

Appendix

Table A1: Changes from version 1.

Item	Version 1	Version 2	Basis
Development period	2013–2019	October 2014 to October 2020	Zhang et al. [60]
Stage-1 accuracy	88%	86% (44 of 51)	Zhang et al. [60]
Radiologist experience	4 years post-fellowship	3 years post-fellowship at the time of annotation	Zhang et al. [60]
Benchmark AUC	0.70 (0.64–0.75)	0.69 (0.64, 0.75)	Weng et al. [50]
Annotated patients	320	378	320 is the number of test-set patients
Table 1	“Training dataset”; age rows inconsistent	Characteristics of every set; sex of the 4,354 queried patients kept; ages not reported	Sex counts sum to the 4,354 queried patients; the per-sex mean ages do not reproduce the overall mean; the source table was not available

Appendix

Table A2: Changes from version 1 (continued).

		Test set (378 nodules)
Set and batch/epochs	CV pooled out-of-fold AUC (95% CI)	Ensemble AUC (95% CI)	Fold models, mean $\pm$ SD	Highest fold model	P vs radiologists	P vs averaged score	P vs benchmark
Manual Set
128/200	0.730 (0.633, 0.827)	0.612 (0.553, 0.671)	0.605 $\pm$ 0.020	0.632	.15–.65	.001	.001
256/250	0.767 (0.672, 0.863)	0.607 (0.547, 0.666)	0.600 $\pm$ 0.015	0.629	.11–.54	< .001	.001
MADLaP Set
256/300	0.709 (0.660, 0.758) *	0.662 (0.605, 0.719)	0.638 $\pm$ 0.016	0.660	.28–.98	.073	.19

Appendix

Table A3: All trained configurations.

	vs Manual 128/200				vs Manual 256/250
Automatic configuration	Difference (95% CI)	P	Holm P, DeLong	Holm P, clustered	Difference (95% CI)	P	Holm P, DeLong	Holm P, clustered
MADLaP 256/300	0.050 (0.019, 0.081)	.002	.015	.013	0.055 (0.022, 0.088)	.001	.011	.013
MADLaP 512/250	0.045 (0.020, 0.071)	< .001	.009	.010	0.051 (0.020, 0.081)	.001	.011	.013
MADLaP 512/300	0.047 (0.021, 0.073)	< .001	.006	.009	0.053 (0.022, 0.083)	< .001	.009	.013
MADLaP 512/340	0.030 (0.003, 0.057)	.027	.048	.051	0.036 (0.005, 0.067)	.024	.048	.051
MADLaP 512/350	0.053 (0.026, 0.081)	< .001	.004	.004	0.059 (0.026, 0.092)	< .001	.009	.010

Appendix

Table A4: Automatic minus manual difference for all 22 configuration pairs.

Figure A1 : Test AUC (378 nodules) of the ten fold models (dots), their mean (tick) and their ensemble (large marker, 95% DeLong CI) for every complete configuration (batch size/epochs). Rings mark the fold model whose test AUC version 1 reported for that configuration; these fold models exceeded their ensembles by 0.010 to 0.037. Band: range of four radiologists; dashed line: benchmark network.

Figure A2 : Cross-validation. (a) Held-out image-level AUC during a 10-fold cross-validation run on the MADLaP Set (batch size 128, 250 epochs) whose models were not archived: folds (thin; one fold’s log ends at 210 epochs), mean

\pm

SD (band), training AUC (dashed). (b) Pooled out-of-fold nodule-level AUC (95% DeLong CI) of every complete configuration under its identified fold assignment (404, 2,605 and 2,066 nodules for the Manual, MADLaP and S1 Sets). Each set is scored against its own labels.

Analysis	Difference (95% CI)	P	Note
MADLaP 512/250 minus Manual 256/250
Main analysis (378 nodules)	0.051 (0.020, 0.081)	.001	paired DeLong
383 nodules (original labels)	0.053 (0.023, 0.084)	< .001	pooled MADLaP minus Manual 0.052 (0.024, 0.080), P < .001; positive pairs 22 of 22; Holm-significant 22 (DeLong), 22 (clustered)
381 nodules with radiologist scores	0.054 (0.024, 0.085)	< .001	pooled MADLaP minus Manual 0.054 (0.026, 0.082), P < .001; positive pairs 22 of 22; Holm-significant 22 (DeLong), 22 (clustered)
377 nodules without label disagreement	0.050 (0.019, 0.080)	.001	pooled MADLaP minus Manual 0.048 (0.020, 0.076), P < .001; positive pairs 22 of 22; Holm-significant 20 (DeLong), 20 (clustered)
Image level (756 images)	0.049 (0.022, 0.076)	< .001	AUC 0.649 vs 0.600; nodule-cluster bootstrap CI; P from variance clustered by nodule

Appendix

Table A5: Sensitivity analyses.

Radiologist decision set	Biopsies n / nodules N	Radiologist sensitivity / specificity	Sensitivity / specificity	Additional cancers	McNemar P	Holm P	Sensitivity / specificity	Additional cancers	McNemar P	Holm P
			MADLaP 512/250				Manual 256/250
Radiologists’ own recommendations
Radiologist 1	273 / 378	0.735 / 0.286	0.823 / 0.342	13	.098	.59	0.782 / 0.316	7	.43	> .99
Radiologist 2	305 / 378	0.830 / 0.208	0.878 / 0.238	7	.31	> .99	0.844 / 0.216	2	.87	> .99
Radiologist 3	274 / 378	0.769 / 0.303	0.830 / 0.342	9	.23	> .99	0.782 / 0.312	2	.89	> .99
Radiologist 4	270 / 378	0.687 / 0.268	0.810 / 0.346	18	.027	.19	0.782 / 0.329	14	.10	.72

Appendix

Table A6: Operating points at the radiologists’ numbers of biopsies.

Block	Operation	Output	Parameters
Input	—	160 $\times$ 160 $\times$ 1	—
Conv 1	3 $\times$ 3, 8; pool	80 $\times$ 80 $\times$ 8	80
Conv 2	3 $\times$ 3, 12; pool	40 $\times$ 40 $\times$ 12	876
Conv 3	3 $\times$ 3, 16; pool	20 $\times$ 20 $\times$ 16	1,744
Conv 4	3 $\times$ 3, 24; pool	10 $\times$ 10 $\times$ 24	3,480
Conv 5	3 $\times$ 3, 32; pool	5 $\times$ 5 $\times$ 32	6,944

Appendix

Table A7 : Network.

Thyroid Nodule Detection

See all 9 papers

Calibrated Selective Prediction Using Deep Ensembles for ROI-Based Thyroid Nodule Ultrasound Classification Under Dataset Shift: A Retrospective Evaluation
Jul 13, 2026Md. Sadibul Hasan Sadib, Md. Mohayminul Mukit, Rahmatul Kabir Rasel Sarker +2Probability Calibration Selective Prediction
Department of Computer Science and Engineering, Daffodil International University, Dhaka, Bangladesh. · Department of Computer Science and Engineering, Daffodil International University, Savar, Dhaka, Bangladesh. · Department of Public Health, Daffodil International University, Dhaka, Bangladesh. +1
CMCNet: Aligning Ultrasound Image Embeddings with Textual TI-RADS Representations for Fine-Grained Thyroid Classification
Aug 14, 2026Bingxin Yu, Xueli Wang, Jerry Zhou +6Thyroid Nodule Detection Ultrasound Image Classification
China-Japan Union Hospital of Jilin University, Changchun, 130012, P.R. China. · College of Computer Science and Technology, Jilin University, Changchun, 130012, P.R. China. · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun, 130012, P.R. China. +3
Prototype-Enhanced Multi-View Learning for Thyroid Nodule Ultrasound Classification
Mar 30, 2026Yangmei Chen, Zhongyuan Zhang, Xikun Zhang +4Thyroid Nodule Detection Ultrasound Image Classification
Guangdong Institute of Smart Education, Jinan University, Guangzhou 510632, China

Jul 13, 2026J/K move · Enter open · S save

Calibrated Selective Prediction Using Deep Ensembles for ROI-Based Thyroid Nodule Ultrasound Classification Under Dataset Shift: A Retrospective Evaluation

Md. Sadibul Hasan Sadib, Md. Mohayminul Mukit, Rahmatul Kabir Rasel Sarker, Tahmid Alam Tamim+1

Department of Computer Science and Engineering, Daffodil International University, Dhaka, Bangladesh. · Department of Computer Science and Engineering, Daffodil International University, Savar, Dhaka, Bangladesh. · Department of Public Health, Daffodil International University, Dhaka, Bangladesh.+1

Probability Calibration Selective Prediction Clinical Decision Support Thyroid Nodule Detection Deep Ensembles Medical Image Classification

Open paper AskPDF arXiv

Background: Deep learning models can classify thyroid nodules on ultrasound, but reliable clinical decision support also requires calibrated probabilities, uncertainty estimation, and selective referral, particularly under dataset shift. Methods: We developed a calibrated deterministic five-member deep ensemble for ROI-based thyroid nodule classification and selective image-based triage. TN5000 was used for model development, five-fold cross-validation, member-wise vector-scaling calibration, and fold-specific threshold selection. TN3K served as an independent external dataset-shift evaluation. The framework used ConvNeXt-Tiny with squeeze-and-excitation attention, ensemble-mean malignancy probability, and mutual information (MI) as an ensemble-disagreement score. A three-tier policy assigned images to No-FNA suggestion, FNA recommendation, or radiologist review. Results: On pooled out-of-fold TN5000 predictions, the ensemble achieved AUC-ROC 0.9395, AP 0.9715, ECE 0.0088, and Brier score 0.0813. At 50% nominal MI retention, 7.2% of cases received a No-FNA suggestion, 39.9% an FNA recommendation, and 52.9% radiologist review, with 98.3% No-FNA NPV and 99.83% malignancy capture. On TN3K, AUC-ROC decreased to 0.7870, AP to 0.7254, ECE increased to 0.1899, and Brier score to 0.2281. The frozen TN5000 policy assigned 83.7% to review, 1.0% to No-FNA, and 15.3% to FNA recommendation. No malignant image entered the No-FNA pathway, but FNA-recommendation PPV fell to 76.6%. Conclusion: The framework showed strong internal discrimination and calibration, but limited external threshold transportability. Selective prediction may help identify images unsuitable for automated triage, but local recalibration, threshold validation, and prospective clinical evaluation are required before deployment.

Abstract

Thyroid Nodule Detection

Explore similar work

Calibrated Selective Prediction Using Deep Ensembles for ROI-Based Thyroid Nodule Ultrasound Classification Under Dataset Shift: A Retrospective Evaluation

CMCNet: Aligning Ultrasound Image Embeddings with Textual TI-RADS Representations for Fine-Grained Thyroid Classification

Prototype-Enhanced Multi-View Learning for Thyroid Nodule Ultrasound Classification