On Cross-Validation for Hyperparameter Optimization of Deep Learning Image Classifiers

Ljubomir Buturovic

cs.CVAug 10, 2026

On Cross-Validation for Hyperparameter Optimization of Deep Learning Image Classifiers

Authors: Ljubomir Buturovic

Organizations: East Palo Alto, United States · East Palo Alto, California

Abstract

Hyperparameter optimization (HPO) can materially affect the performance of deep learning (DL) image classifiers, but there is little empirical guidance on how to derive the validation signal that drives it, especially for the small sample sizes common in fields such as medical imaging. We compared three HPO protocols in terms of {\em absolute performance-estimation error} (AEE; the absolute difference between the winning configuration's validation AUROC and its test AUROC): fixed holdout (F), reshuffled holdout (R), and 5-fold cross-validation (C). The search space, sampler, training procedure, architecture, and test set were held identical across protocols. We evaluated the protocols on three public datasets spanning two regimes: binary medical imaging (RSNA pneumonia radiographs and binarized HAM10000 skin lesions) and 200-class natural imaging (Tiny ImageNet), across a range of development set sizes $n$ and two backbones (ResNet-18 on all datasets, Vision Transformer (ViT-S/16) on RSNA). On the medical datasets, every point estimate favored cross-validation over both holdout protocols, with reductions in AEE largest at small sample sizes and diminishing as $n$ increased. This pattern remained robust under conservative family-wise adjustment. On Tiny ImageNet, AEE was negligible under all three protocols. Test AUROC was generally similar among protocols. Fixed holdout had lower mean AEE than reshuffled holdout in 11 of 12 medical conditions, although this secondary finding was less uniformly supported. For small-sample medical image classification, we recommend cross-validation-based HPO when computational resources permit because it trades additional computation for a more reliable development-time estimate of subsequent test performance.

Figures & tables

Dataset	Classes	$n$ values	Test images
RSNA	2	100 / 300 / 1000 / 3000	21,348
BHAM	2	100 / 300 / 1000 / 3000	2,003
Tiny ImageNet	200	2000 / 5000 / 10000	10,000

Table 1: Datasets used in the analyses: subsample sizes

n

and the size of the shared held-out test set.

Figure 1: Comparison of test set performance and absolute performance estimation error for different datasets, HPO methods and architectures. First three rows are RSNA, BHAM and TIN, respectively, using ResNet-18 backbone; last row is RSNA using ViT backbone. The x axis is the subsample size

n

(the number of labeled images drawn for the run). For C , all

n

images are used in cross-validation; for F / R ,

n

is split 80/20 into training and validation, so the plotted value

n

includes the held-out validation images (the model trains on

0.8\,n

). The y axis is test set AUROC in the first column and AEE in the second. Points show means across subsamples and random seeds; lines are included to show trends with development sample size. Formal paired comparisons between protocols are reported in Table 2 and in the text.

Dataset	$n$	Comparison	$\Delta$ AEE
RSNA (ResNet-18)	100	C $-$ F	$-0.116\,[-0.152,-0.081]$
		C $-$ R	$-0.150\,[-0.186,-0.116]$
	300	C $-$ F	$-0.030\,[-0.068,+0.008]$
		C $-$ R	$-0.080\,[-0.104,-0.054]$
	1000	C $-$ F	$-0.018\,[-0.032,-0.003]$
		C $-$ R	$-0.041\,[-0.050,-0.032]$

Table 2: Paired differences in absolute estimation error between cross-validation and holdout protocols on the medical datasets. Differences are calculated as first protocol minus second protocol; negative values therefore favor the first protocol by indicating lower AEE. Values are mean paired differences with pointwise 95% confidence intervals from crossed random-effects models with random intercepts for development subsample and seed; confidence intervals were obtained by parametric bootstrap with 10,000 replicates. Each comparison comprises 25 paired runs (five subsamples crossed with five seeds).

Dataset	$n$	$\Delta$ AEE: F $-$ R
RSNA (ResNet-18)	100	$-0.035\,[-0.076,+0.008]$
	300	$-0.049\,[-0.080,-0.018]$
	1000	$-0.023\,[-0.040,-0.005]$
	3000	$-0.013\,[-0.020,-0.005]$
BHAM (ResNet-18)	100	$+0.003\,[-0.060,+0.066]$
	300	$-0.030\,[-0.045,-0.016]$

Table 3: Paired differences in absolute estimation error between fixed and reshuffled holdout on the medical datasets. Differences are calculated as F minus R ; negative values therefore favor fixed holdout by indicating lower AEE. Values are mean paired differences with pointwise 95% confidence intervals from crossed random-effects models with random intercepts for development subsample and seed; confidence intervals were obtained by parametric bootstrap with 10,000 replicates. Each comparison comprises 25 paired runs (five subsamples crossed with five seeds).

Figure 2: Comparison of top-1 accuracy for TIN dataset for different sample sizes.

Metric	Comparison	Concordant	Total
AEE	C $-$ F	15	15
AEE	C $-$ R	15	15
AEE	F $-$ R	14	15
Test AUROC	C $-$ F	9	15
Test AUROC	C $-$ R	9	15
Test AUROC	F $-$ R	13	15

Table 4: Agreement between the primary and sensitivity analyses. A comparison was classified as concordant when all three pointwise 95% confidence intervals either included zero or excluded zero in the same direction. Each row comprises 15 comparisons across the applicable datasets, architectures, and sample sizes.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	$n$	Comparison	$\Delta$ AEE	$\Delta$ test AUROC
RSNA (ResNet-18)	100	C $-$ F	$-0.116\,[-0.152,-0.081]$	$+0.051\,[+0.014,+0.088]$
		C $-$ R	$-0.150\,[-0.186,-0.116]$	$+0.040\,[+0.008,+0.071]$
		F $-$ R	$-0.035\,[-0.076,+0.008]$	$-0.011\,[-0.067,+0.045]$
	300	C $-$ F	$-0.030\,[-0.068,+0.008]$	$+0.012\,[+0.001,+0.023]$
		C $-$ R	$-0.080\,[-0.104,-0.054]$	$+0.016\,[+0.008,+0.024]$
		F $-$ R	$-0.049\,[-0.080,-0.018]$	$+0.004\,[-0.009,+0.017]$

Appendix

Table 5: Complete paired comparisons of absolute estimation error and test AUROC across all experimental conditions, for RSNA and BHAM, using ResNet-18. Differences are calculated as the first protocol minus the second protocol. Negative values in

\Delta

AEE favor the first protocol because they indicate lower absolute estimation error; positive values in

\Delta

test AUROC favor the first protocol. Values are mean paired differences with pointwise 95% confidence intervals from crossed random-effects models with random intercepts for development subsample and seed; confidence intervals were obtained by parametric bootstrap with 10,000 replicates. The comparisons comprise 25 paired runs (five subsamples crossed with five seeds).

Dataset	$n$	Comparison	$\Delta$ AEE	$\Delta$ test AUROC
RSNA (ViT)	100	C $-$ F	$-0.109\,[-0.164,-0.053]$	$+0.058\,[-0.021,+0.138]$
		C $-$ R	$-0.130\,[-0.152,-0.109]$	$+0.004\,[-0.013,+0.020]$
		F $-$ R	$-0.022\,[-0.095,+0.049]$	$-0.055\,[-0.143,+0.035]$
	300	C $-$ F	$-0.042\,[-0.060,-0.023]$	$+0.021\,[+0.000,+0.042]$
		C $-$ R	$-0.060\,[-0.099,-0.023]$	$+0.010\,[+0.001,+0.019]$
		F $-$ R	$-0.018\,[-0.057,+0.019]$	$-0.011\,[-0.035,+0.015]$

Appendix

Table 6: Complete paired comparisons of absolute estimation error and test AUROC across all experimental conditions for RSNA using ViT architecture, and TIN. Differences are calculated as the first protocol minus the second protocol. Negative values in

\Delta

AEE favor the first protocol because they indicate lower absolute estimation error; positive values in

\Delta

test AUROC favor the first protocol. Values are mean paired differences with pointwise 95% confidence intervals from crossed random-effects models with random intercepts for development subsample and seed; confidence intervals were obtained by parametric bootstrap with 10,000 replicates. RSNA comparisons comprise 25 paired runs (five subsamples crossed with five seeds), and Tiny ImageNet comparisons comprise 15 paired runs (five subsamples crossed with three seeds).

Hyperparameter Optimization

Hyperparameter optimization selects model and training settings to improve predictive performance, efficiency, or reliability under limited evaluation budgets.

See all 162 papers

Oct 20, 2026

A Multi-Agent Framework for Automated Machine Learning with Evidence-based Planning for Medical Image Analysis

Deep learning has become the default tool for medical image analysis, driven by the open-source movement. However, achieving the best performance on a given task remains difficult, with architecture selection and hyperparameter optimization (HPO) still considered "more of an art than a science". Automated machine learning (AutoML) automates parts of model training, yet helps little in medical imaging: general-purpose approaches lack medical architectures and preprocessing, and medical-focused ones demand engineering expertise and remain confined to narrow tasks such as 3D segmentation. We introduce MedicalMLOmni, a multi-agent AutoML framework that automates model training for diverse single- and multi-modal medical tasks through a natural language interface. Specifically, MedicalMLOmni performs evidence-based planning by retrieving published results to guide model selection and configuration, and employs parallel HPO where multiple agents explore adjustment strategies concurrently. MedicalMLOmni achieves the best score on 19 of 20 benchmarks; PMC-VQA is the exception, where a vision-language model with broader pre-training leads. Backbone-matched controls attribute the gains to the framework, not the backbone. By removing coding requirements, MedicalMLOmni lowers the barrier for domain experts with rich datasets but limited engineering expertise.

Yiqing Shen, Mathias Unberath

Department of Computer Science, Johns Hopkins University

Automated Machine Learning Medical Image Analysis

Ask PDF

Sep 29, 2026

Evaluation Choices Shape Biomedical ML Claims: A Pediatric Pneumonia Benchmark Case Study

Biomedical machine learning papers often compress model performance into one headline number. That number can look like a property of the model even when it depends strongly on how the benchmark was evaluated. We study this problem on the widely used Kermany pediatric chest radiograph dataset using nine image classifiers and a controlled evaluation protocol. Under the same protocol, the eight pretrained backbones differ by only 0.026 AUROC. In contrast, changing whether the backbone is frozen or fine-tuned changes AUROC by 0.044 on average, and changing the decision threshold changes balanced accuracy by 0.090 on average. The official test split is also measurably different from the training pool: a partition classifier distinguishes them at AUC 0.697, rising to 0.898 for normal radiographs. Most strikingly, a classifier using only file properties, with no image anatomy, reaches 0.992 balanced accuracy within the training pool but falls to 0.496 on the official test split. Validation-fitted thresholds and calibration also transfer imperfectly. These results show that a high benchmark score can support different conclusions when the split, training policy, threshold, metric, calibration, and uncertainty are not communicated with it. We end with a seven-item reporting recommendation in which each item is tied to an effect measured in the study

Bhanu Prakash Vangala, Sowmya Guda, Latha Peddi +1

University of Missouri · Government Degree College · Amar Bio Tech Pvt Ltd

Data Leakage Chest X-Ray Classification Shortcut Learning Medical Image Benchmarks

Ask PDF

Jul 1, 2026

CPR: Chained Perceptual Refinement for Coarse-to-Fine Medical Image Classification

High resolution medical images contain fine grained, spatially sparse cues that are critical for diagnosis, yet preserving full resolution incurs substantial computational and memory costs. Most deep models process images uniformly, leading to redundant computation or loss of diagnostic detail under downsampling. We propose Chained Perceptual Refinement, CPR, a coarse to fine framework that formulates medical image analysis as a sequential global to local decision process. Starting from a low resolution global view, CPR dynamically predicts the location and spatial extent of refinement regions, extracts high resolution evidence from the original image, and incrementally integrates it with global context. By keeping the backbone input size fixed while contracting the perceptual field, CPR preserves diagnostic fidelity with constant peak GPU memory. Extensive experiments on five medical imaging datasets and multiple backbone architectures demonstrate that CPR consistently outperforms both fixed resolution and multi scale state of the art baselines, achieving improvements of up to 2.27 percentage points over the second best method. It also achieves up to a 19.6 fold reduction in GFLOPs at matched accuracy, establishing a superior accuracy and efficiency trade off for high resolution medical image analysis. The code is available on GitHub.

Si-Yuan Lu, Hanruo Zhu, Ziquan Zhu +6

Nanjing University of Posts and Telecommunications, Nanjing, China · University of Leicester, Leicester, UK · University of Exeter, Exeter, UK +1

Visual Attention Medical Image Analysis Medical Image Classification Image Classification

Ask PDF

Abstract

Hyperparameter Optimization

Explore similar work

A Multi-Agent Framework for Automated Machine Learning with Evidence-based Planning for Medical Image Analysis

Evaluation Choices Shape Biomedical ML Claims: A Pediatric Pneumonia Benchmark Case Study

CPR: Chained Perceptual Refinement for Coarse-to-Fine Medical Image Classification