Effect of Patch Size on Fine-Tuning Vision Transformers in Two-Dimensional and Three-Dimensional Medical Image Classification

Massoud Dehghan; Ramona Woitek; Amirreza Mahbod

cs.CVFeb 20, 2026

Effect of Patch Size on Fine-Tuning Vision Transformers in Two-Dimensional and Three-Dimensional Medical Image Classification

Authors: Massoud Dehghan, Ramona Woitek, Amirreza Mahbod

Organizations: Research Center for Medical Image Analysis and Artificial Intelligence, Department of Medicine, Faculty of Medicine and Dentistry, Danube Private University, Krems an der Donau, 3500, Austria

Abstract

Vision Transformers (ViTs) and their variants have become state-of-the-art in many computer vision tasks and are widely used as backbones in large-scale vision and vision-language foundation models. While substantial research has focused on architectural improvements, the impact of patch size, a crucial initial design choice in ViTs, remains underexplored, particularly in medical domains where both two-dimensional (2D) and three-dimensional (3D) imaging modalities exist. In this study, using 12 medical imaging datasets from various imaging modalities (including seven 2D and five 3D datasets), we conduct a thorough evaluation of how different patch sizes affect ViT classification performance. Using a single graphical processing unit (GPU) and a range of patch sizes (1, 2, 4, 7, 14, 28), we fine-tune ViT models and observe consistent improvements in classification performance with smaller patch sizes (1, 2, and 4), which achieve the best results across nearly all datasets. More specifically, our results indicate improvements in balanced accuracy of up to 12.78% for 2D datasets (patch size 2 vs. 28) and up to 23.78% for 3D datasets (patch size 1 vs. 14), at the cost of increased computational expense. Moreover, by applying a straightforward ensemble strategy that fuses the predictions of the models trained with patch sizes 1, 2, and 4, we demonstrate a further boost in performance in most cases, especially for the 2D datasets. Our implementation is publicly available on GitHub: https://github.com/HealMaDe/MedViT

Figures & tables

Dataset	Type	# train/val/test	classes	Modality
Breast	2D	546/78/156	2	Ultrasound
Retina	2D	1080/120/400	5	Fundus imaging
Blood	2D	11959/1712/3421	8	Microscopy
Derma	2D	7007/1003/2005	7	Dermatoscopy
OCT	2D	97477/10832/1000	4	Retinal OCT
OrganS	2D	13932/2452/8827	11	Abdominal CT

Table 1: Summary of the selected datasets from MedMNIST V2 collection [ 37 ] .

Panel of Figure 1 — Figure 1: Overview of the 12 selected datasets from the MedMNIST V2 collection used in this study. Each sub-figure shows some representative example images along with the dataset name, imaging modality, and classification task. For the 3D datasets, the middle slices from random samples are shown.

Figure 2: Example of splitting the image into patches with different sizes for 2D models.

Figure 3: Example of splitting the image into patches with different sizes for 3D models.

Patch size	Acc. (%)	Bal. Acc. (%)	AUC (%)	GFLOPs
1	84.40 $\pm$ 0.98	77.05 $\pm$ 1.55	89.60 $\pm$ 2.02	16.71
2	85.04 $\pm$ 1.48	77.24 $\pm$ 0.38	90.07 $\pm$ 2.68	4.19
4	84.62 $\pm$ 1.93	76.94 $\pm$ 4.69	90.54 $\pm$ 0.90	1.06
7	85.26 $\pm$ 0.64	75.63 $\pm$ 0.44	87.29 $\pm$ 2.73	0.36
14	83.55 $\pm$ 1.34	76.21 $\pm$ 3.07	88.25 $\pm$ 1.75	0.11
28	83.76 $\pm$ 1.96	75.36 $\pm$ 2.59	84.11 $\pm$ 2.18	0.04

Table 2: Effect of patch size on classification performance for the BreastMNIST dataset. Acc.: Accuracy, Bal. Acc.: Balanced Accuracy, AUC: Area under the ROC curve, GFLOPs: Giga Floating-Point Operations.

Patch size	Acc. (%)	Bal. Acc. (%)	AUC (%)	GFLOPs
1	54.67 $\pm$ 1.63	36.71 $\pm$ 2.11	74.44 $\pm$ 0.57	16.71
2	51.08 $\pm$ 1.01	32.02 $\pm$ 2.67	71.78 $\pm$ 0.90	4.19
4	52.17 $\pm$ 0.76	34.51 $\pm$ 0.80	73.77 $\pm$ 1.94	1.06
7	53.25 $\pm$ 1.75	36.01 $\pm$ 0.51	73.68 $\pm$ 1.33	0.36
14	53.33 $\pm$ 2.32	35.58 $\pm$ 2.83	72.63 $\pm$ 1.72	0.11
28	54.08 $\pm$ 0.72	36.99 $\pm$ 0.50	72.73 $\pm$ 0.79	0.04

Table 3: Effect of patch size on classification performance for the RetinaMNIST dataset. Acc.: Accuracy, Bal. Acc.: Balanced Accuracy, AUC: Area under the ROC curve, GFLOPs: Giga Floating-Point Operations.

Patch size	Acc. (%)	Bal. Acc. (%)	AUC (%)	GFLOPs
1	96.50 $\pm$ 0.13	96.03 $\pm$ 0.12	99.82 $\pm$ 0.03	16.71
2	96.99 $\pm$ 0.28	96.56 $\pm$ 0.41	99.87 $\pm$ 0.02	4.19
4	96.62 $\pm$ 0.92	96.09 $\pm$ 1.31	99.85 $\pm$ 0.03	1.06
7	96.46 $\pm$ 0.26	95.95 $\pm$ 0.44	99.82 $\pm$ 0.02	0.36
14	94.96 $\pm$ 0.14	94.33 $\pm$ 0.41	99.67 $\pm$ 0.02	0.11
28	93.59 $\pm$ 0.46	92.36 $\pm$ 0.60	99.50 $\pm$ 0.05	0.04

Table 4: Effect of patch size on classification performance for the BloodMNIST dataset. Acc.: Accuracy, Bal. Acc.: Balanced Accuracy, AUC: Area under the ROC curve, GFLOPs: Giga Floating-Point Operations.

Patch size	Acc. (%)	Bal. Acc. (%)	AUC (%)	GFLOPs
1	78.70 $\pm$ 0.64	56.16 $\pm$ 3.02	94.18 $\pm$ 0.20	16.71
2	78.12 $\pm$ 0.33	54.86 $\pm$ 3.75	94.16 $\pm$ 0.36	4.19
4	78.65 $\pm$ 0.99	56.64 $\pm$ 2.30	94.37 $\pm$ 0.67	1.06
7	77.18 $\pm$ 0.94	54.38 $\pm$ 1.90	93.95 $\pm$ 0.38	0.36
14	75.86 $\pm$ 0.09	48.76 $\pm$ 3.03	92.16 $\pm$ 0.41	0.11
28	73.76 $\pm$ 0.56	42.23 $\pm$ 6.01	90.63 $\pm$ 0.51	0.04

Table 5: Effect of patch size on classification performance for the DermaMNIST dataset. Acc.: Accuracy, Bal. Acc.: Balanced Accuracy, AUC: Area under the ROC curve, GFLOPs: Giga Floating-Point Operations.

Patch size	Acc. (%)	Bal. Acc. (%)	AUC (%)	GFLOPs
1	76.60 $\pm$ 2.10	76.60 $\pm$ 2.10	95.66 $\pm$ 0.28	16.71
2	79.10 $\pm$ 0.85	79.10 $\pm$ 0.85	96.68 $\pm$ 0.39	4.19
4	78.63 $\pm$ 1.26	78.63 $\pm$ 1.26	96.25 $\pm$ 0.23	1.06
7	78.03 $\pm$ 0.15	78.03 $\pm$ 0.15	95.62 $\pm$ 0.05	0.36
14	71.70 $\pm$ 0.66	71.70 $\pm$ 0.66	93.02 $\pm$ 0.41	0.11
28	67.37 $\pm$ 0.06	67.37 $\pm$ 0.06	91.24 $\pm$ 0.07	0.04

Table 6: Effect of patch size on classification performance for the OCTMNIST dataset. Acc.: Accuracy, Bal. Acc.: Balanced Accuracy, AUC: Area under the ROC curve, GFLOPs: Giga Floating-Point Operations.

Patch size	Acc. (%)	Bal. Acc. (%)	AUC (%)	GFLOPs
1	78.63 $\pm$ 0.23	74.96 $\pm$ 0.57	97.65 $\pm$ 0.07	16.71
2	80.54 $\pm$ 0.29	76.51 $\pm$ 0.10	98.08 $\pm$ 0.11	4.19
4	80.50 $\pm$ 0.37	76.35 $\pm$ 0.57	97.98 $\pm$ 0.11	1.06
7	79.06 $\pm$ 0.80	74.42 $\pm$ 0.95	97.71 $\pm$ 0.14	0.36
14	74.82 $\pm$ 1.25	70.71 $\pm$ 1.29	96.95 $\pm$ 0.33	0.11
28	68.42 $\pm$ 0.77	63.73 $\pm$ 1.25	95.29 $\pm$ 0.09	0.04

Table 7: Effect of patch size on classification performance for the OrganMNIST dataset. Acc.: Accuracy, Bal. Acc.: Balanced Accuracy, AUC: Area under the ROC curve, GFLOPs: Giga Floating-Point Operations.

Patch size	Acc. (%)	Bal. Acc. (%)	AUC (%)	GFLOPs
1	90.28 $\pm$ 0.67	87.27 $\pm$ 0.84	98.53 $\pm$ 0.55	16.71
2	91.93 $\pm$ 1.53	89.62 $\pm$ 2.12	98.70 $\pm$ 0.07	4.19
4	90.33 $\pm$ 0.34	87.45 $\pm$ 0.45	98.55 $\pm$ 0.08	1.06
7	89.47 $\pm$ 0.49	86.25 $\pm$ 0.86	97.91 $\pm$ 0.09	0.36
14	90.38 $\pm$ 1.00	87.66 $\pm$ 1.51	97.66 $\pm$ 0.21	0.11
28	89.10 $\pm$ 0.73	85.84 $\pm$ 1.01	97.49 $\pm$ 0.20	0.04

Table 8: Effect of patch size on classification performance for the PneumoniaMNIST dataset. Acc.: Accuracy, Bal. Acc.: Balanced Accuracy, AUC: Area under the ROC curve, GFLOPs: Giga Floating-Point Operations).

Patch size	Acc. (%)	Bal. Acc. (%)	AUC (%)	GFLOPs
1	80.20 $\pm$ 2.51	68.72 $\pm$ 3.61	82.56 $\pm$ 2.89	800.85
2	81.88 $\pm$ 0.72	65.76 $\pm$ 1.15	79.16 $\pm$ 1.47	117.83
4	78.64 $\pm$ 0.63	61.12 $\pm$ 2.28	73.92 $\pm$ 1.59	19.90
7	76.29 $\pm$ 0.88	57.74 $\pm$ 1.23	76.47 $\pm$ 2.28	3.10
14	77.29 $\pm$ 1.61	60.59 $\pm$ 3.42	78.58 $\pm$ 1.97	0.57
28	76.96 $\pm$ 1.04	58.68 $\pm$ 1.53	78.76 $\pm$ 1.28	0.40

Table 9: Effect of patch size on classification performance for the AdrenalMNIST3D dataset. Acc.: Accuracy, Bal. Acc.: Balanced Accuracy, AUC: Area under the ROC curve, GFLOPs: Giga Floating-Point Operations.

Patch size	Acc. (%)	Bal. Acc. (%)	AUC (%)	GFLOPs
1	56.11 $\pm$ 2.21	52.56 $\pm$ 5.22	73.79 $\pm$ 2.13	816.97
2	53.33 $\pm$ 1.80	49.51 $\pm$ 2.45	69.23 $\pm$ 2.33	120.2
4	55.97 $\pm$ 3.07	47.79 $\pm$ 2.99	68.78 $\pm$ 0.61	20.3
7	49.86 $\pm$ 1.61	44.51 $\pm$ 2.67	64.57 $\pm$ 1.44	3.16
14	46.11 $\pm$ 2.73	40.05 $\pm$ 1.54	61.42 $\pm$ 1.61	0.58
28	47.22 $\pm$ 1.53	42.34 $\pm$ 0.95	62.25 $\pm$ 0.42	0.41

Table 10: Effect of patch size on classification performance for the FractureMNIST3D dataset. Acc.: Accuracy, Bal. Acc.: Balanced Accuracy, AUC: Area under the ROC curve, GFLOPs: Giga Floating-Point Operations.

Patch size	Acc. (%)	Bal. Acc. (%)	AUC (%)	GFLOPs
1	85.38 $\pm$ 1.06	76.91 $\pm$ 2.58	89.41 $\pm$ 0.39	811.74
2	84.84 $\pm$ 0.70	78.50 $\pm$ 0.54	88.79 $\pm$ 1.61	119.43
4	84.84 $\pm$ 0.70	75.23 $\pm$ 0.42	85.75 $\pm$ 0.87	20.17
7	83.87 $\pm$ 0.95	72.69 $\pm$ 0.82	84.84 $\pm$ 0.35	3.14
14	83.87 $\pm$ 0.70	72.11 $\pm$ 1.58	82.06 $\pm$ 0.90	0.57
28	84.73 $\pm$ 0.40	72.85 $\pm$ 2.37	81.49 $\pm$ 1.32	0.40

Table 11: Effect of patch size on classification performance for the NoduleMNIST3D dataset. Acc.: Accuracy, Bal. Acc.: Balanced Accuracy, AUC: Area under the ROC curve, GFLOPs: Giga Floating-Point Operations.

Patch size	Acc. (%)	Bal. Acc. (%)	AUC (%)	GFLOPs
1	71.02 $\pm$ 0.70	56.93 $\pm$ 1.84	64.12 $\pm$ 1.27	816.97
2	70.74 $\pm$ 1.62	53.42 $\pm$ 2.32	58.76 $\pm$ 2.43	120.20
4	72.73 $\pm$ 0.40	50.80 $\pm$ 0.65	56.72 $\pm$ 4.38	20.30
7	73.58 $\pm$ 0.80	51.05 $\pm$ 1.49	57.60 $\pm$ 2.04	3.16
14	73.01 $\pm$ 0.00	50.00 $\pm$ 0.00	52.98 $\pm$ 0.51	0.58
28	69.70 $\pm$ 1.14	50.27 $\pm$ 1.05	57.22 $\pm$ 0.24	0.41

Table 12: Effect of patch size on classification performance for the SynapseMNIST3D dataset. Acc.: Accuracy, Bal. Acc.: Balanced Accuracy, AUC: Area under the ROC curve, GFLOPs: Giga Floating-Point Operations.

Patch size	Acc. (%)	Bal. Acc. (%)	AUC (%)	GFLOPs
1	92.76 $\pm$ 0.33	79.67 $\pm$ 1.41	94.28 $\pm$ 1.48	812.72
2	91.19 $\pm$ 0.65	72.70 $\pm$ 5.38	90.74 $\pm$ 1.05	119.57
4	90.49 $\pm$ 1.23	70.95 $\pm$ 5.23	87.49 $\pm$ 2.96	20.19
7	88.05 $\pm$ 1.58	57.05 $\pm$ 1.92	73.89 $\pm$ 1.55	3.14
14	87.78 $\pm$ 0.49	55.89 $\pm$ 3.09	74.91 $\pm$ 0.32	0.57
28	88.31 $\pm$ 0.99	57.88 $\pm$ 5.05	74.59 $\pm$ 0.82	0.40

Table 13: Effect of patch size on classification performance for the VesselMNIST3D dataset. Acc.: Accuracy, Bal. Acc.: Balanced Accuracy, AUC: Area under the ROC curve, GFLOPs: Giga Floating-Point Operations.

Figure 4: Comparison of attention maps for patch sizes of 2 and 28 for some example images. The figure shows original images with patch grids (left three columns) and corresponding attention heatmaps (right two columns). Models with smaller patch sizes (P2) consistently achieve correct predictions with more focused attention patterns, while larger patch sizes (P28) lead to incorrect predictions with less informative attention distributions.

Figure 5: Effect of patch size on vision transformer classification performance across 12 medical image datasets (seven 2D and five 3D datasets from the MedMNIST V2 collection)

Figure 6: Effect of patch size on vision transformer performance: average across different dataset groups

Explore similar work

VariViT: A Vision Transformer for Variable Image Sizes
Feb 16, 2026Aswathi Varma, Suprosanna Shit, Chinmay Prabhakar +5Vision Transformer Medical Image Representation Learning
Department of Neuroradiology, Technical University of Munich · Institute for Artificial Intelligence and Informatics in Medicine, Technical University of Munich · Department of Quantitative Biomedicine, University of Zurich +1
ZACH-ViT: Regime-Dependent Inductive Bias in Compact Vision Transformers for Medical Imaging
Feb 20, 2026Athanasios AngelakisVision Transformer Efficient ViTs
BioML Lab, RI CODE, UniBw, Munich, Germany · EDS, Amsterdam UMC, Amsterdam, Netherlands
Hierarchical Vision Transformer Enhanced by Graph Convolutional Network for Image Classification
Apr 18, 2026Haibin JiaoVision Transformer Hierarchical Representation Learning
Independent Researcher

Feb 16, 2026J/K move · Enter open · S save

VariViT: A Vision Transformer for Variable Image Sizes

Aswathi Varma, Suprosanna Shit, Chinmay Prabhakar, Daniel Scholz+4

Department of Neuroradiology, Technical University of Munich · Institute for Artificial Intelligence and Informatics in Medicine, Technical University of Munich · Department of Quantitative Biomedicine, University of Zurich+1

Vision Transformer Medical Image Representation Learning Efficient ViTs Medical Image Classification Brain Tumor Classification

Open paper AskPDF arXiv

Vision Transformers (ViTs) have emerged as the state-of-the-art architecture in representation learning, leveraging self-attention mechanisms to excel in various tasks. ViTs split images into fixed-size patches, constraining them to a predefined size and necessitating pre-processing steps like resizing, padding, or cropping. This poses challenges in medical imaging, particularly with irregularly shaped structures like tumors. A fixed bounding box crop size produces input images with highly variable foreground-to-background ratios. Resizing medical images can degrade information and introduce artefacts, impacting diagnosis. Hence, tailoring variable-sized crops to regions of interest can enhance feature representation capabilities. Moreover, large images are computationally expensive, and smaller sizes risk information loss, presenting a computation-accuracy tradeoff. We propose VariViT, an improved ViT model crafted to handle variable image sizes while maintaining a consistent patch size. VariViT employs a novel positional embedding resizing scheme for a variable number of patches. We also implement a new batching strategy within VariViT to reduce computational complexity, resulting in faster training and inference times. In our evaluations on two 3D brain MRI datasets, VariViT surpasses vanilla ViTs and ResNet in glioma genotype prediction and brain tumor classification. It achieves F1-scores of 75.5% and 76.3%, respectively, learning more discriminative features. Our proposed batching strategy reduces computation time by up to 30% compared to conventional architectures. These findings underscore the efficacy of VariViT in image representation learning. Our code can be found here: https://github.com/Aswathi-Varma/varivit

Abstract

Explore similar work

VariViT: A Vision Transformer for Variable Image Sizes

ZACH-ViT: Regime-Dependent Inductive Bias in Compact Vision Transformers for Medical Imaging

Hierarchical Vision Transformer Enhanced by Graph Convolutional Network for Image Classification