Dynamic self-ensemble on multi-exit residual vision transformer for small-data image classification

Stephen Ekaputra Limantoro; Yan-Kai Wang; Ching-Chun Huang; Hong-Han Shuai; Wen-Huang Cheng; Jungwoo Huh; Sanghoon Lee

cs.CVOct 17, 2026

Dynamic self-ensemble on multi-exit residual vision transformer for small-data image classification

Authors: Stephen Ekaputra Limantoro, Yan-Kai Wang, Ching-Chun Huang, Hong-Han Shuai, Wen-Huang Cheng, Jungwoo Huh, Sanghoon Lee

Organizations: aDepartment of Electrical Engineering and Computer Science, National Yang Ming Chiao Tung University, Hsinchu, 30010, Taiwan · bDepartment of Computer Science, National Yang Ming Chiao Tung University, Hsinchu, 30010, Taiwan · cDepartment of Computer Science and Information Engineering, National Taiwan University, Taipei, 10617, Taiwan · dCollege of Engineering and Computer Science, VinUniversity, Hanoi, Vietnam ·

Abstract

Vision Transformer (ViT) has garnered increased attention for image classification in recent years. However, ViT is notorious for its data-hungry problem as it achieves high-performance results when only trained on large-size datasets. In this paper, we study how ViT specifically works on small-scale datasets with limited data where the availability of image data is often constrained in practice, especially in fields such as medical imaging. To mitigate the issue, we designed a multi-stage residual ViT, dubbed ReSEViT, with the self-ensemble method to utilize multi-level information from intermediate features. Due to the complex decision boundaries for classifications, relying on a single classifier can lead to prediction errors. Hence, we propose a new learning method for ViT, named self-ensemble learning, which ensembles multi-stage classifiers in a single model. Our self-ensemble comprises a novel approach, Dynamic Learner Selection (DLS), to adaptively select the importance of different stages. ReSEViT incorporates residual token embeddings which is a ResNet-like block to model effective local structures alongside with transformer blocks in each stage. The integration of residual tokens and self-ensemble learning into a multi-stage ViT highlights the reduction in redundancy, prediction error and variance while maintaining an efficient model size for small-scale image datasets.

Figures & tables

Figure 1: Illustration of traditional ensemble and our self-ensemble learning for our ReSEViT. The traditional ensemble methods integrating multiple models ( e.g. (a) two CvTs, (b) ResNet and CvT) separately are computationally heavier and only suggest high-level representation producing high-similar semantic outputs. (c) The prediction of ReSEViT is more accurate as multi-level representation evades the risk of semantic redundancy where the transformer in each stage inherits stronger local inductive biases from the residual token. In addition, both training and testing time could be drastically reduced with the self-ensemble learning.

Method	Pos. Emb.	Token Emb.	Projections	Multi-Stage	Small-Scale	Efficient Training
ViT [ vit ]	yes	standard	linear	no	no	no
DeiT [ deit ]	yes	standard	linear	no	no	distillation
CvT [ cvt ]	no	convolution	convolution	yes	no	no
CCT [ cct ]	optional	convolution	linear	no	yes	no
NesT [ nest ]	yes	convolution	linear	yes	yes	no
TransXNet [ transxnet ]	yes	convolution	linear	yes	no	no

Table 1: Comparison Types of Vision Transformer

Method	Training Losses	Inference Phase	Stage Classifier Design	Purpose
BranchyNet [ branchynet ]	$\mathcal{L}=\sum^{N}_{n=1}(w_{n}\times\mathcal{L}_{CE}(Z_{n},y))$ (w/ fixed weight)	$Z$	Extra convolution layers	faster inference
Self-distill [ selfdistillation ]	$\mathcal{L}=\sum^{N}_{n=1}(\mathcal{L}_{CE}(Z_{n},y)+\mathcal{L}_{distill})$	$\sum^{N}_{n=1}Z_{n}$	Extra attention module & layers	knowledge distillation
Self-ensemble (Ours)	$\mathcal{L}=\sum^{N}_{n=1}(\beta_{n}\times\mathcal{L}_{CE}(Z_{n},y))$ (w/ dynamic weight)	$\sum^{N}_{n=1}\beta^{f}_{n}\times Z_{n}$	No extra layer	ensemble

Table 2: Comparison with early and multi-exit networks

Figure 2: (a) Overview of ReSEViT (the multi-exit ViT network architecture with self-ensemble learning for small-scale datasets). ReSEViT is composed of (b) residual token embeddings and (c) convolution transformer blocks. In the residual token embedding, the dashed line denotes a shortcut with an increased dimension (for the second and third-stage), while the solid line is a shortcut with the same dimension. (d) ReSEViT stage classifier consists of a Layer Norm and a FC layer while the general model stage classifier consists of a Pooling and a FC layer.

Figure 3: Feature visualization of ReSEViT-L model (3 stages) with (a) convolution token; (b) residual token.

Small Resolution (32 $\times$ 32)				Large Resolution (224 $\times$ 224)
Output	Layer Name	ReSEViT-T	ReSEViT-S	Output	ReSEViT-L
32×32	Conv. Stem	3×3, 64, $s$ =1	3×3, 64, $s$ =1	56×56	7×7, 64, $s$ =4
32×32	Res. Token 1	$\Big[$ 3×3, 64, $s$ =1 3×3, 64, $s$ =1 $\Big]\times 2$	$\Big[$ 3×3, 64, $s$ =1 3×3, 64, $s$ =1 $\Big]\times 2$	56×56	$\Big[$ 3×3, 64, $s$ =1 3×3, 64, $s$ =1 $\Big]\times 2$
32×32	Transformer Block 1	$\Big[$ 3×3, 64, $s_{k,v}$ =2 $T_{1}$ =32, $h_{1}$ =4 $\Big]\times 1$	$\Big[$ 3×3, 64, $s_{k,v}$ =1 $T_{1}$ =32, $h_{1}$ =8 $\Big]\times 2$	56×56	$\Big[$ 3×3, 64, $s_{k,v}$ =2 $T_{1}$ =56, $h_{1}$ =1 $\Big]\times 2$
16×16	Res. Token 2	$\Big[$ 3×3, 128, $s$ =2 3×3, 128, $s$ =1 $\Big]\times 2$	$\Big[$ 3×3, 128, $s$ =2 3×3, 128, $s$ =1 $\Big]\times 2$	28×28	$\Big[$ 3×3, 192, $s$ =2 3×3, 192, $s$ =1 $\Big]\times 2$
16×16	Transformer Block 2	$\Big[$ 3×3, 128, $s_{k,v}$ =2 $T_{2}$ =16, $h_{2}$ =4 $\Big]\times 1$	$\Big[$ 3×3, 128, $s_{k,v}$ =1 $T_{2}$ =16, $h_{2}$ =8 $\Big]\times 2$	28×28	$\Big[$ 3×3, 192, $s_{k,v}$ =2 $T_{2}$ =28, $h_{2}$ =3 $\Big]\times 2$

Table 3: Detailed Settings of ReSEViT Variants.

Dataset	Train	Test	Dimension	# Classes
General Datasets
CIFAR10 [ cifar ]	50,000	10,000	32 $\times$ 32	10
CIFAR100 [ cifar ]	50,000	10,000	32 $\times$ 32	100
Tiny-ImageNet [ tinyimagenet ]	100,000	10,000	64 $\times$ 64	200
ImageNet100 [ cmc ]	130,000	5,000	224 $\times$ 224	100
Medical Datasets

Table 4: Statistics of Small-Scale Datasets.

Method	CIFAR10	CIFAR100	Tiny-ImageNet	Avg. Acc $\uparrow$	Params. $\downarrow$	FLOPs $\downarrow$	GPU $\downarrow$	Throughput $\uparrow$
Convolution Networks
ResNet18 † [ resnet ]	94.37%	75.41%	52.47%	74.08%	11.22M	0.56G	2045M	10351
ResNet34 † [ resnet ]	95.26%	77.08%	54.87%	75.74%	21.33M	1.16G	2081M	7360
DenseNet 100-12 [ densenet ]	95.90%	79.80%	54.81%	76.84%	7.00M	1.85G	5347M	1251
WRN-40-4 [ wideresnet ]	95.47%	78.90%	56.86%	77.08%	8.98M	1.42G	2035M	5812
ConvNeXt-T † [ convnext ]	94.55%	74.02%	51.15%	73.24%	27.94M	1.45G	4446M	3455

Table 5: Test accuracy comparison on small-scale image datasets with small resolution (32

\times

32). † represents a modified stem for small resolution datasets, while ⋆ represents the original architecture without modification. We report accuracy, parameter size, FLOPs, GPU training memory usage, and inference throughput (images/s).

Method	Top-1 $\uparrow$	Params. $\downarrow$	FLOPs $\downarrow$	Throughput $\uparrow$
ViT-B (ICLR’21) [ vit ]	71.38%	85.85M	16.85G	605
T2T-ViT-14 (ICCV’21) [ t2t ]	78.48%	21.12M	4.33G	1170
Swin-T (ICCV’21) [ swin ]	80.66%	27.60M	4.35G	930
CvT-13 (ICCV’21) [ cvt ]	81.57%	19.69M	4.05G	1034
EfficientFormer-L3 (NeurIPS’22) [ efficientformer ]	79.21%	30.48M	3.95G	916
PoolFormer-S36 (CVPR’22) [ poolformer ]	81.25%	30.40M	4.99G	680

Table 6: Test accuracy comparison on ImageNet100 (224

\times

224). We report accuracy, parameter size, FLOPs, and inference throughput (images/s). ‡ applies pre-trained RegNetY-16GF from ImageNet-1K hard distillation [ efficientformer ] with top-1 88.46

\%

accuracy on ImageNet100.

Datasets	Metrics	CvT-13 †	CCT-7/3 $\times$ 1	NesT-T	ReSEViT-T
PathMNIST	AUC	0.9951	0.9953	0.9956	0.9958
PathMNIST	Top-1	0.9560	0.9554	0.9620	0.9692
ChestMNIST	AUC	0.5742	0.4764	0.5679	0.5849
ChestMNIST	Top-1	0.9496	0.9493	0.9495	0.9496
DermaMNIST	AUC	0.7356	0.7619	0.6591	0.7645
DermaMNIST	Top-1	0.7292	0.7360	0.6686	0.7399

Table 7: Comparative performance of ViT variants across MedMNIST sub-datasets. AUC and Top-1 accuracy are reported.

Metrics	ViT-B	CvT-13	CCT-14/7 $\times$ 2	NesT-T	ReSEViT-L
Metrics	ViT-B	CvT-13	CCT-14/7 $\times$ 2	NesT-T	SL	ML	ML+DLS
AUC	0.8410	0.8766	0.7728	0.8798	0.8972	0.9113	0.9176
Top-1	0.5837	0.6407	0.6733	0.6483	0.6707	0.6895	0.7066

Table 8: Comparative performance of ViT variants on ISIC 2018 (224

\times

224). AUC and Top-1 Accuracy are reported.

Figure 4: Comparison of ViTs against data size on ImageNet100.

Ensemble Method	$M$	Acc $\uparrow$	Params. $\downarrow$	Train $\downarrow$	Test $\downarrow$
Baseline	1	78.65%	4.11M	1.00 $\times$	1.00 $\times$
Model Avg.	2	79.89%	13.74M ⧫	1.05 $\times$	1.02 $\times$
Bagging [ bagging ]	2	78.76%	8.22M ⧫	2.41 $\times$	1.05 $\times$
SSE [ snapshot ]	4	79.41%	4.11M	1.00 $\times$	1.17 $\times$
FGE [ fastgeometric ]	12	79.51%	4.11M	1.00 $\times$	2.86 $\times$
FiLM [ filmensemble ]	2	80.07%	4.12M	2.18 $\times$	1.6 $\times$

Table 9: Comparison with Other Ensemble Methods in ReSEViT-S on CIFAR100.

M

is the number of ensemble models. ⧫ refers to a parallel ensemble type.

Figure 5: Gradient norm per training epochs with ReSEViT-T model on CIFAR100. The column headers represent the token embedding variants, while the row headers represent the learning methods. SL, ML, and ML+DLS represent Single Loss, Multistage Ensemble Loss, and Multistage Ensemble Loss with Dynamic Learner Selection, respectively.

Figure 6: Prediction variance of the proposed ReSEViT. SL, ML, and ML+DLS represent Single Loss, Multistage Ensemble Loss, and Multistage Ensemble Loss with Dynamic Learner Selection, respectively.

Method	$\mathcal{L}_{WSE}$	CIFAR100	ImageNet100
Conv. Token (Baseline)	✗	77.28%	82.60%
	✓	78.62% (+1.34)	83.23% (+0.63)
Residual Token	✗	78.65%	83.31%
	✓	81.05% (+2.40)	85.23% (+1.92)

Table 10: Effects of Residual Token Embeddings.

Method	Stage 1	Stage 2	Stage 3	Ensemble
CIFAR100
SL	12.22%	34.17%	78.65%	76.62%
ML	65.96%	76.38%	78.22%	80.33%
ML+DLS	65.01%	76.70%	78.59%	81.05%
ImageNet100
SL	4.31%	13.39%	83.31%	82.52%

Table 11: Top-1 Accuracy of Self-Ensemble Method.

Method	Training	Inference	CIFAR100	ImageNet100
ML	✗	✗	80.33%	84.12%
ML+BT	✓	✗	80.75%	84.85%
ML+BI	✗	✓	80.71%	84.44%
ML+DLS	✓	✓	81.05%	85.23%

Table 12: Effects of Dynamic Weights (Beta).

Num. of cls.	Stage 1	Stage 2	Stage 3	CIFAR100	ImageNet100
	✓	✗	✗	68.49%	65.70%
1	✗	✓	✗	76.41%	82.67%
	✗	✗	✓	78.65%	83.31%
	✓	✓	✗	77.60%	82.63%
2	✓	✗	✓	79.66%	83.82%
	✗	✓	✓	80.80%	84.81%

Table 13: Number of Stage Classifiers.

Figure 7: ReSEViT-S class prediction distribution in 2-D on CIFAR10 test samples using PCA visualization. Top: classifiers class prediction distinguished by color (10 classes). Bottom: classifiers Top-1 prediction (correct predictions in blue color and incorrect predictions in red color). Best viewed in color.

Figure 8: Visualization of (a) CvT-13 (baseline) and (b) ReSEViT-L GradCAM results. Both models are trained on ImageNet100. We present visualization for each stage. Images contain single and multiple objects with various scales. The class names are Red Fox, American Coot, and Kuvasz. Best viewed in color.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Output	Layer Name	ReSEViT-L	Layer Name	CvT-13
56×56	Conv. Stem	7×7, 64, $s$ =4	Conv. Token 1	7×7, 64, $s$ =4
56×56	Res. Token 1	$\Big[$ 3×3, 64, $s$ =1 3×3, 64, $s$ =1 $\Big]\times 2$	Conv. Token 1	7×7, 64, $s$ =4
56×56	Transformer 1	$\Big[$ 3×3, 64, $s_{k,v}$ =2 $T_{1}$ =56, $h_{1}$ =1 $\Big]\times 2$	Transformer 1	$\Big[$ 3×3, 64, $s_{k,v}$ =2 $T_{1}$ =56, $h_{1}$ =1 $\Big]\times 1$
28×28	Res. Token 2	$\Big[$ 3×3, 192, $s$ =2 3×3, 192, $s$ =1 $\Big]\times 2$	Conv. Token 2	3×3, 192, $s$ =2
28×28	Transformer 2	$\Big[$ 3×3, 192, $s_{k,v}$ =2 $T_{2}$ =28, $h_{2}$ =3 $\Big]\times 2$	Transformer 2	$\Big[$ 3×3, 192, $s_{k,v}$ =2 $T_{2}$ =28, $h_{2}$ =3 $\Big]\times 2$
14×14	Res. Token 3	$\Big[$ 3×3, 384, $s$ =2 3×3, 384, $s$ =1 $\Big]\times 2$	Conv. Token 3	3×3, 384, $s$ =2

Appendix

Table 15: Detailed network specification for ReSEViT-T, CvT-13, and ResNet18 on a large-resolution dataset (224x224).

ReSEViT-S	[CLS]	GAP	SeqPool
CIFAR100 Top-1	78.65 $\%$ / 81.05%	77.99 $\%$ / 80.47 $\%$	78.83% / 80.63 $\%$

Appendix

Table 16: Necessity of CLS Token. ’/’ Symbol Separates the Method without and with Self-Ensemble.

Figure 9: Class Imbalance Distribution across various imbalance ratio ( $r$ ).

Method	CIFAR-100
	( $r=$ 0.01)		( $r=$ 0.1)		( $r=$ 1.0)
	Top-1 $\uparrow$	ECE $\downarrow$	Top-1 $\uparrow$	ECE $\downarrow$	Top-1 $\uparrow$	ECE $\downarrow$
ResNet18 † [ resnet ]	36.82 $\%$	0.145	56.71 $\%$	0.144	75.41 $\%$	0.158
ResNet34 † [ resnet ]	38.67 $\%$	0.168	58.36 $\%$	0.141	77.08 $\%$	0.126
ConvNeXt-T † [ convnext ]	34.45 $\%$	0.221	54.96 $\%$	0.199	74.02 $\%$	0.113
CvT-13 † [ cvt ]	38.45 $\%$	0.221	56.77 $\%$	0.151	73.56 $\%$	0.115

Appendix

Table 17: Comparison with other models under class imbalance. † refers to a modified stem. We report accuracy and ECE.

Roadmap	CvT-13	$\rightarrow$	patchify ”stem”	$\rightarrow$	stage ratio	$\rightarrow$	res. token (ResNet)	$\rightarrow$	ReSEViT-T
Note	(large resolution setting)		(small resolution setting)		(base form)		(wide conv. ⋆ / residual block)		(final form)
Accuracy ( $\uparrow$ )	58.48%		73.56%		67.69%		75.86% ⋆ / 77.65%		79.49%
Throughput ( $\uparrow$ )	2781		2560		7130		4254 ⋆ / 4841		4832
Params. ( $\downarrow$ )	19.66M		19.65M		1.57M		5.85M ⋆ / 3.86M		3.88M
FLOPs ( $\downarrow$ )	0.10G		1.32G		0.22G		0.84G ⋆ / 0.56G		0.56G
Roadmap	ResNet18	$\rightarrow$	patchify ”stem”	$\rightarrow$	w/o 4 th stage	$\rightarrow$	transformer	$\rightarrow$	ReSEViT-T

Appendix

Table 18: A roadmap to ReSEViT on CIFAR100. Top-1 accuracy, parameter size, floating point operations per second (FLOPs), and inference throughput (images/s) are reported. ⋆ represents the use of convolution tokens with wider channel sizes for comparison.

Figure 10: Priority scores of ReSEViT-S and ResNet18 with different numbers of epochs on CIFAR100 and MNIST datasets.

Model	$S$	Top-1 (%)	Params. (M) $\downarrow$	FLOPs. (G) $\downarrow$
SqueezeNet [ squeezenet ]	4	58.08	0.78	0.05
SE-SqueezeNet	4	66.48 (+8.40)	0.90 (+0.12)	0.07 (+0.02)
ResNet18 † [ resnet ]	4	75.41	11.22	0.56
SE-ResNet18 †	4	77.05 (+1.64)	11.27 (+0.05)	0.56
ResNet34 † [ resnet ]	4	77.08	21.33	1.16
SE-ResNet34 †	4	78.43 (+1.35)	21.37 (+0.04)	1.16

Appendix

Table 19: Our Self-Ensemble Method on Various Deep Learning Models. ‘SE-’ Variants Utilize Self-Ensemble. † refers to the modified stem.

S

denotes the number of stages.

Method	Training set
	10 $\%$		100 $\%$
	Top-1	Top-5	Top-1	Top-5
ViT-B	33.68 $\%$	61.04 $\%$	83.74 $\%$	95.42 $\%$
CvT-13	56.66 $\%$	81.28 $\%$	89.70 $\%$	97.72 $\%$
ReSEViT-L	64.20 $\%$	86.46 $\%$	90.02 $\%$	98.18 $\%$

Appendix

Table 20: Performance of ViT variants on ImageNet100 with timm library.

Figure 11: Comparison of ViTs against image resolution on ImageNet100.

Method	56 $\times$ 56			112 $\times$ 112			224 $\times$ 224
Method	Param.	FLOPs	FPS	Param.	FLOPs	FPS	Param.	FLOPs	FPS
ViT-B [ vit ]	85.29M	16.74G	595	85.41M	16.76G	592	85.85M	16.85G	605
CvT-13 [ cvt ]	19.65M	4.02G	990	19.65M	4.02G	988	19.69M	4.05G	1034
ReSeViT-L	12.42M	4.06G	866	12.42M	4.06G	870	12.43M	4.08G	890

Appendix

Table 21: Comparison between image resolution and model computation across various stem designs on ImageNet100.

Dataset	CIFAR (32 $\times$ 32), Tiny-ImageNet (64 $\times$ 64),	ImageNet100	ISIC 2018
	MedMNIST (28 $\times$ 28)
optimizer	AdamW	AdamW	AdamW
init lr	5e-4	5e-4	5e-4
batch size	64	64	64
weight decay	3e-2	3e-2	3e-2
momentum	(0.9, 0.999)	(0.9, 0.999)	(0.9, 0.999)

Appendix

Table 22: Training config.

Figure 12: Details of various multi-exit model architectures. (a) SE-SqueezeNet; (b) Applies to SE-ResNets and SE-PreAct-ResNet1001; (c) SE-CvT-13 † ; (d) SE-ConvNeXt-T † ; (e) SE-PoolFormer-S36 † ; (f) SE-NesT-T; (g) SE-Conv2Former-N † .

Vision Transformer

Vision Transformers (ViTs) use self-attention over image patches to learn visual representations for recognition, segmentation, and other tasks.

See all 370 papers

Self-supervised pretraining for an iterative image size agnostic vision transformer
Apr 22, 2026Nedyalko Prisadnikov, Danda Pani Paudel, Yuqian Fu +1Self-Supervised Pre-Training Vision Transformer
INSAIT, Sofia University "St. Kliment Ohridski", Bulgaria
VariViT: A Vision Transformer for Variable Image Sizes
Feb 16, 2026Aswathi Varma, Suprosanna Shit, Chinmay Prabhakar +5Vision Transformer Medical Image Representation Learning
Department of Neuroradiology, Technical University of Munich · Institute for Artificial Intelligence and Informatics in Medicine, Technical University of Munich · Department of Quantitative Biomedicine, University of Zurich +1
ConvRoute-ViT: Convolutional Token Routing for Efficient Vision Transformers
Oct 28, 2026Manish Das, Shiv Ram Dubey, Satish Kumar SinghVision Transformer Efficient ViTs
IIIT Allahabad, Prayagraj, India

Apr 22, 2026J/K move · Enter open · S save

Self-supervised pretraining for an iterative image size agnostic vision transformer

Nedyalko Prisadnikov, Danda Pani Paudel, Yuqian Fu, Luc Van Gool

INSAIT, Sofia University "St. Kliment Ohridski", Bulgaria

Self-Supervised Pre-Training Vision Transformer Efficient ViTs Self-Supervised Visual Representation Learning

Open paper AskPDF arXiv

Vision Transformers (ViTs) dominate self-supervised learning (SSL). While they have proven highly effective for large-scale pretraining, they are computationally inefficient and scale poorly with image size. Consequently, foundational models like DINO are constrained to low-resolution processing. A recent foveal-inspired transformer achieves resolution agnosticism by iteratively processing a fixed-size context of multi-zoom patches. This model demonstrated promising results via supervised learning, utilizing a sequential, recurrent-like process without backpropagation through time. To unlock its potential as a foundational backbone, we introduce a novel sequential-to-global SSL framework based on DINO's self-distillation objective. Supported by an efficient integral-image patch extraction method, our approach enables large-scale pretraining for image-size agnostic vision encoders. We achieve competitive performance on ImageNet-1K and downstream classification tasks, maintaining a constant computational budget regardless of input resolution.

Abstract

Vision Transformer

Explore similar work

Self-supervised pretraining for an iterative image size agnostic vision transformer

VariViT: A Vision Transformer for Variable Image Sizes

ConvRoute-ViT: Convolutional Token Routing for Efficient Vision Transformers