MedTokenBudget: Lesion-Preserving Token Routing for Dermoscopic Image Classification

Zhexiang Li

cs.CVSep 24, 2026

MedTokenBudget: Lesion-Preserving Token Routing for Dermoscopic Image Classification

Authors: Zhexiang Li

Organizations: University of California, Los Angeles

Abstract

Dermoscopy classifiers built on Vision Transformers process all image patches uniformly, although diagnostic evidence is concentrated in the lesion region. Existing token pruning methods reduce tokens using generic saliency or similarity signals, but rarely ask whether the retained subset still contains the lesion. This paper introduces MedTokenBudget, a supervised post-backbone token routing framework that learns to construct compact lesion-enriched representations when auxiliary lesion masks are available. Its Lesion-Aware Token Scoring (LATS) module fuses attention entropy, feature norm, and local feature contrast through a learned scorer, then routes the top- $K$ patches under a target budget. LATS is trained with budget curriculum learning, diversity regularization, attention distillation, and lesion-mask supervision. The trained router is evaluated with a lesion retention rate that directly measures how much ground-truth lesion evidence survives the token budget. On ISIC 2019, mask-supervised LATS consistently outperforms Random and ToMe at headline budgets while retaining substantially more lesion patches. Code is provided for reproducibility, and complete tabulated results are included in the supplementary material.

Figures & tables

Figure 1: MedTokenBudget overview. A frozen ViT produces patch embeddings; LATS scores patches with attention, norm, and local-contrast signals; the top-

K

router keeps a budgeted subset for MLP classification.

Method	$\beta{=}0.1$	$\beta{=}0.2$	$\beta{=}0.3$	$\beta{=}0.5$	$\beta{=}1.0$	[email protected]
NoPruning	77.5	77.5	77.5	77.5	77.5	100.0
Random	76.2	77.0	76.8	77.1	77.0	29.5
NormBased	69.7	71.8	73.1	74.5	75.7	21.1
ToMe	75.2	76.5	76.9	77.0	77.0	—
AttnEnt	70.9	72.9	73.9	75.2	76.8	34.0
LocalCtr	73.9	74.8	75.7	75.8	76.5	19.3

Table 1: Single-run classification accuracy (%) and lesion retention (%) at selected budgets on ISIC 2019. Bold : best per budget excluding NoPruning, which uses all tokens. [email protected] is a representative retention column at

\beta{=}0.3

on the 155-image mask-overlap subset; full retention curves appear in Figure 2 . ToMe retention is unavailable because merged tokens do not preserve one-to-one patch masks. Random uses one draw. Three-seed results appear in Table 2 .

$\beta$	Method	Acc. (%)	Macro F1 (%)	Bal. Acc. (%)	Ret. (%)
0.1	LATS	$77.77{\pm}0.27$	$67.08{\pm}0.25$	$65.89{\pm}0.49$	$28.9{\pm}0.1$
0.1	Random	$75.75{\pm}0.44$	$59.50{\pm}1.34$	$55.39{\pm}1.31$	$9.9{\pm}0.1$
0.1	ToMe	$75.94{\pm}0.61$	$60.96{\pm}1.56$	$56.90{\pm}1.57$	—
0.3	LATS	$80.38{\pm}0.18$	$69.17{\pm}1.28$	$66.40{\pm}0.91$	$67.4{\pm}0.2$
0.3	Random	$76.57{\pm}0.30$	$60.95{\pm}1.06$	$56.72{\pm}1.81$	$29.7{\pm}0.3$
0.3	ToMe	$76.93{\pm}0.03$	$61.92{\pm}0.23$	$57.81{\pm}0.48$	—

Table 2: Aligned three-seed ISIC robustness for the headline methods and budgets. Retention is computed on the 155-image mask-overlap validation subset for methods with one-to-one retained patches; ToMe merges tokens, so patch retention is not reported.

Figure 2: Lesion retention vs. token budget on ISIC 2019.

Figure 3: Qualitative routing examples. LATS assigns high scores to lesion regions and retains compact patch subsets concentrated over visible lesion evidence.

Method	Acc. (%)	F1 (%)	Ret. (%)
LATS	80.2	68.0	67.4
AttnEnt	73.9	57.3	34.0
NormBased	73.1	54.3	21.1
LocalContrast	75.7	57.1	19.3

Table 3: Single-run single-signal selector baseline comparison at

\beta=0.3

on ISIC 2019. Each baseline uses one LATS signal in isolation with an independently trained head; values come from the budget-sweep baseline runs.

Training	$\beta{=}0.1$	$\beta{=}0.2$	$\beta{=}0.3$	$\beta{=}0.5$	$\beta{=}1.0$
Curriculum	77.5	80.1	80.2	79.3	75.1
Fixed ( $\beta{=}0.5$ )	72.4	77.4	79.0	80.0	76.4

Table 4: Single-run curriculum vs. fixed-budget training on ISIC 2019.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Subset	Images	MEL	NV	BCC	AK	BKL	DF	VASC	SCC
Full validation	5067	910	2568	673	167	516	45	60	128
Mask-overlap validation	155	35	120	0	0	0	0	0	0
Mask-overlap training	617	129	488	0	0	0	0	0	0

Appendix

Table 6: ISIC class distribution for the full validation split and the ISIC 2018 mask-overlap subsets used for auxiliary training supervision and retention evaluation. Counts are derived from the experiment validation manifest and official ISIC 2019 labels. The mask-overlap subset is class-limited (MEL/NV), which is why the main paper treats retention as a focused evidence-preservation analysis rather than a full-class validation metric.

Statistic	Lesion patches	Prevalence
Minimum	6	2.3%
25th percentile	35	13.7%
Median	71	27.7%
75th percentile	104	40.6%
Maximum	248	96.9%
Mean $\pm$ std.	$81.1{\pm}58.3$	$31.6{\pm}22.8\%$

Appendix

Table 7: Lesion-patch prevalence after resizing the 155 validation masks to the

16\times 16

token grid.

Method	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1.0
NoPruning	77.5	77.5	77.5	77.5	77.5	77.5	77.5	77.5	77.5	77.5
Random	76.2	77.0	76.8	77.0	77.1	76.9	76.9	77.1	77.1	77.0
NormBased	69.7	71.8	73.1	74.0	74.5	74.9	75.4	75.3	75.5	75.7
ToMe	75.2	76.5	76.9	77.0	77.0	76.8	77.0	76.9	77.0	77.0
AttnEnt	70.9	72.9	73.9	74.5	75.2	76.1	76.5	76.7	76.6	76.8
LocalCtr	73.9	74.8	75.7	75.9	75.8	76.1	76.2	76.4	76.4	76.5

Appendix

Table 8: Complete budget sweep: accuracy (%) on ISIC 2019.

Method	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1.0
NoPruning	91.6	91.6	91.6	91.6	91.6	91.6	91.6	91.6	91.6	91.6
Random	90.2	91.6	90.9	90.8	91.4	91.2	91.4	91.5	91.4	91.4
NormBased	84.1	86.2	87.9	89.8	90.4	90.4	90.3	90.4	90.4	90.4
ToMe	89.2	92.0	91.8	91.9	91.6	91.6	91.5	91.2	91.2	91.4
AttnEnt	83.2	85.6	88.1	90.0	89.8	90.5	90.9	91.3	91.2	91.6
LocalCtr	87.3	89.4	89.9	90.4	90.6	90.4	91.1	90.8	91.0	90.8

Appendix

Table 9: Auxiliary accuracy-only sweep on Kvasir v2.

$\beta$	Method	Acc. (%)	Macro F1 (%)	Bal. Acc. (%)	Ret. (%)
0.1	LATS	$77.77{\pm}0.27$	$67.08{\pm}0.25$	$65.89{\pm}0.49$	$28.9{\pm}0.1$
0.1	Random	$75.75{\pm}0.44$	$59.50{\pm}1.34$	$55.39{\pm}1.31$	$9.9{\pm}0.1$
0.1	ToMe	$75.94{\pm}0.61$	$60.96{\pm}1.56$	$56.90{\pm}1.57$	—
0.3	LATS	$80.38{\pm}0.18$	$69.17{\pm}1.28$	$66.40{\pm}0.91$	$67.4{\pm}0.2$
0.3	Random	$76.57{\pm}0.30$	$60.95{\pm}1.06$	$56.72{\pm}1.81$	$29.7{\pm}0.3$
0.3	ToMe	$76.93{\pm}0.03$	$61.92{\pm}0.23$	$57.81{\pm}0.48$	—

Appendix

Table 10: Three-seed ISIC robustness at headline budgets.

$\beta$	Acc. (%)	Macro F1 (%)	Ret. (%)	Prec. (%)	Enrich.
0.1	$66.5{\pm}3.0$	$51.2{\pm}3.1$	$19.5{\pm}14.2$	$63.1{\pm}46.1$	$2.00{\pm}1.46$
0.3	$73.2{\pm}1.9$	$55.9{\pm}2.7$	$46.6{\pm}29.1$	$49.7{\pm}31.0$	$1.57{\pm}0.98$
0.5	$74.8{\pm}0.8$	$56.7{\pm}2.7$	$62.7{\pm}29.4$	$39.6{\pm}18.6$	$1.25{\pm}0.59$

Appendix

Table 11: Mask-free LATS control over three seeds on ISIC. The architecture, epochs, and budgets match LATS, but

\mathcal{L}_{\text{lesion}}

is disabled. Precision and enrichment are diagnostic retained-patch metrics computed only where masks are available.

$\beta$	Comparison	$\Delta$ Acc.	$\Delta$ F1	$\Delta$ Ret.
0.1	LATS–Random	$2.02{\pm}0.60$	$7.58{\pm}1.46$	$19.03{\pm}0.19$
0.3	LATS–Random	$3.80{\pm}0.35$	$8.22{\pm}1.46$	$37.63{\pm}0.24$
0.5	LATS–Random	$3.05{\pm}0.84$	$5.93{\pm}2.29$	$35.82{\pm}0.23$
0.1	LATS–ToMe	$1.84{\pm}0.38$	$6.12{\pm}1.44$	—
0.3	LATS–ToMe	$3.45{\pm}0.16$	$7.25{\pm}1.25$	—
0.5	LATS–ToMe	$2.66{\pm}0.37$	$4.79{\pm}1.39$	—

Appendix

Table 12: Paired seed-wise mean differences in percentage points over seeds 42, 43, and 44. Retention differences are reported only for methods with one-to-one retained patch masks.

Explore similar work

REDI: Corpus Aware Patch Ranking for DINOv3 Token Reduction
Jun 30, 2026Chanjong Im, Sebastian Diem, Thomas MandlEfficient ViTs Visual Token Pruning
University of Magdeburg · University of Hildesheim
Privileged Lesion-Context Relational Distillation for Mask-Free Skin Lesion Classification
Jul 21, 2026Abu Mukaddim Rahi, Md Mithun Hossain, Md Zulficar Hasan Joy +2Skin Lesion Classification Medical Image Classification
Department of IT in Artificial Intelligence, Victoria University, Footscray, VIC, Australia · BUBT Resarch Graduate School, Bangladesh University of Business and Technology, Dhaka, Bangladesh · Department of Artificial Intelligence, Daffodil Institute of Information Technology, Dhaka, Bangladesh +2
MDSkin-Net: Multi-Task Skin Lesion Analysis Driven by Pattern Analysis Priors and Spatial Alignment Regularization
Sep 25, 2026Yijian Li, Saad Bedros, Paul Bigliardi +3Skin Lesion Classification Skin Lesion Segmentation
University of Minnesota, Minneapolis, MN, USA

Jun 30, 2026J/K move · Enter open · S save

REDI: Corpus Aware Patch Ranking for DINOv3 Token Reduction

Chanjong Im, Sebastian Diem, Thomas Mandl

University of Magdeburg · University of Hildesheim

Efficient ViTs Visual Token Pruning Image Classification Visual Token Compression

Open paper AskPDF arXiv

Most token reduction methods for Vision Transformers seek favorable tradeoffs between accuracy and efficiency by pruning, merging, or pooling patch tokens. REDI (Relevance for DINOv3 Token Reduction) studies this question through a controlled supervised reference: how should a fixed token budget be allocated across patches for image classification? REDI quantizes final block DINOv3 patch representations into a visual vocabulary and derives class conditioned corpus scores using supervised TF-IDF over visual words. For each validation image, the ground truth class selects a row of the TF-IDF table, and four transformed views produce a TF-IDF map aligned to a reference center crop. A separate dense pass on the same crop provides an attention map. After independent min max normalization, their elementwise product defines the REDI score. A fixed keep, merge, and compress operator then uses score rank to assign patch roles and score magnitude to weight merging and compression. With precomputed REDI scores, a frozen DINOv3 ViT-B/16 backbone, and the same linear classifier used for dense evaluation, the operator reduces the sequence length from 201 to 107 tokens, a 46.8% sequence reduction. The REDI variant based on incoming attention mass achieves 84.706% Top-1 accuracy on ImageNet-1K, compared with 83.514% for the dense baseline, 82.634% for incoming attention mass alone, and 81.796% for supervised TF-IDF alone. The same corpus term also improves reduced classification for three alternative attention formulations relative to their attention only counterparts. Together, these controlled comparisons indicate that class specific corpus statistics and image specific attention provide complementary signals for patch ranking in this setting.

Abstract

Explore similar work

REDI: Corpus Aware Patch Ranking for DINOv3 Token Reduction

Privileged Lesion-Context Relational Distillation for Mask-Free Skin Lesion Classification

MDSkin-Net: Multi-Task Skin Lesion Analysis Driven by Pattern Analysis Priors and Spatial Alignment Regularization