MedVTok: A General-Purpose Medical Visual Tokenizer

Chenglong Ma; Yuanfeng Ji; Junzhi Ning; Jiyao Liu; Yujie Wei; Shaohao Rui; Chaoyang Zhang; Wenjie Li

eess.IVOct 26, 2026

MedVTok: A General-Purpose Medical Visual Tokenizer

Authors: Chenglong Ma, Yuanfeng Ji, Junzhi Ning, Jiyao Liu, Yujie Wei, Shaohao Rui, Chaoyang Zhang, Wenjie Li,

Organizations: Fudan University, China · Shanghai Innovation Institute, China · Stanford University School of Medicine, United States · Ant Healthcare, Ant Group, China ·

Abstract

Multimodal medical AI requires a common visual interface for encoding heterogeneous imaging evidence for patient condition modeling. However, existing medical visual tokenizers are often tied to a specific dimension and imaging modality, forcing multimodal systems to rely on fragmented and poorly reusable representations. We introduce MedVTok, a general-purpose visual tokenizer that maps 2D images and 3D volumes from diverse imaging modalities into a unified token space. Building such a tokenizer poses two key challenges: preserving anatomical structure across dimensions and capturing clinical semantics across modalities. To this end, MedVTok introduces (i) slice differential regularization, which explicitly models inter-slice anatomical coherence missing in previous per-pixel/per-voxel supervision, and (ii) multi-expert representation alignment, which integrates knowledge from multiple medical expert encoders while retaining modality-specific complementary knowledge. Trained on 28M images and 70K volumes across 9 imaging modalities, MedVTok supports classification, retrieval, segmentation, synthesis, visual question answering, and medical report generation through a single tokenizer. Across 24 benchmarks, MedVTok achieves state-of-the-art performance, demonstrating a scalable token interface that unifies medical image dimension, modality, and task usage.

Figures & tables

Figure 1: MedVTok provides a unified token interface for multimodal modeling with medical imaging data. Trained on 28M images and 70K volumes, MedVTok achieves state-of-the-art performance across a broad range of downstream tasks covering 24 2D/3D benchmarks.

Figure 2: Overview of the MedVTok training framework. MedVTok learns both (1) structural fidelity across 2D and 3D inputs under the reconstruction loss $\mathcal{L}_{\text{R}}$ plus the slice differential regularization $\mathcal{L}_{\text{SliDR}}$ , and (2) clinical semantics across modalities driven by $\mathcal{L}_{\text{MERA}}=\mathcal{L}_{\text{shared}}{+}\mathcal{L}_{\text{expert}}$ .

Metrics	Method	CFP	CT	Derm.	Endo.	MRI	OCT	Path.	US	X-ray	Avg.	3D-MRI	3D-CT
PSNR ( $\uparrow$ )	SDVAE [ 29 ]	37.62	36.38	34.68	33.57	34.31	29.17	24.43	32.13	35.29	33.06	–	–
	TokenFlow [ 4 ]	30.57	25.45	29.54	25.75	24.51	24.79	20.17	22.82	26.48	25.56	–	–
	MingTok [ 32 ]	35.58	31.23	32.43	29.85	29.37	26.92	21.87	26.42	31.25	29.43	–	–
	UniTok [ 10 ]	34.02	31.32	31.89	30.34	30.82	28.00	24.40	29.48	31.57	30.21	–	–
	AToken [ 9 ]	41.08	41.60	39.00	39.69	41.64	32.94	28.57	39.41	40.02	38.22	31.46	35.68
	PUMIT [ 8 ]	32.03	32.86	27.05	28.73	32.88	27.06	19.43	23.26	26.69	27.78	29.96	31.87

Table 1: Token fidelity evaluation via autoencoding reconstruction [PSNR (dB)

\uparrow

, SSIM (%)

\uparrow

]. 3D columns are excluded from the 2D average. MedVAE uses separate models for 2D and 3D inputs. The best result is marked in bold and the second-best is underlined .

Figure 3: Qualitative reconstruction for token fidelity evaluation. For each example, we show per-tokenizer absolute error maps with PSNR/SSIM values reported per panel.

Metrics	Method	CFP	CT	Derm.	Endo.	MRI	OCT	Path.	US	X-ray	Avg.
Acc. ( $\uparrow$ )	SDVAE [ 29 ]	18.79	45.68	60.23	47.59	77.47	62.94	47.56	43.97	78.69	53.66
	TokenFlow [ 4 ]	35.73	46.10	61.95	42.83	79.73	68.53	45.38	46.10	86.91	57.03
	MingTok [ 32 ]	50.71	48.10	67.86	44.93	77.70	64.39	64.87	45.04	83.65	60.81
	UniTok [ 10 ]	66.95	74.79	70.63	44.82	80.53	72.88	69.23	46.28	91.03	68.57
	AToken [ 9 ]	64.12	65.82	72.31	46.63	88.13	66.46	66.54	46.19	91.29	67.50
	BiomedCLIP ∗ [ 15 ]	60.59	66.03	68.78	46.03	88.43	73.91	79.10	48.32	93.80	69.45

Table 2: Image classification and retrieval evaluation [Accuracy (%)

\uparrow

, Recall@1 (%)

\uparrow

]. *: semantic encoder.

Metrics	Method	CFP	CT	Derm.	Endo.	MRI	Path.	US	X-ray	Avg.
Dice ( $\uparrow$ )	SDVAE [ 29 ]	73.03	14.88	62.37	24.67	12.69	49.11	84.19	35.56	44.56
	TokenFlow [ 4 ]	70.39	19.05	72.62	40.53	30.49	52.34	91.33	49.52	53.28
	MingTok [ 32 ]	85.36	27.03	83.91	57.32	52.54	67.68	92.91	54.71	65.18
	UniTok [ 10 ]	60.93	33.13	86.25	67.25	54.90	72.51	93.35	57.60	65.74
	AToken [ 9 ]	97.09	37.33	87.45	69.88	63.57	74.30	94.90	59.69	73.03
	BiomedCLIP ∗ [ 15 ]	50.22	31.83	87.48	60.64	57.95	73.65	95.41	59.83	64.63

Table 3: Image segmentation evaluation [Dice (%)

\uparrow

]. *: semantic encoder.

Method	CFP	CT	Derm.	Endo.	MRI	Path.	X-ray	Avg.
PUMIT [ 8 ]	140.40	183.49	272.26	191.49	227.64	377.12	200.14	227.51
MedVAE [ 5 ]	118.13	154.42	202.09	170.29	145.25	325.87	88.55	172.08
MedITok [ 6 ]	90.65	142.46	176.94	58.38	122.96	99.66	77.24	109.76
MedVTok	59.85	146.39	142.87	82.39	116.78	94.65	69.17	101.73

Table 4: Image synthesis evaluation (FID

\downarrow

). A latent diffusion model is trained from scratch in each frozen tokenizer’s latent space with identical architecture and data.

Figure 4: Image synthesis examples. A latent diffusion model trained from scratch on MedVTok’s tokens produces samples with sharper anatomical structure and fewer artifacts than competing tokenizers across modalities, including fundus, X-ray, and dermoscopy.

Method	FundusMM	MM-Skin	Kvasir	PathVQA	SLAKE	GMAI-MM	Avg.
PUMIT [ 8 ]	29.84 $\pm$ 0.42	34.74 $\pm$ 0.38	69.50 $\pm$ 0.51	52.97 $\pm$ 0.63	36.44 $\pm$ 0.55	28.16 $\pm$ 0.47	41.94 $\pm$ 0.33
MedVAE [ 5 ]	28.39 $\pm$ 0.56	34.21 $\pm$ 0.44	66.64 $\pm$ 0.72	51.15 $\pm$ 0.58	31.57 $\pm$ 0.67	27.61 $\pm$ 0.53	39.93 $\pm$ 0.41
MedITok [ 6 ]	31.29 $\pm$ 0.35	36.18 $\pm$ 0.41	70.63 $\pm$ 0.29	58.46 $\pm$ 0.52	45.61 $\pm$ 0.38	30.15 $\pm$ 0.26	45.39 $\pm$ 0.28
MedVTok	33.39 $\pm$ 0.31	37.49 $\pm$ 0.28	72.98 $\pm$ 0.34	57.24 $\pm$ 0.45	45.94 $\pm$ 0.37	31.81 $\pm$ 0.29	46.48 $\pm$ 0.22

Table 5: Visual question answering evaluation [Accuracy (%)

\uparrow

]. Each frozen tokenizer is plugged into the LLaVA-Med [ 58 ] framework for evaluation.

Metrics	Method	Abdomen	Chest	Pelvis	Avg.
BLEU-2 ( $\uparrow$ )	PUMIT [ 8 ]	0.289 $\pm$ 0.008	0.193 $\pm$ 0.003	0.164 $\pm$ 0.007	0.215 $\pm$ 0.005
	MedVAE-3D [ 5 ]	0.249 $\pm$ 0.038	0.185 $\pm$ 0.020	0.152 $\pm$ 0.010	0.195 $\pm$ 0.013
	MedITok [ 6 ]	0.256 $\pm$ 0.013	0.219 $\pm$ 0.020	0.294 $\pm$ 0.017	0.257 $\pm$ 0.012
	MedVTok	0.272 $\pm$ 0.005	0.263 $\pm$ 0.045	0.288 $\pm$ 0.017	0.274 $\pm$ 0.009
RaTEScore ( $\uparrow$ )	PUMIT [ 8 ]	0.604 $\pm$ 0.005	0.511 $\pm$ 0.035	0.201 $\pm$ 0.015	0.439 $\pm$ 0.006
	MedVAE-3D [ 5 ]	0.613 $\pm$ 0.022	0.497 $\pm$ 0.053	0.187 $\pm$ 0.003	0.432 $\pm$ 0.011

Table 6: Medical report generation evaluation. Each frozen tokenizer is integrated into the LaMed [ 59 ] framework for evaluation, generating findings in abdomen, chest, and pelvis regions.

Figure 5: Ablation studies.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Index	Expert Encoder	Domains	$d_{j}$
0	DermLIP [ 21 ]	Dermoscopy	768
1	RetCLIP [ 22 ]	CFP	768
2	Conch [ 20 ]	Pathology	1024
3	PeskaVLP [ 24 ]	Endoscopy	2048
4–8	MedImageInsight [ 16 ]	CT, MR, US, X-ray, Other	2048
9	Spectre [ 25 ]	CT 3D	2160

Appendix

Table 7: Expert encoders used in MERA.

Dataset	Modality	#Samples	#Cls	Task(s)
2D Perception – Reconstruction / Classification / Retrieval
ChestCTScan [ 34 ]	CT	1,000	4	Rec, Cls, Ret
JSIEC [ 33 ]	CFP	1,000	39	Rec, Cls, Ret
HAM10K [ 35 ]	Dermoscopy	10,015	7	Rec, Cls, Ret
PolypDiag [ 37 ]	Endoscopy	2,202	2	Rec, Cls, Ret
BRISC [ 36 ]	MRI	5,023	4	Rec, Cls, Ret

Appendix

Table 8: Evaluation datasets grouped by downstream task. Rec: reconstruction, Cls: classification, Ret: retrieval, Seg: segmentation, Syn: image synthesis, VQA: visual question answering, MRG: medical report generation. #Cls denotes the number of target classes. Datasets appearing in multiple task groups are listed separately.

Figure 6: Additional reconstruction visualization.

Figure 7: 3D Reconstruction visualization.

Metrics	Method	CFP	CT	Derm.	Endo.	MRI	Path.	US	X-ray	Avg.
Dice ( $\uparrow$ )	SDVAE [ 29 ]	73.03	14.88	62.37	24.67	12.69	49.11	84.19	35.56	44.56
	TokenFlow ‡ [ 4 ]	70.39	19.05	72.62	40.53	30.49	52.34	91.33	49.52	53.28
	MingTok ‡ [ 32 ]	85.36	27.03	83.91	57.32	52.54	67.68	92.91	54.71	65.18
	UniTok ‡ [ 10 ]	60.93	33.13	86.25	67.25	54.90	72.51	93.35	57.60	65.74
	AToken ‡ [ 9 ]	97.09	37.33	87.45	69.88	63.57	74.30	94.90	59.69	73.03
	BiomedCLIP ∗ [ 15 ]	50.22	31.83	87.48	60.64	57.95	73.65	95.41	59.83	64.63

Appendix

Table 9: Additional results on image segmentation evaluation. Method type: ∗ semantic encoder; ‡ unified tokenizer.

Figure 8: Qualitative comparison on the image segmentation task. Segmentation powered by MedVTok achieves the best performance, while other tokenizers lead to false segmentation in the background. In each image, the mask is highlighted in cyan; the Dice score is shown in the lower right corner.

Model	CFP	CT	Derm.	Endo.	MRI	OCT	Path.	US	X-ray
RetCLIP	91.95 $\pm$ 1.05	–	–	–	–	–	–	–	–
DermLIP	–	–	79.72 $\pm$ 0.66	–	–	–	–	–	–
PeskaVLP	–	–	–	48.64 $\pm$ 2.29	–	–	–	–	–
Conch	–	–	–	–	–	–	93.08 $\pm$ 2.44	–	–
MedImageInsight	75.64 $\pm$ 2.12	75.32 $\pm$ 2.83	85.87 $\pm$ 0.38	44.65 $\pm$ 0.74	95.47 $\pm$ 0.29	90.79 $\pm$ 1.18	71.79 $\pm$ 2.92	50.18 $\pm$ 2.67	96.21 $\pm$ 0.83
MedVTok	90.54 $\pm$ 1.05	87.03 $\pm$ 1.64	75.27 $\pm$ 0.99	51.00 $\pm$ 1.85	95.03 $\pm$ 0.52	94.00 $\pm$ 1.18	78.97 $\pm$ 1.65	52.92 $\pm$ 1.68	97.22 $\pm$ 0.61

Appendix

Table 10: Classification accuracy (%) compared with MERA experts. Specialized experts are evaluated on their respective modalities. Values are means

\pm

95% confidence intervals.

Model	CFP	CT	Derm.	Endo.	MRI	Path.	US	X-ray
RetCLIP	92.46 $\pm$ 1.69	–	–	–	–	–	–	–
DermLIP	–	–	89.05 $\pm$ 1.86	–	–	–	–	–
PeskaVLP	–	–	–	65.06 $\pm$ 1.51	–	–	–	–
Conch	–	–	–	–	–	83.77 $\pm$ 1.39	–	–
MedImageInsight	92.34 $\pm$ 1.39	25.60 $\pm$ 1.35	85.45 $\pm$ 2.10	66.96 $\pm$ 1.32	55.81 $\pm$ 1.31	42.93 $\pm$ 1.86	95.80 $\pm$ 2.07	50.16 $\pm$ 1.48
MedVTok	97.44 $\pm$ 1.59	42.91 $\pm$ 1.80	90.25 $\pm$ 0.77	72.52 $\pm$ 1.15	69.62 $\pm$ 1.07	77.74 $\pm$ 1.56	96.91 $\pm$ 1.80	61.13 $\pm$ 1.17

Appendix

Table 11: Segmentation Dice (%) compared with MERA experts. Specialized experts are evaluated on their respective modalities. Values are means

\pm

95% confidence intervals.

	AMOS-MM CT				MRISegmenter MRI
Organ	PUMIT	MedVAE 3D	AToken	MedVTok	PUMIT	MedVAE 3D	AToken	MedVTok
Overall	28.67	29.71	36.89	50.13	44.57 $\pm$ 0.76	43.70 $\pm$ 0.80	54.63 $\pm$ 0.68	57.20 $\pm$ 0.76
Spleen	63.68 $\pm$ 1.58	42.94 $\pm$ 2.64	55.76 $\pm$ 2.46	70.61 $\pm$ 2.30	40.83 $\pm$ 2.12	40.68 $\pm$ 2.14	63.28 $\pm$ 2.71	70.60 $\pm$ 2.85
Kidney	48.94 $\pm$ 1.96	45.95 $\pm$ 2.36	59.56 $\pm$ 2.44	74.93 $\pm$ 1.74	81.60 $\pm$ 1.29	77.46 $\pm$ 1.77	83.96 $\pm$ 0.67	86.48 $\pm$ 0.95
Liver	76.63 $\pm$ 2.68	73.73 $\pm$ 2.23	80.50 $\pm$ 1.72	86.31 $\pm$ 1.00	58.11 $\pm$ 1.32	59.11 $\pm$ 1.23	85.71 $\pm$ 0.94	85.83 $\pm$ 0.70
Pancreas	13.42 $\pm$ 1.99	23.51 $\pm$ 2.60	31.64 $\pm$ 2.72	41.90 $\pm$ 2.36	49.43 $\pm$ 2.22	43.98 $\pm$ 2.08	49.23 $\pm$ 1.92	49.49 $\pm$ 2.71

Appendix

Table 12: 3D segmentation Dice (%). Each tokenizer is frozen while a lightweight 3D convolutional head is trained. Values include reported 95% confidence intervals.

Model	MRISegmenter	AMOS-MM MRI
PUMIT	44.57 $\pm$ 0.76	31.44 $\pm$ 1.52
MedVAE-3D	43.70 $\pm$ 0.80	33.52 $\pm$ 3.82
AToken	54.63 $\pm$ 0.68	44.39 $\pm$ 1.75
MedVTok	57.20 $\pm$ 0.76	47.24 $\pm$ 1.47

Appendix

Table 13: Cross-dataset 3D segmentation Dice (%). Models trained on MRISegmenter are evaluated directly on AMOS-MM MRI. Values are means

\pm

95% confidence intervals.

Metrics	Method	CFP	CT	Derm.	Endo.	MRI	Path.	X-ray	Avg.
KID ( $\downarrow$ )	PUMIT [ 8 ]	0.098	0.110	0.096	0.110	0.067	0.296	0.080	0.123
	MedVAE [ 5 ]	0.116	0.084	0.135	0.154	0.077	0.288	0.068	0.132
	MedITok [ 6 ]	0.256	0.099	0.078	0.024	0.054	0.055	0.055	0.089
	MedVTok	0.027	0.076	0.077	0.035	0.048	0.139	0.019	0.060
FID ( $\downarrow$ )	PUMIT [ 8 ]	140.40	183.49	272.26	191.49	227.64	377.12	200.14	227.51
	MedVAE [ 5 ]	118.13	154.42	202.09	170.29	145.25	325.87	88.55	172.08

Appendix

Table 14: Additional results on image synthesis evaluation.

Model	Connector only Qwen3 judge	Standard Qwen3 judge	Standard GPT-5 judge
PUMIT	23.05 $\pm$ 0.44	41.94 $\pm$ 0.33	41.83 $\pm$ 0.33
MedVAE	17.88 $\pm$ 0.48	39.93 $\pm$ 0.41	39.58 $\pm$ 0.42
MedITok	24.76 $\pm$ 0.39	45.39 $\pm$ 0.28	44.82 $\pm$ 0.26
MedVTok	29.86 $\pm$ 0.36	46.48 $\pm$ 0.22	46.02 $\pm$ 0.24

Appendix

Table 15: VQA accuracy (%) under training and judge controls. Connector-only training freezes the language model and uses Qwen3-235B-A22B for grading. Standard training updates the connector and language-model LoRA adapters. Values are means

\pm

95% confidence intervals.

Metrics	Method	Abdomen	Chest	Pelvis	Avg.
BLEU ( $\uparrow$ )	PUMIT [ 8 ]	0.289 $\pm$ 0.008	0.193 $\pm$ 0.003	0.164 $\pm$ 0.007	0.215 $\pm$ 0.005
	MedVAE-3D [ 5 ]	0.249 $\pm$ 0.038	0.185 $\pm$ 0.020	0.152 $\pm$ 0.010	0.195 $\pm$ 0.013
	MedITok [ 6 ]	0.256 $\pm$ 0.013	0.219 $\pm$ 0.020	0.294 $\pm$ 0.017	0.257 $\pm$ 0.012
	MedVTok	0.272 $\pm$ 0.005	0.263 $\pm$ 0.045	0.288 $\pm$ 0.017	0.274 $\pm$ 0.009
ROUGE-L ( $\uparrow$ )	PUMIT [ 8 ]	0.415 $\pm$ 0.005	0.277 $\pm$ 0.005	0.207 $\pm$ 0.007	0.300 $\pm$ 0.003
	MedVAE-3D [ 5 ]	0.398 $\pm$ 0.004	0.266 $\pm$ 0.003	0.183 $\pm$ 0.004	0.282 $\pm$ 0.000

Appendix

Table 16: Additional results on medical report generation evaluation.

Figure 9: Ablation study of

\lambda_{\text{MERA}}

. Classification accuracy and segmentation Dice improve with increasing MERA weight, while reconstruction fidelity (PSNR, SSIM) gradually degrades.

\lambda_{\text{MERA}}=1

balances semantic quality and reconstruction.

Figure 10: Ablation study of

\lambda_{\text{SliDR}}

. SliDR improves 3D reconstruction without affecting 2D fidelity.

\lambda_{\text{SliDR}}=0.1

provides the best trade-off, as larger values degrade classification (left).

	AMOS-MM CT		MRISegmenter MRI
Objective	PSNR (dB)	SSIM (%)	PSNR (dB)	SSIM (%)
Direct gradient	33.24 $\pm$ 0.22	93.01 $\pm$ 0.25	26.11 $\pm$ 0.14	78.62 $\pm$ 0.44
TV	33.05 $\pm$ 0.21	93.08 $\pm$ 0.24	26.04 $\pm$ 0.14	78.00 $\pm$ 0.45
Laplacian	32.99 $\pm$ 0.21	93.08 $\pm$ 0.24	26.20 $\pm$ 0.15	77.86 $\pm$ 0.45
SliDR	33.46 $\pm$ 0.22	93.21 $\pm$ 0.25	26.23 $\pm$ 0.15	78.70 $\pm$ 0.44

Appendix

Table 17: Comparison of depth regularization objectives. All variants use the same 2D-trained checkpoint and a controlled 10K-volume training subset. Higher PSNR and SSIM indicate better reconstruction. The best mean in each column is marked in bold .

Model	$k$ -NN	Linear probe
MedVAE	40.47 $\pm$ 0.15	79.47 $\pm$ 1.35
PUMIT	36.98 $\pm$ 0.55	83.25 $\pm$ 0.84
MedITok	90.73 $\pm$ 0.62	94.62 $\pm$ 0.76
AToken	92.00 $\pm$ 2.07	92.84 $\pm$ 0.87
MedVTok	94.78 $\pm$ 1.81	96.84 $\pm$ 0.59

Appendix

Table 18: Classification accuracy (%) on the unseen PBC modality. Both protocols use frozen tokenizer features. Values are means

\pm

95% confidence intervals.

Model	PSNR (dB)	Accuracy (%)	Dice (%)
AToken	39.35	63.65	69.57
MedITok	36.83	66.59	67.01
MedVTok	39.71	71.70	73.21

Appendix

Table 19: Comparison on the same 2M training images.

Figure 11: $t$ -SNE visualization of frozen tokenizer representations. (a) Out-of-distribution microscopy blood cell dataset. (b) JSIEC fundus dataset (showing the top 16 retinal conditions). MedVTok produces the most clearly separated and coherent clusters in both settings.

Figure 12: Representative responses from the LLaVA-Med model integrating MedVTok as the visual backbone.

Figure 13: Representative medical reports generated by the LaMed model integrating MedVTok as the visual backbone.

Figure 14: Training data distribution across imaging modalities (2D corpus). 2D corpus comprising 28.7M images across 9 modalities and 115 datasets. The colored mid-ring encodes modality with arc length proportional to its sample count; outer spokes correspond to source datasets, with spoke length scaled by

\log_{10}(\text{count})

. Cross-modality datasets ( e.g. , RadImageNet, BIOMEDICA, GLIS-RT) are split equally across their constituent modalities. Long dataset names are abbreviated for layout; full names are listed in Table 20 .

Figure 15: Training data distribution across imaging modalities (3D corpus). 3D corpus comprising 70K volumes across 2 modalities and 37 datasets. The colored mid-ring encodes modality with arc length proportional to its sample count; outer spokes correspond to source datasets, with spoke length scaled by

\log_{10}(\text{count})

. Cross-modality datasets are split equally across their constituent modalities. Long dataset names are abbreviated for layout; full names are listed in Table 21 .

Dataset	Modality	Count
RSNA STR Pulmonary Embolism Detection [ 69 ]	CT	2,854,455
MELA [ 70 ]	CT	1,198,641
RibFrac2020 [ 71 ]	CT	675,992
ATM2022 [ 72 ]	CT	419,489
LIDC-IDRI-ALL-CT [ 73 ]	CT	403,462
FLARE22 [ 74 ]	CT	236,577

Appendix

Table 20: 2D training datasets.

Dataset	Modality	Count
CT-RATE [ 178 ]	CT	15,000
RSNA STR Pulmonary Embolism Detection [ 69 ]	CT	7,148
FLARE22 [ 74 ]	CT	1,874
LIDC-IDRI-ALL-CT [ 73 ]	CT	967
AbdomenCT1K [ 77 ]	CT	882
QIN-HEADNECK [ 75 ]	CT	816

Appendix

Table 21: 3D training datasets.

Image Tokenization

Image tokenization converts images into compact discrete or continuous visual units for efficient generation, compression, and multimodal reasoning.

See all 57 papers

A More Word-like Image Tokenization for MLLMs
May 18, 2026Hyun Lee, Hyemin Jeong, Yejin Kim +4Image Tokenization Efficient VLM Inference
Seoul National University · Ewha Womans University
Token-Sparse Medical Multimodal Reasoning via Dual-Stream Reinforcement Learning
Jun 30, 2026Kaitao Chen, Weiqian Zhao, Jiamin Wu +6Efficient Multimodal Inference Medical VQA
1Fudan University · 2Shanghai Artificial Intelligence Laboratory · 3Shanghai Jiao Tong University +1
What Makes a Good Medical Image Tokenizer? Rethinking Reconstruction and Generation in Medical Image Tokenization
Sep 21, 2026Niklas Bubeck, Yundi Zhang, Vasiliki Sideri-Lampretsa +4Image Tokenization Medical Image Synthesis
School of Computation, Information and Technology, Technical University of Munich, Germany · Munich Center for Machine Learning, Technical University of Munich, Germany · School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Germany +3

May 18, 2026J/K move · Enter open · S save

A More Word-like Image Tokenization for MLLMs

Hyun Lee, Hyemin Jeong, Yejin Kim, Hyungwook Choi+3

Seoul National University · Ewha Womans University

Image Tokenization Efficient VLM Inference Visual Tokenization Multimodal Large Language Models Multimodal Foundation Models Visual Token Compression+1

Open paper AskPDF arXiv

Modern multimodal large language models (MLLMs) typically keep the language model fixed and train a visual projector that maps the pixels into a sequence of tokens in its embedding space, so that images can be presented in essentially the same form as text. However, the language model has been optimized to operate on discrete, semantically meaningful tokens, while prevailing visual projectors transform an image into a long stream of continuous and highly correlated embeddings. This causes the visual tokens to behave differently from the word-like units that LLMs are originally trained to understand. We propose a novel Disentangled Visual Tokenization (DiVT) that clusters patch embeddings into coherent semantic units, so each token corresponds to a distinct visual concept instead of a rigid grid cell. DiVT further adapts its token budget to image complexity, providing an explicit accuracy-compute trade-off modifying neither the vision encoder nor the language model. Across diverse multimodal benchmarks, DiVT matches or surpasses baselines with significantly fewer visual tokens, demonstrating robustness under limited token budgets, significantly reducing memory cost and latency while making visual inputs more compatible with LLMs. Our code is available at https://github.com/snuviplab/DiVT.

Abstract

Image Tokenization

Explore similar work

A More Word-like Image Tokenization for MLLMs

Token-Sparse Medical Multimodal Reasoning via Dual-Stream Reinforcement Learning

What Makes a Good Medical Image Tokenizer? Rethinking Reconstruction and Generation in Medical Image Tokenization