Pretraining of Medical Visual Encoders Toward Multi-modal Large Language Models

Tianyou Jiang

cs.AISep 20, 2026

Pretraining of Medical Visual Encoders Toward Multi-modal Large Language Models

Authors: Tianyou Jiang

Organizations: University of Bern, Bern, Switzerland

Abstract

Multimodal Large Language Models (MLLMs) commonly reuse visual encoders pretrained with CLIP, although the features of these ViTs are ultimately consumed by autoregressive LLMs. We refer to this mismatch as the semantic-interface gap and introduce MedMLIP, a framework that pretrains the visual encoder through report generation with a frozen LLM, while employing Local Relational Distillation (LRD) to preserve relationships among visual patches to avoid visual collapse. We pretrain MedMLIP on IU-Xray and Open-PMC-300K and evaluate the resulting encoders on VQA-RAD and SLAKE. Only the ViT is transferred, while the guiding LLM and projector are replaced, allowing us to assess cross-LLM transferability. Our cross-LLM transfer experiments demonstrate the value of pretraining visual encoders for their autoregressive LLM interface while trying to preserve more fine-grained visual information. Code and the pretrained model are available at https://github.com/SkyCol/MedMLIP

Figures & tables

Figure 1: Representative vision–language alignment paradigms in MLLMs: (a) projection-based token fusion, (b) query-based cross-modal fusion, and (c) layer-wise deep fusion. Although differing in how visual information is mapped and integrated, all paradigms fundamentally depend on the quality of the visual representations provided to the LLM.

Visual Encoder / Objective	Pretraining Data	Guiding LLM	Downstream LLM	VQA-RAD ACC	SLAKE ACC
CLIP	–	–	Qwen0.5B	55.15	79.77
BioMedCLIP	–	–	Qwen0.5B	61.50	72.38
CLIP	IU-Xray	–	Qwen0.5B	57.14	80.61
MedMLIP ( $\mathcal{L}_{\mathrm{gen}}$ )	IU-Xray	Qwen3B	Qwen0.5B	60.71	76.68
MedMLIP ( $\mathcal{L}_{\mathrm{gen}}+\mathcal{L}_{\mathrm{LRD}}$ )	IU-Xray	Qwen3B	Qwen0.5B	61.87	83.14
MedMLIP ( $\mathcal{L}_{\mathrm{gen}}+\mathcal{L}_{\mathrm{LRD}}$ )	Open-PMC-300K	Qwen3B	Qwen0.5B	66.66	86.79

Table 1: Transfer performance of different visual encoders on VQA-RAD and SLAKE. For MedMLIP, the visual encoder is transferred across different LLMs: the ViT pretrained with Qwen0.5B is evaluated with Qwen3B, and vice versa. Only the visual encoder is transferred, while the downstream projector and LLM are replaced.

Figure 2: Effect of downstream training-set size on VQA-RAD performance. We evaluate different visual encoder initialization strategies using 5%, 25%, 50%, and 100% of the VQA-RAD training set with Qwen3B and Qwen0.5B as downstream language models. Highlighted cells indicate the best result within each visual encoder setting. Performance is notably non-monotonic with respect to training-set size, suggesting that the composition of downstream supervision can have a substantial effect beyond the raw number of training examples.

Transfer	VQA-RAD			SLAKE
	$\mathcal{L}_{\mathrm{gen}}$	$+\mathcal{L}_{\mathrm{LRD}}$	$\Delta$	$\mathcal{L}_{\mathrm{gen}}$	$+\mathcal{L}_{\mathrm{LRD}}$	$\Delta$
Qwen0.5B $\rightarrow$ Qwen3B	69.84	74.20	+4.36	76.12	80.08	+3.96
Qwen3B $\rightarrow$ Qwen0.5B	60.71	61.87	+1.16	76.68	83.14	+6.46
Average gain	–	–	+2.76	–	–	+5.21

Table 2: Ablation of Local Relational Distillation (LRD). Adding

\mathcal{L}_{\mathrm{LRD}}

consistently improves downstream transfer over autoregressive generation supervision alone.

Medical VLMs

Medical vision-language models connect clinical images with text for diagnosis, reporting, question answering, and image grounding.

See all 475 papers

SCALPEL: Semantic Cross-modal Alignment via LLM-Powered Encoder Learning for Medical Vision-Language Representation
Jul 29, 2026Yunzhan Fu, Enyu Bao, Xiangyu Shen +4Radiology VLMs Medical VLMs
School of Computer Science, Hangzhou Dianzi University, China
How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining
Sep 28, 2026Lin Chen, Bolin Ni, Qi Yang +6Multimodal Pretraining Multimodal Large Language Models
CASIA · UCAS · Foundation Model Department, Tencent
Let ViT Speak: Generative Language-Image Pre-training
May 1, 2026Yan Fang, Mengcheng Lan, Zilong Huang +7Cross-Modal Learning Vision Transformer
1Beijing Jiaotong University · 2ByteDance · 3Nanyang Technological University

Jul 29, 2026J/K move · Enter open · S save

SCALPEL: Semantic Cross-modal Alignment via LLM-Powered Encoder Learning for Medical Vision-Language Representation

Yunzhan Fu, Enyu Bao, Xiangyu Shen, Yihao Wu+3

School of Computer Science, Hangzhou Dianzi University, China

Radiology VLMs Medical VLMs Vision-Language Alignment Multimodal Contrastive Learning Medical Image Representation Learning

Open paper AskPDF arXiv

Vision-language pre-training (VLP) serves as a cornerstone for medical multimodal representation learning. However, existing medical VLP frameworks are often constrained by the limited context windows and shallow representational capacities of lightweight text encoders when processing lengthy, terminology-dense clinical reports. While integrating medical large language models (LLMs) offers unprecedented clinical reasoning capabilities, it introduces three major bottlenecks: (i) the anisotropic representational collapse of generative LLMs under standard contrastive objectives, (ii) the prohibitive memory overhead of joint end-to-end training with large batch sizes, and (iii) the medical hallucinations induced by vanilla contrastive losses that ignore fine-grained anatomical laterality and negation modifiers. To address these challenges, we propose SCALPEL, a Semantic Cross-modal Alignment framework via LLM-Powered Encoder Learning. First, Clinical Report Contrastive fine-tuning converts a generative LLM into an isotropic encoder via domain-specific clinical text adaptation. Second, an asymmetric alignment strategy leverages offline feature caching to enable efficient training. Critically, we formulate an Anatomy-Negation Aware Objective that explicitly penalizes mismatched image-text pairs involving laterality confusion or false negations. Extensive experiments across MIMIC-CXR, CheXpert, and IU X-Ray benchmarks demonstrate that SCALPEL achieves state-of-the-art performance in cross-modal retrieval, zero-shot disease classification and medical visual question answering.

Abstract

Medical VLMs

Explore similar work

SCALPEL: Semantic Cross-modal Alignment via LLM-Powered Encoder Learning for Medical Vision-Language Representation

How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining

Let ViT Speak: Generative Language-Image Pre-training