VIVID-Med: LLM-Supervised Structured Pretraining for Deployable Medical ViTs

Xiyao Wang; Xiaoyu Tan; Yang Dai; Yuxuan Fu; Shuo Li; Xihe Qiu

cs.CVMar 10, 2026

VIVID-Med: LLM-Supervised Structured Pretraining for Deployable Medical ViTs

Authors: Xiyao Wang, Xiaoyu Tan, Yang Dai, Yuxuan Fu, Shuo Li, Xihe Qiu

Organizations: Shanghai University of Engineering Science, China · National University of Singapore, Singapore · Case Western Reserve University, United States

Abstract

Vision-language pretraining has driven significant progress in medical image analysis. However, current methods typically supervise visual encoders using one-hot labels or free-form text, neither of which effectively captures the complex semantic relationships among clinical findings. In this study, we introduce VIVID-Med, a novel framework that leverages a frozen large language model (LLM) as a structured semantic teacher to pretrain medical vision transformers (ViTs). VIVID-Med translates clinical findings into verifiable JSON field-state pairs via a Unified Medical Schema (UMS), utilizing answerability-aware masking to focus optimization. It then employs Structured Prediction Decomposition (SPD) to partition cross-attention into orthogonality-regularized query groups, extracting complementary visual aspects. Crucially, the LLM is discarded post-training, yielding a lightweight, deployable ViT-only backbone. We evaluated VIVID-Med across multiple settings: on CheXpert linear probing, it achieves a macro-AUC of 0.8588, outperforming BiomedCLIP by +6.65 points while using 500x less data. It also demonstrates robust zero-shot cross-domain transfer to NIH ChestX-ray14 (0.7225 macro-AUC) and strong cross-modality generalization to CT, achieving 0.8413 AUC on LIDC-IDRI lung nodule classification and 0.9969 macro-AUC on OrganAMNIST 11-organ classification. VIVID-Med offers a highly efficient, scalable alternative to deploying resource-heavy vision-language models in clinical settings.

Figures & tables

Figure 1: VIVID-Med overview. The frozen LLM provides structured token supervision during training only. Gradients update the parameters of the ViT and SPD projector, while the deployed model retains only the lightweight ViT backbone.

Figure 2: Structured Prediction Decomposition (SPD). Multiple query groups perform cross-attention over shared ViT tokens. Orthogonality regularization encourages complementary branches before shared projection to the LLM embedding space.

Figure 3: SPD cross-attention maps for a frontal (top) and lateral (bottom) CXR. Each group attends to distinct anatomical regions.

Method	CheXpert-12 (in-domain)		NIH-8 (cross-domain)
Method	Macro-AUC	Macro-F1	Macro-AUC	Macro-F1
ImageNet supervised [ 4 ]	$0.7746\,{\scriptstyle\pm\,0.0081}$	$0.8813\,{\scriptstyle\pm\,0.0046}$	$0.6147\,{\scriptstyle\pm\,0.0012}$	$0.2090\,{\scriptstyle\pm\,0.0111}$
MAE [ 6 ]	$0.7211\,{\scriptstyle\pm\,0.0031}$	$0.8657\,{\scriptstyle\pm\,0.0096}$	$0.5945\,{\scriptstyle\pm\,0.0013}$	$0.1847\,{\scriptstyle\pm\,0.0005}$
DINOv3 [ 19 ]	$0.6876\,{\scriptstyle\pm\,0.0053}$	$0.8760\,{\scriptstyle\pm\,0.0014}$	$0.6287\,{\scriptstyle\pm\,0.0023}$	$0.2049\,{\scriptstyle\pm\,0.0050}$
BiomedCLIP [ 24 ]	$\underline{0.7923\,{\scriptstyle\pm\,0.0249}}$	$\underline{0.8989\,{\scriptstyle\pm\,0.0050}}$	$\underline{0.6725\,{\scriptstyle\pm\,0.0076}}$	$\underline{0.2246\,{\scriptstyle\pm\,0.0058}}$
Random-mask proxy (75%)	$0.7396\,{\scriptstyle\pm\,0.0020}$	$0.8705\,{\scriptstyle\pm\,0.0156}$	$0.6000\,{\scriptstyle\pm\,0.0018}$	$0.1969\,{\scriptstyle\pm\,0.0031}$

Table 1: Quantitative performance of VIVID-Med and baselines on in-domain (CheXpert) and cross-domain (NIH) tasks. Results are reported as mean

\pm

std over 3 seeds. NIH results are evaluated purely zero-shot without retraining.

Figure 4: t-SNE of CLS embeddings on CheXpert. VIVID-Med produces tighter, more separable clusters than ImageNet supervised and BiomedCLIP baselines.

Method	LIDC-IDRI (binary)		OrganAMNIST (11-class)
Method	AUC	F1	Macro-AUC	Macro-F1
ImageNet supervised [ 4 ]	$0.7933\,{\scriptstyle\pm\,0.0336}$	$0.6760\,{\scriptstyle\pm\,0.0251}$	$\underline{0.9928\,{\scriptstyle\pm\,0.0001}}$	$\underline{0.8846\,{\scriptstyle\pm\,0.0007}}$
MAE [ 6 ]	$0.7607\,{\scriptstyle\pm\,0.0507}$	$0.5341\,{\scriptstyle\pm\,0.0350}$	$0.9826\,{\scriptstyle\pm\,0.0001}$	$0.7743\,{\scriptstyle\pm\,0.0024}$
DINOv3 [ 19 ]	$0.6675\,{\scriptstyle\pm\,0.0668}$	$0.4389\,{\scriptstyle\pm\,0.0191}$	$0.9221\,{\scriptstyle\pm\,0.0009}$	$0.4570\,{\scriptstyle\pm\,0.0065}$
BiomedCLIP [ 24 ]	$\mathbf{0.8465\,{\scriptstyle\pm\,0.0148}}$	$\underline{0.7235\,{\scriptstyle\pm\,0.0040}}$	$0.9913\,{\scriptstyle\pm\,0.0002}$	$0.8732\,{\scriptstyle\pm\,0.0052}$
VIVID-Med (ours)	$\underline{0.8413\,{\scriptstyle\pm\,0.0260}}$	$\mathbf{0.7563\,{\scriptstyle\pm\,0.0357}}$	$\mathbf{0.9969\,{\scriptstyle\pm\,0.0001}}$	$\mathbf{0.9322\,{\scriptstyle\pm\,0.0004}}$

Table 2: Cross-modality transfer: CXR-pretrained ViT

\rightarrow

CT slice classification (mean

\pm

std, 3 seeds). No CT data is used during pretraining.

Setting	UMS	SPD	Macro-AUC	Macro-F1
VIVID-Med (free-text)			$0.8253\,{\scriptstyle\pm\,0.0028}$	$0.8877\,{\scriptstyle\pm\,0.0027}$
VIVID-Med (no SPD)	✓		$\underline{0.8431\,{\scriptstyle\pm\,0.0035}}$	$0.8963\,{\scriptstyle\pm\,0.0012}$
VIVID-Med	✓	✓	$\mathbf{0.8588\,{\scriptstyle\pm\,0.0033}}$	$\underline{0.9016\,{\scriptstyle\pm\,0.0060}}$
Q-Former proxy	✓	$\triangle$	$0.8182\,{\scriptstyle\pm\,0.0195}$	$\mathbf{0.9167\,{\scriptstyle\pm\,0.0016}}$

Table 3: Ablation studies isolating the impact of supervision and architecture on the CheXpert linear probe task. Results are reported as mean

\pm

std over 3 seeds.

Explore similar work

Sep 5, 2026

Beyond Classification: Structured Supervision Aligns Visual Evidence with Medical Semantics

Vision Transformers (ViTs) have shown immense potential in medical image analysis. However, standard pre-training via global image classification suffers from spatial collapse, where models rely heavily on background shortcuts rather than localising critical foreground lesions. To overcome this limitation and align visual evidence with precise medical semantics, we systematically investigate alternative pre-training paradigms.Specifically, we evaluate three independent forms of structured supervision: topological priors via graph self-supervision, dense pixel-level constraints via segmentation, and cross-modal semantic grounding via image-text pairs. Notably, our empirical analysis reveals that while all three forms of structured supervision successfully alleviate the global pooling bottleneck and steer visual attention towards foreground regions, image-text alignment achieves the most superior performance. By embedding high-dimensional diagnostic logic, the cross-modal approach not only anchors attention on precise visual evidence but also enables profound abstract reasoning. Extensive experiments demonstrate that this semantically enriched pre-training fundamentally enhances the model's feature representation. Consequently, when fine-tuned for downstream clinical classification tasks, our models achieve superior accuracy and yield highly interpretable attention maps focused on true pathological features, vastly outperforming vanilla classification baselines.

Hexiang Bai, Hanyang Xu, Xiaoxue Li +4

Zhejiang University Hangzhou, China · Henan Normal University Xinxiang, China · University of Southampton Southampton, United Kingdom +3

Medical Image Representation Learning Medical Image Classification Multimodal Pretraining Medical VLMs Explainable Medical Image Analysis

Ask PDF

Jun 4, 2026

MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models

Medical large vision-language models (Med-LVLMs) have recently achieved remarkable progress in vision-language comprehension and medical image segmentation. However, existing models still struggle to unify these two capabilities, which is essential for achieving clinically reasoning that connects visual findings with semantic interpretation. We present MedSIGHT, a unified framework that equips Med-LVLMs with structured, pixel-level understanding for grounded visual comprehension. MedSIGHT introduces a novel Region Perceiver module that produces region-centric tokens, encoding spatial information directly into representation space of the language model. We further propose a medical region codebook into the LLM vocabulary, allowing the model to generate discrete region codes as symbolic representations of anatomical and pathological regions. These codes are decoded through the Region Perceiver to reconstruct segmentation mask, achieving end-to-end spatial grounding. Lastly, MedSIGHT combines Region Perceiver, Codebook and LLM using our proposed progressive training strategy to gradually aligns these modules stably. Trained on only 72K multimodal instruction pairs, MedSIGHT achieves state-of-the-art performance across diverse imaging modalities on both medical comprehension and segmentation tasks.

Aofei Chang, Le Huang, Alex James Boyd +4

GE HealthCare · College of Information Sciences and Technology, The Pennsylvania State University, State College, PA, USA · GE Healthcare, Bellevue, WA, USA.

Medical VLMs Medical Image Segmentation Medical Image Grounding Visual Grounding

Ask PDF

Mar 10, 2026

MedKCO: Medical Vision-Language Pretraining via Knowledge-Driven Cognitive Orchestration

Medical vision-language pretraining (VLP) models have recently been investigated for their generalization to diverse downstream tasks. However, current medical VLP methods typically force the model to learn simple and complex concepts simultaneously. This anti-cognitive process leads to suboptimal feature representations, especially under distribution shift. To address this limitation, we propose a Knowledge-driven Cognitive Orchestration for Medical VLP (MedKCO) that involves both the ordering of the pretraining data and the learning objective of vision-language contrast. Specifically, we design a two level curriculum by incorporating diagnostic sensitivity and intra-class sample representativeness for the ordering of the pretraining data. Moreover, considering the inter-class similarity of medical images, we introduce a self-paced asymmetric contrastive loss to dynamically adjust the participation of the pretraining objective. We evaluate the proposed pretraining method on three medical imaging scenarios in multiple vision-language downstream tasks, and compare it with several curriculum learning methods. Extensive experiments show that our method significantly surpasses all baselines. https://github.com/Mr-Talon/MedKCO.

Chenran Zhang, Ruiqi Wu, Tao Zhou +1

School of Computer Science and Engineering, Southeast University, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, China · School of Computer Science and Engineering, Nanjing University of Science and Technology, China

Medical VLMs Vision-Language Pretraining Contrastive Learning

Ask PDF