MedKCO: Medical Vision-Language Pretraining via Knowledge-Driven Cognitive Orchestration

Chenran Zhang; Ruiqi Wu; Tao Zhou; Yi Zhou

cs.CVMar 10, 2026

MedKCO: Medical Vision-Language Pretraining via Knowledge-Driven Cognitive Orchestration

Authors: Chenran Zhang, Ruiqi Wu, Tao Zhou, Yi Zhou

Organizations: School of Computer Science and Engineering, Southeast University, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, China · School of Computer Science and Engineering, Nanjing University of Science and Technology, China

Abstract

Medical vision-language pretraining (VLP) models have recently been investigated for their generalization to diverse downstream tasks. However, current medical VLP methods typically force the model to learn simple and complex concepts simultaneously. This anti-cognitive process leads to suboptimal feature representations, especially under distribution shift. To address this limitation, we propose a Knowledge-driven Cognitive Orchestration for Medical VLP (MedKCO) that involves both the ordering of the pretraining data and the learning objective of vision-language contrast. Specifically, we design a two level curriculum by incorporating diagnostic sensitivity and intra-class sample representativeness for the ordering of the pretraining data. Moreover, considering the inter-class similarity of medical images, we introduce a self-paced asymmetric contrastive loss to dynamically adjust the participation of the pretraining objective. We evaluate the proposed pretraining method on three medical imaging scenarios in multiple vision-language downstream tasks, and compare it with several curriculum learning methods. Extensive experiments show that our method significantly surpasses all baselines. https://github.com/Mr-Talon/MedKCO.

Figures & tables

Figure 1: Motivations of knowledge-driven cognitive orchestration. (a) Diagnostic sensitivity varies among different diseases. (b) Representativeness of intra-class samples exhibits variation. (c) High inter-class similarity in medical images at the beginning of the pretraining.

Figure 2: Overview of the two-level curriculum. The pretraining data is divided into two distinct levels, label-level (left) and description-level (right). The label-level data is categorized into three stages according to the sensitivity of each modality to detect specific disease. The description-level data is clustered into the most relevant categories based on their textual descriptions. Subsequently, within each category, samples are divided into multiple stages according to the representativeness of their image features.

Modality	Dataset	Metric	CLIP	CL-log [ 15 ]	CL-logit [ 3 ]	MedKCO	FILIP	CL-log [ 15 ]	CL-logit [ 3 ]	MedKCO
CFP	ODIR200×3	ACC	0.772	0.835	0.567	0.863	0.457	0.563	0.528	0.833
	REFUGE	ACC	0.897	0.861	0.803	0.947	0.92	0.875	0.832	0.891
	FIVES	AUC	0.676	0.682	0.659	0.729	0.554	0.661	0.419	0.575
OCT	OCTID	ACC	0.709	0.688	0.743	0.778	0.600	0.600	0.600	0.667
OCT	OCTDL	ACC	0.306	0.351	0.359	0.388	0.307	0.322	0.292	0.420
CXR	CheXpert5*200	ACC	0.384	0.466	0.443	0.526	0.227	0.416	0.389	0.548

Table 1: Result of zero-shot classification. The best result is bolded and the second-best is underlined .

Models	OpenI							MIMIC-CXR							AVG
Models	BLEU1	BLEU2	BLEU3	BLEU4	METEOR	ROUGE	CIDER	BLEU1	BLEU2	BLEU3	BLEU4	METEOR	ROUGE	CIDER	AVG
CLIP	0.362	0.216	0.146	0.105	0.140	0.311	0.293	0.259	0.157	0.102	0.072	0.112	0.246	0.104	0.188
CL-logit [ 15 ]	0.359	0.211	0.140	0.098	0.138	0.309	0.255	0.258	0.154	0.099	0.070	0.111	0.242	0.107	0.182
CL-log [ 3 ]	0.328	0.197	0.134	0.097	0.132	0.303	0.286	0.269	0.162	0.105	0.074	0.114	0.246	0.106	0.182
MedKCO	0.389	0.232	0.157	0.112	0.147	0.322	0.303	0.279	0.168	0.109	0.076	0.117	0.247	0.109	0.198
FILIP	0.378	0.232	0.158	0.114	0.149	0.326	0.316	0.252	0.151	0.099	0.070	0.109	0.242	0.097	0.192

Table 2: Result of report generation. The best result is bolded and the second-best is underlined .

Figure 3: Visualization of image feature at different stages on ODIR200×3 dataset under the CLIP framework.

Models	OpenI			MIMIC-CXR			AVG
Models	R@1	R@5	R@10	R@1	R@5	R@10	AVG
CLIP	0.3	2.0	3.5	3.0	11.5	18.3	6.4
CL-logit [ 15 ]	0.6	2.1	3.6	3.4	11.9	18.5	6.7
CL-log [ 3 ]	1.0	3.7	6.1	5.0	18.0	27.2	10.2
MedKCO	1.2	4.0	6.4	6.4	21.7	31.7	11.9
FILIP	0.7	2.1	3.3	6.9	21.1	31.3	10.9

Table 3: Result of zero-shot image-to-text retrieval. The best result is bolded and the second-best is underlined .

Table 8

Modality	Label-level	Description-level
CFP	EYEPACS, IDRID, RFMid, DEN, LAG, ODIR, PAPILA, PARAGUAY, STARE, ARIA, AGAR300, APTOS, FUND-OCT, JICHI,	MM-Retinal CFP
	DiaRetDB1, DRIONS-DB, Drishti-GS1, E-ophta, G1020, HRF, ORIGA, ROC, BRSET, OIA-DDR, AIROGS, SUSTech-SYSU,
	CHAKSU, DR1-2, Cataract, ScarDat
OCT	RetinalOCT_C8, Large_Dataset_of_Labeled_OCT, GAMMA1, STAGE1, STAGE2, glaucoma_detection, GOALS, OIMHS,	MM-Retinal OCT
OCT	OCTA_500, DUKE_DME, BIOMISA_Retinal_Image_Database_for_Macular_Disorders	MM-Retinal OCT
CXR	CheXpert	MIMIC-CXR

Table 8: Pretraining datasets of each modality.

Modality	Stage	Label
CFP	Stage 1	hard exudates, soft exudates, microaneurysms, haemorrhages, media haze, drusens, tessellation, laser scar,
		optic disc cupping, tortuous vessels, asteroid hyalosis, optic disc pallor, exudates, cotton wool spots,
		colobomas, preretinal haemorrhage, myelinated nerve fibers, tilted disc, vitreous haemorrhage, large optic cup,
		optic atrophy, fibrosis, silicon oil, scar, nevus, red small dots
	Stage 2	no diabetic retinopathy, mild diabetic retinopathy, moderate diabetic retinopathy, severe diabetic retinopathy,
		proliferative diabetic retinopathy, age-related macular degeneration, pathologic myopia, macular scar, shunt,

Table 9: The disease at different stage of each modality in label-level curriculum

Figure 5: Visualization of text feature at different stages on ODIR200×3 dataset under the CLIP framework.

Explore similar work

VIVID-Med: LLM-Supervised Structured Pretraining for Deployable Medical ViTs
Mar 10, 2026Xiyao Wang, Xiaoyu Tan, Yang Dai +3Medical Image Representation Learning Vision-Language Pretraining
Shanghai University of Engineering Science, China · National University of Singapore, Singapore · Case Western Reserve University, United States
Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicine
Mar 2, 2026Yuan Wu, Zongxian Yang, Jiayu Qian +5Medical VLMs VLM Reasoning
Department of Computer Science, City University of Hong Kong (Dongguan) · College of Computing and Data Science (CCDS), Nanyang Technological University · School of Computer Science, Northwestern Polytechnical University
OpenMedQ: Broad Open Pretraining for Medical Vision-Language Models
Jun 11, 2026Ibrahim Gulluk, Max Van Puyvelde, Olivier GevaertMedical VQA Medical Image Classification
Department of Electrical Engineering, Stanford University · Department of Biomedical Data Science, Stanford University School of Medicine · Department of Mathematical Modelling, Statistics and Bioinformatics, Ghent University

Mar 10, 2026J/K move · Enter open · S save

VIVID-Med: LLM-Supervised Structured Pretraining for Deployable Medical ViTs

Xiyao Wang, Xiaoyu Tan, Yang Dai, Yuxuan Fu+2

Shanghai University of Engineering Science, China · National University of Singapore, Singapore · Case Western Reserve University, United States

Medical Image Representation Learning Vision-Language Pretraining Medical Imaging Vision Transformer

Open paper AskPDF arXiv

Vision-language pretraining has driven significant progress in medical image analysis. However, current methods typically supervise visual encoders using one-hot labels or free-form text, neither of which effectively captures the complex semantic relationships among clinical findings. In this study, we introduce VIVID-Med, a novel framework that leverages a frozen large language model (LLM) as a structured semantic teacher to pretrain medical vision transformers (ViTs). VIVID-Med translates clinical findings into verifiable JSON field-state pairs via a Unified Medical Schema (UMS), utilizing answerability-aware masking to focus optimization. It then employs Structured Prediction Decomposition (SPD) to partition cross-attention into orthogonality-regularized query groups, extracting complementary visual aspects. Crucially, the LLM is discarded post-training, yielding a lightweight, deployable ViT-only backbone. We evaluated VIVID-Med across multiple settings: on CheXpert linear probing, it achieves a macro-AUC of 0.8588, outperforming BiomedCLIP by +6.65 points while using 500x less data. It also demonstrates robust zero-shot cross-domain transfer to NIH ChestX-ray14 (0.7225 macro-AUC) and strong cross-modality generalization to CT, achieving 0.8413 AUC on LIDC-IDRI lung nodule classification and 0.9969 macro-AUC on OrganAMNIST 11-organ classification. VIVID-Med offers a highly efficient, scalable alternative to deploying resource-heavy vision-language models in clinical settings.

Abstract

Explore similar work

VIVID-Med: LLM-Supervised Structured Pretraining for Deployable Medical ViTs

Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicine

OpenMedQ: Broad Open Pretraining for Medical Vision-Language Models