MicroCLIP: Connecting Text and Microscopy Images via Contrastive Language--Image Pretraining

Yanran Ruan; Zhixiang Wei; Shiyao Hong; Xiaoxiao Ma; Yuxuan Gu; Liming Ye; Huaian Chen; Yi Jin

cs.CVOct 13, 2026

MicroCLIP: Connecting Text and Microscopy Images via Contrastive Language--Image Pretraining

Authors: Yanran Ruan, Zhixiang Wei, Shiyao Hong, Xiaoxiao Ma, Yuxuan Gu, Liming Ye, Huaian Chen, Yi Jin

Organizations: University of Science and Technology of China

Abstract

Understanding fluorescence microscopy images is fundamental to automating biological research. However, existing methods typically learn predefined concepts from closed-set data, limiting transfer across tasks and biological contexts. In this work, we introduce MicroCLIP, a vision-language model that learns generalizable cellular and subcellular representations from natural language supervision for open-set image classification and retrieval. To pretrain MicroCLIP, we construct MicroPool-2M, a dataset of approximately 2.5 million fluorescence microscopy image-text pairs spanning diverse cellular structures, cell lines, and imaging conditions. Aligning microscopy images with biological concepts in natural language is challenging because most images lack descriptive captions and biological knowledge annotations. We therefore progressively establish image-text associations at three levels to provide MicroCLIP with richer semantic supervision: (1) category-level alignment links images to standardized labels defined by subcellular structure-cell-line combinations; (2) knowledge-level alignment incorporates biological priors derived from these labels to capture relationships across species and cell types; and (3) detail-level alignment uses a multimodal large language model to recaption and verify image-text pairs at scale, emphasizing fine-grained visual characteristics. Extensive experiments demonstrate that MicroCLIP achieves leading performance in zero-shot classification and text-image retrieval on microscopy images. It retrieves images with specific cellular characteristics from thousands of candidates and generalizes to unseen biological concepts without additional supervision. These results highlight MicroCLIP's potential as a foundation model for fluorescence microscopy.

Figures & tables

Figure 1: (1) We connect fluorescence microscopy images with text, so that subcellular structures and cell lines can both be represented in a shared semantic space rather than as isolated class IDs. (2) Building on this shared semantic space, MicroCLIP generalizes to unseen structure–cell-line pairs, whereas conventional closed-set methods are restricted to seen pairs. (3) Beyond recognition, this shared representation further enables language-based retrieval, allowing users to search microscopy images through natural language descriptions.

Figure 2: Illustration of the biological grounding gap. (a–c) GPT-6 responses to biological knowledge, visual description, and biological identification queries, respectively. The predicted identity in (c) differs from the reference annotation. (d) Joint structure–cell-line classification accuracy of DFN2B-CLIP, GPT-6, and MicroCLIP on diagnostic subsets of IID and Compositional Shift, with 10 candidate categories and 30 images per split.

Figure 3: MicroCLIP learns a shared image–text representation from category-level, knowledge-level and detail-level supervision. Each microscopy image is paired with one sampled caption for CLIP training. Composition balancing and rare-class augmentation adjust training exposure, while general-domain image–text pairs provide auxiliary supervision.

Figure 4: Classification evaluation subsets: (a) representative images displayed using pseudocolor; (b) category and image counts for the four internal splits, the external Source Shift benchmark, and the training set.

Method	Params.	In-domain classification	Out-of-domain classification				Retrieval
Method	Params.	IID	Compositional Shift	Cell-Line Shift	Organelle-Set Shift	Source Shift	I2T	T2I
Closed-set methods
Fully Supervised Classifier	–	97.74	–	–	–	–	–	–
Cell-DINO	304M	36.06	0.98	–	–	–	–	–
Open-set methods
CLIP (OpenAI)	150M	2.97	5.12	1.80	2.54	8.38	0.7	0.2

Table 1: Comparison of classification and cross-modal retrieval performance. Classification results are Top-1 accuracy (%), retrieval results are Recall@1 (%). Best and second-best results are shown in bold and underlined, respectively.

Method	Cat.	Det.	Know.	Bal.	Aux.	In-domain classification	Out-of-domain classification				Retrieval
Method	Cat.	Det.	Know.	Bal.	Aux.	IID	Compositional Shift	Cell-Line Shift	Organelle-Set Shift	Source Shift	I2T	T2I
Category-level only	✓					97.37	56.47	23.25	40.74	37.31	15.8	12.6
+ Detail-level captions	✓	✓				98.90	63.94	28.69	41.63	40.59	28.4	27.8
+ Knowledge-level captions	✓	✓	✓			98.86	66.36	25.41	48.31	36.36	37.1	37.4
+ Data Balancing	✓	✓	✓	✓		98.24	64.95	31.53	49.91	36.44	39.4	41.5
+ VLM-0.7M	✓	✓	✓	✓	✓	98.46	67.70	34.07	46.13	36.53	61.5	63.3

Table 2: MicroCLIP component ablations. Cat./Det./Know.: category/detail/knowledge captions. Bal.: composition balancing. Aux.: general-domain image–text pairs from VLM-0.7M.

Balancing threshold $\tau_{a}$	Retained (%)	In-domain classification	Out-of-domain classification				Retrieval
Balancing threshold $\tau_{a}$	Retained (%)	IID	Compositional Shift	Cell-Line Shift	Organelle-Set Shift	Source Shift	I2T	T2I
off (no balancing)	100.0	98.76	66.64	26.61	46.78	30.05	32.4	35.7
$50{,}000$	49.8	98.83	64.73	28.44	47.35	39.29	47.0	47.6
$20{,}000$	30.7	98.24	65.52	27.88	47.80	34.54	44.5	45.5
$5{,}000$	10.5	98.46	67.70	34.07	46.13	36.53	61.5	63.3

Table 3: Effect of the composition-balancing threshold

\tau_{a}

(Equation 5 ). Retained (%) denotes the online sample acceptance fraction. Metrics and highlighting follow Table 1 .

Training data	Samples (M)	In-domain classification	Out-of-domain classification				Retrieval
Training data	Samples (M)	IID	Compositional Shift	Cell-Line Shift	Organelle-Set Shift	Source Shift	I2T	T2I
$25\%$	0.63	98.76	60.39	33.05	44.56	40.93	32.1	32.0
$50\%$	1.25	98.49	65.05	30.44	45.19	39.81	32.4	30.8
$100\%$	2.51	98.54	66.97	30.05	45.54	37.31	60.1	60.7
$100\%$ $+$ VLM-0.7M	3.11	98.46	67.70	34.07	46.13	36.53	61.5	63.3

Table 4: Training-data scale and the full recipe. Samples (M) counts images before augmentation and balancing. Metrics and highlighting follow Table 1 .

Figure 5: Qualitative comparison of (a) image-to-text and (b) text-to-image retrieval.

Figure 6: Joint image–text embeddings visualized with 3D PCA. (a) DFN2B-CLIP and (b) MicroCLIP. Each pair of views uses the same projection, highlighting images or text by category and showing the other modality in gray. Colors identify categories consistently across panels; marker shapes identify images and text levels. DFN2B-CLIP shows category text only, while MicroCLIP includes all three text levels.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Publication	Subcellular Structure	Cell Line	Image Size	Data Number
BBBC026	Logan et al. Logan et al. (2016)	DNA	Human Hepatocyte and Murine Fibroblast cells	1040 $\times$ 1392	864
BBBC001	Moffat et al. Moffat et al. (2006)	DNA	HT-29	512 $\times$ 512	64512
		Actin	HT-29	512 $\times$ 512	64512
		pH3	HT-29	512 $\times$ 512	64512
BBBC020	Wenzel et al. Wenzel et al. (2011)	CD11b	Bone Marrow-Derived Macrophages from C57BL/6 mice	1388 $\times$ 3 $\times$ 1040	25
		Nuclei	Bone Marrow-Derived Macrophages from C57BL/6 mice	1388 $\times$ 3 $\times$ 1040	25

Appendix

Table 5: Source datasets for MicroPool-2M. The table lists each dataset, source publication, biological target, cell line or specimen, image dimensions, and record count. Values describe the collected source data before MicroPool-specific extraction, filtering, harmonization, and deduplication. For datasets summarized in a single row, the data number is the sum over the listed structure–cell-line entries.

Subset	Classes	Struct.	Cells	Images	Shift
IID	59	21	12	5,900	none
Compositional Shift	64	16	16	6,400	pair
Cell-Line Shift	59	22	8	5,900	cell line
Organelle-Set Shift	54	29	6	5,400	structure label
Source Shift (external)	13	3	5	1,158	source

Appendix

Table 6: Classification evaluation data. Structure counts refer to complete structure labels, including composite labels. Each internal evaluation category contains 100 images. Source Shift is an external benchmark and is not balanced to 100 images per category.

Figure 7: Training category frequencies, ranked by image count. Dashed lines mark the augmentation and retention thresholds.

Figure 8: Prompt for generating image-independent biological knowledge about subcellular structures and markers with GPT.

Figure 9: Prompt for generating image-independent biological knowledge about cell lines with GPT.

Figure 10: Prompt for generating detail-level captions with Qwen from a microscopy image and its category annotation.

Figure 11: Three-level supervision for one fluorescence microscopy image. The category caption identifies the annotated structure and cell line; the knowledge caption describes image-independent biological properties of these entities; the detail caption describes the individual image.

Caption level	Median	P95	Truncated at 77 (%)	Truncated at 256 (%)
Category	15	19	0.00	0.00
Detail	225	270	99.83	12.97
Knowledge	109	147	100.00	$<0.01$

Appendix

Table 7: Token-length statistics for the three caption levels.

Model	IID	Compositional Shift	Overall
GPT-6	12/30 (40.00%)	10/30 (33.33%)	22/60 (36.67%)
DFN2B-CLIP	6/30 (20.00%)	4/30 (13.33%)	10/60 (16.67%)
MicroCLIP	30/30 (100.00%)	27/30 (90.00%)	57/60 (95.00%)

Appendix

Table 8: Sampled biological grounding diagnostic. Each split contains 10 categories with three images per category. Entries give correct predictions / images and joint Top-1 accuracy. MicroCLIP is evaluated after 15 training epochs.

Backbone	Params.	In-domain classification	Out-of-domain classification				Retrieval
Backbone	Params.	IID	Compositional Shift	Cell-Line Shift	Organelle-Set Shift	Source Shift	I2T	T2I
PE-Core-B/16	448M	98.80	40.94	24.27	25.24	21.68	59.9	61.5
PE-Core-L/14-336	671M	99.07	49.44	17.17	32.22	21.93	60.5	61.8
PE-Core-G/14-448	2419M	99.20	74.34	29.86	54.70	35.06	54.7	54.5

Appendix

Table 9: MicroCLIP performance across backbone architectures and model sizes. Classification is zero-shot Top-1 accuracy (%), and retrieval is Recall@1 (%) for image-to-text (I2T) and text-to-image (T2I) retrieval. Best and second-best results are shown in bold and underline , respectively.

Method	Sampling ratio			In-domain classification	Out-of-domain classification				Retrieval
Method	Cat.	Det.	Know.	IID	Compositional Shift	Cell-Line Shift	Organelle-Set Shift	Source Shift	I2T	T2I
Category-level only	1.00	0.00	0.00	97.37	56.47	23.25	40.74	37.31	15.80	12.60
Detail-level only	0.00	1.00	0.00	56.53	26.53	10.56	26.72	10.28	48.10	48.50
Knowledge-level only	0.00	0.00	1.00	39.95	12.08	14.88	12.00	30.31	1.30	0.10
Category $+$ Detail	0.75	0.25	0.00	98.90	63.94	28.69	41.63	40.59	28.40	27.80
Category $+$ Knowledge	0.75	0.00	0.25	97.47	63.17	29.31	46.78	27.72	14.90	9.90

Appendix

Table 10: Effect of caption sampling ratios. We compare single-level and mixed supervision on classification and cross-modal retrieval.

Model	Top-1 accuracy (%)
CLIP (OpenAI) ViT-B/16	12.50
SigLIP ViT-B/16	22.50
SigLIP2 ViT-B/16	18.75
DFN2B-CLIP ViT-B/16	17.50
PE-Core B/16	41.88
MicroCLIP (Ours)	88.12

Appendix

Table 11: Zero-shot classification of five subcellular structures on 160 Allen WTC-11 volumes using maximum-intensity projections. All models receive the same 2D projections; chance accuracy is 20%.

Figure 12: 3D cell classification examples. Each row shows five axial slices, the maximum-intensity projection used as model input, and predictions from six models. Green and red indicate correct and incorrect predictions, respectively. All models misclassify the displayed microtubule example as actin.

Figure 13: IID classification examples comparing MicroCLIP and baseline predictions with ground-truth labels on new images of structure–cell-line pairs observed during training.

Figure 14: Compositional Shift examples comparing model predictions on held-out structure–cell-line pairs whose individual constituents are represented in training.

Figure 15: Cell-Line Shift examples comparing model predictions on cell lines absent from training, with structure labels represented in training.

Figure 16: Organelle-Set Shift examples comparing model predictions on complete structure labels absent from training, with cell lines represented in training.

Figure 17: Source Shift examples comparing MicroCLIP and baseline predictions with ground-truth labels on images from the external Light My Cells dataset.

Figure 18: Image-to-text retrieval examples: microscopy image queries and retrieved descriptions.

Figure 19: Text-to-image retrieval examples: textual queries and retrieved microscopy images.

Explore similar work

Jul 21, 2026

scMIR: a vision-language foundation model for single-cell light microscopy image representation

Single-cell light microscopy images have become an important data source for characterizing cell phenotypes, but their complexity and heterogeneity pose challenges to high-throughput automated analysis. Existing representation learning methods mostly rely on task-oriented modeling, which is limited by specific datasets and predefined tasks, making them difficult to generalize across different cell types and microscopy modalities, and experimental conditions. Although general-purpose methods have improved the generalization ability of image representation in recent years, their limited utilization of experimental background and biological context information still poses challenges in complex phenotypic analysis. Here, we propose scMIR, a vision-language foundation model for single-cell light microscopy image representation. By synergistically combining self-supervised image reconstruction with text-guided cross-modal alignment, scMIR can simultaneously encode morphological and biological semantic information in a unified representation space. scMIR is pre-trained on 207,957 image-text pairs, covering various cell types, microscopy modalities, and perturbation conditions. scMIR outperforms existing general models and task-oriented methods as systematically evaluated on various complex tasks using 16 benchmark datasets, including cell classification, clustering, phenotype inference, and batch effect correction tasks. Furthermore, scMIR shows a strong generalization ability across various tasks without requiring task-specific fine-tuning. With its unique advantages, we envision scMIR may promote the standardization and automation of high-throughput phenotyping workflows through supporting various downstream analysis tasks.

Yifan Shang, Jiahui Tan, Xiangxiang Zeng +1

Laser Metrology and Biomedicine Lab, Department of Biomedical Engineering, The Chinese University of Hong Kong, Hong Kong, China · State Key Laboratory of Chemo and Biosensing, College of Computer Science and Electronic Engineering, Hunan University, Changsha 410082, China

Cross-Modal Representation Learning Visual Representation Learning Multimodal Pretraining Self-Supervised Visual Representation Learning Microscopy Image Analysis

Ask PDF

Aug 24, 2026

MorphoCLIP: Text-Supervised Contrastive Learning for Perturbation Matching in Cell Painting Images

Cell Painting microscopy captures how cells change after a chemical or genetic perturbation. Connecting these images to the perturbations that produced them could make large imaging screens easier to search and interpret, but the task remains difficult because biological effects are subtle and technical variation is substantial. We introduce MorphoCLIP, a contrastive model that links Cell Painting profiles with text descriptions of compounds, CRISPR knockouts, and ORF overexpressions. The model keeps its vision and language backbones frozen and trains only a compact cross-channel module and projection layers, so it can be trained on a single consumer GPU. On held-out CPJUMP1 data, MorphoCLIP searches in both directions: from a cell image to its perturbation description and from a description to matching cell images. In both cases, a correct match appears among the top ten results much more often than expected by chance. Adding a replicate-alignment loss makes profiles from repeated experiments more consistent, although this improvement does not yet translate into reliable gene-compound matching. Gene-aware labels and plate correction also show no consistent retrieval benefit. These findings suggest that text supervision can help organize chemical and genetic Cell Painting data. Matching compounds with genetic perturbations, however, remains an open problem.

Sukhrobbek Ilyosbekov, Shubham Gajjar, Rongfei Jin

Northeastern University · Northeastern University Portland, ME, USA

Image-Text Retrieval Microscopy Image Analysis Multimodal Contrastive Learning

Ask PDF

May 19, 2026

A Human-in-the-Loop Framework for Efficient Prompt Selection in Microscopy Vision-Language Models

Deep-learning pipelines for microscopy image classification often require expensive, labor- and time-intensive expert annotation to produce high-quality ground truth for training. Recent work has shown that prompt tuning of vision-language models (VLMs) can reduce manual annotation by constructing a small prompt set of expert-verified image-caption exemplars that is reused as few-shot context to classify all remaining images at inference time. To further reduce effort, the VLM can draft captions for candidate exemplars, which experts then verify and lightly edit instead of writing text de novo. However, two practical questions remain unaddressed: (1) which unlabeled images should be prioritized for verification, and (2) how many verified exemplars are needed to reach a performance target. In this work, we address these questions by formulating prompt-set construction as a target-driven active learning problem that prioritizes which images to annotate. We study three complementary selection criteria under strict low-resource constraints with small unlabeled pools. Experiments show that our methods reach the target performance with substantially fewer expert-verified images than random selection, achieving 100% test accuracy with as few as 20 annotated images on average. More broadly, our human-in-the-loop framework demonstrates a human-centered use of generative AI in biomedical image analysis, where experts remain actively involved in verifying and refining model output while significantly reducing annotation cost. Code and data will be publicly available.

Abhiram Kandiyana, Ankur Mali, Lawrence O. Hall +2

University of South Florida, USA · SRC Biosciences, USA

Vision-Language Models Human-in-the-Loop Annotation Few-Shot Image Classification Microscopy Image Analysis

Ask PDF