MicroCLIP: Connecting Text and Microscopy Images via Contrastive Language--Image Pretraining

Yanran Ruan; Zhixiang Wei; Shiyao Hong; Xiaoxiao Ma; Yuxuan Gu; Liming Ye; Huaian Chen; Yi Jin

cs.CVOct 13, 2026

MicroCLIP: Connecting Text and Microscopy Images via Contrastive Language--Image Pretraining

Authors: Yanran Ruan, Zhixiang Wei, Shiyao Hong, Xiaoxiao Ma, Yuxuan Gu, Liming Ye, Huaian Chen, Yi Jin

Organizations: University of Science and Technology of China

Abstract

Understanding fluorescence microscopy images is fundamental to automating biological research. However, existing methods typically learn predefined concepts from closed-set data, limiting transfer across tasks and biological contexts. In this work, we introduce MicroCLIP, a vision-language model that learns generalizable cellular and subcellular representations from natural language supervision for open-set image classification and retrieval. To pretrain MicroCLIP, we construct MicroPool-2M, a dataset of approximately 2.5 million fluorescence microscopy image-text pairs spanning diverse cellular structures, cell lines, and imaging conditions. Aligning microscopy images with biological concepts in natural language is challenging because most images lack descriptive captions and biological knowledge annotations. We therefore progressively establish image-text associations at three levels to provide MicroCLIP with richer semantic supervision: (1) category-level alignment links images to standardized labels defined by subcellular structure-cell-line combinations; (2) knowledge-level alignment incorporates biological priors derived from these labels to capture relationships across species and cell types; and (3) detail-level alignment uses a multimodal large language model to recaption and verify image-text pairs at scale, emphasizing fine-grained visual characteristics. Extensive experiments demonstrate that MicroCLIP achieves leading performance in zero-shot classification and text-image retrieval on microscopy images. It retrieves images with specific cellular characteristics from thousands of candidates and generalizes to unseen biological concepts without additional supervision. These results highlight MicroCLIP's potential as a foundation model for fluorescence microscopy.

Figures & tables

Figure 1: (1) We connect fluorescence microscopy images with text, so that subcellular structures and cell lines can both be represented in a shared semantic space rather than as isolated class IDs. (2) Building on this shared semantic space, MicroCLIP generalizes to unseen structure–cell-line pairs, whereas conventional closed-set methods are restricted to seen pairs. (3) Beyond recognition, this shared representation further enables language-based retrieval, allowing users to search microscopy images through natural language descriptions.

Figure 2: Illustration of the biological grounding gap. (a–c) GPT-6 responses to biological knowledge, visual description, and biological identification queries, respectively. The predicted identity in (c) differs from the reference annotation. (d) Joint structure–cell-line classification accuracy of DFN2B-CLIP, GPT-6, and MicroCLIP on diagnostic subsets of IID and Compositional Shift, with 10 candidate categories and 30 images per split.

Figure 3: MicroCLIP learns a shared image–text representation from category-level, knowledge-level and detail-level supervision. Each microscopy image is paired with one sampled caption for CLIP training. Composition balancing and rare-class augmentation adjust training exposure, while general-domain image–text pairs provide auxiliary supervision.

Figure 4: Classification evaluation subsets: (a) representative images displayed using pseudocolor; (b) category and image counts for the four internal splits, the external Source Shift benchmark, and the training set.

Method	Params.	In-domain classification	Out-of-domain classification				Retrieval
Method	Params.	IID	Compositional Shift	Cell-Line Shift	Organelle-Set Shift	Source Shift	I2T	T2I
Closed-set methods
Fully Supervised Classifier	–	97.74	–	–	–	–	–	–
Cell-DINO	304M	36.06	0.98	–	–	–	–	–
Open-set methods
CLIP (OpenAI)	150M	2.97	5.12	1.80	2.54	8.38	0.7	0.2

Table 1: Comparison of classification and cross-modal retrieval performance. Classification results are Top-1 accuracy (%), retrieval results are Recall@1 (%). Best and second-best results are shown in bold and underlined, respectively.

Method	Cat.	Det.	Know.	Bal.	Aux.	In-domain classification	Out-of-domain classification				Retrieval
Method	Cat.	Det.	Know.	Bal.	Aux.	IID	Compositional Shift	Cell-Line Shift	Organelle-Set Shift	Source Shift	I2T	T2I
Category-level only	✓					97.37	56.47	23.25	40.74	37.31	15.8	12.6
+ Detail-level captions	✓	✓				98.90	63.94	28.69	41.63	40.59	28.4	27.8
+ Knowledge-level captions	✓	✓	✓			98.86	66.36	25.41	48.31	36.36	37.1	37.4
+ Data Balancing	✓	✓	✓	✓		98.24	64.95	31.53	49.91	36.44	39.4	41.5
+ VLM-0.7M	✓	✓	✓	✓	✓	98.46	67.70	34.07	46.13	36.53	61.5	63.3

Table 2: MicroCLIP component ablations. Cat./Det./Know.: category/detail/knowledge captions. Bal.: composition balancing. Aux.: general-domain image–text pairs from VLM-0.7M.

Balancing threshold $\tau_{a}$	Retained (%)	In-domain classification	Out-of-domain classification				Retrieval
Balancing threshold $\tau_{a}$	Retained (%)	IID	Compositional Shift	Cell-Line Shift	Organelle-Set Shift	Source Shift	I2T	T2I
off (no balancing)	100.0	98.76	66.64	26.61	46.78	30.05	32.4	35.7
$50{,}000$	49.8	98.83	64.73	28.44	47.35	39.29	47.0	47.6
$20{,}000$	30.7	98.24	65.52	27.88	47.80	34.54	44.5	45.5
$5{,}000$	10.5	98.46	67.70	34.07	46.13	36.53	61.5	63.3

Table 3: Effect of the composition-balancing threshold

\tau_{a}

(Equation 5 ). Retained (%) denotes the online sample acceptance fraction. Metrics and highlighting follow Table 1 .

Training data	Samples (M)	In-domain classification	Out-of-domain classification				Retrieval
Training data	Samples (M)	IID	Compositional Shift	Cell-Line Shift	Organelle-Set Shift	Source Shift	I2T	T2I
$25\%$	0.63	98.76	60.39	33.05	44.56	40.93	32.1	32.0
$50\%$	1.25	98.49	65.05	30.44	45.19	39.81	32.4	30.8
$100\%$	2.51	98.54	66.97	30.05	45.54	37.31	60.1	60.7
$100\%$ $+$ VLM-0.7M	3.11	98.46	67.70	34.07	46.13	36.53	61.5	63.3

Table 4: Training-data scale and the full recipe. Samples (M) counts images before augmentation and balancing. Metrics and highlighting follow Table 1 .

Figure 5: Qualitative comparison of (a) image-to-text and (b) text-to-image retrieval.

Figure 6: Joint image–text embeddings visualized with 3D PCA. (a) DFN2B-CLIP and (b) MicroCLIP. Each pair of views uses the same projection, highlighting images or text by category and showing the other modality in gray. Colors identify categories consistently across panels; marker shapes identify images and text levels. DFN2B-CLIP shows category text only, while MicroCLIP includes all three text levels.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Publication	Subcellular Structure	Cell Line	Image Size	Data Number
BBBC026	Logan et al. Logan et al. (2016)	DNA	Human Hepatocyte and Murine Fibroblast cells	1040 $\times$ 1392	864
BBBC001	Moffat et al. Moffat et al. (2006)	DNA	HT-29	512 $\times$ 512	64512
		Actin	HT-29	512 $\times$ 512	64512
		pH3	HT-29	512 $\times$ 512	64512
BBBC020	Wenzel et al. Wenzel et al. (2011)	CD11b	Bone Marrow-Derived Macrophages from C57BL/6 mice	1388 $\times$ 3 $\times$ 1040	25
		Nuclei	Bone Marrow-Derived Macrophages from C57BL/6 mice	1388 $\times$ 3 $\times$ 1040	25

Appendix

Table 5: Source datasets for MicroPool-2M. The table lists each dataset, source publication, biological target, cell line or specimen, image dimensions, and record count. Values describe the collected source data before MicroPool-specific extraction, filtering, harmonization, and deduplication. For datasets summarized in a single row, the data number is the sum over the listed structure–cell-line entries.

Subset	Classes	Struct.	Cells	Images	Shift
IID	59	21	12	5,900	none
Compositional Shift	64	16	16	6,400	pair
Cell-Line Shift	59	22	8	5,900	cell line
Organelle-Set Shift	54	29	6	5,400	structure label
Source Shift (external)	13	3	5	1,158	source

Appendix

Table 6: Classification evaluation data. Structure counts refer to complete structure labels, including composite labels. Each internal evaluation category contains 100 images. Source Shift is an external benchmark and is not balanced to 100 images per category.

Figure 7: Training category frequencies, ranked by image count. Dashed lines mark the augmentation and retention thresholds.

Figure 8: Prompt for generating image-independent biological knowledge about subcellular structures and markers with GPT.

Figure 9: Prompt for generating image-independent biological knowledge about cell lines with GPT.

Figure 10: Prompt for generating detail-level captions with Qwen from a microscopy image and its category annotation.

Figure 11: Three-level supervision for one fluorescence microscopy image. The category caption identifies the annotated structure and cell line; the knowledge caption describes image-independent biological properties of these entities; the detail caption describes the individual image.

Caption level	Median	P95	Truncated at 77 (%)	Truncated at 256 (%)
Category	15	19	0.00	0.00
Detail	225	270	99.83	12.97
Knowledge	109	147	100.00	$<0.01$

Appendix

Table 7: Token-length statistics for the three caption levels.

Model	IID	Compositional Shift	Overall
GPT-6	12/30 (40.00%)	10/30 (33.33%)	22/60 (36.67%)
DFN2B-CLIP	6/30 (20.00%)	4/30 (13.33%)	10/60 (16.67%)
MicroCLIP	30/30 (100.00%)	27/30 (90.00%)	57/60 (95.00%)

Appendix

Table 8: Sampled biological grounding diagnostic. Each split contains 10 categories with three images per category. Entries give correct predictions / images and joint Top-1 accuracy. MicroCLIP is evaluated after 15 training epochs.

Backbone	Params.	In-domain classification	Out-of-domain classification				Retrieval
Backbone	Params.	IID	Compositional Shift	Cell-Line Shift	Organelle-Set Shift	Source Shift	I2T	T2I
PE-Core-B/16	448M	98.80	40.94	24.27	25.24	21.68	59.9	61.5
PE-Core-L/14-336	671M	99.07	49.44	17.17	32.22	21.93	60.5	61.8
PE-Core-G/14-448	2419M	99.20	74.34	29.86	54.70	35.06	54.7	54.5

Appendix

Table 9: MicroCLIP performance across backbone architectures and model sizes. Classification is zero-shot Top-1 accuracy (%), and retrieval is Recall@1 (%) for image-to-text (I2T) and text-to-image (T2I) retrieval. Best and second-best results are shown in bold and underline , respectively.

Method	Sampling ratio			In-domain classification	Out-of-domain classification				Retrieval
Method	Cat.	Det.	Know.	IID	Compositional Shift	Cell-Line Shift	Organelle-Set Shift	Source Shift	I2T	T2I
Category-level only	1.00	0.00	0.00	97.37	56.47	23.25	40.74	37.31	15.80	12.60
Detail-level only	0.00	1.00	0.00	56.53	26.53	10.56	26.72	10.28	48.10	48.50
Knowledge-level only	0.00	0.00	1.00	39.95	12.08	14.88	12.00	30.31	1.30	0.10
Category $+$ Detail	0.75	0.25	0.00	98.90	63.94	28.69	41.63	40.59	28.40	27.80
Category $+$ Knowledge	0.75	0.00	0.25	97.47	63.17	29.31	46.78	27.72	14.90	9.90

Appendix

Table 10: Effect of caption sampling ratios. We compare single-level and mixed supervision on classification and cross-modal retrieval.

Model	Top-1 accuracy (%)
CLIP (OpenAI) ViT-B/16	12.50
SigLIP ViT-B/16	22.50
SigLIP2 ViT-B/16	18.75
DFN2B-CLIP ViT-B/16	17.50
PE-Core B/16	41.88
MicroCLIP (Ours)	88.12

Appendix

Table 11: Zero-shot classification of five subcellular structures on 160 Allen WTC-11 volumes using maximum-intensity projections. All models receive the same 2D projections; chance accuracy is 20%.

Figure 12: 3D cell classification examples. Each row shows five axial slices, the maximum-intensity projection used as model input, and predictions from six models. Green and red indicate correct and incorrect predictions, respectively. All models misclassify the displayed microtubule example as actin.

Figure 13: IID classification examples comparing MicroCLIP and baseline predictions with ground-truth labels on new images of structure–cell-line pairs observed during training.

Figure 14: Compositional Shift examples comparing model predictions on held-out structure–cell-line pairs whose individual constituents are represented in training.

Figure 15: Cell-Line Shift examples comparing model predictions on cell lines absent from training, with structure labels represented in training.

Figure 16: Organelle-Set Shift examples comparing model predictions on complete structure labels absent from training, with cell lines represented in training.

Figure 17: Source Shift examples comparing MicroCLIP and baseline predictions with ground-truth labels on images from the external Light My Cells dataset.

Figure 18: Image-to-text retrieval examples: microscopy image queries and retrieved descriptions.

Figure 19: Text-to-image retrieval examples: textual queries and retrieved microscopy images.

Explore similar work

scMIR: a vision-language foundation model for single-cell light microscopy image representation
Jul 21, 2026Yifan Shang, Jiahui Tan, Xiangxiang Zeng +1Cross-Modal Representation Learning Visual Representation Learning
Laser Metrology and Biomedicine Lab, Department of Biomedical Engineering, The Chinese University of Hong Kong, Hong Kong, China · State Key Laboratory of Chemo and Biosensing, College of Computer Science and Electronic Engineering, Hunan University, Changsha 410082, China
MorphoCLIP: Text-Supervised Contrastive Learning for Perturbation Matching in Cell Painting Images
Aug 24, 2026Sukhrobbek Ilyosbekov, Shubham Gajjar, Rongfei JinImage-Text Retrieval Microscopy Image Analysis
Northeastern University · Northeastern University Portland, ME, USA
A Human-in-the-Loop Framework for Efficient Prompt Selection in Microscopy Vision-Language Models
May 19, 2026Abhiram Kandiyana, Ankur Mali, Lawrence O. Hall +2Vision-Language Models Human-in-the-Loop Annotation
University of South Florida, USA · SRC Biosciences, USA

Jul 21, 2026J/K move · Enter open · S save

scMIR: a vision-language foundation model for single-cell light microscopy image representation

Yifan Shang, Jiahui Tan, Xiangxiang Zeng, Renjie Zhou

Laser Metrology and Biomedicine Lab, Department of Biomedical Engineering, The Chinese University of Hong Kong, Hong Kong, China · State Key Laboratory of Chemo and Biosensing, College of Computer Science and Electronic Engineering, Hunan University, Changsha 410082, China

Cross-Modal Representation Learning Visual Representation Learning Multimodal Pretraining Self-Supervised Visual Representation Learning Microscopy Image Analysis

Open paper AskPDF arXiv

Single-cell light microscopy images have become an important data source for characterizing cell phenotypes, but their complexity and heterogeneity pose challenges to high-throughput automated analysis. Existing representation learning methods mostly rely on task-oriented modeling, which is limited by specific datasets and predefined tasks, making them difficult to generalize across different cell types and microscopy modalities, and experimental conditions. Although general-purpose methods have improved the generalization ability of image representation in recent years, their limited utilization of experimental background and biological context information still poses challenges in complex phenotypic analysis. Here, we propose scMIR, a vision-language foundation model for single-cell light microscopy image representation. By synergistically combining self-supervised image reconstruction with text-guided cross-modal alignment, scMIR can simultaneously encode morphological and biological semantic information in a unified representation space. scMIR is pre-trained on 207,957 image-text pairs, covering various cell types, microscopy modalities, and perturbation conditions. scMIR outperforms existing general models and task-oriented methods as systematically evaluated on various complex tasks using 16 benchmark datasets, including cell classification, clustering, phenotype inference, and batch effect correction tasks. Furthermore, scMIR shows a strong generalization ability across various tasks without requiring task-specific fine-tuning. With its unique advantages, we envision scMIR may promote the standardization and automation of high-throughput phenotyping workflows through supporting various downstream analysis tasks.

Abstract

Explore similar work

scMIR: a vision-language foundation model for single-cell light microscopy image representation

MorphoCLIP: Text-Supervised Contrastive Learning for Perturbation Matching in Cell Painting Images

A Human-in-the-Loop Framework for Efficient Prompt Selection in Microscopy Vision-Language Models