Cytoarchitecture in Words: Weakly Supervised Vision-Language Modeling for Human Brain Microscopy
Authors: Matthew Sutton, Katrin Amunts, Timo Dickscheid, Christian Schiffer
Organizations: Institute of Neuroscience and Medicine (INM-1), Research Centre Jülich, Jülich, Germany · Helmholtz AI, Research Centre Jülich, Jülich, Germany · C´ecile & Oskar Vogt Institute for Brain Research, University Hospital Düsseldorf, Düsseldorf, Germany · Computer Vision, Institute for Computational Visualistics, University of Koblenz, Koblenz, Germany
Vision foundation models increasingly support interactive scientific workflows, but natural-language interaction requires coupling visual representations to language. Curated image-text pairs for this coupling are scarce in many biomedical domains. This is the case for cell-body-stained histological sections of the human brain, where microscopic image patches encode cytoarchitecture: cellular density, morphology, laminar structure, and areal organization. We propose retrieve-and-enrich supervision, a weakly supervised scheme for training image-conditioned language models without curated image-text pairs. The method retrieves label-level text from the literature via a shared anatomical label, then enriches it with image-specific properties such as cortical layer thickness and cell density. Labels provide training targets only, not model inputs. We use this scheme to couple CytoNet, a cytoarchitectonic vision foundation model, to an open-weight large language model via a lightweight Flamingo-style adapter. Across 57 brain areas, the resulting model produces plausible cytoarchitectonic descriptions, supports open-set use by rejecting out-of-scope areas, and predicts the correct area for in-scope patches with 90.6% accuracy. Removing explicit area names from generated text still leaves descriptions sufficient for a language model to recover the area in an 8-way test with 68.6% accuracy. A second instruction-tuned model trained with image-specific targets recovers cortical layer thickness and cell-densities from individual patches, allowing the model to read out information beyond canonical area descriptions. Our results show that retrieve-and-enrich supervision offers a practical route to vision-language training in specialized imaging domains where labels and expert text exist but image-level captions do not.
Figures & tables
Figure 1: Architecture overview. Image-conditioned text generation is based on Flamingo [ 2 ] . Condition images are embedded by the frozen vision model, CytoNet [ 30 ] . Embeddings are linearly projected into image tokens. A gated cross-attention block [ 2 ] after every 4th encoder block of the large langauge model conditions text generation.
Figure 2: Retrieve-and-enrich supervision. Starting from unlabeled images (e.g., microscopic image patches), a label classifier assigns a label to each image patch. In the retrieve stage, this label indexes literature-derived statement pools for the corresponding label, and an LLM converts sampled statements into label-level descriptions. In the enrich stage, a property regressor provides image-specific measurements (e.g., cortical layer thickness and cell density), which are verbalized by an LLM as image-specific description. The final synthetic description combines label-level literature supervision with patch-specific properties.
Model
# Parameters
Medical fine-tuning?
Score
First author performance
n.a.
n.a.
32.0%
BioGPT [ 28 ]
0.35b
✓
24.1%
BioGPT-Large [ 28 ]
1.5b
✓
26.3%
Qwen3 4b [ 33 ]
4b
✗
54.1%
PMC Llama 7b [ 32 ]
7b
✓
31.8%
Llama-3-8b [ 20 ]
8b
✗
61.7%
Table 1: Performance of LLMs on the created QA dataset for cytoarchiteture. Scores are the share of correctly answered multiple choice questions, out of a set of 1,735. For comparison, the first author answered a random subset of 50 questions.
Figure 3: Examples for generated descriptions of given microscopic image patches. The text indicates the respective brain areas, as well as descriptions of area-specific characteristics.
Figure 4: Predicted cortical layer thicknesses and densities against ground truths for all considered areas (top) and area hOc5 (bottom). The model can accurately predict cortical layer thicknesses and cell densities, with individual outliers in the thicknesses of layers II, III and IV. The results for area hOc5 demonstrate that the model is able to accurately capture intra-area variations.
Figure 5: Comparison of outputs to the same prompt from the base language model (Qwen3-4b) and from our trained vision–language model. Qwen3-4b [ 33 ] was provided with a template of information replicating what was supplied to the model during generation of the synthetic training data. Our model was shown only the associated image patch and given no further information. On tasks involving reasoning about other areas of the brain, our trained model shows more certainty and precision, demonstrating a broader understanding of cytoarchitecture. The base model made a substantial error when discussing the motor cortex, highlighted in red.
The functions of different regions of the human brain are closely linked to their distinct cytoarchitecture, which is defined by the spatial arrangement and morphology of the cells. Identifying brain regions by their cytoarchitecture enables various scientific analyses of the brain. However, delineating these areas manually in brain histological sections is time-consuming and requires specialized knowledge. An automated approach is necessary to minimize the effort needed from human experts. To address this, we propose CytoCLIP, a suite of vision-language models derived from pre-trained Contrastive Language-Image Pre-Training (CLIP) frameworks to learn joint visual-text representations of brain cytoarchitecture. CytoCLIP comprises two model variants: one is trained using low-resolution whole-region images to understand the overall cytoarchitectural pattern of an area, and the other is trained on high-resolution image tiles for detailed cellular-level representation. The training dataset is created from NISSL-stained histological sections of developing fetal brains of different gestational weeks. It includes 86 distinct regions for low-resolution images and 379 brain regions for high-resolution tiles. We evaluate the model's understanding of the cytoarchitecture and generalization ability using region classification and cross-modal retrieval tasks. Multiple experiments are performed under various data setups, including data from samples of different ages and sectioning planes. Experimental results demonstrate that CytoCLIP outperforms existing methods. It achieves a weighted F1 score of 0.87 for whole-region classification and 0.91 for high-resolution image tile classification.
Pralaypati Ta, Sriram Venkatesaperumal, Keerthi Ram +1
Sudha Gopalakrishnan Brain Centre, Indian Institute of Technology, Madras, Chennai, India. · Department of Electrical Engineering, Indian Institute of Technology, Madras, Chennai, India.
Vision-language models (VLMs) achieve strong zero-shot (ZS) classification on histology images but do not perform as well on cytology, whose stains and cell morphology differ markedly compared to histology. Conditional random fields (CRFs) can refine noisy VLM predictions by propagating information across patches, but existing CRF frameworks were designed for histopathology and do not transfer to cytology datasets, released as independent patch pools spanning multiple staining protocols. We introduce CytoCRF, which adapts the pairwise terms to cytology by targeting chromatin and cytology-specific staining, and further enrich the neighborhood of each potential term by combining multiple backbones. Across ten cytology datasets, CytoCRF outperforms existing CRF frameworks at every annotation budget, reaching +13.6 percentage points over the best baseline and +33.7 over ZS with only 50 annotations. Combining information from multiple backbones brings further gains, showing that the neighborhood topology matters more than the pairwise potential computed over it.
Manon Dausort, Tiffanie Godelaine, Karim El Khoury +3
ICTEAM, Universit´e Catholique de Louvain, Belgium
Understanding fluorescence microscopy images is fundamental to automating biological research. However, existing methods typically learn predefined concepts from closed-set data, limiting transfer across tasks and biological contexts. In this work, we introduce MicroCLIP, a vision-language model that learns generalizable cellular and subcellular representations from natural language supervision for open-set image classification and retrieval. To pretrain MicroCLIP, we construct MicroPool-2M, a dataset of approximately 2.5 million fluorescence microscopy image-text pairs spanning diverse cellular structures, cell lines, and imaging conditions. Aligning microscopy images with biological concepts in natural language is challenging because most images lack descriptive captions and biological knowledge annotations. We therefore progressively establish image-text associations at three levels to provide MicroCLIP with richer semantic supervision: (1) category-level alignment links images to standardized labels defined by subcellular structure-cell-line combinations; (2) knowledge-level alignment incorporates biological priors derived from these labels to capture relationships across species and cell types; and (3) detail-level alignment uses a multimodal large language model to recaption and verify image-text pairs at scale, emphasizing fine-grained visual characteristics. Extensive experiments demonstrate that MicroCLIP achieves leading performance in zero-shot classification and text-image retrieval on microscopy images. It retrieves images with specific cellular characteristics from thousands of candidates and generalizes to unseen biological concepts without additional supervision. These results highlight MicroCLIP's potential as a foundation model for fluorescence microscopy.