cs.CVApr 11, 2026

SIMPLER: H&E-Informed Representation Learning for Structured Illumination Microscopy

Authors: Abu Zahid Bin Aziz, Syed Fahim Ahmed, Gnanesh Rasineni, Mei Wang, Olcaytu Hatipoglu, Marisa Ricci, Malaiyah Shaw, Guang Li, 

Organizations: Kahlert School of Computing, University of Utah, Salt Lake City, UT 84112, USA · Scientific Computing & Imaging Institute, University of Utah, Salt Lake City, UT 84112, USA · Instapath Inc., Houston, TX 77021 · Tulane University, Department of Biomedical Engineering, New Orleans, Louisiana, United States

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Structured Illumination Microscopy (SIM) enables rapid, high-contrast optical sectioning of fresh tissue without staining or physical sectioning, making it promising for intraoperative and point-of-care diagnostics. Recent foundation and large-scale self-supervised models in digital pathology have demonstrated strong performance on section-based modalities such as Hematoxylin and Eosin (H&E) and immunohistochemistry (IHC). However, these approaches are predominantly trained on thin tissue sections and do not explicitly address thick-tissue fluorescence modalities such as SIM. When transferred directly to SIM, performance is constrained by substantial modality shift, and naive fine-tuning often overfits to modality-specific appearance rather than underlying histological structure. We introduce SIMPLER (Structured Illumination Microscopy-Powered Learning for Embedding Representations), a cross-modality self-supervised pretraining framework that leverages H&E as a semantic anchor to learn reusable SIM representations. H&E encodes rich cellular and glandular structure aligned with established clinical annotations, while SIM provides rapid, nondestructive imaging of fresh tissue. During pretraining, SIM and H&E are progressively aligned through adversarial, contrastive, and reconstruction-based objectives, encouraging SIM embeddings to internalize histological structure from H&E without collapsing modality-specific characteristics. A single pretrained SIMPLER encoder transfers across multiple downstream tasks, including multiple instance learning and morphological clustering, consistently outperforming SIM models trained from scratch or H&E-only pretraining. These results suggest that histology-guided cross-modal pretraining yields biologically grounded SIM embeddings suitable for broad downstream reuse.

Figures & tables

Figure 1: UMAP of feature embeddings. Each stage progressively aligns H&E and SIM features, reducing modality gap and enhancing shared representation.
Figure 2: SIMPLER Overview. The proposed progressive cross-modality curriculum used to align H&E and SIM representations. Each stage introduces increasingly structured alignment constraints, from feature robustness to structural cross-reconstruction.
Table 3
Figure 3: Results. (A) Confusion matrices for H&E and SIM domains comparing UNI and SIMPLER. (B) Qualitative heatmap comparison of attention scores.
Figure 4: Clustering results. (A) Representative patches from two clusters identified by SIMPLER show consistent structure. (B) Cluster maps using UNI (middle) and SIMPLER (bottom). SIMPLER achieves strong cross-modal consistency
More in this topic

Medical Image Representation Learning

Medical image representation learning builds reusable visual features for diagnosis, segmentation, retrieval, and other clinical imaging tasks.

See all 247 papers

Explore similar work

CardsList
Apr 26, 2026

VitaminP: cross-modal learning enables whole-cell segmentation from routine histology

Accurate whole-cell and nuclear segmentation is essential for precision pathology and spatial omics, yet routine hematoxylin and eosin (H&E) staining provides limited cytoplasmic contrast, restricting analyses to nuclei. Multiplex immunofluorescence (mIF) facilitates precise whole-cell delineation but remains constrained by cost and accessibility. We introduce VitaminP, a cross-modal learning framework enabling whole cell segmentation from H&E images. By learning from paired H&E-mIF data, VitaminP transfers molecular boundary information from mIF to overcome cytoplasmic contrast in H&E, establishing cross-modal supervision as a general strategy for recovering missing biological structure. We train VitaminP on 14 public datasets covering 34 cancer types and over 7 million instances, integrating publicly available labels with extensive annotations generated in this study, forming one of the largest resources for segmentation. VitaminP outperforms four state-of-the-art methods and generalizes to unseen datasets, including an in-house dataset spanning 24 rare cancer types. We further developed VitaminPScope, an open-source platform providing an interface for scalable inference and enabling broad adoption.

Yasin Shokrollahi, Karina B. Pinao Gonzales, Elizve N. Barrientos Toro +5

Department of Translational Molecular Pathology, Division of Pathology and Laboratory Medicine, The University of Texas MD Anderson Cancer Center, Houston, TX, USA. · Institute for Data Science in Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA. · The University of Texas MD Anderson Cancer Center, Houston, TX, USA.

Cross-Modal LearningComputational PathologyCross-Modality Medical Image SegmentationImage SegmentationHistopathology Image SegmentationMultimodal Learning
AskPDF
May 12, 2026

Bridging the Modality Bottleneck in Pathology MIL through Virtual Molecular Staining

Multiple instance learning (MIL) is the dominant framework for whole-slide image analysis in computational pathology, typically combining a frozen patch encoder, a projection layer, and a slide-level aggregator. While encoders and aggregators have been extensively studied, the projection layer remains a largely morphology-only bottleneck. This limits endpoints such as biomarker status and survival, which are governed by a molecular state that is not fully captured by H&E morphology. We introduce Molecularly Informed Staining Transform (MIST), a plug-in replacement for the MIL projection layer that uses paired spatial transcriptomics only during training to construct virtual molecular stains. MIST clusters gene expression profiles into cross-modal prototypes, anchors them in the frozen foundation model feature space, and uses them to reorganize H&E patch features along molecularly guided axes. It requires no transcriptomics at inference and can be inserted before standard MIL aggregators. We evaluate MIST across 23 downstream tasks and 8 MIL aggregators. MIST improves 240 of 256 configurations over the standard projection layer, with an average gain of +3.5%, observed consistently across endpoint types: +5.2% on survival prediction, +3.3% on tissue subtyping, and +2.6% on biomarker prediction. Ablations confirm that gene-derived prototypes are the primary source of the gains, while spatial, biological, and pathological analyses show that cross-modal prototype affinities capture spatially coherent molecular programs from H&E alone.

Yucheng Xing, Pei Liu, Jingying Ma +6

National University of Singapore, Singapore · Hunan University, China · Peking Union Medical College Hospital (PUMCH), China +1

Computational PathologyCross-Modal Representation LearningVirtual StainingMultiple Instance LearningWhole-Slide Image Analysis
AskPDF
May 19, 2026

HAPS: Rethinking Image Similarity for Virtual Staining

Virtual staining of histopathology images (e.g., H&E-IHC) is an emerging tool in digital pathology, enabling faster and cheaper workflows by synthesizing target stains from routinely acquired slides. Yet, the quality of virtual staining models is still predominantly assessed with generic metrics such as SSIM, PSNR, and LPIPS. Originally developed for natural images, these metrics are inherently misaligned with the domain-specific characteristics of histological data, failing to capture tissue morphology preservation and biomarker expression patterns. Consequently, a robust, domain-specific standard for quantifying similarity across diverse histological modalities remains a critical gap in the field. In this work, we formalize histology image similarity as a standalone problem and systematically evaluate a broad set of full-reference metrics against a dataset of H&E-IHC patch pairs annotated with expert similarity scores. We further analyze metrics sensitivity to controlled geometric distortions (shifts, rotations and non-rigid deformations) that mimic realistic registration errors between serial sections. Guided by these observations, we propose the Histology-Aware Perceptual Similarity (HAPS) metric. HAPS computes distances in the feature space of a frozen encoder pretrained on histopathology data, adding a linear head to aggregate feature-level differences into a final score that aligns with expert assessments. Finally, we demonstrate the practical value of HAPS for quality control of training data. By quantifying the similarity of training pairs in the MIST dataset and filtering low-scoring samples, we create a cleaner training set. Virtual staining models trained on this refined data outperform those trained on the original, unfiltered dataset.

Fedor Gubanov, Svetlana Illarionova, Vlad Kozlovskiy +6

1*Skolkovo Institute of Science and Technology, Moscow, 121205, Russia. · 2BIMAI-Lab, Biomedically Informed Artificial Intelligence Laboratory,2026 University of Sharjah, Sharjah, United Arab Emirates. · 3National Medical Research Radiological Centre of the Ministry of Health of the Russian Federation, Moscow, Russia.

Computational PathologyMedical Image AnalysisVirtual StainingImage Quality Assessment
AskPDF