Microscopy Image Analysis

Follow this topic Ask about this topic

Microscopy image analysis turns images from biological and materials microscopes into measurements, such as cell counts, tissue structure, particle properties, and cell motion. Common approaches include classical image processing and deep learning for segmentation, detection, classification, tracking, and 2D or 3D reconstruction; newer work also uses self-supervised and foundation models, physics-informed methods, synthetic data, and expert interaction. These tools can make analysis more scalable and reproducible, but performance often depends on microscope settings, sample type, image quality, and the amount of expert annotation available. A central challenge is establishing reliable generalization and evaluation: strong benchmark scores may not ensure robust measurements on new acquisitions, and visual quality metrics may miss errors that affect downstream scientific conclusions.

Latest papers

199 papers15 papers in the last four weeks, down 42% on the four weeks before. 0.1% of all new papers.

Apr 17, 2026

Automated Palynological Analysis System: Integrating Deep Metric Learning, Detection and Classification in Bright Field Microscopy

Traditional melissopalynology is a time-consuming and subjective process, often taking 4-6 hours per sample. We present an automated, high-throughput microscopy system that integrates H_\infty robust mechanical control with advanced deep learning pipelines for the precise counting, classification, and morphological analysis of pollen grains from Bio Bio region in south central territory in Chile. Our system employs U^2-Net for salient object detection and a DINOv2 Vision Transformer backbone trained via Deep Metric Learning for classification. By integrating Gradient-Weighted Attention, the model provides human-interpretable texture and diagnostic feature annotations. The system achieves a 95.8% classification recall and at least 6x processing speedup compared to manual expert analysis.

J. Staforelli-Vivanco, R. Jofré, P. Coelho +12

Facultad de Ingeniería, Universidad Andres Bello, Talcahuano, CCP-THNO 7100 Chile · Universidad Arturo Prat, Sede Victoria, Chile · Laboratory of Palynology and Plant Ecology, School of Sciences and Technologies, University of Concepción, Los Angeles Campus

Salient Object Detection Explainable Image Classification Microscopy Image Analysis Image Classification

Apr 16, 2026

From Boundaries to Semantics: Prompt-Guided Multi-Task Learning for Petrographic Thin-section Segmentation

Grain-edge segmentation (GES) and lithology semantic segmentation (LSS) are two pivotal tasks for quantifying rock fabric and composition. However, these two tasks are often treated separately, and the segmentation quality is implausible albeit expensive, time-consuming, and expert-annotated datasets have been used. Recently, foundation models, especially the Segment Anything Model (SAM), have demonstrated impressive robustness for boundary alignment. However, directly adapting SAM to joint GES and LSS is nontrivial due to 1) severe domain gap induced by extinction-dependent color variations and ultra-fine grain boundaries, and 2) lacking novel modules for joint learning on multi-angle petrographic image stacks. In this paper, we propose Petro-SAM, a novel two-stage, multi-task framework that can achieve high-quality joint GES and LSS on petrographic images. Specifically, based on SAM, we introduce a Merge Block to integrate seven polarized views, effectively solving the extinction issue. Moreover, we introduce multi-scale feature fusion and color-entropy priors to refine the detection.

Yili Ren, Shiqi Wen, Li Hou +8

Research Institute of Petroleum Exploration and Development (RIPED) · The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology +1

Image Segmentation Promptable Image Segmentation Multi-Task Learning Semantic Segmentation Microscopy Image Analysis

Apr 14, 2026

Multitasking Embedding for Embryo Blastocyst Grading Prediction (MEmEBG)

Reliable evaluation of blastocyst quality is critical for the success of in vitro fertilization (IVF) treatments. Current embryo grading practices primarily rely on visual assessment of morphological features, which introduces subjectivity, inter-embryologist variability, and challenges in standardizing quality assurance. In this study, we propose a multitask embedding-based approach for the automated analysis and prediction of key blastocyst components, including the trophectoderm (TE), inner cell mass (ICM), and blastocyst expansion (EXP). The method leverages biological and physical characteristics extracted from images of day-5 human embryos. A pretrained ResNet-18 architecture, enhanced with an embedding layer, is employed to learn discriminative representations from a limited dataset and to automatically identify TE and ICM regions along with their corresponding grades, structures that are visually similar and inherently difficult to distinguish. Experimental results demonstrate the promise of the multitask embedding approach and potential for robust and consistent blastocyst quality assessment.

Nahid Khoshk Angabini, Mohsen Tajgardan, Mahesh Madhavan +3

⋆Department of Computer Science and Media Technology, Malmö University, Malmö, Sweden · ◦Faculty of Electrical and Computer Engineering, Qom University of Technology, Iran · †Kepler Universitätsklinikum, Linz, Austria

Automated Grading Microscopy Image Analysis Multi-Task Learning

Apr 14, 2026

Predicting Blastocyst Formation in IVF: Integrating DINOv2 and Attention-Based LSTM on Time-Lapse Embryo Images

The selection of the optimal embryo for transfer is a critical yet challenging step in in vitro fertilization (IVF), primarily due to its reliance on the manual inspection of extensive time-lapse imaging data. A key obstacle in this process is predicting blastocyst formation from the limited number of daily images available. Many clinics also lack complete time-lapse systems, so full videos are often unavailable. In this study, we aimed to predict which embryos will develop into blastocysts using limited daily images from time-lapse recordings. We propose a novel hybrid model that combines DINOv2, a transformer-based vision model, with an enhanced long short-term memory (LSTM) network featuring a multi-head attention layer. DINOv2 extracts meaningful features from embryo images, and the LSTM model then uses these features to analyze embryo development over time and generate final predictions. We tested our model on a real dataset of 704 embryo videos. The model achieved 96.4% accuracy, surpassing existing methods. It also performs well with missing frames, making it valuable for many IVF laboratories with limited imaging systems. Our approach can assist embryologists in selecting better embryos more efficiently and with greater confidence.

Zahra Asghari Varzaneh, Niclas Wölner-Hanssen, Reza Khoshkangini +2

Sustainable Digitalisation Research Center, Department of Computer Science and Media Technology, Malmö University, Malmö, Sweden · School of Information Technology, Halmstad University, Halmstad, Sweden · Kepler Universitätsklinikum, Linz, Austria +1

Microscopy Image Analysis Recurrent Neural Networks

Apr 13, 2026

Using Deep Learning Models Pretrained by Self-Supervised Learning for Protein Localization

Background: Task-specific microscopy datasets are often small, making it difficult to train deep learning models that learn robust features. While self-supervised learning (SSL) has shown promise through pretraining on large, domain-specific datasets, generalizability across datasets with differing staining protocols and channel configurations remains underexplored. We investigated the generalizability of SSL models pretrained on ImageNet-1k and HPA FOV, evaluating their embeddings on OpenCell with and without fine-tuning, two channel-mismatch strategies, and varying fine-tuning data fractions. We additionally analyzed single-cell embeddings on a labeled OpenCell subset. Result: DINO-based ViT backbones pretrained on HPA FOV or ImageNet-1k transfer well to OpenCell even without fine-tuning. Fine-tuning further improved performance to 0.704

\pm

0.027 (17 classes, multi-class labels). At the single-cell level, the HPA single-cell-pretrained model achieved the highest k-nearest neighbor performance across all neighborhood sizes (macro

F_1

\geq

0.515). Conclusion: SSL methods like DINO, pretrained on large domain-relevant datasets, enable effective use of deep learning features for fine-tuning on small, task-specific microscopy datasets.

Ben Isselmann, Dilara Göksu, Heinz Neumann +1

Department of Mathematics and Natural Sciences, Hochschule Darmstadt, Schoeﬀerstraße 3, Darmstadt, 64295, Hessen, Germany. · Department of Chemical Engineering and Biotechnology, Hochschule Darmstadt, Stephanstrasse 7, Darmstadt, 64295, Hessen, Germany. · European University of Technology, European Union. +1

Supervised Fine-Tuning Self-Supervised Pre-Training Self-Supervised Visual Representation Learning Microscopy Image Analysis

Apr 12, 2026

Self-supervised Pretraining of Cell Segmentation Models

Instance segmentation enables the analysis of spatial and temporal properties of cells in microscopy images by identifying the pixels belonging to each cell. However, progress is constrained by the scarcity of high-quality labeled microscopy datasets. Many recent approaches address this challenge by initializing models with segmentation-pretrained weights from large-scale natural-image models such as Segment Anything Model (SAM). However, representations learned from natural images often encode objectness and texture priors that are poorly aligned with microscopy data, leading to degraded performance under domain shift. We propose DINOCell, a self-supervised framework for cell instance segmentation that leverages representations from DINOv2 and adapts them to microscopy through continued self-supervised training on unlabeled cell images prior to supervised fine-tuning. On the LIVECell benchmark, DINOCell achieves a SEG score of 0.784, improving by 10.42% over leading SAM-based models, and demonstrates strong zero-shot performance on three out-of-distribution microscopy datasets. These results highlight the benefits of domain-adapted self-supervised pretraining for robust cell segmentation.

Kaden Stillwagon, Alexandra Dunnum VandeLoo, Benjamin Magondu +1

School of Computer Science, Georgia Institute of Technology · School of Materials Science and Engineering, Georgia Institute of Technology · Department of Biomedical Engineering, Georgia Institute of Technology +1

Instance Segmentation Microscopy Image Analysis Self-Supervised Pre-Training

Apr 11, 2026

SIMPLER: H&E-Informed Representation Learning for Structured Illumination Microscopy

Structured Illumination Microscopy (SIM) enables rapid, high-contrast optical sectioning of fresh tissue without staining or physical sectioning, making it promising for intraoperative and point-of-care diagnostics. Recent foundation and large-scale self-supervised models in digital pathology have demonstrated strong performance on section-based modalities such as Hematoxylin and Eosin (H&E) and immunohistochemistry (IHC). However, these approaches are predominantly trained on thin tissue sections and do not explicitly address thick-tissue fluorescence modalities such as SIM. When transferred directly to SIM, performance is constrained by substantial modality shift, and naive fine-tuning often overfits to modality-specific appearance rather than underlying histological structure. We introduce SIMPLER (Structured Illumination Microscopy-Powered Learning for Embedding Representations), a cross-modality self-supervised pretraining framework that leverages H&E as a semantic anchor to learn reusable SIM representations. H&E encodes rich cellular and glandular structure aligned with established clinical annotations, while SIM provides rapid, nondestructive imaging of fresh tissue. During pretraining, SIM and H&E are progressively aligned through adversarial, contrastive, and reconstruction-based objectives, encouraging SIM embeddings to internalize histological structure from H&E without collapsing modality-specific characteristics. A single pretrained SIMPLER encoder transfers across multiple downstream tasks, including multiple instance learning and morphological clustering, consistently outperforming SIM models trained from scratch or H&E-only pretraining. These results suggest that histology-guided cross-modal pretraining yields biologically grounded SIM embeddings suitable for broad downstream reuse.

Abu Zahid Bin Aziz, Syed Fahim Ahmed, Gnanesh Rasineni +8

Kahlert School of Computing, University of Utah, Salt Lake City, UT 84112, USA · Scientific Computing & Imaging Institute, University of Utah, Salt Lake City, UT 84112, USA · Instapath Inc., Houston, TX 77021 +1

Medical Image Representation Learning Microscopy Image Analysis Cross-Modal Representation Learning Self-Supervised Visual Representation Learning Multimodal Pretraining Fluorescence Microscopy

Apr 10, 2026

Cluster-First Labelling: An Automated Pipeline for Segmentation and Morphological Clustering in Histology Whole Slide Images

Labelling tissue components in histology whole slide images (WSIs) is prohibitively labour-intensive: a single slide may contain tens of thousands of structures--cells, nuclei, and other morphologically distinct objects--each requiring manual boundary delineation and classification. We present a cloudnative, end-to-end pipeline that automates this process through a cluster-first paradigm. Our system tiles WSIs, filters out tiles deemed unlikely to contain valuable information, segments tissue components with Cellpose-SAM (including cells, nuclei, and other morphologically similar structures), extracts neural embeddings via a pretrained ResNet-50, reduces dimensionality with UMAP, and groups morphologically similar objects using DBSCAN clustering. Under this paradigm, a human annotator labels representative clusters rather than individual objects, reducing annotation effort by orders of magnitude. We evaluate the pipeline on 3,696 tissue components across 13 diverse tissue types from three species (human, rat, rabbit), measuring how well unsupervised clusters align with independent human labels via per-tile Hungarian-algorithm matching. Our system achieves a weighted cluster-label alignment accuracy of 96.8%, with 7 of 13 tissue types reaching perfect agreement. The pipeline, a companion labelling web application, and all evaluation code are released as open-source software.

Muhammad Haseeb Ahmad, Sharmila Rajendran, Damion Young +1

Department of Engineering Science, University of Oxford, United Kingdom · Department of Physiology, Anatomy and Genetics, University of Oxford, United Kingdom · Medical Sciences Division, University of Oxford, United Kingdom

Whole-Slide Image Analysis Microscopy Image Analysis Computational Pathology Histopathology Image Segmentation Image Clustering

Apr 9, 2026

Sampling-Aware 3D Spatial Analysis in Multiplexed Imaging

Highly multiplexed microscopy enables rich spatial characterization of tissues at single-cell resolution, yet most analyses rely on two-dimensional sections despite inherently three-dimensional tissue organization. Acquiring dense volumetric data in spatial proteomics remains costly and technically challenging, leaving practitioners to choose between 2D sections or 3D serial sections under limited imaging budgets. In this work, we study how sampling geometry impacts the stability of commonly used spatial statistics, and we introduce a geometry-aware reconstruction module that enables sparse yet consistent 3D analysis from serial sections. Using controlled simulations, we show that planar sampling reliably recovers global cell-type abundance but exhibits high variance for local statistics such as cell clustering and cell-cell interactions, particularly for rare or spatially localized populations. We observe consistent behavior in real multiplexed datasets, where interaction metrics and neighborhood relationships fluctuate substantially across individual sections. To support sparse 3D analysis in practice, we present a reconstruction approach that links cell projections across adjacent sections using phenotype and proximity constraints and recovers single-cell 3D centroids using cell-type-specific shape priors. We further analyze the trade-off between section spacing, coverage, and redundancy, identifying acquisition regimes that maximize reconstruction utility under fixed imaging budgets. We validate the reconstruction module on a public imaging mass cytometry dataset with dense axial sampling and demonstrate its downstream utility on an in-house CODEX dataset by enabling structure-level 3D analyses that are unreliable in 2D. Together, our results provide diagnostic tools and practical guidance for deciding when 2D sampling suffices and when sparse 3D reconstruction is warranted.

Ido Harlev, Tamar Oukhanov, Raz Ben-Uri +2

Weizmann Institute of Science, Israel

Microscopy Image Analysis 3D Reconstruction Sparse-View 3D Reconstruction

Apr 7, 2026

A Mixture of Experts Foundation Model for Scanning Electron Microscopy Image Analysis

Scanning Electron Microscopy (SEM) is indispensable in modern materials science, enabling high-resolution imaging across a wide range of structural, chemical, and functional investigations. However, SEM imaging remains constrained by task-specific models and labor-intensive acquisition processes that limit its scalability across diverse applications. Here, we introduce the first foundation model for SEM images, pretrained on a large corpus of multi-instrument, multi-condition scientific micrographs, enabling generalization across diverse material systems and imaging conditions. Leveraging a self-supervised transformer architecture, our model learns rich and transferable representations that can be fine-tuned or adapted to a wide range of downstream tasks. As a compelling demonstration, we focus on defocus-to-focus image translation-an essential yet underexplored challenge in automated microscopy pipelines. Our method not only restores focused detail from defocused inputs without paired supervision but also outperforms state-of-the-art techniques across multiple evaluation metrics. This work lays the groundwork for a new class of adaptable SEM models, accelerating materials discovery by bridging foundational representation learning with real-world imaging needs.

Sk Miraj Ahmed, Yuewei Lin, Chuntian Cao +9

Computing and Data Sciences, Brookhaven National Laboratory, Upton, 11973, New York, USA. · Center for Functional Nanomaterials, Brookhaven National Laboratory, Upton, 11973, New York, USA. · Department of Material Sciences and Engineering, The University of Texas at Dallas, Richardson, 75080, Texas, USA.

Microscopy Image Analysis Electron Microscopy Self-Supervised Visual Representation Learning Unpaired Image-to-Image Translation

Apr 6, 2026

3D Gaussian Splatting for Annular Dark Field Scanning Transmission Electron Microscopy Tomography Reconstruction

Analytical Dark Field Scanning Transmission Electron Microscopy (ADF-STEM) tomography reconstructs nanoscale materials in 3D by integrating multi-view tilt-series images, enabling precise analysis of their structural and compositional features. Although integrating more tilt views improves 3D reconstruction, it requires extended electron exposure that risks damaging dose-sensitive materials and introduces drift and misalignment, making it difficult to balance reconstruction fidelity with sample preservation. In practice, sparse-view acquisition is frequently required, yet conventional ADF-STEM methods degrade under limited views, exhibiting artifacts and reduced structural fidelity. To resolve these issues, in this paper, we adapt 3D GS to this domain with three key components. We first model the local scattering strength as a learnable scalar field, denza, to address the mismatch between 3DGS and ADF-STEM imaging physics. Then we introduce a coefficient

γ

to stabilize scattering across tilt angles, ensuring consistent denza via scattering view normalization. Finally, We incorporate a loss function that includes a 2D Fourier amplitude term to suppress missing wedge artifacts in sparse-view reconstruction. Experiments on 45-view and 15-view tilt series show that DenZa-Gaussian produces high-fidelity reconstructions and 2D projections that align more closely with original tilts, demonstrating superior robustness under sparse-view conditions.

Beiyuan Zhang, Hesong Li, Ruiwen Shao +1

Beijing Institute of Technology

Scanning Transmission Electron Microscopy Computed Tomography Sparse-View 3D Reconstruction Microscopy Image Analysis 3D Gaussian Splatting

Apr 4, 2026

SAGE-GAN: Towards Realistic and Robust Segmentation of Spatially Ordered Nanoparticles via Attention-Guided GANs

Precise analysis of nanoparticles for characterization in electron microscopy images is essential for advancing nanomaterial development. Yet it remains challenging due to the time-consuming nature of manual methods and the shortcomings of traditional automated segmentation techniques, especially when dealing with complex shapes and imaging artifacts. While conventional methods yield promising results, they depend on a large volume of labeled training data, which is both difficult to acquire and highly time-consuming to generate. In order to overcome these challenges, we have developed a two-step solution: Firstly, our system learns to segment the key features of nanoparticles from a dataset of real images using a self-attention driven U-Net architecture that focuses on important physical and morphological details while ignoring background features and noise. Secondly, this trained Attention U-Net is embedded in a cycle-consistent generative adversarial network (CycleGAN) framework, inspired by the cGAN-Seg model introduced by Abzargar et al. This integration allows for the creation of highly realistic synthetic electron microscopy image-mask pairs that naturally reflect the structural patterns learned by the Attention U-Net. Consequently, the model can accurately detect features in a diverse array of real-world nanoparticle images and autonomously augment the training dataset without requiring human input. Cycle consistency enforces a direct correspondence between synthetic images and ground-truth masks, ensuring realistic features, which is crucial for accurate segmentation training.

Anindya Pal, Varun Ajith, Saumik Bhattacharya +1

National Institute of Technology Durgapur, Department of Physics · Indian Institute of Technology Kharagpur

Electron Microscopy Image Segmentation Microscopy Image Analysis Synthetic Data Augmentation Convolutional Neural Networks

Apr 1, 2026

AI-assisted Human-in-the-Loop Web Platform for Structural Characterization in Hard drive design

Scanning transmission electron microscopy (STEM) has become a cornerstone instrument for semiconductor materials metrology, enabling nanoscale analysis of complex multilayer structures that define device performance. Developing effective metrology workflows for such systems requires balancing automation with flexibility; rigid pipelines are brittle to sample variability, while purely manual approaches are slow and subjective. Here, we present a tunable human-AI-assisted workflow framework that enables modular and adaptive analysis of STEM images for device characterization. As an illustrative example, we demonstrate a workflow for automated layer thickness and interface roughness quantification in multilayer thin films. The system integrates gradient-based peak detection with interactive correction modules, allowing human input at the design stage while maintaining fully automated execution across samples. Implemented as a web-based interface, it processes TEM/EMD files directly, applies noise reduction and interface tracking algorithms, and outputs statistical roughness and thickness metrics with nanometer precision. This architecture exemplifies a general approach toward adaptive, reusable metrology workflows - bridging human insight and machine precision for scalable, standardized analysis in semiconductor manufacturing. The code is made available at https://github.com/utkarshp1161/thickness-mapping-webapp

Utkarsh Pratiush, Huaixun Huyan, Maryam Zahiri Azar +4

Department of Materials Science and Engineering, University of Tennessee, Knoxville, TN 37996, USA · Western Digital corporation, 5601 Great Oaks Parkway, San Jose, CA 95119, USA

Microscopy Image Analysis Scientific Workflow Automation Human-in-the-Loop AI Semiconductor Manufacturing Scanning Transmission Electron Microscopy

Mar 31, 2026

Toward Generalizable Whole Brain Representations with High-Resolution Light-Sheet Data

Unprecedented visual details of biological structures are being revealed by subcellular-resolution whole-brain 3D microscopy data, enabled by recent advances in intact tissue processing and light-sheet fluorescence microscopy (LSFM). These volumetric data offer rich morphological and spatial cellular information, however, the lack of scalable data processing and analysis methods tailored to these petabyte-scale data poses a substantial challenge for accurate interpretation. Further, existing models for visual tasks such as object detection and classification struggle to generalize to this type of data. To accelerate the development of suitable methods and foundational models, we present CANVAS, a comprehensive set of high-resolution whole mouse brain LSFM benchmark data, encompassing six neuronal and immune cell-type markers, along with cell annotations and a leaderboard. We also demonstrate challenges in generalization of baseline models built on existing architectures, especially due to the heterogeneity in cellular morphology across phenotypes and anatomical locations in the brain. To the best of our knowledge, CANVAS is the first and largest LSFM benchmark that captures intact mouse brain tissue at subcellular level, and includes extensive annotations of cells throughout the brain.

Minyoung E. Kim, Dae Hee Yun, Aditi V. Patel +4

LifeCanvas Technologies

Microscopy Image Analysis Object Detection

Mar 25, 2026

Elucidating the Design Space of Flow Matching for Cellular Microscopy

Flow-matching generative models are increasingly used to simulate cell responses to biological perturbations. However, the design space for building such models is large and underexplored. We systematically analyse the design space of flow matching models for cell-microscopy images, finding that many popular techniques are unnecessary and can even hurt performance. We develop a simple, stable, and scalable recipe which we use to train our foundation model. We scale our model to two orders of magnitude larger than prior methods, achieving a two-fold FID and ten-fold KID improvement over prior methods. We then fine-tune our model with pre-trained molecular embeddings to achieve state-of-the-art performance simulating responses to unseen molecules. Code is available at https://github.com/valence-labs/microscopy-flow-matching

Charles Jones, Emmanuel Noutahi, Jason Hartford +1

Imperial College London · Valence Labs · University of Manchester

Flow Matching Image Generation Single-Cell Perturbation Modeling Microscopy Image Analysis

Mar 24, 2026

λSplit: Self-Supervised Content-Aware Spectral Unmixing for Fluorescence Microscopy

In fluorescence microscopy, spectral unmixing aims to recover individual fluorophore concentrations from spectral images that capture mixed fluorophore emissions. Since classical methods operate pixel-wise and rely on least-squares fitting, their performance degrades with increasingly overlapping emission spectra and higher levels of noise, suggesting that a data-driven approach that can learn and utilize a structural prior might lead to improved results. Learning-based approaches for spectral imaging do exist, but they are either not optimized for microscopy data or are developed for very specific cases that are not applicable to fluorescence microscopy settings. To address this, we propose λSplit, a physics-informed deep generative model that learns a conditional distribution over concentration maps using a hierarchical Variational Autoencoder. A fully differentiable Spectral Mixer enforces consistency with the image formation process, while the learned structural priors enable state-of-the-art unmixing and implicit noise removal. We demonstrate λSplit on 3 real-world datasets that we synthetically cast into a total of 66 challenging spectral unmixing benchmarks. We compare our results against a total of 10 baseline methods, including classical methods and a range of learning-based methods. Our results consistently show competitive performance and improved robustness in high noise regimes, when spectra overlap considerably, or when the spectral dimensionality is lowered, making λSplit a new state-of-the-art for spectral unmixing of fluorescent microscopy data. Importantly, λSplit is compatible with spectral data produced by standard confocal microscopes, enabling immediate adoption without specialized hardware modifications.

Federico Carrara, Talley Lambert, Mehdi Seifi +1

Fondazione Human Technopole, Milan, Italy · Università Campus Bio-Medico, Rome, Italy · Harvard Medical School, Boston, MA, USA

Spectral Unmixing Microscopy Image Analysis Fluorescence Microscopy Hyperspectral Imaging Physics-Informed ML

Mar 22, 2026

Boundary-Aware Instance Segmentation in Microscopy Imaging

Accurate delineation of individual cells in microscopy videos is essential for studying cellular dynamics, yet separating touching or overlapping instances remains a persistent challenge. Although foundation-model for segmentation such as SAM have broadened the accessibility of image segmentation, they still struggle to separate nearby cell instances in dense microscopy scenes without extensive prompting. We propose a prompt-free, boundary-aware instance segmentation framework that predicts signed distance functions (SDFs) instead of binary masks, enabling smooth and geometry-consistent modeling of cell contours. A learned sigmoid mapping converts SDFs into probability maps, yielding sharp boundary localization and robust separation of adjacent instances. Training is guided by a unified Modified Hausdorff Distance (MHD) loss that integrates region- and boundary-based terms. Evaluations on both public and private high-throughput microscopy datasets demonstrate improved boundary accuracy and instance-level performance compared to recent SAM-based and foundation-model approaches. Source code is available at: https://github.com/ThomasMendelson/BAISeg.git

Thomas Mendelson, Joshua Francois, Galit Lahav +1

The School of Electrical and Computer Engineering, Ben-Gurion University of the Negev · Department of Systems Biology, Harvard Medical School

Instance Segmentation Boundary-Aware Segmentation Microscopy Image Analysis

Mar 20, 2026

Evaluating Vision Foundation Models for Pixel and Object Classification in Microscopy

Deep learning underlies most modern approaches and tools in computer vision, including biomedical imaging. However, for interactive semantic segmentation (often called pixel classification in this context) and interactive object-level classification (object classification), feature-based shallow learning remains widely used. This is due to the diversity of data in this domain, the lack of large pretraining datasets, and the need for computational and label efficiency. In contrast, state-of-the-art tools for many other vision tasks in microscopy - most notably cellular instance segmentation - already rely on deep learning and have recently benefited substantially from vision foundation models (VFMs), particularly SAM. Here, we investigate whether VFMs can also improve pixel and object classification compared to current approaches. To this end, we evaluate several VFMs, including general-purpose models (SAM, SAM2, SAM3, DINOv3) and domain-specific ones (

μ

SAM, PathoSAM, KRONOS), in combination with shallow learning and attentive probing on five diverse and challenging datasets. Our results demonstrate consistent improvements over hand-crafted features and provide a clear pathway toward practical improvements. Furthermore, our study establishes a benchmark for VFMs in microscopy and informs future developments in this area.

Carolin Teuber, Anwai Archit, Tobias Boothe +3

Georg-August-University Göttingen, Institute of Computer Science · Department of Tissue Dynamics and Regeneration, Max Planck Institute for Multidisciplinary Sciences, Göttingen · Georg-August-University Göttingen, Faculty of Biology and Psychology +3

Vision Foundation Models Semantic Segmentation Microscopy Image Analysis

Mar 19, 2026

Ca2+ transient detection and segmentation with the Astronomically motivated algorithm for Background Estimation And Transient Segmentation (Astro-BEATS)

Fluorescence-based Ca

^{2+}

-imaging is a powerful tool for studying localized neuronal activity, including miniature Synaptic Calcium Transients, providing real-time insights into synaptic activity. These transients induce only subtle changes in the fluorescence signal, often barely above baseline, which poses a significant challenge for automated synaptic transient detection and segmentation. Detecting astronomical transients similarly requires efficient algorithms that will remain robust over a large field of view with varying noise properties. We leverage techniques used in astronomical transient detection for miniature Synaptic Calcium Transient detection in fluorescence microscopy. We present Astro-BEATS, an automatic miniature Synaptic Calcium Transient segmentation algorithm that incorporates image estimation and source-finding techniques used in astronomy and designed for Ca

^{2+}

-imaging videos. Astro-BEATS outperforms current threshold-based approaches for synaptic Ca

^{2+}

transient detection and segmentation. The produced segmentation masks can be used to train a supervised deep learning algorithm for improved synaptic Ca

^{2+}

transient detection in Ca

^{2+}

-imaging data. The speed of Astro-BEATS and its applicability to previously unseen datasets without re-optimization makes it particularly useful for generating training datasets for deep learning-based approaches.

Bolin Fan, Anthony Bilodeau, Frederic Beaupre +4

Dunlap Institute for Astronomy and Astrophysics, University of Toronto, Toronto, Canada · David A. Dunlap Department for Astronomy and Astrophysics, University of Toronto, Toronto, Canada · CERVO Brain Research Center, Québec, Canada +4

Microscopy Image Analysis Image Segmentation Fluorescence Microscopy

Mar 19, 2026

Statistical Characteristic-Guided Denoising for Rapid High-Resolution Transmission Electron Microscopy Imaging

High-Resolution Transmission Electron Microscopy (HRTEM) enables atomic-scale observation of nucleation dynamics, which boosts the studies of advanced solid materials. Nonetheless, due to the millisecond-scale rapid change of nucleation, it requires short-exposure rapid imaging, leading to severe noise that obscures atomic positions. In this work, we propose a statistical characteristic-guided denoising network, which utilizes statistical characteristics to guide the denoising process in both spatial and frequency domains. In the spatial domain, we present spatial deviation-guided weighting to select appropriate convolution operations for each spatial position based on deviation characteristic. In the frequency domain, we present frequency band-guided weighting to enhance signals and suppress noise based on band characteristics. We also develop an HRTEM-specific noise calibration method and generate a dataset with disordered structures and realistic HRTEM image noises. It can ensure the denoising performance of models on real images for nucleation observation. Experiments on synthetic and real data show our method outperforms the state-of-the-art methods in HRTEM image denoising, with effectiveness in the localization downstream task. Code will be available at https://github.com/HeasonLee/SCGN.

Hesong Li, Ziqi Wu, Ruiwen Shao +1

Beijing Institute of Technology

Scanning Transmission Electron Microscopy Image Denoising Microscopy Image Analysis Electron Microscopy

Mar 18, 2026

Revisiting foundation models for cell instance segmentation

Cell segmentation is a fundamental task in microscopy image analysis. Several foundation models for cell segmentation have been introduced, virtually all of them are extensions of Segment Anything Model (SAM), improving it for microscopy data. Recently, SAM2 and SAM3 have been published, further improving and extending the capabilities of general-purpose segmentation foundation models. Here, we comprehensively evaluate foundation models for cell segmentation (CellPoseSAM, CellSAM,

μ

SAM) and for general-purpose segmentation (SAM, SAM2, SAM3) on a diverse set of (light) microscopy datasets, for tasks including cell, nucleus and organoid segmentation. Furthermore, we introduce a new instance segmentation strategy called automatic prompt generation (APG) that can be used to further improve SAM-based microscopy foundation models. APG consistently improves segmentation results for

μ

SAM, which is used as the base model, and is competitive with the state-of-the-art model CellPoseSAM. Moreover, our work provides important lessons for adaptation strategies of SAM-style models to microscopy and provides a strategy for creating even more powerful microscopy foundation models. Our code is publicly available at https://github.com/computational-cell-analytics/micro-sam.

Anwai Archit, Constantin Pape

Georg-August-University Göttingen, Institute of Computer Science · CAIMed - Lower Saxony Center for AI & Causal Methods in Medicine, Göttingen · Cluster of Excellence Multiscale Bioimaging (MBExC), Georg-August-University Göttingen

Instance Segmentation Microscopy Image Analysis

Mar 17, 2026

Novelty-Driven Target-Space Discovery in Automated Electron and Scanning Probe Microscopy

Modern automated microscopy faces a fundamental discovery challenge: in many systems, the most important scientific information does not reside in the immediately visible image features, but in the target space of sequentially acquired spectra or functional responses, making it essential to develop strategies that can actively search for new behaviors rather than simply optimize known objectives. Here, we developed a deep-kernel-learning BEACON framework that is explicitly designed to guide discovery in the target space by learning structure-property relationships during the experiment and using that evolving model to seek diverse response regimes. We first established the method through demonstration workflows built on pre-acquired ground-truth datasets, which enabled direct benchmarking against classical acquisition strategies and allowed us to define a set of monitoring functions for comparing exploration quality, target-space coverage, and surrogate-model behavior in a transparent and reproducible manner. This benchmarking framework provides a practical basis for evaluating discovery-driven algorithms, not just optimization performance. We then operationalized and deployed the workflow on STEM, showing that the approach can transition from offline validation to real experimental implementation. To support adoption and extension by the broader community, the associated notebooks are available, allowing users to reproduce the workflows, test the benchmarks, and adapt the method to their own instruments and datasets.

Utkarsh Pratiush, Kamyar Barakati, Boris N. Slautin +4

Department of Materials Science and Engineering, University of Tennessee, Knoxville, TN 37996, USA · Department of Chemistry, University of Washington, Seattle, WA 98195, USA · Pacific Northwest National Laboratory, Richland, WA 99354, USA

Deep Kernel Learning Scanning Transmission Electron Microscopy Autonomous Experimentation Electron Microscopy Microscopy Image Analysis

Mar 17, 2026

Understanding Cell Fate Decisions with Temporal Attention

Understanding non-genetic determinants of cell fate is critical for developing and improving cancer therapies, as genetically identical cells can exhibit divergent outcomes under the same treatment conditions. In this work, we present a deep learning approach for cell fate prediction from raw long-term live-cell recordings of cancer cell populations under chemotherapeutic treatment. Our Transformer model is trained to predict cell fate directly from raw image sequences, without relying on predefined morphological or molecular features. Beyond classification, we introduce a comprehensive explainability framework for interpreting the temporal and morphological cues guiding the model's predictions. We demonstrate that prediction of cell outcomes is possible based on the video only, our model achieves balanced accuracy of 0.94 and an F1-score of 0.93. Attention and masking experiments further indicate that the signal predictive of the cell fate is not uniquely located in the final frames of a cell trajectory, as reliable predictions are possible up to 10 h before the event. Our analysis reveals distinct temporal distribution of predictive information in the mitotic and apoptotic sequences, as well as the role of cell morphology and p53 signaling in determining cell outcomes. Together, these findings demonstrate that attention-based temporal models enable accurate cell fate prediction while providing biologically interpretable insights into non-genetic determinants of cellular decision-making. The code is available at https://github.com/bozeklab/Cell-Fate-Prediction.

Florian Bürger, Martim Dias Gomes, Adrián E. Granada +2

Institute for Biomedical Informatics, Faculty of Medicine and University Hospital Cologne, University of Cologne, Cologne, Germany · Faculty of Mathematics and Natural Sciences, University of Cologne, Cologne, Germany · Cologne Excellence Cluster on Cellular Stress Responses in Aging-Associated Diseases (CECAD), University of Cologne, Cologne, Germany +2

Microscopy Image Analysis Interpretable ML

Mar 17, 2026

Bridging the Simulation-to-Reality Gap in Electron Microscope Calibration via VAE-EM Estimation

Electron microscopy has enabled many scientific breakthroughs across multiple fields. A key challenge is the tuning of microscope parameters based on images to overcome optical aberrations that deteriorate image quality. This calibration problem is challenging due to the high-dimensional and noisy nature of the diagnostic images, and the fact that optimal parameters cannot be identified from a single image. We tackle the calibration problem for Scanning Transmission Electron Microscopes (STEM) by employing variational autoencoders (VAEs), trained on simulated data, to learn low-dimensional representations of images, whereas most existing methods extract only scalar values. We then simultaneously estimate the model that maps calibration parameters to encoded representations and the optimal calibration parameters using an expectation maximization (EM) approach. This joint estimation explicitly addresses the simulation-to-reality gap inherent in data-driven methods that train on simulated data from a digital twin. We leverage the known symmetry property of the optical system to establish global identifiability of the joint estimation problem, ensuring that a unique optimum exists. We demonstrate that our approach is substantially faster and more consistent than existing methods on a real STEM, achieving a 2x reduction in estimation error while requiring fewer observations. This represents a notable advance in automated STEM calibration and demonstrates the potential of VAEs for information compression in images. Beyond microscopy, the VAE-EM framework applies to inverse problems where simulated training data introduces a reality gap and where non-injective mappings would otherwise prevent unique solutions.

Jilles S. van Hulst, W. P. M. H. Heemels, Duarte J. Antunes

The authors are with the Control Systems Technology Section, Department of Mechanical Engineering, Eindhoven University of Technology, 5600 MB Eindhoven, The Netherlands.

Electron Microscopy Scanning Transmission Electron Microscopy Microscopy Image Analysis Sim-to-Real Transfer

Mar 16, 2026

Self-Supervised ImageNet Representations for In Vivo Confocal Microscopy: Tortuosity Grading without Segmentation Maps

The tortuosity of corneal nerve fibers are used as indication for different diseases. Current state-of-the-art methods for grading the tortuosity heavily rely on expensive segmentation maps of these nerve fibers. In this paper, we demonstrate that self-supervised pretrained features from ImageNet are transferable to the domain of in vivo confocal microscopy. We show that DINO should not be disregarded as a deep learning model for medical imaging, although it was superseded by two later versions. After careful fine-tuning, DINO improves upon the state-of-the-art in terms of accuracy (84,25%) and sensitivity (77,97%). Our fine-tuned model focuses on the key morphological elements in grading without the use of segmentation maps.

Kim Ouan, Noémie Moreau, Katarzyna Bozek

Faculty of Mathematics and Natural Sciences, University of Cologne, Germany · Institute for Biomedical Informatics, Faculty of Medicine and University Hospital Cologne, University of Cologne, Germany · Center for Molecular Medicine Cologne (CMMC), Faculty of Medicine and University Hospital Cologne, University of Cologne, Germany +1

Medical Image Classification Microscopy Image Analysis

Mar 13, 2026

A protocol for evaluating robustness to H&E staining variation in computational pathology models

Sensitivity to staining variation remains a major barrier to deploying computational pathology (CPath) models as hematoxylin and eosin (H&E) staining varies across laboratories, requiring systematic assessment of how this variability affects model prediction. In this work, we developed a three-step protocol for evaluating robustness to H&E staining variation in CPath models. Step 1: Select reference staining conditions, Step 2: Characterize test set staining properties, Step 3: Apply CPath model(s) under simulated reference staining conditions. Here, we first created a new reference staining library based on the PLISM dataset. As an exemplary use case, we applied the protocol to assess the robustness properties of 306 microsatellite instability (MSI) classification models on the unseen SurGen colorectal cancer dataset (n=738), including 300 attention-based multiple instance learning models trained on the TCGA-COAD/READ datasets across three feature extractors (UNI2-h, H-Optimus-1, Virchow2), alongside six public MSI classification models. Classification performance was measured as AUC, and robustness as the min-max AUC range across four simulated staining conditions (low/high H&E intensity, low/high H&E color similarity). Across models and staining conditions, classification performance ranged from AUC 0.769-0.911 (

Δ

= 0.142). Robustness ranged from 0.007-0.079 (

Δ

= 0.072), and showed a weak inverse correlation with classification performance (Pearson r=-0.22, 95% CI [-0.34, -0.11]). Thus, we show that the proposed evaluation protocol enables robustness-informed CPath model selection and provides insight into performance shifts across H&E staining conditions, supporting the identification of operational ranges for reliable model deployment. Code is available at https://github.com/CTPLab/staining-robustness-evaluation .

Lydia A. Schönpflug, Nikki van den Berg, Sonali Andani +5

Computational Pathology Microscopy Image Analysis Histopathology Image Classification

Mar 12, 2026

Nyxus: A Next Generation Image Feature Extraction Library for the Big Data and AI Era

Modern imaging instruments can produce terabytes to petabytes of data for a single experiment. The biggest barrier to processing big image datasets has been computational, where image analysis algorithms often lack the efficiency needed to process such large datasets or make tradeoffs in robustness and accuracy. Deep learning algorithms have vastly improved the accuracy of the first step in an analysis workflow (region segmentation), but the expansion of domain specific feature extraction libraries across scientific disciplines has made it difficult to compare the performance and accuracy of extracted features. To address these needs, we developed a novel feature extraction library called Nyxus. Nyxus is designed from the ground up for scalable out-of-core feature extraction for 2D and 3D image data and rigorously tested against established standards. The comprehensive feature set of Nyxus covers multiple biomedical domains including radiomics and cellular analysis, and is designed for computational scalability across CPUs and GPUs. Nyxus has been packaged to be accessible to users of various skill sets and needs: as a Python package for code developers, a command line tool, as a Napari plugin for low to no-code users or users that want to visualize results, and as an Open Container Initiative (OCI) compliant container that can be used in cloud or super-computing workflows aimed at processing large data sets. Further, Nyxus enables a new methodological approach to feature extraction allowing for programmatic tuning of many features sets for optimal computational efficiency or coverage for use in novel machine learning and deep learning applications.

Nicholas Schaub, Andriy Kharchenko, Hamdah Abbasi +3

DSBU, Axle Research, 6116 Executive Blvd Suite 400, Rockville, 20852, MD, USA. · NGRF, NovaGen Research Fund, 6116 Executive Blvd Suite 400, Rockville, 20852, MD, USA. · NCATS, 9800 Medical Center Dr, Rockville, 20850, MD, USA.

Medical Image Analysis Microscopy Image Analysis

Mar 12, 2026

Leveraging Phytolith Research using Artificial Intelligence

Phytolith analysis is a crucial tool for reconstructing past vegetation and human activities, but traditional methods are severely limited by labour-intensive, time-consuming manual microscopy. To address this bottleneck, we present Sorometry: a comprehensive end-to-end artificial intelligence pipeline for the high-throughput digitisation, inference, and interpretation of phytoliths. Our workflow processes z-stacked optical microscope scans to automatically generate synchronised 2D orthoimages and 3D point clouds of individual microscopic particles. We developed a multimodal fusion model that combines ConvNeXt for 2D image analysis and PointNet++ for 3D point cloud analysis, supported by a graphical user interface for expert annotation and review. Tested on reference collections and archaeological samples from the Bolivian Amazon, our fusion model achieved a global classification accuracy of 77.9% across 24 diagnostic morphotypes and 84.5% for segmentation quality. Crucially, the integration of 3D data proved essential for distinguishing complex morphotypes (such as grass silica short cell phytoliths) whose diagnostic features are often obscured by their orientation in 2D projections. Beyond individual object classification, Sorometry incorporates Bayesian finite mixture modelling to predict overall plant source contributions at the assemblage level, successfully identifying specific plants like maize and palms in complex mixed samples. This integrated platform transforms phytolith research into an "omics"-scale discipline, dramatically expanding analytical capacity, standardising expert judgements, and enabling reproducible, population-level characterisations of archaeological and paleoecological assemblages.

Andrés G. Mejía Ramón, Kate Dudgeon, Nina Witteveen +6

Institut de Ciència i Tecnologia Ambientals, Universitat Autònoma de Barcelona Carrer de les Columnes s/n, Cerdanyola del Vallès, Barcelona, 08193, Espanya · Department of Anthropology, Smithsonian National Museum of Natural History, 1000 Madison Drive NW, Washington, D.C., 20560, United States · Smithsonian Tropical Research Institute, Luis Clement Avenue, Bldg. 401 Tupper, Ancón, Ciudad de Panamá, República de Panamá +5

Microscopy Image Analysis

Mar 11, 2026

Single molecule localization microscopy challenge: a biologically inspired benchmark for long-sequence modeling

State space models (SSMs) have recently achieved strong performance on long sequence modeling tasks while offering improved memory and computational efficiency compared to transformer based architectures. However, their evaluation has been largely limited to synthetic benchmarks and application domains such as language and audio, leaving their behavior on sparse and stochastic temporal processes in biological imaging unexplored. In this work, we introduce the Single Molecule Localization Microscopy Challenge (SMLM-C), a benchmark dataset consisting of ten SMLM simulations spanning dSTORM and DNA-PAINT modalities with varying hyperparameter designed to evaluate state space models on biologically realistic spatiotemporal point process data with known ground truth. Using a controlled subset of these simulations, we evaluate state space models and find that performance degrades substantially as temporal discontinuity increases, revealing fundamental challenges in modeling heavy-tailed blinking dynamics. These results highlight the need for sequence models better suited to sparse, irregular temporal processes encountered in real world scientific imaging data.

Fatemeh Valeh, Monika Farsang, Radu Grosu +1

Department of Biophysics, Technische Universität Wien, Vienna, Austria · Department of Informatics, Technische Universität Wien, Vienna, Austria

Sequence Modeling State Space Models Irregular Time-Series Modeling Microscopy Image Analysis

Mar 11, 2026

Combining Microscopy Data and Metadata for Reconstruction of Cellular Traction Forces Using a Hybrid Vision Transformer-U-Net

Traction force microscopy (TFM) is a widely used technique for quantifying the forces that cells exert on their surrounding extracellular matrix. Although deep learning methods have recently been applied to TFM data analysis, several challenges remain-particularly achieving reliable inference across multiple spatial scales and integrating additional contextual information such as cell type to improve accuracy. In this study, we propose ViT+UNet, a robust deep learning architecture that integrates a U-Net with a Vision Transformer. Our results demonstrate that this hybrid model outperforms both standalone U-Net and Vision Transformer architectures in predicting traction force fields. Furthermore, ViT+UNet exhibits superior generalization across diverse spatial scales and varying noise levels, enabling its application to TFM datasets obtained from different experimental setups and imaging systems. By appropriately structuring the input data, our approach also allows the inclusion of metadata, in our case cell-type information, to enhance prediction specificity and accuracy.

Yunfei Huang, Elena Van der Vorst, Alexander Richard +1

Fakultät Physik, Technische Universität Dortmund, Dortmund, 44227, Germany. · Faculty of Physics and Center for NanoScience, Ludwig-Maximilians-Universität München, Munich, 80752, Germany. · Department of Veterinary Sciences, Ludwig-Maximilians-Universität München, Munich, 80752, Germany.

Microscopy Image Analysis U-Net Hybrid CNN-Transformer Architectures