Agent-Driven Generative Priors for Pathology Image Synthesis and Segmentation

Sicheng Yang; Rui Tang; Haipeng Zhou; Hongqiu Wang; Zhaohu Xing; Jianda Ma; Lie Dai; Guang Yang

cs.CVNov 4, 2026

Agent-Driven Generative Priors for Pathology Image Synthesis and Segmentation

Authors: Sicheng Yang, Rui Tang, Haipeng Zhou, Hongqiu Wang, Zhaohu Xing, Jianda Ma, Lie Dai, Guang Yang,

Organizations: Sicheng Yang, Haipeng Zhou, Hongqiu Wang, and Zhaohu Xing are with the Hong Kong University of Science and Technology (Guangzhou), China. · Rui Tang is with The University of Hong Kong, Pokfulam, Hong Kong SAR, China. · Jianda Ma and Lie Dai are with Department of Rheumatology and Immunology, Sun Yat-Sen Memorial Hospital, Sun Yat-Sen University, China. · Guang Yang is with Imperial College London, UK. ·

Abstract

Deep learning methods for pathology are constrained by the limited availability and high cost of large-scale pixel-level annotations. Synthetic data can alleviate this limitation, but existing pipelines often lack reliable quality control and remain weakly connected to downstream analysis. To address these issues, we propose a three-stage framework that integrates automated data screening, generative representation learning, and efficient adaptation to downstream tasks. First, we develop a pathology agent system to assess image quality and filter samples, resulting in TCGA-200K, a dataset containing 217,030 pathology patches selected from the public TCGA repository. Second, we train a generative model on the screened data. In addition to image synthesis, its backbone learns structural and semantic representations that can be transferred to downstream tasks. Third, we adapt the pretrained generative backbone to semantic segmentation using a lightweight task-specific head. We further introduce PathSeg-380K, a semantic segmentation dataset with expert annotations for approximately 380,000 cells across 12 cell types. Experiments on multiple pathology datasets show that the proposed framework achieves state-of-the-art segmentation performance. Our code is publicly available at https://github.com/script-Yang/Agent-GenPath.

Figures & tables

Fig. 1: Overview of the datasets and data distributions. (a) Source composition of TCGA-200K, utilized for pre-training the generative prior. (b) Class distributions for downstream segmentation tasks, comparing the publicly available PanNuke dataset with our large-scale, annotated PathSeg-380K dataset, containing

\sim

380,000 annotated cells.

Fig. 2: Overview of the proposed unified three-stage framework. In Stage I, an agent-driven screening pipeline evaluates pathology patches from multiple quality dimensions and selects high-quality samples to construct the TCGA-200K dataset. In Stage II, the curated data are used to learn pathology-specific generative priors through DiT blocks. In Stage III, the learned generative priors are transferred to downstream segmentation on PanNuke and PathSeg-380K, where the pretrained backbone provides multi-level pathology representations for accurate pathology segmentation.

Fig. 3: Stage I: Agent-driven high-quality pathology image selection. Given an input pathology patch $x_{i}$ , $\mathcal{A}_{P}$ initializes the memory state $m_{i}$ , $\mathcal{A}_{R}$ evaluates cell presence, stain quality, and histologic clarity, and $\mathcal{A}_{V}$ performs closed-loop verification to produce the final quality scores.

Fig. 4: Score distributions before and after the integration of our pathology expert agent across three quality dimensions. The distributions shift toward higher scores after agent-driven filtering, indicating an improvement in overall data quality.

Fig. 5: Comparison of synthetic (left) and real (right) pathology images. The generated samples faithfully reproduce complex tissue morphology, cellular structures, and spatial organization, showing strong visual consistency with real clinical images.

Fig. 6: Stage II: Learning pathology generative priors via Flow Matching. Given an image patch and timestep $t$ , the DiT generator $G$ is trained to predict the target flow field.

Fig. 7: Visualizing the refinement of feature activations in the DiT generative prior. Moving from shallow to deep layers, the model sequentially learns to disentangle semantic structures at various scales, such as individual cells, glandular tissues, and the surrounding background, which provides a crucial foundation for robust semantic segmentation.

Fig. 8: Stage III: Transferring generative priors to downstream segmentation. Multi-level features are extracted from the pathology-pretrained DiT backbone, transformed and fused into a unified representation, and then fed into the segmentation head.

Task-specific Segmentation Methods
	PanNuke				PathSeg-380K
Method	mIOU (%) $\uparrow$	mDICE (%) $\uparrow$	p-Acc (%) $\uparrow$	mASD $\downarrow$	mIOU (%) $\uparrow$	mDICE (%) $\uparrow$	p-Acc (%) $\uparrow$	mASD $\downarrow$
U-Net [ 42 ]	$48.09_{\scriptscriptstyle\pm 0.25}$	$61.44_{\scriptscriptstyle\pm 0.42}$	$91.56_{\scriptscriptstyle\pm 0.14}$	$29.71_{\scriptscriptstyle\pm 0.56}$	$48.62_{\scriptscriptstyle\pm 0.94}$	$60.84_{\scriptscriptstyle\pm 0.70}$	$92.65_{\scriptscriptstyle\pm 0.19}$	$6.40_{\scriptscriptstyle\pm 0.15}$
TransUNet [ 43 ]	$46.63_{\scriptscriptstyle\pm 1.01}$	$58.23_{\scriptscriptstyle\pm 0.93}$	$91.88_{\scriptscriptstyle\pm 0.28}$	$23.05_{\scriptscriptstyle\pm 0.67}$	$46.68_{\scriptscriptstyle\pm 0.96}$	$59.27_{\scriptscriptstyle\pm 0.71}$	$92.29_{\scriptscriptstyle\pm 0.21}$	$10.33_{\scriptscriptstyle\pm 0.19}$
HoVer-Net [ 44 ]	$51.38_{\scriptscriptstyle\pm 0.35}$	$64.83_{\scriptscriptstyle\pm 0.54}$	$91.74_{\scriptscriptstyle\pm 0.20}$	$22.42_{\scriptscriptstyle\pm 0.93}$	$52.41_{\scriptscriptstyle\pm 0.75}$	$65.73_{\scriptscriptstyle\pm 0.56}$	$92.56_{\scriptscriptstyle\pm 0.16}$	$7.89_{\scriptscriptstyle\pm 0.13}$
CPP-Net [ 45 ]	$47.58_{\scriptscriptstyle\pm 0.17}$	$60.82_{\scriptscriptstyle\pm 0.21}$	$91.53_{\scriptscriptstyle\pm 0.14}$	$29.45_{\scriptscriptstyle\pm 0.28}$	$50.17_{\scriptscriptstyle\pm 0.77}$	$62.21_{\scriptscriptstyle\pm 0.57}$	$92.73_{\scriptscriptstyle\pm 0.16}$	$7.58_{\scriptscriptstyle\pm 0.15}$

TABLE I: Quantitative comparison on PanNuke and PathSeg-380K. * denotes a statistically significant improvement over the corresponding second-best result (

p<0.05

Fig. 9: Visual comparison of segmentation results across pathology tissue types. Green boxes highlight challenging regions where our framework better matches the ground truth (GT), particularly under cluttered backgrounds and varying cell scales.

Evaluator	Spearman’s $\rho$ $\uparrow$			MAE $\downarrow$	$\boldsymbol{\kappa}$ $\uparrow$
	Cell	Stain	Clarity	Avg.	Avg.
Pathologist 1 vs. Pathologist 2	0.815	0.835	0.816	0.306	0.827
Agent vs. Consensus	0.782	0.811	0.829	0.326	0.814
Regressor vs. Consensus	0.759	0.797	0.796	0.378	0.789
Regressor vs. Agent	0.909	0.914	0.913	0.170	0.912

TABLE II: Reliability of pathology quality assessment against pathologist consensus on an independent test set.

Curation Setting	FID $\downarrow$	KID $\downarrow$	MOS $\uparrow$	mIoU (%) $\uparrow$
From Scratch	–	–	–	47.19
w/o Agent (100%)	38.75	0.056	3.12	53.41
w/ Agent (Top 75%)	33.61	0.032	4.06	54.42
w/ Agent (Top 50%)	30.08	0.028	3.87	54.46
w/ Agent (Top 25%)	24.94	0.024	4.20	55.12
w/ Agent (Top 10%)	23.96	0.021	4.43	55.46

TABLE III: Effect of agent-based data curation on generation quality and downstream segmentation performance on PanNuke.

				PanNuke		PathSeg-380K
Method	Gen. Prior	Agent	Seg. Head	mIoU (%) $\uparrow$	mASD $\downarrow$	mIoU (%) $\uparrow$	mASD $\downarrow$
Baseline				44.58	30.93	44.23	8.86
M1	✓			52.17	25.86	52.08	6.58
M2	✓	✓		53.85	24.42	53.72	5.37
M3	✓		✓	53.41	24.01	53.19	5.51
Ours (Full)	✓	✓	✓	55.46	22.91	55.45	4.81

TABLE IV: Ablation study of the proposed components.

Fig. 10: Effect of hyperparameter

t

on mIoU, mDice, and mASD. Yellow stars highlight the best-performing settings for each metric.

Histopathology Image Synthesis

See all 8 papers

CytoSyn: a Foundation Diffusion Model for Histopathology -- Tech Report
Mar 18, 2026Thomas Duboudin, Xavier Fontaine, Etienne Andrier +7Histopathology Image Synthesis Medical Image Synthesis
Owkin, Inc
Assessment of Conditional Diffusion Model for Synthetic Histopathology Image Generation
Aug 4, 2026Seyed Kahaki, Shijie Li, Weijie Chen +1Medical Image Synthesis Image Generation Evaluation
Division of Imaging, Diagnostics, and Software Reliability, U.S. Food and Drug Administration (FDA)
Controllable Diffusion-Based Lesion Inpainting for Scalable Histopathology Data Augmentation
Jan 13, 2026Mohamad Koohi-Moghadam, Mohammad-Ali Nikouei Mahani, Rex K. H. Au-Yeung +6Computational Pathology Synthetic Data Augmentation
Department of Diagnostic Radiology, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Pok Fu Lam, Hong Kong SAR, PR China · Department of Pathology, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Hong Kong SAR, China · Department of Anatomical and Cellular Pathology, Prince of Wales Hospital, The Chinese University of Hong Kong, Hong Kong SAR, China. +2

Mar 18, 2026J/K move · Enter open · S save

CytoSyn: a Foundation Diffusion Model for Histopathology -- Tech Report

Thomas Duboudin, Xavier Fontaine, Etienne Andrier, Lionel Guillou+6

Owkin, Inc

Histopathology Image Synthesis Medical Image Synthesis Pathology Foundation Models Computational Pathology Latent Diffusion Models

Open paper AskPDF arXiv

Computational pathology has made significant progress in recent years, fueling advances in both fundamental disease understanding and clinically ready tools. This evolution is driven by the availability of large amounts of digitized slides and specialized deep learning methods and models. Multiple self-supervised foundation feature extractors have been developed, enabling downstream predictive applications from cell segmentation to tumor sub-typing and survival analysis. In contrast, generative foundation models designed specifically for histopathology remain scarce. Such models could address tasks that are beyond the capabilities of feature extractors, such as virtual staining. In this paper, we introduce CytoSyn, a state-of-the-art foundation latent diffusion model that enables the guided generation of highly realistic and diverse histopathology H&E-stained images, as shown in an extensive benchmark. We explored methodological improvements, training set scaling, sampling strategies and slide-level overfitting, culminating in the improved CytoSyn-v2, and compared our work to PixCell, a state-of-the-art model, in an in-depth manner. This comparison highlighted the strong sensitivity of both diffusion models and performance metrics to preprocessing-specific details such as JPEG compression. Our model has been trained on a dataset obtained from more than 10,000 TCGA diagnostic whole-slide images of 32 different cancer types. Despite being trained only on oncology slides, it maintains state-of-the-art performance generating inflammatory bowel disease images. To support the research community, we publicly release CytoSyn's weights, its training and validation datasets, and a sample of synthetic images in this repository: https://huggingface.co/Owkin-Bioptimus/CytoSyn.

Abstract

Histopathology Image Synthesis

Explore similar work

CytoSyn: a Foundation Diffusion Model for Histopathology -- Tech Report

Assessment of Conditional Diffusion Model for Synthetic Histopathology Image Generation

Controllable Diffusion-Based Lesion Inpainting for Scalable Histopathology Data Augmentation