Layout-Guided Controllable Pathology Image Generation with In-Context Diffusion Transformers

Yuntao Shou; Xiangyong Cao; Qian Zhao; Deyu Meng

cs.CVMar 11, 2026

Layout-Guided Controllable Pathology Image Generation with In-Context Diffusion Transformers

Authors: Yuntao Shou, Xiangyong Cao, Qian Zhao, Deyu Meng

Organizations: Xi’an Jiaotong University

Abstract

Controllable pathology image synthesis requires reliable regulation of spatial layout, tissue morphology, and semantic detail. However, existing text-guided diffusion models offer only coarse global control and lack the ability to enforce fine-grained structural constraints. Progress is further limited by the absence of large datasets that pair patch-level spatial layouts with detailed diagnostic descriptions, since generating such annotations for gigapixel whole-slide images is prohibitively time-consuming for human experts. To overcome these challenges, we first develop a scalable multi-agent LVLM annotation framework that integrates image description, diagnostic step extraction, and automatic quality judgment into a coordinated pipeline, and we evaluate the reliability of the system through a human verification process. This framework enables efficient construction of fine-grained and clinically aligned supervision at scale. Building on the curated data, we propose In-Context Diffusion Transformer (IC-DiT), a layout-aware generative model that incorporates spatial layouts, textual descriptions, and visual embeddings into a unified diffusion transformer. Through hierarchical multimodal attention, IC-DiT maintains global semantic coherence while accurately preserving structural and morphological details. Extensive experiments on five histopathology datasets show that IC-DiT achieves higher fidelity, stronger spatial controllability, and better diagnostic consistency than existing methods. In addition, the generated images serve as effective data augmentation resources for downstream tasks such as cancer classification and survival analysis.

Figures & tables

Figure 1: Comparison between traditional diffusion-based pathology synthesis and the proposed IC-DiT. (a) Conventional latent diffusion models rely on global cross-attention, offering limited control over spatial structure and semantics. (b) IC-DiT integrates multi-modal in-context conditioning with image, layout, visual, and text tokens, enabling anatomically coherent and spatially controllable pathology generation.

Figure 2: Overview of the multi-agent framework for pathological image description generation. It includes: (1) Pathological Feature Extraction, where vision-language models convert images into visual-textual descriptions; (2) Key Step Extraction, where language models break down diagnostic reasoning into structured steps; and (3) Human Verification, which uses automated scores and expert feedback for accuracy assessment. A Judge Agent quantifies reliability through recognition and coherence metrics.

Figure 3: Overview of the In-Context Diffusion Transformer (IC-DiT) architecture. This model integrates text, image, layout, and embedding tokens through separate encoders (T5, VAE, iBOT) and processes them with multi-modal attention (MM-Attention) mechanisms. The attention mechanism is applied across different modalities to fuse them in a shared latent space, allowing for high-fidelity pathology image generation based on semantic descriptions and spatial layouts.

Methods	BLCA			BRCA			GBMLGG			LUAD
Methods	Vanilla FID	CLIP FID	Similarity	Vanilla FID	CLIP FID	Similarity	Vanilla FID	CLIP FID	Emb Similarity	Vanilla FID	CLIP FID	Similarity
20x Magnification
MFDiffusion	115.73	101.88	0.24	104.72	94.31	0.29	127.90	115.43	0.18	145.72	131.93	0.15
Medfusion	57.62	50.09	0.33	68.52	59.02	0.31	70.59	66.33	0.27	73.48	67.65	0.24
Pathldm	18.62	15.47	0.46	21.38	17.59	0.43	22.54	19.41	0.41	24.10	22.03	0.38
SSL-guided LDM	10.47	7.43	0.56	12.57	8.41	0.49	14.59	13.36	0.46	15.03	14.17	0.44

Table 1: Quantitative evaluation of pathological image generation methods under 20× and 5× magnification across four cancer datasets. The best and second best results are highlighted in bold and underline , respectively.

20x Magnification
Methods	BLCA	BRCA	GBMLGG	LUAD
ControlNet	20.67	19.76	20.03	20.34
PathDiff	22.48	21.35	21.92	22.15
IC-DiT	24.83	24.64	23.72	25.67
5x Magnification
ControlNet	17.23	18.91	17.34	18.45

Table 2: Quantitatively evaluation the image-text similarity of four cancer datasets at 20x and 5x magnification. The best and second best results are highlighted in bold and underline , respectively.

20x Magnification
Methods	BLCA	BRCA	GBMLGG	LUAD
Diffmix	0.7123	0.7312	0.7089	0.7245
SDM	0.6532	0.6891	0.6245	0.6978
ControlNet	0.7425	0.7034	0.7289	0.7167
PathDiff	0.7856	0.7912	0.7734	0.8015
IC-DiT	0.8319	0.8412	0.8352	0.8363

Table 3: Quantitatively evaluation the faithfulness scores of four cancer datasets at 20x and 5x magnification. The best and second best results are highlighted in bold and underline , respectively.

Components			BLCA	BRCA	GBMLGG	LUAD
Global Caption	Layout	Embeddings Similarity	Vanilla FID	Vanilla FID	Vanilla FID	Vanilla FID
$\checkmark$			74.59	79.02	76.60	80.45
	$\checkmark$		70.05	72.31	72.29	76.53
		$\checkmark$	12.15	15.77	18.68	20.03
$\checkmark$	$\checkmark$		66.14	67.51	68.20	73.37
$\checkmark$		$\checkmark$	10.09	12.41	14.57	17.30

Table 4: The results of ablation studies in TCGA datasets. We highlight the best performing scores in bold .

Figure 4: Qualitative comparison of layout-guided pathology image generation. Each column shows the input caption, corresponding mask, real image, and generated image.

Figure 5: Ablation study on layout control. Comparison between real pathology patches and images generated without layout-mask constraints.

Methods	Train Data	Datasets
Methods	Train Data	BLCA	BRCA	GBMLGG	LUAD	UCEC
Porpoise	Real	0.6461 $\pm$ 0.0338	0.6207 $\pm$ 0.0544	0.8479 $\pm$ 0.0128	0.6403 $\pm$ 0.0412	0.6918 $\pm$ 0.0488
w/PathDiff	Synth	0.5735 $\pm$ 0.0273	0.5581 $\pm$ 0.0373	0.7383 $\pm$ 0.0470	0.5637 $\pm$ 0.0509	0.6049 $\pm$ 0.0286
w/PathDiff	Real+Synth	0.6523 $\pm$ 0.0179	0.6291 $\pm$ 0.0307	0.8585 $\pm$ 0.0288	0.6524 $\pm$ 0.0283	0.6997 $\pm$ 0.0416
w/IC-DiT	Synth	0.5970 $\pm$ 0.0319	0.5891 $\pm$ 0.0274	0.7613 $\pm$ 0.0482	0.5933 $\pm$ 0.0144	0.6318 $\pm$ 0.0337
w/IC-DiT	Real+Synth	0.6694 $\pm$ 0.0456	0.6440 $\pm$ 0.0734	0.8702 $\pm$ 0.0173	0.6636 $\pm$ 0.0556	0.7151 $\pm$ 0.0659

Table 5: Comparison of survival prediction models trained with and without synthetic pathology data. C-index (mean

\pm

std) is reported across five histopathology datasets under three training settings: (1) real data only, (2) synthetic data only and (3) real data with one-fold synthetic augmentation. Best results are highlighted in bold .

Methods	Train Data	BLCA	BRCA	GBMLGG	LUAD	UCEC	Avg.
Methods	Train Data	Datasets
ViLa-MIL	Real	84.59	81.06	90.35	82.06	91.52	85.92
w/PathDiff	Synth	78.41	74.53	85.99	76.28	87.65	80.57
w/PathDiff	Real+Synth	85.26	82.34	90.69	83.44	91.80	86.71
w/IC-DiT	Synth	80.48	77.13	87.49	79.05	89.08	82.66
w/IC-DiT	Real+Synth	88.10	84.73	91.58	84.13	92.30	88.17

Table 6: Performance comparison of the classification model on downstream tasks. Accuracy is reported across five histopathology datasets under three training settings: (1) real data only, (2) synthetic data only and (3) real data with one-fold synthetic augmentation. We highlight the best performing scores in bold .

Image Generation

Image generation studies models that create or edit images from text, examples, layouts, and other structured controls.

See all 463 papers

Jan 13, 2026

Controllable Diffusion-Based Lesion Inpainting for Scalable Histopathology Data Augmentation

Expert-annotated training data remains the critical bottleneck for AI in histopathology, particularly for rare pathologies where even dozens of cases may be unavailable. While data augmentation offers a solution, existing methods fail to generate sufficiently realistic lesion morphologies that preserve tissue-specific architectures. Here we present PathoGen, a diffusion-based generative model enabling controllable, high-fidelity lesion inpainting into benign histopathology images. We validate PathoGen across four datasets representing kidney, skin, breast, and prostate pathology. Quantitative assessment confirms PathoGen outperforms state-of-the-art baselines in image fidelity and distributional similarity. Evaluation by six expert pathologists revealed that synthetic images by PathoGen were only marginally distinguished from real tissue image slightly above chance (57.75% accuracy), demonstrating strong perceptual realism of PathoGen-generated lesions. PathoGen achieved the highest win rate (35.4%) when pathologists ranked generation quality against all baselines. Crucially, augmenting training sets with PathoGen-synthesized lesions improves segmentation Dice scores by up to 0.18 compared to traditional augmentations, with maximum benefit in data-scarce regimes. By simultaneously generating realistic morphology and pixel-level annotations, PathoGen effectively addresses both data scarcity and annotation cost, two critical bottlenecks in computational pathology development.

Mohamad Koohi-Moghadam, Mohammad-Ali Nikouei Mahani, Rex K. H. Au-Yeung +6

Department of Diagnostic Radiology, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Pok Fu Lam, Hong Kong SAR, PR China · Department of Pathology, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Hong Kong SAR, China · Department of Anatomical and Cellular Pathology, Prince of Wales Hospital, The Chinese University of Hong Kong, Hong Kong SAR, China. +2

Computational Pathology Synthetic Data Augmentation Diffusion-Based Image Generation Image Inpainting Medical Image Synthesis

Ask PDF

Mar 4, 2026

Structure-Guided Histopathology Synthesis via Dual-LoRA Diffusion

Histopathology image synthesis plays an important role in tissue restoration, data augmentation, and modeling of tumor microenvironments. However, existing generative methods typically address restoration and generation as separate tasks, although both share the same objective of structure-consistent tissue synthesis under varying degrees of missingness, and often rely on weak or inconsistent structural priors that limit realistic cellular organization. We propose Dual-LoRA Controllable Diffusion, a unified centroid-guided diffusion framework that jointly supports Local Structure Completion and Global Structure Synthesis within a single model. Multi-class nuclei centroids serve as lightweight and annotation-efficient spatial priors, providing biologically meaningful guidance under both partial and complete image absence. Two task-specific LoRA adapters specialize the shared backbone for local and global objectives without retraining separate diffusion models. Extensive experiments demonstrate consistent improvements over state-of-the-art GAN and diffusion baselines across restoration and synthesis tasks. For local completion, LPIPS computed within the masked region improves from 0.1797 (HARP) to 0.1524, and for global synthesis, FID improves from 225.15 (CoSys) to 76.04, indicating improved structural fidelity and realism. Our approach achieves more faithful structural recovery in masked regions and substantially improved realism and morphology consistency in full synthesis, supporting scalable pan-cancer histopathology modeling.

Xuan Xu, Prateek Prasanna

Stony Brook University

Medical Image Synthesis Conditional Diffusion Models Image Inpainting

Ask PDF

Jun 5, 2026

STREAM: Stochastic Riemannian Flow Matching with Anisotropic Decoder for Digital Histopathology Image Generation

Synthetic histopathology image generation addresses patient-privacy concerns and the growing data demands of foundation models. Existing state-of-the-art histopathology generative models use pretrained Vision Foundation Models (VFMs) as conditioning signals. We show this yields conditioning-dominated diversity: on TCGA-BRCA, 62-75% of their output diversity is attributable to the conditioning signal rather than the learned latent space, while de novo synthesis still requires a VFM at inference. We instead use histopathology VFMs as the latent space itself: their patch tokens are

\ell_2

-normalized on the unit hypersphere

\mathcal{S}^{d-1}

with strong angular dominance and intrinsic curvature, motivating a Riemannian formulation. We present STREAM, the first framework to apply Riemannian flow matching in the histopathology domain, in two stages: 1) a bridge-type stochastic perturbation that establishes per-token rectifiability on

\mathcal{S}^{d-1}

for training a Diffusion Transformer, and 2) a novel decoder training design whose noise covariance is anisotropic in the left-singular basis of the per-token tangent-projected velocity-field Jacobian, spending a large robustness budget on its low-response directions and a small one on its high-response directions. Across TCGA-BRCA and TCGA-COADREAD, STREAM achieves state-of-the-art gFID and ranks first on nearly all histopathology-specific metrics as well. Code and a public gallery of generated images are available at https://chokevin8.github.io/STREAM-Patho/.

Won June Cho, Daeky Jeong, Hyeongyeol Lim +1

DEEPNOID Inc.

Flow Matching Diffusion Transformer Riemannian Flow Matching Medical Image Synthesis

Ask PDF

Abstract

Image Generation

Explore similar work

Controllable Diffusion-Based Lesion Inpainting for Scalable Histopathology Data Augmentation

Structure-Guided Histopathology Synthesis via Dual-LoRA Diffusion

STREAM: Stochastic Riemannian Flow Matching with Anisotropic Decoder for Digital Histopathology Image Generation