GESTALT: Fast, Generalizable Histopathology Image Synthesis from Gene Expression with Pretrained Pathology Foundation Models
Organizations: Stony Brook University, Stony Brook, NY
Abstract
Spatial transcriptomics measures gene expression while preserving where each cell sits in the tissue. The inverse, what a given transcriptomic state looks like, is less explored but equally useful: it lets researchers visualize the morphological consequence of a molecular state and interpret a spatial assay's molecular calls against real tissue structure. GeneFlow, the current state of the art on the Xenium gene expression-to-image benchmark, trains a model from scratch, incurring a high cost on every new target dataset it is applied to. We instead propose GESTALT, which reformulates gene expression-to-image synthesis as predicting a pathology foundation model's image embedding. A lightweight gene-token transformer predicts this embedding from multi-cell gene expression, and a frozen or LoRA-adapted diffusion decoder (PixCell) renders it as an image. Because this visual knowledge is reused rather than relearned, GESTALT trains in a fraction of the time. In both multi-cell and single-cell settings, our method outperforms the SoTA on perceptual metrics across all three Xenium datasets. Beyond improved image quality, GESTALT natively synthesizes at 1024 x 1024, and is evaluated in three generalization settings: zero-shot transfer across gene panels and datasets, held-out patients and disease states, and transfer across studies and tissue sources. Coupled with a statistical model of spatial expression, GESTALT also provides, to our knowledge, the first paired synthesis of cell-resolved transcriptomic fields and tissue images with explicit control over cell type, density, and composition. Code is available at https://github.com/trongnhandev1011/Gestalt.
Figures & tables
| FID | SSIM | FeatDist | embCOS | |
| GeneFlow [ 27 ] (multi-cell, reported, ) | ||||
| Xenium C1 | 47.95 | 0.28 | 14.50 | – |
| Xenium C2 | 45.50 | 0.40 | 15.61 | – |
| Xenium P1 | 34.31 | 0.23 | 13.41 | – |
| Ours (LoRA, ) | ||||
| Xenium C1 | 15.75 | 0.350 | 12.66 | 0.838 |
| Direction | GeneFlow | Gestalt | Oracle | ||
|---|---|---|---|---|---|
| FID | FeatDist | FID | FeatDist | FID | |
| C1 C2 | 112.25 | 17.18 | 62.71 | 16.23 | 15.24 |
| C2 C1 | 146.78 | 18.36 | 138.65 | 18.08 | 25.65 |
| C2 P1 | 148.15 | 17.59 | 89.18 | 16.47 | 19.54 |
| P1 C2 | 127.79 | 17.27 | 75.24 | 15.62 | 15.04 |
| C1 P1 | 79.86 | 15.80 | 56.62 | 16.53 | 11.52 |
| FID | SSIM | FeatDist | |
| NCBI882 (held-out diseased section) | |||
| Gestalt | 38.22 | 0.162 | 12.69 |
| Oracle | 28.64 | 0.168 | 11.23 |
| NCBI861 (held-out diseased section) | |||
| Gestalt | 59.37 | 0.176 | 13.07 |
| Oracle | 45.09 | 0.180 | 11.62 |
| Pred vs. GT | Oracle vs. GT | |||
|---|---|---|---|---|
| Crop FID | CLIP FID | Crop FID | CLIP FID | |
| Xenium C1 | 71.59 | 5.29 | 59.12 | 3.62 |
| Xenium C2 | 50.87 | 3.33 | 39.20 | 2.08 |
| Xenium P1 | 49.87 | 2.98 | 47.92 | 2.91 |
| Params (M) | ||||
|---|---|---|---|---|
| Method | Wall-clock / | |||
| GeneFlow [ 27 ] | 51.0 | 53.7 | 178.5 | 12h / – |
| Gestalt | ||||
| Mapper | 7.7 | 7.7 | 9.5 | 55–80 / 90–142 min |
| Decoder LoRA | 26.8 | 26.8 | 26.8 | 34 / 34 min |
| Total | 34.5 | 34.5 | 36.3 | 89–114 / 124–176 min |
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| FID | SSIM | FeatDist | |
| GeneFlow [ 27 ] (single-cell, reported) | |||
| Xenium C1 | |||
| Xenium C2 | |||
| Xenium P1 | |||
| Gestalt (ours, LoRA, single-cell) | |||
| Xenium C1 | |||
| FID | SSIM | FeatDist | |
|---|---|---|---|
| Xenium C1 | 28.7 | 0.22 | 12.6 |
| Xenium C2 | 29.8 | 0.26 | 12.0 |
| Xenium P1 | 24.3 | 0.19 | 12.1 |
| Dataset | Conditioning | FID | SSIM | FeatDist | embCOS |
|---|---|---|---|---|---|
| Xenium C1 | GeneFlow | – | |||
| Ours | |||||
| Oracle | 14.27 0.06 | 0.3525 0.0005 | 11.94 0.01 | – | |
| Xenium C2 | GeneFlow | – | |||
| Ours | |||||
| Oracle | 14.75 0.04 | 0.3996 0.0006 | 11.69 0.05 | – |
| Crop FID | CLIP FID | ||
| Xenium C1 | 75.96 | 8.54 | |
| 71.59 | 5.29 | ||
| 66.37 | 5.84 | ||
| Oracle | 59.12 | 3.62 | |
| Xenium C2 | 63.93 | 5.70 | |
| 50.87 | 3.33 |
| FID | SSIM | FeatDist | embCOS | |
| NCBI884 (unseen healthy patient) | ||||
| Gestalt | 45.86 | 0.203 | 13.70 | 0.732 |
| Oracle | 30.07 | 0.221 | 11.98 | – |
| NCBI875 (unseen healthy patient) | ||||
| Gestalt | 89.65 | 0.313 | 14.31 | 0.659 |
| Oracle | 47.08 | 0.324 | 11.91 | – |
| FID | SSIM | FeatDist | |
| NCBI884 (unseen healthy patient) | |||
| Constant mean | 86.47 | 0.142 | 15.06 |
| Permuted prediction | 45.86 | 0.165 | 15.24 |
| Top-10 retrieval | 69.53 | 0.139 | 14.59 |
| Gestalt | 45.86 | 0.203 | 13.70 |
| Oracle (reference) | 30.07 | 0.221 | 11.98 |
| FID | SSIM | FeatDist | |
| NCBI884 (unseen healthy patient) | |||
| No positional embedding | 48.50 | 0.182 | 14.13 |
| No graph attention | 46.57 | 0.218 | 14.02 |
| Mean-pool MLP | 68.51 | 0.224 | 14.56 |
| Detected-mean MLP | 69.28 | 0.160 | 14.34 |
| Gestalt | 45.86 | 0.203 | 13.70 |
| NCBI875 | NCBI884 | |||
|---|---|---|---|---|
| FID | SSIM | FID | SSIM | |
| 4 | 107.75 | 0.316 | 45.65 | 0.210 |
| 8 | 89.65 | 0.313 | 45.86 | 0.203 |
| 16 | 97.36 | 0.316 | 47.73 | 0.206 |
| 32 | 90.30 | 0.306 | 48.40 | 0.202 |
| NCBI875 | NCBI884 | |||
|---|---|---|---|---|
| FM | FID | SSIM | FID | SSIM |
| Virchow2 | 101.58 | 0.318 | 44.16 | 0.207 |
| UNI-v1 | 111.19 | 0.310 | 51.12 | 0.213 |
| CONCH | 86.67 | 0.321 | 45.56 | 0.206 |
| UNI2-h | 89.65 | 0.313 | 45.86 | 0.203 |
| Setting | Overlap | % | Chance % | ||
|---|---|---|---|---|---|
| Lung | 50 | 43/50 | 86 | 15 | |
| (343 genes) | 100 | 90/100 | 90 | 29 | |
| Cross-study | 50 | 40–46/50 | 80–92 | 18 | |
| (280 genes) | 100 | 86–98/100 | 86–98 | 36 |
| Requested | Generated | Real tile | Real |
|---|---|---|---|
| (CellViT) | (CellViT) | ||
| 19 | 32 | 19 | 14 |
| 28 | 38 | 24 | 25 |
| 38 | 61 | 38 | 57 |
| 48 | 67 | 48 | 61 |
| 66 | 79 | 66 | 72 |
| Gestalt | Oracle | |||||||
|---|---|---|---|---|---|---|---|---|
| Test slide | Patient | FID | SSIM | FeatDist | embCOS | FID | SSIM | FeatDist |
| NCBI783 | 5 | 97.06 | 0.313 | 17.54 | 0.479 | 28.28 | 0.287 | 13.80 |
| NCBI784 | 4 | 103.35 | 0.318 | 16.08 | 0.408 | 40.83 | 0.363 | 14.23 |
| NCBI785 | 4 | 105.59 | 0.376 | 16.36 | 0.403 | 28.75 | 0.428 | 13.19 |
Medical Image Synthesis
Medical image synthesis creates, translates, reconstructs, and edits clinical images to fill missing data and support analysis.
See all 331 papers