VGL-stGen: Learning a shared multimodal model for lung histology and spatial transcriptomics
Organizations: Portrai, Inc., Seoul, Republic of Korea
Abstract
VGL-stGen is a multimodal large language model adapted from a pretrained medical vision-language model using paired lung H&E and spatial transcriptomics with single-cell references. Training integrates image-expression and expression-cell-type pairs, and gene-conditioned pseudo-labels transfer transcriptomic cell-type supervision to histology images. One deployed checkpoint supports thirteen evaluated tasks spanning tissue and cell-type classification, gene prediction and conditional expression completion from combinations of histology, cell identity and observed gene expression. Task-specific decoding produces categorical labels, gene lists and structured expression levels. For conditional completion, fixing queried gene identities and predicting their levels autoregressively improves structured prediction. Controlled adaptation shows that restricting supervision to requested expression-level tokens can further improve image-conditioned completion. Image-permutation experiments indicate that correctly paired histology adds information beyond observed expression. External classification relies partly on computational references with shared annotation lineage, and some H&E sources were exposed during cancer training or model development. Within these conditions, VGL-stGen demonstrates how a medical vision-language backbone can learn from spatial transcriptomics and histology to connect tissue morphology and molecular states through shared generative modeling.
Figures & tables
| ID | Input output | Metric | Evaluation condition |
|---|---|---|---|
| 1 | Genes cell type | BA / F1m | External |
| 2 | Image + genes cell type | BA | H&E overlap |
| 3 | Image cell type | BA | H&E overlap |
| 4 | Image tumour | BA | 452 slides |
| 5 | Image tissue | F1m | 39-slide pool |
| 6 | Image genes | F1g | H&E overlap |
| Cohort | Evaluated units | Reference / exposure |
|---|---|---|
| Visium | 9,964 spots / 2 sections | Transferred cell types; one section supplied H&E for cancer training |
| Xenium | 214,246 cells / 2 sections | Transferred cell types; both sections supplied H&E for cancer training |
| TCGA-LUAD binary | 357,692 patches / 452 slides / 398 patients | ARA-CNN labels; includes development subset |
| TCGA tissue classification | 30,719 patches / 39 slides | Development subset; pooled transductive correction |
| Masked expression benchmark | 1,000 rows/task | Internal; 7 image samples / 1 cell-type sample identifier |
| Model | T1 BA / F1m | T2 BA | T3 BA | T4 BA | T5 F1m |
|---|---|---|---|---|---|
| VGL-stGen | 0.6293 / 0.6023 | 0.5394 | 0.2012 | 0.7524 | 0.1200 0.2801 |
| MedGemma-1.5-4B | 0.1836 / 0.1576 | 0.2070 | 0.1105 | 0.5954 | 0.0911 0.1711 |
| Gemma3-4B | 0.2254 / 0.1738 | 0.1853 | 0.1250 | 0.6637 | 0.1334 0.1347 |
| Qwen3-VL-8B | 0.4662 / 0.3148 | 0.4572 | 0.0995 | 0.5182 | 0.1102 0.1630 |
| DeepSeek-VL2-small | 0.1314 / 0.1140 | 0.1380 | 0.1100 | 0.5351 | 0.0649 0.1321 |
| GLM-4.6V-Flash | 0.3444 / 0.2775 | 0.3160 | 0.1008 | 0.5808 | 0.1411 0.1733 |
| Model | T6 | T7 | T8 | T9 | T10 | T11 | T12 | T13 |
|---|---|---|---|---|---|---|---|---|
| VGL-stGen | 0.1173 | 0.0965 | 0.0604 | 0.0484 | 0.2828 | 0.6596 | 0.6858 | 0.7420 |
| MedGemma-1.5-4B | 0.0000 | 0.0005 | 0.0000 | 0.0000 | 0.0003 | 0.5733 | 0.5540 | 0.0000 |
| Gemma3-4B | 0.0088 | 0.0345 | 0.0108 | 0.0000 | 0.0058 | 0.5072 | 0.4996 | 0.3500 |
| Qwen3-VL-8B | 0.0000 | 0.0864 | 0.0210 | 0.0380 | 0.0032 | 0.1634 | 0.2372 | 0.3500 |
| DeepSeek-VL2-small | 0.0000 | 0.0080 | 0.0026 | 0.0000 | 0.0042 | 0.1600 | 0.2165 | 0.0630 |
| GLM-4.6V-Flash | 0.0044 | 0.0397 | 0.0140 | 0.0237 | 0.0030 | 0.3662 | 0.3627 | 0.4260 |
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| Variant | Rows | Source |
|---|---|---|
| Genes cell type | 116,278 | Single-cell atlas |
| Cell type genes | 115,207 | Single-cell atlas |
| Image genes | 1,047,500 | Spatial samples; stain-expanded |
| Task | Model | Level macro-F1 | Non-Absent accuracy |
|---|---|---|---|
| Image | Deployed | 0.3780 | 0.2276 |
| Image | Full-answer | 0.4269 0.0024 | 0.2664 0.0038 |
| Image | Level-only | 0.4463 0.0015 | 0.2930 0.0021 |
| Image | Full-input MLP | 0.4586 0.0021 | 0.3160 0.0011 |
| Cell type | Deployed | 0.4625 | 0.3611 |
| Cell type | Full-answer | 0.5445 0.0008 | 0.4433 0.0049 |
| Evaluation | Seed | Full-answer | Level-only | Difference |
|---|---|---|---|---|
| Image pooled | 42 | 0.67096 | 0.68032 | +0.00936 |
| Image pooled | 7 | 0.67252 | 0.68164 | +0.00912 |
| Image pooled | 2026 | 0.67252 | 0.67948 | +0.00696 |
| Xenium | 42 | 0.71717 | 0.72523 | +0.00806 |
| Xenium | 7 | 0.71705 | 0.72493 | +0.00788 |
| Xenium | 2026 | 0.71735 | 0.72343 | +0.00608 |
| Supervision | Seed | Intact | Permuted | Difference |
|---|---|---|---|---|
| Full-answer | 42 | 0.67096 | 0.66592 | +0.00504 |
| Full-answer | 7 | 0.67252 | 0.66620 | +0.00632 |
| Full-answer | 2026 | 0.67252 | 0.66596 | +0.00656 |
| Level-only | 42 | 0.68032 | 0.67236 | +0.00796 |
| Level-only | 7 | 0.68164 | 0.67452 | +0.00712 |
| Level-only | 2026 | 0.67948 | 0.67128 | +0.00820 |
| Obs. | Model / image | Pair-F1 | Level F1m | Non-Absent acc. |
|---|---|---|---|---|
| 75 | VGL-stGen / Intact | 0.6803 | 0.4466 | 0.2930 |
| 75 | VGL-stGen / Permuted | 0.6724 | 0.4308 | 0.2727 |
| 75 | Image MLP / Intact | 0.6811 0.0038 | 0.4586 0.0021 | 0.3160 0.0011 |
| 75 | Image MLP / Permuted | 0.6700 0.0030 | 0.4375 0.0005 | 0.2903 0.0026 |
| 75 | Context MLP / — | 0.6695 0.0092 | 0.4524 0.0086 | 0.3187 0.0055 |
| 50 | VGL-stGen / Intact | 0.6681 | 0.4173 | 0.2593 |
| Tasks | Items per task | Output limit | Batch / shards |
|---|---|---|---|
| 1 | 224,210 cells/spots | 8 tokens | 16 / 4 |
| 2–3 | 224,210 cells/spots | 16 tokens | 16 mixed / 4 |
| 4 | 357,692 patches / 452 slides | 16 tokens | 32 / 8 slide shards |
| 5 | 30,719 patches / 39 slides | 16 tokens | 32 |
| 6–7 | 1,000 rows | 160 tokens | 8 / 4 |
| 8–9 | 1,000 rows | 56 tokens | 8 / 4 |
| Model | Repository | Image processing |
|---|---|---|
| VGL-stGen | google/medgemma-1.5-4b-it | 896 896; mean/std 0.5 |
| MedGemma-1.5-4B | google/medgemma-1.5-4b-it | 896 896; mean/std 0.5 |
| Gemma3-4B | google/gemma-3-4b-it | 896 896; mean/std 0.5 |
| Qwen3-VL-8B | Qwen/Qwen3-VL-8B-Instruct | Dynamic resolution; 16-pixel patches; merge 2 |
| Qwen3-VL-32B | Qwen/Qwen3-VL-32B-Instruct | Dynamic resolution; 16-pixel patches; merge 2 |
| Gemma4-12B | google/gemma-4-12B-it | Variable resolution; 16-pixel patches; max 280 soft tokens |
| Task | Comparator | Difference | 95% interval |
|---|---|---|---|
| 1 | Gemma4-31B | +0.1258 | [+0.1204, +0.1309] |
| 2 | Qwen3-VL-32B | +0.0654 | [+0.0598, +0.0709] |
| 3 | Gemma4-31B | +0.0360 | [+0.0332, +0.0386] |
| 4 | Gemma4-12B | +0.0538 | [+0.0491, +0.0587] |
| 5 | Qwen3.8-27B | +0.0188 | [+0.0133, +0.0238] |
| 6 | Gemma4-31B | +0.0617 | [+0.0568, +0.0665] |
| Model | T6 | T7 | T8 | T9 | T10 | T13 |
|---|---|---|---|---|---|---|
| VGL-stGen | 0.1173 | 0.0965 | 0.0604 | 0.0484 | 0.2828 | 0.7420 |
| MedGemma-1.5-4B | 0.0005 | 0.0140 | 0.0065 | 0.0141 | 0.0011 | 0.3620 |
| Gemma3-4B | 0.0186 | 0.0625 | 0.0088 | 0.0180 | 0.0057 | 0.3530 |
| Qwen3-VL-8B | 0.0024 | 0.0850 | 0.0210 | 0.0380 | 0.0032 | 0.3500 |
| DeepSeek-VL2-small | 0.0000 | 0.0103 | 0.0020 | 0.0028 | 0.0043 | 0.0630 |
| GLM-4.6V-Flash | 0.0082 | 0.0501 | 0.0135 | 0.0262 | 0.0040 | 0.4220 |
| Model / supervision | Image | Cell type |
|---|---|---|
| Deployed checkpoint | 0.6596 | 0.6858 |
| Full-answer adaptation | 0.6720 0.0009 | 0.7221 0.0007 |
| Level-only adaptation | 0.6805 0.0011 | 0.7201 0.0004 |
| Full-input MLP | 0.6811 0.0038 | 0.7300 0.0008 |
| Readout | Balanced accuracy | Macro-F1 | Trainable parameters |
|---|---|---|---|
| Discriminative short run | 0.5554 | 0.5209 | 32,819,212 |
| Generative short run | 0.5737 | 0.5683 | 32,788,480 |
| ID | Task | Score | Metric | Condition |
|---|---|---|---|---|
| 1 | Genes cell type | 0.6293 / 0.6023 | BA / F1m | External |
| 2 | Image + genes cell type | 0.5394 | BA | H&E overlap |
| 3 | Image cell type | 0.2012 | BA | H&E overlap |
| 4 | Image tumour | 0.7532 | BA | 452 slides |
| 5 | Image tissue | 0.1200 0.2801 | F1m | 39-slide pool |
| 6 | Image genes | 0.1173 | F1g | H&E overlap |
Gene Expression Prediction from Histopathology Images
Gene expression prediction from histopathology images links H&E tissue morphology to molecular profiles, enabling virtual spatial transcriptomics without sequencing every sample.
See all 45 papers