Anatomy-Activated Mixture-of-Experts for 3D Medical Vision-Language Pre-training
Organizations: Jagiellonian University · Helmholtz Munich · Technical University of Munich · Munich Center for Machine Learning ·
Abstract
Medical Vision-Language Pre-training (VLP) leverages the semantic richness of radiology reports to learn 3D representations, yet current architectures remain fundamentally anatomy-agnostic. While clinical diagnostics rely on organ-specific context, standard encoders apply a uniform, dense parameterization to all volumetric patches, and existing Mixture-of-Experts (MoE) routers gate on isolated local features without explicit access to the organ composition of the scan or crop. We propose Anatomy-Activated Mixture-of-Experts (AMoE), a framework that incorporates anatomical inductive biases within the model's computational path. AMoE introduces two key components: (1) a Histogram Representation Router (HRR) that conditions expert selection on global visual context and a predicted anatomical composition histogram, encouraging anatomy-dependent expert weighting; and (2) an Organ-Query Multi-Scale Projector (OQMSP) that utilizes learnable queries to extract organ-level embeddings for fine-grained alignment without requiring manual annotations at inference. By routing at the patch level rather than the token level, AMoE reduces routing complexity by orders of magnitude while satisfying load balancing by construction. Pre-trained on over 60,000 multi-phase CT scans, AMoE achieves state-of-the-art performance across segmentation, classification, and detection benchmarks. Furthermore, we demonstrate that our anatomically grounded inductive biases support robust generalization to MRI and ultrasound, establishing a scalable foundation for medical representation learning. The code is available at https://github.com/gmum/A2MoE.
Figures & tables
| Method | AMOS-CT [ 39 ] | PANORAMA [ 2 ] | AMOS-MRI [ 39 ] | GEPAR3D [ 80 ] | MRISegmenter [ 98 ] | MVSeg [ 19 ] | ||||||||
| Params | VRAM ‡ | DSC | HD95 | DSC | HD95 | DSC | HD95 | DSC | HD95 | DSC | HD95 | DSC | HD95 | |
| VoCo † [ 84 , 85 ] | 53.0 | 4.4 | 87.1 | 16.4 | 71.6 | 15.9 | 80.1 | 10.5 | 83.5 | 6.8 | 73.0 | 7.7 | 84.3 | 5.1 |
| uC 3DU-Net [ 37 ] | 21.7 | 3.1 | 83.1 | 16.0 | 76.6 | 8.4 | 74.4 | 16.9 | 83.8 | 12.5 | 76.7 | 5.9 | 83.9 | 4.7 |
| SuPreM † [ 50 ] | 62.2 | 4.4 | 87.1 | 7.7 | 76.6 | 4.7 | 75.8 | 19.5 | 80.6 | 11.0 | 70.8 | 8.4 | 84.3 | 4.9 |
| Mamba-HoME [ 68 ] | 170.1 | 3.7 | 86.3 | 18.7 | 77.5 | 4.8 | 77.7 | 9.9 | 86.1 | 6.5 | 75.2 | 6.5 | 84.5 | 4.8 |
| Merlin † [ 13 ] | 252.3 | 3.0 | 85.6 | 29.8 | 77.5 | 9.4 | 79.1 | 10.4 | 86.5 | 2.7 | 75.8 | 6.0 | 84.1 | 3.9 |
| Config | AMOS-CT | AMOS-MRI | GEPAR3D | MVSeg |
| MoE (Baseline) | 86.0 (14.4) | 77.4 (14.5) | 85.6 (9.8) | 84.0 (5.0) |
| 87.0 (12.0) | 77.9 (14.0) | 85.8 (8.5) | 84.3 (4.8) | |
| 87.5 (7.6) | 78.5 (12.2) | 86.1 (7.9) | 84.5 (4.4) |
| # | Experts | Params | mDSC (%) | |||||
| A-CT | PANO | A-MRI | GEP | MRISeg | MVSeg | |||
| 1 | [4, 4, 4, 4] | 83.8 | 87.5 | 77.9 | 78.0 | 85.9 | 77.7 | 84.8 |
| 2 | [8, 8, 8, 8] | 87.0 | 87.3 | 75.1 | 77.9 | 85.0 | 77.5 | 85.0 |
| 3 | [4, 8, 16, 32] | 102.3 | 87.5 | 77.4 | 77.4 | 86.1 | 77.6 | 84.5 |
| 4 | [16, 16, 16, 16] | 93.3 | 86.9 | 78.2 | 78.5 | 86.2 | 77.4 | 84.9 |
Appendix figures & tables34 assets
Supplementary material from the paper’s appendix.
Appendix
| Class | Anatomy |
| 0 | Background |
| 1 | Spleen |
| 2 | Left kidney |
| 3 | Right kidney |
| 4 | Gallbladder |
| 5 | Liver |
| Dataset | Scans | Exams | Patients |
| Merlin [ 13 ] | 25,494 | 25,494 | 18,317 |
| CancerVerse [ 49 ] | 24,422 | 22,344 | 14,599 |
| Proprietary set | 17,415 | 5,805 | 4,004 |
| Total | 67,331 | 53,643 | 37,989 |
| Config | Organ Classes (DSC %) | Overall | ||||||||||||||||||
| # | B | MoE | T | Spl | RK | LK | GB | Eso | Liv | Sto | Aor | IVC | Pan | RAG | LAG | Duo | Bla | P/U | mDSC | mHD95 |
| 1 | ✓ | ✗ | ✗ | 95.0 | 95.5 | 95.4 | 75.5 | 80.7 | 96.2 | 88.2 | 93.5 | 88.9 | 82.7 | 75.1 | 73.6 | 75.6 | 86.0 | 80.1 | 85.5 | 17.3 |
| 2 | ✓ | ✓ | ✗ | 95.8 | 95.9 | 94.9 | 82.7 | 83.6 | 96.8 | 91.5 | 94.1 | 90.0 | 85.6 | 75.1 | 75.5 | 79.7 | 89.3 | 81.6 | 87.5 | 7.6 |
| 3 | ✓ | ✓ | ✓ | 96.7 | 96.1 | 95.6 | 84.8 | 84.9 | 97.7 | 92.2 | 94.3 | 90.6 | 87.3 | 77.0 | 77.6 | 82.9 | 89.5 | 83.6 | 88.7 | 6.2 |
| Config | NIH | MSD Pancreas | Proprietary | Overall | ||||||
| # | B | MoE | T | Pancreas | PDAC | Pancreas | PDAC | Pancreas | mDSC | mHD95 |
| 1 | ✓ | ✗ | ✗ | 92.7 | 47.9 | 88.1 | 54.0 | 81.9 | 76.0 | 8.6 |
| 2 | ✓ | ✓ | ✗ | 92.0 | 53.6 | 87.7 | 65.8 | 81.4 | 78.2 | 7.1 |
| 3 | ✓ | ✓ | ✓ | 93.1 | 55.2 | 89.1 | 67.2 | 82.2 | 79.1 | 5.8 |
| Config | Organ Classes (DSC %) | Overall | ||||||||||||||||
| # | B | MoE | T | Spl | RK | LK | GB | Eso | Liv | Sto | Aor | IVC | Pan | RAG | LAG | Duo | mDSC | mHD95 |
| 1 | ✓ | ✗ | ✗ | 86.4 | 89.8 | 89.6 | 62.5 | 70.5 | 95.7 | 82.4 | 87.0 | 81.0 | 83.5 | 59.8 | 64.2 | 64.4 | 76.7 | 9.3 |
| 2 | ✓ | ✓ | ✗ | 91.0 | 90.6 | 90.7 | 67.3 | 72.3 | 96.5 | 81.4 | 87.5 | 83.4 | 80.6 | 57.4 | 59.5 | 62.8 | 78.5 | 12.2 |
| 3 | ✓ | ✓ | ✓ | 91.5 | 90.9 | 91.2 | 78.9 | 75.0 | 97.4 | 86.8 | 87.6 | 84.4 | 86.5 | 64.5 | 67.0 | 68.1 | 82.3 | 8.9 |
| Config | Upper Right (Q1) | Upper Left (Q2) | Lower Left (Q3) | Lower Right (Q4) | Overall | ||||||||||||||||||||||||||||||||
| # | B | MoE | T | 18 | 17 | 16 | 15 | 14 | 13 | 12 | 11 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 38 | 37 | 36 | 35 | 34 | 33 | 32 | 31 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | mDSC | mHD95 |
| 1 | ✓ | ✗ | ✗ | 68.0 | 87.9 | 90.2 | 85.3 | 85.2 | 91.7 | 91.0 | 93.7 | 93.1 | 90.9 | 89.2 | 86.1 | 90.0 | 93.0 | 85.3 | 70.6 | 66.9 | 78.3 | 84.5 | 89.4 | 89.4 | 85.4 | 87.2 | 86.0 | 80.3 | 89.1 | 92.6 | 89.1 | 93.3 | 84.8 | 81.4 | 56.6 | 85.3 | 10.6 |
| 2 | ✓ | ✓ | ✗ | 67.9 | 87.6 | 90.8 | 88.0 | 85.6 | 91.4 | 89.3 | 93.0 | 93.3 | 91.7 | 89.4 | 84.0 | 92.0 | 92.4 | 91.4 | 74.0 | 68.1 | 88.2 | 88.6 | 91.4 | 88.6 | 84.7 | 85.4 | 80.7 | 83.5 | 88.9 | 91.4 | 88.6 | 92.2 | 88.7 | 83.2 | 59.8 | 86.2 | 8.3 |
| 3 | ✓ | ✓ | ✓ | 70.4 | 92.2 | 89.1 | 88.3 | 91.0 | 93.5 | 90.4 | 93.6 | 91.5 | 91.7 | 88.7 | 87.4 | 91.3 | 94.2 | 90.7 | 76.7 | 64.3 | 87.1 | 87.5 | 89.8 | 91.1 | 86.5 | 89.5 | 86.0 | 84.5 | 88.5 | 87.6 | 91.2 | 93.8 | 90.0 | 88.3 | 64.3 | 87.7 | 8.1 |
| Config | Anatomical Classes (DSC %) | |||||||||||||||||||||||||||||||||
| # | B | MoE | T | 1 Spleen | 2 Kid R | 3 Kid L | 4 Gallb | 5 Liver | 6 Esoph | 7 Stom | 8 Aorta | 9 IVC | 10 P&S Vn | 11 Pancr | 12 Adr R | 13 Adr L | 14 Lung R | 15 Lung L | 16 Sm Bow | 17 Duod | 18 Colon | 19 Ilia Art L | 20 Ilia Art R | 21 Ilia Vn L | 22 Ilia Vn R | 23 GluMax L | 24 GluMax R | 25 GluMed L | 26 GluMed R | 27 Auto L | 28 Auto R | 29 Ilio L | 30 Ilio R | 31 Hip L |
| 1 | ✓ | ✗ | ✗ | 90.7 | 96.4 | 86.6 | 73.6 | 98.4 | 84.7 | 92.6 | 95.2 | 91.9 | 61.7 | 87.1 | 60.2 | 58.0 | 97.1 | 96.1 | 91.0 | 82.6 | 89.8 | 75.9 | 80.2 | 67.1 | 59.7 | 63.5 | 60.4 | 84.6 | 78.9 | 97.3 | 97.1 | 95.3 | 95.8 | 80.5 |
| 2 | ✓ | ✓ | ✗ | 91.5 | 96.3 | 91.5 | 78.6 | 98.5 | 85.4 | 93.6 | 95.5 | 93.1 | 64.0 | 88.1 | 61.0 | 59.2 | 97.5 | 96.6 | 92.2 | 84.2 | 91.6 | 76.8 | 77.1 | 66.5 | 58.2 | 70.0 | 59.1 | 86.8 | 79.2 | 97.5 | 97.4 | 95.5 | 96.4 | 83.4 |
| 3 | ✓ | ✓ | ✓ | 92.8 | 96.7 | 93.0 | 83.1 | 98.6 | 85.6 | 94.4 | 95.4 | 93.2 | 63.7 | 89.3 | 61.3 | 60.1 | 97.6 | 96.7 | 93.9 | 84.7 | 93.7 | 81.6 | 83.9 | 78.4 | 78.1 | 79.4 | 78.5 | 90.7 | 87.8 | 97.5 | 97.4 | 95.8 | 96.5 | 86.6 |
| Config | Organ Classes (DSC %) | Overall | |||||
| # | B | MoE | T | PL | AL | mDSC | mHD95 |
| 1 | ✓ | ✗ | ✗ | 83.0 | 84.7 | 83.8 | 5.8 |
| 2 | ✓ | ✓ | ✗ | 84.2 | 85.6 | 84.9 | 4.3 |
| 3 | ✓ | ✓ | ✓ | 84.5 | 86.0 | 85.4 | 4.0 |
| Organ Classes (DSC %) | Overall | |||||||||||||||||
| # | Experts | Spl | RK | LK | GB | Eso | Liv | Sto | Aor | IVC | Pan | RAG | LAG | Duo | Bla | P/U | mDSC | mHD95 |
| 1 | [4, 4, 4, 4] | 95.6 | 94.4 | 95.3 | 83.9 | 83.5 | 96.9 | 91.0 | 94.1 | 89.9 | 85.9 | 74.2 | 75.7 | 79.6 | 89.2 | 82.9 | 87.5 | 8.4 |
| 2 | [8, 8, 8, 8] | 95.8 | 95.6 | 93.2 | 80.7 | 83.6 | 96.6 | 91.8 | 94.2 | 90.0 | 85.8 | 75.1 | 76.2 | 79.9 | 89.1 | 82.2 | 87.3 | 8.7 |
| 3 | [4, 8, 16, 32] | 95.8 | 95.9 | 94.9 | 82.7 | 83.6 | 96.8 | 91.5 | 94.1 | 90.0 | 85.6 | 75.1 | 75.5 | 79.7 | 89.3 | 81.6 | 87.5 | 7.6 |
| 4 | [16, 16, 16, 16] | 95.7 | 95.4 | 92.9 | 80.5 | 83.1 | 95.3 | 91.3 | 94.1 | 89.8 | 84.7 | 74.9 | 76.0 | 79.7 | 88.3 | 81.9 | 86.9 | 9.2 |
| NIH | MSD Pancreas | Proprietary | Overall | |||||
| # | Experts | Pancreas | PDAC | Pancreas | PDAC | Pancreas | mDSC | mHD95 |
| 1 | [4, 4, 4, 4] | 92.4 | 50.7 | 88.9 | 63.7 | 81.8 | 77.9 | 7.6 |
| 2 | [8, 8, 8, 8] | 92.0 | 50.4 | 85.4 | 63.3 | 79.5 | 76.1 | 8.5 |
| 3 | [4, 8, 16, 32] | 92.3 | 51.4 | 88.1 | 60.5 | 81.8 | 77.4 | 8.2 |
| 4 | [16, 16, 16, 16] | 92.0 | 53.6 | 87.7 | 65.8 | 81.4 | 78.2 | 7.1 |
| Organ Classes (DSC %) | Overall | |||||||||||||||
| # | Experts | Spl | RK | LK | GB | Eso | Liv | Sto | Aor | IVC | Pan | RAG | LAG | Duo | mDSC | mHD95 |
| 1 | [4, 4, 4, 4] | 90.5 | 90.3 | 90.5 | 68.4 | 72.3 | 96.7 | 80.9 | 87.0 | 83.1 | 80.3 | 58.5 | 55.8 | 60.2 | 78.0 | 13.4 |
| 2 | [8, 8, 8, 8] | 90.6 | 90.7 | 90.5 | 67.3 | 70.8 | 96.3 | 81.5 | 86.5 | 82.7 | 79.2 | 58.5 | 56.5 | 61.9 | 77.9 | 16.9 |
| 3 | [4, 8, 16, 32] | 90.4 | 90.7 | 90.5 | 63.3 | 70.9 | 96.4 | 79.6 | 87.2 | 83.5 | 81.0 | 57.7 | 53.6 | 61.6 | 77.4 | 12.2 |
| 4 | [16, 16, 16, 16] | 91.0 | 90.6 | 90.7 | 67.3 | 72.3 | 96.5 | 81.4 | 87.5 | 83.4 | 80.6 | 57.4 | 59.5 | 62.8 | 78.5 | 12.2 |
| Upper Right (Q1) | Upper Left (Q2) | Lower Left (Q3) | Lower Right (Q4) | Overall | |||||||||||||||||||||||||||||||
| # | Experts | 18 | 17 | 16 | 15 | 14 | 13 | 12 | 11 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 38 | 37 | 36 | 35 | 34 | 33 | 32 | 31 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | mDSC | mHD95 |
| 1 | [4, 4, 4, 4] | 68.1 | 89.6 | 88.8 | 86.6 | 84.7 | 91.3 | 90.5 | 93.8 | 91.4 | 88.5 | 88.7 | 85.5 | 92.0 | 94.4 | 89.4 | 73.1 | 67.6 | 86.0 | 85.5 | 91.0 | 91.3 | 86.1 | 87.1 | 81.7 | 80.8 | 89.4 | 91.5 | 88.3 | 91.3 | 85.4 | 83.9 | 58.0 | 85.9 | 7.7 |
| 2 | [8, 8, 8, 8] | 60.6 | 88.1 | 90.7 | 87.2 | 88.1 | 88.8 | 89.9 | 90.3 | 89.0 | 91.4 | 88.5 | 83.7 | 90.4 | 92.7 | 88.2 | 65.0 | 63.4 | 82.1 | 84.4 | 90.5 | 91.0 | 86.1 | 86.7 | 86.1 | 84.3 | 86.3 | 90.8 | 88.8 | 92.3 | 87.7 | 82.1 | 59.1 | 85.0 | 9.4 |
| 3 | [4, 8, 16, 32] | 67.4 | 90.8 | 86.9 | 86.6 | 85.5 | 91.7 | 91.0 | 93.7 | 93.1 | 90.9 | 89.2 | 86.1 | 90.0 | 93.0 | 92.2 | 73.0 | 71.0 | 86.5 | 85.9 | 89.8 | 89.0 | 85.1 | 86.4 | 83.1 | 81.0 | 88.8 | 92.6 | 89.4 | 91.4 | 88.9 | 83.0 | 60.9 | 86.1 | 7.9 |
| 4 | [16, 16, 16, 16] | 67.9 | 87.6 | 90.8 | 88.0 | 85.6 | 91.4 | 89.3 | 93.0 | 93.3 | 91.7 | 89.4 | 84.0 | 92.0 | 92.4 | 91.4 | 74.0 | 68.1 | 88.2 | 88.6 | 91.4 | 88.6 | 84.7 | 85.4 | 80.7 | 83.5 | 88.9 | 91.4 | 88.6 | 92.2 | 88.7 | 83.2 | 59.8 | 86.2 | 8.3 |
| Experts | 1 Spleen | 2 Kid R | 3 Kid L | 4 Gallb | 5 Liver | 6 Esoph | 7 Stom | 8 Aorta | 9 IVC | 10 P&S Vn | 11 Pancr | 12 Adr R | 13 Adr L | 14 Lung R | 15 Lung L | 16 Sm Bow | 17 Duod | 18 Colon | 19 Ilia Art L | 20 Ilia Art R | 21 Ilia Vn L | 22 Ilia Vn R | 23 GluMax L | 24 GluMax R | 25 GluMed L | 26 GluMed R | 27 Auto L | 28 Auto R | 29 Ilio L | 30 Ilio R | 31 Hip L |
| [4, 4, 4, 4] | 91.1 | 96.5 | 89.1 | 79.0 | 98.5 | 85.3 | 93.5 | 95.5 | 93.1 | 65.5 | 87.5 | 60.6 | 59.6 | 97.5 | 96.5 | 92.5 | 84.0 | 91.7 | 77.4 | 79.1 | 65.5 | 58.0 | 63.8 | 71.6 | 85.2 | 79.9 | 97.6 | 97.4 | 95.6 | 96.4 | 80.1 |
| [8, 8, 8, 8] | 93.4 | 96.4 | 89.4 | 78.0 | 98.5 | 85.3 | 93.4 | 95.4 | 92.9 | 64.0 | 88.2 | 60.8 | 59.7 | 97.4 | 96.4 | 92.4 | 84.4 | 91.8 | 76.6 | 77.1 | 65.5 | 56.6 | 72.3 | 56.9 | 84.2 | 80.7 | 97.4 | 97.4 | 95.6 | 96.3 | 80.9 |
| [4, 8, 16, 32] | 91.5 | 96.3 | 91.5 | 78.6 | 98.5 | 85.4 | 93.6 | 95.5 | 93.1 | 64.0 | 88.1 | 61.0 | 59.2 | 97.5 | 96.6 | 92.2 | 84.2 | 91.6 | 76.8 | 77.1 | 66.5 | 58.2 | 70.0 | 59.1 | 86.8 | 79.2 | 97.5 | 97.4 | 95.5 | 96.4 | 83.4 |
| [16, 16, 16, 16] | 91.4 | 96.5 | 88.7 | 80.0 | 98.5 | 85.3 | 93.7 | 95.5 | 93.0 | 64.5 | 87.9 | 60.9 | 58.0 | 97.5 | 96.5 | 92.2 | 83.7 | 91.7 | 76.7 | 77.2 | 60.8 | 56.1 | 72.7 | 64.5 | 84.9 | 80.3 | 97.5 | 97.4 | 95.6 | 96.4 | 83.0 |
| Organ Classes (DSC %) | Overall | ||||
| # | Experts | PL | AL | mDSC | mHD95 |
| 1 | [4, 4, 4, 4] | 84.1 | 85.5 | 84.8 | 4.8 |
| 2 | [8, 8, 8, 8] | 84.2 | 85.7 | 85.0 | 4.2 |
| 3 | [4, 8, 16, 32] | 83.4 | 85.5 | 84.5 | 4.4 |
| 4 | [16, 16, 16, 16] | 84.2 | 85.6 | 84.9 | 4.3 |
| Organ Classes (DSC %) | Overall | ||||||||||||||||
| Method | Spl | RK | LK | GB | Eso | Liv | Sto | Aor | IVC | Pan | RAG | LAG | Duo | Bla | P/U | mDSC | mHD95 |
| VoCo † [ 84 , 85 ] | 96.2 | 94.3 | 95.4 | 81.6 | 83.2 | 96.0 | 91.1 | 93.9 | 89.7 | 85.5 | 76.0 | 75.7 | 79.8 | 88.0 | 79.4 | 87.1 | 16.4 |
| uC 3DU-Net [ 37 ] | 94.8 | 93.1 | 94.7 | 70.1 | 79.6 | 89.2 | 86.7 | 93.0 | 86.9 | 77.8 | 73.1 | 73.3 | 72.6 | 84.5 | 76.6 | 83.1 | 16.0 |
| SuPreM † [ 50 ] | 96.3 | 95.7 | 95.8 | 81.2 | 83.5 | 97.3 | 90.9 | 93.6 | 89.4 | 85.5 | 74.4 | 75.2 | 79.6 | 88.4 | 80.0 | 87.1 | 7.7 |
| SegMamba [ 88 ] | 95.0 | 95.5 | 95.4 | 75.5 | 80.7 | 96.2 | 88.2 | 93.5 | 88.9 | 82.7 | 75.1 | 73.6 | 75.6 | 86.0 | 80.1 | 85.5 | 17.3 |
| Mamba-HoME [ 68 ] | 96.0 | 95.0 | 94.4 | 81.7 | 82.0 | 94.7 | 90.2 | 93.8 | 88.9 | 84.0 | 74.4 | 74.1 | 78.4 | 83.8 | 83.2 | 86.3 | 18.7 |
| NIH | MSD Pancreas | Proprietary | Overall | ||||
| Method | Pancreas | PDAC | Pancreas | PDAC | Pancreas | mDSC | mHD95 |
| VoCo † [ 84 , 85 ] | 90.1 | 38.7 | 86.5 | 43.3 | 80.3 | 71.6 | 15.9 |
| SegMamba [ 88 ] | 92.7 | 47.9 | 88.1 | 54.0 | 81.9 | 76.0 | 8.6 |
| uC 3DU-Net [ 37 ] | 92.6 | 49.0 | 88.9 | 56.8 | 80.2 | 76.6 | 4.7 |
| SuPreM † [ 50 ] | 92.3 | 46.0 | 88.6 | 61.4 | 82.0 | 76.6 | 4.7 |
| Mamba-HoME [ 68 ] | 92.9 | 51.2 | 88.4 | 60.8 | 81.7 | 77.5 | 4.8 |
| Organ Classes (DSC %) | Overall | ||||||||||||||
| Method | Spl | RK | LK | GB | Eso | Liv | Sto | Aor | IVC | Pan | RAG | LAG | Duo | mDSC | mHD95 |
| VoCo † [ 84 , 85 ] | 91.2 | 90.5 | 91.1 | 71.2 | 74.9 | 97.2 | 84.7 | 87.1 | 83.9 | 82.9 | 62.0 | 59.5 | 65.0 | 80.1 | 10.5 |
| uC 3DU-Net [ 37 ] | 88.7 | 89.7 | 88.7 | 52.0 | 70.2 | 95.7 | 76.6 | 85.4 | 81.2 | 76.4 | 56.3 | 52.2 | 53.8 | 74.4 | 16.9 |
| SuPreM † [ 50 ] | 89.9 | 89.8 | 89.2 | 67.7 | 67.3 | 95.6 | 79.5 | 85.1 | 79.4 | 79.8 | 55.2 | 54.1 | 58.0 | 75.8 | 19.5 |
| SegMamba [ 88 ] | 86.4 | 89.8 | 89.6 | 62.5 | 70.5 | 95.7 | 82.4 | 87.0 | 81.0 | 83.5 | 59.8 | 64.2 | 64.4 | 76.7 | 9.3 |
| Mamba-HoME [ 68 ] | 87.4 | 88.9 | 88.3 | 69.8 | 71.7 | 95.9 | 82.7 | 85.3 | 82.3 | 83.4 | 59.0 | 64.2 | 63.5 | 77.7 | 9.9 |
| Upper Right (Q1) | Upper Left (Q2) | Lower Left (Q3) | Lower Right (Q4) | Overall | ||||||||||||||||||||||||||||||
| Method | 18 | 17 | 16 | 15 | 14 | 13 | 12 | 11 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 38 | 37 | 36 | 35 | 34 | 33 | 32 | 31 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | mDSC | mHD95 |
| VoCo † [ 84 , 85 ] | 63.9 | 89.6 | 88.4 | 86.8 | 88.6 | 92.7 | 89.4 | 92.3 | 90.0 | 86.7 | 86.3 | 86.3 | 89.8 | 91.8 | 89.2 | 65.4 | 64.3 | 86.6 | 85.9 | 87.4 | 85.5 | 78.5 | 81.2 | 74.7 | 77.8 | 85.8 | 86.4 | 84.5 | 89.1 | 84.6 | 83.5 | 56.2 | 83.5 | 6.8 |
| uC 3DU-Net [ 37 ] | 67.7 | 89.5 | 87.7 | 84.6 | 83.0 | 87.3 | 88.5 | 92.3 | 90.7 | 87.9 | 87.7 | 86.5 | 88.8 | 91.0 | 88.2 | 66.4 | 64.9 | 79.0 | 82.8 | 86.0 | 85.9 | 83.8 | 87.5 | 82.8 | 82.6 | 89.5 | 89.0 | 84.5 | 87.5 | 87.1 | 82.4 | 53.6 | 83.8 | 12.5 |
| SuPreM † [ 50 ] | 62.7 | 85.9 | 87.0 | 82.1 | 75.1 | 81.9 | 86.8 | 84.6 | 87.0 | 86.1 | 86.1 | 83.0 | 88.4 | 89.5 | 85.3 | 66.8 | 61.4 | 79.5 | 83.7 | 83.2 | 76.7 | 81.0 | 81.5 | 76.3 | 75.8 | 81.6 | 88.0 | 78.6 | 83.8 | 85.8 | 79.2 | 58.1 | 80.6 | 11.0 |
| SegMamba [ 88 ] | 68.0 | 87.9 | 90.2 | 85.3 | 85.2 | 91.7 | 91.0 | 93.7 | 93.1 | 90.9 | 89.2 | 86.1 | 90.0 | 93.0 | 85.3 | 70.6 | 66.9 | 78.3 | 84.5 | 89.4 | 89.4 | 85.4 | 87.2 | 86.0 | 80.3 | 89.1 | 92.6 | 89.1 | 93.3 | 84.8 | 81.4 | 56.6 | 85.3 | 10.6 |
| Mamba-HoME [ 68 ] | 69.7 | 90.6 | 91.2 | 86.5 | 84.2 | 93.9 | 91.1 | 93.9 | 91.4 | 92.0 | 89.9 | 86.8 | 90.9 | 94.4 | 90.1 | 69.6 | 66.8 | 84.0 | 83.5 | 88.9 | 87.4 | 85.8 | 85.9 | 81.7 | 84.4 | 89.4 | 92.3 | 89.8 | 93.5 | 85.4 | 84.2 | 60.0 | 86.1 | 6.5 |
| Method | 1 Spleen | 2 Kid R | 3 Kid L | 4 Gallb | 5 Liver | 6 Esoph | 7 Stom | 8 Aorta | 9 IVC | 10 P&S Vn | 11 Pancr | 12 Adr R | 13 Adr L | 14 Lung R | 15 Lung L | 16 Sm Bow | 17 Duod | 18 Colon | 19 Ilia Art L | 20 Ilia Art R | 21 Ilia Vn L | 22 Ilia Vn R | 23 GluMax L | 24 GluMax R | 25 GluMed L | 26 GluMed R | 27 Auto L | 28 Auto R | 29 Ilio L | 30 Ilio R | 31 Hip L |
| VoCo [ 84 ] | 91.0 | 96.3 | 88.9 | 74.4 | 98.4 | 84.2 | 93.0 | 95.2 | 92.1 | 62.0 | 86.2 | 59.7 | 55.1 | 97.2 | 96.0 | 90.9 | 81.9 | 89.6 | 74.9 | 75.0 | 59.3 | 50.8 | 63.6 | 57.1 | 83.6 | 76.9 | 97.2 | 97.1 | 95.2 | 95.8 | 73.7 |
| SuPreM [ 50 ] | 90.2 | 96.1 | 85.7 | 70.9 | 98.2 | 83.1 | 91.1 | 95.1 | 91.5 | 61.6 | 85.4 | 59.1 | 54.5 | 97.1 | 95.7 | 88.6 | 80.3 | 86.8 | 69.7 | 72.3 | 55.2 | 49.4 | 50.0 | 43.2 | 83.1 | 75.4 | 97.0 | 96.8 | 94.6 | 95.6 | 72.4 |
| uC 3DU-Net [ 37 ] | 91.0 | 96.2 | 87.3 | 79.2 | 98.5 | 85.4 | 92.9 | 95.3 | 92.8 | 67.0 | 87.9 | 59.5 | 57.1 | 97.5 | 96.6 | 92.0 | 82.2 | 91.3 | 76.1 | 79.5 | 66.0 | 57.5 | 66.8 | 58.5 | 84.5 | 79.0 | 97.5 | 97.3 | 95.6 | 96.4 | 84.3 |
| SegMamba [ 88 ] | 90.7 | 96.4 | 86.6 | 73.6 | 98.4 | 84.7 | 92.6 | 95.2 | 91.9 | 61.7 | 87.1 | 60.2 | 58.0 | 97.1 | 96.1 | 91.0 | 82.6 | 89.8 | 75.9 | 80.2 | 67.1 | 59.7 | 63.5 | 60.4 | 84.6 | 78.9 | 97.3 | 97.1 | 95.3 | 95.8 | 80.5 |
| Mamba-HoME [ 68 ] | 91.3 | 96.3 | 90.3 | 76.9 | 98.5 | 84.7 | 93.4 | 95.1 | 92.3 | 65.4 | 88.2 | 60.2 | 58.1 | 97.4 | 96.5 | 92.5 | 83.2 | 91.7 | 77.3 | 79.8 | 74.3 | 72.6 | 74.2 | 72.8 | 87.2 | 84.0 | 97.3 | 97.2 | 95.4 | 95.9 | 82.6 |
| Merlin [ 13 ] | 91.5 | 96.3 | 91.7 | 78.1 | 98.5 | 84.7 | 93.7 | 95.1 | 92.5 | 66.8 | 88.6 | 60.3 | 58.3 | 97.5 | 96.6 | 93.0 | 83.4 | 92.4 | 77.8 | 79.7 | 77.0 | 77.4 | 78.1 | 77.3 | 88.1 | 85.9 | 97.3 | 97.2 | 95.5 | 96.0 | 83.4 |
| Organ Classes (DSC %) | Overall | |||
| Method | PL | AL | mDSC | mHD95 |
| VoCo † [ 84 , 85 ] | 83.3 | 85.3 | 84.3 | 5.1 |
| uC 3DU-Net [ 37 ] | 82.9 | 85.0 | 83.9 | 4.7 |
| SuPreM † [ 50 ] | 83.2 | 85.4 | 84.3 | 4.9 |
| SegMamba [ 88 ] | 83.0 | 84.7 | 83.8 | 5.8 |
| Mamba-HoME [ 68 ] | 83.6 | 85.3 | 84.5 | 4.8 |
| Method | Bowel | Liver | Kidney | Spleen | Mean |
| VoCo [ 84 ] | 64.2 | 67.1 | 88.1 | 86.2 | 76.4 |
| SuPreM [ 50 ] | 72.6 | 76.2 | 90.4 | 92.5 | 82.9 |
| BTB3D [ 33 ] | 70.1 | 74.8 | 89.4 | 87.2 | 80.4 |
| Merlin [ 13 ] | 79.8 | 81.2 | 94.2 | 90.8 | 86.5 |
| A 2 MoE | 82.4 | 84.6 | 93.8 | 92.1 | 88.2 |
| VoCo [ 84 ] | SuPreM [ 50 ] | BTB3D [ 33 ] | Merlin [ 13 ] | A 2 MoE | ||||||||||||
| Organ | Grading | AUC | Sens. | Spec. | AUC | Sens. | Spec. | AUC | Sens. | Spec. | AUC | Sens. | Spec. | AUC | Sens. | Spec. |
| Bowel | 0 vs. 1 | 64.2 | 61.9 | 63.1 | 72.6 | 71.4 | 68.4 | 70.1 | 66.7 | 65.8 | 79.8 | 81.0 | 74.2 | 82.4 | 76.2 | 79.5 |
| Liver | 0 vs. 1+2 | 67.1 | 62.8 | 64.2 | 76.2 | 69.1 | 72.8 | 74.8 | 67.0 | 71.5 | 81.2 | 74.5 | 77.9 | 84.6 | 78.7 | 80.8 |
| 0 vs. 1 | 63.8 | 60.5 | 64.2 | 72.4 | 65.8 | 72.8 | 70.9 | 63.2 | 71.5 | 78.1 | 69.7 | 77.9 | 77.3 | 72.4 | 74.2 | |
| 0 vs. 2 | 76.8 | 72.2 | 64.2 | 85.4 | 83.3 | 72.8 | 83.2 | 77.8 | 71.5 | 89.1 | 88.9 | 77.9 | 91.8 | 83.3 | 80.8 | |
| Kidney | 0 vs. 1+2 | 88.1 | 76.4 | 82.3 | 90.4 | 80.0 | 89.2 | 89.4 | 78.2 | 83.5 | 94.2 | 83.6 | 87.1 | 93.8 | 85.5 | 88.6 |
| Pancreatic Tumor (%) | Kidney Tumor (%) | Liver Tumor (%) | |||||||||
| Model | Sen. ( 2 cm) | Sens. ( 2 cm) | Spec. | Sens. ( 2 cm) | Sens. ( 2 cm) | Spec. | Sens. ( 2 cm) | Sens. ( 2 cm) | Spec. | ||
| CT-CHAT | 66.7 | 51.9 | 61.2 | 31.1 | 32.8 | 74.2 | 5.7 | 3.2 | 94.7 | ||
| CT2Rep | 0.0 | 0.0 | 92.5 | 36.5 | 39.3 | 70.4 | 35.8 | 49.2 | 70.4 | ||
| M3D | 0.0 | 7.4 | 97.2 | 8.1 | 16.4 | 84.1 | 9.4 | 12.7 | 86.0 | ||
| Merlin | 33.3 | 51.9 | 71.8 | 28.4 | 45.9 | 86.6 | 30.2 | 41.3 | 95.9 | ||
| RadFM | 0.0 | 0.0 | 99.9 | 3.7 | 6.3 | 95.6 | 3.3 | 5.7 | 93.9 | ||
Medical VLMs
Medical vision-language models connect clinical images with text for diagnosis, reporting, question answering, and image grounding.
See all 511 papers