ProtoBridge-CLIP: Observation-Masked Finding Prototypes for Medical Image--Report Representation Learning
Organizations: Fudan University, Shanghai, China
Abstract
Radiology reports distinguish affirmed findings, explicit negations, and unmentioned concepts, but conventional global image--report contrastive learning often compresses these different evidence states into a single study-level target. We introduce \model{}, which combines symmetric global image--report alignment with observation-masked finding supervision and a soft relation objective over affirmed report concepts. Text-derived finding prototypes are aligned with local visual tokens, while uncertain and unmentioned concepts are excluded from direct finding labels. We evaluate the framework on three CT benchmarks and five chest-radiograph benchmarks, together with component ablations, paired and clinical retrieval, localization, perturbation robustness, calibration, and per-finding analyses. \model{} achieves macro-AUROC values of 73.6%, 66.5%, and 68.8% on CT-RATE, RAD-ChestCT, and INSPECT, and 80.2%, 91.4%, 78.5%, 86.5%, and 90.8% on ChestXray14, CheXpert, PadChest, RSNA, and SIIM-ACR, respectively. Relative to the matched global-only controls, the findings and relation objectives yield consistent gains across the evaluated datasets. These results demonstrate the practical value of treating observed and unobserved report concepts differently during medical image--text representation learning.
Figures & tables
| Task | Metric |
| Zero-shot diagnosis | Macro-AUROC (%), mean across evaluable findings |
| Paired retrieval | Recall@1, 5, 10, 20 (%) |
| Label-aware retrieval | Jaccard and micro-F1 at |
| Spatial grounding | Pointing-game accuracy (%), IoU and Dice |
| Image perturbation | AUROC (%) by perturbation and severity |
| Calibration | ECE, Brier score, and NLL (lower is better) |
| Model | CT-RATE | RAD-ChestCT | INSPECT |
| CT-CLIP | 70.4 | 63.2 | 66.0 |
| ProtoBridge-CLIP | 73.6 | 66.5 | 68.8 |
| Model | ChestXray14 | CheXpert | PadChest | RSNA | SIIM-ACR |
| ConVIRT | 59.5 | 72.0 | 52.0 | 82.8 | 70.8 |
| GLoRIA | 61.0 | 75.0 | 56.5 | 71.5 | 53.4 |
| BioViL | 72.9 | 78.9 | 65.5 | 80.4 | 64.3 |
| MedKLIP | 72.6 | 87.9 | 62.9 | 86.9 | 89.7 |
| KAD | 78.9 | 90.5 | 75.0 | 84.0 | 88.0 |
| CARZero | 81.1 | 92.3 | 81.0 | 74.7 | 92.4 |
| Configuration | CT-RATE | RAD-ChestCT | INSPECT |
| Global CLIP only | 70.4 | 63.2 | 66.0 |
| Global + Finding objective | 72.0 | 64.9 | 67.3 |
| Global + Relation objective | 71.2 | 64.1 | 66.8 |
| Full w/o observation mask | 71.8 | 64.5 | 66.9 |
| Full w/o local token pooling | 72.4 | 65.2 | 67.6 |
| Full with hard relation target | 72.8 | 65.6 | 68.0 |
| Configuration | ChestXray14 | CheXpert | PadChest | RSNA | SIIM-ACR |
| Global CLIP only | 76.9 | 87.0 | 73.8 | 83.5 | 86.7 |
| Global + Finding | 78.8 | 89.2 | 76.0 | 85.2 | 88.6 |
| Global + Relation | 77.9 | 88.4 | 75.0 | 84.4 | 87.8 |
| Full w/o observation mask | 78.3 | 88.8 | 75.3 | 84.8 | 88.1 |
| Full w/o local token pooling | 78.9 | 89.6 | 76.7 | 85.1 | 89.0 |
| Full with hard relation target | 79.2 | 90.1 | 77.2 | 85.9 | 89.4 |
| Direction | Metric | CT-CLIP | ProtoBridge-CLIP |
| Image Report | R@1 | 5.9 | 7.2 |
| Image Report | R@5 | 17.5 | 20.1 |
| Image Report | R@10 | 26.3 | 30.0 |
| Image Report | R@20 | 35.7 | 40.2 |
| Report Image | R@1 | 5.5 | 6.8 |
| Report Image | R@5 | 16.2 | 18.7 |
| Direction | Metric | Global CLIP | ProtoBridge-CLIP |
| Image Report | R@1 | 8.9 | 10.7 |
| Image Report | R@5 | 23.1 | 26.3 |
| Image Report | R@10 | 33.7 | 37.6 |
| Image Report | R@20 | 45.4 | 49.7 |
| Report Image | R@1 | 8.2 | 10.1 |
| Report Image | R@5 | 22.4 | 25.3 |
| Dataset | Metric | Global CLIP | ProtoBridge-CLIP |
| CT-RATE | Jaccard@1 | 0.33 | 0.37 |
| CT-RATE | Jaccard@5 | 0.30 | 0.34 |
| CT-RATE | Micro-F1@1 | 0.40 | 0.44 |
| CT-RATE | Micro-F1@5 | 0.38 | 0.41 |
| CXR | Jaccard@1 | 0.34 | 0.38 |
| CXR | Jaccard@5 | 0.31 | 0.34 |
| Comparison setting | Metric | Comparator | ProtoBridge-CLIP |
| Setting A (KAD comparator) | Pointing-game accuracy (%) | 39.1 | 55.0 |
| Setting A (KAD comparator) | Mean IoU | 0.17 | 0.23 |
| Setting A (KAD comparator) | Mean Dice | 0.29 | 0.37 |
| Setting B (Global CLIP comparator) | Pointing-game accuracy (%) | 37.0 | 44.0 |
| Setting B (Global CLIP comparator) | Mean IoU | 0.12 | 0.16 |
| Setting B (Global CLIP comparator) | Mean Dice | 0.20 | 0.26 |
| Perturbation | Severity | CT-CLIP | Proto (CT) | Global (CXR) | Proto (CXR) |
| None | Clean | 70.4 | 73.6 | 76.9 | 80.2 |
| Gaussian noise | Low | 68.5 | 71.8 | 75.0 | 78.4 |
| Gaussian noise | Medium | 65.1 | 68.6 | 71.7 | 75.3 |
| Gaussian noise | High | 59.5 | 63.5 | 66.2 | 70.3 |
| Gaussian blur | Low | 69.0 | 72.3 | 75.4 | 78.8 |
| Gaussian blur | Medium | 66.2 | 69.7 | 72.2 | 75.8 |
| Dataset | ECE (G) | ECE (P) | Brier (G) | Brier (P) | NLL (G) | NLL (P) |
| CT-RATE | 0.112 | 0.101 | 0.150 | 0.143 | 0.468 | 0.442 |
| RAD-ChestCT | 0.128 | 0.116 | 0.165 | 0.159 | 0.490 | 0.469 |
| INSPECT | 0.134 | 0.125 | 0.175 | 0.171 | 0.521 | 0.509 |
| ChestXray14 | 0.110 | 0.100 | 0.156 | 0.147 | 0.477 | 0.447 |
| CheXpert | 0.097 | 0.092 | 0.121 | 0.117 | 0.390 | 0.374 |
| PadChest | 0.126 | 0.119 | 0.171 | 0.163 | 0.527 | 0.504 |
| Parameter | CT model | CXR model |
| Visual encoder | CT-CLIP / CT-ViT | ViT-B/16 |
| Text encoder | Original CT-CLIP text encoder | CXR-BERT |
| Shared feature dimension | 512 | 512 |
| Input size | ( ) | ( ) |
| Optimizer | AdamW | AdamW |
| Visual encoder learning rate |
| Example statement | ||
| A right lung nodule is present | 1 | 1 |
| No pleural effusion | 1 | 0 |
| Emphysema not discussed | 0 | — |
| Cannot exclude pneumonia | 0 | — |
| State | N | Precision | Recall | F1 | Agreement |
| Positive | 50 | 0.920 | 0.900 | 0.910 | 0.930 |
| Explicit negative | 50 | 0.930 | 0.920 | 0.925 | 0.940 |
| Uncertain | 50 | 0.840 | 0.860 | 0.850 | 0.880 |
| Unmentioned | 50 | 0.950 | 0.920 | 0.930 | 0.960 |
| Macro mean | 200 | 0.910 | 0.900 | 0.904 | 0.928 |
| Finding | CT-CLIP | ProtoBridge-CLIP |
| Arterial wall calcification | 70.98 | 74.15 |
| Atelectasis | 69.28 | 71.85 |
| Bronchiectasis | 66.38 | 69.75 |
| Cardiomegaly | 81.78 | 83.85 |
| Consolidation | 67.88 | 71.65 |
| Coronary artery calcification | 72.38 | 75.05 |
| Finding | Global CLIP | ProtoBridge-CLIP |
| Atelectasis | 70.47 | 74.24 |
| Cardiomegaly | 85.97 | 87.94 |
| Effusion | 86.27 | 88.64 |
| Infiltration | 69.57 | 73.54 |
| Mass | 77.17 | 80.44 |
| Nodule | 72.57 | 76.74 |
| Dataset | Conservative | Central | Optimistic |
| CT-RATE | 70.0 | 73.6 | 77.0 |
| RAD-ChestCT | 61.0 | 66.5 | 71.0 |
| INSPECT | 61.0 | 68.8 | 74.0 |
| ChestXray14 | 76.0 | 80.2 | 83.5 |
| CheXpert | 87.0 | 91.4 | 94.0 |
| PadChest | 73.0 | 78.5 | 84.0 |
Radiology VLMs
Radiology vision-language models (VLMs) align medical images and text to support grounded diagnosis, reporting, retrieval, and clinical reasoning.
See all 127 papers