Counterfactual Analysis of Extent Leakage from Annotation-Derived Prompts in Vision-Language Medical Image Segmentation
Organizations: Department of Information Physics and Computing, Graduate School of Information Science and Technology, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8656, Japan · Department of Computer Science and Engineering, Indian Institute of Technology Hyderabad, Kandi, Sangareddy, Telangana, 502284, India
Abstract
Text-guided segmentation models receive an image and a short caption about the lesion, and on public benchmarks the caption improves accuracy considerably. On colonoscopy and breast ultrasound, however, the size, count and location words of each caption were computed from the mask that later scores the prediction, and the chest X-ray captions were written by benchmark annotators. We tested what two published models take from them. On colonoscopy and chest X-ray, it is mostly the lesion's extent. Keeping only the size word or the lung-zone phrase recovers 84 to 98 percent of the improvement over an empty caption, and removing it loses 52 to 77 percent. The rest matters little, and a polyp called a rock is segmented just as well. We call this extent leakage. On breast ultrasound, nearly half the captions also name the diagnosis, which carries part of the benefit. Retraining on three datasets shows that models pick up this habit from their training captions, though only partly on breast ultrasound, and that it weakens their segmentation from the image alone. The leak can be measured without training a model, since caption words alone predict lesion area (coefficient of determination 0.54 to 0.77 on six of seven test sets). Without the test mask, the best method we tested, which estimates extent from the image, reaches 74.59 Dice on colonoscopy against the published 77.23. We propose an evaluation protocol that keeps what a caption says about the image apart from what it copies from the label.
Figures & tables
| Dataset | Modality | Total | Train | Val | Test | Extent attribute ∗ |
|---|---|---|---|---|---|---|
| Kvasir-SEG | colonoscopy | 1000 | 800 | 100 | 100 | size word |
| CVC-ClinicDB | colonoscopy | 612 | (490) | (61) | 61 | size word |
| CVC-300 | colonoscopy | 66 | (6) | - | 60 | size word |
| CVC-ColonDB | colonoscopy | 380 | (20) | - | 360 | size word |
| BKAI-IGH | colonoscopy | 999 | (799) | (100) | 100 | size word |
| BUSI | ultrasound | 780 | 624 | 78 | 78 | size word |
| Test set | Empty | Mism. | Match. | |||||
|---|---|---|---|---|---|---|---|---|
| MedCLIPSeg | ||||||||
| Kvasir-SEG | 81.04 | 81.04 | 90.15 | 9.11 | 0.1 | 87.7 | 51.8 | 0.70 |
| CVC-ClinicDB | 69.92 | 76.06 | 80.80 | 10.87 | 56.5 | 84.1 | 67.4 | 0.54 |
| CVC-300 | 65.32 | 79.49 | 80.84 | 15.53 | 91.3 | 85.9 | 67.2 | 0.24 |
| BKAI-IGH | 54.55 | 73.75 | 79.16 | 24.61 | 78.0 | 92.0 | 77.2 | 0.59 |
| CVC-ColonDB | 59.17 | 60.96 | 71.89 | 12.72 | 14.1 | 98.3 | 68.2 | 0.71 |
| Criterion | Kvasir-SEG ( ) | BUSI ( ; R1-R3, : 65) | QaTa-COV19 ( ) | |
|---|---|---|---|---|
| R0 | Original within of the published checkpoint | 76.00 vs. 77.23 ( 1.22) yes | 85.26 vs. 85.73 ( 0.47) yes | 83.66 vs. 83.41 (+0.25) yes |
| R1 | Size-blind size response of Original | 1.2% (0.13 vs. 11.1 pp) yes | 1.3% (0.11 vs. 8.0 pp) yes | 0.0% (0.00 vs. 7.9 pp) yes |
| R2 | Size-blind retains of (CI) | 37.4% [28.5, 46.8] yes | 39.1% [21.9, 58.3] consistent | 9.0% [7.1, 11.0] yes |
| R3 | Size-only retains of (CI) | 89.7% [80.4, 99.8] yes | 49.9% [37.6, 60.7] no | 91.0% [88.8, 93.2] yes |
| R4 | Mismatched: ; empty-caption score within of Text-free | [ 0.78, 0.28]; [ 0.18, +0.80] yes | [ 2.52, 0.10]; [ 0.46, +1.74] no | - |
| Size-blind better from the image alone ( , CI) | +2.39 [+1.62, +3.16] (not registered) | +8.43 [+4.38, +13.02] yes | +11.28 [+10.55, +12.00] yes |
| Captions | train | train | test |
|---|---|---|---|
| Kvasir-SEG | 0.77 | 0.56 | 0.57 |
| CVC-ClinicDB | 0.68 | 0.44 | 0.48 |
| CVC-300 | - | - | 0.02 ∗ |
| BKAI-IGH | 0.48 | 0.18 | 0.28 |
| CVC-ColonDB | - | - | 0.35 |
| Dice | |||
| Method | all | unseen | , all (95% CI) |
| Annotation-free | |||
| Published, empty caption | 63.21 | 60.14 | 14.02 [ 15.63, 12.41] |
| Edge-aligned threshold | 73.84 | 71.41 | 3.39 [ 4.45, 2.39] |
| Image-only describer | 72.38 | 69.71 | 4.84 [ 5.91, 3.80] |
| Text-free (trained without text) | 72.65 | 70.14 | 4.57 [ 5.67, 3.52] |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Criteria | Written (dated record) | First commit containing them | First run | First result seen | Deciding result |
|---|---|---|---|---|---|
| Retraining, Kvasir-SEG (R0-R4) | research plan, dated by its author 22 Sep about 16:30 IST, while the first Original run was training | e19ee2a , 22 Sep 23:06 IST (audit script with the bars) | 22 Sep 16:16 IST (first Original run, by the project log; its log was last written at 17:20 IST) | seed 1: R0 22 Sep 18:40, R1-R2 21:12, R3 23:35 IST; R4 23 Sep 02:10 IST | three seeds: R0-R3 23 Sep 13:02-15:03 IST; R4 7 Oct |
| Retraining, BUSI and QaTa-COV19 (R0-R4, ) | declaration, 5 Oct; Amendment 1 (R4 intervals, , consistent ) the same day | 9f31c22 / e2c2051 , 5 Oct 16:14-16:15 JST; amendment 0e94675 , 16:40 JST | queue started 5 Oct 16:20 JST with two Kvasir-SEG Mismatched seeds; first BUSI run after 17:15 JST; first QaTa-COV19 run 6 Oct | BUSI 7 Oct; QaTa-COV19 10 Oct | BUSI 7 Oct; QaTa-COV19 10 Oct |
| ExtentSeg against the published model, and its area loss | architecture plan, 23 Sep 01:28 IST, before any output of the model | e484275 , 23 Sep 15:27 IST | first run queued 23 Sep 01:31 IST | seed 1: 23 Sep 11:40 IST | three seeds: 24 Sep 01:40 IST |
| ExtentSeg against its extent-free control | research plan, 23 Sep, before any output of the control | d7160da , 23 Sep 21:13 IST (audit script with the bar) | control built 23 Sep 12:50 IST | seed 1: 23 Sep 21:20 IST | three seeds: after 24 Sep 01:25 IST, when seeds 2-3 of the control were launched |
| RecLMIS (reproduction, extent necessity) | research plan, 22 Sep, before either was run | e19ee2a , 22 Sep 23:06 IST | - (published checkpoint) | 22 Sep | 22 Sep |
| Caption style | research plan, 22 Sep, before any rewritten caption was generated | e19ee2a , 22 Sep 23:06 IST | - (published checkpoints) | 22 Sep | 22 Sep |
| Criterion | Bar | Result | Verdict |
|---|---|---|---|
| R0-R4 and , retraining on three training sets | Section 4.4 | Table 3 | Kvasir-SEG: R0-R4 met ( not registered). QaTa-COV19: every criterion run was met. BUSI: R0, R1 and met; R2 consistent, and met on the second population specified in advance (Section 5.2 ); R3 and R4 not met |
| ExtentSeg against the published model | ExtentSeg without area loss (the registered model) minus the published model with its matched caption, unseen centres: interval above zero | 3.58 [ 4.76, 2.36]; with the area loss, 2.74 [ 3.84, 1.61] | not met |
| Area loss of ExtentSeg | ExtentSeg minus ExtentSeg without area loss, unseen centres: at least , interval excluding zero | +0.84 [+0.23, +1.43] | met |
| Extent input of ExtentSeg | ExtentSeg minus its extent-free control, unseen centres: at least , interval excluding zero | +0.59 [ 0.02, +1.18] | not met: positive in seven of nine seed pairings, interval includes zero (Section 5.5 ) |
| RecLMIS reproduction | Dice with its own captions within 1.0 of the reported 85.22 | 85.21 | met |
| RecLMIS extent necessity | deleting the zone phrase costs at least 50% of the text gain, deletion written in MedCLIPSeg’s wording | 43.6% [41.3, 45.8] | not met; in RecLMIS’s own wording the necessity is 58.0% (Table 2 ) |
| Colonoscopy, all ( ) | Unseen centres ( ) | Without “no polyp” ( ) | BUSI lesions ( ) | |||||
|---|---|---|---|---|---|---|---|---|
| Quantity | value (95% CI) | change | value (95% CI) | change | value (95% CI) | change | value (95% CI) | change |
| (points) | 13.92 [12.20, 15.69] | 0.10 | 14.84 [12.90, 16.79] | 0.03 | 14.03 [12.40, 15.71] | +0.01 | 10.13 [5.32, 15.37] | 0.51 |
| (points) | +4.86 [+2.65, +7.03] | 0.71 | +5.78 [+3.30, +8.25] | 0.74 | +5.62 [+3.54, +7.65] | +0.06 | 0.63 [ 7.51, +5.94] | +3.27 |
| (%) | 34.9 [20.7, 47.0] | 4.8 | 39.0 [24.2, 51.6] | 4.9 | 40.1 [27.4, 51.0] | +0.4 | 6.3 [ 101.1, 54.2] | +30.5 |
| (%) | 94.3 [90.7, 97.9] | +0.8 | 95.0 [91.1, 99.0] | +0.9 | 94.1 [90.7, 97.4] | +0.6 | 20.4 [ 53.2, 63.2] | 10.2 |
| (%) | 69.0 [63.1, 74.6] | +0.2 | 71.0 [64.7, 77.1] | +0.4 | 69.1 [63.5, 74.4] | +0.3 | 67.7 [35.4, 138.6] | +10.6 |
| Quantity ( ) | Estimate | Images | Sequences, 0.6 | Sequences, 0.5 | Sequences, 0.7 |
|---|---|---|---|---|---|
| (points) | 14.02 | [12.41, 15.63] | [11.32, 16.57] | [11.11, 16.80] | [11.52, 16.39] |
| (points) | +5.56 | [+3.53, +7.61] | [+0.59, +9.17] | [ 0.08, +9.56] | [+1.20, +8.92] |
| (%) | 39.7 | [27.3, 50.8] | [4.9, 58.6] | [ 0.7, 59.6] | [10.0, 58.0] |
| (%) | 93.5 | [90.0, 97.0] | [88.9, 99.4] | [88.6, 100.1] | [88.7, 99.5] |
| (%) | 68.8 | [63.3, 74.2] | [62.2, 77.1] | [61.7, 77.2] | [62.1, 75.3] |
| R2 (%) | 37.4 | [28.5, 46.8] | [26.3, 51.5] | [26.7, 51.4] | [25.0, 53.1] |
| Test set | Empty | Mism. | Match. | (95% CI) | (95% CI) | (95% CI) | (95% CI) |
|---|---|---|---|---|---|---|---|
| MedCLIPSeg, normalised surface Dice | |||||||
| Kvasir-SEG | 82.84 | 83.10 | 92.33 | 9.49 [5.88, 13.16] | 2.7 [ 55.0, 36.7] | 88.1 [78.5, 95.3] | 52.3 [32.1, 69.8] |
| CVC-ClinicDB | 73.20 | 80.20 | 85.16 | 11.96 [6.16, 18.30] | 58.6 [8.9, 91.4] | 84.1 [64.5, 95.2] | 69.7 [43.1, 86.1] |
| CVC-300 | 68.07 | 82.94 | 84.28 | 16.20 [12.43, 20.18] | 91.8 [81.4, 100.9] | 85.8 [78.0, 92.9] | 67.3 [54.6, 79.2] |
| BKAI-IGH | 55.55 | 75.41 | 80.92 | 25.37 [20.10, 30.92] | 78.3 [65.7, 90.4] | 92.2 [86.7, 96.9] | 77.5 [67.5, 87.0] |
| CVC-ColonDB | 61.36 | 63.74 | 74.95 | 13.59 [11.25, 15.92] | 17.5 [ 7.3, 37.2] | 97.6 [92.1, 103.3] | 68.5 [60.0, 76.6] |
| Same content, other style | Content slots emptied | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Test set | Released | Terse | Verbose | Clinical | List | Clinical | List | Empty | |
| Kvasir-SEG | 100 | 90.15 | 90.04 | 90.04 | 88.97 | 89.45 | 80.52 | 83.73 | 81.04 |
| CVC-ClinicDB | 61 | 80.80 | 80.17 | 80.34 | 81.18 | 80.85 | 68.73 | 72.22 | 69.92 |
| CVC-300 | 60 | 80.84 | 80.84 | 80.95 | 77.14 | 78.27 | 63.08 | 67.76 | 65.32 |
| BKAI-IGH | 100 | 79.16 | 79.37 | 79.26 | 75.64 | 77.12 | 54.18 | 60.02 | 54.55 |
| CVC-ColonDB | 360 | 71.89 | 71.86 | 72.68 | 72.81 | 73.33 | 57.25 | 61.98 | 59.17 |
| Rule | Kvasir-SEG | CVC-ClinicDB | CVC-300 | BKAI-IGH | CVC-ColonDB | Colonoscopy, all | Unseen | BUSI | QaTa-COV19 |
|---|---|---|---|---|---|---|---|---|---|
| half | 85.03 | 75.79 | 71.01 | 62.98 | 64.84 | 69.06 | 66.31 | 63.06 | 76.04 |
| Otsu | 82.28 | 70.55 | 65.94 | 51.08 | 58.25 | 62.50 | 59.10 | 62.68 | 76.22 |
| drop | 82.42 | 74.44 | 77.42 | 77.62 | 63.25 | 70.42 | 68.36 | 62.84 | 75.05 |
| plateau | 2.67 | 4.44 | 6.35 | 5.08 | 7.07 | 5.83 | 6.38 | 4.07 | 5.29 |
| agree | 85.11 | 75.13 | 70.88 | 62.96 | 65.02 | 69.09 | 66.33 | 62.92 | 76.13 |
| grad ∗ | 87.93 | 77.64 | 74.30 | 77.19 | 68.27 | 73.84 | 71.41 | 63.95 | 75.66 |
| Dice, empty caption | Dice, matched caption | Text gain | |||||||
| Model | seed 1 | seed 2 | seed 3 | seed 1 | seed 2 | seed 3 | seed 1 | seed 2 | seed 3 |
| Kvasir-SEG, five colonoscopy test sets ( ) | |||||||||
| Original | 69.54 | 69.20 | 70.74 | 77.07 | 75.04 | 75.90 | 7.53 | 5.84 | 5.15 |
| Size-blind | 72.18 | 71.79 | 72.67 | 74.54 | 74.16 | 74.88 | 2.36 | 2.37 | 2.21 |
| Size-only | 69.79 | 68.94 | 70.49 | 75.88 | 76.41 | 73.54 | 6.09 | 7.47 | 3.06 |
| Mismatched | 72.86 | 73.13 | 72.90 | 72.42 | 72.83 | 72.06 | 0.44 | 0.30 | 0.84 |