Seeing Is Not Believing: Counterfactual Lesion Erasure Teaches a Small Medical Vision-Language Agent to Report Only What the Image Shows
Organizations: Independent Researcher
Abstract
Medical vision-language models (VLMs) often report a finding because the question names it; standard benchmarks miss this. From public segmentation datasets we build counterfactual views without manual annotation: the lesion erased by inpainting, a same-shape control edit elsewhere, and unedited lesion-free images. The evidence sensitivity gap (ESG) is the rate of absent-or-refer answers after erasure minus the rate after the control edit; it is zero for a model that ignores the image or reacts to both edits alike. MedScope-4B (Qwen3.5-4B, LoRA, image tools) was trained on label-derived trajectories, selected by a pre-registered rule and tested once against nine frontier and six open VLMs; four pre-registered checks then probed the mechanism. On 507 internal cases across five imaging types, MedScope-4B reached ESG 0.758 and claimed an erased lesion in 9.6% of cases, against 0.27-0.47 and 41-62% for the frontier models (all p<0.001), without significant loss of accuracy on real lesions. Without counterfactual views ESG fell to 0.279+-0.107 (three seeds); the effect replicated on MedGemma 1.5 4B and two unseen inpainting tools. At four external centres the advantage held but frontier models detected more real lesions; on a fresh external test MedScope-4B detected 0 of 48 pneumothorax films, with net utility below gemini-4-argon's (-0.36 [-0.50, -0.23]). Matching the control edit was not shown to matter; over three seeds no fresh-external ESG advantage over removal-based DPO was established; a paired-view consistency objective did not beat a matched-compute control, while that continuation alone cut erased-lesion claims on unseen tools at every seed, at a cost in sensitivity at one. Training on erased and edited images gives a small model an evidence-gated disposition that prompting does not give frontier models; it does not improve perception, and rare critical findings need coverage.
Figures & tables
| Work | Image changed | Control edit | Lesion-free symmetry | Trains | Medical | Compared here |
|---|---|---|---|---|---|---|
| POPE [ 9 ] , CARES [ 11 ] , MedVH [ 13 ] , MedHEval [ 14 ] | no | – | – | no | partly | – |
| Causal VQA edits [ 18 ] | yes | unmatched | no | no | no | unmatched arm (§ 5.3 ) |
| Object removal + preference optimisation [ 10 ] | yes | no | no | yes | no | removal-based DPO arm (§ 5.3 ) |
| Counter-evidence and counterfactual grounding detectors [ 16 , 17 ] | at inference | no | no | no | partly | – |
| This work | yes | matched and unmatched | yes | yes | yes | – |
| View | Correct / absent answer | Wrong, or claims finding | Refer | Clarify |
|---|---|---|---|---|
| Original, random removal | ( critical) | |||
| Target removal, natural negative | ||||
| Degraded |
| Source | Imaging | Cases with mask | Lesion-free | Split level |
|---|---|---|---|---|
| SLAKE [ 7 ] | CT / MRI | 17 | 90 | near-copy groups |
| Kvasir-SEG [ 37 ] + HyperKvasir [ 38 ] | Colonoscopy | 83 | 91 | near-copy groups |
| ChestX-Det [ 39 ] | Chest X-ray | 41 | 50 | patient |
| BUSI [ 41 ] | Breast ultrasound | 42 | 15 | near-copy groups |
| TCGA-LGG [ 42 ] | Brain MRI (FLAIR) | 36 | 42 | patient |
| Total | 219 | 288 | 507 cases, 1,115 views |
| Test set | Cases | Sources (cases) | Purpose |
|---|---|---|---|
| External centres | 169 | CVC-ClinicDB [ 43 ] (40), QaTa-COV19 [ 44 ] (20), BUS-BRA [ 45 ] (45), BraTS-Africa 2023 [ 46 , 47 ] (64, with 50 tumour-free slices) | New scanners, populations |
| Fresh internal | 76 | ChestX-Det, unused test-split patients (36 with a lesion, 40 lesion-free) | Confirmatory |
| Fresh external | 316 | BraTS 2023 meningioma [ 48 ] (82), BUS-UCLM [ 49 ] (99), SIIM-ACR pneumothorax [ 50 ] (96), ETIS-LaribPolypDB [ 51 ] (39) | Confirmatory, new sources |
| Held-out wordings | 373 | Main test with three test-only paraphrases | Question wording |
| Standard VQA | 1,051 | VQA-RAD test (451), SLAKE English test (300 sampled), PathVQA test (300 sampled) | No regression |
| Model | CCR | ESG | Erased % | Healthy % | Net utility |
|---|---|---|---|---|---|
| MedScope-4B (this work) | 0.849 | 0.758 | 9.6 | 7.6 | 0.732 [0.649, 0.813] |
| gemini-4-argon | 0.874 | 0.470 | 41.1 | 9.0 | 0.565 |
| Claude Sonnet 5.5 | 0.870 | 0.402 | 48.4 | 16.7 | 0.381 |
| gemini-3.1-pro-preview | 0.849 | 0.384 | 51.6 | 26.7 | 0.306 |
| gpt-6-sol | 0.829 | 0.342 | 51.6 | 25.7 | 0.261 |
| Claude Fable 5.1 | 0.856 | 0.342 | 52.1 | 23.3 | 0.255 |
| Compared with | ESG | Erased-lesion claims | Healthy-image claims | CCR | Net utility |
|---|---|---|---|---|---|
| gemini-4-argon | 0.288 [0.208, 0.367] | 0.315 [ 0.385, 0.243] | 0.014 [ 0.059, 0.028] | 0.025 [ 0.070, 0.017] | 0.168 [0.083, 0.254] |
| Claude Sonnet 5.5 | 0.356 [0.284, 0.431] | 0.388 [ 0.459, 0.318] | 0.090 [ 0.137, 0.045] | 0.021 [ 0.069, 0.026] | 0.351 [0.267, 0.437] |
| gemini-3.1-pro-prev. | 0.374 [0.298, 0.448] | 0.420 [ 0.491, 0.350] | 0.191 [ 0.252, 0.133] | 0.000 [ 0.043, 0.044] | 0.426 [0.337, 0.519] |
| gpt-6-sol | 0.416 [0.341, 0.489] | 0.420 [ 0.491, 0.346] | 0.181 [ 0.235, 0.126] | 0.021 [ 0.022, 0.063] | 0.472 [0.389, 0.554] |
| Claude Fable 5.1 | 0.416 [0.339, 0.496] | 0.425 [ 0.500, 0.355] | 0.156 [ 0.212, 0.102] | 0.007 [ 0.054, 0.039] | 0.477 [0.384, 0.566] |
| Claude Opus 5.5 | 0.429 [0.345, 0.507] | 0.443 [ 0.515, 0.368] | 0.160 [ 0.217, 0.103] | 0.014 [ 0.066, 0.036] | 0.374 [0.283, 0.466] |
| Imaging (cases) | MedScope-4B | gemini-4-argon | Claude Opus 5.5 | gpt-6-astra |
|---|---|---|---|---|
| Colonoscopy (174) | 0.95 / 0.01 / 1.06 | 0.69 / 0.31 / 0.83 | 0.57 / 0.40 / 0.53 | 0.40 / 0.58 / 0.40 |
| Brain MRI (78) | 0.75 / 0.14 / 0.75 | 0.42 / 0.47 / 0.56 | 0.31 / 0.61 / 0.51 | 0.50 / 0.31 / 0.43 |
| Breast ultrasound (57) | 0.74 / 0.14 / 0.78 | 0.40 / 0.48 / 0.53 | 0.29 / 0.62 / 0.40 | 0.12 / 0.83 / 0.28 |
| Chest X-ray (91) | 0.51 / 0.17 / 0.40 | 0.20 / 0.49 / 0.21 | 0.05 / 0.66 / 0.12 | 0.15 / 0.66 / 0.14 |
| SLAKE CT/MRI (107) | 0.47 / 0.12 / 0.22 | 0.35 / 0.41 / 0.38 | 0.24 / 0.59 / 0.32 | 0.29 / 0.47 / 0.17 |
| Seed | Endpoint | CMCC | Control | Difference [95% CI] | |
| 0 | Erased-lesion claims | 0.122 | 0.151 | [ , ] | 0.085 |
| 1 | Erased-lesion claims | 0.137 | 0.078 | [ , ] | |
| 2 | Erased-lesion claims | 0.120 | 0.115 | [ , ] | 0.818 |
| Pooled | Erased-lesion claims | 0.126 | 0.115 | [ , ] | 0.265 |
| Pooled | ESG | 0.729 | 0.709 | [ , ] | 0.143 |
| Pooled | Control-edit denial | 0.143 | 0.176 | [ , ] | 0.001 |
| Control edits | Insertions | CCR | ESG | Erased % | Healthy % | Denies after Telea edit % |
|---|---|---|---|---|---|---|
| no | no (erasures only) | 0.788 | 0.708 | 6.8 | 5.9 | 43.2 |
| no | yes | 0.806 | 0.735 | 7.3 | 11.1 | 43.2 |
| yes | no | 0.897 | 0.744 | 15.1 | 11.8 | 16.0 |
| yes | yes (the recipe) | 0.847 | 0.735 | 11.9 | 9.4 | 23.2 |
| Set | Arm | CCR | ESG | Erased % | Denial % | Net utility |
|---|---|---|---|---|---|---|
| Main test | Matched control (factorial arm) | 0.897 | 0.744 | 15.1 | 10.5 | 0.727 |
| Unmatched control | 0.904 | 0.753 | 15.1 | 9.1 | 0.739 | |
| Recipe | 0.847 | 0.735 | 11.9 | 14.6 | 0.692 | |
| No CF, then removal-based DPO | 0.900 | 0.658 | 23.7 | 10.0 | 0.602 | |
| Recipe, then removal-based DPO (C3) | 0.872 | 0.753 | 11.4 | 13.2 | 0.700 | |
| Held-out tools | Matched control | 0.859 | 0.749 | 11.0 | 13.7 | 0.733 |
| Arm | CCR | ESG | Erased % | Healthy % | Net utility |
| Qwen3.5-4B, with counterfactual views (3 seeds) | 0.871 0.022 | 0.741 0.019 | 13.2 3.2 | 13.9 4.0 | 0.684 0.021 |
| Qwen3.5-4B, without (3 seeds) | 0.955 0.008 | 0.279 0.107 | 67.6 11.9 | 35.9 3.9 | 0.341 0.090 |
| Without, “absent” share matched (seed 0) | 0.945 | 0.215 | 73.5 | 31.6 | 0.321 |
| Erasures only (seed 0) | 0.788 | 0.708 | 6.8 | 5.9 | 0.667 |
| Control edits only, matched (seed 0) | 0.897 | 0.744 | 15.1 | 11.8 | 0.727 |
| Control edits only, unmatched (seed 0) | 0.904 | 0.753 | 15.1 | 12.5 | 0.739 |
| Model | LaMa (training) | Transplant (training) | Telea (unseen) | Stable Diffusion (unseen) |
|---|---|---|---|---|
| MedScope-4B | 0.752 / 7.2 | 0.720 / 8.0 | 0.664 / 7.2 | 0.640 / 17.6 |
| gemini-4-argon | 0.536 / 32.8 | 0.488 / 32.8 | 0.560 / 8.8 | 0.392 / 40.0 |
| Claude Opus 5.5 | 0.376 / 45.6 | 0.264 / 56.8 | 0.296 / 41.6 | 0.352 / 46.4 |
| gpt-6-astra | 0.312 / 55.2 | 0.240 / 64.0 | 0.256 / 60.8 | 0.256 / 61.6 |
| Stage 1, the recipe | 0.720 / 12.0 | 0.744 / 8.0 | 0.696 / 7.2 | 0.568 / 24.8 |
| Stage 1, control edits only | 0.712 / 17.6 | 0.736 / 9.6 | 0.768 / 7.2 | 0.704 / 16.0 |
| Model | Default | Cautious | Solved examples | Highest reasoning | Highest image setting | Plain prompt |
|---|---|---|---|---|---|---|
| gemini-4-argon | 0.441 / 41.2 | 0.412 / 42.6 | 0.456 / 36.8 | 0.456 / 39.7 | 0.485 / 38.2 | 0.338 / 52.9 |
| Claude Opus 5.5 | 0.338 / 54.4 | 0.382 / 44.1 | 0.309 / 52.9 | 0.397 / 51.5 | – | 0.412 / 38.2 |
| gpt-6-astra | 0.338 / 52.9 | 0.397 / 47.1 | 0.500 / 38.2 | 0.324 / 58.8 | 0.397 / 51.5 | 0.250 / 66.2 |
| Test set | Model | CCR | ESG | Erased % | Healthy % | Net utility |
|---|---|---|---|---|---|---|
| External centres (169) | MedScope-4B | 0.727 | 0.597 | 11.8 | 4.0 | 0.530 |
| gemini-4-argon | 0.950 | 0.370 | 58.8 | 18.0 | 0.544 | |
| Claude Opus 5.5 | 0.966 | 0.412 | 53.8 | 30.0 | 0.544 | |
| gpt-6-astra | 0.945 | 0.277 | 66.4 | 24.0 | 0.424 | |
| Fresh internal (76) | MedScope-4B | 0.681 | 0.472 | 19.4 | 12.5 | 0.402 |
| gemini-4-argon | 0.639 | 0.056 | 61.1 | 12.5 | 0.048 |
| Source (cases) | MedScope-4B | gemini-4-argon | Claude Opus 5.5 | gpt-6-astra |
|---|---|---|---|---|
| BUS-UCLM, breast ultrasound (99) | 0.73 / 0.67 | 0.71 / 0.55 | 0.93 / 0.48 | 0.98 / 0.18 |
| ETIS-Larib, colonoscopy (39) | 0.69 / 0.38 | 0.97 / 0.75 | 0.92 / 0.59 | 0.73 / 0.35 |
| BraTS 2023 meningioma, MRI (82) | 0.53 / 0.03 | 0.62 / 0.09 | 0.69 / 0.00 | 0.50 / 0.21 |
| SIIM-ACR pneumothorax, chest X-ray (96) | 0.00 / 1.10 | 0.66 / 0.06 | 0.28 / 0.68 | 0.29 / 0.62 |
| Model | VQA-RAD (451) | SLAKE (300) | PathVQA (300) |
|---|---|---|---|
| MedScope-4B | 0.741 / 0.230 / 0.514 | 0.881 / 0.644 / 0.730 | 0.801 / 0.125 / 0.477 |
| gemini-4-argon | 0.884 / 0.410 / 0.674 | 0.927 / 0.696 / 0.780 | 0.712 / 0.097 / 0.417 |
| Claude Opus 5.5 | 0.876 / 0.380 / 0.656 | 0.927 / 0.602 / 0.720 | 0.865 / 0.153 / 0.523 |
| gpt-6-astra | 0.797 / 0.255 / 0.557 | 0.835 / 0.476 / 0.607 | 0.526 / 0.062 / 0.303 |
| Qwen3.5-4B (base) | 0.657 / 0.260 / 0.481 | 0.826 / 0.424 / 0.570 | 0.545 / 0.021 / 0.293 |
| Lingshu-7B | 0.705 / 0.195 / 0.479 | 0.798 / 0.545 / 0.637 | 0.712 / 0.042 / 0.390 |
| Model | 256-pixel copy | Original resolution |
|---|---|---|
| Qwen3.5-4B (base) | 0% | 12% |
| Stage 1 without counterfactual views | 0% | 5% |
| Stage 1 with counterfactual views | 19% | 40% |
| Stage 2 | not run | 40% |
| MedScope-4B (stage 3) | 0% | 0% |
| gemini-4-argon | 69% | 69% |
Medical VLMs
Medical vision-language models connect clinical images with text for diagnosis, reporting, question answering, and image grounding.
See all 500 papers