Supervision-Aware Evaluation of MedGemma for Reference-Assisted Radiotherapy Contour Ranking
Organizations: Dartmouth College, Hanover, NH, USA · Mayo Clinic, Phoenix, AZ, USA
Abstract
Radiotherapy contour ranking can support quality assurance, but vision-language model (VLM) gains may reflect supervision, score construction, available inputs, or representation access. We evaluated frozen MedGemma 1.5 in a single-institution study, a 12-patient held-out HaN-Seg test, five-fold cross-fitting on 30 non-test HaN-Seg patients, and transfer without refitting to 120 SegRap2023 patients. On held-out HaN-Seg, a supervised readout of the final prompt state exceeded a supervised readout of the model's yes/no output probabilities by +0.3047 pooled average precision (AP; 95% patient-bootstrap interval [+0.2822, +0.3269]); the advantage persisted against a parameter-count-matched nonlinear control. A readout of projected visual tokens exceeded the final-state readout in cross-fitting and transfer (+0.0706 AP averaged equally across patients on SegRap; 95% interval [+0.0607, +0.0804]), while a structured panel-feature readout remained competitive with visual tokens. The anatomy-conditioned readout did not demonstrate an advantage over the native-feature comparator. These results motivate visual-token and structured-feature controls in medical-imaging VLM evaluations; outcomes were computational or synthetic, not clinical acceptability.
Figures & tables
| Interface or contrast | HaN-Seg cross-fit | SegRap zero-refit transfer | |
|---|---|---|---|
| Full population (primary) | Panel-identifiable sensitivity | ||
| Absolute patient-macro AP | |||
| P1-MLP: nonlinear output-space | 0.3630 | 0.3708 | 0.4074 |
| P2: final prompt state | 0.6339 | 0.5877 | 0.7266 |
| P4: pooled projected visual tokens | 0.7229 | 0.6583 | 0.8171 |
| P3+-MLP: all-panel structured features | 0.7896 | 0.6702 | 0.8290 |
VLM Evaluation
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
See all 1,060 papers