cs.CVMar 15, 2026
SaveHow Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images
Organizations: Singapore University of Technology and Design, Singapore · Department of Neurology, National Neuroscience Institute, Singapore · Duke-NUS Medical School, Singapore
Abstract
Generalist multimodal large language models (MLLMs) have achieved impressive performance across a wide range of vision-language tasks. However, their performance on medical tasks, particularly in zero-shot settings where generalization is critical, remains suboptimal. A key research gap is the limited understanding of why medical MLLMs underperform in medical image interpretation. In this work, we present a pioneering systematic investigation into the visual grounding capabilities of state-of-the-art medical MLLMs. To disentangle visual grounding from semantic grounding, we design VGMED, a novel evaluation dataset developed with expert clinical guidance, explicitly assessing the visual grounding capability of medical MLLMs. We introduce new quantitative metrics and conduct detailed qualitative analyses. Our study across eight state-of-the-art (SOTA) medical MLLMs validates that they often fail to ground their predictions in clinically relevant image regions. We note that this finding is specific to medical image analysis; in contrast, prior work has shown that MLLMs are capable of grounding their predictions in the correct image regions when applied to natural scene images. Motivated by these findings, we propose VGRefine, a simple yet effective inference-time method that refines attention distribution to improve visual grounding in medical settings. Our approach achieves SOTA performance across 6 diverse Med-VQA benchmarks (over 110K VQA samples from 8 imaging modalities) without requiring additional training or external expert models. Overall, our work, for the first time, systematically validates inadequate visual grounding as one of the key contributing factors for medical MLLMs' under-performance. Additional experiments are included in the Supp.
Figures & tables
Figure 1: Visual grounding issues in state-of-the-art medical MLLMs. (a) Column 1 shows input medical images with expert-annotated ground-truth regions (red boxes). Columns 2–5 display attention distributions from representative medical MLLMs. (b) Column 1 shows natural scene images with annotated ground-truth bounding boxes, and column 2 shows attention distributions from LLaVA-v1.5. For the first time, we systematically validate that state-of-the-art medical MLLMs often suffer from inadequate visual grounding —they fail to accurately localize and interpret image regions that are clinically relevant to the question. We note that, in contrast, when applied to natural images, MLLMs are capable of grounding their predictions in the correct image regions ( Zhang et al., 2025a ) . Attention maps are taken from the LLM layers identified as most relevant to visual grounding (see Sec. 2 for details).
Figure 2: Co-creation of VGMED with clinicians for visual grounding assessment. Existing Med-VQA datasets often include questions about image modality or plane, which can be answered without referencing specific image regions. They also contain many abnormality- or knowledge-based questions that require substantial medical expertise to determine what to look for. As a result, existing datasets are not well-suited for analyzing visual grounding. In contrast, our dataset leverages LLM prompting and clinical expert guidance to generate clinically meaningful localization and attribute questions that are explicitly grounded in annotated image regions, enabling rigorous assessment of the visual grounding capabilities of medical MLLMs. Best viewed in color and with zoom.
Figure 3: Medical MLLMs demonstrate suboptimal visual grounding when applied to medical images. Analysis using our proposed VGMED dataset—designed specifically to assess visual grounding in medical MLLMs—shows that all evaluated medical MLLMs exhibit substantial weaker alignment between their attention distributions and ground-truth annotations on medical images compared to natural scene images (from MS COCO). Additional comparison with general domain MLLM LLaVA-v1.5 on natural images (below the dashed line) further confirms that medical MLLMs consistently exhibit reduced alignment with annotated regions. Best viewed in color and with zoom.
Figure 4: Illustration of the proposed VGRefine method : a two-step inference-time method to improve visual grounding in medical MLLMs. In Step I (Attention Triage) , we aggregate attention from the model’s most visually sensitive heads and suppress low-confident attention, obtaining a binary mask. In Step II (Attention Knockout) , we use this mask to refine the model’s attention distribution, improving its focus on relevant regions during inference. In the lower triangular attention matrix, each row represents the attention score of a query token to all key tokens.
| Model | VQA-RAD | SLAKE | PathVQA | PMC-VQA | Avg. |
| Qwen-VL-Chat | 47.0 | 56.0 | 55.1 | 36.6 | 48.9 |
| LLaVA-v1.6-7B | 52.6 | 57.9 | 47.9 | 35.5 | 48.5 |
| Med-Flamingo | 45.4 | 43.5 | 54.7 | 23.3 | 41.7 |
| RadFM | 50.6 | 34.6 | 38.7 | 25.9 | 37.5 |
| LLaVA-Med-7B | 51.4 | 48.6 | 56.8 | 24.7 | 45.4 |
| LLaVA-Tri | 59.8 | 43.4 | 59.0 | - | - |
Table 1: Accuracy on medical VQA datasets. To align with the evaluation protocol with HuatuoGPT-V ( Chen et al., 2024a ) , we specifically used the closed-ended subset for evaluation. Evaluation on other subsets in Supp.
| Model | BMS | CM | DLM | P | PH | Avg. |
| Qwen-VL-Chat | 36.5 | 31.7 | 32.7 | 28.4 | 34.6 | 32.7 |
| LLaVA-v1.6-7B | 40.5 | 36.9 | 32.1 | 32.3 | 26.9 | 33.1 |
| Med-Flamingo | 29.6 | 28.1 | 24.8 | 25.3 | 31.2 | 28.3 |
| RadFM | 27.5 | 26.8 | 25.8 | 24.7 | 29.1 | 27.0 |
| LLaVA-Med-7B | 39.9 | 39.1 | 34.6 | 37.4 | 34.0 | 36.9 |
| LLaVA-Tri | 37.1 | - | 27.8 | - | - | - |
Table 2: Accuracy on MMMU Health & Medicine benchmark. BMS , CM , DLM , P , PH denote Basic Medical Science, Clinical Medicine, Diagnostics & Laboratory Medicine, Pharmacy, Public Health respectively.
| Model | CT | FP | MRI | OCT | Der | Mic | X-Ray | US | Avg. |
| Qwen-VL-Chat ( Bai et al., 2023 ) | 51.5 | 45.4 | 43.9 | 54.0 | 55.4 | 49.5 | 63.1 | 33.5 | 49.5 |
| LLaVA-v1.6-7B ( Liu et al., 2024a ) | 40.1 | 39.5 | 54.8 | 58.4 | 54.0 | 48.8 | 53.3 | 47.9 | 49.6 |
| Med-Flamingo ( Moor et al., 2023 ) | 34.6 | 33.3 | 27.5 | 26.0 | 28.3 | 28.1 | 30.1 | 33.2 | 30.2 |
| RadFM ( Wu et al., 2023 ) | 33.3 | 35.0 | 22.0 | 31.3 | 36.3 | 28.0 | 31.5 | 26.1 | 30.5 |
| LLaVA-Med-7B ( Li et al., 2023a ) | 25.3 | 48.4 | 35.9 | 42.1 | 45.2 | 44.0 | 31.7 | 34.4 | 35.8 |
| VILA-M3-8B ( Nath et al., 2024 ) | 60.2 | 35.7 | 51.5 | 56.9 | 51.5 | 51.7 | 65.4 | 46.1 | 53.0 |
Table 3: The accuracy of OmniMedVQA within different modalities. Specifically, FP denotes Fundus Photography , MRI denotes Magnetic Resonance Imaging , OCT denotes Optical Coherence Tomography , Der denotes Dermoscopy , Mic denotes Microscopy Images , US denotes Ultrasound .
| VQA-RAD | SLAKE | PathVQA | PMC-VQA | Avg. | (%) | VQA-RAD | SLAKE | PathVQA | PMC-VQA | Avg. | |
| 1 | 68.62 | 75.81 | 64.85 | 53.65 | 68.28 | 30 | 70.78 | 76.84 | 67.55 | 55.70 | 68.22 |
| 2 | 69.78 | 76.63 | 63.58 | 53.90 | 68.21 | 40 | 70.70 | 76.66 | 67.28 | 56.00 | 68.12 |
| 5 | 70.09 | 76.56 | 66.52 | 54.30 | 68.08 | 50 | 71.24 | 76.88 | 67.61 | 56.20 | 68.42 |
| 8 | 70.70 | 76.52 | 66.64 | 55.60 | 68.12 | 60 | 70.70 | 76.66 | 67.28 | 55.65 | 68.05 |
| 10 | 70.86 | 76.81 | 67.67 | 56.05 | 68.34 | 70 | 70.47 | 76.66 | 67.61 | 55.80 | 68.17 |
| 15 | 70.78 | 76.84 | 67.43 | 55.40 | 68.11 | 80 | 70.78 | 76.73 | 67.55 | 55.80 | 68.21 |
Table 4: Ablation study on the choice of top heads and precentile of magnitude-based filtering.
Figure A.1: Our proposed inference-time method VGRefine achieve state-of-the-art performance on OmniMedVQA ( Hu et al., 2024 ) and MMMU (Health & Medicine track) ( Yue et al., 2024 ) . Many existing medical MLLMs remain to underperform on medical VQA tasks in the zero-shot setting as shown in this figure, but there is a lack of systematic study to understand the reasons. Compared to existing medical MLLMs, our proposed VGRefine demonstrates consistently stronger zero-shot performance across all modalities and sub-domains, highlighting its effectiveness in mitigating the issue of inadequate visual grounding as revealed in our study.
Appendix figures & tables34 assets
Supplementary material from the paper’s appendix.
Appendix
| Related works | No. of Samples | Data Source |
| MLLMs Know ( Zhang et al., 2025a ) | 4,370 | Text-VQA |
| Your LVLM ( Kang et al., 2025 ) | 1,000 | RefCOCO |
| What’s in the Image ( Kaduri et al., 2024 ) | 81 | COCO |
| Hallucination Attribution ( Yang et al., 2025 ) | 1,500 | COCO |
| Devils in LVLM ( Jiang et al., 2025 ) | 2,000 | COCO |
| FastV ( Chen et al., 2025a ) | 1,000 | 4 VL Tasks |
Table B.1: Number of samples used in related works.
| Dataset | Modality | Anatomical Structures |
| AMOS2022 ( Ji et al., 2022 ) | CT, MR | Abdomen, Thorax, Pelvic |
| ATM2022 ( Zhang et al., 2023a ) | CT | Thorax |
| AbdomenomenCT-1K ( Ma et al., 2022b ) | CT | Abdomen |
| BTCV ( Igelsias et al., 2015 ) | CT | Thorax, Abdomen, Pelvic |
| BraTS2013 ( Menze et al., 2015 ) | MR | Head & neck |
| BraTS2015 ( Menze et al., 2015 ) | MR | Head & neck |
Table C.2: Detailed information about the 44 datasets incorporated into VGMED. In the ”Dataset” column, names such as ”StructSeg2019 (Task 1)” represent specific task-based subsets. In the ”Anatomical Structures” column, ”Others” signifies datasets lacking detailed anatomical data from their original sources.
Figure D.1: We conduct an experiment to analyze the alignment between attention distributions from different attention heads and layers and the ground truth annotations in the images. This follows the evaluation setup described in Section 2.3 of the main paper. The medical MLLM evaluated is HuatuoGPT-V-7B. Each cell in the above figures reflects the degree of alignment, measured using our proposed KL Divergence metric (lower is better). This analysis helps identify the specific heads and layers that are most relevant to visual grounding. COCO is used for natural scene image analysis, and our dataset VGMED is used for medical image analysis. Interestingly, we find that the attention heads most relevant to visual grounding in natural scene images are often also the most relevant for medical images. However, despite this overlap, the overall visual grounding performance on medical images remains lower than on natural scenes, consistent with the findings presented in Figure 3 (main paper). Based on this analysis, we identify the top attention heads with the strongest alignment (i.e., lowest KL divergence) and aggregate their attention distributions to compute a refined attention map. Notably, we select the top heads using randomly sampled natural scene images from COCO dataset, to avoid data leakage from medical evaluation benchmarks. This setup also demonstrates that our method generalizes effectively from natural images to the biomedical domain.
| Model | VQA-RAD | SLAKE | PathVQA | |||||||||
| Metric | BLEU-1 | BERT | OpenRecall | Avg. | BLEU-1 | BERT | OpenRecall | Avg. | BLEU-1 | BERT | OpenRecall | Avg. |
| Qwen-VL-Chat | 28.6 | 63.4 | 27.0 | 39.7 | 28.9 | 52.0 | 33.6 | 38.2 | 18.7 | 45.1 | 9.9 | 24.6 |
| LLaVA-v1.6-7B | 22.1 | 58.0 | 21.9 | 34.0 | 30.8 | 52.7 | 36.4 | 40.0 | 22.8 | 47.7 | 11.2 | 27.2 |
| Med-Flamingo | 27.4 | 61.9 | 12.7 | 34.0 | 11.8 | 40.2 | 21.1 | 24.4 | 24.3 | 50.4 | 2.4 | 25.7 |
| RadFM | 30.5 | 64.1 | 41.6 | 45.4 | 38.6 | 61.0 | 44.2 | 47.9 | 24.8 | 51.4 | 10.1 | 29.8 |
| LLaVA-Med-7B | 21.6 | 40.5 | 28.2 | 30.1 | 37.0 | 58.4 | 39.2 | 44.9 | 28.5 | 60.1 | 12.3 | 33.6 |
Table E.3: Performance comparison on full medical VQA datasets for open-ended medical VQA. We evaluate all models under the zero-shot setting. These results underscore that enhanced visual grounding with our inference-time method VGRefine contributes to better performance on medical VQA tasks. It is important to note that VILA-M3 ( Nath et al., 2024 ) , MedPLIB ( Huang et al., 2025 ) and LLaVA-Tri ( Xie et al., 2025 ) incorporate training data from VQA-RAD, SLAKE, and PathVQA, and thus making zero-shot evaluation unfair, and are excluded from our zero-shot comparison.
| Model | VQA-RAD | SLAKE | PathVQA | PMC-VQA | Avg. |
| HuatuoGPT-V-7B (Baseline) | 67.4 | 76.5 | 60.7 | 53.9 | 65.3 |
| PAI ( Liu et al., 2024c ) | 43.7 | 24.48 | 20.8 | 52.8 | 33.3 |
| AdaptVis ( Chen et al., 2025b ) | 68.6 | 75.1 | 67.6 | 52.9 | 66.7 |
| ViCrop ( Zhang et al., 2025a ) | 68.9 | 70.9 | 66.7 | 54.6 | 65.5 |
| VGRefine (Ours) | 71.2 | 76.9 | 67.6 | 56.2 | 68.4 |
Table F.4: Accuracy on closed-ended medical VQA datasets.
| Benchmarks | Subset | Metric | LLaVA-v1.6-34B | VILA-M3-13B | MedRegA-34B | HuatuoGPT-V-34B | VGRefine-34B (Ours) |
| VQA-RAD | - | CloseAcc | 58.6 | - | - | 68.1 | 72.9 |
| BLEU-1 | 44.5 | - | - | 50.5 | 52.6 | ||
| BERT | 69.2 | - | - | 74.8 | 74.8 | ||
| OpenRecall | 43.6 | - | - | 51.7 | 52.8 | ||
| SLAKE | - | CloseAcc | 67.3 | - | - | 76.9 | 79.1 |
| BLEU-1 | 48.6 | - | - | 56.3 | 57.2 |
Table G.5: Experiment results of larger models (more than 10B parameters). We evaluate all models under the zero-shot setting. Our inference-time method VGRefine outperforms other state-of-the-art medical MLLMs in most cases. These results underscore that enhanced visual grounding contributes to better performance on medical VQA tasks. It is important to note that VILA-M3 ( Nath et al., 2024 ) , MedRegA ( Wang et al., 2025 ) incorporate training data from VQA-RAD ( Lau et al., 2018 ) , SLAKE ( Liu et al., 2021a ) , PathVQA ( He et al., 2020 ) , and PMC-VQA ( Zhang et al., 2023b ) , thus making zero-shot evaluation unfair, and are excluded from our zero-shot comparison of these benchmarks.
Figure I.2: The question from a predefined question set is sampled for generating localization questions in both the COCO and VGMED datasets. {label} represents the object (in COCO) or organ/lesion (in VGMED) identified by a bounding box in the corresponding image.
Figure I.3: For attribute questions in VGMED, we use a specific prompt for each modality. In the prompt, {modality} denotes the modality of the image. {label} denotes the organ or lesion labeled by a bounding box in the image.
Figure I.4: For attribute questions in VGMED, we use a specific prompt for each modality. In the prompt, {modality} denotes the modality of the image. {label} denotes the organ or lesion labeled by a bounding box in the image.
Figure I.5: For attribute questions in VGMED, we use a specific prompt for each modality. In the prompt, {modality} denotes the modality of the image. {label} denotes the organ or lesion labeled by a bounding box in the image.
Figure I.6: For attribute questions in VGMED, we use a specific prompt for each modality. In the prompt, {modality} denotes the modality of the image. {label} denotes the organ or lesion labeled by a bounding box in the image.
Figure I.7: For attribute questions in VGMED, we use a specific prompt for each modality. In the prompt, {modality} denotes the modality of the image. {label} denotes the organ or lesion labeled by a bounding box in the image.
Figure I.8: For attribute questions in VGMED, we use a specific prompt for each modality. In the prompt, {modality} denotes the modality of the image. {label} denotes the organ or lesion labeled by a bounding box in the image.
Figure I.9: For attribute questions in VGMED, we use a specific prompt for each modality. In the prompt, {modality} denotes the modality of the image. {label} denotes the organ or lesion labeled by a bounding box in the image.
Figure I.10: For attribute questions in VGMED, we use a specific prompt for each modality. In the prompt, {modality} denotes the modality of the image. {label} denotes the organ or lesion labeled by a bounding box in the image.
Figure I.11: In order to compare the results between our VGMED datasets and natural scene images, we have also generated the attribute questions for COCO examples. {label} refers to the object label from the image’s bounding boxes.
Figure I.12: Prompt for evaluating the open and closed-ended questions in VQA-RAD, SLAKE, and PathVQA benchmarks.
Figure I.13: Prompt for evaluating the multiple-choice VQA benchmarks.
Figure J.14: Example of a blinded human evaluation case, showing a medical image with a VQA question, baseline attention map, and VGRefine attention map, assessed by an experienced clinician for clinical reasonableness. Clinician feedback highlighted that VGRefine attention maps were less noisy, better localized, and more aligned with expected clinical focus points.
Figure J.15: Qualitative evaluation. We visualize attention maps across different layers, including those with the lowest KL divergence (highlighted with an orange boundary), which are indicative of layers most relevant to visual grounding in MLLMs. For medical images, the attention maps of medical MLLMs show limited alignment with the ground-truth annotated regions. In contrast, a general-domain MLLM LLaVA-v1.5 applied to natural images exhibits strong alignment with relevant regions, consistent with other study of general-domain MLLMs ( Zhang et al., 2025a ) . This highlights a gap in MLLM’s visual grounding performance between the medical and natural image domains. Best viewed in color and with zoom. Additional results in Supp J.2 .
Figure J.16: Qualitative evaluation of (i) medical MLLMs HuatuoGPT-V, LLaVA-Tri and MedRegA on COCO, and (ii) LLaVA-v1.5 on VGMED. We visualize attention maps across different layers, including those with the lowest KL divergence (highlighted with an orange boundary), which are indicative of layers most relevant to visual grounding in MLLMs. We observe that LLaVA-v1.5 fails to ground predictions in clinically relevant regions when operating on medical images and medical VQA tasks. Furthermore, medical-domain models can ground their predictions when applied to natural images. This is consistent with our quantitative analysis in Fig. 3 of the main paper. Together, they show that medical MLLMs possess good visual grounding capabilities in general-domain settings. Overall, this confirms that the grounding failure is not due to model weakness, but is fundamentally specific to the medical domain, consistent with our central findings. Inadequate visual grounding is a medical-domain failure mode.
Figure J.17: Qualitative evaluation. We visualize attention maps across different layers, including those with the lowest KL divergence (highlighted with an orange boundary), which are indicative of layers most relevant to visual grounding in MLLMs. For medical images, the attention maps of medical MLLMs show limited alignment with the ground-truth annotated regions. In contrast, a general-domain MLLM LLaVA-v1.5 applied to natural images exhibits strong alignment with relevant regions, consistent with other study of general-domain MLLMs Zhang et al. (2025a) . This highlights a gap in MLLM’s visual grounding performance between the medical and natural image domains.
Figure J.18: Qualitative evaluation. We visualize attention maps across different layers, including those with the lowest KL divergence (highlighted with an orange boundary), which are indicative of layers most relevant to visual grounding in MLLMs. For medical images, the attention maps of medical MLLMs show limited alignment with the ground-truth annotated regions. In contrast, a general-domain MLLM LLaVA-v1.5 applied to natural images exhibits strong alignment with relevant regions, consistent with other study of general-domain MLLMs Zhang et al. (2025a) . This highlights a gap in MLLM’s visual grounding performance between the medical and natural image domains.
Figure J.19: Qualitative evaluation. We visualize attention maps across different layers, including those with the lowest KL divergence (highlighted with an orange boundary), which are indicative of layers most relevant to visual grounding in MLLMs. For medical images, the attention maps of HuatuoGPT-V-Bio with a specialized vision encoder (BiomedCLIP) show limited alignment with the ground-truth annotated regions. In contrast, a general-domain MLLM LLaVA-v1.5 applied to natural images exhibits strong alignment with relevant regions, consistent with other study of general-domain MLLMs Zhang et al. (2025a) .
Figure J.20: Qualitative evaluation. We visualize attention maps across different layers, including those with the lowest KL divergence (highlighted with an orange boundary), which are indicative of layers most relevant to visual grounding in MLLMs. For medical images, the attention maps of medical MLLMs show limited alignment with the ground-truth annotated regions. In contrast, a general-domain MLLM LLaVA-v1.5 applied to natural images exhibits strong alignment with relevant regions, consistent with other study of general-domain MLLMs Zhang et al. (2025a) . This highlights a gap in MLLM’s visual grounding performance between the medical and natural image domains.
Figure J.21: Qualitative evaluation. We visualize attention maps across different layers, including those with the lowest KL divergence (highlighted with an orange boundary), which are indicative of layers most relevant to visual grounding in MLLMs. For medical images, the attention maps of medical MLLMs show limited alignment with the ground-truth annotated regions. In contrast, a general-domain MLLM LLaVA-v1.5 applied to natural images exhibits strong alignment with relevant regions, consistent with other study of general-domain MLLMs Zhang et al. (2025a) . This highlights a gap in MLLM’s visual grounding performance between the medical and natural image domains.
Figure J.22: Qualitative evaluation. We visualize attention maps across different layers, including those with the lowest KL divergence (highlighted with an orange boundary), which are indicative of layers most relevant to visual grounding in MLLMs. For medical images, the attention maps of medical MLLMs show limited alignment with the ground-truth annotated regions. In contrast, a general-domain MLLM LLaVA-v1.5 applied to natural images exhibits strong alignment with relevant regions, consistent with other study of general-domain MLLMs Zhang et al. (2025a) . This highlights a gap in MLLM’s visual grounding performance between the medical and natural image domains.
Figure J.23: Qualitative evaluation. We visualize attention maps across different layers, including those with the lowest KL divergence (highlighted with an orange boundary), which are indicative of layers most relevant to visual grounding in MLLMs. For medical images, the attention maps of medical MLLMs show limited alignment with the ground-truth annotated regions. In contrast, a general-domain MLLM LLaVA-v1.5 applied to natural images exhibits strong alignment with relevant regions, consistent with other study of general-domain MLLMs Zhang et al. (2025a) . This highlights a gap in MLLM’s visual grounding performance between the medical and natural image domains.
Figure J.24: Qualitative evaluation. We visualize attention maps across different layers, including those with the lowest KL divergence (highlighted with an orange boundary), which are indicative of layers most relevant to visual grounding in MLLMs. For medical images, the attention maps of medical MLLMs show limited alignment with the ground-truth annotated regions. In contrast, a general-domain MLLM LLaVA-v1.5 applied to natural images exhibits strong alignment with relevant regions, consistent with other study of general-domain MLLMs Zhang et al. (2025a) . This highlights a gap in MLLM’s visual grounding performance between the medical and natural image domains.
Figure K.25: (a) Quantitative and (b) qualitative evaluation of InternVL3-8B and Qwen2.5-VL-7B on VGMED and COCO. We observe that the visual grounding deficiency in medical domain persists even in these latest general-purpose models.
Figure L.26: Comparison of visual grounding when using all input tokens , question-only tokens , or the last token to derive attention maps. Using two representative MLLMs (HuatuoGPT-V-7B and LLaVA-v1.5), we evaluate how different token-selection strategies affect attention alignment on VGMED and COCO. Across all metrics and layers, attention maps computed from the last token achieves equal or better alignment with ground-truth regions compared to the alternative options.
Figure M.27: Representative failure cases of HuatuoGPT-Vision on medical benchmarks. (a) The model correctly interprets the question but attends to the wrong anatomical region, leading to an incorrect answer. After applying VGRefine, the model’s attention shifts toward more clinically relevant region, resulting in the correct prediction. (b) The model misunderstand the question, resulting in both semantic and visual grounding failure. (c) Additionally, we include examples from LLaVA-v1.5 on natural images as a reference of accurate visual grounding. While multiple factors contribute to poor generalization, weak visual grounding consistently emerges as a major and measurable issue, though not the sole cause.
Figure N.28: Example of the clinician rating interface used during VGMED curation.
| Type | Category | Rating 3 (%) | Rating 4 (%) | Rating 5 (%) |
| Attribute | Clinical Relevance | 3.31 | 4.11 | 92.58 |
| Min. Semantic Grounding | 0.37 | 10.38 | 89.25 | |
| Visual Grounding | 4.04 | 12.18 | 83.77 | |
| Localization | Clinical Relevance | 0.02 | 0.52 | 99.46 |
| Min. Semantic Grounding | 0.05 | 5.76 | 94.19 | |
| Visual Grounding | 3.96 | 11.79 | 84.25 |
Table N.6: Percentage distribution of clinician ratings (3–5) across all axes for Attribute and Localization questions.