From Reasoning to Pixels: Grounded Medical Multimodal LLMs for VQA and Segmentation

Haowen Gu; Gensheng Pei; Junzhu Mao; Qiong Wang; Mingwu Ren; Yazhou Yao

cs.CVAug 27, 2026

From Reasoning to Pixels: Grounded Medical Multimodal LLMs for VQA and Segmentation

Authors: Haowen Gu, Gensheng Pei, Junzhu Mao, Qiong Wang, Mingwu Ren, Yazhou Yao

Organizations: Nanjing University of Science and Technology, Nanjing, China · State Key Laboratory of Intelligent Manufacturing of Advanced Construction Machinery, Nanjing, China · Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, Korea

Abstract

Although Multimodal Large Language Models (MLLMs) have demonstrated impressive performance in Medical Visual Question Answering (Med-VQA), their reliance on global image features often lacks precise pixel-level grounding, thereby limiting clinical trustworthiness. To bridge the semantic gap between high-level clinical reasoning and spatial localization, we propose MedREAL (Medical REasoning-driven Answering and Localization), a unified framework that seamlessly aligns linguistic reasoning with spatial grounding. Specifically, MedREAL introduces Seg Anchored Reasoning Pooling (SARP) to distill task-relevant semantic evidence directly from [SEG] tokens within the MLLM's hidden states. Furthermore, a Reasoning-to-Visual (R2V) fusion mechanism is proposed to effectively inject these reasoning-aware features into a segmentation pipeline for accurate mask decoding. To facilitate this paradigm, we construct MedRAVS-13K, a comprehensive dataset comprising 13,824 expertly validated samples across four diverse imaging modalities. Extensive experiments demonstrate that MedREAL significantly outperforms state-of-the-arts, achieving 68.49% gIoU and 70.47% cIoU on benchmark evaluations. By generating evidence masks that are strictly consistent with textual diagnoses, MedREAL provides a robust, interpretable framework for reasoning-driven medical image analysis.

Figures & tables

Figure 1: Comparison between Medical VQA, promptable segmentation, and the proposed MedREAL framework. MedREAL introduces an explicit evidence token to extract reasoning-aligned features, which are fused with visual context to produce answer-consistent evidence localization.

Figure 2: Overview of MedREAL . A medical image and question are processed by a VLM producing answer tokens including [SEG]. SARP distills reasoning-relevant features from [SEG], which are fused with global visual features via R2V Fusion to condition the mask decoder, producing an evidence-aligned segmentation mask guided by both reasoning and visual context.

Figure 4: Dataset Generation Pipeline of MedRAVS-13K. The process illustrates the integration of raw images and masks to generate reasoning-aware text, followed by rigorous quality assurance.

Dataset	Class	Train Samples	Val Samples	Referring Phrase
BUSI [ 2 ]	Benign	915	396	benign breast mass
	Malignant	441	189	malignant breast tumor
	Normal	276	123	normal breast tissue
COVID-QU-Ex [ 48 ]	COVID-19	2330	582	COVID-19 lung infection
	Non-COVID	1162	291	non-COVID lung infection
	Normal	1164	291	normal lung region

Table 1: Referring phrases used for different datasets and semantic classes. These ground-truth derived templates serve as explicit prompts for non-reasoning baselines.

Refering-based Segmentation
Method	BUSI		COVID-QU-Ex		ISIC-2018		Kvasir-SEG		Overall
Method	gIoU	cIoU	gIoU	cIoU	gIoU	cIoU	gIoU	cIoU	gIoU	cIoU
OVSeg [ 30 ]	4.72	4.79	17.59	16.75	26.19	24.56	13.88	13.58	13.48	18.75
SAM3 [ 10 ]	17.69	0.68	0.00	0.00	1.04	1.78	0.00	0.00	5.43	1.39
MedSAM3 [ 32 ]	62.93	58.97	71.56	75.03	83.03	80.43	81.82	82.02	70.99	78.28
Agent-based Segmentation

Table 2: Comparison of gIoU and cIoU across different datasets. Best results are highlighted in bold , and second-best are underlined .

	BLEU					ROUGE
Method	BLEU	BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE-1	ROUGE-2	ROUGE-L
LISA [ 26 ]	24.04	56.94	29.69	18.88	13.15	59.81	33.41	48.11
MedREAL ( Ours )	89.12	98.75	97.74	97.00	96.26	96.69	95.84	96.69

Table 3: Comparison of text generation performance between LISA and our method.

Box Guide	SARP	R2V	gIoU (%)	cIoU (%)
✗	✗	✗	11.35	19.69
$\checkmark$	✗	✗	45.02	52.84
✗	$\checkmark$	$\checkmark$	36.24	37.74
$\checkmark$	✗	$\checkmark$	23.22	47.12
$\checkmark$	$\checkmark$	✗	45.50	46.96
$\checkmark$	$\checkmark$	$\checkmark$	68.49	70.47

Table 4: Ablation study isolating the contributions of bounding-box guidance, SARP, and R2V modules.

Backbone MLLM	gIoU (%)	cIoU (%)
Qwen3-VL-2B-Instruct	68.49	70.47
Qwen3-VL-4B-Instruct	75.96	70.26

Table 5: Effect of MLLM scaling on reasoning-driven segmentation performance. Upgrading the backbone directly translates to superior localization.

Figure 5: Qualitative comparison with competing methods across four diverse medical imaging modalities (Top to Bottom: BUSI [ 2 ] , COVID-QU-Ex [ 48 ] , ISIC-2018 [ 12 ] , and Kvasir-SEG [ 22 ] ). MedREAL consistently produces precise evidence masks that are robust to structural artifacts and highly aligned with the underlying clinical reasoning.

Panel of Figure 6 — Figure 6: Qualitative comparison between MedREAL and LISA. MedREAL demonstrates superior semantic consistency between the generated VQA text and the spatial segmentation mask.

Medical VLMs

Medical vision-language models connect clinical images with text for diagnosis, reporting, question answering, and image grounding.

See all 458 papers

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering
Apr 10, 2026Suyang Xi, Songtao Hu, Yuxiang Lai +4Medical VQA Medical VLMs
Department of Radiation Oncology and Winship Cancer Institute, Emory University School of Medicine, Atlanta, 30322, GA, USA · Department of Biostatistics and Bioinformatics, Emory University, Atlanta, 30322, GA, USA
MedPixel: A Unified Pixel-Language Model for Medical Reasoning and Segmentation
Aug 10, 2026Haoyu Yang, Meixing Shi, Zengjie Chen +5Medical VLMs VLM Reasoning
Zhejiang University · Fudan University · Kuaishou +2
LocAnyMed: Vision-Language Grounding for Multimodal Medical Images
Aug 4, 2026Zihan Wang, Tong Liu, Zhiwei Wang +6Medical VLMs Medical Image Grounding
School of Computer Science, Wuhan University, Wuhan, China · College of Computing and Data Science, Nanyang Technological University, Singapore · Australian Artificial Intelligence Institute, School of Computer Science, University of Technology Sydney, Sydney, Australia +1

Apr 10, 2026J/K move · Enter open · S save

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering

Suyang Xi, Songtao Hu, Yuxiang Lai, Wangyun Dan+3

Department of Radiation Oncology and Winship Cancer Institute, Emory University School of Medicine, Atlanta, 30322, GA, USA · Department of Biostatistics and Bioinformatics, Emory University, Atlanta, 30322, GA, USA

Medical VQA Medical VLMs Latent Visual Reasoning RL for VLMs

Open paper AskPDF arXiv

Medical vision--language models (VLMs) have shown strong potential for medical visual question answering (VQA), yet their reasoning remains largely text-centric: images are encoded once as static context, and subsequent inference is dominated by language. This paradigm is fundamentally limited in clinical scenarios, where accurate answers often depend on subtle, localized visual evidence that cannot be reliably preserved in static embeddings. We propose MedLVR, a latent visual reasoning framework that introduces an explicit visual evidence state into autoregressive decoding. Instead of relying solely on text-based intermediate reasoning, MedLVR interleaves a short latent reasoning segment within the decoder by reusing hidden states as continuous latent steps, enabling iterative preservation and refinement of query-relevant visual evidence before answer generation. To support effective visual supervision, we adopt a two-stage training strategy: region of interest (ROI)-supervised fine-tuning aligns latent states with clinically relevant image evidence, and Visual-Latent Policy Optimization (VLPO) further optimizes latent reasoning and answer generation under outcome-level rewards. Experiments on OmniMedVQA and five external medical VQA benchmarks show that MedLVR consistently outperforms recent reasoning baselines and improves the average score over the Qwen2.5-VL-7B backbone from 48.3% to 53.4%. These results show that latent visual reasoning provides an effective mechanism for preserving diagnostically relevant visual evidence and improving the reliability of medical VQA.

Abstract

Medical VLMs

Explore similar work

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering

MedPixel: A Unified Pixel-Language Model for Medical Reasoning and Segmentation

LocAnyMed: Vision-Language Grounding for Multimodal Medical Images