MedVeriSeg: Teaching LISA-Like Medical Segmentation Models to Verify Query Validity Without Extra Training

Qinyue Tong; Xiaozhen Wang; Ziqian Lu; Jun Liu; Yunlong Yu; Zheming Lu

cs.CVApr 11, 2026

MedVeriSeg: Teaching LISA-Like Medical Segmentation Models to Verify Query Validity Without Extra Training

Authors: Qinyue Tong, Xiaozhen Wang, Ziqian Lu, Jun Liu, Yunlong Yu, Zheming Lu

Organizations: School of Aeronautics and Astronautics, Zhejiang University, Hangzhou 310027, China · Southern Medical University, Guangzhou 510515, Guangdong, China · School of Computer Science and Technology (School of Artificial Intelligence), Zhejiang Sci-Tech University, Hangzhou 310018, Zhejiang, China · College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, 310027 China

Abstract

Despite recent progress in text-prompt-based medical image segmentation, existing LISA-like MLLM-based methods typically generate masks regardless of whether the target specified in the query is present, leading to hallucinated segmentation. In this work, we propose MedVeriSeg, a training-free query verification framework that enables LISA-like medical segmentation models to reject false segmentation queries. MedVeriSeg first quantifies the response quality between the [SEG] token and image features through a Similarity Response Quality Scoring Module. To further improve robustness, it employs a Lightweight Routed Multi-Agent Verification Module, which fuses quantitative score evidence with qualitative agent evidence to comprehensively verify the validity of the query. To support systematic evaluation, we construct MedVeriSeg-Bench, a benchmark designed for query verification in medical image segmentation. Experimental results demonstrate that MedVeriSeg effectively identifies false segmentation queries and reduces hallucinated segmentation, while maintaining a high acceptance rate for valid queries, thereby largely preserving the segmentation utility of LISA-like medical segmentation models.

Figures & tables

Fig. 1: Example of hallucinated segmentation by LISA-like models under false queries. MedVeriSeg enables these models to effectively identify false queries and prevent hallucinated mask generation.

Fig. 2: Distribution patterns of similarity heatmaps under true and false query cases. We use MediSee [ 11 ] as backbone for the visual analysis. The similarity matrix is computed between the hidden-state representation of the [SEG] token and the image features extracted from the final hidden layer of the MLLM.

Fig. 3: Distributions of 200 positive and 200 negative samples across the three SRQS dimensions: strength , compactness , and purity . The local overlap between positive and negative samples indicates limited separability under SRQS alone.

Fig. 4: Revisiting the LISA-style segmentation structure (a), using MediSee [ 11 ] as an example, and overview of the proposed MedVeriSeg framework (b). MedVeriSeg consists of two main components: the Similarity Response Quality Scoring Module (SRQS) and the Lightweight Routed Multi-Agent Verification Module (LMAV). SRQS derives quantitative evidence from the similarity matrix, while LMAV integrates it with qualitative evidence from other agents for comprehensive analysis.

Fig. 5: Clinician-in-the-Loop Negative Sample Construction for MedVeriSeg-Bench, which consists of both positive and negative samples. The negative samples are constructed using the above automatic pipeline, followed by clinician review to correct erroneous samples and refine overly simple samples.

Fig. 6: Analysis of the data structure in MedVeriSeg-Bench dataset. (a) Hierarchical distribution of imaging modalities and source datasets; (b) Word cloud of MedVeriSeg-Bench. Please zoom in for improved clarity .

Methods	Modalities								All Set
	ct		mr_flair		mr_t1		mr_t1ce		overall
	Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1
Non-MLLM Verification Baselines
Random Verification [ 40 ]	50.29	44.34	50.52	55.78	50.26	52.01	48.91	57.14	50.37	49.91
Public-User Verification	43.61	36.32	35.51	40.19	43.48	43.87	44.54	49.40	42.05	40.05
Specialist-User Verification	90.05	87.80	87.21	89.44	82.96	84.69	85.15	87.55	88.23	88.00

TABLE I: Comparison of true–false query verification performance between MedVeriSeg and baseline methods across major medical imaging modalities in MedVeriSeg-Bench. For MLLM-based methods, Image and Heatmap indicate the visual information used for verification, and Simple Query and CoT Query denote different levels of textual guidance. This table reports the overall verification performance across positive and negative samples. The heatmap is produced by MediSee [ 11 ] .

Setting	Value
Agent MLLM Backbone	Qwen-3-VL
Python Version	3.10.6
CUDA Version	11.7
GPU Type	NVIDIA RTX A6000
GPU Memory Usage	about 19 GB
Token Cost per Sample	about 1600 tokens

TABLE II: Implementation details of MedVeriSeg (based on MediSee).

Category	Methods
	DeepSeek-Janus-Pro	LLaVA-Med	Qwen-3-VL	MedVeriSeg (kimi-k2.6)	MedVeriSeg (Qwen-3-VL)
	[ 26 ]	[ 17 ]	[ 25 ]	Ours	Ours
edema	(18.80, 99.64, 51.42)	(30.53, 97.47, 57.54)	(69.84, 91.88, 78.73)	(93.28, 69.13, 83.54)	(98.29, 73.29, 88.20)
enhancing tumor	(1.67, 100.00, 33.80)	(4.17, 97.00, 34.50)	(43.54, 91.85, 59.33)	(96.46, 69.53, 87.66)	(99.17, 71.24, 90.04)
non enhancing tumor	(1.07, 98.50, 41.65)	(3.48, 96.63, 42.28)	(31.55, 86.89, 54.60)	(89.04, 69.66, 80.97)	(91.98, 73.41, 84.24)
liver	(44.78, 91.45, 77.17)	(52.24, 89.47, 78.08)	(59.70, 79.61, 73.52)	(82.09, 85.53, 84.47)	(88.06, 79.61, 82.19)

TABLE III: Category-wise comparison of true–false query verification performance between MedVeriSeg and baseline methods on MedVeriSeg-Bench. Accuracy (Acc.) is used as the evaluation metric. Each triplet denotes the accuracy on positive samples, negative samples, and all samples, respectively. This table reports results on a subset of 16 categories from the dataset. For the compared baselines, the visual information is the image, and the query type is simple.

Method	Accuracy	F1	Precision	Recall	Average
Closed-source MLLMs
GPT-4o [ 43 ]	78.60	79.52	76.30	83.01	79.36
Gemini-2.5-Pro [ 44 ]	84.71	85.49	81.38	90.05	85.41
Open-source MLLMs
kimi-k2.6 [ 41 ]	81.37	82.36	78.26	86.93	82.23
Qwen-3-VL [ 25 ]	83.93	84.84	80.30	89.95	84.76

TABLE IV: Ablation study on the choice of agent MLLM backbones in the multi-agent module.

Fig. 7: Ablation study of the three evidence-providing agents in MedVeriSeg. The values reported in the figure denote accuracy (Acc), reflecting the overall performance across both positive and negative samples.

$\mathcal{S}$	$\mathcal{C}$	$\mathcal{P}$	Easy Pos. + Easy Neg. Cases
			ct		mr_flair		mr_t1		mr_t1ce		Overall
			Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1
✓			46.08	58.31	72.32	80.00	65.22	73.05	69.43	77.64	57.01	67.26
	✓		79.94	73.12	78.33	84.54	76.87	81.24	74.67	80.98	78.61	78.95
		✓	71.25	70.35	67.62	79.23	66.96	76.43	69.65	79.59	69.74	74.97
✓	✓		82.47	88.03	87.02	92.11	84.57	90.07	83.89	89.74	84.43	89.88

TABLE V: Ablation study of different score dimensions in the Similarity Response Quality Scoring Module.

\mathcal{S}

: Strength,

\mathcal{C}

: Compactness,

\mathcal{P}

: Purity.

Fig. 8: Qualitative analysis and case studies of MedVeriSeg. (a) Easy positive cases, (b) easy negative cases, and (c) ambiguous cases.

Fig. 9: Human evaluation results in terms of (a) human-query identification performance (Accuracy) and (b) practical utility assessment. In figure (b), scores are rated on a 1–5 scale, with higher scores indicating better performance.

Setting	Accuracy	F1	Precision	Recall
EWA Direct Decision	79.01	77.50	83.60	72.23
Ours	83.93	84.84	80.30	89.95

TABLE VI: Effect of quantitative weighted evidence fusion in the LMAV module. “EWA Direct Decision” denotes the Evidence Weighting Agent directly makes the final prediction.

Baseline	Setting	Positive Acc.	Negative Acc.	Average Acc.	Neg. Gain
Medical Model (test on MedVeriSeg-Bench)
MediSee [ 11 ]	w/o plug-in	100.00	0.00	50.00	–
MediSee	+ Ours	89.95	77.89	83.92	+ 77.89
MedPLIB [ 12 ]	w/o plug-in	100.00	0.00	50.00	–
MedPLIB	+ Ours	83.90	72.24	78.07	+ 72.24
General Model (test on 175 natural images)

TABLE VII: Performance of MedVeriSeg as a plug-in module for other LISA-like segmentation models.

Medical Image Segmentation

Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.

See all 788 papers

From Reasoning to Pixels: Grounded Medical Multimodal LLMs for VQA and Segmentation
Aug 27, 2026Haowen Gu, Gensheng Pei, Junzhu Mao +3Medical VLMs Medical Image Grounding
Nanjing University of Science and Technology, Nanjing, China · State Key Laboratory of Intelligent Manufacturing of Advanced Construction Machinery, Nanjing, China · Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, Korea
MediRound: Multi-Round Entity-Level Reasoning Segmentation in Medical Images
Nov 15, 2025Qinyue Tong, Ziqian Lu, Jun Liu +3Interactive Medical Image Segmentation Text-Guided Image Segmentation
Zhejiang University, Hangzhou, Zhejiang, China · Zhejiang Sci-Tech University, Hangzhou, Zhejiang, China · National University of Singapore, Singapore
Quality-Guided Semi-Supervised Learning for Medical Image Segmentation
Jun 1, 2026Kumar Abhishek, Ghassan HamarnehSemi-Supervised Medical Image Segmentation Medical Image Segmentation
School of Computing Science, Simon Fraser University, Canada

Aug 27, 2026J/K move · Enter open · S save

From Reasoning to Pixels: Grounded Medical Multimodal LLMs for VQA and Segmentation

Haowen Gu, Gensheng Pei, Junzhu Mao, Qiong Wang+2

Nanjing University of Science and Technology, Nanjing, China · State Key Laboratory of Intelligent Manufacturing of Advanced Construction Machinery, Nanjing, China · Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, Korea

Medical VLMs Medical Image Grounding Medical VQA Medical Image Segmentation Multimodal Reasoning

Open paper AskPDF arXiv

Although Multimodal Large Language Models (MLLMs) have demonstrated impressive performance in Medical Visual Question Answering (Med-VQA), their reliance on global image features often lacks precise pixel-level grounding, thereby limiting clinical trustworthiness. To bridge the semantic gap between high-level clinical reasoning and spatial localization, we propose MedREAL (Medical REasoning-driven Answering and Localization), a unified framework that seamlessly aligns linguistic reasoning with spatial grounding. Specifically, MedREAL introduces Seg Anchored Reasoning Pooling (SARP) to distill task-relevant semantic evidence directly from [SEG] tokens within the MLLM's hidden states. Furthermore, a Reasoning-to-Visual (R2V) fusion mechanism is proposed to effectively inject these reasoning-aware features into a segmentation pipeline for accurate mask decoding. To facilitate this paradigm, we construct MedRAVS-13K, a comprehensive dataset comprising 13,824 expertly validated samples across four diverse imaging modalities. Extensive experiments demonstrate that MedREAL significantly outperforms state-of-the-arts, achieving 68.49% gIoU and 70.47% cIoU on benchmark evaluations. By generating evidence masks that are strictly consistent with textual diagnoses, MedREAL provides a robust, interpretable framework for reasoning-driven medical image analysis.

Abstract

Medical Image Segmentation

Explore similar work

From Reasoning to Pixels: Grounded Medical Multimodal LLMs for VQA and Segmentation

MediRound: Multi-Round Entity-Level Reasoning Segmentation in Medical Images

Quality-Guided Semi-Supervised Learning for Medical Image Segmentation