MedVeriSeg: Teaching LISA-Like Medical Segmentation Models to Verify Query Validity Without Extra Training
Authors: Qinyue Tong, Xiaozhen Wang, Ziqian Lu, Jun Liu, Yunlong Yu, Zheming Lu
Organizations: School of Aeronautics and Astronautics, Zhejiang University, Hangzhou 310027, China · Southern Medical University, Guangzhou 510515, Guangdong, China · School of Computer Science and Technology (School of Artificial Intelligence), Zhejiang Sci-Tech University, Hangzhou 310018, Zhejiang, China · College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, 310027 China
Despite recent progress in text-prompt-based medical image segmentation, existing LISA-like MLLM-based methods typically generate masks regardless of whether the target specified in the query is present, leading to hallucinated segmentation. In this work, we propose MedVeriSeg, a training-free query verification framework that enables LISA-like medical segmentation models to reject false segmentation queries. MedVeriSeg first quantifies the response quality between the [SEG] token and image features through a Similarity Response Quality Scoring Module. To further improve robustness, it employs a Lightweight Routed Multi-Agent Verification Module, which fuses quantitative score evidence with qualitative agent evidence to comprehensively verify the validity of the query. To support systematic evaluation, we construct MedVeriSeg-Bench, a benchmark designed for query verification in medical image segmentation. Experimental results demonstrate that MedVeriSeg effectively identifies false segmentation queries and reduces hallucinated segmentation, while maintaining a high acceptance rate for valid queries, thereby largely preserving the segmentation utility of LISA-like medical segmentation models.
Figures & tables
Fig. 1: Example of hallucinated segmentation by LISA-like models under false queries. MedVeriSeg enables these models to effectively identify false queries and prevent hallucinated mask generation.
Fig. 2: Distribution patterns of similarity heatmaps under true and false query cases. We use MediSee [ 11 ] as backbone for the visual analysis. The similarity matrix is computed between the hidden-state representation of the [SEG] token and the image features extracted from the final hidden layer of the MLLM.
Fig. 3: Distributions of 200 positive and 200 negative samples across the three SRQS dimensions: strength , compactness , and purity . The local overlap between positive and negative samples indicates limited separability under SRQS alone.
Fig. 4: Revisiting the LISA-style segmentation structure (a), using MediSee [ 11 ] as an example, and overview of the proposed MedVeriSeg framework (b). MedVeriSeg consists of two main components: the Similarity Response Quality Scoring Module (SRQS) and the Lightweight Routed Multi-Agent Verification Module (LMAV). SRQS derives quantitative evidence from the similarity matrix, while LMAV integrates it with qualitative evidence from other agents for comprehensive analysis.
Fig. 5: Clinician-in-the-Loop Negative Sample Construction for MedVeriSeg-Bench, which consists of both positive and negative samples. The negative samples are constructed using the above automatic pipeline, followed by clinician review to correct erroneous samples and refine overly simple samples.
Fig. 6: Analysis of the data structure in MedVeriSeg-Bench dataset. (a) Hierarchical distribution of imaging modalities and source datasets; (b) Word cloud of MedVeriSeg-Bench. Please zoom in for improved clarity .
Methods
Modalities
All Set
ct
mr_flair
mr_t1
mr_t1ce
overall
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Non-MLLM Verification Baselines
Random Verification [ 40 ]
50.29
44.34
50.52
55.78
50.26
52.01
48.91
57.14
50.37
49.91
Public-User Verification
43.61
36.32
35.51
40.19
43.48
43.87
44.54
49.40
42.05
40.05
Specialist-User Verification
90.05
87.80
87.21
89.44
82.96
84.69
85.15
87.55
88.23
88.00
TABLE I: Comparison of true–false query verification performance between MedVeriSeg and baseline methods across major medical imaging modalities in MedVeriSeg-Bench. For MLLM-based methods, Image and Heatmap indicate the visual information used for verification, and Simple Query and CoT Query denote different levels of textual guidance. This table reports the overall verification performance across positive and negative samples. The heatmap is produced by MediSee [ 11 ] .
Setting
Value
Agent MLLM Backbone
Qwen-3-VL
Python Version
3.10.6
CUDA Version
11.7
GPU Type
NVIDIA RTX A6000
GPU Memory Usage
about 19 GB
Token Cost per Sample
about 1600 tokens
TABLE II: Implementation details of MedVeriSeg (based on MediSee).
Category
Methods
DeepSeek-Janus-Pro
LLaVA-Med
Qwen-3-VL
MedVeriSeg (kimi-k2.6)
MedVeriSeg (Qwen-3-VL)
[ 26 ]
[ 17 ]
[ 25 ]
Ours
Ours
edema
(18.80, 99.64, 51.42)
(30.53, 97.47, 57.54)
(69.84, 91.88, 78.73)
(93.28, 69.13, 83.54)
(98.29, 73.29, 88.20)
enhancing tumor
(1.67, 100.00, 33.80)
(4.17, 97.00, 34.50)
(43.54, 91.85, 59.33)
(96.46, 69.53, 87.66)
(99.17, 71.24, 90.04)
non enhancing tumor
(1.07, 98.50, 41.65)
(3.48, 96.63, 42.28)
(31.55, 86.89, 54.60)
(89.04, 69.66, 80.97)
(91.98, 73.41, 84.24)
liver
(44.78, 91.45, 77.17)
(52.24, 89.47, 78.08)
(59.70, 79.61, 73.52)
(82.09, 85.53, 84.47)
(88.06, 79.61, 82.19)
TABLE III: Category-wise comparison of true–false query verification performance between MedVeriSeg and baseline methods on MedVeriSeg-Bench. Accuracy (Acc.) is used as the evaluation metric. Each triplet denotes the accuracy on positive samples, negative samples, and all samples, respectively. This table reports results on a subset of 16 categories from the dataset. For the compared baselines, the visual information is the image, and the query type is simple.
Method
Accuracy
F1
Precision
Recall
Average
Closed-source MLLMs
GPT-4o [ 43 ]
78.60
79.52
76.30
83.01
79.36
Gemini-2.5-Pro [ 44 ]
84.71
85.49
81.38
90.05
85.41
Open-source MLLMs
kimi-k2.6 [ 41 ]
81.37
82.36
78.26
86.93
82.23
Qwen-3-VL [ 25 ]
83.93
84.84
80.30
89.95
84.76
TABLE IV: Ablation study on the choice of agent MLLM backbones in the multi-agent module.
Fig. 7: Ablation study of the three evidence-providing agents in MedVeriSeg. The values reported in the figure denote accuracy (Acc), reflecting the overall performance across both positive and negative samples.
S
C
P
Easy Pos. + Easy Neg. Cases
ct
mr_flair
mr_t1
mr_t1ce
Overall
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
✓
46.08
58.31
72.32
80.00
65.22
73.05
69.43
77.64
57.01
67.26
✓
79.94
73.12
78.33
84.54
76.87
81.24
74.67
80.98
78.61
78.95
✓
71.25
70.35
67.62
79.23
66.96
76.43
69.65
79.59
69.74
74.97
✓
✓
82.47
88.03
87.02
92.11
84.57
90.07
83.89
89.74
84.43
89.88
TABLE V: Ablation study of different score dimensions in the Similarity Response Quality Scoring Module. S : Strength, C : Compactness, P : Purity.
Fig. 8: Qualitative analysis and case studies of MedVeriSeg. (a) Easy positive cases, (b) easy negative cases, and (c) ambiguous cases.
Fig. 9: Human evaluation results in terms of (a) human-query identification performance (Accuracy) and (b) practical utility assessment. In figure (b), scores are rated on a 1–5 scale, with higher scores indicating better performance.
Setting
Accuracy
F1
Precision
Recall
EWA Direct Decision
79.01
77.50
83.60
72.23
Ours
83.93
84.84
80.30
89.95
TABLE VI: Effect of quantitative weighted evidence fusion in the LMAV module. “EWA Direct Decision” denotes the Evidence Weighting Agent directly makes the final prediction.
Baseline
Setting
Positive Acc.
Negative Acc.
Average Acc.
Neg. Gain
Medical Model (test on MedVeriSeg-Bench)
MediSee [ 11 ]
w/o plug-in
100.00
0.00
50.00
–
MediSee
+ Ours
89.95
77.89
83.92
+ 77.89
MedPLIB [ 12 ]
w/o plug-in
100.00
0.00
50.00
–
MedPLIB
+ Ours
83.90
72.24
78.07
+ 72.24
General Model (test on 175 natural images)
TABLE VII: Performance of MedVeriSeg as a plug-in module for other LISA-like segmentation models.
Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.
Although Multimodal Large Language Models (MLLMs) have demonstrated impressive performance in Medical Visual Question Answering (Med-VQA), their reliance on global image features often lacks precise pixel-level grounding, thereby limiting clinical trustworthiness. To bridge the semantic gap between high-level clinical reasoning and spatial localization, we propose MedREAL (Medical REasoning-driven Answering and Localization), a unified framework that seamlessly aligns linguistic reasoning with spatial grounding. Specifically, MedREAL introduces Seg Anchored Reasoning Pooling (SARP) to distill task-relevant semantic evidence directly from [SEG] tokens within the MLLM's hidden states. Furthermore, a Reasoning-to-Visual (R2V) fusion mechanism is proposed to effectively inject these reasoning-aware features into a segmentation pipeline for accurate mask decoding. To facilitate this paradigm, we construct MedRAVS-13K, a comprehensive dataset comprising 13,824 expertly validated samples across four diverse imaging modalities. Extensive experiments demonstrate that MedREAL significantly outperforms state-of-the-arts, achieving 68.49% gIoU and 70.47% cIoU on benchmark evaluations. By generating evidence masks that are strictly consistent with textual diagnoses, MedREAL provides a robust, interpretable framework for reasoning-driven medical image analysis.
Haowen Gu, Gensheng Pei, Junzhu Mao +3
Nanjing University of Science and Technology, Nanjing, China · State Key Laboratory of Intelligent Manufacturing of Advanced Construction Machinery, Nanjing, China · Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, Korea
Despite notable progress in text-guided medical image segmentation nowadays, these methods are limited to single-round dialogues and fail to support multi-round reasoning, which is important for medical education scenarios. In this work, we introduce Multi-Round Entity-Level Medical Reasoning Segmentation (MEMR-Seg), a new task that requires generating segmentation masks through multi-round queries with entity-level reasoning, helping learners progressively develop their understanding of medical knowledge. To support this task, we construct MR-MedSeg, a large-scale dataset of 177K multi-round medical segmentation dialogues, featuring entity-based reasoning across rounds. Furthermore, we propose MediRound, an effective baseline model designed for multi-round medical reasoning segmentation. To mitigate the inherent error propagation within the chain-like pipeline of multi-round segmentation, we introduce a lightweight yet effective Judgment & Correction Mechanism during model inference. Experimental results demonstrate that our method effectively addresses the MEMR-Seg task and outperforms conventional medical referring segmentation methods. The project is available at https://github.com/Edisonhimself/MediRound.
Qinyue Tong, Ziqian Lu, Jun Liu +3
Zhejiang University, Hangzhou, Zhejiang, China · Zhejiang Sci-Tech University, Hangzhou, Zhejiang, China · National University of Singapore, Singapore
Training accurate medical image segmentation models requires large amounts of densely annotated data, which is costly and time-consuming to obtain. Semi-supervised learning (SSL) alleviates this by learning from both abundant unlabeled data and limited labeled data. However, most modern SSL methods rely on pseudolabels for unlabeled data, and typically assess their reliability through model confidence or uncertainty, measures that are self-referential and lack explicit grounding in segmentation quality. Instead, we propose a quality-guided SSL framework that trains a dedicated network to estimate segmentation quality from image-mask pairs. The predictor is trained on variable-quality masks generated through synthetic corruptions augmented with imperfect outputs from partially trained segmentation models, capturing realistic error patterns encountered during training. We integrate the quality predictor into SSL through two complementary mechanisms: a quality-aware regularization loss and a quality-based pseudolabel sample reweighting scheme. We show that our method serves as a drop-in enhancement to existing SSL frameworks. Extensive experiments across five datasets and multiple architectures demonstrate consistent improvements over competing SSL methods, advancing the state-of-the-art in semi-supervised medical image segmentation.
Kumar Abhishek, Ghassan Hamarneh
School of Computing Science, Simon Fraser University, Canada