SegBanana: Steering Unified Multimodal Models into Medical Segmenters

Xiaoye Liang; Ye Yan; Mingze Yin; Shikun Feng; Mai Xu; Haiguang Liu; Lai Jiang; Yiheng Zhu

cs.CVSep 28, 2026

SegBanana: Steering Unified Multimodal Models into Medical Segmenters

Authors: Xiaoye Liang, Ye Yan, Mingze Yin, Shikun Feng, Mai Xu, Haiguang Liu, Lai Jiang, Yiheng Zhu

Organizations: Beihang University · Zhongguancun Academy · Zhejiang University

Abstract

Medical image segmentation remains challenging in practical deployment, as models often struggle to generalize beyond the distributions covered by their training data and high-quality pixel-level annotations are typically unavailable for adaptation. Inspired by the cross-task transferability of large language models, we investigate whether unified multimodal models (UMMs) can transfer their pretrained visual understanding, reasoning, and generation capabilities to medical image segmentation without task-specific post-training. By recasting segmentation as structured visual generation, we find that frontier UMMs (e.g., Nano Banana) already exhibit basic segmentation capabilities across diverse clinical scenarios, but still struggle with challenging tasks requiring specialized anatomical or domain-specific knowledge. We further show that these limitations can be effectively mitigated by incorporating visual anatomical knowledge from in-context exemplars, expanding candidate solutions through repeated sampling, and refining suboptimal predictions via targeted editing.Motivated by these observations, we propose SegBanana, to our knowledge, the first agentic visual generation framework for training-free medical image segmentation. SegBanana builds on a frozen UMM as the core generative model, augmented with Anatomy-Aware Knowledge Retrieval and Comparative Quality Critique to unlock its potential segmentation capability. A State-Aware Multimodal Controller maintains structured state and iteratively orchestrates these tools, repeatedly refining intermediate predictions toward higher-quality masks. Across eight medical segmentation datasets, SegBanana achieves an average mDice of 77.45%, outperforming representative generalist (SAM3 and SegGPT) and medical-specific (BiomedParse and MedSAM3) baselines by at least 14.93 points, while remaining robust to out-of-domain visual supports.

Figures & tables

Figure 1: Comparison of generalization paradigms for medical image segmentation, including medical-specific methods, generalist methods, and our SegBanana.

Dataset	Task Characteristics					mDice (%)
Dataset	Modality	Target	Size	Structure	Main Challenge	Text-only	Random	Retrieved
ACDC ( Bernard et al., 2018 )	MRI	RV / MYO / LV	Small–Med.	Adjacent	Anatomical coupling	36.89	31.79	45.30
Drishti-GS ( Sivaswamy et al., 2014 )	Fundus	Disc / Cup	Small	Nested	Subtle boundaries	43.73	53.57	57.49
ISIC ( Codella et al., 2019 )	Dermoscopy	Lesion	Med.–Large	Irregular	Appearance variation	74.68	79.62	80.65

Table 1: Task characteristics and performance under text-only and support-augmented prompting, where Random samples a support and Retrieved selects a query-relevant one.

Figure 2: Inference-time exploration and refinement. (a) Best-of- $N$ performance with increasing sampling. (b) Improvement under varying editing guidance. (c) Before–after Dice across error levels and prompt types, with shaded regions indicating their distributions.

Figure 3: Overview of SegBanana. A State-Aware Multimodal Controller orchestrates Anatomy-Aware Knowledge Retrieval, UMM-based Mask Generation and Refinement, and Comparative Quality Critique, using state and feedback to adapt inference.

Medical-Specific Methods
Method	TNBC	RAVIR	Drishti-GS	ISIC	BUS-UCLM	Kvasir	ACDC	BraTS	Avg
UniverSeg $\blacktriangle$	24.48	28.45	40.68	39.64	13.38	19.91	21.48	15.48	25.44
BiomedParse $\blacklozenge$	11.81	5.67	78.19	87.59	63.81	88.95	89.03	75.11	62.52
MedSAM3 $\blacklozenge$	68.21	4.21	34.13	89.80	57.68	91.17	16.74	55.42	52.17
IBISAgent $\blacklozenge$	1.00	1.65	4.14	87.47	57.06	78.86	14.87	60.14	38.15
MedSAMAgent $\blacklozenge$	5.78	0.30	66.41	83.47	49.16	83.92	80.50	21.38	48.87

Table 2: Comparison of SegBanana against medical-specific and generalist baselines across eight medical segmentation datasets in terms of mDice (%).

\blacklozenge

\blacktriangle

denote zero-/few-shot settings. Gray denotes medical-specific results lower than SegBanana. Bold / underline indicate the best/second-best results within each method category.

Figure 4: In-domain vs. out-of-domain mDice (%) across datasets. Points closer to the diagonal indicate greater robustness to support-domain shift.

Framework	Controller	TNBC	RAVIR	Drishti-GS	ISIC	BUS-UCLM	Kvasir	ACDC	BraTS	Avg.
SegBanana	Gemini-3.5	74.75	80.93	87.09	82.06	78.07	89.53	61.68	65.52	77.45
w/o Retrieval	Gemini-3.5	76.26	81.22	60.55	82.78	77.92	88.19	51.56	61.67	72.52
w/o Critic	Gemini-3.5	72.08	78.05	63.88	81.16	75.07	86.02	56.68	65.66	72.33
w/o State Context	Gemini-3.5	73.86	77.30	65.07	81.85	76.18	87.74	56.36	63.43	72.72
SegBanana	Qwen-3.8	70.06	41.94	83.28	79.03	69.37	82.31	51.31	54.32	66.45
SegBanana	GPT-5.6	62.38	54.36	88.15	82.78	77.54	85.39	55.39	61.88	70.98

Table 3: Ablation of framework components and multimodal controller backbones across eight datasets. Results are reported in mDice (%).

Figure 5: Iterative segmentation performance across inference steps, with the $\Delta$ mDice of critique-selected predictions and the oracle performance among all masks generated up to each iteration.

Figure 6: Case study of SegBanana, illustrating its iterative segmentation process and comparison with the top five methods from medical-specific and generalist baselines.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Modality	Target	Structure	Primary Challenge
ACDC	MRI	LV / RV / MYO	Adjacent	Anatomical coupling
Drishti-GS	Fundus	Optic disc / cup	Nested	Subtle boundary discrimination
ISIC	Dermoscopy	Skin lesions	Irregular	Appearance variation
TNBC	Histopathology	Cell nuclei	Irregular, Adjacent	Dense clustering & appearance variation
RAVIR	Fundus	Retinal vessels	Irregular	Subtle boundaries & topology preservation
BUS-UCLM	Ultrasound	Breast lesions	Irregular	Appearance variation & weak boundaries

Appendix

Table 4: Characteristics of the evaluation datasets in terms of imaging modality, segmentation target, spatial structure, and primary segmentation challenge.

Modality	UniverSeg	BiomedParse	MedSAM3	IBISAgent	MedSAM-Agent
MRI	BraTS , BrainDevelopment, ISLES, LGG, MCIC, OASIS, PPMI, WMH, I2CVB, NCI-ISBI, PROMISE12, FeTA, SpineWeb, ACDC , CDEMRIS	BraTS2023 , PROMISE12, LGG, ACDC , M&Ms	MRI corpus ‡	ACDC , LGG, M&Ms, MSD Brain Tumor, MSD Heart, MSD Hippocampus, MSD Prostate	ACDC , LGG, AMOS-MRI
CT	AbdomenCT-1K, BTCV, KiTS, LiTS, LUNA, SegTHOR, WORD	AbdomenCT-1K, BTCV, KiTS23, TotalSegmentator, COVID-19 CT, LIDC-IDRI, FUMPE	CT corpus ‡	COVID-19 CT, KiTS23, LIDC-IDRI, MSD Liver, MSD Spleen, MSD Pancreas, MSD Colon, MSD Lung, MSD Hepatic Vessel	FLARE22, KiTS, LIDC-IDRI, BTCV, AMOS-CT, WORD
MRI & CT	AMOS, CHAOS, MSD	AMOS22, MSD	–	AMOS22, MSD task groups	–
Fundus / Retinal	DRIVE, STARE, IDRID, e-Ophtha	DRIVE, REFUGE, G1020	RIM-ONE	G1020, REFUGE	REFUGE
X-ray	CheXplanation, PanDental	COVID-QU-Ex, QaTa-COV19, SIIM-ACR Pneumothorax, Chest Xray Masks and Labels, CDD-CESM, Radiography	X-ray corpus ‡	COVID-QU-Ex, QaTa-COV19, Radiography, CDD-CESM, SIIM-ACR Pneumothorax, CXR Masks and Labels	CXRMask, Radiography series, CDD-CESM
Ultrasound	CAMUS, HMC-QU, BUS, TUCC	CAMUS, BUSI, US Simulation & Segmentation, BreastUS, LiverUS, FH-PS-AOP	BUSI; broader US corpus ‡	CAMUS, BreastUS, LiverUS, FH-PS-AOP	BreastUS, LiverUS, FH-PS-AOP

Appendix

Table 5: Reported medical data exposure of the compared medical-specific methods. Datasets used during any model-development stage, including pretraining, fine-tuning/SFT, and RL, are counted as exposure. Gray rows denote modalities included in our evaluation benchmark, while red dataset names indicate benchmark-level overlap with our evaluation datasets.

\ddagger

indicates that exposure to the modality is explicitly reported, but the complete dataset-level training list is not publicly disclosed.

Figure 7: Out-of-domain support construction across six medical segmentation tasks. For each task, the query dataset (left) is paired with an external annotated dataset (right) sharing the same or closely corresponding segmentation target, while differing in acquisition conditions, appearance, population, or annotation characteristics. Example image–mask pairs illustrate the resulting domain shift while preserving task-relevant anatomical semantics.

Dataset	Task Description
ACDC	Segment the right ventricular cavity, myocardium, and left ventricular cavity.
BUS-UCLM	Segment the breast lesion or mass.
Kvasir	Segment the gastrointestinal polyp.
RAVIR	Segment the retinal vessels.
BraTS	Segment the brain tumor.
TNBC	Segment the cell nuclei.

Appendix

Table 6: Dataset-specific task descriptions used in prompts.

Figure 8: Knowledge Sufficiency Assessment prompt. The controller assesses whether additional task-specific anatomical knowledge is required from the task description, query image, and compact structured state, and returns a fixed JSON decision indicating whether support retrieval is needed.

Figure 9: Generation Instruction-Synthesis prompt. The controller combines the task description, stage-relevant structured state, query image, and optional retrieved support to produce an executable generation instruction for the frozen UMM.

Figure 10: Editing Instruction-Synthesis prompt. The controller transforms the retained prediction, residual-error diagnosis, and relevant interaction trajectory into a targeted correction instruction specifying the structure, error, and desired local revision.

Figure 11: Generation Execution prompt. The frozen UMM receives the query image, synthesized generation instruction, optional selected support pair, and output contract, and directly generates an RGB segmentation mask.

Figure 12: Editing Execution prompt. The frozen UMM locally revises the retained best mask using the query image and synthesized correction instruction while preserving reliable regions and following the output contract.

Figure 13: Comparative Quality Critique prompt. A frozen multimodal critic compares a baseline or current champion against a challenger, performs label-wise quality assessment and residual-error diagnosis, and returns the comparison in a fixed structured format.

Figure 14: Action Selection prompt. Based on the compact structured state and comparative critique, the controller selects Generate , Edit , or Accept , and specifies the target residual error when editing is chosen.

Figure 15: Additional analyses for the observations in the main text. (a) Diversity-aware retrieval broadens the accessible hypothesis space: query–support similarity and support redundancy are compared between similarity-only and MMR-based retrieval; marker color indicates hypothesis dispersion measured by the standard deviation of Dice scores across sampled predictions, and the inset shows its correlation with Best-of-4 improvement. (b) Editing and resampling provide complementary refinement mechanisms: the blue curve reports the mean Dice improvement and the orange curve reports the proportion of improved cases under regeneration, targeted editing, and their combination.

Dataset	Iter.	Nano Banana Calls	Ret. Calls
ACDC	2.47	4.52	3.88
BraTS	1.76	3.40	2.26
BUS-UCLM	1.60	3.18	1.78
Drishti-GS	2.40	4.76	3.28
ISIC	1.70	3.24	1.52
Kvasir	1.56	3.12	0.92

Appendix

Table 7: Average inference cost of SegBanana across datasets. We report the average number of inference iterations, Nano Banana calls, and retrieval calls per sample.

Dataset	Stop@1	Stop@2	Stop@3	Generate	Edit	Accept	Retrieved
ACDC	15.0%	23.3%	61.7%	76.6%	14.4%	9.0%	100.0%
BraTS	58.0%	8.0%	34.0%	63.5%	15.3%	21.2%	96.0%
BUS-UCLM	66.0%	8.0%	26.0%	55.8%	17.6%	26.7%	90.0%
Drishti-GS	28.0%	4.0%	68.0%	65.3%	24.8%	9.9%	100.0%
ISIC	58.0%	14.0%	28.0%	56.5%	20.6%	22.9%	62.0%
Kvasir	64.0%	16.0%	20.0%	52.5%	20.3%	27.2%	44.0%

Appendix

Table 8: Adaptive stopping, action selection, and retrieval behavior of SegBanana. Stop@

k

denotes the percentage of samples terminating after iteration

k

. Action columns report the distribution of controller decisions after capping Generate to at most two actions per iteration, and Retrieved denotes the percentage of samples invoking support retrieval at least once.

Figure 16: Additional qualitative comparisons on ACDC, TNBC, BUS-UCLM, and Kvasir. We compare SegBanana with representative medical-specific and generalist segmentation methods across cardiac MRI, histopathology, ultrasound, and endoscopy.

Figure 17: Additional qualitative comparisons on Drishti-GS, RAVIR, ISIC, and BraTS. The examples cover nested anatomical structures, fine vascular structures, skin lesions, and brain tumors, illustrating segmentation behavior across diverse targets and imaging modalities.

Explore similar work

Jul 14, 2026cs.CV

UniMedSeg: Unified In-Context Learning for Multi-Paradigm 2D/3D Medical Image Segmentation

Medical image segmentation foundation models are expected to generalize across diverse clinical scenarios, yet existing universal methods remain fragmented by prompt paradigms and spatial dimensions. Visual in-context learning, interactive segmentation, and language-guided segmentation are typically handled by paradigm-specific models, while 2D and 3D images are also modeled separately. Such isolation prevents heterogeneous annotations and data from being jointly absorbed by a single scalable model and limits cross-paradigm knowledge transfer. To address this bottleneck, we propose UniMedSeg, a Transformer-centric universal segmentation framework that maps visual examples, geometric interactions, language instructions, and 2D/3D images into a shared sequence space, enabling heterogeneous medical supervision to be jointly learned through a unified in-context interface without prompt- or dimension-specific branches. To overcome the long-sequence memory bottleneck caused by visual contexts, we introduce Decoupled Split Attention, which reduces attention complexity to linear while preserving hardware-friendly computation and focused context-target interaction. Extensively trained and evaluated on a large corpus curated from 27 public datasets, UniMedSeg achieves state-of-the-art performance across visual in-context, interactive, and language-guided segmentation without task-specific fine-tuning, demonstrating strong generalization on diverse held-out tasks. The code and model weights are publicly available at https://github.com/Lii1228/UniMedSeg

Yunzhou Li, Jiesi Hu, Yanwu Yang +5

Harbin Institute of Technology at Shenzhen, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China · University Hospital Tübingen, Tübingen, Germany +2

Semi-Supervised Medical Image Segmentation In-Context Learning

Ask PDF

Nov 24, 2025cs.CV

MedSAM3: Delving into Segment Anything with Medical Concepts

Medical image segmentation is fundamental for biomedical discovery. Existing methods lack generalizability and demand extensive, time-consuming manual annotation for new clinical application. Here, we propose MedSAM-3, a text promptable medical segmentation model for medical image and video segmentation. By fine-tuning the Segment Anything Model (SAM) 3 architecture on medical images paired with semantic conceptual labels, our MedSAM-3 enables medical Promptable Concept Segmentation (PCS), allowing precise targeting of anatomical structures via open-vocabulary text descriptions rather than solely geometric prompts. We further introduce the MedSAM-3 Agent, a framework that integrates Multimodal Large Language Models (MLLMs) to perform complex reasoning and iterative refinement in an agent-in-the-loop workflow. Comprehensive experiments across diverse medical imaging modalities, including X-ray, MRI, Ultrasound, CT, and video, demonstrate that our approach significantly outperforms existing specialist and foundation models. We will release our code and model at https://github.com/Joey-S-Liu/MedSAM3.

Anglin Liu, Xu R. Cao, Yifan Shen +4

The Hong Kong University of Science and Technology (Guangzhou) · University of Illinois Urbana-Champaign · Southeast University +1

Segment Anything Model Medical Imaging

Ask PDF

Sep 29, 2026cs.CV

SAM Meets VLM: Parameter-Decoupled Full-Parameter Training for Unified Medical Reasoning and Segmentation

Medical multimodal large language models (MLLMs) are increasingly expected not only to answer clinical questions, but also to localize the visual evidence behind their predictions. A common strategy connects a vision--language model (VLM) with SAM-style segmentation through a special <SEG> token, yet full-parameter training of this unified architecture is difficult because image-level reasoning and pixel-level segmentation impose different requirements on the shared representation space. To address this issue, we propose a parameter-decoupled training framework for unified medical reasoning and segmentation. The framework treats the <SEG> hidden state as a semantic-to-spatial prompt for the mask decoder and encourages it to become separable from generic language states, reducing ambiguous segmentation prompts and potential disruption to reasoning representations. It first performs medical shallow alignment to adapt visual features to clinical language without disturbing the LLM; then controlled instruction tuning shapes separable <SEG> prompt states, monitored by the Davies--Bouldin Index (DBI), while scaling segmentation gradients entering the language backbone; finally, the SAM branch is specialized with the VLM frozen to improve mask precision without altering reasoning parameters. Experiments on medical referring segmentation, grounding, visual QA, and textual QA benchmarks show that our framework achieves strong language-conditioned segmentation while preserving competitive reasoning ability. Ablations show that two-phase instruction tuning, gradient scaling, and segmentation specialization all contribute to the model.

Xuyang Cao, Enyou Liu, Jun Zhao +3

JDH Algo, JD Health International Inc.

Medical Vision-Language Models Multimodal Large Language Models Segment Anything Model

Ask PDF