ROSE: Retrieval-Oriented Segmentation Enhancement

Song Tang; Guangquan Jie; Henghui Ding; Yu-Gang Jiang

cs.CVApr 15, 2026

ROSE: Retrieval-Oriented Segmentation Enhancement

Authors: Song Tang, Guangquan Jie, Henghui Ding, Yu-Gang Jiang

Organizations: Fudan University, China · Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China

Abstract

Existing segmentation models based on multimodal large language models (MLLMs), such as LISA, often struggle with novel or emerging entities due to their inability to incorporate up-to-date knowledge. To address this challenge, we introduce the Novel Emerging Segmentation Task (NEST), which focuses on segmenting (i) novel entities that MLLMs fail to recognize due to their absence from training data, and (ii) emerging entities that exist within the model's knowledge but demand up-to-date external information for accurate recognition. To support the study of NEST, we construct a NEST benchmark using an automated pipeline that generates news-related data samples for comprehensive evaluation. Additionally, we propose ROSE: Retrieval-Oriented Segmentation Enhancement, a plug-and-play framework designed to augment any MLLM-based segmentation model. ROSE comprises four key components. First, an Internet Retrieval-Augmented Generation module is introduced to employ user-provided multimodal inputs to retrieve real-time web information. Then, a Textual Prompt Enhancer enriches the model with up-to-date information and rich background knowledge, improving the model's perception ability for emerging entities. Furthermore, a Visual Prompt Enhancer is proposed to compensate for MLLMs' lack of exposure to novel entities by leveraging internet-sourced images. To maintain efficiency, a WebSense module is introduced to intelligently decide when to invoke retrieval mechanisms based on user input. Experimental results demonstrate that ROSE significantly boosts performance on the NEST benchmark, outperforming a strong Gemini-2.0 Flash-based retrieval baseline by 19.2 in gIoU.

Figures & tables

Figure 2: NEST Data Engine. We introduce an automated annotation pipeline that efficiently generates high-quality evaluation samples for the novel emerging segmentation task. The pipeline leverages time-specific queries to continuously collect news content and corresponding relevant images for constructing VQA pairs and automatically generating mask annotations, enabling a comprehensive and reliable evaluation of models’ abilities to segment emerging entities.

Figure 3: Architecture overview of ROSE. Given a user input (image and question), ROSE first employs the WebSense module to determine whether internet retrieval is needed. If so, the Internet Retrieval-Augmented Generation module retrieves relevant textual and visual data from the web. The retrieved content is then processed by the Textual and Visual Prompt Enhancer to generate enriched prompts for the MLLM-based segmentation model, which ultimately produces accurate segmentation masks for novel and emerging entities.

Method	RAG	Acc.	Novel Entity		Emerging Entity		Overall
			gIoU	cIoU	gIoU	cIoU	gIoU	cIoU
CRIS [ 32 ]	✗	-	36.8	27.3	40.5	30.3	38.9	29.1
GRES [ 24 ]	✗	-	37.8	36.6	44.2	36.1	41.4	36.3
Grounded-SAM [ 30 ]	✗	-	39.0	31.7	53.8	40.3	47.4	36.7
SEEM [ 37 ]	✗	-	41.8	27.4	47.1	38.2	44.8	33.7
LISA-7B [ 19 ]	✗	-	38.4	28.5	56.5	47.5	48.7	39.3

Table 1: Novel Emerging Segmentation results on NEST dataset. Furthermore, we partition NEST into a novel entity split and an emerging entity split based on LLaVA-v1.5-7B’s [ 26 ] knowledge, which is the foundation model employed by LISA-7B [ 19 ] , SESAME-7B [ 33 ] , and READ-7B [ 28 ] .

Method	RAG	NEST		ReasonSeg		RefSeg		Overall
		gIoU	cIoU	gIoU	cIoU	gIoU	cIoU	gIoU	cIoU
CRIS [ 32 ]	✗	39.6	29.2	18.2	19.7	56.0	50.4	40.5	26.6
GRES [ 24 ]	✗	42.7	35.6	20.3	17.7	60.8	56.2	43.8	30.0
Grounded-SAM [ 30 ]	✗	47.8	34.4	20.6	15.6	42.4	31.4	43.8	27.7
SEEM [ 37 ]	✗	47.3	36.6	33.2	25.4	21.6	17.3	41.7	29.7
LISA-7B [ 19 ]	✗	51.1	39.0	42.5	44.2	54.9	53.7	50.9	40.6

Table 2: Mixed Dataset NEST+ Results. ReasonSeg is sourced from [ 19 ] , while RefSeg is derived from RefCOCO, RefCOCO+ [ 17 ] and RefCOCOg [ 35 ] datasets.

Figure 4: Qualitative results comparing LISA [ 19 ] , READ [ 28 ] , and ROSE on novel and emerging entities. ROSE accurately segments unseen and newly emerging targets, while LISA struggles due to a lack of up-to-date knowledge or inability to recognize new entities.

Method	Novel Entity		Emerging Entity		Overall
	gIoU	cIoU	gIoU	cIoU	gIoU	cIoU
LISA-7B [ 19 ]	38.4	28.5	56.5	47.5	48.7	39.3
+ROSE (IRAG only)	40.4	30.9	67.1	62.7	55.7	49.1
+ROSE (IRAG+TPE)	41.3	31.6	73.3	67.2	59.6	51.8
+ROSE (IRAG+VPE)	64.9	61.7	71.7	65.5	68.7	63.8
+ROSE (Full)	68.6	67.2	79.4	72.3	74.7	70.1

Table 3: Ablation Study. Impact of different ROSE components on novel emerging segmentation, conducted on the NEST dataset. The data split follows the same approach as in Tab. 1 .

Semantic Segmentation

Semantic segmentation assigns a class label to every image pixel, enabling detailed scene understanding across domains and sensors.

See all 414 papers

LLMSeg: Training-Free Text-Prompted Image Segmentation with B-Splines
Oct 21, 2026Abdullah HamdiImage Segmentation Text-Guided Image Segmentation
King Abdullah University of Science and Technology (KAUST)
Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token
Mar 19, 2026Anqi Zhang, Xiaokang Ji, Guangyu Gao +3Image Segmentation Semantic Segmentation
Beijing Institute of Technology · University of Birmingham · Beijing Jiaotong University +1
UniEvo-RS: Omni-Prompt Unified Remote Sensing Segmentation with Representative Exemplar-Driven Prototype Evolution
Aug 4, 2026Kunquan Zhang, Peilang Li, Xikun Hu +4Remote Sensing Image Segmentation Test-Time Adaptation
Sun Yat-sen University · National University of Defense Technology

Oct 21, 2026J/K move · Enter open · S save

LLMSeg: Training-Free Text-Prompted Image Segmentation with B-Splines

Abdullah Hamdi

King Abdullah University of Science and Technology (KAUST)

Image Segmentation Text-Guided Image Segmentation Referring Image Segmentation

Open paper AskPDF arXiv

Promptable segmentation models such as SAM 3 achieve remarkable open-vocabulary performance but require dedicated architectures, large-scale annotation engines, and task-specific training. We ask a different question: can a general-purpose multimodal large language model (MLLM) segment images directly, without an added segmentation head, mask decoder, or training? We present LLMSeg, a training-free pipeline that turns a vision-capable LLM into a text-prompted image segmenter. LLMSeg decomposes segmentation into three LLM-driven stages. First, detect localizes all instances matching a free-form text query and describes their coarse morphology. Next, segment traces each instance using closed periodic B-splines confined to its detected box. These splines define an outer boundary and optional hole loops. Finally, refine iteratively proposes focus boxes around residual errors and retraces confined splines. The process ends when the mask stabilizes or the pass budget is exhausted. The output is a binary mask with inspectable intermediate geometry. We evaluate LLMSeg across ten referring and reasoning segmentation splits (ReasonSeg and RefCOCO/+/g). We also evaluate open-vocabulary instance segmentation (LVIS, COCO, and SA-Co), semantic segmentation, and zero-shot detection (ODinW13 and RF-100VL). With an off-the-shelf Gemini-3.7-Flash backend, LLMSeg is competitive with and often surpasses the specialist SAM 3 model on identical items. It reaches 71.5 cIoU on ReasonSeg val without additional segmentation-specific training. Backend model updates can improve accuracy without retraining the pipeline. The code is available at https://abdullahamdi.com/llmseg .

Abstract

Semantic Segmentation

Explore similar work

LLMSeg: Training-Free Text-Prompted Image Segmentation with B-Splines

Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token

UniEvo-RS: Omni-Prompt Unified Remote Sensing Segmentation with Representative Exemplar-Driven Prototype Evolution