Moondream Segmentation: From Words to Masks

Ethan Reid

cs.CVApr 3, 2026

Moondream Segmentation: From Words to Masks

Authors: Ethan Reid

Organizations: M87 Labs, San Francisco, CA, USA.

Abstract

We present Moondream Segmentation, a referring image segmentation extension of Moondream 3, a vision-language model. Given an image and a referring expression, the model autoregressively decodes a vector path and iteratively refines the rasterized mask into a final detailed mask. We introduce a reinforcement learning stage that resolves ambiguity in the supervised signal by directly optimizing mask quality. Rollouts from this stage produce coarse-to-ground-truth targets for the refiner. To mitigate evaluation noise from polygon annotations, we release RefCOCO-M, a cleaned RefCOCO validation split with boundary-accurate masks. Moondream Segmentation achieves a cIoU of 80.2% on RefCOCO (val) and 62.6% mIoU on LVIS (val).

Figures & tables

Figure 1: Example masks produced by Moondream Segmentation. Prompts are shown in white boxes.

Figure 2: High-level overview of Moondream Segmentation. The VLM decodes a vector path from the image and prompt, which is rasterized into a coarse mask. An iterative refiner conditioned on frozen vision features produces the final mask.

Figure 3: Training data generation pipeline. Web images are labeled by an ensemble of VLMs with text annotations and bounding boxes, verified by Moondream, filtered for consistency and accuracy, and passed to a segmentation model to propose masks. Surviving image–text–box–mask tuples are added to the final dataset; rejected samples are discarded.

	RefCOCO			RefCOCO+			RefCOCOg		RefCOCO-M
Model	val	testA	testB	val	testA	testB	val	test	val	[email protected]
LISA (LLaVA 7B)	74.9	79.1	72.3	65.1	70.8	58.1	67.9	70.6	72.7	68.1
Text4Seg (InternVL2-8B)	74.7	77.4	71.6	68.5	63.6	62.9	70.7	71.6	70.4	68.9
SimpleSeg (Qwen2.5-VL)	80.9	77.8	75.2	72.4	77.3	66.1	73.3	74.1	85.2	68.9
Gemini 2.5 Flash	63.7	64.8	65.6	52.8	57.3	50.8	56.2	57.4	70.4	67.5
SAM 3 Agent (Gemini 2.5 Pro)	75.5	77.6	71.0	67.3	71.1	63.4	73.4	74.0	86.7	81.5

Table 1: RIS benchmark comparison (%; higher is better). RefCOCO/+/g and RefCOCO-M report cIoU; the final RefCOCO-M column reports [email protected].

Figure 4: Original RefCOCO polygon masks (top) and RefCOCO-M refined masks (bottom). RefCOCO-M tightens boundaries and recovers fine structure that is often missing from the original annotations.

Model	LVIS
Gemini 2.5 Flash	45.5
SAM 3 Agent (Gemini 2.5 Pro)	59.3
SAM 3	62.6
Moondream Seg.	62.6

Table 2: LVIS (val) instance segmentation comparison (mIoU, %; higher is better).

Figure 5: Boundary-focused qualitative comparison (prompt: car ). Moondream masks are typically sharper at edges and better preserve fine structure than SAM 3.

Figure 6: Qualitative comparison against Gemini 2.5 Flash (prompt: car ). Gemini masks can contain small isolated false positives (salt-and-pepper noise) and boundary noise, while Moondream produces cleaner masks with sharper edges.

Figure 7: mIoU on a held-out set of RefCOCO-M rollout-derived coarse masks conditioned on the ground-truth bounding box, as a function of refinement steps.

Refiner	RefCOCO-M
SAM 2	90.8
HQ-SAM 2	91.7
Moondream Seg.	93.8

Table 3: Refiner comparison. mIoU (%) on a held-out set of RefCOCO-M rollout-derived coarse masks conditioned on the ground-truth bounding box (higher is better).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Examples of referring expressions removed from RefCOCO-M by the safety pipeline.

Figure 9: Additional qualitative samples produced by Moondream Segmentation. Prompts are shown in white boxes.

Figure 10: RefCOCO validation samples produced by Moondream Segmentation. Prompts are shown in white boxes.

Figure 11: LVIS validation samples produced by Moondream Segmentation. Prompts are shown in white boxes.

Referring Image Segmentation

Referring image segmentation maps a natural-language description of an object or region to its pixels in an image.

See all 43 papers

LlamaSeg: Image Segmentation via Autoregressive Mask Generation
May 26, 2025Jiru Deng, Tengjin Weng, Tianyu Yang +3Image Segmentation Text-Guided Image Segmentation
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Tsinghua University · Shenzhen University +2
AMLRIS: Alignment-aware Masked Learning for Referring Image Segmentation
Feb 26, 2026Tongfei Chen, Shuo Yang, Yuguang Yang +7Referring Image Segmentation Vision-Language Alignment
School of Artificial Intelligence, Beihang University, China · School of Electronic Information Engineering, Beihang University, China · AI Research, Qihoo 360, China +3
VespaSeg: A Resource-Aware Ground-then-Segment Pipeline for Referring Expression Segmentation
Aug 2, 2026Savindu Dilshan WickramasinghePromptable Image Segmentation Efficient VLM Inference
University of Moratuwa · Department of Electronic and Telecommunication Engineering, University of Moratuwa Moratuwa, Sri Lanka

May 26, 2025J/K move · Enter open · S save

LlamaSeg: Image Segmentation via Autoregressive Mask Generation

Jiru Deng, Tengjin Weng, Tianyu Yang, Wenhan Luo+2

Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Tsinghua University · Shenzhen University+2

Image Segmentation Text-Guided Image Segmentation Promptable Image Segmentation Open-Vocabulary Semantic Segmentation

Open paper AskPDF arXiv

We present LlamaSeg, a visual autoregressive framework that unifies multiple image segmentation tasks via natural language instructions. By reformulating segmentation as visual generation, LlamaSeg encodes masks as visual tokens and uses a LLaMA-style Transformer for direct next-token prediction, naturally fitting segmentation into autoregressive architectures. To support large-scale training, we introduce a data annotation pipeline and construct the SA-OVRS dataset, which contains 2M segmentation masks annotated with over 5,800 open vocabulary labels or diverse textual descriptions, spanning diverse real-world scenarios. This enables our model to localize objects in images based on text prompts and to generate fine-grained masks. We further introduce the composite metric average Hausdorff Distance (

d_{\mathrm{AHD}}

) to evaluate mask contour fidelity for generative models better. Experiments show that LlamaSeg consistently outperforms existing generative approaches on multiple segmentation benchmarks and delivers finer, more accurate segmentation masks. Code and dataset are available at https://github.com/GML-FMGroup/llamaseg.

Abstract

Referring Image Segmentation

Explore similar work

LlamaSeg: Image Segmentation via Autoregressive Mask Generation

AMLRIS: Alignment-aware Masked Learning for Referring Image Segmentation

VespaSeg: A Resource-Aware Ground-then-Segment Pipeline for Referring Expression Segmentation