Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token

Anqi Zhang; Xiaokang Ji; Guangyu Gao; Jianbo Jiao; Chi Harold Liu; Yunchao Wei

cs.CVMar 19, 2026

Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token

Authors: Anqi Zhang, Xiaokang Ji, Guangyu Gao, Jianbo Jiao, Chi Harold Liu, Yunchao Wei

Organizations: Beijing Institute of Technology · University of Birmingham · Beijing Jiaotong University · Beijing Academy of Artificial Intelligence

Abstract

Recent segmentation methods leveraging Multi-modal Large Language Models (MLLMs) have shown reliable object-level segmentation and enhanced spatial perception. However, almost all previous methods predominantly rely on specialist mask decoders to interpret masks from generated segmentation-related embeddings and visual features, or incorporate multiple additional tokens to assist. This paper aims to investigate whether and how we can unlock segmentation from MLLM itSELF with 1 segmentation Embedding (SELF1E) while achieving competitive results, which eliminates the need for external decoders. To this end, our approach targets the fundamental limitation of resolution reduction in pixel-shuffled image features from MLLMs. First, we retain image features at their original uncompressed resolution, and refill them with residual features extracted from MLLM-processed compressed features, thereby improving feature precision. Subsequently, we integrate pixel-unshuffle operations on image features with and without LLM processing, respectively, to unleash the details of compressed features and amplify the residual features under uncompressed resolution, which further enhances the resolution of refilled features. Moreover, we redesign the attention mask with dual perception pathways, i.e., image-to-image and image-to-segmentation, enabling rich feature interaction between pixels and the segmentation token. Comprehensive experiments across multiple segmentation tasks validate that SELF1E achieves performance competitive with specialist mask decoder-based methods, demonstrating the feasibility of decoder-free segmentation in MLLMs. Project page: https://github.com/ANDYZAQ/SELF1E.

Figures & tables

Panel of Figure 1 — Figure 1: Comparison of different MLLM-based segmentation paradigms. Almost all previous methods follow (a) and (b), which rely on the specialist mask decoder. Limited approaches directly predict mask from the MLLM, yet they still require multiple [SEG] tokens for guidance. Our approach in (d) takes advantage of higher resolution pre-compressed features and integrates the accumulated residual features, enabling MLLM-based segmentation without additional specialist decoders and [SEG] tokens.

Figure 2: The additional branch of pre-compressed image features self-replication for uncompressed features. The compressed features for LLM follow the original process.

Figure 3: An overview of RFR and RFA operations. The residual features are amplified from the restored compressed features with and without LLM processing. The fusion of restored uncompressed features and the amplified residual features simultaneously achieves higher resolution and fine-grained representations.

Method	w/o SMD	1-Token	RefCOCO			RefCOCO+			RefCOCOg
Method	w/o SMD	1-Token	val	testA	testB	val	testA	testB	val	test
LISA-7B(ft) [ 31 ]	$\times$	$\checkmark$	74.9	79.1	72.3	65.1	70.8	58.1	67.9	70.6
PixelLM-7B [ 49 ]	$\times$	$\checkmark$	73.0	76.5	68.2	66.3	71.7	58.3	69.3	70.5
GSVA-7B [ 66 ]	$\times$	$\checkmark$	76.4	77.4	72.8	64.5	67.7	58.6	71.1	72.0
GSVA-7B(ft) [ 66 ]	$\times$	$\checkmark$	77.2	78.9	73.5	65.9	69.6	59.8	72.7	73.3
u-LLaVA [ 68 ]	$\times$	$\checkmark$	83.0	85.1	80.5	77.1	81.7	70.6	77.1	78.0

Table 1: Comparison of cIoU on the Referring Expression Segmentation benchmarks (RefCOCO/+/g). SMD represents Specialist Mask Decoder, and 1-Token represents using a single special token for segmentation. “ft” denotes the model is finetuned on the specific dataset. Results in bold are the best, while underlined are the second best.

Method	ZS	val		testA		testB
Method	ZS	cIoU	gIoU	cIoU	gIoU	cIoU	gIoU
LISA-7B [ 31 ]	$\times$	38.7	32.2	52.6	48.5	44.8	39.7
LISA-7B(ft) [ 31 ]	$\times$	61.8	61.6	68.5	66.3	60.6	58.8
GSVA-7B [ 66 ]	$\times$	61.7	63.3	69.2	70.1	60.3	61.3
GSVA-7B(ft) [ 66 ]	$\times$	63.3	66.5	69.9	71.1	60.5	62.2
SAM4MLLM-8B [ 8 ]	$\times$	67.8	71.9	72.2	74.2	63.4	65.3

Table 2: Comparison on Generalized Referring Expression Segmentation. ZS denotes whether the method is zero-shot.

Method	val		test
Method	gIoU	cIoU	gIoU	cIoU
LISA-7B [ 31 ]	44.4	46.0	36.8	34.1
LISA-7B(ft) [ 31 ]	52.9	54.0	47.3	48.4
LaSagnA [ 63 ]	48.8	47.2	-	-
VISA-7B [ 70 ]	52.7	57.8	-	-
SAM4MLLM-8B [ 8 ]	58.4	60.4	-	-

Table 3: Comparison of our approach and other state-of-the-art methods on Reasoning Segmentation.

	A150	PC59	PC459	PAS20
PSALM [ 79 ]	18.2	48.5	10.2	81.3
HiMTok-8B [ 62 ]	25.0	43.9	-	82.0
HyperSeg [ 64 ]	22.3	64.6	-	92.1
SELF1E-2B	50.2	64.6	42.4	89.9
SELF1E-2B*	24.0	36.9	18.9	44.8

Table 4: Comparison with SOTA methods on open-vocabulary semantic segmentation benchmarks. We use mIoU as the evaluation metric for semantic segmentation. The datasets are abbreviated as: ADE20k-150 (A150), Pascal Context-59 (PC59), Pascal Context-459 (PC459), Pascal VOC-20 (PAS20). “*” denotes that the model is trained without any data from the corresponding datasets.

Figure 4: Visualization results on RefCOCO demonstrate the effectiveness of the modules. ‘HR’ stands for Higher Resolution of uncompressed image features, ‘Residual’ refers to the use of residual features from the LLM, and ‘PUS’ represents MLP-with-Pixel-Unshuffle. The bottom row illustrates the resolution of the fused image features and the predicted mask before interpolation to the original image size.

$\bm{F}_{V_{1}}^{HQ}$	RFR	PUS	val	testA	testB
			76.6	78.3	73.6
✓			78.7	81.1	75.2
✓	✓		79.0	81.3	76.3
		✓	78.5	80.3	75.1
✓	✓	✓	79.8	82.0	77.1

Table 5: Ablation of RFR and Pixel-Unshuffle on RefCOCO.

	RefCOCO	RefCOCO+	RefCOCOg
1. Baseline	78.9	73.5	76.3
2. 1+Part Unshuffle	79.3	73.9	76.7
3. 1+RFA(MLP $\times$ 1)	79.4	74.3	76.7
4. 1+RFA(MLP $\times$ 2)	80.0	74.3	77.4
5. 1+RFA(MLP $\times$ 3)	79.7	74.2	76.6

Table 6: Ablation study of RFA on Referring Expression Segmentation tasks. The results are the average of cIoU among the subsets.

	RefCOCO	RefCOCO+	RefCOCOg
Causal	78.4	72.7	75.5
+ [IMG] $\rightarrow$ [IMG]	79.0	73.5	75.8
+ [IMG] $\rightarrow$ [SEG]	80.0	74.3	77.4
+ [IMG] $\rightarrow$ [TEXT]	80.2	74.9	77.0
Bidirectional	80.2	75.1	76.9

Table 7: Ablation study on the attention masks for segmentation. The causal mask and bidirectional mask denote two widely used attention masks. The “

\rightarrow

” represents the additional visible regions, e.g. , [IMG]

\rightarrow

[SEG] means the [SEG] tokens are visible to the previous [IMG] tokens.

Task	Dataset	Samples	SEG-rates	SEG-samples
VQA	VQAv2 [ 20 ]	100k	1 $\times$	100k
	OKVQA [ 43 ]	9k		9k
	TextVQA [ 53 ]	35k		35k
	VizWiz [ 22 ]	20k		20k
	GQA [ 26 ]	100k		100k
	LLaVA-150k [ 39 ]	157k		157k

Table 8: Details of the multiple datasets for training. SEG-rates represent the magnification of the dataset samples for the training of SELF1E-SEG version.

Methods	VQAv2	OKVQA	VizWiz	GQA	TextVQA	POPE	MMB-en	MMB-cn	MME
InternVL3-2B [ 83 ]	80.1	56.1	56.6	60.7	77.0	89.6	81.1	78.4	2221.2
InternVL3-8B [ 83 ]	81.8	61.9	63.4	63.2	78.9	91.1	83.4	82.2	2415.4
SELF1E-2B	77.7	48.5	67.8	61.5	71.7	89.4	69.6	66.2	2014.6
SELF1E-8B	80.2	54.0	69.9	64.1	72.3	91.1	74.6	72.0	2265.1

Table 9: Comparison of the VQA performance of our approach with their original base MLLMs.

Method	MLLM	RefCOCO			RefCOCO+			RefCOCOg
Method	MLLM	val	testA	testB	val	testA	testB	val	test
SELF1E-2B	InternVL3-2B	80.2	82.1	77.6	74.6	79.1	69.2	77.0	77.8
	InternVL2-2B	77.7	80.6	74.6	71.5	76.7	66.9	74.3	74.7
	InternVL2.5-2B	80.1	82.2	78.0	74.7	78.7	69.8	76.5	77.6
SELF1E-SEG-2B	InternVL3-2B	84.3	85.4	82.3	78.9	83.5	75.1	80.4	81.0
	InternVL2-2B	83.5	85.5	81.4	77.7	82.0	72.9	79.5	80.1

Table 10: Comparison of results with different MLLMs as base model on the Referring Expression Segmentation benchmarks (RefCOCO/+/g).

	RefCOCO	RefCOCO+	RefCOCOg
Baseline	76.2	72.3	74.5
Scanning	78.3	73.3	76.3
Self-Replication	78.9	73.5	76.3

Table 11: Ablation study on the operations for retaining the resolution.

Method	Inference (ms)	FPS	Memory (GB)
LISA-7B	250.0	4.00	19.2
UFO-7B	961.5	1.04	14.4
SELF1E-8B	105.0	9.52	17.7

Table 12: Efficiency comparison among LISA, UFO, and SELF1E.

	Method	Exist.	Count	Pos.	Color	Post.	Celeb.	Scene	Landm.	Artw.	OCR
Perp.	SELF1E	190.00	140.00	153.33	175.00	164.29	159.41	160.25	156.50	140.50	102.50
Perp.	InternVL3-2B	195.00	165.00	136.67	170.00	157.48	155.00	156.25	163.75	156.75	155.00
	Method	Comm.		Num.		Trans.		Code		Total Score
Cog.	SELF1E	97.86		100.00		162.50		112.50		2014.64
Cog.	InternVL3-2B	115.71		105.00		185.00		147.50		2164.11

Table 13: MME Benchmark Performance Comparison

Figure 5: Representative VQA results comparison.

Figure 6: Visualization results on ReasonSeg demonstrate outstanding reasoning ability of SELF1E. “Pred” denotes the predictions from SELF1E, “GT” denotes the ground-truth masks.

Figure 7: Visualization results on open-vocabulary segmentation. “OG” refers to the original image without overlays.

Figure 8: Visualization results show the attention maps of [SEG] to [IMG] tokens under different attention-mask designs. “ [IMG] $\rightarrow$ [IMG] ” indicates that all image tokens use a bidirectional attention mask, while all other tokens follow a causal mask. “ [IMG] $\rightarrow$ [SEG] ” means that, in addition to the bidirectional mask among image tokens, all image tokens are also allowed to interact with the [SEG] token.

Image Segmentation

Image segmentation assigns pixels or voxels to objects and regions, supporting measurement, scene understanding, and decisions across image domains.

See all 390 papers

From Drop-off to Recovery: A Mechanistic Analysis of Segmentation in MLLMs
Mar 18, 2026Boyong Wu, Sanghwan Kim, Zeynep AkataMultimodal Large Language Models Semantic Segmentation
Technical University of Munich · Helmholtz Munich · Munich Center for Machine Learning (MCML)
Better, Stronger, Faster, and Broader: Structured All-Mask Prediction for MLLM-Based Segmentation
Aug 3, 2026Jiazhen Liu, Mingkuan Feng, Long ChenInstance Segmentation Image Segmentation
The Hong Kong University of Science and Technology (HKUST), Hong Kong.
Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning
Jul 7, 2026Yake Wei, Yuan Wang, Fengyun Rao +2Vision-Language Grounding VLM Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China · Laboratory of Research on Large Models and Intelligent Governance · Tsinghua University +1

Mar 18, 2026J/K move · Enter open · S save

From Drop-off to Recovery: A Mechanistic Analysis of Segmentation in MLLMs

Boyong Wu, Sanghwan Kim, Zeynep Akata

Technical University of Munich · Helmholtz Munich · Munich Center for Machine Learning (MCML)

Multimodal Large Language Models Semantic Segmentation VLM Interpretability Mechanistic Interpretability

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) are increasingly applied to pixel-level vision tasks, yet their intrinsic capacity for spatial understanding remains poorly understood. We investigate segmentation capacity through a layerwise linear probing evaluation across the entire MLLM pipeline: vision encoder, adapter, and LLM. We further conduct an intervention based attention knockout analysis to test whether cross-token attention progressively refines visual representations, and an evaluation of bidirectional attention among image tokens on spatial consistency. Our analysis reveals that the adapter introduces a segmentation representation drop-off, but LLM layers progressively recover through attention-mediated refinement, where correctly classified tokens steer misclassified neighbors toward the correct label. At early image token positions, this recovery is bounded by causal attention, which bidirectional attention among image tokens alleviates. These findings provide a mechanistic account of how MLLMs process visual information for segmentation, informing the design of future segmentation-capable models.

Abstract

Image Segmentation

Explore similar work

From Drop-off to Recovery: A Mechanistic Analysis of Segmentation in MLLMs

Better, Stronger, Faster, and Broader: Structured All-Mask Prediction for MLLM-Based Segmentation

Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning