From Drop-off to Recovery: A Mechanistic Analysis of Segmentation in MLLMs

Boyong Wu; Sanghwan Kim; Zeynep Akata

cs.CVMar 18, 2026

From Drop-off to Recovery: A Mechanistic Analysis of Segmentation in MLLMs

Authors: Boyong Wu, Sanghwan Kim, Zeynep Akata

Organizations: Technical University of Munich · Helmholtz Munich · Munich Center for Machine Learning (MCML)

Abstract

Multimodal Large Language Models (MLLMs) are increasingly applied to pixel-level vision tasks, yet their intrinsic capacity for spatial understanding remains poorly understood. We investigate segmentation capacity through a layerwise linear probing evaluation across the entire MLLM pipeline: vision encoder, adapter, and LLM. We further conduct an intervention based attention knockout analysis to test whether cross-token attention progressively refines visual representations, and an evaluation of bidirectional attention among image tokens on spatial consistency. Our analysis reveals that the adapter introduces a segmentation representation drop-off, but LLM layers progressively recover through attention-mediated refinement, where correctly classified tokens steer misclassified neighbors toward the correct label. At early image token positions, this recovery is bounded by causal attention, which bidirectional attention among image tokens alleviates. These findings provide a mechanistic account of how MLLMs process visual information for segmentation, informing the design of future segmentation-capable models.

Figures & tables

Figure 2: Overview of the three analysis methods. (a) Layerwise linear probing: given an input image, the vision encoder produces patch token embeddings (brown), which are projected by the adapter into the LLM’s embedding space. Inside the LLM, image tokens are processed jointly with text prompt tokens (yellow). At each layer $\ell$ , we extract only the image token representations and train an independent linear probe to predict per-patch semantic classes, reassembled into a 2D segmentation map. (b) Attention knockout: we selectively block attention to incorrectly classified tokens (left) or correctly classified tokens (right) across all LLM layers, testing whether cross-token attention drives self-refinement. (c) Bidirectional attention mask: image tokens attend to each other bidirectionally while all other token pairs retain causal masking, alleviating context starvation at early image positions.

Figure 3: Layerwise linear probing results across the MLLM stack. mIoU on ADE20K for CLIP, DINOv2, and SigLIP encoders paired with Vicuna-7B, measured at the vision encoder output, adapter output, and each LLM layer. All three encoders exhibit a drop at the adapter followed by progressive recovery across LLM layers. Dashed lines mark the best-performing layer; values on the right indicate the total mIoU improvement across LLM layers.

Figure 4: Qualitative segmentation predictions across the MLLM stack. From left to right: input image, ground truth, linear probe prediction at the vision encoder output, adapter output, and at an intermediate LLM layer. Representation drop-off at the adapter but deeper layers appear to resolve class confusions (e.g., wall vs. bed) and produce more spatially coherent predictions.

Figure 5: UMAP projections of patch-level hidden states across the MLLM stack. Each point represents one of 576 image patch tokens, colored by semantic class. At the adapter output, classes are interleaved, by layer 20, same-class patches form distinct clusters, illustrating the progressive emergence of semantic structure through the LLM layers.

Figure 6: Global attention knockout: layerwise segmentation comparison. Top row: unmodified baseline. Middle row: incorrect class blocked (e.g. sky). Bottom row: correct class blocked (e.g. ceiling). Blocking the incorrect class accelerates the resolution of misclassified patches across layers, while blocking the correct class impairs self-correction and leaves more errors in mid-to-late layers.

Figure 7: Context starvation under causal attention. Per-patch pixel accuracy for the first 50 tokens in the LLM layer of the MLLM. Accuracy Gap: Bidirectional minus causal accuracy. Early patches suffer severe context starvation under causal masking.

Figure 8: Causal vs. bidirectional attention: Layerwise probing on ADE20K across the MLLM Stack for different vision encoders. Bidirectional attention yields modest peak mIoU gains for CLIP and DINOv2, while SigLIP shows a larger effect.

Vision Encoder	Attention	GQA	MMB	MME P	MME C	MMMU	POPE	SQA I	TextVQA	VizWiz
CLIP ViT-L/14	Causal	62.6	66.4	1483	284	35.3	86.8	68.7	46.9	56.0
CLIP ViT-L/14	Bidirectional	62.7	65.5	1538	288	36.2	86.9	69.7	47.0	57.5
DINOv2 ViT-L/14	Causal	62.1	57.7	1304	324	34.6	87.2	66.1	14.0	51.4
DINOv2 ViT-L/14	Bidirectional	60.5	55.3	1247	326	32.4	85.1	66.3	13.7	45.8
SigLIP SO400M/14	Causal	61.1	63.7	1414	275	34.7	84.4	70.4	50.2	58.2
SigLIP SO400M/14	Bidirectional	62.1	66.9	1402	298	34.9	84.8	70.7	53.6	53.3

Table 1: VQA benchmark: causal vs. bidirectional LLaVA 1.5 (7B) across three vision encoders. All models use Vicuna-7B as the LLM backbone. Bidirectional attention preserves language understanding with segmentation gains that vary across encoders. DINOv2 shows broader regressions, consistent with its lack of text alignment.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Layerwise linear probing on Cityscapes (left) and PASCAL VOC (right). mIoU across the MLLM stack for CLIP, DINOv2, and SigLIP encoders paired with Vicuna-7B.

Encoder	Attention	Adapter	Peak LLM	$\Delta_{\text{enc}}$	Recovery
CLIP ViT-L/14@336	Causal	33.22	40.74	$+$ 6.36	$+$ 7.52
CLIP ViT-L/14@336	Bidirectional	32.68	41.16	$+$ 6.78	$+$ 8.48
DINOv2 Large	Causal	42.90	44.50	$-$ 1.08	$+$ 1.60
DINOv2 Large	Bidirectional	40.47	44.82	$-$ 0.76	$+$ 4.35
SigLIP SO400M/14	Causal	31.44	35.39	$-$ 3.31	$+$ 3.95
SigLIP SO400M/14	Bidirectional	32.39	41.26	$+$ 2.56	$+$ 8.87

Appendix

Table 2: Causal vs. bidirectional attention: Layerwise probing on ADE20K. Adapter and Peak LLM report mIoU (%).

\Delta_{\text{enc}}

: Peak LLM mIoU minus vision encoder baseline (positive

=

surpasses encoder). Recovery: Peak LLM mIoU minus adapter mIoU.

	Causal	Bidirectional	$\Delta$	% Impr.
Patch 0	0.6195	0.7630	$+$ 0.1435	$+$ 23.2%
Patch 1	0.6851	0.7933	$+$ 0.1082	$+$ 15.8%
Patch 2	0.7069	0.7931	$+$ 0.0862	$+$ 12.2%

Appendix

Table 3: Context starvation under causal attention. Per-patch pixel accuracy for the first three tokens.

\Delta

: Bidirectional minus causal accuracy. % Impr.: relative improvement over causal. Early patches suffer severe context starvation under causal masking.

Figure 10: Layerwise probing across different MLLMs. $\Delta$ mIoU relative to adapter output on ADE20K for LLaVA-1.5 (7B), LLaVA-OneVision (7B), and DeepSeek-VL (7B), each using its default vision encoder (CLIP for LLaVA-1.5, SigLIP for OneVision, SigLIP+SAM-B for DeepSeek-VL). Curves span different numbers of layers due to differences in LLM backbone depth. The adapter drop-off and LLM self-refinement pattern is consistent across architectures.

Figure 11: Aggregate attention knockout across three misclassified classes. Percentage of incorrectly predicted patches remaining across LLM layers, normalized to 100% at layer 0 and averaged over all images exhibiting the misclassification. Blocking the incorrect class (red) accelerates self-correction, blocking the correct class (blue) impairs it and can amplify errors beyond the initial count.

Panel of Figure 12 — Figure 12: Qualitative attention knockout examples. Layerwise segmentation predictions under three conditions (rows: no knockout, knockout incorrect, knockout correct) for three class-confusion pairs. Top: sky vs. windowpane. Middle: ceiling vs. wall. Bottom: grass vs. road. Blocking the incorrect class produces cleaner segmentation maps at earlier layers, while blocking the correct class leaves errors unresolved or amplified.

Figure 13: Extended qualitative segmentation predictions across the MLLM stack. Seven ADE20K validation images showing, from left to right: input image, ground truth, linear probe prediction at the vision encoder output, adapter output, and at an intermediate LLM layer. The representation drop-off at the adapter is visible as increased noise and class confusion compared to the vision encoder output, while deeper LLM layers produce more spatially coherent predictions.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,069 papers

Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token
Mar 19, 2026Anqi Zhang, Xiaokang Ji, Guangyu Gao +3Image Segmentation Semantic Segmentation
Beijing Institute of Technology · University of Birmingham · Beijing Jiaotong University +1
What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models
Feb 28, 2026Yingqi Fan, Junlong Tong, Anhao Zhao +1Multimodal Large Language Models Multimodal Model Interpretability
Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative · Shanghai Jiao Tong University +1
Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning
Jul 7, 2026Yake Wei, Yuan Wang, Fengyun Rao +2Vision-Language Grounding VLM Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China · Laboratory of Research on Large Models and Intelligent Governance · Tsinghua University +1

Mar 19, 2026J/K move · Enter open · S save

Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token

Anqi Zhang, Xiaokang Ji, Guangyu Gao, Jianbo Jiao+2

Beijing Institute of Technology · University of Birmingham · Beijing Jiaotong University+1

Image Segmentation Semantic Segmentation Multimodal Large Language Models

Open paper AskPDF arXiv

Recent segmentation methods leveraging Multi-modal Large Language Models (MLLMs) have shown reliable object-level segmentation and enhanced spatial perception. However, almost all previous methods predominantly rely on specialist mask decoders to interpret masks from generated segmentation-related embeddings and visual features, or incorporate multiple additional tokens to assist. This paper aims to investigate whether and how we can unlock segmentation from MLLM itSELF with 1 segmentation Embedding (SELF1E) while achieving competitive results, which eliminates the need for external decoders. To this end, our approach targets the fundamental limitation of resolution reduction in pixel-shuffled image features from MLLMs. First, we retain image features at their original uncompressed resolution, and refill them with residual features extracted from MLLM-processed compressed features, thereby improving feature precision. Subsequently, we integrate pixel-unshuffle operations on image features with and without LLM processing, respectively, to unleash the details of compressed features and amplify the residual features under uncompressed resolution, which further enhances the resolution of refilled features. Moreover, we redesign the attention mask with dual perception pathways, i.e., image-to-image and image-to-segmentation, enabling rich feature interaction between pixels and the segmentation token. Comprehensive experiments across multiple segmentation tasks validate that SELF1E achieves performance competitive with specialist mask decoder-based methods, demonstrating the feasibility of decoder-free segmentation in MLLMs. Project page: https://github.com/ANDYZAQ/SELF1E.

Abstract

Multimodal Large Language Models

Explore similar work

Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token

What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models

Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning