Multiple works have emerged to push the boundaries of multi-modal large language models (MLLMs) towards pixel-level understanding. The current trend is to train MLLMs with pixel-level grounding supervision in terms of masks on large-scale labelled data and specialized decoders for the segmentation task. However, we show that such MLLMs when evaluated on recent challenging vision-centric benchmarks, exhibit a weak ability in visual question answering (VQA). Surprisingly, some of these methods even downgrade the grounding ability of MLLMs that were never trained with such pixel-level supervision. In this work, we propose two novel challenging benchmarks with paired evaluation for both VQA and grounding. We demonstrate that simple baselines that are not unified achieve performance that matches or surpasses some of the pixel-level MLLMs. Our paired benchmarks and evaluation enable additional analysis on whether the failure stems from VQA and/or grounding abilities. Furthermore, we propose a paired prompt sensitivity analysis on both the language and visual prompts tailored for the grounding task. Our benchmarking pushes pixel-level MLLMs to improve in both accuracy and robustness, without sacrificing language abilities. More importantly, we study the research question of ``When does grounding emerge in MLLMs with respect to the output tokens?'' We propose a post-hoc interpretability tool that can be plugged into any MLLM to study the aforementioned question. We show that grounding does not necessarily coincide with the exact referring expression in the output, but can coincide with the object parts, its location, appearance, context or state. Code and datasets are publicly available at https://msiam.github.io/PixFoundationSeries/.
Figures & tables
Figure 1: Our focus : (i) the grounding & VQA accuracy and robustness of pixel-level MLLMs (prompt variations include spelling errors) (left) , (ii) when does grounding emerge in standard MLLMs with respect to the output tokens? (right) . The latter shows the noun phrases and their corresponding predicted segmentation, highlighted in red. These are extracted from LLaVA 1.5 attention maps with three masks due to the point prompt ambiguity from the maximum attention, highlighted as a black circle.
Figure 2: Examples of ground-truth annotations for referring expressions in the respective object of interest in the question and their segmentation masks. First row: PixMMVP examples, Second row: PixCV-Bench examples. Ground-truth highlighted in green.
Figure 3: Failures of pixel-level MLLMs. (a) The first failure is the degraded performance in visual question answering in some of these models. (b) The second which relates to the first, is the degraded performance in instruction following, where the question is instructing the model to generate one letter from the options but fails to do so. (c) The third is the degraded performance in pixel-level visual grounding in some of these models. The predicted masks corresponding to the [SEG] token/s are highlighted in red.
Figure 4: Prompt sensitivity in visual grounding. Two examples showing the prompt sensitivity in relation to grounding, emphasizing the importance of language in visual grounding. The examples are generated using Qwen2.5-VL. Prompt is (a) “Locate the dog’s face and output all the coordinates in JSON format.”, (b)“Locate the dog’s face, output its bbox coordinates using JSON format.”, (c) “Identify the front of the school bus [x1, y1, x2, y2] in JSON format.”, (d)“Locate the front of the school bus then output this box coordinates using JSON format.”.
Table 1: Language prompt sensitivity. Left: templates used in the VQA. QUES, CHOICES, INSTRUCT correspond to the question, choices and the instruction to generate an option’s letter, resp. Right: templates used in the visual grounding. According to the model’s output, the prompt either uses (segment, mask, masks), or (detect, box, boxes) in the prompt. EXPR corresponds to the referring expression.
Figure 5: Visual prompt sensitivity. It shows the best two MLLMs in visual grounding sensitive to perturbations in the visual prompt using: Left: guided masking and Right: guided cropping, to motivate this study. The original performance of these MLLMs is presented as an upper bound (dashed line). It also shows example prompts with these variations.
Figure 6: Our Interpretability mechanism showing concept categories where the grounding emerges in PixMMVP using LLaVA 1.5 (7B). Top: referring expression, output response, noun phrases and concepts corresponding to the grounding using the oracle selection. Bottom: the four images with predicted segmentation mask, highlighted in red, using the oracle selection and max. attention, highlighted as a black circle. It shows the referring segmentation emerging in different noun phrases than the queried expression.
Method
Venue
Grounding Output
PixMMVP
PixMMVP Prompt Sensitivity
A†
A
M†
M
S
A
A w/ GPT
M
S
Pixel-level MLLMs w/ Mask Output
OMG LLaVA (7B) ⋄
NeurIPS 2024
Mask
12.0
12.0
17.8 (13.8)
38.0 (38.8)
18.2 (18.3)
5.9
11.8
29.7
9.8
GLAMM (7B)
CVPR 2024
Mask
1.3
2.7
31.5 (30.6)
47.4 (48.3)
5.1 (5.1)
-
-
41.9
-
LISA (7B)
CVPR 2024
Mask
7.3
-
18.1 (14.1)
42.9 (42.9)
12.5 (12.5)
-
-
35.0
-
LLaVA-G (7B)
ECCV 2024
Mask
9.3
-
19.2 (15.2)
28.8 (29.1)
14.0 (14.1)
-
-
26.8
-
Table 2: PixMMVP evaluation. VQA accuracy using the first and third probing (i.e., A† & A resp.). Visual grounding w/ mask output using the first two probing (i.e., M† & M resp.). ∗,⋄,⋆ : models using Llama 3 (8B), InternLM2 (7B), and Qwen2.5 (7B), resp., unlike the rest that are relying on Vicuna (7B and 13B) for the base LLM. - : indicates either the model can not be evaluated in that setting, or has low results below 1%. (.): is the mIoU while discarding all the images that were labelled with no objects of interest. S : denotes the score of the MLLM, which is the harmonic mean of VQA and grounding accuracies. The oracle results are highlighted in red , the best and second best are bolded and underlined , resp.
Figure 7: Ablations on PixMMVP benchmark. (a) Point accuracy ( PA ) ablation of (L-7B, L-13B, C-8B) corresponding to LLaVA 1.5 (7B & 13B) and Cambrian (8B), resp., PixF: PixFoundation. It shows consistently that our PixFoundation automatic and oracle variants surpass the attend and segment method, confirming that visual grounding occurs in noun phrases that are not necessarily the closest to the referring expression. (b) Language prompt sensitivity in the visual grounding task of RGA and Qwen2.5-VL, showing two modes (w/o Expr: variations occur to the prompt without the referring expressions, Expr: variations occur only to the referring expressions), using intersection over union ( M ). It shows how the state-of-the-art model Qwen2.5-VL is severely impacted by variations in the language prompt, impacting its visual grounding ability. (c) Random baseline vs. maximum attention point, second maximum and third maximum ablation using intersection over union ( M ). It shows the stability of our results regardless of using the first, second or third maximum and confirms its superiority with respect to a random baseline variant. All variants are evaluated within the oracle selection.
Method
Grounding Output
PixCV-Bench
A†
A
M
S
Pixel-level MLLMs w/ Mask Output
OMG LLaVA (7B) ⋄
Mask
12.0
42.1
50.5
45.9
GLAMM (7B)
Mask
-
-
51.9
-
LISA (7B)
Mask
3.7
-
48.1
6.7
LLaVA-G (7B)
Mask
14.1
4.4
17.6
15.8
Table 3: PixCV-Bench evaluation. VQA accuracy using the first and third probing (i.e., A† & A resp.). Visual grounding w/ mask output using mIoU in the second probing (i.e., M ).
Figure 8: PixMMVP qualitative comparison in visual grounding following the second probing. The referred expression is on top. It shows that mining for grounding within the attention maps of standard MLLMs w/ oracle mask selection is better than MLLMs trained with mask supervision, without degrading their VQA abilities. Thus, questioning the current training recipes and design choices of pixel-level MLLMs to fully utilize the potential in their base MLLMs.
Figure 9: PixMMVP Frequency of failures in both visual grounding and VQA vs. VQA failures only vs. grounding only. For visual grounding, IoU <0.5 is considered a failure.
Figure 10: Analysis of when grounding emerges on the PixMMVP benchmark using the three base MLLMs, LLaVA 1.5 (7, 13B) and Cambrian (8B), that were not trained with pixel-level grounding supervision. We follow the second probing, then report the oracle selection. Analysis on: (a) the output location and (b) the output concept category, which coincides with the best segmentation.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Model Name
Model Checkpoint
LISA (7B)
xinlai/LISA-7B-v1-explanatory
GLAMM (7B)
MBZUAI/GLaMM-FullScope
RGA (7B)
SurplusDeficit/UniGR-7B
LLaVA-G (7B)
Haozhangcx/llava_grounding_gd_vp
LLaVA 1.5 (7B)
liuhaotian/llava-v1.5-7b
LLaVA 1.5 (13B)
liuhaotian/llava-v1.5-13b
Appendix
Table S1 : Additional details about implementation and output. Left: Hugging Face model checkpoints used. Right: The average output length across PixMMVP dataset for the three base MLLMs using the first and second probing techniques.
Method
PixMMVP
A†
A
M†
M
S
OMG LLaVA (7B) ⋄
12.0
12.0
13.8
38.8
18.3
LLaVA 1.5 (7B) + (a+s)
27.3
28.0
7.8
14.1
18.8
LLaVA 1.5 (13B) + (a+s)
39.3
30
8.5
13.9
20.5
Cambrian (8B) ∗ + (a+s)
52.0
52.0
15.5
13.3
23.9
Cambrian (8B) ∗ + PixFoundation (Ours)
52.0
52.0
44.7
41.4
48.1
Appendix
Table S2 : Open-source automatic mask selection. Comparison of pixel-level MLLMs to our automatic baseline that relies on Cambrian (8B), an open-source model, for the automatic selection (PixFoundation ⋆ ) on PixMMVP. Instead of using GPT-5.1 which is closed source (PixFoundation). Best results are bolded.
Method
refCOCO
refCOCO+
refCOCOg
val
testA
testB
val
testA
testB
val
test
LISA
74.1
76.5
71.1
62.4
67.4
56.5
66.4
68.5
LISA (ft)
74.9
79.1
72.3
65.1
70.8
58.1
67.9
70.6
OMG LLaVA (7B) ⋄
75.6
77.7
71.2
65.6
69.7
58.9
70.7
70.2
OMG LLaVA (7B) ⋄ (ft)
78.0
80.3
74.1
69.1
73.1
63.0
72.9
72.9
Cambrian (8B) ∗ + PixFoundation †
77.8
81.1
72.8
73.2
79.5
70.1
75.5
77.5
Appendix
Table S3 : RefCOCO benchmark comparison of pixel-level MLLMs to one of our provided baselines and our upper bound using cIoU. Best results are bolded.
Figure S1 : Analysis on when grounding emerges on the PixCV-Bench benchmark using the three base MLLMs, LLaVA 1.5 (7, 13B) and Cambrian (8B), that were not trained with pixel-level grounding supervision. We follow the second probing and then report the oracle selection. Analysis on: (a) the output location and (b) the output concept category, which coincides with the best segmentation.
Method
Training
Inference
Seperate Tasks Prompts
Joint Tasks Prompt
w/ Mask Supervision
w/ Seg. Decoder
A†,A
PA↑,Dnorm↓
A†,A
PA↑,Dnorm↓
Set A
RGA- Full
✓
✓
5.3, 51.3
72.1 , 0.08
1.3, -
56.3, 0.17
RGA - Part
✓
✗
5.3, 51.3
72.1 , 0.18
1.3, -
57.5 , 0.21
Qwen2.5-VL
✗
✗
54.7 , 38.0
57.9, 0.22
32.7 , -
42.5, 0.23
Set B
LISA - Full
✓
✓
7.3, -
57.1, 0.12
-, -
50.6, 0.17
LISA - Part
✓
✗
7.3, -
51.4, 0.24
-, -
48.6, 0.26
Appendix
Table S4 : Step-wise ablation of pixel-level MLLMs with vs. without mask supervision during the MLLM training, and with vs. without the segmentation decoder used during inference. We report both the VQA and visual grounding metrics for point accuracy and normalized distance. We evaluate under our proposed three probing techniques for the separate tasks prompts. We also evaluate with a fourth probing technique that prompts the model to generate both the question’s answer and grounding, under the joint tasks prompt. Best results are bolded.
Method
S
Ssensitivity
Run-time (s)
Component-wise
Total
PixFoundation - Point & Noun Phrase Ext.
33.6
25.0
1.0
2.9
PixFoundation - SAM
0.5
PixFoundation (GPT-5.1)- Filtration
0.2
PixFoundation (GPT-5.1) - Mask Selection
1.2
(a+s) - Point & Noun Phrase Ext.
18.8
17.5
1.0
1.52
Appendix
Table S5 : Runtime ablation for the components used in the simple baseline, (a+s) and our PixFoundation vs. LISA, one of the unified pixel-level MLLMs. The accuracy and prompt variations robustness scores, S,Ssensitivity , are reported on PixMMVP in standard and prompt sensitivity settings. It shows our non-unified simple baseline surpassing all other models in terms of scores and run-time, except for our PixFoundation scores, motivating our benchmarking effort. It also shows our PixFoundation, which is mainly meant as an interpretability mechanism, as the best in terms of the scores, yet entails a reasonable computational overhead. Best and second best results are bolded and underlined, resp.
Figure S2 : Relative TSNE to show the out-of-distribution nature of PixMMVP and PixCV-Bench (ADE20K) part with respect to RefCOCO variants, where the latter dataset is used in standard training & evaluation of MLLMs equipped with visual grounding.
Figure S3 : Qualitative ablation for the baseline and our interpretability tool using the base MLLM, Cambrian (8B), ablating the different schemes for mask selection. We use the second probing to prompt the MLLM to identify the referred expression. The referring expressions for these examples are as follows: (a) the key “z”, (b) the key “z”, (c) people, (d) the elderly person. Predictions are highlighted in red.
Figure S4 : Normalized attention maps visualization showing the noun phrase and its corresponding attention in the output text for two PixMMVP examples using Cambrian (8B) base MLLM. While the attention maps can not be directly used as segmentation, yet it provides initial locations for the maximally attended pixels corresponding to what the model is looking at. In certain scenarios, it exactly aligns with the noun phrase describing it, as in the two examples. Yet in certain scenarios, as we showed earlier, the grounding of the referred expression in question emerges with other noun phrases describing it.
Figure S5 : First example of when grounding emerges , corresponding to Image 1 in Fig. 6 main. Each column corresponds to a noun phrase and shows three potential SAM predicted masks highlighted in red using the maximum attention point of this noun phrase as a point prompt, highlighted as a black circle. It shows the output from mining the attention maps for pixel-level grounding using LLaVA 1.5 (7B) base MLLM. Noun phrases: (a) the image, (b) a butterfly, (c) orange wings.
Figure S6 : Second example of when grounding emerges, corresponding to Image 3 in Fig. 6 main. Each column corresponds to a noun phrase and shows three potential SAM predicted masks highlighted in red using the maximum attention point of this noun phrase as a point prompt, highlighted as a black circle. It shows the output from mining the attention maps for pixel-level grounding using LLaVA 1.5 (7B) base MLLM. Noun phrases: (a) The flame, (b) the match, (c) the top, (d) the image, (e) darkness
Figure S7 : Third example of when grounding emerges, corresponding to Image 6 in Fig. 6 main. Each column corresponds to a noun phrase and shows three potential SAM predicted masks highlighted in red using the maximum attention point of this noun phrase as a point prompt, highlighted as a black circle. It shows the output from mining the attention maps for pixel-level grounding using LLaVA 1.5 (7B) base MLLM. Noun phrases:(a) the dog’s face, (b) the scene, (c) a black and white dog, (d) a black nose.
Figure S8 : Fourth example of when grounding emerges, corresponding to Image 161 in Fig. 6 main. Each column corresponds to a noun phrase and shows three potential SAM predicted masks highlighted in red using the maximum attention point of this noun phrase as a point prompt, highlighted as a black circle. It shows the output from mining the attention maps for pixel-level grounding using LLaVA 1.5 (7B) base MLLM. Noun phrases: (a) The minute hand, (b) the clock, (c) the scene, (d) the 12 o’clock position.
Figure S9 : Additional examples of when grounding emerges with LLaVA-1.5 (13B), first example . Each column corresponds to a noun phrase and shows three potential SAM predicted masks highlighted in red using the maximum attention point of this noun phrase as a point prompt. It shows the output from mining the attention maps for pixel-level grounding using LLaVA 1.5 (13B) base MLLM. Noun phrases: (a) The elderly person, (b) the scene, (c) a man.
Figure S10 : Additional examples of when grounding emerges with LLaVA-1.5 (13B), second example . Each column corresponds to a noun phrase and shows three potential SAM predicted masks highlighted in red using the maximum attention point of this noun phrase as a point prompt. It shows the output from mining the attention maps for pixel-level grounding using LLaVA 1.5 (13B) base MLLM. Noun phrases: (a) The elderly person, (b) the scene, (c) a man.
Figure S11 : PixMMVP qualitative comparison between the pixel-level visual grounding following the second probing. The referred expression used in the segmentation is shown on top of each row. It shows persistently that mining for the grounding within attention maps of MLLMs that were not trained with pixel-level grounding supervision and using the oracle selection outperforms the pixel-level MLLMs. It clearly shows the oracle excels in identifying fine-grained object parts and descriptions that other pixel-level MLLMs are not necessarily capable of. The second best performance is GLAMM, yet we showed it is completely incapable of performing visual question answering.
Figure S12 : PixCV-Bench qualitative comparison between the pixel-level visual grounding following the second probing. The referred expression used in the segmentation is shown on top of each row. It shows that, similar to PixMMVP, mining for the grounding within MLLMs that were not trained with pixel-level grounding supervision paired with the oracle selection outperforms pixel-level MLLMs.
Figure S13 : PixMMVP fine-grained analysis of the studied models’ performance across the different visual patterns, showing the model’s accuracy with each pattern.
Figure S14 : PixCV-Bench fine-grained analysis of the studied models’ performance across the different visual patterns in ADE20K and COCO, showing the model’s accuracy with each pattern.
Figure S15 : Failures of the oracle upper bound, PixFoundation † , using Cambrian (8B) as base MLLM on PixMMVP. It shows that failures mostly emerge in quantity or counting tasks. It also shows that the upper bound inherits SAM failures and the ambiguity arising in the referred expression itself, e.g., “the wall behind the bed”, which direction does “behind” indicate?
Establishing a clear link between model predictions and the visual evidence that supports them is critical for transparency and reliability in multimodal reasoning, yet current multimodal large language model (MLLM) evaluations do not explicitly enforce this alignment. Existing benchmarks assess either textual answer correctness or pixel-level localization in isolation, leaving the coupling of reasoning and grounding an open challenge. We introduce VISTAQA, a comprehensive benchmark for joint evaluation of free-form answer correctness and pixel-level evidence grounding in visual question answering. VISTAQA comprises 1,157 expert-curated samples spanning six task types and six visual domains, ranging from direct perception to compositional and relational reasoning. VISTAQA requires models to not only answer correctly, but to also provide precise segmentation masks that support their answers. It also includes hallucination-aware examples where no valid visual evidence exists. To support this enhanced evaluation, we introduce GROVE, a unified evaluation metric that enforces joint correctness by combining textual accuracy and grounding quality via a per-sample geometric mean, ensuring neither dimension can compensate for deficiencies in the other. Comprehensive experiments across grounding-aware models and hybrid pipelines with general-purpose MLLMs reveal that even the strongest systems achieve limited performance under GROVE, highlighting a substantial gap between answer accuracy and visual evidence alignment.
Mozhgan Nasr Azadani, Yimu Wang, Yongpeng Zhu +5
University of Waterloo · Stanford University · NVIDIA
Multi-modal large language models (MLLMs) have shown impressive generalization across tasks using images and text modalities. While their extension to video has enabled tasks such as video question answering and video captioning, their pixel-level visual grounding abilities are less studied. In this work, we raise the pertinent question of whether motion is used in pixel-level visual grounding and whether video MLLMs can segment objects based on natural language expressions describing their motion patterns. We identify the shortcomings in the current benchmarks, where we show that a single frame can often suffice for capturing the referring expression without any temporal reasoning. To address this, we introduce novel motion-centric probing, particularly designed for the visual grounding task, to study video MLLMs' ability to identify true motion from a fake one and their ability to grasp the ordering of motion. Consequently, we introduce MoCentric-Bench, a motion-centric benchmark designed to evaluate video MLLMs based on their ability to capture the interaction between motion and language, rather than relying primarily on static cues. We further establish strong single-image baselines that either match or surpass prior methods. Finally, we explore a simple motion-centric adaptation that provides state-of-the-art performance on our MoCentric-Bench. Code is available at https://msiam.github.io/PixFoundationSeries/.
Recent advancements in Multimodal Large Language Models (MLLMs) have evolved from static perception to interleaved visual-language reasoning, often referred to as ``thinking with images''. A basic operation in this reasoning process is to zoom in on regions of interest (often represented with bounding boxes) to acquire finer visual details. In this paper, we propose Segmentation before Answering (SegAnswer), which shifts the unit of zoom-in from the popular bounding box to pixel-level segmentation mask. By employing fine-grained masks to isolate the target area from cluttered environments, segmented visual input yields a more precise region of interest, effectively filtering out redundant background and interfering objects. Furthermore, the discrete patches of segmented visual input align more seamlessly with how MLLMs structure visual tokens via positional embeddings. In experiments, we evaluate SegAnswer across diverse benchmarks, including high-resolution perception, general perception, and hallucination. It achieves consistent improvements and also exhibits considerable performance on segmentation tasks, validating its capability for reliable pixel grounding.
Yake Wei, Yuan Wang, Fengyun Rao +2
Gaoling School of Artificial Intelligence, Renmin University of China · Laboratory of Research on Large Models and Intelligent Governance · Tsinghua University +1