PixFoundation: Are We Heading in the Right Direction with Pixel-level Vision Foundation Models?

Mennatullah Siam

cs.CVFeb 6, 2025

PixFoundation: Are We Heading in the Right Direction with Pixel-level Vision Foundation Models?

Authors: Mennatullah Siam

Abstract

Multiple works have emerged to push the boundaries of multi-modal large language models (MLLMs) towards pixel-level understanding. The current trend is to train MLLMs with pixel-level grounding supervision in terms of masks on large-scale labelled data and specialized decoders for the segmentation task. However, we show that such MLLMs when evaluated on recent challenging vision-centric benchmarks, exhibit a weak ability in visual question answering (VQA). Surprisingly, some of these methods even downgrade the grounding ability of MLLMs that were never trained with such pixel-level supervision. In this work, we propose two novel challenging benchmarks with paired evaluation for both VQA and grounding. We demonstrate that simple baselines that are not unified achieve performance that matches or surpasses some of the pixel-level MLLMs. Our paired benchmarks and evaluation enable additional analysis on whether the failure stems from VQA and/or grounding abilities. Furthermore, we propose a paired prompt sensitivity analysis on both the language and visual prompts tailored for the grounding task. Our benchmarking pushes pixel-level MLLMs to improve in both accuracy and robustness, without sacrificing language abilities. More importantly, we study the research question of ``When does grounding emerge in MLLMs with respect to the output tokens?'' We propose a post-hoc interpretability tool that can be plugged into any MLLM to study the aforementioned question. We show that grounding does not necessarily coincide with the exact referring expression in the output, but can coincide with the object parts, its location, appearance, context or state. Code and datasets are publicly available at https://msiam.github.io/PixFoundationSeries/.

Figures & tables

Figure 1: Our focus : (i) the grounding & VQA accuracy and robustness of pixel-level MLLMs (prompt variations include spelling errors) (left) , (ii) when does grounding emerge in standard MLLMs with respect to the output tokens? (right) . The latter shows the noun phrases and their corresponding predicted segmentation, highlighted in red. These are extracted from LLaVA 1.5 attention maps with three masks due to the point prompt ambiguity from the maximum attention, highlighted as a black circle.

Panel of Figure 2 — Figure 2: Examples of ground-truth annotations for referring expressions in the respective object of interest in the question and their segmentation masks. First row: PixMMVP examples, Second row: PixCV-Bench examples. Ground-truth highlighted in green.

Figure 3: Failures of pixel-level MLLMs. (a) The first failure is the degraded performance in visual question answering in some of these models. (b) The second which relates to the first, is the degraded performance in instruction following, where the question is instructing the model to generate one letter from the options but fails to do so. (c) The third is the degraded performance in pixel-level visual grounding in some of these models. The predicted masks corresponding to the [SEG] token/s are highlighted in red.

Panel of Figure 4 — Figure 4: Prompt sensitivity in visual grounding. Two examples showing the prompt sensitivity in relation to grounding, emphasizing the importance of language in visual grounding. The examples are generated using Qwen2.5-VL. Prompt is (a) “Locate the dog’s face and output all the coordinates in JSON format.”, (b)“Locate the dog’s face, output its bbox coordinates using JSON format.”, (c) “Identify the front of the school bus [x1, y1, x2, y2] in JSON format.”, (d)“Locate the front of the school bus then output this box coordinates using JSON format.”.

#	Prompt Template
1	Ques:QUES\n $\|$ CHOICES \nInstruct: $\|$ INSTRUCT\nAns:
2	QUES \n $\|$ CHOICES \n $\|$ INSTRUCT \nAnswer:
3	QUES $\|$ CHOICES $\|$ INSTRUCT
4	Question-QUES $\|$ CHOICES. $\|$ INSTRUCT. Answer-
5	QUES \t $\|$ CHOICES \t $\|$ INSTRUCT Answer::
6	Question, QUES $\|$ CHOICES. $\|$ INSTRUCT. Answer,

Table 1: Language prompt sensitivity. Left: templates used in the VQA. QUES, CHOICES, INSTRUCT correspond to the question, choices and the instruction to generate an option’s letter, resp. Right: templates used in the visual grounding. According to the model’s output, the prompt either uses (segment, mask, masks), or (detect, box, boxes) in the prompt. EXPR corresponds to the referring expression.

Panel of Figure 5 — Figure 5: Visual prompt sensitivity. It shows the best two MLLMs in visual grounding sensitive to perturbations in the visual prompt using: Left: guided masking and Right: guided cropping, to motivate this study. The original performance of these MLLMs is presented as an upper bound (dashed line). It also shows example prompts with these variations.

Panel of Figure 6 — Figure 6: Our Interpretability mechanism showing concept categories where the grounding emerges in PixMMVP using LLaVA 1.5 (7B). Top: referring expression, output response, noun phrases and concepts corresponding to the grounding using the oracle selection. Bottom: the four images with predicted segmentation mask, highlighted in red, using the oracle selection and max. attention, highlighted as a black circle. It shows the referring segmentation emerging in different noun phrases than the queried expression.

Method	Venue	Grounding Output	PixMMVP					PixMMVP Prompt Sensitivity
			$\mathcal{A}\dagger$	$\mathcal{A}$	$\mathcal{M}\dagger$	$\mathcal{M}$	$\mathcal{S}$	$\mathcal{A}$	$\mathcal{A}$ w/ GPT	$\mathcal{M}$	$\mathcal{S}$
Pixel-level MLLMs w/ Mask Output
OMG LLaVA (7B) $\diamond$	NeurIPS 2024	Mask	12.0	12.0	17.8 (13.8)	38.0 (38.8)	18.2 (18.3)	5.9	11.8	29.7	9.8
GLAMM (7B)	CVPR 2024	Mask	1.3	2.7	31.5 (30.6)	47.4 (48.3)	5.1 (5.1)	-	-	41.9	-
LISA (7B)	CVPR 2024	Mask	7.3	-	18.1 (14.1)	42.9 (42.9)	12.5 (12.5)	-	-	35.0	-
LLaVA-G (7B)	ECCV 2024	Mask	9.3	-	19.2 (15.2)	28.8 (29.1)	14.0 (14.1)	-	-	26.8	-

Table 2: PixMMVP evaluation. VQA accuracy using the first and third probing (i.e.,

\mathcal{A}\dagger

\mathcal{A}

resp.). Visual grounding w/ mask output using the first two probing (i.e.,

\mathcal{M}\dagger

\mathcal{M}

resp.).

\ast,\diamond,\star

: models using Llama 3 (8B), InternLM2 (7B), and Qwen2.5 (7B), resp., unlike the rest that are relying on Vicuna (7B and 13B) for the base LLM. - : indicates either the model can not be evaluated in that setting, or has low results below 1%. (.): is the mIoU while discarding all the images that were labelled with no objects of interest.

\mathcal{S}

: denotes the score of the MLLM, which is the harmonic mean of VQA and grounding accuracies. The oracle results are highlighted in red , the best and second best are bolded and underlined , resp.

Panel of Figure 7 — Figure 7: Ablations on PixMMVP benchmark. (a) Point accuracy ( $\mathcal{PA}$ ) ablation of (L-7B, L-13B, C-8B) corresponding to LLaVA 1.5 (7B & 13B) and Cambrian (8B), resp., PixF: PixFoundation. It shows consistently that our PixFoundation automatic and oracle variants surpass the attend and segment method, confirming that visual grounding occurs in noun phrases that are not necessarily the closest to the referring expression. (b) Language prompt sensitivity in the visual grounding task of RGA and Qwen2.5-VL, showing two modes (w/o Expr: variations occur to the prompt without the referring expressions, Expr: variations occur only to the referring expressions), using intersection over union ( $\mathcal{M}$ ). It shows how the state-of-the-art model Qwen2.5-VL is severely impacted by variations in the language prompt, impacting its visual grounding ability. (c) Random baseline vs. maximum attention point, second maximum and third maximum ablation using intersection over union ( $\mathcal{M}$ ). It shows the stability of our results regardless of using the first, second or third maximum and confirms its superiority with respect to a random baseline variant. All variants are evaluated within the oracle selection.

Method	Grounding Output	PixCV-Bench
		$\mathcal{A}\dagger$	$\mathcal{A}$	$\mathcal{M}$	$\mathcal{S}$
Pixel-level MLLMs w/ Mask Output
OMG LLaVA (7B) $\diamond$	Mask	12.0	42.1	50.5	45.9
GLAMM (7B)	Mask	-	-	51.9	-
LISA (7B)	Mask	3.7	-	48.1	6.7
LLaVA-G (7B)	Mask	14.1	4.4	17.6	15.8

Table 3: PixCV-Bench evaluation. VQA accuracy using the first and third probing (i.e.,

\mathcal{A}\dagger

\mathcal{A}

resp.). Visual grounding w/ mask output using mIoU in the second probing (i.e.,

\mathcal{M}

Panel of Figure 8 — Figure 8: PixMMVP qualitative comparison in visual grounding following the second probing. The referred expression is on top. It shows that mining for grounding within the attention maps of standard MLLMs w/ oracle mask selection is better than MLLMs trained with mask supervision, without degrading their VQA abilities. Thus, questioning the current training recipes and design choices of pixel-level MLLMs to fully utilize the potential in their base MLLMs.

Panel of Figure 9 — Figure 9: PixMMVP Frequency of failures in both visual grounding and VQA vs. VQA failures only vs. grounding only. For visual grounding, IoU $<0.5$ is considered a failure.

Panel of Figure 10 — Figure 10: Analysis of when grounding emerges on the PixMMVP benchmark using the three base MLLMs, LLaVA 1.5 (7, 13B) and Cambrian (8B), that were not trained with pixel-level grounding supervision. We follow the second probing, then report the oracle selection. Analysis on: (a) the output location and (b) the output concept category, which coincides with the best segmentation.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Model Name	Model Checkpoint
LISA (7B)	xinlai/LISA-7B-v1-explanatory
GLAMM (7B)	MBZUAI/GLaMM-FullScope
RGA (7B)	SurplusDeficit/UniGR-7B
LLaVA-G (7B)	Haozhangcx/llava_grounding_gd_vp
LLaVA 1.5 (7B)	liuhaotian/llava-v1.5-7b
LLaVA 1.5 (13B)	liuhaotian/llava-v1.5-13b

Appendix

Table S1 : Additional details about implementation and output. Left: Hugging Face model checkpoints used. Right: The average output length across PixMMVP dataset for the three base MLLMs using the first and second probing techniques.

Method	PixMMVP
	$\mathcal{A}\dagger$	$\mathcal{A}$	$\mathcal{M}\dagger$	$\mathcal{M}$	$\mathcal{S}$
OMG LLaVA (7B) $\diamond$	12.0	12.0	13.8	38.8	18.3
LLaVA 1.5 (7B) + (a+s)	27.3	28.0	7.8	14.1	18.8
LLaVA 1.5 (13B) + (a+s)	39.3	30	8.5	13.9	20.5
Cambrian (8B) $\ast$ + (a+s)	52.0	52.0	15.5	13.3	23.9
Cambrian (8B) $\ast$ + PixFoundation (Ours)	52.0	52.0	44.7	41.4	48.1

Appendix

Table S2 : Open-source automatic mask selection. Comparison of pixel-level MLLMs to our automatic baseline that relies on Cambrian (8B), an open-source model, for the automatic selection (PixFoundation

\star

) on PixMMVP. Instead of using GPT-5.1 which is closed source (PixFoundation). Best results are bolded.

Method	refCOCO			refCOCO+			refCOCOg
	val	testA	testB	val	testA	testB	val	test
LISA	74.1	76.5	71.1	62.4	67.4	56.5	66.4	68.5
LISA (ft)	74.9	79.1	72.3	65.1	70.8	58.1	67.9	70.6
OMG LLaVA (7B) $\diamond$	75.6	77.7	71.2	65.6	69.7	58.9	70.7	70.2
OMG LLaVA (7B) $\diamond$ (ft)	78.0	80.3	74.1	69.1	73.1	63.0	72.9	72.9
Cambrian (8B) $\ast$ + PixFoundation $\dagger$	77.8	81.1	72.8	73.2	79.5	70.1	75.5	77.5

Appendix

Table S3 : RefCOCO benchmark comparison of pixel-level MLLMs to one of our provided baselines and our upper bound using cIoU. Best results are bolded.

Panel of Figure 17 — Figure S1 : Analysis on when grounding emerges on the PixCV-Bench benchmark using the three base MLLMs, LLaVA 1.5 (7, 13B) and Cambrian (8B), that were not trained with pixel-level grounding supervision. We follow the second probing and then report the oracle selection. Analysis on: (a) the output location and (b) the output concept category, which coincides with the best segmentation.

	Method	Training	Inference	Seperate Tasks Prompts		Joint Tasks Prompt
		w/ Mask Supervision	w/ Seg. Decoder	$\mathcal{A}\dagger,\mathcal{A}$	$\mathcal{PA}\uparrow,\mathcal{D}_{\text{norm}}\downarrow$	$\mathcal{A}\dagger,\mathcal{A}$	$\mathcal{PA}\uparrow,\mathcal{D}_{\text{norm}}\downarrow$
Set A	RGA- Full	✓	✓	5.3, 51.3	72.1 , 0.08	1.3, -	56.3, 0.17
	RGA - Part	✓	✗	5.3, 51.3	72.1 , 0.18	1.3, -	57.5 , 0.21
	Qwen2.5-VL	✗	✗	54.7 , 38.0	57.9, 0.22	32.7 , -	42.5, 0.23
Set B	LISA - Full	✓	✓	7.3, -	57.1, 0.12	-, -	50.6, 0.17
	LISA - Part	✓	✗	7.3, -	51.4, 0.24	-, -	48.6, 0.26

Appendix

Table S4 : Step-wise ablation of pixel-level MLLMs with vs. without mask supervision during the MLLM training, and with vs. without the segmentation decoder used during inference. We report both the VQA and visual grounding metrics for point accuracy and normalized distance. We evaluate under our proposed three probing techniques for the separate tasks prompts. We also evaluate with a fourth probing technique that prompts the model to generate both the question’s answer and grounding, under the joint tasks prompt. Best results are bolded.

Method	$\mathcal{S}$	$\mathcal{S}_{\text{sensitivity}}$	Run-time (s)
			Component-wise	Total
PixFoundation - Point & Noun Phrase Ext.	33.6	25.0	1.0	2.9
PixFoundation - SAM			0.5
PixFoundation (GPT-5.1)- Filtration			0.2
PixFoundation (GPT-5.1) - Mask Selection			1.2
(a+s) - Point & Noun Phrase Ext.	18.8	17.5	1.0	1.52

Appendix

Table S5 : Runtime ablation for the components used in the simple baseline, (a+s) and our PixFoundation vs. LISA, one of the unified pixel-level MLLMs. The accuracy and prompt variations robustness scores,

S,S_{\text{sensitivity}}

, are reported on PixMMVP in standard and prompt sensitivity settings. It shows our non-unified simple baseline surpassing all other models in terms of scores and run-time, except for our PixFoundation scores, motivating our benchmarking effort. It also shows our PixFoundation, which is mainly meant as an interpretability mechanism, as the best in terms of the scores, yet entails a reasonable computational overhead. Best and second best results are bolded and underlined, resp.

Panel of Figure 20 — Figure S2 : Relative TSNE to show the out-of-distribution nature of PixMMVP and PixCV-Bench (ADE20K) part with respect to RefCOCO variants, where the latter dataset is used in standard training & evaluation of MLLMs equipped with visual grounding.

Panel of Figure 21 — Figure S3 : Qualitative ablation for the baseline and our interpretability tool using the base MLLM, Cambrian (8B), ablating the different schemes for mask selection. We use the second probing to prompt the MLLM to identify the referred expression. The referring expressions for these examples are as follows: (a) the key “z”, (b) the key “z”, (c) people, (d) the elderly person. Predictions are highlighted in red.

Panel of Figure 22 — Figure S4 : Normalized attention maps visualization showing the noun phrase and its corresponding attention in the output text for two PixMMVP examples using Cambrian (8B) base MLLM. While the attention maps can not be directly used as segmentation, yet it provides initial locations for the maximally attended pixels corresponding to what the model is looking at. In certain scenarios, it exactly aligns with the noun phrase describing it, as in the two examples. Yet in certain scenarios, as we showed earlier, the grounding of the referred expression in question emerges with other noun phrases describing it.

Panel of Figure 23 — Figure S5 : First example of when grounding emerges , corresponding to Image 1 in Fig. 6 main. Each column corresponds to a noun phrase and shows three potential SAM predicted masks highlighted in red using the maximum attention point of this noun phrase as a point prompt, highlighted as a black circle. It shows the output from mining the attention maps for pixel-level grounding using LLaVA 1.5 (7B) base MLLM. Noun phrases: (a) the image, (b) a butterfly, (c) orange wings.

Panel of Figure 24 — Figure S6 : Second example of when grounding emerges, corresponding to Image 3 in Fig. 6 main. Each column corresponds to a noun phrase and shows three potential SAM predicted masks highlighted in red using the maximum attention point of this noun phrase as a point prompt, highlighted as a black circle. It shows the output from mining the attention maps for pixel-level grounding using LLaVA 1.5 (7B) base MLLM. Noun phrases: (a) The flame, (b) the match, (c) the top, (d) the image, (e) darkness

Panel of Figure 25 — Figure S7 : Third example of when grounding emerges, corresponding to Image 6 in Fig. 6 main. Each column corresponds to a noun phrase and shows three potential SAM predicted masks highlighted in red using the maximum attention point of this noun phrase as a point prompt, highlighted as a black circle. It shows the output from mining the attention maps for pixel-level grounding using LLaVA 1.5 (7B) base MLLM. Noun phrases:(a) the dog’s face, (b) the scene, (c) a black and white dog, (d) a black nose.

Panel of Figure 26 — Figure S8 : Fourth example of when grounding emerges, corresponding to Image 161 in Fig. 6 main. Each column corresponds to a noun phrase and shows three potential SAM predicted masks highlighted in red using the maximum attention point of this noun phrase as a point prompt, highlighted as a black circle. It shows the output from mining the attention maps for pixel-level grounding using LLaVA 1.5 (7B) base MLLM. Noun phrases: (a) The minute hand, (b) the clock, (c) the scene, (d) the 12 o’clock position.

Panel of Figure 27 — Figure S9 : Additional examples of when grounding emerges with LLaVA-1.5 (13B), first example . Each column corresponds to a noun phrase and shows three potential SAM predicted masks highlighted in red using the maximum attention point of this noun phrase as a point prompt. It shows the output from mining the attention maps for pixel-level grounding using LLaVA 1.5 (13B) base MLLM. Noun phrases: (a) The elderly person, (b) the scene, (c) a man.

Panel of Figure 28 — Figure S10 : Additional examples of when grounding emerges with LLaVA-1.5 (13B), second example . Each column corresponds to a noun phrase and shows three potential SAM predicted masks highlighted in red using the maximum attention point of this noun phrase as a point prompt. It shows the output from mining the attention maps for pixel-level grounding using LLaVA 1.5 (13B) base MLLM. Noun phrases: (a) The elderly person, (b) the scene, (c) a man.

Panel of Figure 29 — Figure S11 : PixMMVP qualitative comparison between the pixel-level visual grounding following the second probing. The referred expression used in the segmentation is shown on top of each row. It shows persistently that mining for the grounding within attention maps of MLLMs that were not trained with pixel-level grounding supervision and using the oracle selection outperforms the pixel-level MLLMs. It clearly shows the oracle excels in identifying fine-grained object parts and descriptions that other pixel-level MLLMs are not necessarily capable of. The second best performance is GLAMM, yet we showed it is completely incapable of performing visual question answering.

Panel of Figure 30 — Figure S12 : PixCV-Bench qualitative comparison between the pixel-level visual grounding following the second probing. The referred expression used in the segmentation is shown on top of each row. It shows that, similar to PixMMVP, mining for the grounding within MLLMs that were not trained with pixel-level grounding supervision paired with the oracle selection outperforms pixel-level MLLMs.

Panel of Figure 31 — Figure S13 : PixMMVP fine-grained analysis of the studied models’ performance across the different visual patterns, showing the model’s accuracy with each pattern.

Panel of Figure 32 — Figure S14 : PixCV-Bench fine-grained analysis of the studied models’ performance across the different visual patterns in ADE20K and COCO, showing the model’s accuracy with each pattern.

Panel of Figure 33 — Figure S15 : Failures of the oracle upper bound, PixFoundation $\dagger$ , using Cambrian (8B) as base MLLM on PixMMVP. It shows that failures mostly emerge in quantity or counting tasks. It also shows that the upper bound inherits SAM failures and the ambiguity arising in the referred expression itself, e.g., “the wall behind the bed”, which direction does “behind” indicate?

Large Vision-Language Models

Large vision-language models (LVLMs) connect image and video perception with language generation, reasoning, and interactive tasks.

See all 404 papers

VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence
May 20, 2026Mozhgan Nasr Azadani, Yimu Wang, Yongpeng Zhu +5VLM Evaluation Visual Question Answering
University of Waterloo · Stanford University · NVIDIA
PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding?
Sep 2, 2025Mennatullah SiamVision-Language Grounding Video-Language Model Evaluation
Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning
Jul 7, 2026Yake Wei, Yuan Wang, Fengyun Rao +2Vision-Language Grounding VLM Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China · Laboratory of Research on Large Models and Intelligent Governance · Tsinghua University +1

May 20, 2026J/K move · Enter open · S save

VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence

Mozhgan Nasr Azadani, Yimu Wang, Yongpeng Zhu, Lihong Chen+4

University of Waterloo · Stanford University · NVIDIA

VLM Evaluation Visual Question Answering Vision-Language Grounding Visually Grounded Reasoning Visual Grounding Multimodal Understanding

Open paper AskPDF arXiv

Establishing a clear link between model predictions and the visual evidence that supports them is critical for transparency and reliability in multimodal reasoning, yet current multimodal large language model (MLLM) evaluations do not explicitly enforce this alignment. Existing benchmarks assess either textual answer correctness or pixel-level localization in isolation, leaving the coupling of reasoning and grounding an open challenge. We introduce VISTAQA, a comprehensive benchmark for joint evaluation of free-form answer correctness and pixel-level evidence grounding in visual question answering. VISTAQA comprises 1,157 expert-curated samples spanning six task types and six visual domains, ranging from direct perception to compositional and relational reasoning. VISTAQA requires models to not only answer correctly, but to also provide precise segmentation masks that support their answers. It also includes hallucination-aware examples where no valid visual evidence exists. To support this enhanced evaluation, we introduce GROVE, a unified evaluation metric that enforces joint correctness by combining textual accuracy and grounding quality via a per-sample geometric mean, ensuring neither dimension can compensate for deficiencies in the other. Comprehensive experiments across grounding-aware models and hybrid pipelines with general-purpose MLLMs reveal that even the strongest systems achieve limited performance under GROVE, highlighting a substantial gap between answer accuracy and visual evidence alignment.

#	Prompt Template
1	Ques:QUES\n $\|$ CHOICES \nInstruct: $\|$ INSTRUCT\nAns:
2	QUES \n $\|$ CHOICES \n $\|$ INSTRUCT \nAnswer:
3	QUES $\|$ CHOICES $\|$ INSTRUCT
4	Question-QUES $\|$ CHOICES. $\|$ INSTRUCT. Answer-
5	QUES \t $\|$ CHOICES \t $\|$ INSTRUCT Answer::
6	Question, QUES $\|$ CHOICES. $\|$ INSTRUCT. Answer,

Abstract

Large Vision-Language Models

Explore similar work

VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence

PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding?

Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning