PROVE: Probabilistic Visual Grounding via Patch-Level Evidence for Vision-Language Models

Mayank Nautiyal; Tianru Zhang; Li Ju; Ekta Vats; Prashant Singh

cs.CVOct 23, 2026

PROVE: Probabilistic Visual Grounding via Patch-Level Evidence for Vision-Language Models

Authors: Mayank Nautiyal, Tianru Zhang, Li Ju, Ekta Vats, Prashant Singh

Organizations: Department of Information Technology, Uppsala University, Sweden · Science for Life Laboratory, Uppsala University, Sweden

Abstract

Visual grounding with vision-language models such as CLIP typically produces a single deterministic relevance map for an image-query pair, without indicating whether the localisation is reliable or whether the referent is present at all. We introduce PROVE (Probabilistic Visual Grounding via Patch-Level Evidence), a post-hoc evidential adapter that maps frozen CLIP patch features to per-patch Beta distributions without modifying the backbone. PROVE casts grounding as per-patch membership: each patch accumulates positive and negative evidence for whether it belongs to the referent, so a query may match zero, one, or many patches. We train PROVE with an evidential objective on soft patch-coverage targets, and define a prior-pull objective that keeps absent-referent examples near an uninformative prior, so that vacuity serves as an absence signal without a dedicated presence classifier. During inference, a single forward pass yields, in closed form, a posterior-mean grounding map, a spatial vacuity map, and an image-level absence score. We evaluate PROVE on referring-expression benchmarks: it is competitive in localisation with state-of-the-art CLIP grounding methods, its uncertainty tracks localisation error and supports selective prediction under abstention, and its image-level score detects when the referent is absent altogether. Our evidential patch-membership modelling offers a simple, drop-in mechanism for probabilistic verification in frozen vision-language grounding systems.

Figures & tables

Figure 1: Overview of PROVE. Frozen CLIP encoders ( $\Psi_{I}$ , $\Psi_{T}$ ) give patch and text features and a saliency prior (Grad-ECLIP); a trained evidential head maps them to per-patch Betas, giving a mean grounding map and a vacuity map.

Figure 2: Beta evidence and predictive uncertainty. (a) The balance of

\alpha_{i}

and

\beta_{i}

shifts relevance toward foreground or background;

\alpha_{i}\!\approx\!\beta_{i}\!\gg\!1

concentrates around intermediate relevance, and

\alpha_{i}=\beta_{i}=1

is the uniform low-evidence prior. (b) For fixed mean

\mu_{i}

, higher concentration

s_{i}=\alpha_{i}+\beta_{i}

reduces the predictive variance.

	gRefCOCO (people)			gRefCOCO (objects)			Flickr30k Entities			PhraseCut
Method	PG $\uparrow$	EBPG $\uparrow$	IoU@ $k\uparrow$	PG $\uparrow$	EBPG $\uparrow$	IoU@ $k\uparrow$	PG $\uparrow$	EBPG $\uparrow$	IoU@ $k\uparrow$	PG $\uparrow$	EBPG $\uparrow$	IoU@ $k\uparrow$
Grad-CAM [ 24 ]	0.202	0.225	0.179	0.181	0.229	0.209	0.400	0.358	0.320	0.295	0.215	0.234
Grad-ECLIP [ 33 ]	0.552	0.372	0.283	0.522	0.358	0.302	0.820	0.567	0.440	0.586	0.350	0.350
CLIP-Surgery [ 14 ]	0.495	0.263	0.299	0.433	0.265	0.310	0.822	0.400	0.457	0.630	0.245	0.403
Rollout [ 1 ]	0.124	0.200	0.146	0.081	0.166	0.098	0.238	0.303	0.239	0.116	0.151	0.101
PROVE (ours)	$\mathbf{0.855}_{\pm.001}$	$\mathbf{0.459}_{\pm.004}$	$\mathbf{0.616}_{\pm.002}$	$\mathbf{0.759}_{\pm.005}$	$\mathbf{0.412}_{\pm.004}$	$\mathbf{0.538}_{\pm.003}$	$0.764_{\pm.005}$	$\underline{0.502}_{\pm.004}$	$\mathbf{0.527}_{\pm.003}$	$0.512_{\pm.014}$	$\underline{0.274}_{\pm.003}$	$\underline{0.354}_{\pm.011}$

Table 1: Grounding localisation. Best in bold , second best underlined . PROVE: mean

{}_{\pm\text{std}}

over 5 seeds; baselines deterministic.

Figure 3: Qualitative comparison on single-, multi-, and no-target queries. PROVE’s mean map localises the referent(s), while vacuity flags the absent case; the baselines peak even when the referent is absent. Maps are min–max normalised.

Split	Method	AUROC	N-acc	T-acc	Bal
testA	Grad-ECLIP	0.682	0.570	0.711	0.640
	Det. control	0.823	0.643	0.864	0.754
	PROVE	$\mathbf{0.849}_{\pm.010}$	$\mathbf{0.657}_{\pm.029}$	$\mathbf{0.928}_{\pm.017}$	$\mathbf{0.792}_{\pm.006}$
testB	Grad-ECLIP	0.746	0.518	0.853	0.686
	Det. control	0.789	0.671	0.749	0.710
	PROVE	$\mathbf{0.813}_{\pm.006}$	$0.669_{\pm.026}$	$\mathbf{0.869}_{\pm.021}$	$\mathbf{0.769}_{\pm.004}$

Table 2: Absent-referent detection on gRefCOCO testA (people) and testB (objects). Best per split in bold ; PROVE is mean

{}_{\pm\text{std}}

over 5 seeds, baselines deterministic.

Panel of Figure 4 — Figure 4: Uncertainty as a ranking signal. (a) IoU@ $k$ per vacuity bin (bin 1 most certain). (b) IoU@ $k$ on the retained set as the most uncertain images are rejected. Both span four datasets; mean over 5 seeds, shaded $\pm 1$ std.

	People			Objects			Flickr30k			PhraseCut
Method	PG	EBPG	IoU@ $k$	PG	EBPG	IoU@ $k$	PG	EBPG	IoU@ $k$	PG	EBPG	IoU@ $k$
$\mathrm{PROVE}^{\mathrm{det}}$	0.86	0.63	0.63	0.76	0.55	0.55	0.77	0.68	0.55	0.52	0.41	0.36
PROVE	0.86	0.46	0.62	0.76	0.41	0.54	0.76	0.50	0.53	0.51	0.27	0.35

Table 3: Deterministic vs. probabilistic localisation. PROVE: mean over 5 seeds. Higher value per metric in bold (PG comparable across both).

		Loc.		Absence
Split	Variant	PG	IoU@ $k$	AUROC	$T$ -acc	Bal.
testA	Full	0.85	0.61	0.83	0.95	0.78
	$-\mathcal{L}_{\mathrm{nt}}$	0.86	0.63	0.56	1.00	0.50
	$-\mathcal{L}_{\mathrm{reg}}$	0.84	0.59	0.83	0.85	0.78
testB	Full	0.76	0.54	0.80	0.90	0.76
	$-\mathcal{L}_{\mathrm{nt}}$	0.76	0.54	0.57	1.00	0.50

Table 4: Loss ablation on gRefCOCO testA (people) and testB (objects), single-seed runs. Full is the default PROVE configuration; each lower row removes one auxiliary term.

Split	Backbone	PG $\uparrow$	EBPG $\uparrow$	IoU@ $k$ $\uparrow$
testA	ViT-B/16	$0.855{\scriptstyle\,\pm.001}$	$0.459{\scriptstyle\,\pm.004}$	$0.616{\scriptstyle\,\pm.002}$
testA	ViT-L/14	$\mathbf{0.871}{\scriptstyle\,\pm.002}$	$\mathbf{0.479}{\scriptstyle\,\pm.004}$	$\mathbf{0.632}{\scriptstyle\,\pm.002}$
testB	ViT-B/16	$0.759{\scriptstyle\,\pm.005}$	$0.412{\scriptstyle\,\pm.004}$	$0.538{\scriptstyle\,\pm.003}$
testB	ViT-L/14	$\mathbf{0.773}{\scriptstyle\,\pm.003}$	$\mathbf{0.433}{\scriptstyle\,\pm.003}$	$\mathbf{0.560}{\scriptstyle\,\pm.001}$

Table S1 : Localisation across backbones on gRefCOCO. PROVE: mean

\pm

std over five seeds (four for ViT-L/14).

Panel of Figure 10 — Figure S1 : ViT-L/14 uncertainty behaviour on gRefCOCO. (a) IoU@ $k$ across vacuity bins, where bin 1 is the most certain. (b) IoU@ $k$ on the retained set as a function of the rejected fraction. Mean over five seeds, with shading indicating $\pm 1$ standard deviation.

Split	Backbone	AUROC	$N$ -acc	$T$ -acc	Bal.
testA	ViT-B/16	$0.849{\scriptstyle\,\pm.010}$	$0.657{\scriptstyle\,\pm.029}$	$0.928{\scriptstyle\,\pm.017}$	$0.792{\scriptstyle\,\pm.006}$
testA	ViT-L/14	$\mathbf{0.851}{\scriptstyle\,\pm.008}$	$\mathbf{0.660}{\scriptstyle\,\pm.020}$	$\mathbf{0.936}{\scriptstyle\,\pm.012}$	$\mathbf{0.798}{\scriptstyle\,\pm.004}$
testB	ViT-B/16	$0.813{\scriptstyle\,\pm.006}$	$\mathbf{0.669}{\scriptstyle\,\pm.026}$	$0.869{\scriptstyle\,\pm.021}$	$0.769{\scriptstyle\,\pm.004}$
testB	ViT-L/14	$\mathbf{0.822}{\scriptstyle\,\pm.008}$	$0.667{\scriptstyle\,\pm.016}$	$\mathbf{0.888}{\scriptstyle\,\pm.013}$	$\mathbf{0.778}{\scriptstyle\,\pm.003}$

Table S2 : Absent-referent detection across backbones on gRefCOCO,

A=-\max_{i}s_{i}

. PROVE: mean

\pm

std over five seeds (four for ViT-L/14).

Split	Absence score	AUROC	$N$ -acc	$T$ -acc	Bal.
testA (people)	Grad-ECLIP $-\max_{i}\tilde{s}_{i}$	$0.682$	$0.570$	$0.711$	$0.640$
	Det. control $-\max_{i}\mu_{i}^{\mathrm{det}}$	$0.823$	$0.643$	$0.864$	$0.754$
	PROVE $-\max_{i}\alpha_{i}$	$0.829{\scriptstyle\,\pm 0.002}$	$0.644{\scriptstyle\,\pm 0.003}$	$0.883{\scriptstyle\,\pm 0.006}$	$0.763{\scriptstyle\,\pm 0.002}$
	PROVE $-\mathrm{top}\text{-}k\,s_{i}$	$0.847{\scriptstyle\,\pm 0.010}$	$\mathbf{0.661}{\scriptstyle\,\pm 0.024}$	$0.921{\scriptstyle\,\pm 0.013}$	$0.791{\scriptstyle\,\pm 0.006}$
	PROVE $-\max_{i}s_{i}$	$\mathbf{0.849}{\scriptstyle\,\pm 0.010}$	$0.657{\scriptstyle\,\pm 0.029}$	$\mathbf{0.928}{\scriptstyle\,\pm 0.017}$	$\mathbf{0.792}{\scriptstyle\,\pm 0.006}$
testB (objects)	Grad-ECLIP $-\max_{i}\tilde{s}_{i}$	$0.746$	$0.518$	$0.853$	$0.686$

Table S3 : Absence score comparison on gRefCOCO. PROVE results are mean

\pm

std over five seeds; baselines are single runs. The threshold

\tau

is selected on validation with

T\text{-acc}\!\geq\!0.90

. Best result per split and metric is shown in bold .

Split	Metric	$\gamma=5$	$\gamma=10$	$\bm{\gamma=20}$	$\gamma=40$	$\gamma=100$
testA (people)	PG	$0.858$	$\mathbf{0.859}$	$0.857$	$0.856$	$0.854$
	EBPG	$0.374$	$0.423$	$0.463$	$0.481$	$\mathbf{0.504}$
	IoU@ $k$	$\mathbf{0.619}$	$0.617$	$0.616$	$0.615$	$0.615$
	AUROC	$0.823$	$\mathbf{0.849}$	$0.836$	$0.847$	$0.833$
testB (objects)	PG	$0.759$	$\mathbf{0.761}$	$0.755$	$0.757$	$0.757$
	EBPG	$0.340$	$0.380$	$0.415$	$0.427$	$\mathbf{0.449}$

Table S4 : Effect of evidence capacity

\gamma

on gRefCOCO. Best per row in bold .

Figure S2 : Qualitative results on Flickr30k Entities.

Figure S3 : Qualitative results on PhraseCut.

Figure S4 : Qualitative results on gRefCOCO TestA (People).

Figure S5 : Qualitative results on gRefCOCO TestB (Objects).

Vision-Language Grounding

Vision-Language Grounding links words and phrases to the objects, regions, actions, or moments they describe in images, video, and scenes.

See all 418 papers

Apr 9, 2026

Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models

Despite rapid progress, pretrained vision-language models still struggle when answers depend on tiny visual details or on combining clues spread across multiple regions, as in documents and compositional queries. We address this by framing grounding as test-time evidence retrieval: given a query, the model should actively identify where to look next to resolve ambiguity. To this end, we propose a training-free, model-intrinsic grounding method that uses uncertainty as supervision. Specifically, we compute the entropy of the model's next-token distribution and backpropagate it to the visual token embeddings to obtain an entropy-gradient relevance map, without auxiliary detectors or attention-map heuristics. We then extract and rank multiple coherent regions to support multi-evidence queries, and introduce an iterative zoom-and-reground procedure with a spatial-entropy stopping rule to avoid over-refinement. Experiments on seven benchmarks across four VLM architectures demonstrate consistent improvements over existing methods, with the largest gains on detail-critical and high-resolution settings, while also producing more interpretable evidence localizations.

Marcel Gröpl, Jaewoo Jung, Seungryong Kim +2

ETH Zürich · KAIST AI · ETH AI Center

Vision-Language Grounding VLM Interpretability Gradient-Based Attribution

Ask PDF

Jun 29, 2026

H-GRPO: Permutation-Invariant Reinforcement Learning for Improving Spatial Grounded Reasoning

Vision-language models (VLMs) have achieved strong performance on visual reasoning tasks, yet a correct final answer does not necessarily imply that the model followed a correct or visually grounded reasoning process. We study how VLMs can be trained to produce explicit intermediate reasoning steps that are both semantically meaningful and grounded in visual evidence. We represent each reasoning step as a structured triplet consisting of a sub-question, sub-answer, and supporting bounding box, enabling the reasoning process to be evaluated along both semantic and spatial dimensions. To optimize this representation, we introduce Hungarian-GRPO (H-GRPO), a reinforcement-learning framework that aligns predicted and reference reasoning steps using permutation-invariant Hungarian matching, avoiding the assumption that valid reasoning steps must follow a fixed ordering. Experiments show that optimizing grounded reasoning improves both intermediate reasoning quality and answer accuracy, with an H-GRPO variant achieving the strongest overall grounding and decomposition performance. Our analyses show that grounding quality and answer correctness are related but distinct, while counterfactual evidence masking confirms that the localized regions contain information important to the model's predictions. These findings motivate evaluating VLMs beyond answer accuracy and demonstrate the value of explicit, spatially grounded reasoning.

Eric Peh, Debaditya Roy, Chen Li +1

Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur, India +1

RL for VLMs Visual Reasoning VLM Interpretability Reinforcement Learning Vision-Language Grounding VLM Reasoning+2

Ask PDF

May 11, 2026

Grounded or Guessing? LVLM Confidence Estimation via Blind-Image Contrastive Ranking

Large vision-language models suffer from visual ungroundedness: they can produce a fluent, confident, and even correct response driven entirely by language priors, with the image contributing nothing to the prediction. Existing confidence estimation methods cannot detect this, as they observe model behavior under normal inference with no mechanism to determine whether a prediction was shaped by the image or by text alone. We introduce BICR (Blind-Image Contrastive Ranking), a model-agnostic confidence estimation framework that makes this contrast explicit during training by extracting hidden states from a frozen LVLM twice: once with the real image-question pair, and once with the image blacked out while the question is held fixed. A lightweight probe is trained on the real-image hidden state and regularized by a ranking loss that penalizes higher confidence on the blacked-out view, teaching it to treat visual grounding as a signal of reliability at zero additional inference cost. Evaluated across five modern LVLMs and seven baselines on a benchmark covering visual question answering, object hallucination detection, medical imaging, and financial document understanding, BICR achieves the best cross-LVLM average on both calibration and discrimination simultaneously, with statistically significant discrimination gains robust to cluster-aware analysis at 4-18x fewer parameters than the strongest probing baseline.

Reza Khanmohammadi, Erfan Miahi, Simerjot Kaur +4

Michigan State University · Independent AI Researcher · JPMorgan AI Research +1

VLM Evaluation Confidence Estimation for VLMs Vision-Language Models Confidence Estimation in Language Models Vision-Language Grounding Visually Grounded Reasoning+1

Ask PDF

Abstract

Vision-Language Grounding

Explore similar work

Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models

H-GRPO: Permutation-Invariant Reinforcement Learning for Improving Spatial Grounded Reasoning

Grounded or Guessing? LVLM Confidence Estimation via Blind-Image Contrastive Ranking