PROVE: Probabilistic Visual Grounding via Patch-Level Evidence for Vision-Language Models

Mayank Nautiyal; Tianru Zhang; Li Ju; Ekta Vats; Prashant Singh

cs.CVOct 23, 2026

PROVE: Probabilistic Visual Grounding via Patch-Level Evidence for Vision-Language Models

Authors: Mayank Nautiyal, Tianru Zhang, Li Ju, Ekta Vats, Prashant Singh

Organizations: Department of Information Technology, Uppsala University, Sweden · Science for Life Laboratory, Uppsala University, Sweden

Abstract

Visual grounding with vision-language models such as CLIP typically produces a single deterministic relevance map for an image-query pair, without indicating whether the localisation is reliable or whether the referent is present at all. We introduce PROVE (Probabilistic Visual Grounding via Patch-Level Evidence), a post-hoc evidential adapter that maps frozen CLIP patch features to per-patch Beta distributions without modifying the backbone. PROVE casts grounding as per-patch membership: each patch accumulates positive and negative evidence for whether it belongs to the referent, so a query may match zero, one, or many patches. We train PROVE with an evidential objective on soft patch-coverage targets, and define a prior-pull objective that keeps absent-referent examples near an uninformative prior, so that vacuity serves as an absence signal without a dedicated presence classifier. During inference, a single forward pass yields, in closed form, a posterior-mean grounding map, a spatial vacuity map, and an image-level absence score. We evaluate PROVE on referring-expression benchmarks: it is competitive in localisation with state-of-the-art CLIP grounding methods, its uncertainty tracks localisation error and supports selective prediction under abstention, and its image-level score detects when the referent is absent altogether. Our evidential patch-membership modelling offers a simple, drop-in mechanism for probabilistic verification in frozen vision-language grounding systems.

Figures & tables

Figure 1: Overview of PROVE. Frozen CLIP encoders ( $\Psi_{I}$ , $\Psi_{T}$ ) give patch and text features and a saliency prior (Grad-ECLIP); a trained evidential head maps them to per-patch Betas, giving a mean grounding map and a vacuity map.

Figure 2: Beta evidence and predictive uncertainty. (a) The balance of

\alpha_{i}

and

\beta_{i}

shifts relevance toward foreground or background;

\alpha_{i}\!\approx\!\beta_{i}\!\gg\!1

concentrates around intermediate relevance, and

\alpha_{i}=\beta_{i}=1

is the uniform low-evidence prior. (b) For fixed mean

\mu_{i}

, higher concentration

s_{i}=\alpha_{i}+\beta_{i}

reduces the predictive variance.

	gRefCOCO (people)			gRefCOCO (objects)			Flickr30k Entities			PhraseCut
Method	PG $\uparrow$	EBPG $\uparrow$	IoU@ $k\uparrow$	PG $\uparrow$	EBPG $\uparrow$	IoU@ $k\uparrow$	PG $\uparrow$	EBPG $\uparrow$	IoU@ $k\uparrow$	PG $\uparrow$	EBPG $\uparrow$	IoU@ $k\uparrow$
Grad-CAM [ 24 ]	0.202	0.225	0.179	0.181	0.229	0.209	0.400	0.358	0.320	0.295	0.215	0.234
Grad-ECLIP [ 33 ]	0.552	0.372	0.283	0.522	0.358	0.302	0.820	0.567	0.440	0.586	0.350	0.350
CLIP-Surgery [ 14 ]	0.495	0.263	0.299	0.433	0.265	0.310	0.822	0.400	0.457	0.630	0.245	0.403
Rollout [ 1 ]	0.124	0.200	0.146	0.081	0.166	0.098	0.238	0.303	0.239	0.116	0.151	0.101
PROVE (ours)	$\mathbf{0.855}_{\pm.001}$	$\mathbf{0.459}_{\pm.004}$	$\mathbf{0.616}_{\pm.002}$	$\mathbf{0.759}_{\pm.005}$	$\mathbf{0.412}_{\pm.004}$	$\mathbf{0.538}_{\pm.003}$	$0.764_{\pm.005}$	$\underline{0.502}_{\pm.004}$	$\mathbf{0.527}_{\pm.003}$	$0.512_{\pm.014}$	$\underline{0.274}_{\pm.003}$	$\underline{0.354}_{\pm.011}$

Table 1: Grounding localisation. Best in bold , second best underlined . PROVE: mean

{}_{\pm\text{std}}

over 5 seeds; baselines deterministic.

Figure 3: Qualitative comparison on single-, multi-, and no-target queries. PROVE’s mean map localises the referent(s), while vacuity flags the absent case; the baselines peak even when the referent is absent. Maps are min–max normalised.

Split	Method	AUROC	N-acc	T-acc	Bal
testA	Grad-ECLIP	0.682	0.570	0.711	0.640
	Det. control	0.823	0.643	0.864	0.754
	PROVE	$\mathbf{0.849}_{\pm.010}$	$\mathbf{0.657}_{\pm.029}$	$\mathbf{0.928}_{\pm.017}$	$\mathbf{0.792}_{\pm.006}$
testB	Grad-ECLIP	0.746	0.518	0.853	0.686
	Det. control	0.789	0.671	0.749	0.710
	PROVE	$\mathbf{0.813}_{\pm.006}$	$0.669_{\pm.026}$	$\mathbf{0.869}_{\pm.021}$	$\mathbf{0.769}_{\pm.004}$

Table 2: Absent-referent detection on gRefCOCO testA (people) and testB (objects). Best per split in bold ; PROVE is mean

{}_{\pm\text{std}}

over 5 seeds, baselines deterministic.

Panel of Figure 4 — Figure 4: Uncertainty as a ranking signal. (a) IoU@ $k$ per vacuity bin (bin 1 most certain). (b) IoU@ $k$ on the retained set as the most uncertain images are rejected. Both span four datasets; mean over 5 seeds, shaded $\pm 1$ std.

	People			Objects			Flickr30k			PhraseCut
Method	PG	EBPG	IoU@ $k$	PG	EBPG	IoU@ $k$	PG	EBPG	IoU@ $k$	PG	EBPG	IoU@ $k$
$\mathrm{PROVE}^{\mathrm{det}}$	0.86	0.63	0.63	0.76	0.55	0.55	0.77	0.68	0.55	0.52	0.41	0.36
PROVE	0.86	0.46	0.62	0.76	0.41	0.54	0.76	0.50	0.53	0.51	0.27	0.35

Table 3: Deterministic vs. probabilistic localisation. PROVE: mean over 5 seeds. Higher value per metric in bold (PG comparable across both).

		Loc.		Absence
Split	Variant	PG	IoU@ $k$	AUROC	$T$ -acc	Bal.
testA	Full	0.85	0.61	0.83	0.95	0.78
	$-\mathcal{L}_{\mathrm{nt}}$	0.86	0.63	0.56	1.00	0.50
	$-\mathcal{L}_{\mathrm{reg}}$	0.84	0.59	0.83	0.85	0.78
testB	Full	0.76	0.54	0.80	0.90	0.76
	$-\mathcal{L}_{\mathrm{nt}}$	0.76	0.54	0.57	1.00	0.50

Table 4: Loss ablation on gRefCOCO testA (people) and testB (objects), single-seed runs. Full is the default PROVE configuration; each lower row removes one auxiliary term.

Split	Backbone	PG $\uparrow$	EBPG $\uparrow$	IoU@ $k$ $\uparrow$
testA	ViT-B/16	$0.855{\scriptstyle\,\pm.001}$	$0.459{\scriptstyle\,\pm.004}$	$0.616{\scriptstyle\,\pm.002}$
testA	ViT-L/14	$\mathbf{0.871}{\scriptstyle\,\pm.002}$	$\mathbf{0.479}{\scriptstyle\,\pm.004}$	$\mathbf{0.632}{\scriptstyle\,\pm.002}$
testB	ViT-B/16	$0.759{\scriptstyle\,\pm.005}$	$0.412{\scriptstyle\,\pm.004}$	$0.538{\scriptstyle\,\pm.003}$
testB	ViT-L/14	$\mathbf{0.773}{\scriptstyle\,\pm.003}$	$\mathbf{0.433}{\scriptstyle\,\pm.003}$	$\mathbf{0.560}{\scriptstyle\,\pm.001}$

Table S1 : Localisation across backbones on gRefCOCO. PROVE: mean

\pm

std over five seeds (four for ViT-L/14).

Panel of Figure 10 — Figure S1 : ViT-L/14 uncertainty behaviour on gRefCOCO. (a) IoU@ $k$ across vacuity bins, where bin 1 is the most certain. (b) IoU@ $k$ on the retained set as a function of the rejected fraction. Mean over five seeds, with shading indicating $\pm 1$ standard deviation.

Split	Backbone	AUROC	$N$ -acc	$T$ -acc	Bal.
testA	ViT-B/16	$0.849{\scriptstyle\,\pm.010}$	$0.657{\scriptstyle\,\pm.029}$	$0.928{\scriptstyle\,\pm.017}$	$0.792{\scriptstyle\,\pm.006}$
testA	ViT-L/14	$\mathbf{0.851}{\scriptstyle\,\pm.008}$	$\mathbf{0.660}{\scriptstyle\,\pm.020}$	$\mathbf{0.936}{\scriptstyle\,\pm.012}$	$\mathbf{0.798}{\scriptstyle\,\pm.004}$
testB	ViT-B/16	$0.813{\scriptstyle\,\pm.006}$	$\mathbf{0.669}{\scriptstyle\,\pm.026}$	$0.869{\scriptstyle\,\pm.021}$	$0.769{\scriptstyle\,\pm.004}$
testB	ViT-L/14	$\mathbf{0.822}{\scriptstyle\,\pm.008}$	$0.667{\scriptstyle\,\pm.016}$	$\mathbf{0.888}{\scriptstyle\,\pm.013}$	$\mathbf{0.778}{\scriptstyle\,\pm.003}$

Table S2 : Absent-referent detection across backbones on gRefCOCO,

A=-\max_{i}s_{i}

. PROVE: mean

\pm

std over five seeds (four for ViT-L/14).

Split	Absence score	AUROC	$N$ -acc	$T$ -acc	Bal.
testA (people)	Grad-ECLIP $-\max_{i}\tilde{s}_{i}$	$0.682$	$0.570$	$0.711$	$0.640$
	Det. control $-\max_{i}\mu_{i}^{\mathrm{det}}$	$0.823$	$0.643$	$0.864$	$0.754$
	PROVE $-\max_{i}\alpha_{i}$	$0.829{\scriptstyle\,\pm 0.002}$	$0.644{\scriptstyle\,\pm 0.003}$	$0.883{\scriptstyle\,\pm 0.006}$	$0.763{\scriptstyle\,\pm 0.002}$
	PROVE $-\mathrm{top}\text{-}k\,s_{i}$	$0.847{\scriptstyle\,\pm 0.010}$	$\mathbf{0.661}{\scriptstyle\,\pm 0.024}$	$0.921{\scriptstyle\,\pm 0.013}$	$0.791{\scriptstyle\,\pm 0.006}$
	PROVE $-\max_{i}s_{i}$	$\mathbf{0.849}{\scriptstyle\,\pm 0.010}$	$0.657{\scriptstyle\,\pm 0.029}$	$\mathbf{0.928}{\scriptstyle\,\pm 0.017}$	$\mathbf{0.792}{\scriptstyle\,\pm 0.006}$
testB (objects)	Grad-ECLIP $-\max_{i}\tilde{s}_{i}$	$0.746$	$0.518$	$0.853$	$0.686$

Table S3 : Absence score comparison on gRefCOCO. PROVE results are mean

\pm

std over five seeds; baselines are single runs. The threshold

\tau

is selected on validation with

T\text{-acc}\!\geq\!0.90

. Best result per split and metric is shown in bold .

Split	Metric	$\gamma=5$	$\gamma=10$	$\bm{\gamma=20}$	$\gamma=40$	$\gamma=100$
testA (people)	PG	$0.858$	$\mathbf{0.859}$	$0.857$	$0.856$	$0.854$
	EBPG	$0.374$	$0.423$	$0.463$	$0.481$	$\mathbf{0.504}$
	IoU@ $k$	$\mathbf{0.619}$	$0.617$	$0.616$	$0.615$	$0.615$
	AUROC	$0.823$	$\mathbf{0.849}$	$0.836$	$0.847$	$0.833$
testB (objects)	PG	$0.759$	$\mathbf{0.761}$	$0.755$	$0.757$	$0.757$
	EBPG	$0.340$	$0.380$	$0.415$	$0.427$	$\mathbf{0.449}$

Table S4 : Effect of evidence capacity

\gamma

on gRefCOCO. Best per row in bold .

Figure S2 : Qualitative results on Flickr30k Entities.

Figure S3 : Qualitative results on PhraseCut.

Figure S4 : Qualitative results on gRefCOCO TestA (People).

Figure S5 : Qualitative results on gRefCOCO TestB (Objects).

Vision-Language Grounding

Vision-Language Grounding links words and phrases to the objects, regions, actions, or moments they describe in images, video, and scenes.

See all 418 papers

Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models
Apr 9, 2026Marcel Gröpl, Jaewoo Jung, Seungryong Kim +2Vision-Language Grounding VLM Interpretability
ETH Zürich · KAIST AI · ETH AI Center
H-GRPO: Permutation-Invariant Reinforcement Learning for Improving Spatial Grounded Reasoning
Jun 29, 2026Eric Peh, Debaditya Roy, Chen Li +1RL for VLMs Visual Reasoning
Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur, India +1
Grounded or Guessing? LVLM Confidence Estimation via Blind-Image Contrastive Ranking
May 11, 2026Reza Khanmohammadi, Erfan Miahi, Simerjot Kaur +4VLM Evaluation Confidence Estimation for VLMs
Michigan State University · Independent AI Researcher · JPMorgan AI Research +1

Apr 9, 2026J/K move · Enter open · S save

Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models

Marcel Gröpl, Jaewoo Jung, Seungryong Kim, Marc Pollefeys+1

ETH Zürich · KAIST AI · ETH AI Center

Vision-Language Grounding VLM Interpretability Gradient-Based Attribution

Open paper AskPDF arXiv

Despite rapid progress, pretrained vision-language models still struggle when answers depend on tiny visual details or on combining clues spread across multiple regions, as in documents and compositional queries. We address this by framing grounding as test-time evidence retrieval: given a query, the model should actively identify where to look next to resolve ambiguity. To this end, we propose a training-free, model-intrinsic grounding method that uses uncertainty as supervision. Specifically, we compute the entropy of the model's next-token distribution and backpropagate it to the visual token embeddings to obtain an entropy-gradient relevance map, without auxiliary detectors or attention-map heuristics. We then extract and rank multiple coherent regions to support multi-evidence queries, and introduce an iterative zoom-and-reground procedure with a spatial-entropy stopping rule to avoid over-refinement. Experiments on seven benchmarks across four VLM architectures demonstrate consistent improvements over existing methods, with the largest gains on detail-critical and high-resolution settings, while also producing more interpretable evidence localizations.

Abstract

Vision-Language Grounding

Explore similar work

Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models

H-GRPO: Permutation-Invariant Reinforcement Learning for Improving Spatial Grounded Reasoning

Grounded or Guessing? LVLM Confidence Estimation via Blind-Image Contrastive Ranking