SAGE: Sense-Aligned Multimodal Grounding Evaluation with Gloss-Augmented Reasoning

Sunjae Kwon; Suhaas Kiran DG; Zonghai Yao; Talha Chafekar; Deepali Singh; Hong Yu

cs.CLOct 22, 2026

SAGE: Sense-Aligned Multimodal Grounding Evaluation with Gloss-Augmented Reasoning

Authors: Sunjae Kwon, Suhaas Kiran DG, Zonghai Yao, Talha Chafekar, Deepali Singh, Hong Yu

Organizations: UMass Amherst · UMass Lowell · UMass Chan Medical School · VA Bedford Health Care

Abstract

This paper introduces Sense-Aligned Grounding Evaluation (SAGE), a unified 10-candidate image-selection benchmark that reformulates VWSD, VerSe, and WikiDiverse into a shared term-centric protocol covering noun-, verb-, and entity-centered grounding. SAGE includes confusable hard negatives and a context-swapped VerSe diagnostic to probe target-term grounding beyond direct caption--image compatibility. We further propose Gloss-Augmented Reasoning (GAR), a training-free framework combining definition grounding with constraint-guided semantic verification. Experiments show that strong VWSD performance does not transfer uniformly across grounding settings. With Gemma3-12B, definition augmentation improves all three subsets, while constraint-guided reasoning improves VWSD and yields particularly strong gains on VerSe, but is less effective on WikiDiverse. Combining the two signals achieves the best macro-average performance, improving Hits@1 from 65.90 to 71.22. These results show that SAGE supports both unified evaluation and fine-grained analysis of explicit semantic reasoning across grounding settings.

Figures & tables

Figure 1: Original task formulations of the source datasets in SAGE. VWSD selects a candidate image, while VerSe and WikiDiverse map targets to OntoNotes senses and Wikipedia entities, respectively. Red indicates targets; green indicates correct outputs. Alternative entries are illustrative; ellipses indicate omissions.

Figure 2: Illustration of SPIS. Given a context and target verb, the task is to select the candidate image depicting the intended verb sense. SPIS replaces the original caption-paired gold image with another image annotated with the same OntoNotes verb sense, preserving the target meaning while varying objects, participants, and scene context. The swapped image becomes the gold image in the final 10-candidate set.

	VWSD	VerSe	WikiDiverse
Train	12,869	1,042	10,750
Validation	-	-	1,288
Test	463	1,264	1,243

Table 1: Summary statistics of the three datasets in SAGE after reformulation.

Figure 3: Illustrative overview of the proposed GAR framework. Given a target term $t$ and context $C$ , GAR first synthesizes a contextually plausible definition from candidate glosses retrieved from a lexical knowledge base and expands it into visually checkable evidence. The expanded definition is decomposed into semantic constraints, reformulated as binary visual questions, and verified against each candidate image. CGR aggregates the affirmative probabilities into an interpretable semantic score for reranking.

Model Type	Baselines	VWSD		VerSe		WikiDiverse		Macro Avg.
Model Type	Baselines	Hits@1	MRR	Hits@1	MRR	Hits@1	MRR	Hits@1	MRR
	CLIP	57.7 $\pm$ 4.5	72.2 $\pm$ 3.1	43.7 $\pm$ 2.7	60.6 $\pm$ 2.0	48.9 $\pm$ 2.8	68.7 $\pm$ 1.8	50.1 $\pm$ 2.0	67.2 $\pm$ 1.4
	UVWSD	60.9 $\pm$ 4.4	75.2 $\pm$ 2.9	45.7 $\pm$ 2.8	62.2 $\pm$ 2.0	52.3 $\pm$ 2.8	70.5 $\pm$ 1.8	53.0 $\pm$ 2.0	69.3 $\pm$ 1.3
	SigLIP2-0.3B	76.0 $\pm$ 3.9	84.9 $\pm$ 2.5	51.4 $\pm$ 2.8	66.2 $\pm$ 2.0	51.6 $\pm$ 2.8	70.5 $\pm$ 1.8	59.7 $\pm$ 1.9	73.9 $\pm$ 1.2
ITMs	SigLIP2-1B	74.7 $\pm$ 4.0	84.3 $\pm$ 2.5	51.2 $\pm$ 2.8	66.6 $\pm$ 2.0	53.5 $\pm$ 2.8	71.9 $\pm$ 1.8	59.8 $\pm$ 1.9	74.3 $\pm$ 1.2
	InternVL 3.5-2B	55.1 $\pm$ 4.5	69.7 $\pm$ 3.2	67.2 $\pm$ 2.6	78.9 $\pm$ 1.7	44.8 $\pm$ 2.8	63.8 $\pm$ 1.9	55.7 $\pm$ 2.0	70.8 $\pm$ 1.4

Table 2: Experimental results on SAGE. Scores are point estimates

\pm

95% bootstrap confidence-interval half-widths (10,000 item-level resamples). Within each model type, best and second-best scores are bolded and italicized , respectively. Ranking procedures are in Appendix H .

	VWSD	VerSe	WikiDiverse
VWSD	1.00	0.46	0.74
VerSe	0.46	1.00	0.47
WikiDiverse	0.74	0.47	1.00

Table 3: Kendall’s

\tau_{b}

correlations between model rankings across SAGE subsets in Table 2 . All off-diagonal correlations are significant at

p<0.05

Model	Orig.	Orig. (Neu.)	SPIS	SPIS (Neu.)	$\Delta_{\mathrm{SPIS}}\downarrow$	$\Delta_{\mathrm{verb}}\uparrow$
CLIP	91.7	90.7	43.7	40.3	48.0	3.4
Gemma3-12B	91.3	90.1	59.3	41.1	32.0	18.2
GPT-5.6-Sol	92.0	91.2	74.3	41.3	17.7	33.0

Table 4: VerSe Hits@1 under original pairing and SPIS, with and without target-verb neutralization.

\Delta_{\mathrm{SPIS}}=\mathrm{Original}-\mathrm{SPIS}

measures the performance change under sense-preserving image replacement, and

\Delta_{\mathrm{verb}}=\mathrm{SPIS}-\mathrm{SPIS(Neu.)}

measures sensitivity to target-verb information.

Model Type	Method	VWSD		VerSe		WikiDiverse		Macro Avg.
Model Type	Method	Hits@1	MRR	Hits@1	MRR	Hits@1	MRR	Hits@1	MRR
Gemma3-12B	Base	75.6	84.7	59.3	72.7	62.8	76.9	65.9	78.1
	Generated Def.	75.8	84.9	63.2 †	75.6 †	59.5 †	74.7 †	66.2	78.4
	Lexical Def.	79.9 †	87.7 †	67.2 †	78.9 †	64.6 ∗	78.0 ∗	70.6	81.5
	CGR	78.6 ∗	86.2 ∗	71.7 †	81.7 †	56.7 †	71.3 †	69.0	79.7
	Full GAR	81.2 †	88.6 †	69.8 †	81.1 †	62.7	76.8	71.2	82.1

Table 5: Ablation study of GAR. Generated Def. produces a context-specific definition without lexical-resource retrieval; Lexical Def. retrieves lexical knowledge and synthesizes an expanded definition for holistic ranking; CGR decomposes the lexical-semantic representation into visually verifiable constraints; and Full GAR combines Lexical Def. and CGR using RRF.

{}^{*}p<0.05

and

{}^{\dagger}p<0.01

show significant differences from Base.

Axis	VWSD		VerSe		WikiDiverse
Axis	Hits@1	MRR	Hits@1	MRR	Hits@1	MRR
Object	75.4	84.2	69.4	79.8	57.0	71.4
Attribute	74.3	83.5	63.9	76.1	42.0	68.4
State	67.6	77.7	67.8	79.2	32.1	50.9
All	78.6	86.2	71.7	81.7	56.7	71.3

Table 6: Ablation study on semantic axes for CGR with Gemma3-12B. Combining all axes performs best on VWSD and VerSe, while object-level constraints perform comparably to the all-axis variant on WikiDiverse.

Setting	Model	Hits@1	MRR
Training-free	CLIP-ViT-L/14 ( Yang et al., 2024 )	57.5	72.6
	UVWSD ( Kwon et al., 2023 )	60.9	75.2
	GPT_BLIP ( Kritharoula et al., 2023 )	72.4	82.3
	QVWSD ( Qiao et al., 2025 )	62.4	76.6
	DCTP ( Bhattacharya et al., 2026 )	62.2	76.9
	Qwen2.5-VL-72B ( Musacchio et al., 2025 )	79.1	–

Table 7: Comparison with previous VWSD models. Details of the models are provided in Appendix J .

	Hard Conf.	Rand. Conf.
WikiDiverse	14.56%	2.98%
VerSe	26.38%	11.59%

Table 8: Pairwise confusion analysis of hard and random negatives on Gemma3-12B. Each negative is compared individually against the gold image. Hard Conf. and Rand. Conf. denote the percentage of hard–gold and random–gold pairs, respectively, in which the negative is ranked above the gold image.

Error Type	VWSD	VerSe	WikiDiverse
Upstream Semantic / Referent	26%	12%	10%
Decomposition Drift	20%	18%	2%
Cross-Axis Conflict	6%	6%	6%
Non-Discriminative Constraint	32%	48%	76%
Visual Verification	16%	16%	6%

Table 9: Distribution (%) of manually categorized CGR errors (50 per subset; 150 total).

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

	VERB	NOUN	PROPN	Others	All
Count	12	392	54	5	463
Ratio	2.59%	84.67%	11.66%	1.08%	100.00%

Appendix

Table 10: Distribution of POS categories in the VWSD test split.

Figure 4: An example of “play” in VerSe. Borders indicate labels: red = correct , blue = hard negatives , green = random negatives .

Figure 5: An example of “catch” in VerSe. Borders indicate labels: red = correct , blue = hard negatives , green = random negatives .

Figure 6: An example of “Irene” in WikiDiverse. Borders indicate labels: red = correct , blue = hard negatives , green = random negatives .

Figure 7: An example of “Shadow” in WikiDiverse. Borders indicate labels: red = correct , blue = hard negatives , green = random negatives .

Dataset	Unanimous	2-of-3 agreement	All different	Fleiss’ $\kappa$
VerSe	150/200 (75.0%)	43/200 (21.5%)	7/200 (3.5%)	0.801
WikiDiverse	155/200 (77.5%)	40/200 (20.0%)	5/200 (2.5%)	0.823

Appendix

Table 11: Agreement composition among the three independent annotators. Unanimous agreement indicates that all three annotators selected the same image; 2-of-3 agreement indicates that exactly two annotators selected the same image; and all different indicates that the three annotators selected three distinct images. Fleiss’

\kappa

was computed from the original annotations before aggregation.

Mismatch category	Description	Support
Caption-specific vs. sense-level match	Annotators select a candidate that better matches specific participants, objects, or scene details in the original caption, while the SAGE gold preserves the same OntoNotes verb sense under a different visual realization.	7/23 (30.4%)
Competing sense/event cue	A non-gold candidate evokes a different sense or event interpretation of the same target verb, creating sense-level competition.	4/23 (17.4%)
Multiple plausible same-sense realizations	More than one candidate plausibly instantiates the same coarse OntoNotes verb sense or a closely related event configuration.	6/23 (26.1%)
Weak or ambiguous visual evidence	The intended verb event is visually underspecified, weakly depicted, or difficult to verify from the available candidate images.	6/23 (26.1%)

Appendix

Table 12: Breakdown of the 23 human–gold mismatches in the VerSe validation sample. Support is computed over the mismatched cases.

Figure 8: Example of caption-specific versus sense-level matching in VerSe. The target verb is “lean,” defined as inclining from a vertical position. Both the gold and alternative candidates can evoke the broad sense of leaning or being inclined, but they differ in the visual realization of the event and its participants. This illustrates a residual tension in the SPIS setting: the gold preserves the verb sense, while an alternative may appear closer to the original caption details. The red border indicates the SAGE gold label.

Figure 9: Example of a competing sense/event cue in VerSe. The target verb is “cover” in the sense of being overlaid with something, as in donuts covered with icing. A competing candidate can evoke a different protective or shielding sense of “cover,” making the sense distinction visually ambiguous. The red border indicates the SAGE gold label.

Figure 10: Example of multiple plausible same-sense realizations in VerSe. The target verb is “catch” in the sense of seizing or stopping the movement of something. Several candidates depict related catching or intercepting events with different participants and settings, making the verb sense visually plausible in more than one image. The red border indicates the SAGE gold label.

Mismatch category	Description	Support
Fine-grained knowledge requirement	The gold image is plausible, but correct identification requires fine-grained geographic, domain-specific, or relational world knowledge.	4/12 (33.3%)
Benchmark label ambiguity	The gold image conflicts with the textual context, or another candidate image is also clearly valid under the given target and context.	3/12 (25.0%)
Representation ambiguity	The target entity can be visually represented in multiple plausible ways, such as a country, team, organization, logo, building, event, or related scene.	5/12 (41.7%)

Appendix

Table 13: Breakdown of the 12 human–gold mismatches in the WikiDiverse validation sample. Support is computed over the mismatched cases.

Figure 11: Example of fine-grained knowledge requirement in WikiDiverse. Although several candidates depict plausible county buildings or maps, the context specifies Union County in New Mexico, and the gold image is the corresponding New Mexico referent. The red border indicates the SAGE gold label.

Figure 12: Example of benchmark label ambiguity in WikiDiverse. The target mention is “Spanish Grand Prix” in the context of Jenson Button at the Spanish Grand Prix. Although the benchmark gold image corresponds to a circuit-map representation, another candidate depicts an actual race scene from the Spanish Grand Prix, making more than one candidate plausibly valid under the given context. The red border indicates the SAGE gold label.

Figure 13: Example of representation ambiguity in WikiDiverse. The target mention “Parliament” can be visually grounded as a physical building, an institution, or an abstract parliamentary structure. The benchmark gold image depicts the Palace of Westminster, while other candidates provide plausible institutional representations. The red border indicates the SAGE gold label.

	Avg. Hard	Avg. Random
WikiDiverse	4.55	4.45
VerSe	0.85	8.15

Appendix

Table 14: Average numbers of hard and random negatives per query. Each candidate set contains nine negatives in total; available hard negatives are included first and remaining positions are filled with random negatives.

	BabelNet			WordNet
	VWSD	VerSe	WikiDiverse	VWSD	VerSe	WikiDiverse
Coverage (%)	100.00	100.00	95.01	97.84	100.00	31.50
Mean Ambiguity	27.76	19.07	11.11	6.94	27.11	0.84

Appendix

Table 15: Coverage and ambiguity statistics of lexical knowledge bases across datasets. Coverage is measured as the percentage of instances with at least one available sense definition. Ambiguity is measured as the average number of candidate senses per instance.

Subset	Annotator A	Annotator B	Agreement / AC1
VWSD	48/50 (96.0%)	47/50 (94.0%)	47/50 (94.0%) / 0.934
VerSe	46/50 (92.0%)	46/50 (92.0%)	48/50 (96.0%) / 0.953
WikiDiverse	44/50 (88.0%)	46/50 (92.0%)	48/50 (96.0%) / 0.951
Overall	138/150 (92.0%)	139/150 (92.7%)	143/150 (95.3%) / 0.946

Appendix

Table 16: Human assessment of the contextual plausibility of synthesized definitions.

Subset	Target and context	Retrieved-gloss gap	Synthesized definition
WikiDiverse	jets : “One of the jets shadowing the bomber”	Retrieved entries referred to the New York Jets, a musical gang, and sports clubs, but not aircraft.	Jets are fast, powerful aircraft.
WikiDiverse	McClellan : McClellan shakes hands with President Bush …	Retrieved entries provided only a surname and an airport.	McClellan refers to a person who served as the White House Press Secretary.
WikiDiverse	AFC Asian Cup Group A match : Thailand about to take a corner kick against Oman in their AFC Asian Cup Group A match.	No candidate gloss was retrieved.	A competition game between teams in the same group during the AFC Asian Cup tournament.

Appendix

Table 17: Examples in which definition synthesis recovered a plausible meaning absent from the retrieved candidate glosses.

Target	Context (abridged)	Synthesized definition	Issue
VWSD
expression	expression maths	In computer science, an expression is a syntactic entity in a programming language that may be evaluated to determine its value.	Wrong domain
VerSe
perform	A man performing a trick on a skateboard.	To do something in front of an audience, often to entertain.	Wrong action sense
perform	A person with skis performing a jump.	To do something in front of an audience, such as acting or music, often in order to entertain.	Wrong action sense
sit	Two people sitting on top of a truck.	Be located or situated somewhere.	Wrong posture sense

Appendix

Table 18: Synthesized definitions jointly judged contextually implausible by both annotators, grouped by SAGE subset.

Model	Params	Hits@1	MRR
InternVL-3.5-2B	2B	55.1	69.7
InternVL-3.5-8B	8B	66.1	78.6
Gemma3-4B	4B	57.9	72.6
Gemma3-12B	12B	75.6	84.7
Qwen3-VL-2B	2B	62.2	74.6
Qwen3-VL-8B	8B	76.5	85.4

Appendix

Table 19: Open-source VLM performance under pointwise token-likelihood ranking on the VWSD subset. Scores are reported on a 0–100 scale.

Model	Params	Hits@1	MRR
Qwen2.5-VL-3B-Instruct	3B	63.7	–
Qwen2.5-VL-7B-Instruct	7B	71.3	–
Qwen2.5-VL-32B-Instruct	32B	76.5	–
Qwen2.5-VL-72B-Instruct	72B	79.1	–
Aya-Vision-8B	8B	20.5	–
Aya-Vision-32B	32B	13.8	–

Appendix

Table 20: Published open-source VLM results from Musacchio et al. (2025) on VWSD using joint multiple-choice prediction. This method produces a single top-1 selection rather than a complete candidate ranking; MRR is therefore not reported.

Model	Dataset	Scalar score		YES/NO logprob
		Hits@1	MRR	Hits@1	MRR
GPT-4.1-mini	VWSD	78.6	86.9	80.8	88.1
	VerSe	58.2	71.1	62.1	74.4
	WikiDiverse	66.6	79.8	67.9	80.0
Gemma3-12B	VWSD	58.8	74.5	75.6	84.7
	VerSe	47.0	62.3	59.3	72.7

Appendix

Table 21: Sensitivity to scoring interfaces. GPT-4.1-mini and Gemma3-12B are evaluated using both scalar compatibility scores and binary YES/NO token likelihoods. Token-likelihood scoring matches or exceeds scalar scoring for both models.

Figure 14: Upstream semantic error for VWSD #129 ( torch ). The synthesized definition changes the intended plant referent into a portable light source, causing the generated Object constraint to favor the flashlight distractor.

Figure 15: Upstream referent error for WikiDiverse #604 ( Garuda ). The airline referent is replaced by the mythological deity before question generation.

Figure 16: Decomposition drift for VWSD #30 ( lift ). Questions derived from a facelift definition drift to the physical lifting sense.

Figure 17: Decomposition drift for VWSD #211 ( milk ). A dairy-specific Attribute constraint overwhelms the correctly identified river referent.

Figure 18: Cross-axis conflict for WikiDiverse #598. Target identity and airport-scene attributes favor different candidates.

Figure 19: Cross-axis conflict for VerSe #956 ( fly ). State evidence favors the gold image, whereas an animal-oriented Object constraint favors the distractor.

Figure 20: Non-discriminative constraints for WikiDiverse #217. Generic party-logo cues do not distinguish the U.S. Libertarian Party from its U.K. counterpart.

Figure 21: Non-discriminative constraints for WikiDiverse #1002. Broad fighter-aircraft cues fit both the Su-27 and F-16.

Figure 22: Visual verification error for VerSe #494 ( bite ). The VLM affirms tooth use for a standing dog despite the absence of visible biting.

Figure 23: Visual verification error for WikiDiverse #1013. The model accepts the Stobhill constraint even though the visible sign names Moorgreen Hospital.

Figure 24: Pilot runtime profile on VWSD, measured in seconds.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 999 papers

AgroVG: A Large-Scale Multi-Source Benchmark for Agricultural Visual Grounding
May 21, 2026Haocheng Li, Juepeng Zheng, Zenghao Yang +5Agricultural Image Analysis Set Prediction
China Agricultural University · Sun Yat-sen University · National Supercomputing Center in Shenzhen +3
GAVEL: Grounded Caption Error Verification and Localization
Jun 25, 2026Zixian Gao, Atsushi Hashimoto, Kuniaki SaitoVLM Evaluation Text-Image Alignment
OMRON SINIC X Corporation
Vision-Language Grounding as Bidirectional Concept Correspondence
Aug 8, 2026Jieyu Zhang, Ziqi Gao, Luke Zettlemoyer +1Vision-Language Grounding Vision-Language Alignment
University of Washington · Allen Institute for AI · FAIR at Meta

May 21, 2026J/K move · Enter open · S save

AgroVG: A Large-Scale Multi-Source Benchmark for Agricultural Visual Grounding

Haocheng Li, Juepeng Zheng, Zenghao Yang, Kaiqi Du+4

China Agricultural University · Sun Yat-sen University · National Supercomputing Center in Shenzhen+3

Agricultural Image Analysis Set Prediction Visual Grounding

Open paper AskPDF arXiv

Visual grounding, the task of localizing objects described by natural-language expressions, is a foundational capability for agricultural AI systems, enabling applications such as selective weeding, disease monitoring, and targeted harvesting. Reliable evaluation of agricultural visual grounding remains challenging because agricultural targets are often small, repetitive, occluded, or irregularly shaped, and instructions may refer to one, many, or no objects in an image. Evaluating this capability therefore requires jointly testing localization accuracy, target-set completeness, and existence-aware abstention. To address these challenges, we introduce AgroVG, a multi-source benchmark that formulates agricultural grounding as generalized set prediction: given an image and a referring expression, a model must return all matching target instances or abstain when no target is present. AgroVG contains 10{,}071 annotation-grounded image-query pairs from ten source datasets across six target families: crop/weed, fruit, wheat head, pest, plant disease, and tree canopy. It supports bounding-box grounding (T1) across all six families and instance-mask grounding (T2) on sources with reliable instance-level pixel annotations, with queries covering single-target, multi-target, and target-absent regimes. AgroVG further provides task-specific protocols for box-set matching and query-level mask coverage. Zero-shot evaluation of 26 model configurations spanning closed-source MLLMs, open-source VLMs, and specialized grounding systems reveals persistent gaps: the best multi-target Set-

F_1

reaches only 0.35, and the best positive-query mask success rate at [email protected] remains below 0.17. Data and code are available at https://anonymous.4open.science/r/AgroVG-5172/ .

Abstract

VLM Evaluation

Explore similar work

AgroVG: A Large-Scale Multi-Source Benchmark for Agricultural Visual Grounding

GAVEL: Grounded Caption Error Verification and Localization

Vision-Language Grounding as Bidirectional Concept Correspondence