A Multimodal Framework for Aligning Human Linguistic Descriptions with Visual Perceptual Data

Joseph Bingham

cs.AIFeb 23, 2026

A Multimodal Framework for Aligning Human Linguistic Descriptions with Visual Perceptual Data

Authors: Joseph Bingham

Organizations: Faculty of Biology, Technion University, Haifa, Israel.

Abstract

Establishing stable mappings between natural language expressions and visual percepts is a foundational problem for both cognitive science and artificial intelligence. Humans routinely ground linguistic reference in noisy, ambiguous perceptual contexts, yet the mechanisms supporting such cross-modal alignment remain poorly understood. In this work, we introduce a computational framework designed to model core aspects of human referential interpretation by integrating linguistic utterances with perceptual representations derived from large-scale, crowd-sourced imagery. The system approximates human perceptual categorization by combining scale-invariant feature transform (SIFT) alignment with the Universal Quality Index (UQI) to quantify similarity in a cognitively plausible feature space, while a set of linguistic preprocessing and query-transformation operations captures pragmatic variability in referring expressions. We evaluate the model on the Stanford Repeated Reference Game corpus (15,000 utterances paired with tangram stimuli), a paradigm explicitly developed to probe human-level perceptual ambiguity and coordination. Our framework achieves robust referential grounding. It requires 65% fewer utterances than human interlocutors to reach stable mappings and can correctly identify target objects from single referring expressions 41.66% of the time (versus 20% for humans).These results suggest that relatively simple perceptual-linguistic alignment mechanisms can yield human-competitive behavior on a classic cognitive benchmark, and offers insights into models of grounded communication, perceptual inference, and cross-modal concept formation. Code is available at https://anonymous.4open.science/r/metasequoia-9D13/README.md .

Figures & tables

Figure 1: An overview of the repeated reference game and our framework for lexical entrainment and common ground establishment. Our paper addresses the right-hand side of this figure, where the human serves as director and the AI as matcher. The sets $\Gamma,\Xi,\Omega$ maintain the current state of common ground, with $\Gamma$ containing finalized conceptual pacts, $\Xi$ the set of conceptual pacts under negotiation, and $\Omega$ containing any pacts which were rejected.

Figure 2: The symmetric simplicial sets of common ground that exist between human co-performers $A$ and $B$ , and machine co-performer $C$ . Each pair of co-performers has their own common ground, representing shared understanding about themselves, their joint activity, and the environment, represented by the nodes $AB,AC,BC$ at the learned Wasserstein barycenters of mutual alignment. Common ground $ABC$ is shared with all.

Figure 3: An example of the repeated reference problem, the director on the right, the matcher on the left. The director issues an utterance, $\varphi$ , indicating what they perceived the selected tangram stimuli to depict. The matcher can either guess which tangram they believe the director is referring to, pose a clarifying question, or wait for the director to provide more information. The illustrated example uses text from the open corpus.

Figure 4: Changes in the accuracy of our bindings resulting from varying the number of images scraped by the MCP matcher. These accuracy numbers represent achieving lexical entrainment on the referent in a single utterance, as a comparative measure.

Figure 5: An example of the distances of the closest 5 scraped photos to the target. The query text for this is ”tangram figure sitting and looking”. It should be noted that all queries were manually checked to ensure that the scraped images were unique from the tangram figure.

Figure 6: Processing of scraped images to conform with tangram data set from original to comparison measure. The use of grey-scale is in order to prevent color disagreements from disproportionately effecting results.

top k	Human	matcher
$k=1$	20.00	41.66%
$k=3$	N/A	63.01%
$k=5$	N/A	83.56%

Table 1: Top-k accuracy of the game when only the first director’s phrase is used. The human matcher got none of the tangrams correct with just one utterance from the director. This is starkly contrasted by our matcher, which was able to get 41.66% correct with just one phrase from the director.

	Time (ms)		Utterances Needed
Tangram ID	Human	matcher	Human	matcher 1 1 1 This is the average when the matcher gets the correct answer.
A	31737	1.2	2.5	1
B	21156	7.8	3.75	1
C	15311	3.3	2.5	2.3
D	27794	0.4	2.4	1
E	16614	2.9	2.4	1

Table 2: A comparison of the performance between the human matcher from the data set to our matcher give the same input phrases (utterances) from the director.

Referring Expression Comprehension

See all 21 papers

A Dynamic-Semantics Framework for Grounding Human Referring Expressions in Visual Perceptual Data
Aug 9, 2026Joseph BinghamVision-Language Grounding Visual Grounding
Department of Biology, Technion University, Haifa, IL.
MM-Conv: A Multimodal Dataset and Benchmark for Context-Aware Grounding in 3D Dialogue
May 20, 2026Anna Deichler, Jim O'Regan, Fethiye Irmak Dogan +4Query Reformulation Visually Grounded Reasoning
KTH Royal Institute of Technology, Stockholm, Sweden
Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretation
Aug 30, 2026Alvin Wei Ming Tan, Ben Prystawski, Veronica BoycePragmatic Reasoning in Language Models VLM Evaluation
Department of Psychology, Stanford University, USA · Department of Brain and Cognitive Sciences, MIT, USA

Aug 9, 2026J/K move · Enter open · S save

A Dynamic-Semantics Framework for Grounding Human Referring Expressions in Visual Perceptual Data

Joseph Bingham

Department of Biology, Technion University, Haifa, IL.

Vision-Language Grounding Visual Grounding Referring Expression Comprehension

Open paper AskPDF arXiv

Humans converge on shared names for novel, hard-to-describe objects through repeated interaction, a process psycholinguists call lexical entrainment. Leading vision-language models fail at this: recent empirical work documents that they do not shorten references, reuse successful expressions, or maintain stable pact state across turns. We present a framework that addresses the gap by externalizing pact state into three explicit, inspectable sets of referent-object bindings (

Γ, Ξ, Ω

), updated by a dynamic-semantics context-change rule. The symbolic layer sits on top of a lightweight perceptual-alignment pipeline that grounds noisy human referring expressions in crowd-sourced imagery via SIFT homographies and the Universal Quality Index. Evaluated on the Stanford Repeated Reference Game corpus (over 15{,}000 director-matcher utterances on abstract tangram stimuli), the framework places the correct target in its top-5 hypothesis set 83.56% of the time from a single director utterance. Human matcher top-1 accuracy on the same corpus is approximately 77-80%. We also report results on a held-out condition in which obvious tangram-adjacent images are excluded from the retrieved set, which provides a more conservative measurement of the grounding signal. Ablations isolate the contribution of each component: SIFT alignment, UQI, query preprocessing, and image augmentation. The central contribution is the combination: a transparent, auditable symbolic layer that recovers the structure of lexical entrainment turn by turn, paired with a perceptual channel whose behavior can be examined ablation by ablation. We also discuss in detail what the framework does not do. It is not interactive, it does not close the loop with the director, and its retrieval-driven perceptual channel is vulnerable to a class of leakage effects that we quantify and bound rather than wave away.

Abstract

Referring Expression Comprehension

Explore similar work

A Dynamic-Semantics Framework for Grounding Human Referring Expressions in Visual Perceptual Data

MM-Conv: A Multimodal Dataset and Benchmark for Context-Aware Grounding in 3D Dialogue

Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretation