Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretation

Alvin Wei Ming Tan; Ben Prystawski; Veronica Boyce

cs.CLAug 30, 2026

Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretation

Authors: Alvin Wei Ming Tan, Ben Prystawski, Veronica Boyce

Organizations: Department of Psychology, Stanford University, USA · Department of Brain and Cognitive Sciences, MIT, USA

Abstract

Flexible adaptation to context and shared pragmatic intuitions contribute to smooth human conversation. Iterated reference games---in which players repeatedly pick out novel referents using language---present a test case for agents' ability to perform context-sensitive pragmatic reasoning in multi-turn linguistic environments. We tested humans and vision--language models on their ability to identify the intended meaning of descriptions produced in iterated reference games, varying the provided context in terms of amount, order, and relevance. While humans performed well consistently, the models we evaluated could make use of prior context to interpret humans' referring expressions, but they struggled to build up the relevant context to interpret those expressions effectively. Our results suggest that the models we evaluated lack core skills needed for efficient linguistic collaboration.

Figures & tables

Figure 1: (A) Overview of the experimental structure for the original interactive games in Boyce et al. (2024) . (B) User interface for each trial in the experiments with naïve participants.

	Condition	$N_{\textup{trials}}$	Original game	Same game	Trial order	Same tangram seen
Cumulative	Yoked	720	Y	Y	Original	Y
	Backward	720	Y	Y	Reversed	Y
	Shuffled	7056	Y	Y	Permuted	Y
	Random	7704	N	N	Permuted	Y
Non-cumulative	Ablated	720	Y	Y	Original	N
	Other-within	6480	N	Y	Original	Y

Table 1: Conditions of in-context trials seen by models. Columns indicate whether the in-context trials are from the same original game as the test trial, whether the in-context trials are all from the same original game, what the order of the in-context trials is, and whether the tangram in the test trial occurs in the in-context trials.

Figure 2: Matcher accuracy across all cumulative conditions and matcher types (both human and model), with best-fit LOESS curves, shown by repetition number as seen by the matcher. Subplots reflect the type of feedback received by model matchers: (A) interactive limited feedback, (B) human-yoked limited feedback, or (C) full feedback. Error bars indicate bootstrapped 95% confidence intervals. Dashed lines indicate the chance level (0.083).

Figure 3: Matcher accuracy in full feedback across all non-cumulative conditions, with best-fit LOESS curves, shown by repetition number as seen by the matcher except for the no context condition, which is shown by the original repetition number. Error bars indicate bootstrapped 95% confidence intervals. Dashed lines indicate the chance level (0.083).

Figure 4: Trial accuracy as a function of previous-repetition accuracy for the same tangram across matchers for the yoked, backward, shuffled, and random conditions under interactive limited feedback. The top row displays accuracy for trials subsequent to a correct trial, while the bottom row displays accuracy for trials subsequent to an incorrect trial. Error bars indicate bootstrapped 95% confidence intervals. Dashed lines indicate the chance level (0.083).

Figure 5: Difference in accuracy between the text-only and image–text paradigms across all conditions and models under full feedback. Error bars indicate bootstrapped 95% confidence intervals. Dashed lines indicate no difference.

Figure 6: Comparison between naïve human accuracy and model accuracy under interactive limited feedback, with best-fit linear regressions. Shaded regions indicate 95% confidence intervals. Dashed lines indicate perfect calibration ( $y=x$ ).

Figure 7: Matcher accuracy as a function of the proportion of metaphorical and holistic spans in descriptions.

Figure 8: Matcher accuracy for frontier models under interactive limited feedback.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: The image that was presented to the vision–language models, containing all 12 tangram shapes with their letter labels.

Figure 10: Example of the in-context trials presented to a model, with the User providing the trial text, the Assistant providing a guess, and the User responding with the correctness of the guess.

Figure 11: Matcher accuracy for the shuffled, backward, random, and no context conditions across all matcher types (both human and model), with best-fit LOESS curves, shown by repetition number from the original game. Error bars indicate bootstrapped 95% confidence intervals. Dashed lines indicate the chance level (0.083).

	$r$
Condition	Human split-half	Qwen 3 VL	Gemma 3	Llama 3.2	Molmo 2	Kimi VL
Yoked	$.42\;[.32,.50]$	$.08$	$-.07$	$-.02$	$.46$	$.19$
Backward	$.48\;[.40,.55]$	$.20$	$-.07$	$.02$	$.40$	$.19$
Shuffled	–	$.23$	$.02$	$.16$	$.47$	$.31$
Random	–	$.23$	$.04$	$.11$	$.59$	$.29$

Appendix

Table 2: Human split-half correlations and model–human correlations for the yoked, shuffled, backward, and random conditions. Human split-half correlations show the mean and 95% confidence interval over 1000 random splits.

Figure 12: Item-wise variation in accuracy across matchers and conditions. Error bars indicate bootstrapped 95% confidence intervals. Dashed lines indicate the chance level (0.083).

Figure 13: Correlation in confusion matrices across conditions for all models.

Figure 14: Centred discrepancy in model versus human accuracies in the random condition for selected lemmas, categorised by semantic categories. Error bars indicate 95% confidence intervals. Dashed line indicates zero (i.e., discrepancy equal to the grand mean). Empty/filled points indicate whether the confidence intervals do/do not include zero respectively.

Figure 15: (A) Matcher accuracies and (B) likelihood of perseverating on their previous answer for practice conditions. Error bars indicate bootstrapped 95% confidence intervals. Dashed lines indicate the chance level (0.083).

Figure 16: Proportion of invalid responses for frontier models under interactive limited feedback.

Pragmatic Reasoning in Language Models

Pragmatic reasoning in language models studies how models infer and communicate meanings shaped by context, intent, shared knowledge, and social norms.

See all 103 papers

When Contextual Inference Fails: Cancelability in Interactive Instruction Following
Mar 20, 2026Natalia Bila, Kata Naszádi, Alexandra Mayn +1Pragmatic Reasoning in Language Models Instruction Following
Department of Language Science and Technology, Saarland University · Language Technology Lab, University of Amsterdam
DRInQ: Evaluating Conversational Implicature with Controlled Context Variation
May 22, 2026Hirona Jacqueline Arai, Xiang RenLLM Evaluation Pragmatic Reasoning in Language Models
University of Southern California
Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialogue
Jun 30, 2026Nan Li, Albert Gatt, Massimo PoesioVLM Evaluation Vision-Language Grounding
Utrecht University, Utrecht, The Netherlands

Mar 20, 2026J/K move · Enter open · S save

When Contextual Inference Fails: Cancelability in Interactive Instruction Following

Natalia Bila, Kata Naszádi, Alexandra Mayn, Christof Monz

Department of Language Science and Technology, Saarland University · Language Technology Lab, University of Amsterdam

Pragmatic Reasoning in Language Models Instruction Following Clarification Question Generation

Open paper AskPDF arXiv

We investigate the separation of literal interpretation from contextual inference in a collaborative block-building tasks, where an agent must resolve underspecified instructions using context. We adapt an existing two-speaker psycholinguistic paradigm into an interactive benchmark called Build What I Mean (BWIM). This setup contrasts a pragmatically cooperative speaker with one who is only literally reliable. In BWIM, models face underspecified instructions and must choose between making a contextual inference or requesting clarification at a small communication cost. Evaluating several state-of-the-art LLMs, we find a clear dissociation between judgment and action. Although models successfully detect speaker unreliability in explicit confidence ratings, they fail to leverage this awareness when taking action. Instead of deploying efficient clarification strategies, models default to suboptimal behaviors. These include partner-blind over-clarification and question-averse guessing under uncertainty. BWIM provides a controlled environment to evaluate online partner adaptation and contextual reasoning in interactive settings.

Abstract

Pragmatic Reasoning in Language Models

Explore similar work

When Contextual Inference Fails: Cancelability in Interactive Instruction Following

DRInQ: Evaluating Conversational Implicature with Controlled Context Variation

Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialogue