Flexible adaptation to context and shared pragmatic intuitions contribute to smooth human conversation. Iterated reference games---in which players repeatedly pick out novel referents using language---present a test case for agents' ability to perform context-sensitive pragmatic reasoning in multi-turn linguistic environments. We tested humans and vision--language models on their ability to identify the intended meaning of descriptions produced in iterated reference games, varying the provided context in terms of amount, order, and relevance. While humans performed well consistently, the models we evaluated could make use of prior context to interpret humans' referring expressions, but they struggled to build up the relevant context to interpret those expressions effectively. Our results suggest that the models we evaluated lack core skills needed for efficient linguistic collaboration.
Figures & tables
Figure 1: (A) Overview of the experimental structure for the original interactive games in Boyce et al. (2024) . (B) User interface for each trial in the experiments with naïve participants.
Condition
Ntrials
Original game
Same game
Trial order
Same tangram seen
Cumulative
Yoked
720
Y
Y
Original
Y
Backward
720
Y
Y
Reversed
Y
Shuffled
7056
Y
Y
Permuted
Y
Random
7704
N
N
Permuted
Y
Non-cumulative
Ablated
720
Y
Y
Original
N
Other-within
6480
N
Y
Original
Y
Table 1: Conditions of in-context trials seen by models. Columns indicate whether the in-context trials are from the same original game as the test trial, whether the in-context trials are all from the same original game, what the order of the in-context trials is, and whether the tangram in the test trial occurs in the in-context trials.
Figure 2: Matcher accuracy across all cumulative conditions and matcher types (both human and model), with best-fit LOESS curves, shown by repetition number as seen by the matcher. Subplots reflect the type of feedback received by model matchers: (A) interactive limited feedback, (B) human-yoked limited feedback, or (C) full feedback. Error bars indicate bootstrapped 95% confidence intervals. Dashed lines indicate the chance level (0.083).
Figure 3: Matcher accuracy in full feedback across all non-cumulative conditions, with best-fit LOESS curves, shown by repetition number as seen by the matcher except for the no context condition, which is shown by the original repetition number. Error bars indicate bootstrapped 95% confidence intervals. Dashed lines indicate the chance level (0.083).
Figure 4: Trial accuracy as a function of previous-repetition accuracy for the same tangram across matchers for the yoked, backward, shuffled, and random conditions under interactive limited feedback. The top row displays accuracy for trials subsequent to a correct trial, while the bottom row displays accuracy for trials subsequent to an incorrect trial. Error bars indicate bootstrapped 95% confidence intervals. Dashed lines indicate the chance level (0.083).
Figure 5: Difference in accuracy between the text-only and image–text paradigms across all conditions and models under full feedback. Error bars indicate bootstrapped 95% confidence intervals. Dashed lines indicate no difference.
Figure 6: Comparison between naïve human accuracy and model accuracy under interactive limited feedback, with best-fit linear regressions. Shaded regions indicate 95% confidence intervals. Dashed lines indicate perfect calibration ( y=x ).
Figure 7: Matcher accuracy as a function of the proportion of metaphorical and holistic spans in descriptions.
Figure 8: Matcher accuracy for frontier models under interactive limited feedback.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: The image that was presented to the vision–language models, containing all 12 tangram shapes with their letter labels.
Figure 10: Example of the in-context trials presented to a model, with the User providing the trial text, the Assistant providing a guess, and the User responding with the correctness of the guess.
Figure 11: Matcher accuracy for the shuffled, backward, random, and no context conditions across all matcher types (both human and model), with best-fit LOESS curves, shown by repetition number from the original game. Error bars indicate bootstrapped 95% confidence intervals. Dashed lines indicate the chance level (0.083).
r
Condition
Human split-half
Qwen 3 VL
Gemma 3
Llama 3.2
Molmo 2
Kimi VL
Yoked
.42[.32,.50]
.08
−.07
−.02
.46
.19
Backward
.48[.40,.55]
.20
−.07
.02
.40
.19
Shuffled
–
.23
.02
.16
.47
.31
Random
–
.23
.04
.11
.59
.29
Appendix
Table 2: Human split-half correlations and model–human correlations for the yoked, shuffled, backward, and random conditions. Human split-half correlations show the mean and 95% confidence interval over 1000 random splits.
Figure 12: Item-wise variation in accuracy across matchers and conditions. Error bars indicate bootstrapped 95% confidence intervals. Dashed lines indicate the chance level (0.083).
Figure 13: Correlation in confusion matrices across conditions for all models.
Figure 14: Centred discrepancy in model versus human accuracies in the random condition for selected lemmas, categorised by semantic categories. Error bars indicate 95% confidence intervals. Dashed line indicates zero (i.e., discrepancy equal to the grand mean). Empty/filled points indicate whether the confidence intervals do/do not include zero respectively.
Figure 15: (A) Matcher accuracies and (B) likelihood of perseverating on their previous answer for practice conditions. Error bars indicate bootstrapped 95% confidence intervals. Dashed lines indicate the chance level (0.083).
Figure 16: Proportion of invalid responses for frontier models under interactive limited feedback.
Pragmatic reasoning in language models studies how models infer and communicate meanings shaped by context, intent, shared knowledge, and social norms.
We investigate the separation of literal interpretation from contextual inference in a collaborative block-building tasks, where an agent must resolve underspecified instructions using context. We adapt an existing two-speaker psycholinguistic paradigm into an interactive benchmark called Build What I Mean (BWIM). This setup contrasts a pragmatically cooperative speaker with one who is only literally reliable. In BWIM, models face underspecified instructions and must choose between making a contextual inference or requesting clarification at a small communication cost. Evaluating several state-of-the-art LLMs, we find a clear dissociation between judgment and action. Although models successfully detect speaker unreliability in explicit confidence ratings, they fail to leverage this awareness when taking action. Instead of deploying efficient clarification strategies, models default to suboptimal behaviors. These include partner-blind over-clarification and question-averse guessing under uncertainty. BWIM provides a controlled environment to evaluate online partner adaptation and contextual reasoning in interactive settings.
Natalia Bila, Kata Naszádi, Alexandra Mayn +1
Department of Language Science and Technology, Saarland University · Language Technology Lab, University of Amsterdam
Human conversation relies heavily on conversational implicature, in which speakers convey meanings that are suggested rather than explicitly stated. Although recent large language models exhibit strong conversational fluency, they remain unreliable when interpretation depends on reasoning that integrates social and contextual cues, a process rarely articulated in text. We introduce DRinQ, a benchmark for evaluating pragmatic reasoning about conversational implicature in question utterances, designed to isolate pragmatic variation while holding each question's surface form fixed. To support scalable evaluation, we propose a semi-automated pipeline that produces question-context-interpretation instances with systematic variation. Across evaluations, we find a consistent generation-inference asymmetry: while state-of-the-art models can generate plausible pragmatic scenarios when guided, they often fail to recover the intended implication at inference time. For smaller models, structured prompting improves alignment with human judgments. A comparative writing study further reveals complementary strengths: human authors tend to produce safer, predictable contexts, whereas models generate varied scenarios with interpretations that sometimes exceed contextual support. These findings highlight persistent challenges in modeling conversational implicature and motivate more context-sensitive evaluation frameworks.
In collaborative dialogue, shared perception does not guarantee shared interpretation. Mutual understanding must be established through interaction. We investigate whether vision-language models (VLMs) can distinguish what could be shared from what has been shared between dialogue participants through grounding. We formulate this as an interpretation-matching task on 13,077 annotated reference expressions from HCRC MapTask dialogues, and evaluate VLMs under systematically controlled manipulations of dialogue context and map-information access. Our results show that providing authentic map images improves overall performance but shifts models toward over-predicting alignment. Textual descriptions of the same map content reproduce this bias, while non-informative images suppress alignment predictions entirely, indicating that the bias is driven by task-relevant map content, not the visual channel. This improvement comes at the cost of degraded accuracy on non-aligned cases. Calibration analysis and reference-chain tracking further suggest that models rely on static referential cues on the maps rather than tracking how grounding unfolds through dialogue history. We observe these patterns most clearly in Qwen3-VL-8B-Instruct and, to varying degrees, in four additional models from two architecture families. In models that exhibit the bias, map content, whether presented visually or textually, is treated as evidence of mutual understanding, conflating potential with established common ground.