Human communication is fundamentally creative, and often makes use of subtext -- implied meaning that goes beyond the literal content of the text. Here, we systematically study whether language models can use subtext in communicative settings, and introduce four new evaluation suites to assess these capabilities. Our evaluation settings range from writing & interpreting allegories to playing multi-agent and multi-modal games inspired by the rules of board games like Dixit. We find that frontier models generally exhibit a strong bias towards overly literal, explicit communication, and thereby fail to account for nuanced constraints -- even the best performing models generate literal clues 60% of times in one of our environments -- Visual Allusions. However, we find that some models can sometimes make use of common ground with another party to help them communicate with subtext, achieving 30%-50% reduction in overly literal clues; but they struggle at inferring presence of a common ground when not explicitly stated. For allegory understanding, we find paratextual and persona conditions to significantly shift the interpretation of subtext. Overall, our work provides quantifiable measures for an inherently complex and subjective phenomenon like subtext and reveals many weaknesses and idiosyncrasies of current LLMs. We hope this research to inspire future work towards socially grounded creative communication and reasoning.
Figures & tables
Figure 1: Evaluating the use of subtext in creative communication in large language models (LLMs). A. A schematic of the use of subtext, in reference to a scene from the 1987 film The Princess Bride as an illustrative example. Both generating and understanding the text rely on implied meaning beyond the explicit tokens. B. Our evaluation covers multi-agent social communication in games inspired by the rules of Dixit and Wavelength (§ 3 ), as well as model generation and understanding of historical allegories and stories (§ 4 ).
Game Score
Win Rate
Component Scores
Clue Types (%)
Model
Storytelling
Guessing
Distraction
Obvious ( ↓ )
Obscure ( ↓ )
Just-Right ( ↑ )
Gemma-3-27B
25.44
0.26
4.26
4.26
1.23
52.31
7.81
39.88
Gemini-2.5-Flash
26.77
0.40
2.74
5.63
1.91
71.81
1.90
26.29
Gemini-2.5-Pro
27.81
0.41
4.09
4.91
1.78
59.28
3.07
37.64
GPT-5-mini
26.02
0.23
2.42
5.32
1.72
75.64
1.81
22.54
GPT-5
26.65
0.23
2.63
6.05
1.86
74.74
0.94
24.33
Table 1: Game scores, win-rates, component scores, and clue-types for different models playing Visual Allusions .
Figure 2: Performance comparison of different models in the shared context setup. Statistical significance for Spark Coefficients (only calculated for Common Ground = True case) is calculated using a chi-squared test ( p=0.05 ) to reject the null hypothesis of no spark.
Average Score
Storytelling Score
Storytelling Clues
Awareness
Model Name
Shared Belief
Gemini-2.5-Pro
True
27.1
5.4
0.962
0.962
False
29.4
6.0
0.830
0.051
GPT-5
True
29.0
4.8
0.927
0.909
False
26.9
1.2
0.203
0.102
Table 2: Manipulating Shared Belief Prior for Gemini-2.5-Pro and GPT-5 players.
Game Score
Win Rate
MindRead
Model
Gemini-2.5-Flash
15.46
0.31
0.30
Gemini-2.5-Pro
14.80
0.40
0.29
GPT-5
17.14
0.57
0.33
GPT-5-mini
17.89
0.66
0.34
Claude-Haiku-4.5
17.14
0.54
0.37
Table 3: Performance of different models on our environment inspired by the rules of the game Wavelength . Each model plays against the other player 7 times. The maximum achievable score in the game is 20.
Persona Cond.
Information Cond.
Gemini-2.5-Flash
Gemini-2.5-Pro
GPT-5-mini
GPT-5
Claude-Haiku-4.5
Claude-Sonnet-4.5
Critic
Default
0.081
0.256
0.111
0.615
0.141
0.448
Author Name
0.089
0.341 †
0.137
0.700 †
0.159
0.491
Reader Name
0.096
0.289
0.107
0.659
0.130
0.506 †
Deep Reader
Default
0.093
0.259
0.052
0.448
0.211
0.450
Author Name
0.096
0.319 †
0.096 †
0.574 †
0.222
0.513 †
Reader Name
0.115
0.293
0.067
0.611 †
0.148 †
0.543 †
Table 4: Model accuracies for decoding the ground truth event. by persona and information conditions. Bold scores indicate an increase over ‘Default’ information condition. † denotes statistical significance according to the paired T-test. More results in Appx. A.6.4 .
Model
Succeeded
Conformed
Strikes
Time to First Strike
Time of Death
Inquisitor-Critic Agreement
Control Author
0.40
8.35
0.30
10.00 +
10.00 +
0.86
Gemini-2.5-Flash
1.20
5.70
1.30
4.70
10.20
0.75
Gemini-2.5-Pro
0.85
1.35
2.90
2.45
6.15
0.74
GPT-5
2.20
5.15
0.85
5.55
10.00 +
0.63
Claude-Sonnet-4.5
1.30
2.60
2.25
1.80
7.40
0.73
Table 5: Aggregate Results on The Aesopian Author environment. The maximum value for all metrics except the Inquisitor-Critic-Agreement is 10. + implies that the average is greater than the value 10. The results are averaged across 20 different game settings.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Component Scores
Clue Types (%)
Model
Memory
Score
Win Rate
Storytelling
Guessing
Distractor
Obvious ( ↓ )
Obscure ( ↓ )
Just-Right ( ↑ )
GPT-5
False
28.30
0.525
1.275
1.425
0.70
92.61
0.50
6.89
True
27.40
0.25
1.35
1.05
0.40
90.12
3.10
6.79
Gemini-Pro
False
26.60
0.20
1.275
1.80
0.875
91.19
1.86
6.95
True
29.95
0.75
3.00
1.95
0.90
81.05
2.14
16.81
Sonnet-4.5
False
26.50
0.25
1.20
2.025
1.075
92.07
1.27
6.65
Appendix
Table 6: Results for different models playing Visual Allusions with and without game memory. The models remember the last 3 rounds of the game when memory is enabled. Results are averaged across 20 rounds.
Game Score
Win Rate
MindRead
Model
Gemini-2.5-Flash
15.46
0.31
0.30
Gemini-2.5-Pro
14.80
0.40
0.29
GPT-5
17.14
0.57
0.33
GPT-5-mini
17.89
0.66
0.34
Claude-Haiku-4.5
17.14
0.54
0.37
Appendix
Table 7: Performance of different models on our environment inspired by the rules of the game Wavelength . Each model plays against the other player 7 times. The maximum achievable score in the game is 20.
Persona Condition
Information Condition
Gemma-3-27B
Gemini-2.5-Flash
Gemini-2.5-Pro
GPT-5-mini
GPT-5
Claude-Haiku-4.5
Claude-Sonnet-4.5
Critic
Default
0.011
0.081
0.256
0.111
0.615
0.141
0.448
Author Name
0.022
0.089
0.341 †
0.137
0.700 †
0.159
0.491
Reader Name
0.015
0.096
0.289
0.107
0.659
0.130
0.506 †
Deep Reader
Default
0.022
0.093
0.259
0.052
0.448
0.211
0.450
Author Name
0.033
0.096
0.319 †
0.096 †
0.574 †
0.222
0.513 †
Reader Name
0.030
0.115
0.293
0.067
0.611 †
0.148 †
0.543 †
Appendix
Table 8: Model accuracies for decoding the ground truth event by persona and information conditions. Bold scores indicate an increase over ‘Default’ information condition for allegories generated using Gemini-2.5-Flash . † denotes statistical significance ( p<0.05 ) according to the paired T-test.
Persona Cond.
Information Cond.
Gemini-2.5-Flash
Gemini-2.5-Pro
GPT-5
Claude-Sonnet-4.5
Critic
Default
0.037
0.222
0.685
0.500
Author Name
0.093
0.352
0.852 †
0.500
Reader Name
0.093
0.333
0.833 †
0.500
Deep Reader
Default
0.056
0.278
0.593
0.500
Author Name
0.185 †
0.426 †
0.815 †
0.722 †
Reader Name
0.151
0.407
0.796 †
0.741 †
Appendix
Table 9: Model accuracies for decoding the ground truth event by persona and information conditions allegories generated using GPT-5 . Bold scores indicate an increase over ‘Default’ information condition. † denotes statistical significance ( p<0.05 ) according to the paired T-test.
Model
Alignment
Num Succeeded
Num Conformed
Strikes Administered
Time to First Strike
Time of Death
Inquisitor-Critic Agreement
Control Author
Misaligned
0.37
6.50
0.75
9.50
10.00 +
0.72
Aligned
0.42
9.58
0.00
10.00 +
10.00 +
0.96
Gemini-2.5-Flash
Misaligned
1.00
4.33
1.33
1.00
10.00 +
0.57
Aligned
1.29
6.29
1.29
6.29
9.86
0.82
Gemini-2.5-Pro
Misaligned
0.25
1.25
3.00
2.38
5.50
0.82
Aligned
1.25
1.42
2.83
2.50
6.58
0.69
Appendix
Table 10: The Aesopian Author aggregate results by author alignment. The superscript + implies that the average is greater than the value 10 which happens in the cases where a strike is not registered during the game.
Figure 3: Qualitative examples of clues generated by different models while playing as storyteller in Visual Allusions (continues on next page).
Figure 4: Qualitative examples of clues generated by different models while playing as storyteller in Visual Allusions .
Figure 5: Qualitative examples of clues generated by different models under the shared context setting while playing as storyteller in Visual Allusions .
Pragmatic reasoning in language models studies how models infer and communicate meanings shaped by context, intent, shared knowledge, and social norms.
Ambiguity is often treated as a bug for AI systems to resolve---but in human communication and culture, ambiguity can also be a generative resource. From humour to politics to art, people express themselves in words and images that are open enough to invite different interpretations, yet constrained enough to be interpretable. We operationalise this notion of calibrated ambiguity with a task drawn from the parlour game Dixit. We compare differences in clues generated by human vs multimodal language models, based on a novel coding rubric for calibrated ambiguity, and find that models consistently exhibit ambiguity collapse (i.e., their outputs are over-specified, leaving no room for multiple legitimate interpretations). Unlike human clues, AI-generated clues also exhibit cultural flattening; they almost never make reference to culturally-situated knowledge, even when prompted to use allusion and figurative language.
Cody Kommers, Mingrui Ye, Evelyn Gius +4
King’s College London London, United Kingdom · University of Edinburgh Edinburgh, United Kingdom
The game of Taboo requires describing a target word without using a set of forbidden words, so that other players can guess it. This deceptively simple task combines strict lexical constraints with the need for communicatively effective descriptions, making it a compelling playground for examining how LLMs navigate competing demands at inference time. We evaluate two open-weight models under conditions that intervene at progressively deeper levels of the generative process, from prompting to generation-time constraints to internal representations manipulations. We assess their outputs through forbidden word violation detection, LLM-as-a-judge measuring the degree to which generated descriptions successfully evoke the target concept for both human and machine guessers, and examining whether the strategies models adopt under constraint align with those of human players. Our results show that compliance with the rules of the game and communicative effectiveness trade off differently across conditions, and that models remain substantially weaker than humans as guessers, suggesting that lexical grounding under constraint is an open challenge for current language models.
Sara Candussio, Francesca Padovani, Daniel Scalena +1
AILab, MIGe, Univeristy of Trieste, Italy · Center for Language and Cognition (CLCG), University of Groningen, The Netherlands · University of Milano - Bicocca, Italy
Large language models (LLMs) increasingly act as agents with little human oversight, so potential mistakes they make can go unnoticed. Users then depend on the model to report what went wrong. An honest model discloses its mistakes, while a deceptive one conceals them. However, it is unclear how current LLMs behave in such situations. In this study, we prefill LLM trajectories with synthetic mistakes. The trajectories resemble real deployments in chat and agentic settings. Models fail to disclose their mistake in 36.4% of chat and 67.1% of agentic rollouts. In 2.4% and 5.3% of rollouts, respectively, they are aware of the mistake in their chain of thought but still deceptively conceal it. Rates vary by model: for instance, Gemini 3.5 Flash knowingly conceals mistakes in up to 19.9% of agentic rollouts. In 11.9% of chat and 51.8% of agentic rollouts, models show no awareness of mistakes, even though they reliably spot them when reviewing the same transcript as an outside observer. Our results show that, as agents take on more tasks with less oversight, users cannot rely on them to self-report possible mistakes. Developers should instead use independent monitors that review agent trajectories, or specifically train models to check their past actions and disclose what they find.
Lucas Florin, Amelie Knecht, Ulysse Schaller +1
AI Safety Research Group, University of Stuttgart, Germany