Human communication is fundamentally creative, and often makes use of subtext -- implied meaning that goes beyond the literal content of the text. Here, we systematically study whether language models can use subtext in communicative settings, and introduce four new evaluation suites to assess these capabilities. Our evaluation settings range from writing & interpreting allegories to playing multi-agent and multi-modal games inspired by the rules of board games like Dixit. We find that frontier models generally exhibit a strong bias towards overly literal, explicit communication, and thereby fail to account for nuanced constraints -- even the best performing models generate literal clues 60% of times in one of our environments -- Visual Allusions. However, we find that some models can sometimes make use of common ground with another party to help them communicate with subtext, achieving 30%-50% reduction in overly literal clues; but they struggle at inferring presence of a common ground when not explicitly stated. For allegory understanding, we find paratextual and persona conditions to significantly shift the interpretation of subtext. Overall, our work provides quantifiable measures for an inherently complex and subjective phenomenon like subtext and reveals many weaknesses and idiosyncrasies of current LLMs. We hope this research to inspire future work towards socially grounded creative communication and reasoning.
Figures & tables
Figure 1: Evaluating the use of subtext in creative communication in large language models (LLMs). A. A schematic of the use of subtext, in reference to a scene from the 1987 film The Princess Bride as an illustrative example. Both generating and understanding the text rely on implied meaning beyond the explicit tokens. B. Our evaluation covers multi-agent social communication in games inspired by the rules of Dixit and Wavelength (§ 3 ), as well as model generation and understanding of historical allegories and stories (§ 4 ).
Game Score
Win Rate
Component Scores
Clue Types (%)
Model
Storytelling
Guessing
Distraction
Obvious ( ↓ )
Obscure ( ↓ )
Just-Right ( ↑ )
Gemma-3-27B
25.44
0.26
4.26
4.26
1.23
52.31
7.81
39.88
Gemini-2.5-Flash
26.77
0.40
2.74
5.63
1.91
71.81
1.90
26.29
Gemini-2.5-Pro
27.81
0.41
4.09
4.91
1.78
59.28
3.07
37.64
GPT-5-mini
26.02
0.23
2.42
5.32
1.72
75.64
1.81
22.54
GPT-5
26.65
0.23
2.63
6.05
1.86
74.74
0.94
24.33
Table 1: Game scores, win-rates, component scores, and clue-types for different models playing Visual Allusions .
Figure 2: Performance comparison of different models in the shared context setup. Statistical significance for Spark Coefficients (only calculated for Common Ground = True case) is calculated using a chi-squared test ( p=0.05 ) to reject the null hypothesis of no spark.
Average Score
Storytelling Score
Storytelling Clues
Awareness
Model Name
Shared Belief
Gemini-2.5-Pro
True
27.1
5.4
0.962
0.962
False
29.4
6.0
0.830
0.051
GPT-5
True
29.0
4.8
0.927
0.909
False
26.9
1.2
0.203
0.102
Table 2: Manipulating Shared Belief Prior for Gemini-2.5-Pro and GPT-5 players.
Game Score
Win Rate
MindRead
Model
Gemini-2.5-Flash
15.46
0.31
0.30
Gemini-2.5-Pro
14.80
0.40
0.29
GPT-5
17.14
0.57
0.33
GPT-5-mini
17.89
0.66
0.34
Claude-Haiku-4.5
17.14
0.54
0.37
Table 3: Performance of different models on our environment inspired by the rules of the game Wavelength . Each model plays against the other player 7 times. The maximum achievable score in the game is 20.
Persona Cond.
Information Cond.
Gemini-2.5-Flash
Gemini-2.5-Pro
GPT-5-mini
GPT-5
Claude-Haiku-4.5
Claude-Sonnet-4.5
Critic
Default
0.081
0.256
0.111
0.615
0.141
0.448
Author Name
0.089
0.341 †
0.137
0.700 †
0.159
0.491
Reader Name
0.096
0.289
0.107
0.659
0.130
0.506 †
Deep Reader
Default
0.093
0.259
0.052
0.448
0.211
0.450
Author Name
0.096
0.319 †
0.096 †
0.574 †
0.222
0.513 †
Reader Name
0.115
0.293
0.067
0.611 †
0.148 †
0.543 †
Table 4: Model accuracies for decoding the ground truth event. by persona and information conditions. Bold scores indicate an increase over ‘Default’ information condition. † denotes statistical significance according to the paired T-test. More results in Appx. A.6.4 .
Model
Succeeded
Conformed
Strikes
Time to First Strike
Time of Death
Inquisitor-Critic Agreement
Control Author
0.40
8.35
0.30
10.00 +
10.00 +
0.86
Gemini-2.5-Flash
1.20
5.70
1.30
4.70
10.20
0.75
Gemini-2.5-Pro
0.85
1.35
2.90
2.45
6.15
0.74
GPT-5
2.20
5.15
0.85
5.55
10.00 +
0.63
Claude-Sonnet-4.5
1.30
2.60
2.25
1.80
7.40
0.73
Table 5: Aggregate Results on The Aesopian Author environment. The maximum value for all metrics except the Inquisitor-Critic-Agreement is 10. + implies that the average is greater than the value 10. The results are averaged across 20 different game settings.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Component Scores
Clue Types (%)
Model
Memory
Score
Win Rate
Storytelling
Guessing
Distractor
Obvious ( ↓ )
Obscure ( ↓ )
Just-Right ( ↑ )
GPT-5
False
28.30
0.525
1.275
1.425
0.70
92.61
0.50
6.89
True
27.40
0.25
1.35
1.05
0.40
90.12
3.10
6.79
Gemini-Pro
False
26.60
0.20
1.275
1.80
0.875
91.19
1.86
6.95
True
29.95
0.75
3.00
1.95
0.90
81.05
2.14
16.81
Sonnet-4.5
False
26.50
0.25
1.20
2.025
1.075
92.07
1.27
6.65
Appendix
Table 6: Results for different models playing Visual Allusions with and without game memory. The models remember the last 3 rounds of the game when memory is enabled. Results are averaged across 20 rounds.
Game Score
Win Rate
MindRead
Model
Gemini-2.5-Flash
15.46
0.31
0.30
Gemini-2.5-Pro
14.80
0.40
0.29
GPT-5
17.14
0.57
0.33
GPT-5-mini
17.89
0.66
0.34
Claude-Haiku-4.5
17.14
0.54
0.37
Appendix
Table 7: Performance of different models on our environment inspired by the rules of the game Wavelength . Each model plays against the other player 7 times. The maximum achievable score in the game is 20.
Persona Condition
Information Condition
Gemma-3-27B
Gemini-2.5-Flash
Gemini-2.5-Pro
GPT-5-mini
GPT-5
Claude-Haiku-4.5
Claude-Sonnet-4.5
Critic
Default
0.011
0.081
0.256
0.111
0.615
0.141
0.448
Author Name
0.022
0.089
0.341 †
0.137
0.700 †
0.159
0.491
Reader Name
0.015
0.096
0.289
0.107
0.659
0.130
0.506 †
Deep Reader
Default
0.022
0.093
0.259
0.052
0.448
0.211
0.450
Author Name
0.033
0.096
0.319 †
0.096 †
0.574 †
0.222
0.513 †
Reader Name
0.030
0.115
0.293
0.067
0.611 †
0.148 †
0.543 †
Appendix
Table 8: Model accuracies for decoding the ground truth event by persona and information conditions. Bold scores indicate an increase over ‘Default’ information condition for allegories generated using Gemini-2.5-Flash . † denotes statistical significance ( p<0.05 ) according to the paired T-test.
Persona Cond.
Information Cond.
Gemini-2.5-Flash
Gemini-2.5-Pro
GPT-5
Claude-Sonnet-4.5
Critic
Default
0.037
0.222
0.685
0.500
Author Name
0.093
0.352
0.852 †
0.500
Reader Name
0.093
0.333
0.833 †
0.500
Deep Reader
Default
0.056
0.278
0.593
0.500
Author Name
0.185 †
0.426 †
0.815 †
0.722 †
Reader Name
0.151
0.407
0.796 †
0.741 †
Appendix
Table 9: Model accuracies for decoding the ground truth event by persona and information conditions allegories generated using GPT-5 . Bold scores indicate an increase over ‘Default’ information condition. † denotes statistical significance ( p<0.05 ) according to the paired T-test.
Model
Alignment
Num Succeeded
Num Conformed
Strikes Administered
Time to First Strike
Time of Death
Inquisitor-Critic Agreement
Control Author
Misaligned
0.37
6.50
0.75
9.50
10.00 +
0.72
Aligned
0.42
9.58
0.00
10.00 +
10.00 +
0.96
Gemini-2.5-Flash
Misaligned
1.00
4.33
1.33
1.00
10.00 +
0.57
Aligned
1.29
6.29
1.29
6.29
9.86
0.82
Gemini-2.5-Pro
Misaligned
0.25
1.25
3.00
2.38
5.50
0.82
Aligned
1.25
1.42
2.83
2.50
6.58
0.69
Appendix
Table 10: The Aesopian Author aggregate results by author alignment. The superscript + implies that the average is greater than the value 10 which happens in the cases where a strike is not registered during the game.
Figure 3: Qualitative examples of clues generated by different models while playing as storyteller in Visual Allusions (continues on next page).
Figure 4: Qualitative examples of clues generated by different models while playing as storyteller in Visual Allusions .
Figure 5: Qualitative examples of clues generated by different models under the shared context setting while playing as storyteller in Visual Allusions .
Pragmatic reasoning in language models studies how models infer and communicate meanings shaped by context, intent, shared knowledge, and social norms.
AILab, MIGe, Univeristy of Trieste, Italy · Center for Language and Cognition (CLCG), University of Groningen, The Netherlands · University of Milano - Bicocca, Italy