PILAPS: A Bilingual Benchmark for Evaluating Pragmatic Inference across Phenomena and Settings
Organizations: Department of Theoretical and Applied Linguistics, University of Cambridge · Language Technology Lab, University of Cambridge
Abstract
Recent work reports strong pragmatic performance by large language models (LLMs) in English, yet whether this performance generalises across languages and evaluation settings remains unclear. This question is increasingly important as LLMs are deployed across languages, cultures, and conversational contexts. In this paper, we first introduce PILAPS, a bilingual English/Mandarin Chinese benchmark covering eight pragmatic phenomena across purpose-written and dialogue-derived items, with both culturally adapted and independently sourced data. We then evaluate eight LLMs from four families across a range of model sizes and prompting strategies. Our results reveal substantial variation across models and pragmatic phenomena: open-source models continue to struggle particularly with humour, while cross-linguistic performance varies with model origin. Our error analysis shows that, compared with human participants, models are more prone to literal interpretations in conversational contexts. Theory-of-Mind prompting lowers the literal share of errors, whereas Chain-of-Thought prompting slightly raises it. Together, these findings demonstrate that strong pragmatic performance in one evaluation setting does not necessarily generalise across languages or conversational contexts.
Figures & tables
| Source | Cross-lingual | |||||
|---|---|---|---|---|---|---|
| Benchmark | Lang. | Phen. | Constructed | Derived | Adapted | Native |
| PragMega ( Floyd et al., 2023 ) | 1 | 7 | ✓ | ✗ | – | – |
| MultiPragEval ( Park et al., 2024 ) | 4 | 4 | ✓ | ✗ | ✓ | ✗ |
| PUB ( Sravanthi et al., 2024 ) | 1 | 4 | ✓ | ✗ | – | – |
| SwordsmanImp ( Yue et al., 2024 ) | 1 | 1 | ✗ | ✓ | – | – |
| FLUID QA ( Park et al., 2025 ) | 3 | 4 | ✓ | ✗ | ✓ | ✗ |
| Phenomenon | Definition | What the task tests | Num. Options |
|---|---|---|---|
| Coherence | Sentences that fit together into a sensible story | Judging whether two sentences form a coherent story | 2 |
| Deceit | Misleading someone in order to achieve a hidden goal | Recognising an intention to mislead | 4 |
| Humour | A joke ending that is surprising but still fitting | Identifying the intended humorous continuation | 5 |
| Indirect Speech | Expressing a request or intention without stating it outright | Recognising the intended meaning of an indirect utterance | 4 |
| Irony | Saying the opposite of what one actually means | Recognising that a speaker means the opposite of the literal utterance | 4 |
| Maxims | Breaking a conversational rule to imply extra meaning | Inferring meaning from a flouted Gricean maxim | 4 |
| Constructed | Dialogue-derived | |||||||||||||
| Model | Coh | Dec | Hum | Ind | Iro | Max | Met | Avg | Hum | Iro | Max | Imp | Avg | Mean |
| English Dataset | ||||||||||||||
| DeepSeek-V4-Flash | 80.83 | 93.33 | 93.33 | 93.33 | 80.00 | 85.00 | 90.00 | 87.98 | 61.67 | 85.00 | 65.00 | 65.83 | 69.37 | 81.21 |
| Qwen3-32B | 79.17 | 90.00 | 73.33 | 80.00 | 77.33 | 73.33 | 85.00 | 79.74 | 50.00 | 50.00 | 51.67 | 72.84 | 56.13 | 71.15 |
| Qwen3-8B | 71.67 | 65.00 | 56.67 | 85.00 | 52.00 | 56.67 | 76.67 | 66.24 | 28.33 | 46.67 | 43.33 | 66.99 | 46.33 | 59.00 |
| Qwen3-4B | 79.17 | 71.67 | 45.00 | 80.00 | 49.33 | 51.67 | 66.67 | 63.36 | 40.00 | 40.00 | 43.33 | 75.07 | 49.60 | 58.35 |
| Task | Lit. | W-prag. | Assoc. | N-seq. | ||
|---|---|---|---|---|---|---|
| EN | Humour | Models | 37.50 | – | 26.79 | 35.71 |
| Humans | 20.59 | – | 20.59 | 58.82 | ||
| Irony | Models | 54.34 | – | 42.47 | 3.20 | |
| Humans | 44.83 | – | 51.72 | 3.45 | ||
| Maxims | Models | 65.20 | 24.67 | 10.13 | – | |
| Humans | 59.52 | 26.19 | 14.29 | – |
| Model | Vanilla | CoT | ToM |
|---|---|---|---|
| DeepSeek-V4-Flash | 81.21 / 86.18 | 82.87 / 83.61 | 84.90 / 86.61 |
| Qwen3-32B | 71.15 / 75.29 | 73.67 / 75.66 | 74.56 / 78.77 |
| Qwen3-8B | 59.00 / 62.32 | 58.05 / 59.27 | 58.21 / 67.18 |
| Qwen3-4B | 58.35 / 59.05 | 57.12 / 56.75 | 59.60 / 58.23 |
| Gemini-3.5-Flash | 93.46 / 92.87 | 93.33 / 92.22 | 91.82 / 91.45 |
| Llama-3.3-70B | 66.76 / 54.33 | 67.39 / 57.48 | 70.56 / 60.82 |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Dimension | What it Captures | Indicators Considered |
|---|---|---|
| Fluency | Naturalness and readability of the Chinese translation | Idiomatic word choice and everyday usage; grammatical correctness; appropriate tone and register; smooth flow; absence of translationese |
| Adequacy | Preservation of the informational and pragmatic content of the English source, with culturally appropriate substitution permitted | Localised personal names and culturally specific expressions; sufficient contextual cues and speaker-intention information retained |
| Compliance | Adherence to the structural and task-specific requirements of the original item | Multiple-choice format preserved; distractor categories and their interpretive functions maintained; relationship between the target utterance and each option unchanged |
| Lang. | Question and Response | Answer |
|---|---|---|
| EN | Are you a morning person? / I’m more of a night owl. | No |
| ZH | 你是早起的人吗?/我更像一只夜猫子。 | 不是 |
| EN | Are you sad about losing your job? / It turned out to be a blessing in disguise. | No |
| ZH | 你为失去工作而难过吗?/事实证明,这也是因祸得福。 | 没有 |
| Task | Scenario and Question | Answer Options |
|---|---|---|
| Coherence | The lights have been on since last night. The car doesn’t start. | (1) Incoherent; (2) Coherent |
| Deceit | Peter’s mother wants them to start eating healthier and tells him that they will eat boiled sprouts tonight. Peter tells his mother: “I don’t think I can eat, I have a terrible stomach-ache.” Why has Peter responded in such a way? | (1) He does not want to get in trouble with mom for eating boiled sprouts. [wrong-pragmatic] ; (2) He is experiencing stomach pain. [literal] ; (3) He wants to avoid eating boiled sprouts. [correct] ; (4) He wants to have boiled sprouts every night. [wrong-pragmatic] |
| Humour | A woman was telling her friend, “I made my husband a millionaire.” The friend replies, “And what was he before you married him?” The woman replies: | (1) “A lonely guy.” [literal] ; (2) “Watch out, the wind is blowing away your wig!” [non-sequitur] ; (3) “A multi-millionaire.” [correct] ; (4) “We have been married for 10 years.” [associative] ; (5) “My favorite drink is a martini.” [non-sequitur] |
| Indirect Speech | The boys were playing around. Dan accidentally tripped Scott. Scott got very angry. Dan said: “Everybody’s being wild.” What is he trying to convey? | (1) It was not his fault. [correct] ; (2) He likes to live in the city. [associative] ; (3) He likes to live in the wilderness. [associative] ; (4) All of the boys were careless. [literal] |
| Irony | John finished the race way ahead of the other competitors. His friend laughed and said to him, “You are so slow!” What did his friend want to convey? | (1) John is an outstanding runner. [correct] ; (2) John needs new running shoes. [associative] ; (3) John is not a good runner. [literal] ; (4) John needs a new computer. [non-sequitur] |
| Maxims | Lenny is saying goodbye to his date Bianka. Lenny asks Bianka: “Where do you live?” Bianka responds: “I live on the moon.” Why has Bianka responded like this? | (1) Bianka does not want Lenny to know her address. [correct] ; (2) Bianka is an alien. [literal] ; (3) Bianka is not educated. [wrong-pragmatic] ; (4) Bianka tries to make Lenny angry with her. [associative] |
| Task | Scenario and Question | Answer Options |
|---|---|---|
| Humour | Jake just broke up with his girlfriend. Daniel: Look on the bright side. Jake: What’s the bright side? Daniel: … | (1) I once spilled my drink all over a date. [non-sequitur] ; (2) You will find someone better soon. [literal] ; (3) At least you two never moved in together. [associative] ; (4) Only nine more months to Comic-Con. [correct] ; (5) Breakups are always quite hard at first. [non-sequitur] |
| Irony | Alex: Is there something wrong with your neck? Ben: It’s a little stiff. Alex: Have you considered massage? Ben: Yes, I relish the thought of a stranger covering my body with oil and rubbing it. What did Ben want to convey by the last sentence? | (1) Ben thinks massage therapy is an underrated profession. [associative] ; (2) Ben prefers to use a heating pad instead. [non-sequitur] ; (3) Ben finds the idea of being massaged very unappealing. [correct] ; (4) Ben is enthusiastic about getting a massage. [literal] |
| Maxims | Mark: Oh, Tom, there you are, I’m ready to go home. Tom: I just got here. Mark: Good, perfect timing. Why has Mark responded like this? | (1) Mark is pleased about the coincidence of their timing. [literal] ; (2) Mark thinks that Tom should have arrived earlier to make the most of the evening. [wrong-pragmatic] ; (3) Mark is congratulating Tom for arriving in time. [associative] ; (4) Mark is implying that Tom can now take him home straight away. [correct] |
| Model | Origin | Params. | Scale | Access |
|---|---|---|---|---|
| DeepSeek-V4-Flash | Chinese | 284B | Large | API |
| Qwen3-32B | Chinese | 32B | Large | HF |
| Qwen3-8B | Chinese | 8B | Small | HF |
| Qwen3-4B | Chinese | 4B | Small | HF |
| Gemini-3.5-Flash | English | – | Large | API |
| Llama-3.3-70B | English | 70B | Large | HF |
| Model Held Out | ||
|---|---|---|
| DeepSeek-V4-Flash | ||
| Qwen3-32B | ||
| Qwen3-8B | ||
| Qwen3-4B | ||
| Gemini-3.5-Flash | ||
| Llama-3.3-70B |
| Accuracy | vs. Vanilla | |||||
|---|---|---|---|---|---|---|
| Setting | Lang. | Vanilla | CoT | ToM | CoT | ToM |
| Constructed | EN | 73.59 | 74.54 | 75.26 | ||
| ZH | 69.32 | 68.60 | 70.48 | |||
| Extended | EN | 49.58 | 48.26 | 52.85 | ||
| ZH | 47.85 | 49.44 | 53.75 | |||
| Brief | EN | 70.44 | 76.09 | 71.94 | ||
| Method | v0 | v1 | v2 |
|---|---|---|---|
| Vanilla | 65.18 | 64.99 | 65.26 |
| CoT | 65.79 | 66.35 | 64.84 |
| ToM | 66.84 | 66.83 | 68.45 |
| Constructed | Dialogue-derived | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Coh | Dec | Hum | Ind | Iro | Max | Met | Avg | Hum | Iro | Max | Imp | Avg | Mean |
| English Dataset | ||||||||||||||
| Tiny-Aya-Water | 70.00 | 46.67 | 20.00 | 60.00 | 45.33 | 41.67 | 58.33 | 48.86 | 15.00 | 23.33 | 60.00 | 65.04 | 40.84 | 45.94 |
| Ministral3-3B-IT | 48.33 | 61.67 | 25.00 | 65.00 | 49.33 | 45.00 | 65.00 | 51.33 | 15.00 | 31.67 | 53.33 | 52.92 | 38.23 | 46.57 |
| Gemma4-E4B-IT | 68.33 | 41.67 | 30.00 | 51.67 | 46.67 | 36.67 | 53.33 | 46.91 | 28.33 | 21.67 | 38.33 | 76.74 | 41.27 | 44.86 |
| SmolLM3-3B | 60.00 | 60.00 | 30.00 | 66.67 | 50.67 | 40.00 | 48.33 | 50.81 | 23.33 | 30.00 | 38.33 | 70.47 | 40.53 | 47.07 |
| Task | EN | ZH |
|---|---|---|
| Humour | 66.00 | 66.50 |
| Irony | 85.00 | 90.00 |
| Maxims | 79.00 | 91.00 |
| Macro-average | 76.67 | 82.50 |
| Family | Labels |
|---|---|
| Literal | IncorrectLiteral, IncorrectStraightforward, IncorrectPlausibleLiteral |
| Wrong-pragmatic | IncorrectNonLiteral, IncorrectNonLiteralLexicalOverlap, IncorrectNonLiteralSocialConvention |
| Associative | IncorrectAssociate, IncorrectAssociative, IncorrectLexicalOverlap, AssociativeNonSequitur |
| Non-sequitur | IncorrectNonSequitur, FunnyNonSequitur, NeutralNonSequitur |
| Prompt | Lit | W-prag | Assoc | N-seq | Acc | |
|---|---|---|---|---|---|---|
| EN | Vanilla | 78.92 | 7.26 | 6.65 | 7.17 | 73.48 |
| CoT | 80.43 | 8.75 | 4.93 | 5.89 | 74.64 | |
| ToM | 73.05 | 10.02 | 7.41 | 9.52 | 76.22 | |
| ZH | Vanilla | 65.95 | 11.31 | 11.54 | 11.20 | 68.76 |
| CoT | 66.97 | 13.05 | 10.66 | 9.32 | 68.39 | |
| ToM | 60.35 | 14.98 | 11.98 | 12.69 | 70.72 |
| Constructed | Extended dialogue | ||||||||||
| Model | Lit | W-prag | Assoc | N-seq | n | Lit | W-prag | Assoc | N-seq | n | |
| English Dataset | |||||||||||
| DeepSeek-V4-Flash | 85.56 | 4.17 | 3.33 | 6.94 | 42 | 49.16 | 14.29 | 19.81 | 16.75 | 53 | |
| Qwen3-32B | 75.00 | 11.46 | 2.08 | 11.46 | 76 | 53.10 | 6.90 | 23.33 | 16.67 | 89 | |
| Qwen3-8B | 84.16 | 1.28 | 9.62 | 4.95 | 132 | 60.60 | 1.96 | 27.36 | 10.08 | 109 | |
| Qwen3-4B | 79.49 | 5.88 | 8.57 | 6.06 | 149 | 58.61 | 8.82 | 22.39 | 10.19 | 106 | |
| English | Chinese | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Task | Lit. | W-prag. | Assoc. | N-seq. | Lit. | W-prag. | Assoc. | N-seq. | ||
| Deceit | 76.25 | 23.75 | – | – | 80 | 61.29 | 38.71 | – | – | 124 |
| Humour | 47.85 | – | 15.59 | 36.56 | 186 | 28.14 | – | 15.08 | 56.78 | 199 |
| Indirect Speech | 95.31 | – | 4.69 | – | 64 | 86.96 | – | 13.04 | – | 92 |
| Irony | 93.69 | – | 5.83 | 0.49 | 206 | 84.95 | – | 12.14 | 2.91 | 206 |
| Maxims | 72.37 | 13.82 | 13.82 | – | 152 | 61.00 | 10.00 | 29.00 | – | 200 |
Pragmatic Reasoning in Language Models
Pragmatic reasoning in language models studies how models infer and communicate meanings shaped by context, intent, shared knowledge, and social norms.
See all 109 papers