cs.CLNov 19, 2026

PILAPS: A Bilingual Benchmark for Evaluating Pragmatic Inference across Phenomena and Settings

Authors: Jiani Cao, Anna Korhonen, Chen Cecilia Liu

Organizations: Department of Theoretical and Applied Linguistics, University of Cambridge · Language Technology Lab, University of Cambridge

Abstract

Recent work reports strong pragmatic performance by large language models (LLMs) in English, yet whether this performance generalises across languages and evaluation settings remains unclear. This question is increasingly important as LLMs are deployed across languages, cultures, and conversational contexts. In this paper, we first introduce PILAPS, a bilingual English/Mandarin Chinese benchmark covering eight pragmatic phenomena across purpose-written and dialogue-derived items, with both culturally adapted and independently sourced data. We then evaluate eight LLMs from four families across a range of model sizes and prompting strategies. Our results reveal substantial variation across models and pragmatic phenomena: open-source models continue to struggle particularly with humour, while cross-linguistic performance varies with model origin. Our error analysis shows that, compared with human participants, models are more prone to literal interpretations in conversational contexts. Theory-of-Mind prompting lowers the literal share of errors, whereas Chain-of-Thought prompting slightly raises it. Together, these findings demonstrate that strong pragmatic performance in one evaluation setting does not necessarily generalise across languages or conversational contexts.

Figures & tables

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. VakyArth: Evaluating Pragmatic Competence in LLMs across Indic Languages

    Sep 1, 2026Usneek Singh, Poorvaja Veera Balaji Kumar, Parth Nanda +4Multilingual Language Model EvaluationLLM Evaluation

  2. Tracing the ongoing emergence of human-like reasoning in Large Language Models

    May 20, 2026Paolo Morosi, Nikoleta Pantelidou, Fritz Günther +2Logical ReasoningPragmatic Reasoning in Language Models