You Really Didn't Get That? Benchmarking Social Pragmatic Inference for Indirect and Playful Chinese Online Comments

Shiwei Hong; Junjie Ma; Emma Jiren Wang; Ethan Z. Rong; Siying Hu; Haichang Li; Ziying Wang; Zhicong Lu

cs.CLSep 3, 2026

You Really Didn't Get That? Benchmarking Social Pragmatic Inference for Indirect and Playful Chinese Online Comments

Authors: Shiwei Hong, Junjie Ma, Emma Jiren Wang, Ethan Z. Rong, Siying Hu, Haichang Li, Ziying Wang, Zhicong Lu

Organizations: George Mason University · Virginia Tech · University of Toronto · City University of Hong Kong ·

Abstract

Chinese online comments often convey social meaning through indirect and playful language that is hard to interpret without context. Existing evaluations largely organize items around predefined phenomena or controlled pragmatic categories, leaving open whether models can distinguish plausible readings of what a naturally occurring comment is doing in a particular exchange. We introduce a benchmark for evaluating whether LLMs can recover such situated pragmatic meanings. From more than 200,000 public Chinese social media interaction records, we construct 4,735 human-validated diagnostic items, each pairing a target comment with reconstructed preceding context and plausible misreadings. We evaluate eight LLMs as both question writers and solvers in a cross-writer setting. The task is challenging: the strongest model achieves 81.42% leave-writer-out accuracy. Across all eight models, the mean leave-writer-out accuracy is 68.70% while human accuracy was 90.8%. Case analysis shows that models often recognize broad irony or playfulness while misidentifying the mechanism or interactional move.

Figures & tables

Figure 2: Benchmark construction and cross-writer evaluation workflow. Public interaction records are reconstructed into contextual episodes, converted into model-written diagnostic questions, validated by humans, and evaluated in a cross-writer setting. LWO denotes leave-writer-out evaluation.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Background	Most familiar platforms
A1	HCI and NLP background	Zhihu, Xiaohongshu, Douban
A2	HCI background	Xiaohongshu, Tieba
A3	HCI background	Xiaohongshu, Douban
A4	HCI background	Xiaohongshu
A5	HCI background	Zhihu, Douban, Xiaohongshu, Tieba
A6	HCI background	Xiaohongshu

Appendix

Table 2: Annotator background and most familiar source platforms. All annotator identifiers are anonymized.

ID	Preference labeling	Episode screening	Diagnostic question review
A1	Approx. 1,000 of 5,000	Lead adjudication and flagged case discussion	Items 1 to 1,000
A2	Approx. 1,000 of 5,000	Approx. one sixth of 12,000 plus candidates	Items 1,001 to 2,000
A3	Approx. 1,000 of 5,000	Approx. one sixth of 12,000 plus candidates	Items 2,001 to 3,000
A4	Approx. 1,000 of 5,000	Approx. one sixth of 12,000 plus candidates	Items 3,001 to 4,000
A5	Approx. 1,000 of 5,000	Approx. one sixth of 12,000 plus candidates	Items 4,001 to 5,000
A6	Not assigned	Approx. one sixth of 12,000 plus candidates	Items 5,001 to 6,000

Appendix

Table 3: Annotator workload across preference labeling, episode screening, and diagnostic question review. Episode screening was divided approximately evenly across A2 to A7 after classifier assisted triage.

Solver	Full context	Comment only	Drop
GPT 5.5	62.95	28.40	34.55
Claude Opus 4.7	59.70	26.79	32.91
DeepSeek V4 Pro	71.77	31.20	40.57
Qwen 3.5 9B	81.42	33.58	47.84
Kimi K2.6	63.25	24.91	38.34
MiniMax M2.5	65.89	29.69	36.20

Appendix

Table 4: Full-context versus comment-only LWO accuracy (percent). Drop is an absolute percentage-point difference.

	Remove post-level		Remove direct parent		Remove local interaction
Solver	Net loss	% of 4,735	Net loss	% of 4,735	Net loss	% of 4,735
GPT 5.5	302	6.38	65	1.37	191	4.0
Llama 4 Maverick	351	7.41	56	1.18	193	4.1
Qwen 3.5 9B	289	6.10	49	1.03	230	4.9

Appendix

Table 5: Paired leave-one-component-out sensitivity. Net loss is the decrease in correct answers relative to the corresponding full-context predictions on the same 4,735 items.

Pragmatic Reasoning in Language Models

Pragmatic reasoning in language models studies how models infer and communicate meanings shaped by context, intent, shared knowledge, and social norms.

See all 94 papers

Social Meaning in Large Language Models: Structure, Magnitude, and Pragmatic Prompting
Apr 2, 2026Roland MühlenberndPragmatic Reasoning in Language Models Language Model Calibration
Leibniz-Centre General Linguistics, Berlin, Germany
Unveiling the Limits of Large Language Models in Inferring Pragmatic Meaning from Non-Verbal Responses
Jun 1, 2026Sugyeong Eo, Heuiseok LimLLM Evaluation Pragmatic Reasoning in Language Models
Department of Software, Yonsei University Mirae Campus, Republic of Korea · Department of Computer Science and Engineering, Korea University, Republic of Korea
Polite but Misaligned: Evaluating LLM Politeness Judgments Against Human Pragmatic Norms
Sep 24, 2026Rong Wang, Kun Sun, Yadong GuoLLM Evaluation Pragmatic Reasoning in Language Models
University of Tübingen · Tongji University

Apr 2, 2026J/K move · Enter open · S save

Social Meaning in Large Language Models: Structure, Magnitude, and Pragmatic Prompting

Roland Mühlenbernd

Leibniz-Centre General Linguistics, Berlin, Germany

Pragmatic Reasoning in Language Models Language Model Calibration LLM Prompting

Open paper AskPDF arXiv

Large language models (LLMs) increasingly exhibit human-like patterns of pragmatic and social reasoning. This paper addresses two related questions: do LLMs approximate human social meaning not only qualitatively but also quantitatively, and can prompting strategies informed by pragmatic theory improve this approximation? To address the first, we introduce two calibration-focused metrics distinguishing structural fidelity from magnitude calibration: the Effect Size Ratio (ESR) and the Calibration Deviation Score (CDS). To address the second, we derive prompting conditions from two pragmatic assumptions: that social meaning arises from reasoning over linguistic alternatives, and that listeners infer speaker knowledge states and communicative motives. Applied to a case study on numerical (im)precision across three frontier LLMs, we find that all models reliably reproduce the qualitative structure of human social inferences but differ substantially in magnitude calibration. Prompting models to reason about speaker knowledge and motives most consistently reduces magnitude deviation, while prompting for alternative-awareness tends to amplify exaggeration. Combining both components is the only intervention that improves all calibration-sensitive metrics across all models, though fine-grained magnitude calibration remains only partially resolved. LLMs thus capture inferential structure while variably distorting inferential strength, and pragmatic theory provides a useful but incomplete handle for improving that approximation.

Abstract

Pragmatic Reasoning in Language Models

Explore similar work

Social Meaning in Large Language Models: Structure, Magnitude, and Pragmatic Prompting

Unveiling the Limits of Large Language Models in Inferring Pragmatic Meaning from Non-Verbal Responses

Polite but Misaligned: Evaluating LLM Politeness Judgments Against Human Pragmatic Norms