Polite but Misaligned: Evaluating LLM Politeness Judgments Against Human Pragmatic Norms

Rong Wang; Kun Sun; Yadong Guo

cs.CLSep 24, 2026

Polite but Misaligned: Evaluating LLM Politeness Judgments Against Human Pragmatic Norms

Authors: Rong Wang, Kun Sun, Yadong Guo

Organizations: University of Tübingen · Tongji University

Abstract

Despite strong performance on standard benchmarks, it remains unclear whether large language models (LLMs) evaluate social pragmatics in ways that align with human judgments. We evaluate LLM politeness judgments using two English-language datasets with complementary annotation formats: continuous human ratings and three-way categorical labels. Across the seven evaluated models, we find that inter-model agreement is stronger than model--human agreement. Strategy-level analyses suggest that model--human alignment is associated with explicit linguistic cues, while some rapport-building strategies occur more frequently in misaligned cases. In the categorical task, model predictions exhibit systematic neutral compression, characterized by the overproduction of Neutral labels and the underprediction of Impolite labels. This pattern persists when expert consensus is used as the reference on a diagnostic subset. Our findings highlight the need for pragmatic evaluations that go beyond aggregate agreement metrics by examining directional patterns of model--human disagreement across different human references.

Figures & tables

Figure 1: Study overview. Panel A shows Dataset 1 continuous scoring with six prompts, Dataset 2 three-way classification with four prompts (P2, P3, P4, and P6), and the 318-case expert audit. Panel B shows model–human alignment, inter-model consistency, strategy differences, and human-reference comparisons.

Figure 2: Overall LLM–human alignment across continuous scoring and polarity classification. Panels A–B show Dataset 1 results on the normalized score scale; Close Rate is the proportion of items with an absolute model–human difference below 0.5. Panel C reports model-level Cohen’s $\kappa$ values for Dataset 2; colors distinguish proprietary and open-source models.

Figure 3: Strategy-level differences between aligned and misaligned cases across the two evaluation settings. Positive values indicate strategies more frequent in aligned or correctly classified cases.

Panel A: Agreement comparison	Accuracy	Cohen’s $\kappa$	Macro-F1
Mean model-level LLM vs. Expert consensus	0.636	0.395	0.557
Mean model-level LLM vs. Crowd label	0.163	-0.252	–
Crowd label vs. Expert consensus	0.314	-0.029	0.331
Panel B: Label distribution	Neutral	Polite	Impolite
Crowd labels	31.4%	38.1%	30.5%
Expert consensus	44.1%	31.4%	24.6%

Table 1: Expert-audit results on the diagnostic subset of Dataset 2. LLM values in Panels A–B are averages across five model-level results, rather than predictions from a cross-model ensemble. NCI and ISR are computed from marginal label distributions; ENSR measures directional extreme-to-Neutral shifts. pp = percentage points.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Prompt	Model–human $r$	Inter-model $r$	Gap
P1	0.398	0.729	+0.331
P2	0.388	0.741	+0.353
P3	0.393	0.706	+0.313
P4	0.308	0.456	+0.148
P5	0.411	0.837	+0.426
P6	0.412	0.876	+0.464

Appendix

Table 3: Mean model–human and pairwise inter-model Pearson correlations for Dataset 1 by prompt condition. Gap is the inter-model mean minus the model–human mean.

Model	Condition	$\boldsymbol{\sigma^{2}}$	IQR	Central.	$r$
GPT-4.1	Baseline	0.42	0.85	1.73	0.433
	Expert	0.51	1.02	1.42	0.478
	Neutral	0.47	0.93	1.58	0.451
Claude-3.5	Baseline	0.38	0.78	1.81	0.392
	Expert	0.46	0.95	1.48	0.429
	Neutral	0.41	0.84	1.67	0.407

Appendix

Table 4: Prompt-framing ablation on the Dataset 1 subset. Centralization is the ratio of model to human predictions in the normalized interval

[-0.3,0.3]

; values above 1.0 indicate greater model concentration near the center. Pearson

r

measures agreement with human ratings.

Strategy	Baseline $r$	Expert $r$	$\boldsymbol{\Delta r}$
Larger increase in $r$
Multiple hedges	0.31	0.42	+0.11
First-person plural	0.28	0.38	+0.10
Strategic negative words	0.24	0.33	+0.09
Casual register	0.35	0.43	+0.08
Subtle deference	0.37	0.44	+0.07

Appendix

Table 5: Effect of expert prompting on selected politeness strategies.

\Delta r

is the change in Pearson correlation with human ratings.

Strategy	$\boldsymbol{\kappa}$	Precision	Recall
Please	0.97	0.98	0.99
Gratitude	0.94	0.96	0.95
Apologizing	0.91	0.93	0.92
Greeting	0.89	0.91	0.90
Hedges	0.82	0.85	0.84
Modal verbs	0.88	0.90	0.89

Appendix

Table 7: Validation of the rule-based strategy detector against human annotations on 200 requests. Precision and recall treat human annotations as the reference.

Human & Model: Polite
“Could you please help me understand this issue? I would really appreciate your guidance.”
Human: $+1.8$ , Model: $+1.7$
Strategies: modal, please, gratitude, indirectness
“Thank you so much for taking the time to look at this. Your expertise would be incredibly valuable here.”
Human: $+1.9$ , Model: $+1.8$
Strategies: gratitude, deference, positive lexicon

Appendix

Table 8: Selected examples of close model–human agreement in the Dataset 1 continuous-scoring analysis.

Type 1: Model overestimates politeness
“I was wondering if perhaps you might possibly consider taking a look at this when you have a moment?”
Human: $+0.3$ , Model: $+1.5$ , $\Delta=+1.2$
Possible cue conflict: multiple hedges coincide with a higher model score
“Your work is absolutely amazing and perfect in every way. Could you possibly help me with just this tiny little thing?”
Human: $-0.2$ , Model: $+1.3$ , $\Delta=+1.5$
Possible cue conflict: exaggerated praise coincides with a higher model score

Appendix

Table 9: Selected examples of model–human disagreement in the Dataset 1 continuous-scoring analysis. Interpretations are diagnostic rather than annotator-provided explanations.

Figure 4: Model-level agreement with expert consensus. The dashed line marks the mean pairwise expert agreement value ( $\kappa=0.368$ ) as a descriptive reference; it is not a performance ceiling.

Figure 5: Label distributions for crowd annotations, expert consensus, and the mean LLM distribution on the diagnostic expert-audit subset. Model outputs are more strongly concentrated in Neutral and contain fewer Impolite predictions than both human references.

Explore similar work

No Universal Courtesy: A Cross-Linguistic, Multi-Model Study of Politeness Effects on LLMs Using the PLUM Corpus
Apr 17, 2026Hitesh Mehta, Arjit Saxena, Garima Chhikara +1Large Language Model Bias Prompt Engineering
Department of Computer Science Engineering, Delhi Technological University, New Delhi, India · Department of Electronics and Communication Engineering, Delhi Technological University, New Delhi, India
How Hypocritical Is Your LLM judge? Listener-Speaker Asymmetries in the Pragmatic Competence of Large Language Models
Apr 17, 2026Judith Sieker, Sina ZarrießLarge Language Model Judges Linguistics
Computational Linguistics, Department of Linguistics Bielefeld University, Germany
LLMs Can Better Capture Human Judgments--With the Right Prompts
Jun 10, 2026Danica Dillion, Chen Cecilia Liu, Baihui Wang +5Human Judgment Moral Reasoning
Complexity Science Hub, Vienna · The Ohio State University · University of Cambridge +2

Apr 17, 2026, cs.CLJ/K move · Enter open · S save

No Universal Courtesy: A Cross-Linguistic, Multi-Model Study of Politeness Effects on LLMs Using the PLUM Corpus

Hitesh Mehta, Arjit Saxena, Garima Chhikara, Rohit Kumar

Department of Computer Science Engineering, Delhi Technological University, New Delhi, India · Department of Electronics and Communication Engineering, Delhi Technological University, New Delhi, India

Large Language Model Bias Prompt Engineering Multi-Model Study

Open paper AskPDF arXiv

This paper explores the response of Large Language Models (LLMs) to user prompts with different degrees of politeness and impoliteness. The Politeness Theory by Brown and Levinson and the Impoliteness Framework by Culpeper form the basis of experiments conducted across three languages (English, Hindi, Spanish), five models (Gemini-Pro, GPT-4o Mini, Claude 3.7 Sonnet, DeepSeek-Chat, and Llama 3), and three interaction histories between users (raw, polite, and impolite). Our sample consists of 22,500 pairs of prompts and responses of various types, evaluated across five levels of politeness using an eight-factor assessment framework: coherence, clarity, depth, responsiveness, context retention, toxicity, conciseness, and readability. The findings show that model performance is highly influenced by tone, dialogue history, and language. While polite prompts enhance the average response quality by up to ~11% and impolite tones worsen it, these effects are neither consistent nor universal across languages and models. English is best served by courteous or direct tones, Hindi by deferential and indirect tones, and Spanish by assertive tones. Among the models, Llama is the most tone-sensitive (11.5% range), whereas GPT is more robust to adversarial tone. These results indicate that politeness is a quantifiable computational variable that affects LLM behaviour, though its impact is language- and model-dependent rather than universal. To support reproducibility and future work, we additionally release PLUM (Politeness Levels in Utterances, Multilingual), a publicly available corpus of 1,500 human-validated prompts across three languages and five politeness categories, and provide a formal supplementary analysis of six falsifiable hypotheses derived from politeness theory, empirically assessed against the dataset.

Abstract

Explore similar work

No Universal Courtesy: A Cross-Linguistic, Multi-Model Study of Politeness Effects on LLMs Using the PLUM Corpus

How Hypocritical Is Your LLM judge? Listener-Speaker Asymmetries in the Pragmatic Competence of Large Language Models

LLMs Can Better Capture Human Judgments--With the Right Prompts