Social Meaning in Large Language Models: Structure, Magnitude, and Pragmatic Prompting

Roland Mühlenbernd

cs.CLApr 2, 2026

Social Meaning in Large Language Models: Structure, Magnitude, and Pragmatic Prompting

Authors: Roland Mühlenbernd

Organizations: Leibniz-Centre General Linguistics, Berlin, Germany

Abstract

Large language models (LLMs) increasingly exhibit human-like patterns of pragmatic and social reasoning. This paper addresses two related questions: do LLMs approximate human social meaning not only qualitatively but also quantitatively, and can prompting strategies informed by pragmatic theory improve this approximation? To address the first, we introduce two calibration-focused metrics distinguishing structural fidelity from magnitude calibration: the Effect Size Ratio (ESR) and the Calibration Deviation Score (CDS). To address the second, we derive prompting conditions from two pragmatic assumptions: that social meaning arises from reasoning over linguistic alternatives, and that listeners infer speaker knowledge states and communicative motives. Applied to a case study on numerical (im)precision across three frontier LLMs, we find that all models reliably reproduce the qualitative structure of human social inferences but differ substantially in magnitude calibration. Prompting models to reason about speaker knowledge and motives most consistently reduces magnitude deviation, while prompting for alternative-awareness tends to amplify exaggeration. Combining both components is the only intervention that improves all calibration-sensitive metrics across all models, though fine-grained magnitude calibration remains only partially resolved. LLMs thus capture inferential structure while variably distorting inferential strength, and pragmatic theory provides a useful but incomplete handle for improving that approximation.

Figures & tables

	Precise	Approximate
HP (insurance claim)	“The bicycle cost $500.”	“The bicycle cost about $500.”
LP (friend inquiry)	“The bicycle cost $500.”	“The bicycle cost about $500.”

Table 1: Example stimuli from the bicycle scenario across the four experimental conditions. HP context: Jamie reports the cost to an insurance agent. LP context: Jamie answers a friend who is casually considering buying a bicycle. The utterance form (precise vs. approximate) is identical across contexts; only the pragmatic demands differ.

Figure 1: Human vs. model mean ratings across all conditions (scenarios, contexts, utterance forms, and social attributes) for each model and prompting condition. Each point represents one human–model mean rating pair; the dashed identity line ( $H=M$ ) indicates perfect calibration. Points above the line indicate model overestimation; points below indicate underestimation. GPT clusters closely around the identity line across all conditions, reflecting near-calibrated magnitude alignment. Claude shows greater spread, with sensitivity to prompting condition visible in the vertical displacement of individual condition clusters. Gemini displays a characteristic compression along the x-axis with strong vertical spread, reflecting the severe magnitude inflation reported in Table 3 . Prompting conditions: MIN = Minimal (gray circles); ALT = Alternative-Aware (orange squares); KMA = Knowledge-and-Motives-Aware (blue triangles); COM = Combined (green diamonds).

Model	Prompt	Spearman $\rho$	CCC	RMSE
GPT	MIN	0.856	0.703	0.811
	ALT	0.880	0.812	0.565
	KMA	0.829	0.754	0.601
	COM	0.832	0.786	0.547
Claude	MIN	0.849	0.730	0.766
	ALT	0.847	0.754	0.724

Table 2: Global pattern similarity metrics per model and prompting condition. Spearman

\rho

measures rank-order correspondence between model and human mean ratings across all conditions; CCC (Concordance Correlation Coefficient) jointly assesses co-variation and mean-level agreement; RMSE reports average deviation on the 7-point scale. Prompting conditions: MIN = Minimal; ALT = Alternative-Aware; KMA = Knowledge-and-Motives-Aware; COM = Combined. Bold indicates the best value per model per metric.

Model	Prompt	$\textbf{CDS}_{\textbf{m}}$	$\textbf{CDS}_{\textbf{i}}$	CDS
GPT	MIN	0.325	0.460	0.393
	ALT	0.303	0.392	0.348
	KMA	0.360	0.444	0.402
	COM	0.259	0.349	0.304
Claude	MIN	0.845	0.231	0.538
	ALT	1.235	0.205	0.720

Table 3: Calibration Deviation Scores for main effects (

\text{CDS}_{\text{m}}

), interaction effects (

\text{CDS}_{\text{i}}

), and their aggregate ( CDS ). Lower values indicate better magnitude alignment with the human benchmark. Prompting conditions: MIN = Minimal; ALT = Alternative-Aware; KMA = Knowledge-and-Motives-Aware; COM = Combined. Bold indicates the best value per model per metric.

Figure 2: Effect Size Ratios (ESR) per model, prompting condition, and benchmark effect. Rows are grouped into main effects (top) and form $\times$ context interactions (bottom); columns correspond to prompting conditions (MIN, ALT, KMA, COM). Color encodes deviation from perfect calibration (ESR $=1$ , white): blue indicates attenuation, red indicates exaggeration. Values exceeding the colorscale maximum of 4.5 are marked with an asterisk.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	Prompt	RMSE mean	RMSE indiv
GPT	MIN	0.811	1.542
	ALT	0.565	1.429
	KMA	0.601	1.441
	COM	0.547	1.420
Claude	MIN	0.766	1.521
	ALT	0.724	1.497

Appendix

Table 4: RMSE computed against human condition means ( RMSE mean ) vs. individual human ratings ( RMSE indiv ) per model and prompting condition.

Pragmatic Reasoning in Language Models

Pragmatic reasoning in language models studies how models infer and communicate meanings shaped by context, intent, shared knowledge, and social norms.

See all 94 papers

Tracing the ongoing emergence of human-like reasoning in Large Language Models
May 20, 2026Paolo Morosi, Nikoleta Pantelidou, Fritz Günther +2Logical Reasoning Pragmatic Reasoning in Language Models
Departament de Filologia Catalana, Universitat Autònoma de Barcelona, Barcelona 08193, Spain. · Institut für Psychologie, Humboldt-Universitat zu Berlin, Berlin 10099, Germany. · Institució Catalana de Recerca i Estudis Avançats (ICREA), Barcelona 08010, Spain
LLMs Can Better Capture Human Judgments--With the Right Prompts
Jun 10, 2026Danica Dillion, Chen Cecilia Liu, Baihui Wang +5Human Preference Evaluation LLM Alignment
Complexity Science Hub, Vienna · The Ohio State University · University of Cambridge +2
Closing the Confidence-Faithfulness Gap in Large Language Models
Mar 26, 2026Miranda Muqing Miao, Lyle UngarConfidence Estimation in Language Models Language Model Calibration
University of Pennsylvania · Department of Computer and Information Science

May 20, 2026J/K move · Enter open · S save

Tracing the ongoing emergence of human-like reasoning in Large Language Models

Paolo Morosi, Nikoleta Pantelidou, Fritz Günther, Elena Pagliarini+1

Departament de Filologia Catalana, Universitat Autònoma de Barcelona, Barcelona 08193, Spain. · Institut für Psychologie, Humboldt-Universitat zu Berlin, Berlin 10099, Germany. · Institució Catalana de Recerca i Estudis Avançats (ICREA), Barcelona 08010, Spain

Logical Reasoning Pragmatic Reasoning in Language Models LLM Reasoning Natural Language Inference

Open paper AskPDF arXiv

Humans effortlessly go beyond literal meanings: If you mow the lawn, I will give you fifty dollars, is typically understood as implying that the speaker will pay only if the lawn is mowed, whereas If you are hungry, there is pizza in the oven implies that pizza is available regardless of the hearers hunger. Large Language Models - LLMs - show human-like performance on many tasks, yet it remains unclear whether they reason like humans. To address this, we conducted a population-matching experiment assessing how twentyfive LLMs compute conditional inferences across four languages, compared to an equal number of humans per language. We find that humans enrich logical reasoning through pragmatic inferences across languages. Model behavior is more variable. Some LLMs perfectly follow the truth-table of conditionals but they ignore pragmatic inferences, while others deviate from the truth-table, adhering to a single interpretation across the board, thus reflecting accurate rule-based processing but not human-like reasoning. Overall, LLMs are accurate semantic operators, but fail to capture the pragmatic enrichments characteristic of human reasoning. Crucially, LLM accuracy is neither predicted nor boosted by open vs. closed status, training orientation, or architecture type, suggesting that pragmatic reasoning is still an emerging ability in the cognitive toolkit of artificial systems.

Abstract

Pragmatic Reasoning in Language Models

Explore similar work

Tracing the ongoing emergence of human-like reasoning in Large Language Models

LLMs Can Better Capture Human Judgments--With the Right Prompts

Closing the Confidence-Faithfulness Gap in Large Language Models