Why Do LLMs Accept Generics? Explaining Acceptability using Psycholinguistic and Model-Derived Factors

Mervat Abassy; Zhaoxin Feng; Tatsuki Kuribayashi; Kentaro Inui; Emmanuele Chersoni

cs.CLOct 20, 2026

Why Do LLMs Accept Generics? Explaining Acceptability using Psycholinguistic and Model-Derived Factors

Authors: Mervat Abassy, Zhaoxin Feng, Tatsuki Kuribayashi, Kentaro Inui, Emmanuele Chersoni

Organizations: MBZUAI · The Hong Kong Polytechnic University · Tohoku University · RIKEN

Abstract

Psycholinguistic studies explore why humans accept some generic statements but reject others by showing that these judgments depend on more than how common a property is. People consider several other factors such as how well it sets the kind apart from others or whether it is characteristic of the kind. Inspired by the linguistic and cognitive literature, we explore to what extent factors from these studies explain how large language models (LLMs) judge generics. Our results with twelve models on a psycholinguistic dataset and on naturally occurring generics show that models weigh how common a property is against how well it distinguishes the kind, and combine the two much as people do, though less clearly. Like people, they favor characteristic properties, but unlike the accounts of human judgment, they ignore striking ones. What matters most to LLMs is whether a statement is factually true, and much of the influence of human factors reaches their ratings through that judgment. In six 7-9B models, shifting the hidden states along each factor's direction moves the judgment as the factor predicts.

Figures & tables

Figure 1: We explore how far factors from the psycholinguistic literature explain LLM judgments of generics, which further factors the models point to, and whether these factors can steer acceptability inside the models.

Figure 2: IS_A generics: acceptability against prevalence for humans and one Qwen model per group (fitted lines; Pearson

r

, ∗

p<.05

). ConGen: all items (solid) and the half on which models agree most about prevalence (dotted). All models in Appendix C.2 .

Figure 3: Feature-describing generics: acceptability against prevalence (P) for high and low cue validity (C), for humans and Qwen3-8B. Converging lines indicate an interaction ( ∗

p<.05

). Human ratings are means over participants; the model’s are single integer ratings, hence the discrete values. All models in Table 5 .

Figure 4:

R^{2}

of acceptability as factors are added (one Qwen per group); dashed: people’s P + C. All models in Appendix D.5 .

Model	Layer	F	P	C	Ch	O	S	Random
Qwen2.5-7B	15 / 28	.72 ∗	.51 ∗	.30 ∗	.18 ∗	$-$ .11 ∗	.22 ∗	$-$ .06 ∗
Llama-3.1-8B	13 / 32	1.88 ∗	1.23 ∗	1.65 ∗	1.52 ∗	$-$ .16 ∗	.48 ∗	$-$ .09 ∗
Mistral-7B	13 / 32	2.07 ∗	1.82 ∗	1.81 ∗	1.45 ∗	$-$ .88 ∗	.51 ∗	.05 ∗
OLMo-3-7B	13 / 32	2.50 ∗	1.42 ∗	1.44 ∗	.89 ∗	$-$ .57 ∗	.39 ∗	.11 ∗
Gemma-2-9B	17 / 42	.59 ∗	.47 ∗	.59 ∗	.35 ∗	$-$ .64 ∗	.36 ∗	.01
Qwen3-8B	20 / 36	1.22 ∗	1.10 ∗	1.00 ∗	1.11 ∗	$-$ .57 ∗	.43 ∗	.07 ∗

Table 1: Change in expected acceptability per unit of push along each factor’s direction, at factuality’s peak layer. Bold: largest effect and overreach; ∗ beyond two standard errors.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Factor	Prompt
Acceptability	Please rate how acceptable the following sentence is: "<generic>" Answer with a single number from -3 to +3, where -3 = "Definitely reject", 0 = "Not sure", and +3 = "Definitely accept." Respond with only the number, nothing else.
Prevalence	Suppose you encounter <category> X. How likely is it that X <feature>? Answer with a single number from 0 to 100, where 0 = "definitely not" and 100 = "definitely yes." Respond with only the number, nothing else.
Cue validity	Suppose <comparison category> X <feature>. How likely is it that X is <category>? Answer with a single number from -3 to +3, where -3 = "Very unlikely", 0 = "Not sure", and +3 = "Very likely." Respond with only the number, nothing else.
Factuality	How factually accurate is the following statement? "<generic>" Answer with a single number from -3 to +3, where -3 = "factually wrong", 0 = "Not sure", and +3 = "factually correct." Respond with only the number, nothing else.
Overreach	Sentences that generalise about a kind vary in how strong a claim they make. Some say no more than the facts support; others assert more about the kind than is warranted. Does the following sentence claim more than is warranted? "<generic>" Answer with a single number from -3 to +3, where -3 = "claims no more than is warranted", 0 = "Not sure", and +3 = "claims far more than is warranted." Respond with only the number, nothing else.
Characteristicness	Some properties are characteristic of a kind: the kind is disposed to have them, whether or not every member does. Others are merely common among members without being characteristic of the kind itself. Is "<feature>" a characteristic property of <category>? Answer with a single number from -3 to +3, where -3 = "not characteristic at all, just something many happen to have", 0 = "Not sure", and +3 = "strongly characteristic of the kind." Respond with only the number, nothing else.

Appendix

Table 2: Prompts for Ryu et al.’s three questions (top) and the four additional factors (bottom). Placeholders in angle brackets are filled from each item’s materials; a line break marks a blank line in the prompt.

Group	Model	A	P	C
Open, 7–9B	Qwen2.5-7B	.74	.75	.66
	Mistral-7B	.62	.69	.73
	OLMo-3-7B	.47	.38	.41
	Gemma-2-9B	.83	.79	.69
	Qwen3-8B	.60	.61	.54
	Llama-3.1-8B	.72	.65	.34

Appendix

Table 3: Pearson correlations of each model’s ratings of acceptability (A), prevalence (P) and cue validity (C) with the human means, over Ryu et al.’s 64 feature-describing generics. Group average: the ratings averaged over the models of the group. Bold: highest in each group.

		$r_{P,C}$		$r_{A,P}$
						ConGen, prevalence
Group	Rater	Ryu	ConGen	Ryu	ConGen	agreed	disputed
Humans	Ryu et al. (2022)	$-$ .64 ∗	–	.87 ∗	–	–	–
Open, 7–9B	Qwen3-8B	$-$ .68 ∗	.24 ∗	.84 ∗	.51 ∗	.64 ∗	.37 ∗
	Gemma-2-9B	$-$ .72 ∗	.29 ∗	.84 ∗	.59 ∗	.75 ∗	.42 ∗
	Mistral-7B	$-$ .63 ∗	.35 ∗	.82 ∗	.37 ∗	.62 ∗	.08

Appendix

Table 4: IS_A generics: correlation of prevalence (P) with cue validity (C) and with acceptability (A), each rater’s own ratings. Agreed / disputed: the 75 ConGen items on which the twelve models’ prevalence ratings vary least / most (split at the median standard deviation, 22.8 points). The models’ prevalence ratings agree closely on Ryu et al.’s items (mean inter-model

r=.92

; model average vs. human ratings,

r=.98

) but less on ConGen’s (

r=.57

). Bold: within each group, the model closest to the human value reported by Ryu et al. (2022) (

r_{P,C}=-.64

r_{A,P}=.87

). Ryu et al.: 20 items; ConGen: 150. ∗

p<.05

Figure 5: Feature-describing generics: acceptability against prevalence for high and low cue validity, for every model and analysis (fitted lines;

t

of the P

\times

C interaction, ∗

p<.05

). Converging lines: the human interaction.

		Ryu et al.’s items		ConGen
Group	Rater	own	human P, C	own	group avg. P, C	own, ordinal
Humans	Ryu et al. (2022)	$-$ .166 ∗	–	–	–	–
Open, 7–9B	Qwen3-8B	$-$ .242 ∗	$-$ .161	$-$ .022	$-$ .104 ∗	.02
	Gemma-2-9B	$-$ .112	$-$ .124	$-$ .047	$-$ .110 ∗	.00
	Mistral-7B	$-$ .117	$-$ .034	.001	$-$ .038	.04
	Llama-3.1-8B	.205	$-$ .129	$-$ .024	$-$ .066	.05

Appendix

Table 5: Feature-describing generics: standardised prevalence

\times

cue validity interaction in acceptability, with the rater’s own prevalence and cue validity (own), the human ones (human P, C; Ryu et al.’s items only), or those averaged over the model’s group (group avg. P, C). Ordinal: the own-ratings model as an ordered logit, which allows for the ceiling of the rating scale (log-odds). Bold: a significant negative interaction, the human pattern. Ryu et al.: 64 items; ConGen: 582. ∗

p<.05

Rejecting		Accepting
Word	$z$	Word	$z$
not	20.1	clear	10.7
all	14.8	correct	9.6
incorrect	11.5	known	8.3
specific	9.7	accurately	8.3
implies	9.2	verb	8.0

Appendix

Table 6: The words and word pairs that most distinguish the models’ explanations ending in rejection (rating

\leq 0

n=549

) from those ending in acceptance (

\geq 2

n=834

), pooled over the six 7–9B models and three prompts: weighted log-odds with an informative prior ( Monroe et al., 2008 ) , as

z

-scores; higher means more reliably distinctive. The final answer line, the words of the generic itself, common function words (except not , all , no ) and words used in fewer than 15 explanations were removed before counting.

		IS_A				Feature-describing
		Ryu et al.		ConGen		Ryu et al.			ConGen
Group	Rater	P	P+C	P	P+C	P	P+C	P $\times$ C	P	P+C	P $\times$ C
Humans	Ryu et al. (2022)	.71	.82	–	–	.34	.50	.53	–	–	–
Open, 7–9B	Qwen2.5-7B	.26	.23	.22	.23	.16	.23	.14	.23	.26	.25
	Mistral-7B	.54	.48	.11	.11	.12	.12	.07	.24	.26	.26
	OLMo-3-7B	.27	$-$ .84	.12	.13	$-$ .01	$-$ .04	$-$ .06	.22	.23	.23

Appendix

Table 7: Baselines within each generic type: variance explained in held-out items (10-fold cross-validated

R^{2}

, mean of five splits) by prevalence alone (P), prevalence and cue validity (P+C), and Ryu et al.’s rule with their interaction (P

\times

C). Each rater’s own ratings. Bold: the best baseline (ties included). Negative values: worse than predicting the mean. Ryu et al.: 20 IS_A and 64 feature-describing items; ConGen: 150 and 582.

Figure 6: Characteristicness and strikingness against prevalence, each model’s own ratings: one fitted line per model over the range of its prevalence ratings, coloured by group, with the range of correlations in each panel. Ratings on

-3

+3

; prevalence on 0–100.

		Characteristicness				Strikingness
		min. char. $-$ maj. stat.		principled	$\rho$ with	striking
Group	Model	raw	given P	$-$ other	humans	$-$ other
Open, 7–9B	Qwen2.5-7B	.83 ∗	.81 ∗	2.44 ∗	.68	2.03 ∗
	Mistral-7B	2.00 ∗	1.97 ∗	2.41 ∗	.68	3.67 ∗
	OLMo-3-7B	$-$ .17	$-$ .23	1.02 ∗	.68	2.28 ∗
	Gemma-2-9B	.25	.24	2.45 ∗	.93 ∗	4.18 ∗

Appendix

Table 8: Validation of the characteristicness and strikingness ratings on Ryu et al.’s items, which are the materials of Prasada et al. (2013) , twelve of each of seven types. Min. char.

-

maj. stat.: mean characteristicness of minority characteristic generics ( Ducks lay eggs ) minus majority statistical ones ( Cars have radios ), raw and controlling for the model’s own prevalence (in Ryu et al.’s human ratings the former are the less prevalent, 69% vs. 74%). Principled

-

other: quasi-definitions and majority and minority characteristic generics minus the other four types.

\rho

: rank correlation of the model’s seven type means with Prasada et al.’s human ratings of principled connections. Striking

-

other: mean strikingness of the striking generics minus all others. Ratings on

-3

+3

. Bold: the contrast holds with prevalence controlled. ∗

p<.05

Group	Model	$R^{2}_{P,C}$	Ch	S	$R^{2}$	$R^{2}_{P,C}$	Ch	S	$R^{2}$	$R^{2}_{P,C}$	Ch	S	$R^{2}$
		Feature, Ryu et al.				Feature, ConGen				IS_A, ConGen
Open, 7–9B	Qwen2.5-7B	.36	.35 ∗	.26 ∗	.47 ∗	.27	.09 ∗	$-$ .04	.28 ∗	.27	.28 ∗	$-$ .01	.32 ∗
	Mistral-7B	.23	.32 ∗	.17	.33 ∗	.27	.22 ∗	.02	.31 ∗	.14	.27 ∗	$-$ .08	.22 ∗
	OLMo-3-7B	.05	.50 ∗	.02	.26 ∗	.24	.06	$-$ .06	.25	.16	.22 ∗	.06	.20 ∗
	Gemma-2-9B	.37	.39 ∗	.09	.51 ∗	.33	.17 ∗	$-$ .01	.35 ∗	.38	.33 ∗	.01	.45 ∗
	Qwen3-8B	.21	.18	$-$ .10	.24	.18	.13 ∗	.05	.20 ∗	.28	.12	$-$ .09	.30

Appendix

Table 9: Characteristicness (Ch) and strikingness (S) added to the baseline of prevalence and cue validity: standardised weights in each model’s acceptability, each model’s own ratings.

R^{2}_{P,C}

: variance explained by the baseline;

R^{2}

: with Ch and S added, bold when the rise is significant (

F

test). Ryu et al.: 64 items; ConGen: 582 feature-describing and 150 IS_A. ∗

p<.05

Feature, Ryu et al.

Feature, ConGen

IS_A, ConGen

Group

Model

R^{2}_{6}

shared

R^{2}_{6}

shared

R^{2}_{6}

shared

Open, 7–9B

Qwen2.5-7B

.58 ∗

.19 ∗

.68 ∗

90%

.72 ∗

.01

.64 ∗

96%

.72 ∗

.06

.66 ∗

96%

Mistral-7B

.59 ∗

-

.28 ∗

.64 ∗

98%

.58 ∗

-

.12 ∗

.58 ∗

91%

.65 ∗

-

.11

.56 ∗

97%

OLMo-3-7B

.35 ∗

-

.16

.42 ∗

43%

.57 ∗

.03

.49 ∗

85%

.77 ∗

-

.13 ∗

.71 ∗

96%

Gemma-2-9B

.42 ∗

-

.22

.67 ∗

89%

.56 ∗

-

.20 ∗

.63 ∗

97%

.59 ∗

-

.30 ∗

.80 ∗

99%

Qwen3-8B

.59 ∗

-

.25 ∗

.57 ∗

86%

.56 ∗

-

.17 ∗

.51 ∗

94%

.70 ∗

-

.06

.70 ∗

89%

Appendix

Table 10: Factuality (F) and overreach (O) added to the four psychological factors (prevalence, cue validity, characteristicness, strikingness): standardised weights, each model’s own ratings. Factuality is the largest of the six weights in every cell except OLMo-3-7B (characteristicness) and Gemini-2.5-Flash (prevalence) on Ryu et al.’s items. Bold: overreach significantly lowers acceptability.

R^{2}_{6}

: variance explained by all six factors, starred when the rise over the four (the

R^{2}

of Table 9 ) is significant (

F

test). Shared: of what the four factors explain in acceptability, the share that factuality and overreach also explain. ∗

p<.05

		Feature, Ryu et al.			Feature, ConGen			IS_A, ConGen
Group	Model	P $\times$ Ch	P $\times$ S	$\Delta R^{2}_{\mathrm{cv}}$	P $\times$ Ch	P $\times$ S	$\Delta R^{2}_{\mathrm{cv}}$	P $\times$ Ch	P $\times$ S	$\Delta R^{2}_{\mathrm{cv}}$
Open, 7–9B	Qwen2.5-7B	$-$ .09	$-$ .02	$-$ .074	.00	$-$ .06	$-$ .011	$-$ .07	.00	$-$ .073
	Mistral-7B	.01	$-$ .23	$-$ .154	$-$ .04	.00	$-$ .019	$-$ .01	.02	$-$ .115
	OLMo-3-7B	$-$ .12	.25 ∗	$-$ .154	$-$ .01	.09 ∗	$-$ .007	$-$ .17	.01	$-$ .051
	Gemma-2-9B	.04	$-$ .16	$-$ .428	$-$ .01	$-$ .02	$-$ .018	.00	$-$ .11	$-$ .046
	Qwen3-8B	$-$ .09	.13	$-$ .170	$-$ .12 ∗	$-$ .02	$-$ .016	$-$ .04	$-$ .10	$-$ .064

Appendix

Table 11: Interactions predicted by the psychological accounts, each added on its own to the model of Table 9 : prevalence

\times

characteristicness (P

\times

Ch) and prevalence

\times

strikingness (P

\times

S), standardised weights; both are predicted to be negative (bold: significant and negative).

\Delta R^{2}_{\mathrm{cv}}

: change in held-out

R^{2}

(10-fold cross-validation) from adding all pairwise interactions among the factors; negative: they predict new items worse. ∗

p<.05

Figure 7: What the six factors explain in each model’s acceptability, split into the part explained only by the psychological factors (prevalence, cue validity, characteristicness, strikingness), the part explained by either set, and the part explained only by factuality and overreach (in-sample

R^{2}

, each model’s own ratings). Ryu et al.’s 20 IS_A items are too few for this analysis (Appendix D.5 ).

	$r$ with people’s acceptability			P-Acceptability from people’s P and C
				IS_A		Feature-describing
Model	P-Acc.	SLOR	answers	P	C	P	C	P $\times$ C
Qwen2.5-7B	.44 ∗	.43 ∗	.72 ∗	.46	$-$ .16	.20	.32 ∗	$-$ .07
Mistral-7B	.42 ∗	.37 ∗	.67 ∗	.61 ∗	$-$ .08	.12	.45 ∗	$-$ .03
OLMo-3-7B	.44 ∗	.38 ∗	.51 ∗	.41	$-$ .42	.23	.30 ∗	.00
Gemma-2-9B	.46 ∗	.29 ∗	.84 ∗	.77 ∗	.04	.42 ∗	.25 ∗	$-$ .09

Appendix

Table 12: Surprisal-based acceptability of the 7–9B models on Ryu et al.’s items. Left: correlation with people’s acceptability of P-Acceptability ( Cilleruelo et al., 2025 ) , of SLOR ( Lau et al., 2017 ) , and of the model’s answers for comparison. Right: Ryu et al.’s analysis with P-Acceptability as the acceptability measure and people’s prevalence (P) and cue validity (C) as predictors, standardised weights; 20 IS_A and 64 feature-describing items. Bold: a significant effect in the human direction. ∗

p<.05

Model	Layer	F	P	C	Ch	O	S	Random
Qwen2.5-7B	15 / 28	.64 ∗	.50 ∗	.31 ∗	.18 ∗	$-$ .09 ∗	.21 ∗	$-$ .07 ∗
Llama-3.1-8B	13 / 32	1.68 ∗	1.27 ∗	1.57 ∗	1.44 ∗	$-$ .07 ∗	.43 ∗	$-$ .08 ∗
Mistral-7B	13 / 32	1.99 ∗	1.74 ∗	1.40 ∗	1.09 ∗	$-$ .87 ∗	.50 ∗	.05 ∗
OLMo-3-7B	13 / 32	2.44 ∗	1.58 ∗	1.35 ∗	.93 ∗	$-$ .47 ∗	.38 ∗	.11 ∗
Gemma-2-9B	17 / 42	.67 ∗	.51 ∗	.57 ∗	.40 ∗	$-$ .59 ∗	.36 ∗	.01
Qwen3-8B	20 / 36	1.22 ∗	1.11 ∗	.99 ∗	1.10 ∗	$-$ .60 ∗	.43 ∗	$-$ .10 ∗

Appendix

Table 13: As Table 1 , with each factor’s direction fitted to the rating the model wrote instead of its expected rating. Bold: the largest positive effect (factuality), and overreach, which lowers the rating as its meaning predicts. ∗ more than two standard errors from zero.

Figure 8: Mean prevalence of the generics best fitted by Some , Most and All ( Cilleruelo et al., 2025 ) , per model: people’s prevalence on Ryu et al.’s items, the model’s own on ConGen. Marker size: share of items with that best fit. The bare plural is not shown.

Model	$r$	=mode	$r$	=mode	$r$	=mode	$r$	=mode	$r$	=mode	$r$	=mode	$r$	=mode	mass
	A		P		C		F		Ch		O		S
Qwen2.5-7B	.98	90%	.97	80%	.96	86%	.97	86%	.96	73%	.92	79%	.94	88%	1.00
Mistral-7B	.92	87%	.87	71%	.89	83%	.92	86%	.91	77%	.87	74%	.90	87%	1.00
OLMo-3-7B	.87	82%	.87	58%	.81	79%	.86	73%	.89	91%	.77	97%	.91	95%	1.00
Gemma-2-9B	.97	86%	.95	82%	.97	90%	.97	92%	.96	87%	.94	87%	.95	86%	1.00
Qwen3-8B	.99	97%	.98	95%	.99	97%	.99	99%	.99	96%	.97	98%	.98	98%	1.00

Appendix

Table 14: What the 7–9B models write against what their output probabilities say, ConGen’s feature-describing generics.

r

: correlation of the written answer with the expected rating (the probability-weighted mean of the valid answers); =mode: share of items on which the written answer is the single most probable answer; mass: mean probability on valid answers, averaged over the questions. Bold: the lowest agreement for each question. A: acceptability, P: prevalence, C: cue validity, F: factuality, Ch: characteristicness, O: overreach, S: strikingness.

Model	Layer	F	P	C	Ch	O	S	Random
Qwen2.5-7B	15 / 28	.41 ∗	.11 ∗	.04 ∗	.00	.00	.04 ∗	.00
Llama-3.1-8B	13 / 32	1.28 ∗	.72 ∗	.43 ∗	.38 ∗	$-$ .10 ∗	.04 ∗	.00
Mistral-7B	13 / 32	1.79 ∗	.68 ∗	.58 ∗	.26 ∗	$-$ .58 ∗	.04 ∗	.00
OLMo-3-7B	13 / 32	1.37 ∗	.32 ∗	.18 ∗	.10 ∗	$-$ .13 ∗	.02 ∗	.01
Gemma-2-9B	17 / 42	.05	.11 ∗	.06 ∗	.04 ∗	$-$ .19 ∗	.03 ∗	.01
Qwen3-8B	20 / 36	.51 ∗	.46 ∗	.22 ∗	.40 ∗	$-$ .30 ∗	.07 ∗	.02

Appendix

Table 15: Swaps: change in expected acceptability when items in the bottom quarter of a factor take the position along its direction of items in the top quarter, minus the change for the reverse swap, at the layer of Table 1 . Bold: the largest positive shift, and overreach where it lowers the rating. ∗ more than two standard errors from zero.

Pragmatic Reasoning in Language Models

Pragmatic reasoning in language models studies how models infer and communicate meanings shaped by context, intent, shared knowledge, and social norms.

See all 103 papers

Tracing the ongoing emergence of human-like reasoning in Large Language Models
May 20, 2026Paolo Morosi, Nikoleta Pantelidou, Fritz Günther +2Logical Reasoning Pragmatic Reasoning in Language Models
Departament de Filologia Catalana, Universitat Autònoma de Barcelona, Barcelona 08193, Spain. · Institut für Psychologie, Humboldt-Universitat zu Berlin, Berlin 10099, Germany. · Institució Catalana de Recerca i Estudis Avançats (ICREA), Barcelona 08010, Spain
It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief
Jul 20, 2026Kevin Du, Clara Kümpel, Michelle Wastl +1LLM Evaluation Knowledge Conflicts in Language Models
ETH Zürich · University of Zurich · UC San Diego
An LLM-Native Psychometric Instrument Reveals a Self-Report--Behavior Gap Across 25 Models
Apr 24, 2026Juan Manuel ContrerasLLM Evaluation Language Model Self-Assessment
Independent Researcher

May 20, 2026J/K move · Enter open · S save

Tracing the ongoing emergence of human-like reasoning in Large Language Models

Paolo Morosi, Nikoleta Pantelidou, Fritz Günther, Elena Pagliarini+1

Departament de Filologia Catalana, Universitat Autònoma de Barcelona, Barcelona 08193, Spain. · Institut für Psychologie, Humboldt-Universitat zu Berlin, Berlin 10099, Germany. · Institució Catalana de Recerca i Estudis Avançats (ICREA), Barcelona 08010, Spain

Logical Reasoning Pragmatic Reasoning in Language Models LLM Reasoning Natural Language Inference

Open paper AskPDF arXiv

Humans effortlessly go beyond literal meanings: If you mow the lawn, I will give you fifty dollars, is typically understood as implying that the speaker will pay only if the lawn is mowed, whereas If you are hungry, there is pizza in the oven implies that pizza is available regardless of the hearers hunger. Large Language Models - LLMs - show human-like performance on many tasks, yet it remains unclear whether they reason like humans. To address this, we conducted a population-matching experiment assessing how twentyfive LLMs compute conditional inferences across four languages, compared to an equal number of humans per language. We find that humans enrich logical reasoning through pragmatic inferences across languages. Model behavior is more variable. Some LLMs perfectly follow the truth-table of conditionals but they ignore pragmatic inferences, while others deviate from the truth-table, adhering to a single interpretation across the board, thus reflecting accurate rule-based processing but not human-like reasoning. Overall, LLMs are accurate semantic operators, but fail to capture the pragmatic enrichments characteristic of human reasoning. Crucially, LLM accuracy is neither predicted nor boosted by open vs. closed status, training orientation, or architecture type, suggesting that pragmatic reasoning is still an emerging ability in the cognitive toolkit of artificial systems.

Abstract

Pragmatic Reasoning in Language Models

Explore similar work

Tracing the ongoing emergence of human-like reasoning in Large Language Models

It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief

An LLM-Native Psychometric Instrument Reveals a Self-Report--Behavior Gap Across 25 Models