Hey, guess what: CUES lead to inferences about typicality

Alison Chi; Sanghyeok Choi; Hannah Rohde; Emily Allaway

cs.CLOct 23, 2026

Hey, guess what: CUES lead to inferences about typicality

Authors: Alison Chi, Sanghyeok Choi, Hannah Rohde, Emily Allaway

Organizations: University of Edinburgh

Abstract

Throughout life, we accumulate knowledge of how typical certain events are. We use this knowledge in daily conversation, often interpreting meaning beyond what has been said explicitly. In this work, we take steps to better understand this meaning recovery, investigating both atypicality and typicality inferences. First, we construct two corpora, EventTyp and the human-validated EventTypGold. We conduct large-scale human and LLM experiments and find that for humans, three CUES predict inferences about typicality: Commonsense Knowledge, Utterance Emphasis, and Snark. However, in most experimental settings, we find that LLMs lack the prerequisite Commonsense Knowledge necessary to accurately assess an event's general typicality, and therefore struggle to follow these CUES.

Figures & tables

Figure 1: Examples of what can cause an atypicality or typicality inference. Someone making an atypicality inference would answer the question on the left with unlikely , whereas if they made a typicality inference, they would answer the question on the right with likely .

	Definition	Example	Written by		Validated by
			LLM	Human	LLM	Human
Context	The first event utterance	Anahita thanks her boss
TypicFollowUp	Event utterance stating something that	Anahita means it	2,062	292	1,226	1,128
	is generally typical given Context.
AtypicFollowUp	Event utterance stating something that	Anahita is in tears	1,679	675	0	2,354
	is generally atypical given Context.

Table 1: Example (Context, TypicFollowUp, AtypicFollowUp) tuple from EventTyp , along with the number of items per-type written and validated by an LLM and a human. Note that we also count the human-written and human-validated data from Kravtchenko and Demberg (2015) . The total number of tuples in EventTyp is 2,354 .

Figure 2: Setup for our surprisal experiments with examples using the Unlikely likelihood rating option. Surprisal is computed over the Completion, given the Input.

Follow-up	Context example	Question
	Anahita thanks her boss	In general, when
		someone thanks
		their boss, how likely
		is it that they
TypicFollowUp	Anahita means it	…mean it?
AtypicFollowUp	Anahita is in tears	…are in tears?

Table 2: Example of the questions we ask in the Commonsense Knowledge Experiment. Two questions are asked per Context, one for the likelihood of the TypicFollowUp and one for that of the AtypicFollowUp.

	Chedeline and Kathem are having a conversation about their friend Anahita.
	Chedeline: “Anahita thanked her boss.”
	Follow-up Setting
Follow-up	NONE	Plain	HeyGW	Question
TypicFollowUp	–	“She meant it.”	“And hey, guess what,	In general, when Anahita thanks her boss, how likely is it that
			she meant it!”	she means it? (Very Unlikely, Unlikely, Likely, Very Likely)
AtypicFollowUp	–	“She was in tears.”	“And hey, guess what,	In general, In general, when Anahita thanks her boss, how likely is it that

Table 3: Setup for Event-Typicality Inference experiments ( § 6 ). A participant sees the top dialogue (containing a Context), and ONE Follow-up setting (containing either a TypicFollowUp or a AtypicFollowUp), along with its corresponding question. This means that for each tuple, there are 6 possible experiment conditions.

Figure 3: Human-LLM agreement for the Commonsense Knowledge experiment, with

\kappa^{\text{AtypicFollowUp }}

on the X-axis and

\kappa^{\text{TypicFollowUp }}

on the Y-axis.

Figure 4: Human results for TypicFollowUps vs. AtypicFollowUps, where a point’s size denotes the number of InfScores equal to its X value. InfScore > 0 = atypicality inference, and InfScore < 0 = typicality inference.

Figure 5: Human distributions of responses in the NONE vs. HeyGW settings for TypicFollowUps, with non-snarky scenarios (left) and snarky scenarios (right).

Figure 6: Percentage of Follow-ups in the HeyGW setting where an LLM made a typicality (X-axis) vs. atypicality inference (Y-axis). Points in the gray halo are LLM configurations where the atypicality and typicality inferences were both statistically significant (p

<

0.01).

Figure 7: LLM consistency; i.e. the percentage of Follow-ups where responses from the Event-Typicality Inference experiment NONE setting matched those from the Commonsense Knowledge experiment, (X-axis for AtypicFollowUps and Y-Axis for TypicFollowUps).

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

TypicFollowUp Event	Example followup to
Category	Mary thanks her boss
State	Mary is at the office
Subaction	Mary means it
Next-action	Mary’s boss says you’re welcome

Appendix

Table 4: Example TypicFollowUp events for the three utterance categories.

Unnatural for dialogue		Wrong typicality
Prompt(s)	Description	Prompt(s)	Description
Malformed Context (2):	Has errors or is	Abstract (3):	Idiomatic, not concrete, or solely
,	awkward/redundant	, ,	happens in one person’s head
Malformed event utterance (3):	Has errors or	Entailed/almost entailed (5):
, ,	is incoherent	Direct (
		Death ( ),	Event only false if the person dies

Appendix

Table 5: 27 LLM-filtering prompts and their classes (Unnatural for dialogue vs. Wrong typicality)

	Accepted Answer	Norming version	Main version
1	Likely/Very Likely	In general, when someone puts on	Bob and Sarah are having a conversation
		a t-shirt and skirt, how likely is	about their friend Sharon. Bob:
		it that they put clothes on?	“Sharon put on a t-shirt and skirt.
			She put clothes on.” In general,
			when Sharon puts on a t-shirt and skirt,
			how likely is it that she puts clothes on?

Appendix

Table 6: Attention check questions. These qualify as ‘nonsensical items’ per Prolific’s attention check policy.

LLM	Base	Instruct	Small	Big
GPT-5.4 ( OpenAI, 2026 )	x	✓	nano-2026-03-17	2026-03-05
Llama-3.1 ( Grattafiori et al., 2024 )	✓	✓	8b	70b
Olmo-3 ( Olmo et al., 2025 )	✓	✓	7b	32b
Qwen3 ( Yang et al., 2025 )	✓	✓	8b	30b
gemma-3 ( Gemma Team, 2025 )	✓	✓	12b	27b

Appendix

Table 7: LLMs used. For those where prompting was available, we tried prompting both with and without CoT (see § C.2.2 . For details on model version selection, see § C.2.1 . Since OpenAI has not made GPT-5.4’s parameter number publicly available, we just put the timestamp model IDs.

Figure 8: Example of what human participants saw in our Commonsense Knowledge experiment.

Figure 9: Example of what human participants saw during our Event-Typicality Inference experiment. Each participant answered between 30 and 32 questions.

Figure 10: Example of an implicit request question participants saw in the pragmatics pre-test. The expected answer here is d).

Figure 11: Example of a coherence question participants saw in the pragmatics pre-test. The expected answer here is ‘Incoherent.’

	$\beta$	z	p
Follow-up	2.87	39.85	$<$ 0.001
Plain.NONE	1.20	26.15	$<$ 0.001
NONE.HeyGW	-0.27	-5.28	$<$ 0.001
Follow-up:Plain.NONE	-1.05	-11.47	$<$ 0.001
Follow-up:NONE.HeyGW	1.95	18.51	$<$ 0.001

Appendix

Table 8: Regression output for the main mixed effects model fit on human data (

\text{Follow-up}\times\text{setting}

	$\beta$	z	p
Follow-up	2.77	40.57	$<$ 0.001
Plain.NONE	1.18	26.50	$<$ 0.001
NONE.HeyGW	-0.24	-4.72	$<$ 0.001
Snark	-0.22	-3.46	$<$ 0.001
Follow-up:Plain.NONE	-1.02	-11.88	$<$ 0.001
Follow-up:NONE.HGW	1.95	22.69	$<$ 0.001

Appendix

Table 9: Regression output for the snark mixed effects model fit on human data,

\text{Follow-up}\times\text{setting}\times\text{snark}

	$\beta$	z	p
TypicFollowUp
Plain.NONE	0.66	10.06	$<$ 0.001
NONE.HeyGW	0.64	8.77	$<$ 0.001
AtypicFollowUp
Plain.NONE	1.86	26.50	$<$ 0.001
NONE.HeyGW	-1.34	-17.98	$<$ 0.001

Appendix

Table 10: Regression output for the subset mixed effects models on human data.

LLM	Surprisal			Prompting			CoT-Prompting
	AtypInf	TypInf	SnarkEffect	AtypInf	TypInf	SnarkEffect	AtypInf	TypInf	SnarkEffect
GPT-5.4	–	–	–	x	✓	x	✓	✓	✓
Llama-3.1-70B	✓	x	✓	✓	✓	✓	✓	✓	✓
Olmo3-32B	✓	x	x	x	✓	x	✓	✓	x
Qwen3-30B	x	✓	x	x	✓	x	x	✓	x
gemma-3-27b	x	x	x	x	✓	x	✓	✓	✓

Appendix

Table 11: Summary of mixed effects models fit on LLM output data. Atypicality and typicality inferences abbreviated to AtypInf and TypInf. Note that the snark effect is only present if atypicality inferences are made.

Figure 12: GPT-5.4 comparison of the distribution of responses to NONE vs. HeyGW settings for TypicFollowUps, with non-snarky dialogues on the left and snarky dialogues on the right.

Figure 13: Correlation (p<0.01) between Commonsense Knowledge (CKnowledge) human-LLM agreement (

\kappa^{\text{Combined}}

, Appendix G ) and the percentage of Event-Typicality Inference Experiment HeyGW setting tuples where the LLM made both an atypicality inference (on the TypicFollowUp) and a typicality inference (on the AtypicFollowUp).

Figure 14: Percentage of the time a typicality inference was made on events voted as AtypicFollowUp by LLM (X-axis) vs. humans (Y-Axis)

Figure 15: Percentage of the time an atypicality inference was made on events voted as TypicFollowUp by LLM (X-axis) vs. humans (Y-Axis)

Pragmatic Reasoning in Language Models

Pragmatic reasoning in language models studies how models infer and communicate meanings shaped by context, intent, shared knowledge, and social norms.

See all 103 papers

Humans and LLMs Diverge on Probabilistic Inferences
Feb 26, 2026Gaurav Kamath, Sreenath Madathil, Sebastian Schuster +2LLM Reasoning
McGill University · Mila – Quebec AI Institute · University of Vienna +2
Can LLMs Access their World Knowledge for Event Prediction?
Oct 13, 2026Zijun Wu, Zhitian Zhang, Hossein Hajimirsadeghi +2Event Time Prediction LLM Evaluation
RBC Borealis · University of Alberta
It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief
Jul 20, 2026Kevin Du, Clara Kümpel, Michelle Wastl +1LLM Evaluation Knowledge Conflicts in Language Models
ETH Zürich · University of Zurich · UC San Diego

Feb 26, 2026J/K move · Enter open · S save

Humans and LLMs Diverge on Probabilistic Inferences

Gaurav Kamath, Sreenath Madathil, Sebastian Schuster, Marie-Catherine de Marneffe+1

McGill University · Mila – Quebec AI Institute · University of Vienna+2

LLM Reasoning

Open paper AskPDF arXiv

Human reasoning often involves working over limited information to arrive at probabilistic conclusions. In its simplest form, this involves making an inference that is not strictly entailed by a premise, but rather only likely given the premise. While reasoning LLMs have demonstrated strong performance on logical and mathematical tasks, their behavior on such open-ended, non-deterministic inferences remains largely unexplored. We introduce ProbCOPA, a dataset of 210 handcrafted probabilistic inferences in English, each annotated for inference likelihood by 25--30 human participants. We find that human responses are graded and varied, revealing probabilistic judgments of the inferences in our dataset. Comparing these judgments with responses from eight state-of-the-art reasoning LLMs, we show that models consistently fail to produce human-like distributions. Finally, analyzing LLM reasoning chains, we find evidence of a common reasoning pattern used to evaluate such inferences. Our findings reveal persistent differences between humans and LLMs, and underscore the need to evaluate reasoning beyond deterministic settings.

Abstract

Pragmatic Reasoning in Language Models

Explore similar work

Humans and LLMs Diverge on Probabilistic Inferences

Can LLMs Access their World Knowledge for Event Prediction?

It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief