Contextualising (Im)plausible Events Triggers Figurative Language

Annerose Eichel; Tonmoy Rakshit; Sabine Schulte im Walde

cs.CLApr 9, 2026

Contextualising (Im)plausible Events Triggers Figurative Language

Authors: Annerose Eichel, Tonmoy Rakshit, Sabine Schulte im Walde

Organizations: Institute for Natural Language Processing, University of Stuttgart

Abstract

This work explores the connection between (non-)literalness and plausibility at the example of subject-verb-object events in English. We design a systematic setup of plausible and implausible event triples in combination with abstract and concrete constituent categories. Our analysis of human and LLM-generated judgments and example contexts reveals substantial differences between assessments of plausibility. While humans excel at nuanced detection and contextualization of (non-)literal vs. implausible events, LLM results reveal only shallow contextualization patterns with a bias to trade implausibility for non-literal, plausible interpretations.

Figures & tables

Panel of Figure 1 — Figure 1: Analysis of human figurative majority-assigned labels assigned across (a) 27 abstractness combinations ranging from most abstract on the left to most concrete one the right, (b) in comparison with original labels used to create PAP, and (c) PAP majority ratings .

Panel of Figure 2 — Figure 2: Analysis of Qwen3-4B figurative labels (zero-shot), similarly to Figure 1 .

Literal
		Human		Qwen3-4B		Llama3.1-8B		Mistral-7B		Gemma3-4B
Figurative	df	$\chi^{2}$	$V$	$\chi^{2}$	$V$	$\chi^{2}$	$V$	$\chi^{2}$	$V$	$\chi^{2}$	$V$
abstractness	26	63.64***	0.39	23.10	0.24	48.13**	0.34	41.50*	0.32	42.82*	0.32
orig. label	1	6.91**	0.13	37.68	0.30	0.17	0.02	4.64*	0.11	33.10	0.28
PAP rating	3	13.49**	0.18	6.20	0.12	2.88	0.08	1.59	0.06	7.68	0.14
abstractness	26	43.93**	0.33	34.81	0.29	50.88**	0.35	42.67*	0.32	43.75*	0.33

Table 1: Associations between figurative language and abstractness, original label, or PAP ratings.

\chi^{2}

indicates significance (

p<.05

:, $p<.01$ :, $p<.001$ :) and Cramér’s

V

measures strength of association. Model results are based on zero-shot prompts.

model	zero-shot		few-shot
	acc.	$\tau$	acc.	$\tau$
Gemma3-4B	0.27	0.09	0.30	0.01
Qwen3-4B	0.27	-0.15	0.25	-0.16
Mistral-7B	0.20	-0.10	0.21	-0.05
Llama3.1-8B	0.28	0.04	0.32	0.05

Table 2: Model performance for label prediction across prompt types. Predictions are aggregated across prompt templates and five model runs. We report accuracy (acc.) and Kendall’s

\tau

using human majority-assigned decisions as reference value. Bold:

p<0.001

	H1	H2	H3	LL	QW	MI.	GE
Specific	94	92	93	51	52	27	17
Altered (s)	10	8	8	-	-	-	2
Altered (v)	6	13	13	-	-	-	-
Altered (o)	15	8	8	-	-	-	-
Anomalous	-	1	2	6	11	1	22
None	3	4	2	40	34	69	58

Table 3: Human (H) vs. LLM (LL: Llama, QW: Qwen, MI: Mistral, GE: Gemma) context patterns.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

orig. labels	PAP ratings
	plausible	disagree	implausible
plausible	81	64	31
implausible	81	80	77

Appendix

Table 4: Overview of number of target event triples sampled from PAP.

Panel of Figure 4 — Figure 4: Distributions of annotator demographic features.

Figure 5: Zero shot prompt templates, from top to bottom including the task formulation as a question (top), a statement (middle), and as closely as possible based on the instructions for human annotation but condensed in one sentence (bottom).

Figure 6: Few shot prompt template. The prompt is based as closely as possible on the instructions for human annotation while optimizing for the shortest length.

Panel of Figure 7 — Figure 7: Overview of analysis of Qwen3-4B , Mistral-7B , Llama3.1-8B , and Gemma3-4B figurative labels ( zero-shot ), similarly to human analysis in Figure 1 .

Panel of Figure 8 — Figure 8: Overview of analysis of Qwen3-4B , Mistral-7B , Llama3.1-8B , and Gemma3-4B figurative labels ( few-shot ), similarly to human analysis in Figure 1 .

Literal
		Human		Qwen3-4B		Llama3.1-8B		Mistral-7B		Gemma3-4B
Figurative	df	$\chi^{2}$	$V$	$\chi^{2}$	$V$	$\chi^{2}$	$V$	$\chi^{2}$	$V$	$\chi^{2}$	$V$
abstractness	26	63.64***	0.39	43.66*	0.33	46.71**	0.34	51.37**	0.35	53.59**	0.36
orig. label	1	6.91**	0.13	41.05	0.32	7.74**	0.14	4.64*	0.11	13.56***	0.18
PAP_rating	3	13.49**	0.18	15.48**	0.19	4.04	0.10	3.37	0.09	1.23	0.05
abstractness	26	43.93**	0.33	50.84**	0.35	61.28***	0.39	34.08	0.29	57.32***	0.37

Appendix

Table 5: Associations between figurative language and abstractness, original label, or PAP ratings.

\chi^{2}

indicates significance (

p<0.05

:, $p<0.01$ :, $p<0.001$ :) and Cramér’s

V

measures strength of association. Model results are based on few-shot prompts.

	Zero-shot			Few-shot
	question	statement	human instr.	human instr.
Gemma3-4B	67.88	89.29	61.05	22.83
Qwen3-4B	91.00	83.94	63.50	73.90
Mistral-7B	84.91	95.83	78.47	94.51
Llama3.1-8B	23.88	31.39	31.03	61.37

Appendix

Table 6: Overview of share of only figurative labels per prompt, in percent. Note that percentages do not add up to 100% but each number is a share of 100% labels distributed among figurative (shown here), literal , neither , and unclear (not shown). For reference, prompt templates are listed in Figure 6 and Figure 5 .

Figurative Language Understanding

Figurative language understanding studies how people and AI interpret nonliteral meaning in metaphors, idioms, proverbs, puns, and related forms.

See all 45 papers

STRIVE: Probing Reasoning Limits in Graded Plausibility Generation and Evaluation
Aug 5, 2026Bhiman Kumar Baghel, Anna Chrabaszcz, Tessa Warren +3LLM Evaluation Controllable Text Generation
University of Pittsburgh, Pittsburgh, PA, USA · University of Wisconsin–Madison, Madison, WI, USA
Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility
Jul 16, 2025Michael A. Lepori, Jennifer Hu, Ishita Dasgupta +3Representation Geometry in Language Models
Department of Computer Science, Brown University · Google DeepMind · Department of Cognitive & Psychological Sciences, Brown University
Plausibility as Commonsense Reasoning: Humans Succeed, Large Language Models Do not
Apr 6, 2026Sercan KarakaşImplicit Reasoning in Language Models LLM Evaluation
University of Chicago

Aug 5, 2026J/K move · Enter open · S save

STRIVE: Probing Reasoning Limits in Graded Plausibility Generation and Evaluation

Bhiman Kumar Baghel, Anna Chrabaszcz, Tessa Warren, Michael Walsh Dickey+2

University of Pittsburgh, Pittsburgh, PA, USA · University of Wisconsin–Madison, Madison, WI, USA

LLM Evaluation Controllable Text Generation Computational Psycholinguistics Language Model Probing

Open paper AskPDF arXiv

Event knowledge concerns who does what to whom. Psycholinguists use event-plausibility judgments to examine how this knowledge supports human language processing. To isolate plausibility effects, these studies require controlled event sets in which one event slot varies across plausibility levels while all other event features remain fixed. Constructing such sets manually is labor-intensive. We therefore introduce STRIVE, an LLM-based framework for jointly generating and evaluating controlled event sets crossing plausibility class (plausible vs. implausible) with intended classification difficulty (easy vs. hard). Given a verb, STRIVE constructs a shared event frame, then produces one event per condition by varying one slot while holding all others fixed. In experiments with six models across 60 verbs, GPT-5.1 produced high-quality sets only 16.7% of the time using the baseline generation prompt. Adding a global reasoning scratchpad and evaluator-guided refinement raised this rate to 75.0%. Greater reasoning effort also improved evaluator--human agreement. Nevertheless, events near the plausibility boundary remain most difficult. They elicit the greatest human disagreement, and the best evaluator reaches only 57% accuracy on the implausible-hard condition, indicating a need for human input. Overall, STRIVE offers a scalable approach to reducing manual effort by automating initial event-set generation and evaluation for psycholinguistic studies.

Abstract

Figurative Language Understanding

Explore similar work

STRIVE: Probing Reasoning Limits in Graded Plausibility Generation and Evaluation

Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility

Plausibility as Commonsense Reasoning: Humans Succeed, Large Language Models Do not