Learning the Cue or Learning the Word? Analyzing Generalization in Metaphor Detection for Verbs

Sinan Kurtyigit; Sabine Schulte im Walde; Alexander Fraser

cs.CLApr 15, 2026

Learning the Cue or Learning the Word? Analyzing Generalization in Metaphor Detection for Verbs

Authors: Sinan Kurtyigit, Sabine Schulte im Walde, Alexander Fraser

Organizations: School of Computation, Information and Technology, Technical University of Munich · Institute for Natural Language Processing, University of Stuttgart

Abstract

Metaphor detection models achieve strong benchmark performance, yet it remains unclear whether this reflects transferable generalization or lexical memorization. To address this, we analyze generalization in metaphor detection through RoBERTa, the shared backbone of many state-of-the-art systems, focusing on English verbs using the VU Amsterdam Metaphor Corpus. We introduce a controlled lexical hold-out setup where all instances of selected target lemmas are strictly excluded from fine-tuning, and compare predictions on these Held-out lemmas against Exposed lemmas (verbs seen during fine-tuning). While the model performs best on Exposed lemmas, it maintains robust performance on Held-out lemmas. Further analysis reveals that sentence context alone is sufficient to match full-model performance on Held-out lemmas, whereas static verb-level embeddings are not. Together, these results suggest that generalization is primarily driven by "learning the cue" (transferable contextual patterns), while "learning the word" (verb-specific memorization) provides an additive boost when lexical exposure is available.

Figures & tables

Exposed Set ( $n=10$ per lemma)				Held-out Set ( $n=20$ per lemma)
Lemma	$N_{\text{VUA}}$	Met.% ${}_{\text{VUA}}$	Met.% ${}_{\text{Eval}}$	Lemma	$N_{\text{VUA}}$	Met.% ${}_{\text{VUA}}$	Met.% ${}_{\text{Eval}}$
Metaphorical-biased				Metaphorical-biased
face	11	91%	90%	break	25	92%	90%
base	10	90%	90%	share	21	86%	85%
consider	13	85%	80%	feel	82	82%	80%
follow	13	85%	80%	lose	29	79%	80%

Table 1: Target lemmas selected for the Exposed (left) and Held-out (right) evaluation sets.

N_{\text{VUA}}

indicates the original instance count in the VUA Verbs train split. Met.%

{}_{\text{VUA}}

and Met.%

{}_{\text{Eval}}

report the percentage of metaphorical instances before and after stratified downsampling, respectively.

Dataset Split	$N_{\text{Samples}}$	Met. %	$N_{\text{Lemmas}}$
Standard VUA
Train	17240	28%	2190
Test	5873	30%	1111
Lexical Hold-out
Filtered Train	14882	26%	2163
Exposed Eval.	300	49%	30

Table 2: Statistics for the VUA Verbs datasets, reporting the number of samples, proportion of metaphorical labels, and count of unique target lemmas. The top section displays the standard benchmark splits, while the bottom section displays the constructed lexical hold-out splits.

Figure 1: Model architecture. The input sentence is tokenized and passed through the RoBERTa encoder. Hidden states corresponding to the target verb’s sub-word tokens (e.g., unravel + ed ) are aggregated via mean pooling into a Contextualized Target Representation , which is passed to a linear classifier to predict metaphoricity.

Seed	Prec.	Rec.	F1
9999	.738	.676	.706
314	.760	.667	.710
999	.736	.689	.712
7	.706	.721	.713
5555	.674	.763	.716
42	.681	.755	.716

Table 3: Model performance across 10 random seeds on the validation split. The table reports Precision, Recall, and F1 scores for each run, sorted by F1. Summary statistics (Mean, Std) are provided at the bottom, and the selected median seed (42) is highlighted.

Model	Prec.	Rec.	F1
Shared Task Baselines
DeepMet	.789	.819	.804
Go Figure!	.732	.823	.775
Our Models
Ours (Standard)	.681	.755	.716
Ours (Filtered)	.754	.663	.706

Table 4: Performance on the VUA Verbs Test Set reporting Precision, Recall, and F1. The top section lists results from the 2020 VUA Shared Task. The bottom section displays our model trained on the full dataset (Standard) and the filtered train set (filtered).

Evaluation Set	Prec.	Rec.	F1
Full Model
Exposed	.864	.776	.817
Held-out	.811	.573	.672
Context-only
Exposed	.743	.748	.746
Held-out	.714	.650	.681

Table 5: Performance on the Exposed and Held-out evaluation sets, reporting Precision, Recall, and F1. The table is divided by model variant to isolate information sources: the Full Model (top) serves as the primary benchmark; Context-only (middle) assesses the sufficiency of structural cues; and Word-only (bottom) evaluates the predictive power of lexical identity alone. The bottom row presents the Random Baseline.

Evaluation Set	Purity	$k$ -NN F1
Full Model
Exposed	.789	.788
Held-out	.690	.637
Context-only
Exposed	.638	.505
Held-out	.624	.544

Table 6: Geometric analysis of the Contextualized Target Representations using a

k

-Nearest Neighbor probe (

k=10

). We report Neighborhood Purity and

k

-NN F1 score for Exposed and Held-out verbs, comparing the Full Model (top) against the Context-only variant (bottom) to assess the impact of verb identity on the embedding space structure.

Figurative Language Understanding

Figurative language understanding studies how people and AI interpret nonliteral meaning in metaphors, idioms, proverbs, puns, and related forms.

See all 45 papers

Cross-Dataset Stability of Expert-Informed Skill Prompting and Fine-Tuning for Chinese Metaphor Identification
Aug 26, 2026Yufeng Wu, Meichun LiuFigurative Language Understanding Cross-Dataset Generalization
City University of Hong Kong
A Reproducible Multi-Architecture Baseline for Token-Level Chinese Metaphor Identification under the MIPVU Framework
May 8, 2026Yufeng Wu
City University of Hong Kong
M $^3$ R-Bench: A Unified Benchmark for Evidence-Grounded Multimodal Metaphor Understanding
Aug 6, 2026Hong Jiang, Junnan Zhu, Jingwang Huang +9Figurative Language Understanding Multimodal Reasoning
School of Computer Science, Chongqing University, Chongqing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Institute of Advanced Intelligence and Computing (IAIC), Agency for Science, Technology and Research (A*STAR), Singapore +3

Aug 26, 2026J/K move · Enter open · S save

Cross-Dataset Stability of Expert-Informed Skill Prompting and Fine-Tuning for Chinese Metaphor Identification

Yufeng Wu, Meichun Liu

City University of Hong Kong

Figurative Language Understanding Cross-Dataset Generalization Prompt Engineering LLM Fine-Tuning

Open paper AskPDF arXiv

Metaphor-identification performance can change markedly across datasets that differ in text distribution and annotation policy. We examine whether a fixed expert-informed procedure produces a more even cross-dataset profile than task-specific parameter adaptation. Four prespecified conditions are compared for Chinese sentence-level metaphor identification: BERT fine-tuning (BERT-FT), QLoRA-based large language model fine-tuning (LLM-FT), direct zero-shot LLM prompting (LLM-ZS), and zero-shot prompting with a frozen procedural Skill (Skill-ZS). The Skill operationalizes established criteria involving contextual meaning, basic meaning, contrast, and comparison. Evaluation covers CMRE Test and two external datasets, CCIME and CMC. Fine-tuned scores are means over three seeds, whereas each zero-shot score comes from one deterministic configuration. Fine-tuning remains strongest on the native test set: BERT-FT reaches 91.76 Macro-F1. LLM-FT has the highest external mean (83.52), while Skill-ZS is close at 82.92 and has both the highest external floor (82.64) and the smallest observed range across all three datasets (4.08 points). In the matched zero-shot comparison, adding the Skill reduces metaphorical predictions on every dataset. This sharply lowers false positives on CCIME but increases false negatives on CMRE Test and CMC. The results position expert-informed Skill prompting as a complementary route to more even observed cross-dataset performance, while fine-tuning retains its advantage in native-data accuracy. To our knowledge, this is the first study to compare an expert-informed procedural Skill with task-specific fine-tuning in the same cross-dataset evaluation of Chinese sentence-level metaphor identification.

Abstract

Figurative Language Understanding

Explore similar work

Cross-Dataset Stability of Expert-Informed Skill Prompting and Fine-Tuning for Chinese Metaphor Identification

A Reproducible Multi-Architecture Baseline for Token-Level Chinese Metaphor Identification under the MIPVU Framework

M3^33R-Bench: A Unified Benchmark for Evidence-Grounded Multimodal Metaphor Understanding

M $^3$ R-Bench: A Unified Benchmark for Evidence-Grounded Multimodal Metaphor Understanding