Cross-Dataset Stability of Expert-Informed Skill Prompting and Fine-Tuning for Chinese Metaphor Identification

Yufeng Wu; Meichun Liu

cs.CLAug 26, 2026

Cross-Dataset Stability of Expert-Informed Skill Prompting and Fine-Tuning for Chinese Metaphor Identification

Authors: Yufeng Wu, Meichun Liu

Organizations: City University of Hong Kong

Abstract

Metaphor-identification performance can change markedly across datasets that differ in text distribution and annotation policy. We examine whether a fixed expert-informed procedure produces a more even cross-dataset profile than task-specific parameter adaptation. Four prespecified conditions are compared for Chinese sentence-level metaphor identification: BERT fine-tuning (BERT-FT), QLoRA-based large language model fine-tuning (LLM-FT), direct zero-shot LLM prompting (LLM-ZS), and zero-shot prompting with a frozen procedural Skill (Skill-ZS). The Skill operationalizes established criteria involving contextual meaning, basic meaning, contrast, and comparison. Evaluation covers CMRE Test and two external datasets, CCIME and CMC. Fine-tuned scores are means over three seeds, whereas each zero-shot score comes from one deterministic configuration. Fine-tuning remains strongest on the native test set: BERT-FT reaches 91.76 Macro-F1. LLM-FT has the highest external mean (83.52), while Skill-ZS is close at 82.92 and has both the highest external floor (82.64) and the smallest observed range across all three datasets (4.08 points). In the matched zero-shot comparison, adding the Skill reduces metaphorical predictions on every dataset. This sharply lowers false positives on CCIME but increases false negatives on CMRE Test and CMC. The results position expert-informed Skill prompting as a complementary route to more even observed cross-dataset performance, while fine-tuning retains its advantage in native-data accuracy. To our knowledge, this is the first study to compare an expert-informed procedural Skill with task-specific fine-tuning in the same cross-dataset evaluation of Chinese sentence-level metaphor identification.

Figures & tables

Dataset	Total	Metaphorical	Non-metaphorical
CMRE Train	6,794	3,397	3,397
CMRE Dev	850	425	425
CMRE Test	850	425	425
CCIME	1,162	567	595
CMC	698	274	424

TABLE I: Dataset composition after CMRE Train/Dev overlap removal.

Condition	Labels	Updated	Skill	Runs
BERT-FT	CMRE	full model	no	3
LLM-FT	CMRE	QLoRA	no	3
LLM-ZS	none	none	no	1
Skill-ZS	none	none	yes	1

TABLE II: Knowledge-supply regimes and formal run basis.

Fig. 1: Complementary profiles. (A) Macro-F1 by dataset. (B) CMRE Test Macro-F1 versus the observed external floor; the diagonal marks equality. (C) Rates of metaphorical predictions for the matched zero-shot pair and gold prevalence; arrows run from LLM-ZS to Skill-ZS. Whiskers show sample SD only for the three-seed fine-tuned conditions.

Method	Ext. mean	Ext. floor	Native $-$ ext. mean	Range
BERT-FT	80.69	76.70	11.07	15.06
LLM-FT	83.52	76.78	7.65	14.39
LLM-ZS	78.75	66.74	3.12	24.02
Skill-ZS	82.92	82.64	-3.80	4.08

TABLE III: Observed cross-dataset stability criteria in Macro-F1 points, computed from unrounded values. Native

-

ext. mean is signed.

Dataset	Method	Accuracy	P	R	F1	Macro-F1
CMRE Test	BERT-FT	$\mathbf{91.76{\pm}0.51}$	$\mathbf{92.70{\pm}0.16}$	$90.67{\pm}1.30$	$\mathbf{91.67{\pm}0.59}$	$\mathbf{91.76{\pm}0.51}$
	LLM-FT	$91.18{\pm}0.51$	$88.95{\pm}0.62$	$\mathbf{94.04{\pm}0.59}$	$91.42{\pm}0.49$	$91.17{\pm}0.51$
	LLM-ZS	81.88	83.62	79.29	81.40	81.87
	Skill-ZS	79.41	88.58	67.53	76.64	79.12
CCIME	BERT-FT	$76.97{\pm}0.28$	$70.85{\pm}0.67$	$89.71{\pm}1.17$	$79.17{\pm}0.12$	$76.70{\pm}0.34$
	LLM-FT	$77.48{\pm}2.46$	$69.35{\pm}2.57$	$\mathbf{96.77{\pm}1.37}$	$80.77{\pm}1.67$	$76.78{\pm}2.77$

TABLE IV: Classification performance (%). P, R, and F1 refer to the metaphorical class. Fine-tuned values are means

\pm

sample SDs over three seeds; zero-shot values are point estimates. Bold marks the highest observed value in each dataset–metric column.

Dataset	Method	TN	FP	FN	TP	Predicted metaphorical	Gold metaphorical
CMRE Test	LLM-ZS	359	66	88	337	47.41%	50.00%
CMRE Test	Skill-ZS	388	37	138	287	38.12%	50.00%
CCIME	LLM-ZS	276	319	51	516	71.86%	48.80%
CCIME	Skill-ZS	446	149	52	515	57.14%	48.80%
CMC	LLM-ZS	387	37	25	249	40.97%	39.26%
CMC	Skill-ZS	399	25	82	192	31.09%	39.26%

TABLE V: Aggregate confusion counts and rates of metaphorical predictions for matched zero-shot conditions.

Figurative Language Understanding

Figurative language understanding studies how people and AI interpret nonliteral meaning in metaphors, idioms, proverbs, puns, and related forms.

See all 43 papers

Interpretable Chinese Metaphor Identification via LLM-Assisted MIPVU Rule Script Generation: A Comparative Protocol Study
Mar 11, 2026Weihang Huang, Mengna LiuFigurative Language Understanding Interpretable ML
University of Birmingham Birmingham, UK · Guangdong University of Foreign Studies Guangzhou, China
A Reproducible Multi-Architecture Baseline for Token-Level Chinese Metaphor Identification under the MIPVU Framework
May 8, 2026Yufeng Wu
City University of Hong Kong
MetFuse: Figurative Fusion between Metonymy and Metaphor
Apr 14, 2026Saptarshi Ghosh, Tianyu JiangFigurative Language Understanding
University of Cincinnati

Mar 11, 2026J/K move · Enter open · S save

Interpretable Chinese Metaphor Identification via LLM-Assisted MIPVU Rule Script Generation: A Comparative Protocol Study

Weihang Huang, Mengna Liu

University of Birmingham Birmingham, UK · Guangdong University of Foreign Studies Guangzhou, China

Figurative Language Understanding Interpretable ML LLM-Assisted Annotation

Open paper AskPDF arXiv

Metaphor identification is a foundational task in figurative language processing, yet most computational approaches operate as opaque classifiers offering no insight into why an expression is judged metaphorical. This interpretability gap is especially acute for Chinese, where rich figurative traditions, absent morphological cues, and limited annotated resources compound the challenge. We present an LLM-assisted pipeline that operationalises four metaphor identification protocols--MIP/MIPVU lexical analysis, CMDAG conceptual-mapping annotation, emotion-based detection, and simile-oriented identification--as executable, human-auditable rule scripts. Each protocol is a modular chain of deterministic steps interleaved with controlled LLM calls, producing structured rationales alongside every classification decision. We evaluate on seven Chinese metaphor datasets spanning token-, sentence-, and span-level annotation, establishing the first cross-protocol comparison for Chinese metaphor identification. Within-protocol evaluation shows Protocol A (MIP) achieves an F1 of 0.472 on token-level identification, while cross-protocol analysis reveals striking divergence: pairwise Cohen's kappa between Protocols A and D is merely 0.001, whereas Protocols B and C exhibit near-perfect agreement (kappa = 0.986). An interpretability audit shows all protocols achieve 100% deterministic reproducibility, with rationale correctness from 0.40 to 0.87 and editability from 0.80 to 1.00. Error analysis identifies conceptual-domain mismatch and register sensitivity as dominant failure modes. Our results demonstrate that protocol choice is the single largest source of variation in metaphor identification, exceeding model-level variation, and that rule-script architectures achieve competitive performance while maintaining full transparency.

Abstract

Figurative Language Understanding

Explore similar work

Interpretable Chinese Metaphor Identification via LLM-Assisted MIPVU Rule Script Generation: A Comparative Protocol Study

A Reproducible Multi-Architecture Baseline for Token-Level Chinese Metaphor Identification under the MIPVU Framework

MetFuse: Figurative Fusion between Metonymy and Metaphor