cs.CLMar 11, 2026

Interpretable Chinese Metaphor Identification via LLM-Assisted MIPVU Rule Script Generation: A Comparative Protocol Study

Authors: Weihang Huang, Mengna Liu

Organizations: University of Birmingham Birmingham, UK · Guangdong University of Foreign Studies Guangzhou, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Metaphor identification is a foundational task in figurative language processing, yet most computational approaches operate as opaque classifiers offering no insight into why an expression is judged metaphorical. This interpretability gap is especially acute for Chinese, where rich figurative traditions, absent morphological cues, and limited annotated resources compound the challenge. We present an LLM-assisted pipeline that operationalises four metaphor identification protocols--MIP/MIPVU lexical analysis, CMDAG conceptual-mapping annotation, emotion-based detection, and simile-oriented identification--as executable, human-auditable rule scripts. Each protocol is a modular chain of deterministic steps interleaved with controlled LLM calls, producing structured rationales alongside every classification decision. We evaluate on seven Chinese metaphor datasets spanning token-, sentence-, and span-level annotation, establishing the first cross-protocol comparison for Chinese metaphor identification. Within-protocol evaluation shows Protocol A (MIP) achieves an F1 of 0.472 on token-level identification, while cross-protocol analysis reveals striking divergence: pairwise Cohen's kappa between Protocols A and D is merely 0.001, whereas Protocols B and C exhibit near-perfect agreement (kappa = 0.986). An interpretability audit shows all protocols achieve 100% deterministic reproducibility, with rationale correctness from 0.40 to 0.87 and editability from 0.80 to 1.00. Error analysis identifies conceptual-domain mismatch and register sensitivity as dominant failure modes. Our results demonstrate that protocol choice is the single largest source of variation in metaphor identification, exceeding model-level variation, and that rule-script architectures achieve competitive performance while maintaining full transparency.

Figures & tables

DatasetTotalMetaphorLiteralOtherMetaphor%Label Type
PSU CMC35,7453,27232,432419.2%Token
CMC8,0277,312715091.1%Sentence
CMDAG34,46334,46300100%Span
Chinese Simile334,069144,358189,711043.2%Span
NLPCC 2024 T935,46335,46300100%Span
ConFiguRe9,0104,35404,65648.3%Span
Table 1: Dataset statistics. Metaphor% indicates the proportion of metaphor-positive instances. Label type denotes the annotation granularity: Token (word-level binary labels), Sentence (sentence-level binary labels), or Span (annotated text spans such as tenor and vehicle).
ProtocolEvaluationPRF1Acc
A (MIP)Token-level, PSU CMC0.4470.5000.4720.898
A (MIP) by registerAcademic0.6610.5460.5980.900
Fiction0.3100.4280.3600.885
 News0.3430.4900.4040.908
B (CMDAG)Sentence-level, ChineseMCorpus0.6400.2380.3470.423
B (CMDAG) spanVehicle partial F1, CMDAG——0.338—
Table 2: Within-protocol evaluation results. Protocol A is evaluated at token level on PSU CMC; Protocol B at sentence level on ChineseMCorpus and span level on CMDAG; Protocol C at sentence level on ChineseMCorpus; Protocol D at span level on Chinese Simile.
ProtocolPRF1
A (MIP)0.7700.8990.829
B (CMDAG)0.8390.1910.311
C (Emotion)0.8430.1870.307
D (Simile)0.9090.0090.018
Table 3: Cross-protocol comparison on 1,723 PSU CMC sentences (sentence-level classification). A sentence is metaphorical if it contains ≥1\geq 1 MRW token in the gold standard.
 ABCD
A (MIP)1.0000.0630.0610.001
B (CMDAG)0.0631.0000.9860.072
C (Emotion)0.0610.9861.0000.074
D (Simile)0.0010.0720.0741.000
Table 4: Pairwise Cohen’s κ\kappa between protocols on 1,723 PSU CMC sentences. Values interpreted following Landis and Koch [1977] : << 0.00 poor, 0.00–0.20 slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, 0.81–1.00 almost perfect.
SystemF1
Majority baseline0.000
Random baseline0.087
Simple heuristic0.008
Our system Protocol A0.472
Literature estimates
BERT fine-tuned (approx.)∼\sim 0.65
Table 5: Baseline comparison on PSU CMC token-level metaphor identification. Literature estimates are approximate values reported in or derived from comparable studies on Chinese metaphor detection.
ProtocolRationaleDeterministicEditability
A (MIP)0.64100%0.80
B (CMDAG)0.40100%0.90
C (Emotion)0.51100%1.00
D (Simile)0.87100%1.00
Table 6: Interpretability assessment scores across four protocols. Rationale correctness is based on manual evaluation of 50 sampled decisions per protocol. Determinism measures exact reproducibility. Editability measures the success rate of targeted protocol modifications.
More in this topic

Figurative Language Understanding

Figurative language understanding studies how people and AI interpret nonliteral meaning in metaphors, idioms, proverbs, puns, and related forms.

See all 43 papers

Explore similar work

CardsList
May 8, 2026

A Reproducible Multi-Architecture Baseline for Token-Level Chinese Metaphor Identification under the MIPVU Framework

Metaphor is pervasive in everyday language, yet token-level computational identification of metaphor-related words in Chinese under the MIPVU framework remains under-explored relative to English. This paper presents a reproducible multi-architecture baseline for token-level metaphor identification on the PSU Chinese Metaphor Corpus (PSU CMC), the only widely available MIPVU-annotated Chinese corpus. We systematically compare three model families: (i) encoder fine-tuning with Chinese RoBERTa-wwm-ext-large; (ii) MelBERT adapted to Chinese using a newly constructed basic-meaning resource derived from the Modern Chinese Dictionary, 7th edition (MCD7), comprising 74,823 entries with 71.51% PSU CMC vocabulary coverage; and (iii) Qwen3.5-9B fine-tuned with QLoRA as an instruction-tuned generative baseline. Across five fixed seeds, MelBERT MIP-only achieves the strongest performance at 0.7281 +/- 0.0050 test positive F1, marginally above MelBERT Full (0.7270 +/- 0.0069) and clearly above plain RoBERTa (0.7142 +/- 0.0121). The Qwen QLoRA generative configuration trails encoder baselines by approximately 11 F1 points (0.6157 +/- 0.0113). Three findings merit attention: (1) the SPV channel of MelBERT does not contribute reliable positive signal in Chinese, consistent with the dominance of conventional metaphor; (2) the Qwen-encoder gap is concentrated in recall, reflecting the discrete-commitment limitation of generative output; (3) several Qwen task formulations fail due to format design rather than model capacity. We release all split manifests, per-seed outputs, the MCD7 basic-meaning embedding pipeline, and training scripts to serve as a common reference for future Chinese metaphor identification research.

Yufeng Wu

City University of Hong Kong

Topic pending
AskPDF
Aug 26, 2026

Cross-Dataset Stability of Expert-Informed Skill Prompting and Fine-Tuning for Chinese Metaphor Identification

Metaphor-identification performance can change markedly across datasets that differ in text distribution and annotation policy. We examine whether a fixed expert-informed procedure produces a more even cross-dataset profile than task-specific parameter adaptation. Four prespecified conditions are compared for Chinese sentence-level metaphor identification: BERT fine-tuning (BERT-FT), QLoRA-based large language model fine-tuning (LLM-FT), direct zero-shot LLM prompting (LLM-ZS), and zero-shot prompting with a frozen procedural Skill (Skill-ZS). The Skill operationalizes established criteria involving contextual meaning, basic meaning, contrast, and comparison. Evaluation covers CMRE Test and two external datasets, CCIME and CMC. Fine-tuned scores are means over three seeds, whereas each zero-shot score comes from one deterministic configuration. Fine-tuning remains strongest on the native test set: BERT-FT reaches 91.76 Macro-F1. LLM-FT has the highest external mean (83.52), while Skill-ZS is close at 82.92 and has both the highest external floor (82.64) and the smallest observed range across all three datasets (4.08 points). In the matched zero-shot comparison, adding the Skill reduces metaphorical predictions on every dataset. This sharply lowers false positives on CCIME but increases false negatives on CMRE Test and CMC. The results position expert-informed Skill prompting as a complementary route to more even observed cross-dataset performance, while fine-tuning retains its advantage in native-data accuracy. To our knowledge, this is the first study to compare an expert-informed procedural Skill with task-specific fine-tuning in the same cross-dataset evaluation of Chinese sentence-level metaphor identification.

Yufeng Wu, Meichun Liu

City University of Hong Kong

Figurative Language UnderstandingCross-Dataset GeneralizationPrompt EngineeringLLM Fine-Tuning
AskPDF
Sep 15, 2026

AraMIP: Extending MIPVU Towards Metaphor Identification in Arabic

Metaphor research has gained increasing attention due to its relevance to linguistic creativity, language use, cognitive processes, and related areas. While many efforts have been devoted to metaphor identification and annotation in English and other languages, Arabic remains under-resourced in this area. In this work, we propose the Arabic Metaphor Identification Procedure (AraMIP), a novel guideline for Arabic metaphor annotation. AraMIP builds on the widely used Metaphor Identification Procedure Vrije Universiteit (MIPVU) framework, incorporating adaptations that accounts for the language-specific properties of Arabic. We distinguish three major types of Arabic figurative language: Isti'ara (metaphor), kinaya (metonymy/indirect expression), and tashbih (simile), and annotate a pilot dataset of 300 sentences (5277 words). Our analysis reveals key challenges specific to Arabic, including morphological complexity, inconsistencies in dictionary sense ordering, and the absence of standardized contextual materials for annotators. This work contributes a first step toward standardized Arabic figurative instances and facilitates the development of larger annotated resources, thereby supporting future research on figurative language in Arabic.

Mandar Marathe, Manar Ali, Sara Nabhani +3

1SOAS University of London, United Kingdom · 2CRC 1646 ‘Linguistic Creativity in Communication’, Bielefeld University, Germany · 3Computational Linguistics, CLCG, University of Groningen, The Netherlands +2

Figurative Language UnderstandingArabic NLPLow-Resource Language Processing
AskPDF