Span Modeling for Idiomaticity and Figurative Language Detection with Span Contrastive Loss

Blake Matheny; Phuong Minh Nguyen; Minh Le Nguyen

cs.CLMar 24, 2026

Span Modeling for Idiomaticity and Figurative Language Detection with Span Contrastive Loss

Authors: Blake Matheny, Phuong Minh Nguyen, Minh Le Nguyen

Organizations: Information Science, Japan Advanced Institute of Science and Technology, Ishikawa, Japan.

Abstract

The category of figurative language contains many varieties, some of which are non-compositional in nature. This type of phrase or multi-word expression (MWE) includes idioms, which represent a single meaning that does not consist of the sum of its words. For language models, this presents a unique problem due to tokenization and adjacent contextual embeddings. Many large language models have overcome this issue with large phrase vocabulary, though immediate recognition frequently fails without one- or few-shot prompting or instruction finetuning. The best results have been achieved with BERT-based or LSTM finetuning approaches. The model in this paper contains one such variety. We propose BERT- and RoBERTa-based models finetuned with a combination of slot loss and span contrastive loss (SCL) with hard negative reweighting to improve idiomaticity detection, attaining state of the art sequence accuracy performance on existing datasets. Comparative ablation studies show the effectiveness of SCL and its generalizability. The geometric mean of F1 and sequence accuracy (SA) is also proposed to assess a model's span awareness and general performance together.

Figures & tables

Dataset	Task	Primary	Best
EPIE (Formal)	Span tagging (BIO)	Acc.	98.00 *
MAGPIE (random)	Span id. (exact)	Sequence Accuracy	91.51 Yayavaram et al. (2024)
MAGPIE (random)	Span id.	Sequence Accuracy	92.97 Matheny et al. (2025)
MAGPIE (type-aware)	Span id. (unseen types)	Sequence Accuracy	70.47 Zeng and Bhat (2021)
SemEval-2022 2A	Idiomaticity existence	Macro-F1 (All)	93.85 *
FLUTE (Idioms)	Sentence-pair entailment	Accuracy	79.20 *

Table 1: Current SOTA of idiom and figurative language datasets. The asterisk (*) denotes that the result is taken from the corresponding dataset paper.

Panel of Figure 1 — Figure 1: Coefficient ablation results for all datasets.

Figure 2: Difference between test and development set sequence accuracy and F1 evaluation for best coefficient of each model with SCL-neg architecture .

Figure 3: Visualization of F1 cross evaluation results on the test sets of each dataset.

Figure 4: Visualization of SA cross evaluation results on the test sets of each dataset.

Model	$\mu_{\mathrm{SA}}$	$\mu_{\mathrm{F1}}$	$\mu_{\mathrm{GM}}$	Hardest ds.	$R_{\mathrm{SA}}$	$R_{\mathrm{F1}}$	$R_{\mathrm{GM}}$
IFL-C4-A-BERT	0.3762	0.3112	0.3421	FLUTE	0.0968	0.0229	0.0471
IFL-C4-A-RoBERTa	0.3513	0.2940	0.3214	SEMEVAL2022	0.0862	0.0000	0.0000
FLUTE-BERT	0.1910	0.1960	0.1935	SEMEVAL2022	0.0906	0.0206	0.0432
FLUTE-RoBERTa	0.1813	0.1358	0.1569	PIFL-OSCAR	0.0508	0.0249	0.0356
MAGPIE (random)-BERT	0.4575	0.4558	0.4566	SEMEVAL2022	0.1446	0.0714	0.1016
MAGPIE (random)-RoBERTa	0.4459	0.4607	0.4532	SEMEVAL2022	0.1366	0.0689	0.0970

Table 2: Summary statistics computed on test splits only.

\mu

is the mean across the six evaluation datasets (rows). The robustness factor

R

is the model’s performance on the most difficult dataset, as the minimum geometric mean

\sqrt{\mathrm{SA}\cdot\mathrm{F1}}

across datasets;

R_{\mathrm{SA}}

and

R_{\mathrm{F1}}

are the corresponding minima over datasets, and

R_{\mathrm{GM}}=\sqrt{R_{\mathrm{SA}}\cdot R_{\mathrm{F1}}}

Model	$\mu_{\mathrm{SA}}$	$\mu_{\mathrm{F1}}$	$\mu_{\mathrm{GM}}$	Hardest ds.	$R_{\mathrm{SA}}$	$R_{\mathrm{F1}}$	$R_{\mathrm{GM}}$
IFL-C4-A-BERT 1.0	0.3749	0.3104	0.3202	FLUTE	0.0955	0.0285	0.0522
IFL-C4-A-RoBERTa 1.0	0.3595	0.2993	0.2916	SEMEVAL2022	0.0862	0.0000	0.0000
FLUTE-BERT 0.1	0.2253	0.2509	0.2372	SEMEVAL2022	0.0381	0.0258	0.0314
FLUTE-RoBERTa 0.1	0.2320	0.2257	0.2263	SEMEVAL2022	0.0879	0.0317	0.0528
MAGPIE (random)-BERT 0.1	0.4677	0.4811	0.4676	FLUTE	0.1472	0.0783	0.1074
MAGPIE (random)-RoBERTa 0.5	0.4670	0.4805	0.4674	SEMEVAL2022	0.1525	0.0363	0.0744

Table 3: SCL-neg (test only).

\mu

is the mean across evaluation datasets.

\mu_{\mathrm{GM}}

is the mean of

\sqrt{\mathrm{SA}\cdot\mathrm{F1}}

across datasets. The hardest dataset is the dataset attaining the minimum per-dataset GM for the model.

R_{\mathrm{SA}}

and

R_{\mathrm{F1}}

are the minima across datasets (test), and

R_{\mathrm{GM}}=\sqrt{R_{\mathrm{SA}}\cdot R_{\mathrm{F1}}}

Figure 5: Difference between the best coefficient of span contrastive loss and baseline results.

Panel of Figure 6 — Figure 6: t-SNE visualization of word and span embeddings for MAGPIE (random) trained models evaluated on the MAGPIE (random) dataset.

Panel of Figure 7 — Figure 7: PCA visualization of embeddings for MAGPIE (random) trained models evaluated on the MAGPIE (random) dataset.

MAGPIE (random) - Over prediction
Sentence	i suppose you get used to it after a while , sleeping in the day .
Gold Labels - (none)	i suppose you get used to it after a while , sleeping in the day .
BERT baseline - incorrect	I suppose you get used to it after a while , B-idiom[sleeping] I-idiom[in] I-idiom[the] I-idiom[day] .
RoBERTa - incorrect	I suppose you get used to it after a while , B-idiom[sleeping] I-idiom[in] I-idiom[the] I-idiom[day] .
BERT SCL-neg - incorrect	I suppose you get used to it after a while , B-idiom[sleeping] I-idiom[in] I-idiom[the] I-idiom[day] .
RoBERTa SCL-neg - incorrect	I suppose you get used to it after a while , B-idiom[sleeping] I-idiom[in] I-idiom[the] I-idiom[day] .

Table 4: Grouped examples by dataset and instance ID: sentence, gold labels, and predictions from BERT and RoBERTa.

Figurative Language Understanding

Figurative language understanding studies how people and AI interpret nonliteral meaning in metaphors, idioms, proverbs, puns, and related forms.

See all 45 papers

IdiomX A Multilingual Benchmark for Idiom Understanding, Retrieval, and Interpretation
Apr 25, 2026Ayman Ali ShararaFigurative Language Understanding Multilingual Language Model Evaluation
MSc Data Science & Machine Learning (SPOC S21) Data ScienceTech Institute (DSTI)
IdioLink: Retrieving Meaning Beyond Words Across Idiomatic and Literal Expressions
May 21, 2026Kai Golan Hashiloni, Daniel Fadlon, Lior Livyatan +3Figurative Language Understanding Semantic Search
Data Science Institute, Reichman University, Herzliya, Israel · EfiArazi School of Computer Science, Reichman University, Herzliya, Israel · Vrije Universiteit Amsterdam, The Netherlands
When Meaning Isn't Literal: Exploring Idiomatic Meaning Across Languages and Modalities
Apr 12, 2026Sarmistha Das, Shreyas Guha, Suvrayan Bandyopadhyay +3Retrieval-Augmented Reasoning Figurative Language Understanding
Department of Computer Science and Engineering, Indian Institute of Technology Patna, Patna, India · School of Information Technology, King Mongkut’s Institute of Technology Ladkrabang, Bangkok, Thailand

Apr 25, 2026J/K move · Enter open · S save

IdiomX A Multilingual Benchmark for Idiom Understanding, Retrieval, and Interpretation

Ayman Ali Sharara

MSc Data Science & Machine Learning (SPOC S21) Data ScienceTech Institute (DSTI)

Figurative Language Understanding Multilingual Language Model Evaluation Multilingual IR

Open paper AskPDF arXiv

Idiomatic expressions remain a persistent challenge for natural language processing because their meanings are often non-compositional, context-dependent, and difficult to align across languages. Existing idiom resources are often limited in scale, contextual diversity, or multilingual coverage, restricting their utility for modern language models. We introduce IdiomX, a large-scale multilingual benchmark for idiom understanding, retrieval, and interpretation, constructed through a reproducible multi-stage pipeline combining lexical resource extraction, large-scale normalization, controlled large language model enrichment, and structured validation. The resulting dataset contains over 190K contextualized examples spanning 12K+ idioms, with aligned English, Arabic, and French semantic representations, idiomatic and literal usage labels, and rich linguistic metadata. Building on this resource, we define a unified four-task benchmark covering idiom detection, context-to-idiom retrieval, Arabic-to-English idiom retrieval, and idiom interpretation, extending evaluation from figurative recognition to semantic grounding and explainable meaning retrieval. Experiments show that contextual transformer models substantially improve idiom detection, while hybrid retrieval and reranking architectures significantly strengthen both monolingual and cross-lingual idiom retrieval. Results further demonstrate that idiom interpretation can be effectively modeled as a semantic retrieval task, introducing interpretability as a complementary benchmark dimension. Overall, IdiomX provides a scalable benchmark for studying idiomatic language as a progression from detection to retrieval and semantic interpretation, and offers a modular framework extensible to additional languages and figurative reasoning tasks

Abstract

Figurative Language Understanding

Explore similar work

IdiomX A Multilingual Benchmark for Idiom Understanding, Retrieval, and Interpretation

IdioLink: Retrieving Meaning Beyond Words Across Idiomatic and Literal Expressions

When Meaning Isn't Literal: Exploring Idiomatic Meaning Across Languages and Modalities