Rethinking Metrics for Lexical Semantic Change Detection

Roksana Goworek; Haim Dubossarsky

cs.CLFeb 17, 2026

Rethinking Metrics for Lexical Semantic Change Detection

Authors: Roksana Goworek, Haim Dubossarsky

Organizations: Queen Mary University of London · The Alan Turing Institute · University of Cambridge

Abstract

Lexical semantic change detection (LSCD) increasingly relies on contextualised language model embeddings, yet most approaches still quantify change using a small set of semantic change metrics, primarily Average Pairwise Distance (APD) and cosine distance over word prototypes (PRT). We introduce Average Minimum Distance (AMD) and Symmetric Average Minimum Distance (SAMD), new measures that quantify semantic change via local correspondence between word usages across time periods. Across multiple languages, encoder models, and representation spaces, we show that AMD often provides more robust performance, particularly under dimensionality reduction and with non-specialised encoders, while SAMD excels with specialised encoders. We suggest that LSCD may benefit from considering alternative semantic change metrics beyond APD and PRT, with AMD offering a robust option for contextualised embedding-based analysis.

Figures & tables

Panel of Figure 1 — Figure 1: Illustrative schematic contrasting LSCD distance measures on usage embeddings from two corpora (blue: $C_{1}$ , yellow: $C_{2}$ ).

Figure 2: Performance (Spearman correlations) across metrics and spaces, averaged over languages and encoders. Std in brackets.

Figure 3: Performance of APD, PRT, AMD and SAMD using XL-LEXEME embeddings across representation spaces. Each row corresponds to a language; columns show metric–space combinations.

Panel of Figure 4 — Figure 4: Performance of APD, PRT, AMD and SAMD for non-specialised encoders.

Figure 5: Stress test of metrics under progressive dimensionality reduction. Spearman correlations are averaged across languages and encoders as embedding dimensionality is reduced by factors of two using PCA or random dimension selection.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Language	# Words	Defs / Word
English	37	$11.00\pm 5.36$
German	48	$3.75\pm 1.70$
Swedish	31	$4.26\pm 1.48$
Latin	40	$6.20\pm 1.14$
Spanish	100	$6.84\pm 4.14$
Chinese	40	$3.85\pm 1.71$

Appendix

Table 1: Number of target words and average number of generated definitions per word (mean

\pm

standard deviation) for each language.

Language	Prompt
English	Write all dictionary definitions of {WORD} in English. One sense per line. Output only the definitions. Do not add any text before or after the definitions.
German	Schreibe alle Wörterbuchdefinitionen von {WORD} auf Deutsch. Eine Bedeutung pro Zeile. Gib nur die Definitionen aus. Füge keinen Text vor oder nach den Definitionen hinzu.
Swedish	Skriv alla ordboksdefinitioner av {WORD} på svenska. En betydelse per rad. Skriv endast definitionerna. Lägg inte till någon text före eller efter definitionerna.
Latin	Scribe omnes definitiones dictionarii verbi {WORD} Latine. Una significatio per lineam. Redde tantum definitiones. Ne quidquam addas ante aut post definitiones.
Spanish	Escribe todas las definiciones de diccionario de {WORD} en español. Un significado por línea. Devuelve solo las definiciones. No añadas ningún texto antes ni después de las definiciones.
Chinese	用中文写出“ {WORD} ”的所有词典释义。每行一个义项。只输出释义，不要在释义前后添加任何文字。

Appendix

Table 2: Language-specific prompts used for dictionary definition generation.

Word	Earlier-associated definition	Later-associated definition
plane	A tool consisting of a block with a projecting blade, for shaping and smoothing a wooden surface.	To travel in an airplane.
record	To indicate or show (a reading) on a measuring instrument.	To state for the purpose of being set down in an official account.

Appendix

Table 3: Most discriminative definition dimensions identified by LDA in definition-based space for selected words. Negative weights are associated with the earlier corpus and positive weights with the later corpus.

Metric	Full	Definition	PCA	Random
Dominant NN share	.268 (.159)	.259 (.167)	.243 (.134)	.197 (.139)
Unused NN share	.616 (.069)	.578 (.100)	.583 (.090)	.497 (.049)
Average NN load	2.882 (.618)	3.175 (1.521)	2.923 (.941)	2.330 (.648)

Appendix

Table 4: Mean (standard deviation) of hubness measures across spaces, aggregated over languages and encoders.

Semantic Change Detection

Semantic change detection identifies how meanings, concepts, or scene contents differ across time in text, images, and other data.

See all 63 papers

Evaluating the Evaluator: Problems with SemEval-2020 Task 1 for Lexical Semantic Change Detection
Apr 14, 2026Bach Phan-Tat, Kris Heylen, Dirk Geeraerts +2Semantic Change Detection
Department of Linguistics, KU Leuven, Leuven, Belgium · Instituut voor de Nederlandse Taal, Leiden, The Netherlands · Department of Linguistics and Literary Studies, Vrije Universiteit Brussel, Brussels, Belgium
The BD-LSC Dataset: Facilitating the Benchmarking of Models for Lexical Semantic Change Detection in Slang and Standard Usage
Jun 15, 2026Afnan Aloraini, Riza Batista-NavarroSemantic Change Detection Word Sense Disambiguation
Department of Computer Science, The University of Manchester, Oxford Road, Manchester, M13 9PL, Greater Manchester, United Kingdom · Department of Computer Science, Qassim University, Al-Mulaida, 51425, Saudi Arabia
When Lexical Change Misleads: Rethinking Dynamic Topic Model Evaluation with Traditional and LLM-Based Metrics
Aug 13, 2026Charu Karakkaparambil JamesTopic Modeling
RPTU University Kaiserslautern-Landau Kaiserslautern, Germany

Apr 14, 2026J/K move · Enter open · S save

Evaluating the Evaluator: Problems with SemEval-2020 Task 1 for Lexical Semantic Change Detection

Bach Phan-Tat, Kris Heylen, Dirk Geeraerts, Stefano De Pascale+1

Department of Linguistics, KU Leuven, Leuven, Belgium · Instituut voor de Nederlandse Taal, Leiden, The Netherlands · Department of Linguistics and Literary Studies, Vrije Universiteit Brussel, Brussels, Belgium

Semantic Change Detection

Open paper AskPDF arXiv

This discussion paper re-examines SemEval-2020 Task 1, the most influential shared benchmark for lexical semantic change detection, through a three-part evaluative framework: operationalisation, data quality, and benchmark design. First, at the level of operationalisation, we argue that the benchmark models semantic change mainly as gain, loss, or redistribution of discrete senses. While practical for annotation and evaluation, this framing is too narrow to capture gradual, constructional, collocational, and discourse-level change. Also, the gold labels are outcomes of annotation decisions, clustering procedures, and threshold settings, which could potentially limit the validity of the task. Second, at the level of data quality, we show that the benchmark is affected by substantial corpus and preprocessing problems, including OCR noise, malformed characters, truncated sentences, inconsistent lemmatisation, POS-tagging errors, and missed targets. These issues can distort model behaviour, complicate linguistic analysis, and reduce reproducibility. Third, at the level of bench-mark design, we argue the small curated target sets and limited language coverage reduce realism and increase statistical uncertainty. Taken together, these limitations suggest that the benchmark should be treated as a useful but partial test bed rather than a definitive measure of progress. We therefore call for future datasets and shared tasks to adopt broader theories of semantic change, document pre-processing transparently, expand cross-linguistic coverage, and use more realistic evaluation settings. Such steps are necessary for more valid, interpretable, and generalisable progress in lexical semantic change detection

Abstract

Semantic Change Detection

Explore similar work

Evaluating the Evaluator: Problems with SemEval-2020 Task 1 for Lexical Semantic Change Detection

The BD-LSC Dataset: Facilitating the Benchmarking of Models for Lexical Semantic Change Detection in Slang and Standard Usage

When Lexical Change Misleads: Rethinking Dynamic Topic Model Evaluation with Traditional and LLM-Based Metrics