In the LLM era, Word Sense Induction remains unsolved

Anna Mosolova; Marie Candito; Carlos Ramisch

cs.CLMar 12, 2026

In the LLM era, Word Sense Induction remains unsolved

Authors: Anna Mosolova, Marie Candito, Carlos Ramisch

Organizations: Université Paris Cité, CNRS, LLF, Paris, France · Aix Marseille Univ, CNRS, LIS, Marseille, France

Abstract

In the absence of sense-annotated data, word sense induction (WSI) is a compelling alternative to word sense disambiguation, particularly in low-resource or domain-specific settings. In this paper, we emphasize methodological problems in current WSI evaluation. We propose an evaluation on a SemCor-derived dataset, respecting the original corpus polysemy and frequency distributions. We assess pre-trained embeddings and clustering algorithms across parts of speech, and propose and evaluate an LLM-based WSI method for English. We evaluate data augmentation sources (LLM-generated, corpus and lexicon), and semi-supervised scenarios using Wiktionary for data augmentation, must-link constraints, number of clusters per lemma. We find that no unsupervised method (whether ours or previous) surpasses the strong "one cluster per lemma" heuristic (1cpl). We also show that (i) results and best systems may vary across POS, (ii) LLMs have troubles performing this task, (iii) data augmentation is beneficial and (iv) capitalizing on Wiktionary does help. It surpasses previous SOTA system on our test set by 3.3%. WSI is not solved, and calls for a better articulation of lexicons and LLMs' lexical semantics capabilities.

Figures & tables

Metric	H	C	RB	SQ
Rand index	$\surd$	$\surd$	$\times$	$\times$
Paired F-score	$\surd$	$\surd$	$\times$	$\times$
NMI	$\surd$	$\times$	$\times$	$\surd$
V-measure	$\surd$	$\surd$	$\times$	$\surd$
B 3 Precision	$\surd$	$\times$	$\surd$	$\times$
B 3 Recall	$\times$	$\surd$	$\times$	$\surd$

Table 1: Sensibility of clustering metrics to the properties defined by Amigó et al. (2008) . Table 9 in Appendix B illustrates these properties on use cases.

Model	Algo	Verb	Adj	Noun	All POS	w Avg
Unsupervised
BERT-b-u	AG silh	64.8	75.6	72.3	70.6	70.8
BERT-b-u	X-Means	62.9[±0.1]	76.5 [±2.2]	73.7[±0.3]	70.6[±0.8]	71.1
BERT-l-u	AG silh	65.8	75.7	72.3	71.1	71.1
BERT-l-u	X-Means	63.2[±0.6]	75.5[±1.2]	74.8 [±0.1]	70.2[±0.1]	71.5
Self-supervised

Table 3: F-B 3 performance across PLMs and clustering algorithms for each POS, for all POS (All POS), and the average over POS weighted by POS proportion in SemCor ( w Avg). In bold, the best value for each model supervision type (un-, self-, semi-supervised). In blue , the best value for each column, excluding baselines. In red , cases where a baseline is best over the column. Best previous system: PolyLM-large:

73.0

(Table 2 ).

Model	F-B 3
PolyLM-large	72.7
BERT-l-uncased+AG s +150WB	74.0
BERT-l-Wikt+AG wikt +ML+100WB	76.0
1cpl	74.4

Table 6: F-B 3 results on SemCor-WSI test set for all POS of the best previous system (cf. Table 2 ) and the best unsupervised and supervised models from Tables 4 and 5 (for these models we reuse the best layer tuned on the development set).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	Testing set	Trial set
Instances	4664	400
AVG senses/inst.	1.12	4.97

Appendix

Table 8: Testing set details from SemEval 2013 paper Jurgens and Klapaftis (2013) , trial set details computed on the provided dataset. AVG senses/inst. : mean number of applicable senses per instance.

Panel of Table 9 — Table 9: Verification of clustering metrics properties proposed by Amigó et al. (2008) on their use cases (see figure 11 in their paper) for the metrics discussed in the § 3 . We also test the metrics on the 1 cluster per lemma and 1 cluster per instance solutions.

Model	Size on GPU	$\mathbf{\tau}$
Llama 3.1 8B	16GB	40m
Llama 3.3 70B	135GB	-
Llama 3.3 70B 4bit	38GB	3h40m
BERT-large 17 17 17 Number of parameters of BERT-large-uncased: 334M + AG s	3GB	5m45s
GPT-4o	UNK	45 min

Appendix

Table 10: GPU usage of each model, where

\tau

represents the SemCor-WSI processing time. Size on GPU indicates the model’s size, which may double during inference. All values are approximate and may vary slightly.

POS	%
Noun	0.49
Adjective	0.22
Verb	0.30

Appendix

Table 12: Percentage of each POS in SemCor Brown1 and Brown2 from which SemCor-WSI was composed.

Aug	Base	Must-link
	AG s	AG wikt	AG s
No	20	NA	NA
Wiktionary	20	17	20
Llama 3.1 8B 4bit	24	22	17
GPT-4o	24	19	24
WB (10 per l.)	16	24	22

Appendix

Table 14: For Table 4 , best layer for each data augmentation type on BERT-l-u + AG.

Aug	Base	Must-link
	AG s	AG wikt	AG s
No	23	NA	NA
Wiktionary	23	23	23
Llama 3.1 8B 4bit	24	23	22
GPT-4o	24	23	23
WB (10 per l.)	23	22	20

Appendix

Table 15: For Table 5 , best layer for each data augmentation type on BERT-l-Wikt + AG.

Figure 1: Bootstrapping distribution for 1000 runs using AG silh in combination with all PLMs.

Clustering

Clustering groups data points by shared structure, developing methods to discover, validate, and interpret groups across diverse data and constraints.

See all 335 papers

WiC is Not WSD: A Study on LLMs and Lexical Ambiguity Resolution
Sep 17, 2026Yi Zhou, Kiamehr Rezaee, Danushka Bollegala +2Open-Weight Language Models Word Sense Disambiguation
Cardiff University · University of Liverpool · Amazon
An Exploration-Analysis-Disambiguation Reasoning Framework for Word Sense Disambiguation with Low-Parameter LLMs
Mar 5, 2026Deshan Sumanathilaka, Nicholas Micallef, Julian HoughWord Sense Disambiguation Small Language Models
Department of Computer Science, Swansea University, Fabian Way, Crymlyn Burrows, Swansea, Wales, UK
Inspicio: Open-Vocabulary, LLM-Based Sense Retrieval for Historical Languages
Sep 1, 2026Michele CilettiLexical Retrieval Low-Resource Language Processing
Department of Humanities, University of Foggia / Via Arpi 176, 71121 Foggia, Italy

Sep 17, 2026J/K move · Enter open · S save

WiC is Not WSD: A Study on LLMs and Lexical Ambiguity Resolution

Yi Zhou, Kiamehr Rezaee, Danushka Bollegala, Mohammad Taher Pilehvar+1

Cardiff University · University of Liverpool · Amazon

Open-Weight Language Models Word Sense Disambiguation

Open paper AskPDF arXiv

Word-in-Context (WiC) remains challenging for language models, despite recent progress on lexical-semantic tasks. We hypothesise that this difficulty arises not only from comparing two contextual uses of a word, but also from the absence of an explicit sense inventory that specifies the relevant level of semantic granularity. We evaluate open LLMs on WiC and traditional Word Sense Disambiguation (WSD) under similar settings. We find that providing candidate senses, similar to what is done in traditional WSD, improves WiC performance in all settings. In general, explicit sense information helps models make more consistent and targeted judgements. Human evaluation further shows that many apparent WiC errors reflect label ambiguity or mismatches between model and annotator sense boundaries rather than simple failures of lexical understanding. In particular, results show that LLMs overthink the sense distinction often leading to errors based on overly fine-grained distinctions.

Abstract

Clustering

Explore similar work

WiC is Not WSD: A Study on LLMs and Lexical Ambiguity Resolution

An Exploration-Analysis-Disambiguation Reasoning Framework for Word Sense Disambiguation with Low-Parameter LLMs

Inspicio: Open-Vocabulary, LLM-Based Sense Retrieval for Historical Languages