An Exploration-Analysis-Disambiguation Reasoning Framework for Word Sense Disambiguation with Low-Parameter LLMs

Deshan Sumanathilaka; Nicholas Micallef; Julian Hough

cs.CLMar 5, 2026

An Exploration-Analysis-Disambiguation Reasoning Framework for Word Sense Disambiguation with Low-Parameter LLMs

Authors: Deshan Sumanathilaka, Nicholas Micallef, Julian Hough

Organizations: Department of Computer Science, Swansea University, Fabian Way, Crymlyn Burrows, Swansea, Wales, UK

Abstract

Word Sense Disambiguation (WSD) remains a key challenge in Natural Language Processing (NLP), especially when dealing with rare or domain-specific senses that are often misinterpreted. While modern high-parameter Large Language Models (LLMs) such as GPT-4-Turbo have shown state-of-the-art WSD performance, their computational and energy demands limit scalability. This study investigates whether low-parameter LLMs (<4B parameters) can achieve comparable results through fine-tuning strategies that emphasize reasoning-driven sense identification. Using the FEWS dataset augmented with semi-automated, rationale-rich annotations, we fine-tune eight small-scale open-source LLMs (e.g. Gemma and Qwen). Our results reveal that Chain-of-Thought (CoT)-based reasoning combined with neighbour-word analysis achieves performance comparable to GPT-4-Turbo in zero-shot settings. Importantly, Gemma-3-4B and Qwen-3-4B models consistently outperform all medium-parameter baselines and state-of-the-art models on FEWS, with robust generalization to unseen senses. Furthermore, evaluation on the unseen "Fool Me If You Can'' dataset confirms strong cross-domain adaptability without task-specific fine-tuning. This work demonstrates that with carefully crafted reasoning-centric fine-tuning, low-parameter LLMs can deliver accurate WSD while substantially reducing computational and energy demands.

Figures & tables

Dataset	Size (K)	Input		Output
		Max	Avg	Max	Avg
Direct sense	101	511	44.7	251	13.9
COT R	101	1915	226.1	1921	265.9
Advanced R	10	1865	212.6	2477	672.0
Verb R	4.5	1915	222.6	2607	764.4

Table 1: Stats for fine-tuning Variants. R: Reason

Models	Base Inference			Instructional Fine-Tuning
	Precision	Recall	F1 Score	Precision	Recall	F1 Score
Gemma 2 2B	0.7720	0.7341	0.7515	0.7870	0.7781	0.7814
Gemma 3 4B	0.7806	0.7302	0.7534	0.7741	0.7214	0.7458
Llama 3.2 1B	0.6439	0.7503	0.6923	0.7919	0.7474	0.7681
Llama 3.2 3B	0.7616	0.7330	0.7430	0.8079	0.7567	0.7804
SmolLM 3	0.7687	0.7656	0.7662	0.7884	0.7636	0.7749

Table 2: Bert Score based on distilled base uncased against the base model vs finetuned model. The best scores are in bold. * Deepseek-distill-Qwen 1.5B

Fine Tuned with CoT based Neighbour words analysis Approach
Models	Noun	Verb	Adjective	Adverb	Overall
SmolLM 3B (1 epoch)	0.52	0.35	0.50	0.54	0.47
Gemma-2 2B (1 epoch)	0.70	0.61	0.68	0.66	0.67
Qwen 2.5 3B (1 epoch)	0.71	0.64	0.71	0.74	0.69
Gemma 3 4B (1 epoch)	0.79	0.65	0.70	0.80	0.72
Qwen 3 4B (2 epochs)	0.79	0.67	0.75	0.68	0.74

Table 3: Fine Tuning with CoT based reasoning with Neighbour words analysis. The results are presented in the F1 score.

Fine Tuned with COT based Neighbour words analysis Approach (Zero shot)
Models	Noun	Verb	Adjective	Adverb	Overall
Qwen 3 4B (2 epochs)	0.79	0.67	0 .75	0.68	0.74 ∗∗
Gemma 3 4B (2 epochs)	0.81	0.71	0.72	0.76	0.75 ∗∗
Fine Tuned with COT based Neighbour words analysis Approach (Few shot)
Qwen 3 4B (2 epochs)	0.74	0.62	0.68	0.64	0.68
Gemma 3 4B (2 epochs)	0.78	0.68	0.70	0.78	0.72

Table 4: F1 Score for benchmarking Different Reasoning Strategies.

Dataset	MFS	Lesk	SemEq b	SemEq L	ESR b	ESR L	RTWE L	Qwen	Gemma	GlossGPT
Fewshot	51.5	40.9	80.1	82.3	77.8	83.4	78.4	76.52	75.68	90.7
Zeroshot	-	39.0	70.2	72.2	71.6	75.8	69.9	72.66	71.86	79.5

Table 5: F1 score for CoT based Neighbour words analysis against current techniques. FEWS test data used for evaluation. b : Base, L : Large, Our approach in italic (4B models)

Model	Neighbour Word A.	Semantic A.	Syntactic & Semantic A.	Hybrid
Qwen3-4B	0.679 ∗	0.662	0.677	0.602
Gemma3-4B	0.658 ∗∗	0.601	0.628	0.544
* $p<0.001$ , $p<0.01$ , * $p<0.05$

Table 6: F1 score of the VERB task compared using different reasoning methods. Hybrid:Neighbour Word Analysis with Syntactic & Semantic Analysis, A: Analysis

Dataset	ARES	BEM	ESC	EWS	GEN	SYN	CSC	SandWiCH	Gemma	Qwen
42D	41.8	53.2	58.9	43.9	50.2	32.8	56.6	77.1	64.43	78.48
hardEN	0.0	0.0	0.0	0.0	0.0	0.0	7.35	53.4	40.71	54.19

Table 7: Performance comparison (F1 score) against existing WSD models. The compared models are ARES Scarlini et al. (2020) ,BEM Blevins and Zettlemoyer (2020) , ESC Barba et al. (2021a) , EWS Bevilacqua et al. (2020b) , GEN Bevilacqua et al. (2020a) , SYN Scozzafava et al. (2020) ),CSC Barba et al. (2021b) and SandWiCH Guzman-Olivares et al. (2025) . Our best-performing approach, Neighbour words analysis with Finetuned Qwen and Gemma, is reported in Italic.

Models	# Parameters	Set 1	Set 2	Set 3	Set 4
Roberta-base	125M	0.945	0.969	0.888	0.715
Bert-large	340M	0.970	0.978	0.874	0.689
T5-large	770M	0.984	0.987	0.896	0.691
FLAN-T5-large	780M	0.948	0.953	0.852	0.663
T5-xl	3B	0.991	0.992	0.907	0.710
FLAN-T5-xl	3B	0.955	0.958	0.881	0.718

Table 8: F1 score for “Fool me if you can” dataset for binary classification of sense ID. *Our approach is not fine-tuned with training data from “Fool me if you can” dataset.

Table 9: Prompt used to elicit rationales for sense selection during fine-tuning.

You are a helpful assistant tasked with simulating the human thinking process for Word Sense Disambiguation (WSD). Given a sentence containing an ambiguous word, along with the expected sense ID, your goal is to generate a detailed and logical reasoning process as a human would. The given ambiguous word is a VERB. Your response should include: 1. Syntactic Evidence — Summarize the verb’s morphosyntax (tense/aspect/voice), immediate dependents (subject, objects, complements), key function words (auxiliaries, particles, prepositions), and any relevant dependency/constituent patterns that constrain its meaning. 2. Semantic Evidence — Describe selectional preferences and semantic roles implied by the verb, plausible paraphrases, collocations, and context/topic cues (entities, events) that support a specific sense. 3. Decision — State the chosen sense ID and give a clear justification that ties the syntactic and semantic cues to that sense. 4. Elimination of Alternatives — Briefly state why other possible sense IDs do not fit given the observed syntax and semantics. Be thorough, logical, and emulate how a human would naturally think through the disambiguation. Do not include any additional information or instructions outside of the reasoning process.

Table 10: Prompt used to elicit rationales for Verb sense selection during extended study.

You are an expert evaluator assessing the quality of Word Sense Disambiguation (WSD) reasoning generated by an AI system. Task Context: • Original sentence, ambiguous word, and sense definitions: {input_text} • Correct sense ID: {senseid} • Generated Reasoning to evaluate: {reasoning} Evaluation Instructions: Evaluate the generated reasoning on the following four dimensions using a 1-5 scale: 1. Contextual Analysis Quality : Does the reasoning identify and explain relevant contextual clues effectively? • 5: Comprehensively identifies all relevant contextual clues and explains their disambiguation role • 4: Identifies most key contextual elements with clear explanations • 3: Identifies some context but misses important clues or lacks depth • 2: Minimal contextual analysis with superficial observations • 1: Incorrect or irrelevant contextual analysis 2. Sense ID Justification Accuracy : Is the justification for the correct sense logically sound and semantically accurate? • 5: Provides precise, logically sound justification with clear semantic connections • 4: Good justification with minor gaps in reasoning • 3: Adequate justification but lacks depth or contains minor logical errors • 2: Weak justification with significant logical gaps • 1: Incorrect or nonsensical justification 3. Elimination Reasoning Completeness : Does it systematically eliminate alternative senses with clear reasoning? • 5: Systematically eliminates all alternative senses with clear reasoning • 4: Eliminates most alternatives with good reasoning • 3: Partial elimination with some reasoning gaps • 2: Minimal elimination or weak reasoning • 1: No elimination or incorrect reasoning 4. Overall Coherence and Human-likeness : Does the reasoning flow naturally like human thinking? • 5: Reads like natural human reasoning, logically structured and comprehensive • 4: Good flow with minor artificiality • 3: Adequate but somewhat mechanical or repetitive • 2: Disjointed or overly formulaic • 1: Incoherent or completely artificial Output Format (JSON): { "contextual_analysis_score": 1-5, "justification_accuracy_score": 1-5, "elimination_completeness_score": 1-5, "coherence_score": 1-5 }

Table 11: Prompt used for LLM-as-a-Judge evaluation of WSD reasoning quality

Word Sense Disambiguation

See all 13 papers

In the LLM era, Word Sense Induction remains unsolved
Mar 12, 2026Anna Mosolova, Marie Candito, Carlos RamischClustering Distributional Semantics
Université Paris Cité, CNRS, LLF, Paris, France · Aix Marseille Univ, CNRS, LIS, Marseille, France
WiC is Not WSD: A Study on LLMs and Lexical Ambiguity Resolution
Sep 17, 2026Yi Zhou, Kiamehr Rezaee, Danushka Bollegala +2Open-Weight Language Models Word Sense Disambiguation
Cardiff University · University of Liverpool · Amazon
Inspicio: Open-Vocabulary, LLM-Based Sense Retrieval for Historical Languages
Sep 1, 2026Michele CilettiLexical Retrieval Low-Resource Language Processing
Department of Humanities, University of Foggia / Via Arpi 176, 71121 Foggia, Italy

Mar 12, 2026J/K move · Enter open · S save

In the LLM era, Word Sense Induction remains unsolved

Anna Mosolova, Marie Candito, Carlos Ramisch

Université Paris Cité, CNRS, LLF, Paris, France · Aix Marseille Univ, CNRS, LIS, Marseille, France

Clustering Distributional Semantics

Open paper AskPDF arXiv

In the absence of sense-annotated data, word sense induction (WSI) is a compelling alternative to word sense disambiguation, particularly in low-resource or domain-specific settings. In this paper, we emphasize methodological problems in current WSI evaluation. We propose an evaluation on a SemCor-derived dataset, respecting the original corpus polysemy and frequency distributions. We assess pre-trained embeddings and clustering algorithms across parts of speech, and propose and evaluate an LLM-based WSI method for English. We evaluate data augmentation sources (LLM-generated, corpus and lexicon), and semi-supervised scenarios using Wiktionary for data augmentation, must-link constraints, number of clusters per lemma. We find that no unsupervised method (whether ours or previous) surpasses the strong "one cluster per lemma" heuristic (1cpl). We also show that (i) results and best systems may vary across POS, (ii) LLMs have troubles performing this task, (iii) data augmentation is beneficial and (iv) capitalizing on Wiktionary does help. It surpasses previous SOTA system on our test set by 3.3%. WSI is not solved, and calls for a better articulation of lexicons and LLMs' lexical semantics capabilities.

Abstract

Word Sense Disambiguation

Explore similar work

In the LLM era, Word Sense Induction remains unsolved

WiC is Not WSD: A Study on LLMs and Lexical Ambiguity Resolution

Inspicio: Open-Vocabulary, LLM-Based Sense Retrieval for Historical Languages