NCL-UoR at SemEval-2026 Task 5: Embedding-Based Methods, Fine-Tuning, and LLMs for Word Sense Plausibility Rating

Tong Wu; Thanet Markchom; Huizhi Liang

cs.CLMar 9, 2026

NCL-UoR at SemEval-2026 Task 5: Embedding-Based Methods, Fine-Tuning, and LLMs for Word Sense Plausibility Rating

Authors: Tong Wu, Thanet Markchom, Huizhi Liang

Organizations: Independent Researcher · Department of Computer Science, University of Reading, Reading, UK · School of Computing, Newcastle University, Newcastle upon Tyne, UK

Abstract

Word sense plausibility rating requires predicting the human-perceived plausibility of a given word sense on a 1-5 scale in the context of short narrative stories containing ambiguous homonyms. This paper systematically compares three approaches: (1) embedding-based methods pairing sentence embeddings with standard regressors, (2) transformer fine-tuning with parameter-efficient adaptation, and (3) large language model (LLM) prompting with structured reasoning and explicit decision rules. The best-performing system employs a structured prompting strategy that decomposes evaluation into narrative components (precontext, target sentence, ending) and applies explicit decision rules for rating calibration. The analysis reveals that structured prompting with decision rules outperforms both fine-tuned models and embedding-based approaches, and that prompt design matters more than model scale for this task.

Figures & tables

Panel of Figure 1 — Figure 1: System overview of the three approaches. All approaches take the same input (narrative story, homonym, candidate word sense) and output a plausibility rating from 1 to 5.

Approach	System	$\boldsymbol{\rho}$	Acc.
Embedding	MPNet + Ridge	0.174	0.560
Embedding	RoBERTa + XGBoost	0.114	0.537
Fine-tuning	ELECTRA-base	0.491	0.663
	ELECTRA-large + LoRA	0.644	0.709
	DeBERTa-large + LoRA	0.587	0.757
	+ uncertainty loss	0.606	0.767

Table 1: Development set results. P1 = few-shot prompting; P2 = structured prompting with decision rules.

Approach	System	$\boldsymbol{\rho}$	Acc.
Embedding	MPNet + Ridge	0.109	0.513
Embedding	RoBERTa + XGBoost	0.110	0.505
Fine-tuning	ELECTRA-base	0.482	0.625
	ELECTRA-large + LoRA	0.527	0.639
	DeBERTa-large + LoRA	0.492	0.676
	+ uncertainty loss	0.435	0.659

Table 2: Test set results. P1 = few-shot prompting; P2 = structured prompting with decision rules.

Figure 2: Test-set performance landscape (Spearman vs. Accuracy).

Figure 3: Paired bootstrap differences relative to GPT-4o (P2) with bootstrap confidence intervals.

Loss Configuration	$\boldsymbol{\rho}$	Acc.
Huber + RankNet ( $\lambda_{r}{=}0.5$ )	0.587	0.757
MSE + RankNet + Uncertainty ( $\lambda_{r}{=}0.25$ , $\lambda_{u}{=}0.5$ )	0.606	0.767

Table 3: Ablation of loss components for DeBERTa-large + LoRA on the development set.

Prompt Engineering

Prompt engineering designs and evaluates instructions, context, examples, and learned prompts to steer AI model behavior.

See all 124 papers

SwanNLP at SemEval-2026 Task 5: An LLM-based Framework for Plausibility Scoring in Narrative Word Sense Disambiguation
Apr 17, 2026Deshan Sumanathilaka, Nicholas Micallef, Julian Hough +1LLM Evaluation Few-Shot Prompting
Department of Computer Science, Swansea University, Wales, UK
COGNAC at SemEval-2026 Task 5: LLM Ensembles for Human-Level Word Sense Plausibility Rating in Challenging Narratives
Mar 16, 2026Azwad Anjum Islam, Tisa Islam EranaLanguage Model Ensembles
Florida International University Knight Foundation School of Computing and Information Sciences 11200 SW 8th Street, Miami, FL 33199, USA
*-PLUIE: Personalisable metric with Llm Used for Improved Evaluation
Feb 17, 2026Quentin Lemesle, Léane Jourdan, Daisy Munson +4Language Model Generation Evaluation LLM Evaluation
Univ Rennes, CNRS, IRISA, EXPRESSION, 22300 Lannion, France · Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes, France · Univ Rennes, CNRS, IRISA, SOTERN, 22300 Lannion, France +1

Apr 17, 2026J/K move · Enter open · S save

SwanNLP at SemEval-2026 Task 5: An LLM-based Framework for Plausibility Scoring in Narrative Word Sense Disambiguation

Deshan Sumanathilaka, Nicholas Micallef, Julian Hough, Saman Jayasinghe

Department of Computer Science, Swansea University, Wales, UK

LLM Evaluation Few-Shot Prompting Word Sense Disambiguation

Open paper AskPDF arXiv

Recent advances in language models have substantially improved Natural Language Understanding (NLU). Although widely used benchmarks suggest that Large Language Models (LLMs) can effectively disambiguate, their practical applicability in real-world narrative contexts remains underexplored. SemEval-2026 Task 5 addresses this gap by introducing a task that predicts the human-perceived plausibility of a word sense within a short story. In this work, we propose an LLM-based framework for plausibility scoring of homonymous word senses in narrative texts using a structured reasoning mechanism. We examine the impact of fine-tuning low-parameter LLMs with diverse reasoning strategies, alongside dynamic few-shot prompting for large-parameter models, on accurate sense identification and plausibility estimation. Our results show that commercial large-parameter LLMs with dynamic few-shot prompting closely replicate human-like plausibility judgments. Furthermore, model ensembling slightly improves performance, better simulating the agreement patterns of five human annotators compared to single-model predictions

Abstract

Prompt Engineering

Explore similar work

SwanNLP at SemEval-2026 Task 5: An LLM-based Framework for Plausibility Scoring in Narrative Word Sense Disambiguation

COGNAC at SemEval-2026 Task 5: LLM Ensembles for Human-Level Word Sense Plausibility Rating in Challenging Narratives

*-PLUIE: Personalisable metric with Llm Used for Improved Evaluation