COGNAC at SemEval-2026 Task 5: LLM Ensembles for Human-Level Word Sense Plausibility Rating in Challenging Narratives
Authors: Azwad Anjum Islam, Tisa Islam Erana
Organizations: Florida International University Knight Foundation School of Computing and Information Sciences 11200 SW 8th Street, Miami, FL 33199, USA
We describe our system for SemEval-2026 Task 5, which requires rating the plausibility of given word senses of homonyms in short stories on a 5-point Likert scale. Systems are evaluated by the unweighted average of accuracy (within one standard deviation of mean human judgments) and Spearman Rank Correlation. We explore three prompting strategies using multiple closed-source commercial LLMs: (i) a baseline zero-shot setup, (ii) Chain-of-Thought (CoT) style prompting with structured reasoning, and (iii) a comparative prompting strategy for evaluating candidate word senses simultaneously. Furthermore, to account for the substantial inter-annotator variation present in the gold labels, we propose an ensemble setup by averaging model predictions. Our best official system, comprising an ensemble of LLMs across all three prompting strategies, placed 4th on the competition leaderboard with 0.88 accuracy and 0.83 Spearman's rho (0.86 average). Post-competition experiments with additional models further improved this performance to 0.92 accuracy and 0.85 Spearman's rho (0.89 average). We find that comparative prompting consistently improved performance across model families, and model ensembling significantly enhanced alignment with mean human judgments, suggesting that LLM ensembles are especially well suited for subjective semantic evaluation tasks involving multiple annotators.
Figures & tables
Model
Accuracy
Spearman ρ
Average Score
Zero-shot
CoT
Comp.
Zero-shot
CoT
Comp.
Zero-shot
CoT
Comp.
gpt-5-nano
.72
.68
.82
.76
.66
.77
.74
.67 ↓
.80 ↑
gpt-4.1-mini
.72
.73
.74
.68
.75
.74
.70
.74 ↑
.74 ↑
gemini-2.5-flash
.69
.62
.71
.75
.73
.76
.72
.67 ↓
.73 ↑
gpt-4o-mini
.72
.61
.75
.70
.62
.69
.71
.61 ↓
.72 ↑
gemini-2.0-flash
.59
.61
.68
.64
.68
.70
.61
.65 ↑
.69 ↑
Table 1: Individual model performance under different prompting strategies on the development set. Arrows ( ↑ , ↓ ) indicate changes relative to the zero-shot prompting strategy. The horizontal divider separates models used only in post-competition experiments, marked with (*).
Figure 1: Distribution of word sense plausibility scores by human in development and test data in percentage
Ensembles
Accuracy
ρ
Average
Ezeroshot
0.86
0.79
0.83
ECoT
0.80
0.78
0.79
Ecomp
0.87
0.81
0.84
Eall
0.89
0.84
0.87
Including post-competition models:
Ezeroshot
0.91
0.84
0.88
Table 2: Performance summary of different LLM ensembles on development data.
Team
Accuracy
ρ
Avg.
SRCB (1 st )
0.93
0.86
0.89
COGNAC (4 th )
0.88
0.83
0.86
Eall (post-competition)
0.92
0.85
0.89
Table 3: Our (COGNAC) performance on test data: official submission and post-competition improvements.
Figure 2: Model predictions vs. human ratings (mean ± 1 stdev) on test data. Eall (10 LLMs) aligns better than the best individual model ( gpt-5-mini ).
Word sense plausibility rating requires predicting the human-perceived plausibility of a given word sense on a 1-5 scale in the context of short narrative stories containing ambiguous homonyms. This paper systematically compares three approaches: (1) embedding-based methods pairing sentence embeddings with standard regressors, (2) transformer fine-tuning with parameter-efficient adaptation, and (3) large language model (LLM) prompting with structured reasoning and explicit decision rules. The best-performing system employs a structured prompting strategy that decomposes evaluation into narrative components (precontext, target sentence, ending) and applies explicit decision rules for rating calibration. The analysis reveals that structured prompting with decision rules outperforms both fine-tuned models and embedding-based approaches, and that prompt design matters more than model scale for this task.
Tong Wu, Thanet Markchom, Huizhi Liang
Independent Researcher · Department of Computer Science, University of Reading, Reading, UK · School of Computing, Newcastle University, Newcastle upon Tyne, UK
Recent advances in language models have substantially improved Natural Language Understanding (NLU). Although widely used benchmarks suggest that Large Language Models (LLMs) can effectively disambiguate, their practical applicability in real-world narrative contexts remains underexplored. SemEval-2026 Task 5 addresses this gap by introducing a task that predicts the human-perceived plausibility of a word sense within a short story. In this work, we propose an LLM-based framework for plausibility scoring of homonymous word senses in narrative texts using a structured reasoning mechanism. We examine the impact of fine-tuning low-parameter LLMs with diverse reasoning strategies, alongside dynamic few-shot prompting for large-parameter models, on accurate sense identification and plausibility estimation. Our results show that commercial large-parameter LLMs with dynamic few-shot prompting closely replicate human-like plausibility judgments. Furthermore, model ensembling slightly improves performance, better simulating the agreement patterns of five human annotators compared to single-model predictions
Deshan Sumanathilaka, Nicholas Micallef, Julian Hough +1
Department of Computer Science, Swansea University, Wales, UK
We present our winning system for TaskB (generation with reference passages) in SemEval-2026 Task8: MTRAGEval. Our method is a heterogeneous ensemble of seven LLMs with two prompting variants, where a GPT-4o-mini judge selects the best candidate per instance. We ranked 1st out of 26 teams, achieving a conditioned harmonic mean of 0.7827 and outperforming the strongest baseline (gpt-oss-120b, 0.6390). Ablations show that diversity in model families, scales, and prompting strategies is essential, with the ensemble consistently beating any single model. We also introduce Meno-Lite-0.1, a 7B domain-adapted model with a strong cost--performance trade-off, and analyse MTRAGEval, highlighting annotation limitations and directions for improvement. Our code is publicly available: https://github.com/RaguTeam/ragu_mtrag_semeval