Multiperspectivity as a Resource for Narrative Similarity Prediction
Authors: Max Upravitelev, Veronika Solopova, Jing Yang, Charlott Jakob, Premtim Sahitaj, Ariana Sahitaj, Vera Schmitt
Organizations: Technische Universit`‘at Berlin · German Research Center for Artificial Intelligence (DFKI) · BIFOLD – Berlin Institute for the Foundations of Learning and Data · Centre for European Research in Trusted AI (CERTAIN)
Predicting narrative similarity can be understood as an inherently interpretive task: different, equally valid readings of the same text can produce divergent interpretations and thus different similarity judgments, posing a fundamental challenge for semantic evaluation benchmarks that encode a single ground truth. Rather than treating this multiperspectivity as a challenge to overcome, we propose to incorporate it in the decision making process of predictive systems. To explore this strategy, we created an ensemble of 31 LLM personas. These range from practitioners following interpretive frameworks to more intuitive, lay-style characters. Our experiments were conducted on the SemEval-2026 Task 4 dataset, where the system achieved an accuracy score of 0.705. Accuracy improves with ensemble size, consistent with Condorcet Jury Theorem-like dynamics under weakened independence. Practitioner personas perform worse individually but produce less correlated errors, yielding larger ensemble gains under majority voting. Our error analysis reveals a consistent negative association between gender-focused interpretive vocabulary and accuracy across all persona categories, suggesting either attention to dimensions not relevant for the benchmark or valid interpretations absent from the ground truth. This finding underscores the need for evaluation frameworks that account for interpretive plurality.
Figures & tables
LLM Persona
Role
System Prompt
Literary Critic
Practi-tioner
You are a Literary Critic analyzing narrative similarity.
Computational Narratologist
Practi-tioner
You are a Computational Narratologist. You excel at structural analysis of text and the structured extraction of narrative features, such as actors, relations, events, topics, themes, sentiments etc.
…
…
…
Postcolonial Critic
Practi-tioner
You are a Postcolonial Critic examining the texts in terms of global power dynamics.
Feminist Literary Critic
Practi-tioner
You are a Feminist Literary Critic examining gender-related dynamics within texts.
Gender Perspective Analyst
Practi-tioner
You are a Gender Perspective Analyst examining gender-related dynamics.
Table 1: Selected LLM Personas, full list can be found in Appendix 8
Ensemble Size
Qwen3 14B
Gemma 3 27B-it
gpt-oss-20b
All
Majority Vote Accuracy
E=1
69.1±0.4
71.9±0.4
57.0±0.7
66.0±0.3
E=3
71.7±0.5
73.4±0.4
58.4±0.9
69.8±0.4
E=5
72.7±0.6
73.8±0.5
59.0±0.9
71.5±0.4
E=10
73.8±0.7
73.9±0.4
59.8±1.1
72.9±0.6
E=20
74.2±0.7
74.2±0.5
60.2±1.4
74.5±0.6
Table 2: Ensemble accuracy (% ± standard deviation, based on random persona combinations with maximum sample limit of 5000.) by ensemble size, averaged over n=10 runs.
Metric
Qwen3 14B
Gemma 3 27B-it
gpt-oss-20b
K≥1
99.1%±0.4%
97.2%±1.0%
99.8%±0.2%
K≥2
97.8%±0.5%
94.5%±0.8%
99.3%±0.5%
K≥3
96.4%±0.5%
92.5%±0.7%
98.5%±0.7%
K≥4
95.0%±0.5%
90.6%±0.7%
97.5%±0.7%
K≥5
93.2%±0.6%
88.6%±0.9%
95.7%±0.9%
K≥7
90.2%±0.9%
84.7%±0.6%
91.4%±1.3%
Table 3: Cross-model comparison of K correct personas out of 31 in ensemble voting (mean and ± std across runs)
Rank
Persona
All
Qwen3
Gemma
gpt-oss
–
MAJORITY VOTES ALL
75.8
74.3
74.5
60.8
–
MAJORITY [P] only
76.0
74.2
74.1
60.2
–
MAJORITY [P] (subsamp=13)
75.8
74.1
74.1
60.4
–
MAJORITY [L] only
75.3
73.5
74.5
59.1
1
[L] Tour Guide
67.70
71.70
73.60
57.60
2
[P] Activist
67.20
70.30
74.20
57.10
Table 4: Per-Persona Accuracy (%) Comparison Across Models, sorted by the “All” column. [L] is short for Lay people, [P] for Practitioners.
Metric
[P]
[L]
Conclusion
Individual accuracy
71.0%
71.7%
Higher accuracy for [L]
Error correlation ( r )
0.388
0.461
Errors are 19% less correlated for [P]
Double-fault P(both wrong)
0.164
0.173
[L] fail together more often
Table 5: Error diversity analysis: comparison of [P] and [L] groups. Individual metrics are averaged across personas within each category; diversity metrics are averaged across all pairwise persona combinations within each category. Following Kuncheva and Whitaker (2003) .
Direction
Term
rpb
pFDR
Acc (present)
Acc (absent)
N
Negative correlations (gender vocabulary associated with lower accuracy)
↓
female
− 0.046
< 0.001
56.5%
66.6%
4,575
↓
gender roles
− 0.039
< 0.001
42.0%
66.2%
571
↓
objectification
− 0.032
< 0.001
13.6%
66.1%
81
↓
female protagonist
− 0.030
< 0.001
54.9%
66.3%
1,516
↓
sexual violence
− 0.029
< 0.001
34.9%
66.2%
186
Table 6: Point-biserial correlations between presence of gender/feminist-distinctive vocabulary and voting accuracy for Other Expert personas (combined across all three models, all output fields). Terms were identified via lift-based analysis ( lift≥5.0 ) from the two gender/feminist persona outputs. Correlations are FDR-corrected (Benjamini–Hochberg, α=0.05 ). N = number of responses containing the term. Full results for all persona groups and per-model breakdowns are provided in Appendix F .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Persona
All
Qwen3
Gemma
gpt-oss
Yellow Press Journalist
83.3
83.9
91.8
74.2
Fitness Coach
83.1
83.7
93.0
72.7
Computational Narratologist
83.1
83.3
92.1
73.7
Psychologist
82.9
83.5
93.1
72.2
Electrician
82.8
83.4
91.8
73.2
Ethicist
82.8
83.2
93.1
72.1
Appendix
Table 7: Voting consistency (%) per persona and model. Consistency is defined as the fraction of runs matching the modal (most common) vote for each item, averaged across all 200 items. Each persona was evaluated 10 times per item; a consistency of 100% means the persona always gave the same answer across all runs, while 50% indicates a maximally split vote. Personas are sorted by combined consistency (descending).
LLM Persona
Role
System Prompt
Literary Critic
Practitioner
You are a Literary Critic analyzing narrative similarity.
Computational Narratologist
Practitioner
You are a Computational Narratologist. You excel at structural analysis of text and the structured extraction of narrative features, such as actors, relations, events, topics, themes, sentiments etc.
Psychologist
Practitioner
You are a Psychologist analyzing emotional and motivational similarity.
Psychoanalyst
Practitioner
You are a Psychoanalyst analyzing the text from the perspective of engaging with the unconscious and how it relates to society, focusing on authorial intent, symbolism and conflicts.
Storyteller
Practitioner
You are a Storyteller analyzing narrative flow and engagement.
Computer Scientist
Practitioner
You are a Computer Scientist analyzing structural consistency. You don’t care much for literature, but you are able to talk about text.
Table 10: Error diversity analysis comparing [P] and [L] persona categories.
Gender/Fem. (self)
Other Experts
Lay Personas
Model
Field
Sig.
+
−
Sig.
+
−
Sig.
+
−
Qwen3-14B
themes
4
0
4
11
1
10
10
0
10
key_points
5
0
5
12
0
12
11
1
10
explanation
6
0
6
22
0
22
19
0
19
all
17
0
17
37
1
36
25
1
24
Gemma-27B
themes
96
19
77
79
15
64
40
4
36
Appendix
Table 11: Number of significant correlations (FDR-corrected, p<0.05 ) between gender/feminist-distinctive term presence and voting accuracy. Sig. = total significant; + = positive (term presence associated with higher accuracy); − = negative (term presence associated with lower accuracy).
Group
Dir.
Term
rpb
Acc (+)
Acc ( − )
N
Gender/Fem. (self)
↑
1
male friendship
+ .043
91.3%
63.9%
69
2
male relationships
+ .036
82.5%
63.9%
103
3
male bonding
+ .036
83.5%
63.9%
91
↓
1
male character
− .058
45.7%
64.4%
269
2
female
− .057
59.6%
65.7%
3,312
3
affairs political
− .050
0.0%
64.1%
17
Appendix
Table 12: Top significant correlations (FDR-corrected, p<0.05 ) between gender/feminist vocabulary and voting accuracy, combined across all models, all fields. Top 3 positive ( ↑ ) and top 5 negative ( ↓ ) correlations shown per persona group. All pFDR<0.005 . Totals: Gender/Feminist (self): 132 significant (27 + , 105 − ); Other Experts: 119 (26 + , 93 − ); Lay: 63 (3 + , 60 − ).
We describe our system for SemEval-2026 Task 5, which requires rating the plausibility of given word senses of homonyms in short stories on a 5-point Likert scale. Systems are evaluated by the unweighted average of accuracy (within one standard deviation of mean human judgments) and Spearman Rank Correlation. We explore three prompting strategies using multiple closed-source commercial LLMs: (i) a baseline zero-shot setup, (ii) Chain-of-Thought (CoT) style prompting with structured reasoning, and (iii) a comparative prompting strategy for evaluating candidate word senses simultaneously. Furthermore, to account for the substantial inter-annotator variation present in the gold labels, we propose an ensemble setup by averaging model predictions. Our best official system, comprising an ensemble of LLMs across all three prompting strategies, placed 4th on the competition leaderboard with 0.88 accuracy and 0.83 Spearman's rho (0.86 average). Post-competition experiments with additional models further improved this performance to 0.92 accuracy and 0.85 Spearman's rho (0.89 average). We find that comparative prompting consistently improved performance across model families, and model ensembling significantly enhanced alignment with mean human judgments, suggesting that LLM ensembles are especially well suited for subjective semantic evaluation tasks involving multiple annotators.
Azwad Anjum Islam, Tisa Islam Erana
Florida International University Knight Foundation School of Computing and Information Sciences 11200 SW 8th Street, Miami, FL 33199, USA
We present the shared task on narrative similarity and narrative representation learning - NSNRL (pronounced "nass-na-rel"). The task operationalizes narrative similarity as a binary classification problem: determining which of two stories is more similar to an anchor story. We introduce a novel definition of narrative similarity, compatible with both narrative theory and intuitive judgment. Based on the similarity judgments collected under this concept, we also evaluate narrative embedding representations. We collected at least two annotations each for more than 1,000 story summary triples, with each annotation being backed by at least two annotators in agreement. This paper describes the sampling and annotation process for the dataset; further, we give an overview of the submitted systems and the techniques they employ. We received a total of 71 final submissions from 46 teams across our two tracks. In our triple-based classification setup, LLM ensembles make up many of the top-scoring systems, while in the embedding setup, systems with pre- and post-processing on pretrained embedding models perform about on par with custom fine-tuned solutions. Our analysis identifies potential headroom for improvement of automated systems in both tracks. The task website includes visualizations of embeddings alongside instance-level classification results for all teams.
Hans Ole Hatzel, Ekaterina Artemova, Haimo Paul Stiemer +2
University of Hamburg1 · Toloka AI2 · German UDS3 +1
Recent advances in language models have substantially improved Natural Language Understanding (NLU). Although widely used benchmarks suggest that Large Language Models (LLMs) can effectively disambiguate, their practical applicability in real-world narrative contexts remains underexplored. SemEval-2026 Task 5 addresses this gap by introducing a task that predicts the human-perceived plausibility of a word sense within a short story. In this work, we propose an LLM-based framework for plausibility scoring of homonymous word senses in narrative texts using a structured reasoning mechanism. We examine the impact of fine-tuning low-parameter LLMs with diverse reasoning strategies, alongside dynamic few-shot prompting for large-parameter models, on accurate sense identification and plausibility estimation. Our results show that commercial large-parameter LLMs with dynamic few-shot prompting closely replicate human-like plausibility judgments. Furthermore, model ensembling slightly improves performance, better simulating the agreement patterns of five human annotators compared to single-model predictions
Deshan Sumanathilaka, Nicholas Micallef, Julian Hough +1
Department of Computer Science, Swansea University, Wales, UK