Multiperspectivity as a Resource for Narrative Similarity Prediction
Authors: Max Upravitelev, Veronika Solopova, Jing Yang, Charlott Jakob, Premtim Sahitaj, Ariana Sahitaj, Vera Schmitt
Organizations: Technische Universit`‘at Berlin · German Research Center for Artificial Intelligence (DFKI) · BIFOLD – Berlin Institute for the Foundations of Learning and Data · Centre for European Research in Trusted AI (CERTAIN)
Predicting narrative similarity can be understood as an inherently interpretive task: different, equally valid readings of the same text can produce divergent interpretations and thus different similarity judgments, posing a fundamental challenge for semantic evaluation benchmarks that encode a single ground truth. Rather than treating this multiperspectivity as a challenge to overcome, we propose to incorporate it in the decision making process of predictive systems. To explore this strategy, we created an ensemble of 31 LLM personas. These range from practitioners following interpretive frameworks to more intuitive, lay-style characters. Our experiments were conducted on the SemEval-2026 Task 4 dataset, where the system achieved an accuracy score of 0.705. Accuracy improves with ensemble size, consistent with Condorcet Jury Theorem-like dynamics under weakened independence. Practitioner personas perform worse individually but produce less correlated errors, yielding larger ensemble gains under majority voting. Our error analysis reveals a consistent negative association between gender-focused interpretive vocabulary and accuracy across all persona categories, suggesting either attention to dimensions not relevant for the benchmark or valid interpretations absent from the ground truth. This finding underscores the need for evaluation frameworks that account for interpretive plurality.
Figures & tables
LLM Persona
Role
System Prompt
Literary Critic
Practi-tioner
You are a Literary Critic analyzing narrative similarity.
Computational Narratologist
Practi-tioner
You are a Computational Narratologist. You excel at structural analysis of text and the structured extraction of narrative features, such as actors, relations, events, topics, themes, sentiments etc.
…
…
…
Postcolonial Critic
Practi-tioner
You are a Postcolonial Critic examining the texts in terms of global power dynamics.
Feminist Literary Critic
Practi-tioner
You are a Feminist Literary Critic examining gender-related dynamics within texts.
Gender Perspective Analyst
Practi-tioner
You are a Gender Perspective Analyst examining gender-related dynamics.
Table 1: Selected LLM Personas, full list can be found in Appendix 8
Ensemble Size
Qwen3 14B
Gemma 3 27B-it
gpt-oss-20b
All
Majority Vote Accuracy
E=1
69.1±0.4
71.9±0.4
57.0±0.7
66.0±0.3
E=3
71.7±0.5
73.4±0.4
58.4±0.9
69.8±0.4
E=5
72.7±0.6
73.8±0.5
59.0±0.9
71.5±0.4
E=10
73.8±0.7
73.9±0.4
59.8±1.1
72.9±0.6
E=20
74.2±0.7
74.2±0.5
60.2±1.4
74.5±0.6
Table 2: Ensemble accuracy (% ± standard deviation, based on random persona combinations with maximum sample limit of 5000.) by ensemble size, averaged over n=10 runs.
Metric
Qwen3 14B
Gemma 3 27B-it
gpt-oss-20b
K≥1
99.1%±0.4%
97.2%±1.0%
99.8%±0.2%
K≥2
97.8%±0.5%
94.5%±0.8%
99.3%±0.5%
K≥3
96.4%±0.5%
92.5%±0.7%
98.5%±0.7%
K≥4
95.0%±0.5%
90.6%±0.7%
97.5%±0.7%
K≥5
93.2%±0.6%
88.6%±0.9%
95.7%±0.9%
K≥7
90.2%±0.9%
84.7%±0.6%
91.4%±1.3%
Table 3: Cross-model comparison of K correct personas out of 31 in ensemble voting (mean and ± std across runs)
Rank
Persona
All
Qwen3
Gemma
gpt-oss
–
MAJORITY VOTES ALL
75.8
74.3
74.5
60.8
–
MAJORITY [P] only
76.0
74.2
74.1
60.2
–
MAJORITY [P] (subsamp=13)
75.8
74.1
74.1
60.4
–
MAJORITY [L] only
75.3
73.5
74.5
59.1
1
[L] Tour Guide
67.70
71.70
73.60
57.60
2
[P] Activist
67.20
70.30
74.20
57.10
Table 4: Per-Persona Accuracy (%) Comparison Across Models, sorted by the “All” column. [L] is short for Lay people, [P] for Practitioners.
Metric
[P]
[L]
Conclusion
Individual accuracy
71.0%
71.7%
Higher accuracy for [L]
Error correlation ( r )
0.388
0.461
Errors are 19% less correlated for [P]
Double-fault P(both wrong)
0.164
0.173
[L] fail together more often
Table 5: Error diversity analysis: comparison of [P] and [L] groups. Individual metrics are averaged across personas within each category; diversity metrics are averaged across all pairwise persona combinations within each category. Following Kuncheva and Whitaker (2003) .
Direction
Term
rpb
pFDR
Acc (present)
Acc (absent)
N
Negative correlations (gender vocabulary associated with lower accuracy)
↓
female
− 0.046
< 0.001
56.5%
66.6%
4,575
↓
gender roles
− 0.039
< 0.001
42.0%
66.2%
571
↓
objectification
− 0.032
< 0.001
13.6%
66.1%
81
↓
female protagonist
− 0.030
< 0.001
54.9%
66.3%
1,516
↓
sexual violence
− 0.029
< 0.001
34.9%
66.2%
186
Table 6: Point-biserial correlations between presence of gender/feminist-distinctive vocabulary and voting accuracy for Other Expert personas (combined across all three models, all output fields). Terms were identified via lift-based analysis ( lift≥5.0 ) from the two gender/feminist persona outputs. Correlations are FDR-corrected (Benjamini–Hochberg, α=0.05 ). N = number of responses containing the term. Full results for all persona groups and per-model breakdowns are provided in Appendix F .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Persona
All
Qwen3
Gemma
gpt-oss
Yellow Press Journalist
83.3
83.9
91.8
74.2
Fitness Coach
83.1
83.7
93.0
72.7
Computational Narratologist
83.1
83.3
92.1
73.7
Psychologist
82.9
83.5
93.1
72.2
Electrician
82.8
83.4
91.8
73.2
Ethicist
82.8
83.2
93.1
72.1
Appendix
Table 7: Voting consistency (%) per persona and model. Consistency is defined as the fraction of runs matching the modal (most common) vote for each item, averaged across all 200 items. Each persona was evaluated 10 times per item; a consistency of 100% means the persona always gave the same answer across all runs, while 50% indicates a maximally split vote. Personas are sorted by combined consistency (descending).
LLM Persona
Role
System Prompt
Literary Critic
Practitioner
You are a Literary Critic analyzing narrative similarity.
Computational Narratologist
Practitioner
You are a Computational Narratologist. You excel at structural analysis of text and the structured extraction of narrative features, such as actors, relations, events, topics, themes, sentiments etc.
Psychologist
Practitioner
You are a Psychologist analyzing emotional and motivational similarity.
Psychoanalyst
Practitioner
You are a Psychoanalyst analyzing the text from the perspective of engaging with the unconscious and how it relates to society, focusing on authorial intent, symbolism and conflicts.
Storyteller
Practitioner
You are a Storyteller analyzing narrative flow and engagement.
Computer Scientist
Practitioner
You are a Computer Scientist analyzing structural consistency. You don’t care much for literature, but you are able to talk about text.
Table 10: Error diversity analysis comparing [P] and [L] persona categories.
Gender/Fem. (self)
Other Experts
Lay Personas
Model
Field
Sig.
+
−
Sig.
+
−
Sig.
+
−
Qwen3-14B
themes
4
0
4
11
1
10
10
0
10
key_points
5
0
5
12
0
12
11
1
10
explanation
6
0
6
22
0
22
19
0
19
all
17
0
17
37
1
36
25
1
24
Gemma-27B
themes
96
19
77
79
15
64
40
4
36
Appendix
Table 11: Number of significant correlations (FDR-corrected, p<0.05 ) between gender/feminist-distinctive term presence and voting accuracy. Sig. = total significant; + = positive (term presence associated with higher accuracy); − = negative (term presence associated with lower accuracy).
Group
Dir.
Term
rpb
Acc (+)
Acc ( − )
N
Gender/Fem. (self)
↑
1
male friendship
+ .043
91.3%
63.9%
69
2
male relationships
+ .036
82.5%
63.9%
103
3
male bonding
+ .036
83.5%
63.9%
91
↓
1
male character
− .058
45.7%
64.4%
269
2
female
− .057
59.6%
65.7%
3,312
3
affairs political
− .050
0.0%
64.1%
17
Appendix
Table 12: Top significant correlations (FDR-corrected, p<0.05 ) between gender/feminist vocabulary and voting accuracy, combined across all models, all fields. Top 3 positive ( ↑ ) and top 5 negative ( ↓ ) correlations shown per persona group. All pFDR<0.005 . Totals: Gender/Feminist (self): 132 significant (27 + , 105 − ); Other Experts: 119 (26 + , 93 − ); Lay: 63 (3 + , 60 − ).