Polarity detection becomes substantially more challenging under domain shift, particularly in heterogeneous, long-form narratives with complex discourse structure, such as Holocaust oral histories. This paper presents a corpus-scale diagnostic study of off-the-shelf sentiment classifiers on long-form Holocaust oral histories, using three pretrained transformer-based polarity classifiers on a corpus of 107,305 utterances and 579,013 sentences. After assembling model outputs, we introduce an agreement-based stability taxonomy (ABC) to stratify inter-model output stability. We report pairwise percent agreement, Cohen kappa, Fleiss kappa, and row-normalized confusion matrices to localize systematic disagreement. As an auxiliary descriptive signal, a T5-based emotion classifier is applied to stratified samples from each agreement stratum to compare emotion distributions across strata. The combination of multi-model label triangulation and the ABC taxonomy provides a cautious, operational framework for characterizing where and how sentiment models diverge in sensitive historical narratives. Inter-model agreement is low to moderate overall and is driven primarily by boundary decisions around neutrality.
Figures & tables
Level
Model
Neg. (%)
Neu. (%)
Pos. (%)
Utterance 107,305
NLPTown
48.2
19.2
32.6
CardiffNLP
11.3
80.4
8.3
SiEBERT
54.2
—
45.8
Sentence 579,013
NLPTown
45.1
24.0
31.0
Table 1: Polarity distributions across models and granularities.
Level
Metric
Neg. (%)
Neu. (%)
Pos. (%)
Utterance 107,305
Count
49,133
18,162
40,009
Percentage
45.8%
16.9%
37.3%
Sentence 579,013
Count
260,727
113,237
205,049
Percentage
45.0%
19.6%
35.4%
Table 2: Triangulated polarity distribution across granularities.
Level
Cat.
Count ( n )
Share (%)
U ( N=107,305 )
A -1
8,786
8.2
A +1
6,873
6.4
B
73,037
68.1
C
18,609
17.3
S ( N=579,013 )
Table 3: ABC taxonomy prevalence by granularity.
Pair
Agr
κ
N=0
Agr ≠0
κ=0
Utterances ( N=107,305 )
SiEBERT – CardiffNLP
17.8
0.088
21,045
90.9
0.816
SiEBERT – NLPTown
61.7
0.350
86,746
76.3
0.518
CardiffNLP – NLPTown
32.3
0.114
18,649
88.5
0.767
Sentences ( N=579,013 )
SiEBERT – CardiffNLP
28.0
0.144
177,588
91.2
0.801
Table 4: Pairwise agreement (Agr.) and κ for 3-way and polarity-only ( N=0 ) subsets.
Level
N (3-way)
Fleiss’ κ
N=0
Fleiss’ κ=0
Utterance
107,305
0.0535
18,649
0.7835
Sentence
579,013
0.1287
150,755
0.7398
Table 5: Overall three-model agreement (Fleiss’ κ ) on the full three-way label space and on the polarity-only subset ( N=0 ), where units labeled Neutral by CardiffNLP or NLPTown are excluded.
Category
Attribute
Count
%
Gender
Female
270
54.0
Male
230
46.0
Birth cohort
1890s–1910s
120
24.0
1920s
320
64.0
1930s
60
12.0
Top birthplaces
Poland
181
36.2
Table 6: Corpus demographics and background variables ( N=500 ).
Figure 1: T5 emotion distributions (within-group percentages) across analysis groups ( A−1 , A+1 , B, C) at sentence (left) and utterance (right) levels.
Figure 2: T5 mean confidence (uncalibrated certainty proxy) across analysis groups ( A−1 , A+1 , B, C) at sentence (left) and utterance (right) levels.