Finding Errors Is Not Enough: Selective Model Escalation for Entity-Targeted Sentiment Analysis
Authors: Nishan Chatterjee, Matthew Purver, Antoine Doucet, Senja Pollak
Organizations: Jožef Stefan Institute, Ljubljana, Slovenia · Jožef Stefan International Postgraduate School, Ljubljana, Slovenia · University of La Rochelle, La Rochelle, France · Queen Mary University of London, London, United Kingdom · University of Ljubljana, Ljubljana, Slovenia
Improving a sentiment classifier with limited resources involves more than simply deciding how much data to collect or which model to train. A labelled collection may lack examples of an important class. Once trained, a classifier may benefit from another model even when that model is weaker on its own. We study these decisions in document-level Entity-Level Sentiment Analysis (ELSA), primarily using South Slavic news. We first compare acquisition orders as collections grow. We then use uncertainty to choose which predictions to revisit, how many to process, and whether an alternative encoder or local LLM can correct them. Finally, we select complete routing policies on development data for held-out evaluation. Minority-prioritised acquisition helps at some very small training sizes, but not consistently thereafter. On SentiCoref, an oracle correcting all errors in the least-confident 10% without harming correct predictions would add 9.14 Macro-F1 points; however the best evaluated action gains only 0.90. Encoder averaging helps on AspectBench and NewsMTSC but harms SentiCoref, where selective LLM assistance is more promising. Both development-selected policies route 30% of records to the encoder-assisted LLM, yielding observed gains of 1.02 points on SentiCoref (interval includes zero) and 1.71 on NewsMTSC, at substantially different inference costs.
Figures & tables
Dataset
Test n
Neg. / Neu. / Pos. (%)
AspectBench HBS
12,407
7.5 / 49.5 / 43.0
AspectBench SL
7,740
1.2 / 82.6 / 16.2
SentiCoref SL
2,155
12.9 / 75.1 / 12.0
NewsMTSC EN
1,146
37.4 / 39.7 / 22.9
Table 1: Test article–entity counts and sentiment shares (%). HBS groups Serbian, Serbo-Croatian, Bosnian, and Croatian; SL and EN are Slovenian and English.
Dataset
Base
Selected
Concentration
Perfect
error
error
Δ F1
HBS
13.18
42.22
3.20 ×
+5.75
SL
7.60
33.59
4.42 ×
+10.36
SentiCoref
20.05
41.67
2.08 ×
+9.14
NewsMTSC
16.49
45.22
2.74 ×
+5.05
Table 2: E1 error rates (%) overall (Base) and in its least-confident 10% (Selected); concentration is their ratio. Perfect Δ F1 is the oracle gain from correcting all selected errors without harms.
Scope
Dataset
E1
E2
E1+E2 average
Gemma without E1 information
E1-assisted Gemma
All records
AspectBench HBS
82.63
+0.33
+1.22
−22.90
−1.13
AspectBench SL
74.53
+0.72
+1.27
−31.98
−3.80
SentiCoref SL
59.91
−6.41
−2.92
−10.70
+0.40
NewsMTSC EN
82.14
+0.48
+1.89
−5.46
+0.50
Scope
Dataset
E2 replacement
E1+E2 average
More-confident encoder
E1-assisted Gemma masked / unmasked
Perfect correction
Selected 10%
AspectBench HBS
+0.50
+0.77
+0.74
+0.58 / +0.53
+5.75
Table 3: Macro-F1 comparison: E1 is absolute; all other scores are point changes. All-record Gemma retains the target name; E1-assisted Gemma also receives E1’s prediction. Selected-10% actions modify the same queue and retain E1 elsewhere. A slash separates masked and unmasked Gemma; perfect correction is an oracle upper bound.
Dataset
Action
Route
Δ F1 [95% CI]
GPU-min per 1k
SentiCoref
Keep E1
0%
0.00 [–]
0.00
Avg(E1,E2)
10%
−2.52 [ −4.49,−0.76 ]
0.03
Unmasked Gemma
10%
+0.90 [ −0.28,+2.31 ]
7.89
Selected Gemma
30%
+1.02 [ −0.53,+2.77 ]
43.71
NewsMTSC
Keep E1
0%
0.00 [–]
0.00
Avg(E1,E2)
10%
+1.51 [ +0.38,+2.68 ]
0.03
Table 4: Held-out Macro-F1 changes from E1 with 95% paired source-group bootstrap CIs (10,000 resamples), and additional allocated warm GPU-minutes per 1,000 records. Selected Gemma uses a masked prompt on SentiCoref; other Gemma rows retain the target. Compute excludes E1 and deployment overhead.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
Evidence role
Purpose
Analyses
Main comparisons
Directly support the four headline answers under their stated retrospective protocols.
Nested acquisition histories; 30-prediction detection at the fixed 10% comparison budget; corrective actions on identical queues; policies selected on development data and frozen for held-out evaluation.
Reconstructed sensitivities
Reuse cached predictions, model answers, timings, or scenario inputs to ask how a measured result changes under another configuration.
Expose heterogeneity, possible mechanisms, or failure modes without serving as independent confirmation of the headline result.
Extended routing-rate sensitivities; overlapping language and script subsets; contextual routers; structured-output checks; resampled sample-size and class-mixture analyses.
Appendix
Table 6: Evidence roles in the appendix. “Main” identifies the comparisons used for the paper’s principal answers; it does not mean prospectively registered. Reconstructed and exploratory results retain the qualifications given in their sections.
Figure 1: Experimental relationships among the research questions. RQ1 independently reorders already labelled AspectBench records and does not feed each acquisition history into the later experiments. RQ2–RQ4 form the stronger methodological sequence: risk signals define a queue, actions are compared on matched queues, and development data select a complete policy before test.
Scenario component
Basis
Cost (€)
Setup
Guidelines, training, supervision
5,200.00
Production and review
10,000 labels; 7,000 articles
14,791.67
E1 and E2 training
Three checkpoints each
0.60
DSPy calibration
Allocated endpoint time
4.40
Inference
100,000 future decisions
6.90
Illustrative total
Mixed experimental workloads
20,003.56
Appendix
Table 7: Illustrative accounting scenario, not an observed AspectBench annotation bill or a single end-to-end deployment. Direct accelerator rental excludes engineering, experimentation, unsuccessful runs, quality assurance, infrastructure, storage, maintenance, and staff time.
Workload
Model
GPU
Epochs
Train/eval batch
Accum.
Max. length
Precision
Uncertainty
Acquisition SL
SloBERTa
H100
≤30
16 / 32
2
512
fp32
none
Acquisition HBS
BERTić
H100
≤30
16 / 32
2
512
fp32
none
SentiCoref E1
SloBERTa
A40
≤5
8 / 24
4
512
bf16
30 passes
SentiCoref E2
XLM-R
A40
≤5
8 / 24
4
512
bf16
30 passes
NewsMTSC E1
XLM-R
A6000
≤5
8 / 24
4
512
bf16
30 passes
NewsMTSC E2
mDeBERTa-v3
A6000
≤5
8 / 24
4
512
bf16
30 passes
Appendix
Table 8: Encoder training and uncertainty procedure. An epoch is one complete pass through the training data; batch sizes are the numbers of records processed together during training and evaluation; accumulation is the number of training batches combined before a model update; and maximum length is the token limit. “fp32” and “bf16” are 32-bit and 16-bit numerical formats. Acquisition runs stop after three evaluations without improvement. External runs stop after two, use an update size of 2×10−5 , and apply weight decay 0.01 as regularisation. A separately verified update size is unavailable for the acquisition runs, so we do not infer one here.
Component
Setting used
Model producing test answers
Gemma 3 27B, 4-bit quantised
Model helping optimise instructions
Qwen 2.5 72B, 4-bit quantised
Instruction optimisation
DSPy MIPROv2, medium setting
Concurrent optimisation processes
8
Maximum output
1,024 tokens
Maximum article input
10,000 characters
Appendix
Table 9: LLM instruction-optimisation and answer-generation configuration.
Study component
Dataset and runs
Hardware
Measured workload
Time
RQ1 acquisition
HBS, 90 cells
H100
summed single-GPU time
90.08 h
RQ1 acquisition
SL, 90 cells
H100
summed single-GPU time
49.30 h
External E1
NewsMTSC, 3 trained models
A6000
training and prediction
1.16 h
External E2
NewsMTSC, 3 trained models
A6000
training and prediction
1.12 h
External E1
SentiCoref, 3 trained models
A40
training and prediction
0.51 h
External E2
SentiCoref, 3 trained models
A40
training and prediction
0.60 h
Appendix
Table 10: Recorded resource use. Encoder hours add together the time for which each GPU was occupied. LLM rows average the runs that hide or retain the target name while several requests are processed concurrently. They exclude server loading and one-time instruction optimisation, and should not be read as the time for one record or as physical GPU-hours.
E1 configuration
Records/s
A40-h/100k
3 checkpoints × 3 MC passes
56.36
0.493
3 checkpoints × 10 MC passes
16.77
1.656
Appendix
Table 11: Measured warm encoder throughput and extrapolated A40-hours. The 100,000-record column is computed from the 1,024-record benchmark rather than measured in a separate deployment. Nine stochastic predictions means three trained checkpoints with three stochastic forward passes each, not nine independently trained models.
Figure 2: Fixed training and test sizes and sentiment balance. AspectBench rows overlap: HBS contains the SR, SH, BS, and HR language subsets, while SR contains the two script subsets. The BS test slice is diagnostic only ( n=15 ; no negative example). The Slovenian split also has very few negative examples, whereas NewsMTSC is less skewed. These visible differences limit how directly early learning curves and small-subset scores can be compared.
Figure 3: All annotation designs plotted against the number of labelled records actually used. Early curves differ, especially for chronological HBS acquisition, but the designs broadly converge at larger budgets. Because several design choices change together, the lines are not a direct controlled comparison of one isolated factor.
Figure 4: Class-level changes after adding common-class records to a fixed negative-class base. Negative-class F1 can improve even though the number of negative training examples is held fixed. The added neutral and positive records also enlarge contextual and entity coverage, so the figure does not isolate class count as the cause.
Figure 5: Progressively growing acquisition histories with both Macro-F1 and QWK. Chronological HBS acquisition improves more slowly than the random histories, while the broad bands at the smallest SL budgets expose how uncertain a stopping decision is when there are very few negative examples.
Figure 6: Nine stochastic predictions compared with the internally matched 30-prediction reference at a fixed 10% low-confidence gate. Macro-F1 changes are percentage points. Error-capture retention divides the share of frozen 30-prediction E1 errors selected by the nine-prediction queue by the share selected by the reference queue; it can exceed 100%. Queue overlap is the intersection divided by the 10% queue size. AspectBench points average all 120 fixed three-of-ten pass-position subsets and their bars show the 2.5th–97.5th percentile range; external points use the retained pass-level reconstruction and the corresponding subset range. These are sensitivity intervals, not equivalence or non-inferiority tests.
Figure 7: How often predictions are correct at different reported confidence levels across AspectBench subsets and the external datasets. Several curves depart visibly from the diagonal, especially in sparse low-confidence regions, so the reported score should not be read as a literal probability that the prediction is correct.
Figure 8: Performance on the remaining records as increasingly many low-confidence predictions are withheld. Moderate withholding often improves the retained-set score, but high rates become unstable in small or class-skewed subsets. This figure does not apply a second model to the withheld records and therefore measures triage, not correction.
Gated E1+E2 average
Gated Gemma, target retained
Dataset
Δ F1
Correct
Harm
Δ F1
Correct
Harm
AspectBench HBS
+0.77
198
110
+0.53
98
67
AspectBench SL
+1.14
60
36
+1.23
22
11
SentiCoref SL
−2.52
20
42
+0.90
11
9
NewsMTSC EN
+1.51
25
12
+1.28
16
5
Appendix
Table 12: Correction and harm counts for the main same-queue 10% comparison. Macro-F1 changes are percentage points. Counts must be interpreted together with the sentiment classes changed.
Figure 9: Actions applied to the same frozen 216-record SentiCoref queue. Left: corrections of E1 errors and harms to E1-correct predictions. E2 replacement and averaging correct more E1 errors than Gemma but also damage many more correct decisions. Right: record-level correctness of averaging and target-retaining Gemma; Gemma alone is correct on 51 records, whereas averaging alone is correct on 27. These counts describe one frozen queue and do not establish an architectural cause.
Evaluation
E1
E2 on every record
Gemma without E1 information
E1+E2 average on every record
Gated E1+Gemma at 10%
E1+Gemma on every record
Perfect correction at 10%
AspectBench HBS
82.63
+0.33
−22.90
+1.22
+0.58 / +0.53
−3.82 / −1.13
+5.75
AspectBench SL
74.53
+0.72
−31.98
+1.27
+1.06 / +1.23
−1.44 / −3.80
+10.36
AspectBench SR
81.42
+1.31
−22.43 / −19.89
+2.25
+0.00 / +0.29
−2.07 / −1.02
+5.90
AspectBench SH
82.86
+3.53
−12.01 / −14.18
+4.10
+0.99 / +1.55
−0.21 / +0.72
+6.13
AspectBench HR
77.45
+4.26
−26.37 / −23.99
+3.73
−0.28 / −0.76
−1.15 / −1.74
+6.76
AspectBench SR-CYR
74.93
+7.10
−20.83 / −15.21
+7.41
+0.01 / +0.00
−2.19 / +0.52
+4.29
Appendix
Table 13: Consolidated strategy results. E1 is absolute Macro-F1; all remaining cells are point changes from E1. A slash separates Gemma results when the target’s name is hidden or retained. Gated strategies and perfect correction use the least-confident 10% of E1 predictions, whereas the other strategies process every record. AspectBench rows overlap, and the resource budgets of the strategies differ.
Figure 10: Two actions applied to the same least-confident 10% of document–entity ELSA records. Bars show Macro-F1 change from E1 and intervals come from paired source-document resampling. Encoder averaging improves AspectBench and NewsMTSC but harms SentiCoref. Gemma changes less in magnitude across these four comparisons, although the SentiCoref interval includes both improvement and deterioration. The figure supports RQ3: an error-rich queue does not identify the action that should receive it.
Figure 11: Actions on the same least-confident 10% of E1 predictions. Probability averaging is positive for most AspectBench evaluations and NewsMTSC but negative on SentiCoref; Gemma’s changes are smaller and depend on whether the target name is retained. Holding the queue fixed shows that action choice, not only error detection, determines the result.
Figure 12: Maximum possible improvement and the improvement actually achieved by Gemma on the same selected records. The realised orange bars are much smaller than the blue oracle headroom and are sometimes negative, which makes the gap between finding errors and correcting them visible.
Figure 13: SentiCoref actions across progressively larger sets of low-confidence predictions. Replacing E1 with E2 or averaging the encoders becomes increasingly harmful as coverage grows, whereas Gemma with the target name retained provides a modest gain at some intermediate rates.
Figure 14: NewsMTSC actions across progressively larger sets of low-confidence predictions. Encoder replacement and averaging help most at low or moderate coverage, while the unmasked Gemma curve continues to improve across the plotted range. These test-set comparisons describe sensitivity to queue size; they do not select a deployment threshold.
Figure 15: Comparison of fixed Gemma call rates in the internally nested 10% to 50% experiment. The left panel shows the observed Macro-F1 gains, and the middle panel compares every larger rate with 10%. The right panel summarises regret from the best observed point on each test curve. These test-selected maxima describe heterogeneity; they are not deployment thresholds.
Figure 16: Confidence triage and realised gated-escalation utility. As coverage grows, the queue captures more E1 errors but becomes less concentrated for errors. Removing low-confidence records reduces the retained error rate in all nine evaluated collections and subsets, yet neither the retained-set curves nor the gate rate determine how much Macro-F1 Gemma ultimately recovers.
Figure 17: E1, E2, and encoder combinations across overlapping AspectBench language and script subsets. Probability averaging and learned stackers are often strong, but the leading combination changes across subsets; no single combiner dominates every subset.
Figure 18: The pooled HBS model compared with language-specific models on the same test records. The pooled model shows larger gains for SH, HR, and the Serbian Cyrillic subset than for several other subsets, but heterogeneous intervals and different training pools prevent a causal claim that pooling itself produces the gains.
Figure 19: Comparison of variants that use separate fields from the LLM output. The experiment does not show a stable gain over the simpler procedure.
Figure 20: Development-selected policies on held-out test data. Dotted curves show the best discovery-set signal for each action/rate and solid curves the retrospective test envelope; stars are frozen choices and crosses hindsight test-best grid members. SentiCoref selects masked Gemma at 30% ( +1.02 test points) and NewsMTSC target-retaining Gemma at 30% ( +1.71 ). Hindsight gains of +4.15 and +2.26 leave regrets of 3.13 and 0.55 . Strong discovery–test rank correlation on SentiCoref does not ensure the test optimum, while NewsMTSC’s weak negative correlation still yields a near-best selected policy.
Figure 21: Fixed actions and learned per-record routers inside the same lowest-confidence 30% test region. Blue bars are SentiCoref and hatched orange bars are NewsMTSC. The oracle uses reference labels only to show available headroom. E1 and E1+E2 name the router’s input features; E2 replacement, averaging, and Gemma remain candidate actions.
Figure 22: Two sensitivity analyses for the frozen development-selected policy. Top: Macro-F1 changes over 30 predetermined source-group-aware subsets at three sample sizes; small SentiCoref samples can reverse the gain, and variability contracts as more groups are retained. Bottom: five controlled class-mixture scenarios with 30 source-group bootstrap replicates each. The numerical threshold often routes more than the fixed 30% top- k policy, so its gain is not a same-budget comparison. Both panels resample existing test records rather than representing independent deployments or new language, topic, or concept drift.