Finding Errors Is Not Enough: Selective Model Escalation for Entity-Targeted Sentiment Analysis
Organizations: Jožef Stefan Institute, Ljubljana, Slovenia · Jožef Stefan International Postgraduate School, Ljubljana, Slovenia · University of La Rochelle, La Rochelle, France · Queen Mary University of London, London, United Kingdom ·
Abstract
Improving a sentiment classifier with limited resources involves more than simply deciding how much data to collect or which model to train. A labelled collection may lack examples of an important class. Once trained, a classifier may benefit from another model even when that model is weaker on its own. We study these decisions in document-level Entity-Level Sentiment Analysis (ELSA), primarily using South Slavic news. We first compare acquisition orders as collections grow. We then use uncertainty to choose which predictions to revisit, how many to process, and whether an alternative encoder or local LLM can correct them. Finally, we select complete routing policies on development data for held-out evaluation. Minority-prioritised acquisition helps at some very small training sizes, but not consistently thereafter. On SentiCoref, an oracle correcting all errors in the least-confident 10% without harming correct predictions would add 9.14 Macro-F1 points; however the best evaluated action gains only 0.90. Encoder averaging helps on AspectBench and NewsMTSC but harms SentiCoref, where selective LLM assistance is more promising. Both development-selected policies route 30% of records to the encoder-assisted LLM, yielding observed gains of 1.02 points on SentiCoref (interval includes zero) and 1.71 on NewsMTSC, at substantially different inference costs.
Figures & tables
| Dataset | Test | Neg. / Neu. / Pos. (%) |
|---|---|---|
| AspectBench HBS | 12,407 | 7.5 / 49.5 / 43.0 |
| AspectBench SL | 7,740 | 1.2 / 82.6 / 16.2 |
| SentiCoref SL | 2,155 | 12.9 / 75.1 / 12.0 |
| NewsMTSC EN | 1,146 | 37.4 / 39.7 / 22.9 |
| Dataset | Base | Selected | Concentration | Perfect |
|---|---|---|---|---|
| error | error | F1 | ||
| HBS | 13.18 | 42.22 | 3.20 | +5.75 |
| SL | 7.60 | 33.59 | 4.42 | +10.36 |
| SentiCoref | 20.05 | 41.67 | 2.08 | +9.14 |
| NewsMTSC | 16.49 | 45.22 | 2.74 | +5.05 |
| Scope | Dataset | E1 | E2 | E1+E2 average | Gemma without E1 information | E1-assisted Gemma |
|---|---|---|---|---|---|---|
| All records | AspectBench HBS | 82.63 | +0.33 | +1.22 | ||
| AspectBench SL | 74.53 | +0.72 | +1.27 | |||
| SentiCoref SL | 59.91 | +0.40 | ||||
| NewsMTSC EN | 82.14 | +0.48 | +1.89 | +0.50 | ||
| Scope | Dataset | E2 replacement | E1+E2 average | More-confident encoder | E1-assisted Gemma masked / unmasked | Perfect correction |
| Selected 10% | AspectBench HBS | +0.50 | +0.77 | +0.74 | +0.58 / +0.53 | +5.75 |
| Dataset | Action | Route | F1 [95% CI] | GPU-min per 1k |
|---|---|---|---|---|
| SentiCoref | Keep E1 | 0% | 0.00 [–] | 0.00 |
| Avg(E1,E2) | 10% | [ ] | 0.03 | |
| Unmasked Gemma | 10% | [ ] | 7.89 | |
| Selected Gemma | 30% | [ ] | 43.71 | |
| NewsMTSC | Keep E1 | 0% | 0.00 [–] | 0.00 |
| Avg(E1,E2) | 10% | [ ] | 0.03 |
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
| Evidence role | Purpose | Analyses |
|---|---|---|
| Main comparisons | Directly support the four headline answers under their stated retrospective protocols. | Nested acquisition histories; 30-prediction detection at the fixed 10% comparison budget; corrective actions on identical queues; policies selected on development data and frozen for held-out evaluation. |
| Reconstructed sensitivities | Reuse cached predictions, model answers, timings, or scenario inputs to ask how a measured result changes under another configuration. | Reduced-pass queue reconstruction; stored-output routing-rate curves; throughput extrapolations and illustrative deployment accounting. |
| Exploratory diagnostics | Expose heterogeneity, possible mechanisms, or failure modes without serving as independent confirmation of the headline result. | Extended routing-rate sensitivities; overlapping language and script subsets; contextual routers; structured-output checks; resampled sample-size and class-mixture analyses. |
| Scenario component | Basis | Cost (€) |
|---|---|---|
| Setup | Guidelines, training, supervision | 5,200.00 |
| Production and review | 10,000 labels; 7,000 articles | 14,791.67 |
| E1 and E2 training | Three checkpoints each | 0.60 |
| DSPy calibration | Allocated endpoint time | 4.40 |
| Inference | 100,000 future decisions | 6.90 |
| Illustrative total | Mixed experimental workloads | 20,003.56 |
| Workload | Model | GPU | Epochs | Train/eval batch | Accum. | Max. length | Precision | Uncertainty |
|---|---|---|---|---|---|---|---|---|
| Acquisition SL | SloBERTa | H100 | 16 / 32 | 2 | 512 | fp32 | none | |
| Acquisition HBS | BERTić | H100 | 16 / 32 | 2 | 512 | fp32 | none | |
| SentiCoref E1 | SloBERTa | A40 | 8 / 24 | 4 | 512 | bf16 | 30 passes | |
| SentiCoref E2 | XLM-R | A40 | 8 / 24 | 4 | 512 | bf16 | 30 passes | |
| NewsMTSC E1 | XLM-R | A6000 | 8 / 24 | 4 | 512 | bf16 | 30 passes | |
| NewsMTSC E2 | mDeBERTa-v3 | A6000 | 8 / 24 | 4 | 512 | bf16 | 30 passes |
| Component | Setting used |
|---|---|
| Model producing test answers | Gemma 3 27B, 4-bit quantised |
| Model helping optimise instructions | Qwen 2.5 72B, 4-bit quantised |
| Instruction optimisation | DSPy MIPROv2, medium setting |
| Concurrent optimisation processes | 8 |
| Maximum output | 1,024 tokens |
| Maximum article input | 10,000 characters |
| Study component | Dataset and runs | Hardware | Measured workload | Time |
|---|---|---|---|---|
| RQ1 acquisition | HBS, 90 cells | H100 | summed single-GPU time | 90.08 h |
| RQ1 acquisition | SL, 90 cells | H100 | summed single-GPU time | 49.30 h |
| External E1 | NewsMTSC, 3 trained models | A6000 | training and prediction | 1.16 h |
| External E2 | NewsMTSC, 3 trained models | A6000 | training and prediction | 1.12 h |
| External E1 | SentiCoref, 3 trained models | A40 | training and prediction | 0.51 h |
| External E2 | SentiCoref, 3 trained models | A40 | training and prediction | 0.60 h |
| E1 configuration | Records/s | A40-h/100k |
|---|---|---|
| 3 checkpoints 3 MC passes | 56.36 | 0.493 |
| 3 checkpoints 10 MC passes | 16.77 | 1.656 |
| Gated E1+E2 average | Gated Gemma, target retained | |||||
| Dataset | F1 | Correct | Harm | F1 | Correct | Harm |
| AspectBench HBS | +0.77 | 198 | 110 | +0.53 | 98 | 67 |
| AspectBench SL | +1.14 | 60 | 36 | +1.23 | 22 | 11 |
| SentiCoref SL | 20 | 42 | +0.90 | 11 | 9 | |
| NewsMTSC EN | +1.51 | 25 | 12 | +1.28 | 16 | 5 |
| Evaluation | E1 | E2 on every record | Gemma without E1 information | E1+E2 average on every record | Gated E1+Gemma at 10% | E1+Gemma on every record | Perfect correction at 10% |
|---|---|---|---|---|---|---|---|
| AspectBench HBS | 82.63 | +0.33 | +1.22 | +0.58 / +0.53 | / | +5.75 | |
| AspectBench SL | 74.53 | +0.72 | +1.27 | +1.06 / +1.23 | / | +10.36 | |
| AspectBench SR | 81.42 | +1.31 | / | +2.25 | +0.00 / +0.29 | / | +5.90 |
| AspectBench SH | 82.86 | +3.53 | / | +4.10 | +0.99 / +1.55 | / +0.72 | +6.13 |
| AspectBench HR | 77.45 | +4.26 | / | +3.73 | / | / | +6.76 |
| AspectBench SR-CYR | 74.93 | +7.10 | / | +7.41 | +0.01 / +0.00 | / +0.52 | +4.29 |
Sentiment Analysis
Sentiment analysis detects and measures opinions, emotions, and evaluative stance in text and multimodal data.
See all 136 papers