From Entity Mentions to Tone: An LLM-Based Pipeline for Media Bias Analysis
Organizations: Department of Informatics Faculty of Natural Sciences University of Tirana Tirana, Albania
Abstract
This paper presents a pipeline for analyzing media bias and framing in online news. The pipeline groups articles into topics and events, adds named-entity and sentiment annotations, and compares news sources through people mentions, source-level tone, and event-level coverage patterns. We apply it to 8,358 Albanian news articles collected from GDELT and compare the resulting annotations with GDELT's automated annotations. The results show moderate agreement for sentiment and entity extraction, as well as additional person-entity pairs that can potentially support the bias analysis. We compare two annotation prompts and find that stricter sentiment-validation rules remove label-score inconsistencies but increase execution time and reduce annotation coverage. Based on these results, the simpler prompt is used for the rest of the analysis. We have provided sample analysis on source-level framing pro les, person-level tone differences across sources, and event-level gatekeeping and coverage indicators. These outputs show how the same news collection can be used to examine what sources cover, how they describe public figures, and where coverage is concentrated. The approach is particularly useful in settings where manually verified datasets or specialized language tools are limited.
Figures & tables
| Category | Description | Reference |
|---|---|---|
| Selection | Editorial “gatekeeping”; choosing what is deemed newsworthy or omitting specific details. | [ 3 ] |
| Tone | The emotional “lean” or sentiment valence applied to specific actors or events. | [ 9 ] |
| Reporting/Source | The long-term reputation, systemic lean, and ideological history of a news domain. | [ 10 ] |
| Depth | Uneven coverage space or word count dedicated to one side of a story. | [ 4 ] |
| Category | Description | Reference |
|---|---|---|
| Selection | Structural visibility of narrative elements; what information is presented or omitted. | [ 3 , 9 ] |
| Tone | The emotional undertone, spin, or evaluative posture applied to the news narrative. | [ 4 , 9 ] |
| Emphasis | The relative importance and priority given to specific themes or viewpoints within a text. | [ 3 ] |
| Targeted | Entity-specific framing shifts depending on the politician or actor involved. | [ 11 ] |
| Metric | Value |
|---|---|
| Articles | 8,358 |
| Unique news sources | 124 |
| Topic clusters | 50 |
| Event clusters | 3,551 |
| Multi-source events | 1,340 |
| Articles per source (min/max/mean/median) | 1 / 667 / 67.4 / 23 |
| Metric | Prompt v1 | Prompt v2 |
|---|---|---|
| Sentiment Labeled Articles | 51.75% | 37.20% |
| Inconsistent Sentiment Label / Score | 8.53% | 0.0% |
| NER-Labeled Articles (any entity) | 39.97% | 27.20% |
| Articles with Person Entities (PER) | 8.25% | 5.77% |
| Metric | Prompt v1 | Prompt v2 |
|---|---|---|
| Comparable articles | 3,612 | 3,109 |
| Label agreement rate | 58.72% | 62.34% |
| Metric | Prompt v1 | Prompt v2 |
|---|---|---|
| Evaluable pairs | 690 | 483 |
| Macro LLM agreement rate | 36.83% | 36.12% |
| Macro GDELT agreement rate | 71.46% | 70.41% |
| Macro balanced agreement rate | 45.30% | 44.44% |
| Version | LLM people | GDELT people | Precision | Recall | F1 |
|---|---|---|---|---|---|
| v1 | 1860 | 799 | 30.2% | 70.3% | 42.3% |
| v2 | 1309 | 560 | 29.8% | 69.6% | 41.7% |
| Source | Topics | Articles | Pos. | Neu. | Neg. | Mean score | Avg. bias | Bias vs. corpus |
|---|---|---|---|---|---|---|---|---|
| tvkoha.tv | 2 | 10 | 22.7% | 27.3% | 50.0% | +0.100 | -0.857 | -0.665 |
| 3shi.net | 4 | 16 | 40.7% | 29.6% | 29.6% | +0.170 | +0.200 | +0.392 |
| tv21.tv | 5 | 27 | 11.4% | 50.0% | 38.6% | +0.177 | -0.554 | -0.362 |
| kosovapress.com | 4 | 17 | 9.7% | 25.8% | 64.5% | -0.203 | -0.542 | -0.350 |
| rtsh.al | 1 | 33 | 38.5% | 33.3% | 28.2% | +0.108 | +0.151 | +0.344 |
| epokaere.com | 3 | 10 | 27.6% | 34.5% | 37.9% | +0.324 | +0.111 | +0.303 |
| Person | Source | Mentioned articles | Mean score | Tone balance | Pos. | Neg. |
|---|---|---|---|---|---|---|
| Benjamin Netanyahu | balkanweb.com | 5 | +0.000 | -0.800 | 0.0% | 80.0% |
| Benjamin Netanyahu | panorama.com.al | 6 | +0.300 | -0.333 | 16.7% | 50.0% |
| Donald Trump | opinion.al | 8 | -0.300 | -0.750 | 12.5% | 87.5% |
| Donald Trump | lajmi.net | 6 | +0.400 | +0.500 | 50.0% | 0.0% |
| Edi Rama | syri.net | 10 | +0.190 | -1.000 | 0.0% | 100.0% |
| Edi Rama | kohajone.com | 8 | +0.600 | +0.250 | 62.5% | 37.5% |
| Event | Articles | Covering sources | Omitting sources | Gatekeeping score | Framing polarity |
|---|---|---|---|---|---|
| Event 1 | 48 | 21 | 103 | 0.8306 | +0.0227 |
| Event 2 | 46 | 35 | 89 | 0.7177 | +0.1818 |
| Event 3 | 39 | 19 | 105 | 0.8468 | +0.0625 |
| Event 4 | 37 | 22 | 102 | 0.8226 | — |
| Event 5 | 33 | 19 | 105 | 0.8468 | -1.0000 |
| Event 6 | 33 | 10 | 114 | 0.9194 | +0.0588 |