Media bias in news articles operates through subtle linguistic cues---loaded language, selective framing, and strategic omission---that resist single-model detection and have traditionally required large annotated corpora for supervised training. We ask whether structured multi-agent deliberation can serve as a principled, training-free alternative to supervised classification for this task. We introduce MABPD (Multi-Agent Bias Probing & Detection), a pipeline in which three specialized LLM agents analyze an article from complementary perspectives and resolve disagreements through a Structured Argument Debate (SAD) protocol. SAD implements a domain-motivated asymmetric burden of proof---biased claims without grounded textual evidence carry zero weight---combined with role-weighted voting and post-consensus verification, replacing task-specific supervised decision boundaries with explicit deliberative structure. Ablation confirms that this structured deliberation, not mere agent parallelism, drives performance: removing the debate module reduces F1 by up to 10.6 points. On the BABE benchmark (4,121 expert-annotated sentences), MABPD achieves 83.4% macro F1 on the held-out test split---within 0.7 percentage points (pp) of the supervised SOTA (MAGPIE, 84.1% macro F1; Horych et al., 2024)---without any task-specific training or threshold tuning on annotated data. Cross-dataset evaluation on the SemEval 2019 HyperPartisan corpus (644 articles) yields 75.0% zero-shot accuracy, within 7.2 pp of the supervised SOTA accuracy (82.2%; Kiesel et al. 2019), confirming transfer across annotation regimes. We release the full pipeline and evaluation code.
Figures & tables
Framework
Evidence
Verifier Policy
MAD
None
Symmetric
D2D
Implicit
Symmetric
ConfMAD
None
Symmetric
MABPD
Explicit
Asym (B → N)
Table 1: Architectural comparison of MABPD vs existing MAD frameworks.
Figure 1: Architecture overview of the MABPD pipeline. Three functionally independent agents analyze text (BiasAgent first, then EvidenceAgent and FramingAgent without access to its output); their outputs are aggregated through a deterministic scoring agent and a three-path consensus engine (which may invoke the Structured Argument Debate protocol), and a VerifierAgent performs post-consensus validation.
Statistic
Train
Test
Total
Sentences
3,121
1,000
4,121
Biased
1,740
559
2,299
Neutral
1,381
441
1,822
Bias rate (%)
55.8
55.9
55.8
Mean word count
30.7
31.7
31.0
Topics
23
23
23
Table 2: BABE dataset statistics. Both splits maintain a similar bias rate ( ≈ 56%). “Train” = BABE official train partition ( ds[‘‘train’’] ); “Test” = held-out evaluation split ( ds[‘‘test’’] ). Since MABPD uses no BABE data for training or threshold tuning, all 4,121 sentences are used for evaluation.
System
Acc.
Prec.
Rec.
F1
MF1
BERT-base ‡
74.1
71.4
76.8
74.0
—
RoBERTa-base ‡
81.9
80.1
84.5
82.2
—
MAGPIE (MTL:All) †
—
—
—
—
84.1
MABPD (Full)
83.5
88.0
81.6
84.7
83.4
Table 3: Supervised Protocol — Held-out Test Split (1,000 articles). Supervised baselines are fine-tuned on the 3,121-article BABE train partition; MABPD uses no BABE data for training. Separating evaluation regimes ensures that MABPD ’s zero-shot performance is compared against supervised models on identical test data. Bold marks the best value in each column, whichever system attains it. The F1 column is binary F1 and MF1 is macro F1; MAGPIE reports only macro F1, so it is listed in the MF1 column and left blank elsewhere. The regime-matched comparison is therefore macro-to-macro: MABPD 83.4 vs. MAGPIE 84.1 ( − 0.7 pp). † 84.1% confirmed as macro F1 via source code inspection ( F1Score(average=‘‘macro’’) , head.py ). ‡ BERT and RoBERTa fine-tuned on the BABE train split (3,121 articles).
System
Acc.
Prec.
Rec.
F1
MF1
MABPD (Single)
71.1
90.1
54.0
67.6
—
MABPD (No Debate)
78.1
93.1
65.6
76.9
—
MABPD (Full) ∗
84.7
87.2
85.2
86.2
84.6
Table 4: Zero-shot Protocol — Full Dataset (4,121 articles). All three configurations use the same architectural thresholds with no training on BABE data. This regime isolates the contribution of each pipeline component under fully zero-shot conditions. Bold marks the best value in each column, whichever configuration attains it: the No-Debate arm has the highest precision, at a large cost in recall. ∗ Full system significantly outperforms No-Debate ( p<0.001 , paired bootstrap, 10K resamples).
Full dataset (4,121)
Test split (1,000)
Config
Acc
P
R
F1
Δ
Acc
P
R
F1
Δ
Full System
84.7
87.2
85.2
86.2
—
83.5
88.0
81.6
84.7
—
− Debate
78.1
93.1
65.6
76.9
− 9.3
76.0
93.2
61.5
74.1
− 10.6
Single Agent
71.1
90.1
54.0
67.6
− 18.6
69.7
91.0
50.8
65.2
− 19.5
Heterogeneous
—
83.4
88.0
81.4
84.6
− 0.1
Table 5: Ablation study on both BABE splits. P = precision, R = recall; Δ = Δ F1 vs. Full System. Every row was measured in the same serving environment (§ 4.2 ), so all Δ values share a single reference point. The debate module contributes − 9.3 F1 (full) and − 10.6 (test split); multi-agent analysis overall contributes − 18.6 (full) and − 19.5 (test split) vs. single agent. Results are consistent across both evaluation regimes.
Figure 2: F1 and accuracy comparison across ablation scenarios. The full system substantially outperforms both reduced configurations, with the gap driven primarily by recall improvements from the debate module.
Model Setup
Acc
Prec
Rec
Bin F1
Mac F1
Single-Agent (Qwen-3.5)
80.5%
93.5%
69.9%
80.0%
80.5%
MABPD (Ours)
83.5%
88.0%
81.6%
84.7%
83.4%
Table 6: Single-Agent LLM Baseline Comparison on the BABE test split. MABPD demonstrates substantial improvements over a strong external single-agent baseline (Qwen-3.5 122B-A10B MoE via NVIDIA NIM).
Component
Output
Snippet (GT: Biased)
“The concept of changing one’s biological sex is, of course, nonsense, as sex is determined by unalterable chromosomes.”
Single-Agent Baseline
Neutral (Conf: 0.501. Misses loaded implication.)
BiasAgent Evidence
Extracted loaded language (“nonsense”) and ideological framing.
Predicted Neutral (Failed to detect structural bias).
SAD Debate Reasoning
The explicit loaded language and unsupported claims provided stronger evidence, overriding the neutral perspective.
Table 7: Example 1: Epistemological Framing Bias. MABPD captures subtle epistemological bias missed by a strong single-agent baseline.
Parameter
Value
Justification
BiasAgent weight ( w1 )
0.50
2:1:1 agent weighting: the BiasAgent is the primary detector, carrying twice the weight of the supporting agents.
EvidenceAgent weight ( w2 )
0.25
Supporting agent under the 2:1:1 weighting.
FramingAgent weight ( w3 )
0.25
Supporting agent under the 2:1:1 weighting.
Evidence factor f (biased, ∣e∣=0 )
0.00
Burden-of-proof axiom: a bias claim with no grounded textual evidence carries zero weight.
Tier-B neutral veto ceiling
0.30
Deliberately conservative; demands stronger evidence before flagging bias.
Tier-A neutral veto ceiling
0.40
Deliberately conservative; demands stronger evidence before flagging bias.
Table 8: Architectural weights and decision thresholds, with the design rationale for each value.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Sev.
Definition (abbreviated)
Loaded language
High
Words/phrases with strong emotional connotations used to influence perception.
False balance
Med.
Presenting two sides as equally valid when evidence supports one.
Cherry-picking
High
Selective use of facts/data supporting a predetermined narrative.
Political framing
High
Presenting information favoring a particular political position.
Dog whistle
High
Language appearing neutral but carrying coded connotations.
Scapegoating
High
Blaming a specific group for complex problems without evidence.
Appendix
Table 9: Knowledge base bias categories injected into agent prompts.
Component
Output
Snippet (GT: Biased)
“While ordinary families struggle to afford basic necessities, wealthy corporations continue to hoard record profits, shielded by a tax system rigged in their favor.”
Single-Agent Baseline
Neutral (Conf: 0.520. Treats economic claims as factual reporting.)
BiasAgent Evidence
Extracted loaded language (“hoard,” “rigged”) and emotional economic framing.
EvidenceAgent Evidence
Flagged unsupported claim (“rigged in their favor”) and loaded term (“hoard record profits”).
FramingAgent Evidence
Detected economic inequality framing with progressive ideological lean.
SAD Debate Reasoning
Loaded economic language and unattributed systemic claims warranted bias classification despite factual economic context.
Appendix
Table 10: Example 2: Left-Leaning Framing Bias.
Component
Output
Snippet (GT: Biased)
“The flood of undocumented migrants pouring across the border has overwhelmed local communities, yet bureaucrats in Washington remain content to do nothing.”
Single-Agent Baseline
Neutral (Conf: 0.485. Interprets as factual border reporting.)
BiasAgent Evidence
Extracted loaded language (“flood,” “pouring”) and political labeling (“bureaucrats”).
EvidenceAgent Evidence
Flagged dehumanizing metaphor (“flood of … pouring”) and unsupported claim (“content to do nothing”).
FramingAgent Evidence
Detected us/them framing (communities vs. Washington) with conservative ideological lean.
SAD Debate Reasoning
Dehumanizing metaphor, unsupported attribution of intent, and us/them framing constituted strong multi-dimensional bias evidence.
Appendix
Table 11: Example 3: Right-Leaning Framing Bias.
Figure 4: Component F1 drop when each stage is removed. Combined loss vs. single agent: 18.6 points.
Figure 5: Multi-metric radar across ablation configurations.
Configuration
Pred. Biased
Pred. Neutral
Full System
True Biased
1,958 (TP)
341 (FN)
True Neutral
288 (FP)
1,534 (TN)
− Debate
True Biased
1,507 (TP)
792 (FN)
True Neutral
112 (FP)
1,710 (TN)
Single Agent
True Biased
1,242 (TP)
1,057 (FN)
True Neutral
136 (FP)
1,686 (TN)
Appendix
Table 12: Confusion matrices on the full BABE benchmark (4,121 articles).
Table 14: Threshold sensitivity sweep. MF1 peaks at 0.50; full 0.30–0.80 sweep in released code.
Run
Acc.
Prec.
Rec.
F1
Run 1
84.69
87.20
85.04
86.10
Run 2
84.74
87.18
85.17
86.16
Δ
0.05
0.02
0.13
0.06
Appendix
Table 15: Reproducibility across two independent runs (F1 Δ = 0.06 pp).
Figure 6: Heterogeneous LLM architecture pilot on the held-out test split ( N=1,000 ). Top: single-agent baseline versus the heterogeneous multi-agent pipeline. Bottom: confusion matrix of the heterogeneous pipeline.
Media bias detection has predominantly been framed as a classification task: assign a political label to an article or outlet. We argue this framing is too shallow: it identifies that bias exists but not where, how, or crucially, what is structurally omitted. We present NewsLens, a five-agent adversarial pipeline for structured news bias navigation. A Fact Verifier, Progressive Framing Analyst, Conservative Framing Analyst, Propaganda Detector, and Neutral Summarizer collaborate to deconstruct articles into interpretable framing maps, exposing ideological omissions, rhetorical manipulation, and framing boundaries. The system is evaluated on 15 articles across four geopolitical event clusters (India-Pakistan Kashmir, Gaza, Climate Policy, Ukraine) using Qwen2.5-3B-Instruct (4-bit quantised, Google Colab T4), with cross-model validation using Mistral 7B on the Kashmir cluster. Center outlets show the highest mean Perspective Divergence Score (PDS: Qwen 0.907, Mistral 0.729 on Kashmir subset); conservative-framing outlets show the highest mean Manipulation Index (MI: 0.600 across both models). Cross-model comparison shows high consistency for high-propaganda content (Republic World delta-PDS=0.125, MI=0.8 both models) and greater variance for nuanced reporting. Mann-Whitney U tests find no statistically significant between-group differences at n=15, reported honestly as a sample-size limitation confirmed by post-hoc power analysis. A partial ablation removing the Propaganda Detector shows degraded omission precision in the Neutral Summarizer output. The architecture extends prior lexical-geometric bias work to agentic LLM reasoning, and is fully reproducible using open-weight models without API keys.
Media bias detection relies on definitions and examples that specify what counts as bias, yet these specifications often vary across datasets or remain implicit, even when given the same name. Such variation makes it unclear whether models trained for the same bias category learn the same construct or different phenomena, a problem largely overlooked in prior work. We examine how definition choice affects bias annotation in a between-subjects experiment with 354 participants and a parallel evaluation with four LLMs. Participants and models rate six news articles across four bias categories using definitions that vary in conceptual framing and elaboration. Across 8,496 human and 28,800 LLM ratings, we find that the conceptual target of a definition drives annotation divergence, while construct-preserving elaboration does not: conceptual framing significantly shifts annotations for humans and does so even more strongly for LLMs. We discuss implications for construct specification in annotation protocols and prompt-based measurement, and consider how definitional sensitivity may propagate to downstream classification beyond media bias. We also release MUDD, the Multi-Definition Bias Detection Dataset.
Martin Wessel, Timo Spinde, Jürgen Pfeffer +1
Technical University of Munich · Center for Digital Technology and Management (CDTM) · National Institute of Informatics (NII), Tokyo +1
This paper presents a pipeline for analyzing media bias and framing in online news. The pipeline groups articles into topics and events, adds named-entity and sentiment annotations, and compares news sources through people mentions, source-level tone, and event-level coverage patterns. We apply it to 8,358 Albanian news articles collected from GDELT and compare the resulting annotations with GDELT's automated annotations. The results show moderate agreement for sentiment and entity extraction, as well as additional person-entity pairs that can potentially support the bias analysis. We compare two annotation prompts and find that stricter sentiment-validation rules remove label-score inconsistencies but increase execution time and reduce annotation coverage. Based on these results, the simpler prompt is used for the rest of the analysis. We have provided sample analysis on source-level framing pro les, person-level tone differences across sources, and event-level gatekeeping and coverage indicators. These outputs show how the same news collection can be used to examine what sources cover, how they describe public figures, and where coverage is concentrated. The approach is particularly useful in settings where manually verified datasets or specialized language tools are limited.
Klesti Hoxha, Olti Qirici
Department of Informatics Faculty of Natural Sciences University of Tirana Tirana, Albania