cs.CLSep 4, 2026
SaveMABPD: Multi-Agent Bias Probing & Detection via Structured Argument Debate
Organizations: Graphic Era University, Dehradun, India
Abstract
Media bias in news articles operates through subtle linguistic cues---loaded language, selective framing, and strategic omission---that resist single-model detection and have traditionally required large annotated corpora for supervised training. We ask whether structured multi-agent deliberation can serve as a principled, training-free alternative to supervised classification for this task. We introduce MABPD (Multi-Agent Bias Probing & Detection), a pipeline in which three specialized LLM agents analyze an article from complementary perspectives and resolve disagreements through a Structured Argument Debate (SAD) protocol. SAD implements a domain-motivated asymmetric burden of proof---biased claims without grounded textual evidence carry zero weight---combined with role-weighted voting and post-consensus verification, replacing task-specific supervised decision boundaries with explicit deliberative structure. Ablation confirms that this structured deliberation, not mere agent parallelism, drives performance: removing the debate module reduces F1 by up to 10.6 points. On the BABE benchmark (4,121 expert-annotated sentences), MABPD achieves 83.4% macro F1 on the held-out test split---within 0.7 percentage points (pp) of the supervised SOTA (MAGPIE, 84.1% macro F1; Horych et al., 2024)---without any task-specific training or threshold tuning on annotated data. Cross-dataset evaluation on the SemEval 2019 HyperPartisan corpus (644 articles) yields 75.0% zero-shot accuracy, within 7.2 pp of the supervised SOTA accuracy (82.2%; Kiesel et al. 2019), confirming transfer across annotation regimes. We release the full pipeline and evaluation code.
Figures & tables
| Framework | Evidence | Verifier Policy |
|---|---|---|
| MAD | None | Symmetric |
| D2D | Implicit | Symmetric |
| ConfMAD | None | Symmetric |
| MABPD | Explicit | Asym (B N) |
Table 1: Architectural comparison of MABPD vs existing MAD frameworks.
Figure 1: Architecture overview of the MABPD pipeline. Three functionally independent agents analyze text (BiasAgent first, then EvidenceAgent and FramingAgent without access to its output); their outputs are aggregated through a deterministic scoring agent and a three-path consensus engine (which may invoke the Structured Argument Debate protocol), and a VerifierAgent performs post-consensus validation.
| Statistic | Train | Test | Total |
|---|---|---|---|
| Sentences | 3,121 | 1,000 | 4,121 |
| Biased | 1,740 | 559 | 2,299 |
| Neutral | 1,381 | 441 | 1,822 |
| Bias rate (%) | 55.8 | 55.9 | 55.8 |
| Mean word count | 30.7 | 31.7 | 31.0 |
| Topics | 23 | 23 | 23 |
Table 2: BABE dataset statistics. Both splits maintain a similar bias rate ( 56%). “Train” = BABE official train partition ( ds[‘‘train’’] ); “Test” = held-out evaluation split ( ds[‘‘test’’] ). Since MABPD uses no BABE data for training or threshold tuning, all 4,121 sentences are used for evaluation.
| System | Acc. | Prec. | Rec. | F1 | MF1 |
|---|---|---|---|---|---|
| BERT-base ‡ | 74.1 | 71.4 | 76.8 | 74.0 | — |
| RoBERTa-base ‡ | 81.9 | 80.1 | 84.5 | 82.2 | — |
| MAGPIE (MTL:All) † | — | — | — | — | 84.1 |
| MABPD (Full) | 83.5 | 88.0 | 81.6 | 84.7 | 83.4 |
Table 3: Supervised Protocol — Held-out Test Split (1,000 articles). Supervised baselines are fine-tuned on the 3,121-article BABE train partition; MABPD uses no BABE data for training. Separating evaluation regimes ensures that MABPD ’s zero-shot performance is compared against supervised models on identical test data. Bold marks the best value in each column, whichever system attains it. The F1 column is binary F1 and MF1 is macro F1; MAGPIE reports only macro F1, so it is listed in the MF1 column and left blank elsewhere. The regime-matched comparison is therefore macro-to-macro: MABPD 83.4 vs. MAGPIE 84.1 ( 0.7 pp). † 84.1% confirmed as macro F1 via source code inspection ( F1Score(average=‘‘macro’’) , head.py ). ‡ BERT and RoBERTa fine-tuned on the BABE train split (3,121 articles).
| System | Acc. | Prec. | Rec. | F1 | MF1 |
|---|---|---|---|---|---|
| MABPD (Single) | 71.1 | 90.1 | 54.0 | 67.6 | — |
| MABPD (No Debate) | 78.1 | 93.1 | 65.6 | 76.9 | — |
| MABPD (Full) ∗ | 84.7 | 87.2 | 85.2 | 86.2 | 84.6 |
Table 4: Zero-shot Protocol — Full Dataset (4,121 articles). All three configurations use the same architectural thresholds with no training on BABE data. This regime isolates the contribution of each pipeline component under fully zero-shot conditions. Bold marks the best value in each column, whichever configuration attains it: the No-Debate arm has the highest precision, at a large cost in recall. ∗ Full system significantly outperforms No-Debate ( , paired bootstrap, 10K resamples).
| Full dataset (4,121) | Test split (1,000) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Config | Acc | P | R | F1 | Acc | P | R | F1 | ||
| Full System | 84.7 | 87.2 | 85.2 | 86.2 | — | 83.5 | 88.0 | 81.6 | 84.7 | — |
| Debate | 78.1 | 93.1 | 65.6 | 76.9 | 9.3 | 76.0 | 93.2 | 61.5 | 74.1 | 10.6 |
| Single Agent | 71.1 | 90.1 | 54.0 | 67.6 | 18.6 | 69.7 | 91.0 | 50.8 | 65.2 | 19.5 |
| Heterogeneous | — | 83.4 | 88.0 | 81.4 | 84.6 | 0.1 | ||||
Table 5: Ablation study on both BABE splits. P = precision, R = recall; = F1 vs. Full System. Every row was measured in the same serving environment (§ 4.2 ), so all values share a single reference point. The debate module contributes 9.3 F1 (full) and 10.6 (test split); multi-agent analysis overall contributes 18.6 (full) and 19.5 (test split) vs. single agent. Results are consistent across both evaluation regimes.
Figure 2: F1 and accuracy comparison across ablation scenarios. The full system substantially outperforms both reduced configurations, with the gap driven primarily by recall improvements from the debate module.
| Model Setup | Acc | Prec | Rec | Bin F1 | Mac F1 |
|---|---|---|---|---|---|
| Single-Agent (Qwen-3.5) | 80.5% | 93.5% | 69.9% | 80.0% | 80.5% |
| MABPD (Ours) | 83.5% | 88.0% | 81.6% | 84.7% | 83.4% |
Table 6: Single-Agent LLM Baseline Comparison on the BABE test split. MABPD demonstrates substantial improvements over a strong external single-agent baseline (Qwen-3.5 122B-A10B MoE via NVIDIA NIM).
| Component | Output |
|---|---|
| Snippet (GT: Biased) | “The concept of changing one’s biological sex is, of course, nonsense, as sex is determined by unalterable chromosomes.” |
| Single-Agent Baseline | Neutral (Conf: 0.501. Misses loaded implication.) |
| BiasAgent Evidence | Extracted loaded language (“nonsense”) and ideological framing. |
| EvidenceAgent Evidence | Extracted subjective phrase (“of course, nonsense”). |
| FramingAgent Evidence | Predicted Neutral (Failed to detect structural bias). |
| SAD Debate Reasoning | The explicit loaded language and unsupported claims provided stronger evidence, overriding the neutral perspective. |
Table 7: Example 1: Epistemological Framing Bias. MABPD captures subtle epistemological bias missed by a strong single-agent baseline.
| Parameter | Value | Justification |
| BiasAgent weight ( ) | 0.50 | 2:1:1 agent weighting: the BiasAgent is the primary detector, carrying twice the weight of the supporting agents. |
| EvidenceAgent weight ( ) | 0.25 | Supporting agent under the 2:1:1 weighting. |
| FramingAgent weight ( ) | 0.25 | Supporting agent under the 2:1:1 weighting. |
| Evidence factor (biased, ) | 0.00 | Burden-of-proof axiom: a bias claim with no grounded textual evidence carries zero weight. |
| Tier-B neutral veto ceiling | 0.30 | Deliberately conservative; demands stronger evidence before flagging bias. |
| Tier-A neutral veto ceiling | 0.40 | Deliberately conservative; demands stronger evidence before flagging bias. |
Table 8: Architectural weights and decision thresholds, with the design rationale for each value.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Sev. | Definition (abbreviated) |
|---|---|---|
| Loaded language | High | Words/phrases with strong emotional connotations used to influence perception. |
| False balance | Med. | Presenting two sides as equally valid when evidence supports one. |
| Cherry-picking | High | Selective use of facts/data supporting a predetermined narrative. |
| Political framing | High | Presenting information favoring a particular political position. |
| Dog whistle | High | Language appearing neutral but carrying coded connotations. |
| Scapegoating | High | Blaming a specific group for complex problems without evidence. |
Table 9: Knowledge base bias categories injected into agent prompts.
| Component | Output |
|---|---|
| Snippet (GT: Biased) | “While ordinary families struggle to afford basic necessities, wealthy corporations continue to hoard record profits, shielded by a tax system rigged in their favor.” |
| Single-Agent Baseline | Neutral (Conf: 0.520. Treats economic claims as factual reporting.) |
| BiasAgent Evidence | Extracted loaded language (“hoard,” “rigged”) and emotional economic framing. |
| EvidenceAgent Evidence | Flagged unsupported claim (“rigged in their favor”) and loaded term (“hoard record profits”). |
| FramingAgent Evidence | Detected economic inequality framing with progressive ideological lean. |
| SAD Debate Reasoning | Loaded economic language and unattributed systemic claims warranted bias classification despite factual economic context. |
Table 10: Example 2: Left-Leaning Framing Bias.
| Component | Output |
|---|---|
| Snippet (GT: Biased) | “The flood of undocumented migrants pouring across the border has overwhelmed local communities, yet bureaucrats in Washington remain content to do nothing.” |
| Single-Agent Baseline | Neutral (Conf: 0.485. Interprets as factual border reporting.) |
| BiasAgent Evidence | Extracted loaded language (“flood,” “pouring”) and political labeling (“bureaucrats”). |
| EvidenceAgent Evidence | Flagged dehumanizing metaphor (“flood of … pouring”) and unsupported claim (“content to do nothing”). |
| FramingAgent Evidence | Detected us/them framing (communities vs. Washington) with conservative ideological lean. |
| SAD Debate Reasoning | Dehumanizing metaphor, unsupported attribution of intent, and us/them framing constituted strong multi-dimensional bias evidence. |
Table 11: Example 3: Right-Leaning Framing Bias.
Figure 4: Component F1 drop when each stage is removed. Combined loss vs. single agent: 18.6 points.
Figure 5: Multi-metric radar across ablation configurations.
| Configuration | Pred. Biased | Pred. Neutral | |
|---|---|---|---|
| Full System | True Biased | 1,958 (TP) | 341 (FN) |
| True Neutral | 288 (FP) | 1,534 (TN) | |
| Debate | True Biased | 1,507 (TP) | 792 (FN) |
| True Neutral | 112 (FP) | 1,710 (TN) | |
| Single Agent | True Biased | 1,242 (TP) | 1,057 (FN) |
| True Neutral | 136 (FP) | 1,686 (TN) |
Table 12: Confusion matrices on the full BABE benchmark (4,121 articles).
| Configuration | Pred. Biased | Pred. Neutral | |
|---|---|---|---|
| Full System | True Biased | 456 (TP) | 103 (FN) |
| True Neutral | 62 (FP) | 379 (TN) | |
| Debate | True Biased | 344 (TP) | 215 (FN) |
| True Neutral | 25 (FP) | 416 (TN) | |
| Single Agent | True Biased | 284 (TP) | 275 (FN) |
| True Neutral | 28 (FP) | 413 (TN) |
Table 13: Test-split confusion matrices (1,000 articles).
| Threshold | TP | FP | FN | F1 | MF1 |
|---|---|---|---|---|---|
| 0.40 | 465 | 78 | 94 | 84.4 | 82.6 |
| 0.50 | 456 | 62 | 103 | 84.7 | 83.4 |
| 0.60 | 454 | 61 | 105 | 84.5 | 83.3 |
Table 14: Threshold sensitivity sweep. MF1 peaks at 0.50; full 0.30–0.80 sweep in released code.
| Run | Acc. | Prec. | Rec. | F1 |
|---|---|---|---|---|
| Run 1 | 84.69 | 87.20 | 85.04 | 86.10 |
| Run 2 | 84.74 | 87.18 | 85.17 | 86.16 |
| 0.05 | 0.02 | 0.13 | 0.06 |
Table 15: Reproducibility across two independent runs (F1 = 0.06 pp).
Figure 6: Heterogeneous LLM architecture pilot on the held-out test split ( ). Top: single-agent baseline versus the heterogeneous multi-agent pipeline. Bottom: confusion matrix of the heterogeneous pipeline.