The "10th Juror": Open-Set Standpoint Screening for Bureaucratic Bias Detection

Yuchen Miao; Zijun Wang; Chang Han; Yurui Shi; Mingtai Zhang; Siyang Xu

cs.CLOct 8, 2026

The "10th Juror": Open-Set Standpoint Screening for Bureaucratic Bias Detection

Authors: Yuchen Miao, Zijun Wang, Chang Han, Yurui Shi, Mingtai Zhang, Siyang Xu

Organizations: Sydney Smart Technology College, Northeastern University, China · Taiyuan University of Technology · School of Resources, Environment and Materials, Guangxi University · School of Computer and Communication Engineering, Northeastern University at Qinhuangdao, China

Abstract

Presupposing the boundaries of bias is itself a form of bias. We study closed-loop bias governance for Dutch government documents, where a system must detect biased language, ground decisions in legal and contextual evidence, rewrite problematic sentences when intervention is warranted, and verify that the rewrite mitigates harm without distorting meaning. Existing methods face three challenges: (i) discriminative classifiers capture surface regularities but lack normative grounding; (ii) zero-shot LLMs often adopt generic viewpoints and over-flag ambiguous administrative language; and (iii) fixed taxonomies inherit the Closed-World Assumption, missing emerging local targets. We propose MARS-Gov, a standpoint-aware multi-agent framework that combines legal retrieval, open-set target screening, specialized jurors, conservative routing, and rewrite verification. When screening finds an uncovered group, MARS-Gov instantiates a dynamic "10th juror" to deliberate outside the fixed panel. On DGDB, MARS-Gov sets a new SOTA with 0.880 F1, outperforming the strongest zero-shot LLM detector by 20.2 points (29.8% relative) and the best supervised Dutch encoder by 6.8 points, while reducing unnecessary interventions to 2.5%. Leave-One-Category-Out (LOCO) evaluation recovers held-out categories with 85.1% Correct@1 and 93.8% Correct@3.

Figures & tables

Figure 1: Closed-set detection versus MARS-Gov open-set screening. Fixed taxonomies may miss local targets or force them into a known class. MARS-Gov retrieves legal evidence, detects standpoint gaps, adds a dynamic 10th juror when needed, and sends grounded reports to the Judge for an auditable verdict and minimal rewrite.

Figure 2: Overview of MARS-Gov. The framework retrieves normative evidence, gathers juror risk reports, routes only risky cases to a Judge, and verifies any minimal rewrite. Separating evidence, routing, adjudication, and restoration keeps the decision trace inspectable.

Fine-tuned
	Prec.	Rec.	F1	F1 ${}_{\text{OOD}}$	$\Delta$ Drop	OGR $\downarrow$	BMR	SF	BGS
BERTje de Vries et al. (2019)	0.842 $\pm$ .017	0.784 $\pm$ .021	0.812 $\pm$ .014	0.554 $\pm$ .032	-31.8% $\pm$ 1.62	3.1% $\pm$ 0.51	-	-	-
RobBERT Delobelle et al. (2020)	0.839 $\pm$ .019	0.784 $\pm$ .018	0.811 $\pm$ .013	0.499 $\pm$ .034	-38.5% $\pm$ 1.94	3.6% $\pm$ 0.62	-	-	-
I. Prompted LLMs (Zero-/Few-shot)
GPT-4o	0.514 $\pm$ .025	0.882 $\pm$ .021	0.649 $\pm$ .017	0.621 $\pm$ .024	-4.3% $\pm$ 1.07	16.4% $\pm$ 0.83	0.872 $\pm$ .016	0.821 $\pm$ .021	0.631 $\pm$ .021
Grok-3	0.484 $\pm$ .031	0.920 $\pm$ .017	0.634 $\pm$ .019	0.598 $\pm$ .029	-5.7% $\pm$ 1.18	19.1% $\pm$ 0.94	0.864 $\pm$ .019	0.808 $\pm$ .024	0.642 $\pm$ .022

Table 1: Main DGDB results. Darker green marks stronger performance; darker red marks larger

\Delta

Drop or OGR. F1

{}_{\text{OOD}}

is DGDB-Unseen F1;

\Delta

Drop is the relative drop from DGDB F1. Single-Call was evaluated in-domain only. MARS-Gov gives the best mean F1/BGS with low governance intrusion.

Config.	Metric	B1	B2	B3	B4	B5	B6	B7	B8	B9
		(Disability)	(Migration)	(Colonialism)	(Religion)	(Gender)	(LGBTQ+)	(Culture)	(Education/Class)	(Welfare)
w/o Scout	Rec.	0.794 $\pm$ .027	0.814 $\pm$ .024	0.778 $\pm$ .031	0.776 $\pm$ .028	0.821 $\pm$ .023	0.788 $\pm$ .026	0.807 $\pm$ .024	0.764 $\pm$ .029	0.812 $\pm$ .022
w/o Scout	F1	0.804 $\pm$ .021	0.826 $\pm$ .018	0.794 $\pm$ .025	0.789 $\pm$ .022	0.831 $\pm$ .017	0.806 $\pm$ .019	0.813 $\pm$ .018	0.782 $\pm$ .023	0.819 $\pm$ .017
w/ Scout (Ours)	Rec.	0.882 $\pm$ .018	0.901 $\pm$ .015	0.874 $\pm$ .021	0.871 $\pm$ .019	0.894 $\pm$ .014	0.881 $\pm$ .018	0.889 $\pm$ .016	0.866 $\pm$ .021	0.896 $\pm$ .015
	F1	0.873 $\pm$ .015	0.891 $\pm$ .012	0.867 $\pm$ .017	0.864 $\pm$ .015	0.886 $\pm$ .011	0.872 $\pm$ .014	0.881 $\pm$ .013	0.854 $\pm$ .017	0.884 $\pm$ .012
	Correct@1 (%)	83.4 $\pm$ 2.4	87.6 $\pm$ 1.9	82.7 $\pm$ 2.7	86.5 $\pm$ 2.1	88.9 $\pm$ 1.7	84.3 $\pm$ 2.2	81.6 $\pm$ 2.6	83.2 $\pm$ 2.5	87.4 $\pm$ 1.8

Table 2: Leave-One-Category-Out (LOCO) evaluation on Gemini 2.5 Pro. Best F1 scores are bold. Significance: all w/ Scout gains over w/o Scout remain significant after Bonferroni correction over 9 categories (two-sided paired t-test over 3 runs; max corrected

p=2.4{\times}10^{-2}

Configuration	Detection Quality			Governance	Efficiency
Configuration	Prec.	Rec.	F1	BGS	Tokens/Doc
Full MARS-Gov	0.865 $\pm$ .015	0.895 $\pm$ .013	0.880 $\pm$ .012	0.808 $\pm$ .013	2,450 $\pm$ 58
w/o Verify Loop	0.865 $\pm$ .015	0.895 $\pm$ .013	0.880 $\pm$ .012	0.646 $\pm$ .019 ( $\downarrow$ 20.0%) ∗∗	2,100 $\pm$ 49
w/o Reasoning	0.842 $\pm$ .019	0.870 $\pm$ .017	0.856 $\pm$ .016	0.765 $\pm$ .021 ( $\downarrow$ 5.3%) ∗	2,380 $\pm$ 62
w/o NEC	0.802 $\pm$ .021	0.847 $\pm$ .019	0.824 $\pm$ .018	0.710 $\pm$ .022 ( $\downarrow$ 12.1%) ∗∗	2,250 $\pm$ 54
w/o Scout (10th)	0.851 $\pm$ .017	0.840 $\pm$ .019	0.845 $\pm$ .015	0.760 $\pm$ .018 ( $\downarrow$ 5.9%) ∗	2,150 $\pm$ 44

Table 3: Ablation on DGDB with Gemini 2.5 Pro, averaged over 3 runs. Significance: marked BGS drops vs. Full MARS-Gov remain significant after Bonferroni correction (two-sided paired t-test;

{}^{\ast\ast}p<0.01

{}^{\ast}p<0.05

; max corrected

p=2.7{\times}10^{-2}

). w/o ARG is cost-only.

Figure 4: Recall vs. Over-Governance Rate (OGR) trade-off as threshold

\tau

increases.

Baselines (Gemini 2.5 Pro)
Configuration	Calls	Lat. (s)	Clear%	$/1k
Zero-shot LLM	1.0	1.8 (0.3)	—	0.85
Standard Debate	4.0	7.2 (0.9)	—	4.62
SA-RAG	1.0	3.4 (0.5)	—	2.46
MARS-Gov (with ARG)
GPT-4o	12.6 (0.6)	20.5 (2.3)	64.2 (1.7)	10.04

Table 4: Per-sentence inference cost on DGDB (1,000 sentences, 3 runs; mean with

\sigma

). Calls counts all backbone calls; Clear% is the early-stop share.

Configuration	DALC (Dutch, Social Media)			SBIC (English, Social Media)			KoBBQ (Korean, General)
Configuration	Prec.	Rec.	F1	Prec.	Rec.	F1	Prec.	Rec.	F1
Zero-shot LLMs	0.513 $\pm$ .028	0.873 $\pm$ .021	0.646 $\pm$ .018	0.486 $\pm$ .033	0.844 $\pm$ .023	0.617 $\pm$ .019	0.451 $\pm$ .036	0.803 $\pm$ .027	0.578 $\pm$ .024
Standard Debate	0.607 $\pm$ .025	0.851 $\pm$ .020	0.709 $\pm$ .019	0.583 $\pm$ .027	0.819 $\pm$ .021	0.681 $\pm$ .021	0.553 $\pm$ .029	0.811 $\pm$ .024	0.658 $\pm$ .023
SA-RAG	0.653 $\pm$ .021	0.862 $\pm$ .018	0.743 $\pm$ .016	0.626 $\pm$ .023	0.836 $\pm$ .019	0.716 $\pm$ .018	0.593 $\pm$ .027	0.821 $\pm$ .021	0.689 $\pm$ .019
Ours: MARS-Gov	0.823 $\pm$ .015	0.886 $\pm$ .013	0.853 $\pm$ .012	0.803 $\pm$ .018	0.847 $\pm$ .016	0.824 $\pm$ .015	0.776 $\pm$ .021	0.829 $\pm$ .018	0.802 $\pm$ .017

Table 5: Generalization on DALC, SBIC, and KoBBQ using Gemini 2.5 Pro. Results average 3 runs; best F1 scores are bold. Significance: MARS-Gov beats SA-RAG after Bonferroni correction over 3 datasets (two-sided paired t-test; max corrected

p=1.1{\times}10^{-2}

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Statistics	DGDB	DGDB-Unseen
Language	Dutch	Dutch
Domain	Gov. docs	Gov. docs
Unit	Sentence	Sentence
Corpus size	3,747	DGDB-derived
Evaluation size	Test split	Held-out subset
Construction	Original split	Bias-term substitution

Appendix

Table 6: Statistical overview of the DGDB-derived datasets.

Statistics	DALC	SBIC	KoBBQ
Language	Dutch	English	Korean
Domain	Twitter	Social	QA
Unit	Tweet	Post / tuple	MCQ
Corpus size	8,156	44,671 / 147,139	76,048
Evaluation size	1,000	1,000	1,000
Evaluation balance	500 / 500	500 / 500	500 / 500

Appendix

Table 7: Statistical overview of the external evaluation datasets.

Model Family	Model Name	Standard Zero-shot			Ours: MARS-Gov			Governance
Model Family	Model Name	Prec	Rec	F1	Prec	Rec	F1	BGS
Frontier SOTA	Gemini 2.5 Pro	0.524 $\pm$ .024	0.907 $\pm$ .018	0.664 $\pm$ .016	0.865 $\pm$ .015	0.895 $\pm$ .013	0.880 $\pm$ .012	0.808 $\pm$ .013
	DeepSeek-R1	0.549 $\pm$ .023	0.887 $\pm$ .019	0.678 $\pm$ .015	0.872 $\pm$ .016	0.884 $\pm$ .014	0.878 $\pm$ .013	0.806 $\pm$ .015
	Claude 3.5 Sonnet	0.562 $\pm$ .022	0.872 $\pm$ .020	0.683 $\pm$ .016	0.860 $\pm$ .017	0.882 $\pm$ .015	0.871 $\pm$ .013	0.787 $\pm$ .016
	Grok-3	0.484 $\pm$ .031	0.920 $\pm$ .017	0.634 $\pm$ .019	0.836 $\pm$ .019	0.910 $\pm$ .013	0.871 $\pm$ .014	0.779 $\pm$ .018
	GPT-4o	0.514 $\pm$ .025	0.882 $\pm$ .021	0.649 $\pm$ .017	0.844 $\pm$ .017	0.863 $\pm$ .015	0.853 $\pm$ .013	0.757 $\pm$ .014

Appendix

Table 8: Extended Leaderboard on DGDB (In-Domain). Models are ranked by their MARS-Gov F1 score. The red dashed line marks the performance threshold of the fine-tuned BERTje baseline (

F1=0.812

). Observation: While MARS-Gov improves performance across model families (avg.

\Delta

+0.20

, larger for frontier backbones than for smaller ones), a minimum reasoning capability threshold (approx. GPT-4 Turbo level) is required to surpass supervised baselines.

Figure 5: Comparison of F1 scores between Native Dutch and Translated English prompts. Native prompting consistently outperforms English across all backbones, avoiding the semantic loss associated with translating specific administrative terminology ( $p<0.05$ , paired Student’s t-test over 3 independent runs).

Dimension	DGDB NEC retrieval
Primary Objective	Detecting implicit bias in official policy texts.
Core Domain	Legal and political administrative text ( high-stakes ).
Inventory	Dutch constitutional and equal-treatment law; antidiscrimination policy; accessibility and election communication guidance; online-discrimination, institutional-racism, emancipation, and inclusive-language resources; DGDB reasoning entries.
Selection Rationale	Grounds the review in legal norms while covering administrative, accessibility, and inclusive-language contexts that commonly shape implicit bias.

Appendix

Table 9: NEC retrieval construction for DGDB. MARS-Gov prioritizes normative sources and official writing guidance for administrative bias review.

Figure 6: ARG Hyperparameter Grid Search. (a) F1 Score and (b) Over-Governance Rate (OGR) across varying $\tau_{\mathrm{high}}$ and juror-vote thresholds. The selected configuration ( $\tau_{\mathrm{high}}=0.6,N_{\mathrm{soft\_bias}}=2$ ) is highlighted.

Explore similar work

Jun 22, 2026cs.CL

UnBias-Plus: Detect, Explain, and Rewrite Bias

Bias in natural language remains a persistent challenge in both human-written and AI-generated content, affecting domains such as journalism, education, and AI research. Most existing detection methods identify only the presence of bias, with limited support for granular detection, interpretable explanations, neutral rewriting, and openly available trained models. We present UnBias-Plus, an open-source toolkit unifying (1) segment-level multi-class bias classification, (2) biased span localization, (3) neutral text rewriting, and (4) reasoning for each decision. Available via Python, CLI, REST API, and web interfaces, UnBias-Plus supports accessible bias analysis. The toolkit, source code, models, datasets, and documentation are publicly available.

Ahmed Y. Radwan, Ahmed ElKady, Sindhuja Chaduvula +3

Vector Institute for Artificial Intelligence, Toronto, Canada · Independent Researcher

Explainable Artificial Intelligence Social Bias in Language Models Language Model Bias Evaluation

Ask PDF

May 17, 2026cs.CL

NewsLens: A Multi-Agent Framework for Adversarial News Bias Navigation

Media bias detection has predominantly been framed as a classification task: assign a political label to an article or outlet. We argue this framing is too shallow: it identifies that bias exists but not where, how, or crucially, what is structurally omitted. We present NewsLens, a five-agent adversarial pipeline for structured news bias navigation. A Fact Verifier, Progressive Framing Analyst, Conservative Framing Analyst, Propaganda Detector, and Neutral Summarizer collaborate to deconstruct articles into interpretable framing maps, exposing ideological omissions, rhetorical manipulation, and framing boundaries. The system is evaluated on 15 articles across four geopolitical event clusters (India-Pakistan Kashmir, Gaza, Climate Policy, Ukraine) using Qwen2.5-3B-Instruct (4-bit quantised, Google Colab T4), with cross-model validation using Mistral 7B on the Kashmir cluster. Center outlets show the highest mean Perspective Divergence Score (PDS: Qwen 0.907, Mistral 0.729 on Kashmir subset); conservative-framing outlets show the highest mean Manipulation Index (MI: 0.600 across both models). Cross-model comparison shows high consistency for high-propaganda content (Republic World delta-PDS=0.125, MI=0.8 both models) and greater variance for nuanced reporting. Mann-Whitney U tests find no statistically significant between-group differences at n=15, reported honestly as a sample-size limitation confirmed by post-hoc power analysis. A partial ablation removing the Propaganda Detector shows degraded omission precision in the Neutral Summarizer output. The architecture extends prior lexical-geometric bias work to agentic LLM reasoning, and is fully reproducible using open-weight models without API keys.

Joy Bose

Independent Researcher, Bengaluru, India

Multi-Agent Systems

Ask PDF

Feb 6, 2026cs.CL

FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge

Existing LLM-as-a-Judge systems suffer from three fundamental limitations: limited adaptivity to task- and domain-specific evaluation criteria, systematic biases driven by non-semantic cues such as position, length, format, and model provenance, and evaluation inconsistency that leads to contradictory judgments across different evaluation modes (e.g., pointwise versus pairwise). To address these issues, we propose FairJudge, an adaptive, debiased, and consistent LLM-as-a-Judge. Unlike prior approaches that treat the judge as a static evaluator, FairJudge models judging behavior itself as a learnable and regularized policy. From a data-centric perspective, we construct a high-information-density judging dataset that explicitly injects supervision signals aligned with evaluation behavior. Building on this dataset, we adopt a curriculum-style SFT-DPO-GRPO training paradigm that progressively aligns rubric adherence, bias mitigation, and cross-mode consistency, while avoiding catastrophic forgetting. Experimental results on multiple internal and public benchmarks show that FairJudge consistently improves agreement and F1, reduces non-semantic biases, and outperforms substantially larger instruction-tuned LLMs. All resources will be publicly released after acceptance to facilitate future research.

Bo Yang, Lanfei Feng, Yunkui Chen +3

College of Computer Science, Zhejiang University, Hangzhou, China · School of Software Technology, Zhejiang University, Hangzhou, China

LLM Evaluation LLM-as-a-Judge Language Model Bias Evaluation Bias in LLM-as-a-Judge

Ask PDF