The "10th Juror": Open-Set Standpoint Screening for Bureaucratic Bias Detection

Yuchen Miao; Zijun Wang; Chang Han; Yurui Shi; Mingtai Zhang; Siyang Xu

cs.CLOct 8, 2026

The "10th Juror": Open-Set Standpoint Screening for Bureaucratic Bias Detection

Authors: Yuchen Miao, Zijun Wang, Chang Han, Yurui Shi, Mingtai Zhang, Siyang Xu

Organizations: Sydney Smart Technology College, Northeastern University, China · Taiyuan University of Technology · School of Resources, Environment and Materials, Guangxi University · School of Computer and Communication Engineering, Northeastern University at Qinhuangdao, China

Abstract

Presupposing the boundaries of bias is itself a form of bias. We study closed-loop bias governance for Dutch government documents, where a system must detect biased language, ground decisions in legal and contextual evidence, rewrite problematic sentences when intervention is warranted, and verify that the rewrite mitigates harm without distorting meaning. Existing methods face three challenges: (i) discriminative classifiers capture surface regularities but lack normative grounding; (ii) zero-shot LLMs often adopt generic viewpoints and over-flag ambiguous administrative language; and (iii) fixed taxonomies inherit the Closed-World Assumption, missing emerging local targets. We propose MARS-Gov, a standpoint-aware multi-agent framework that combines legal retrieval, open-set target screening, specialized jurors, conservative routing, and rewrite verification. When screening finds an uncovered group, MARS-Gov instantiates a dynamic "10th juror" to deliberate outside the fixed panel. On DGDB, MARS-Gov sets a new SOTA with 0.880 F1, outperforming the strongest zero-shot LLM detector by 20.2 points (29.8% relative) and the best supervised Dutch encoder by 6.8 points, while reducing unnecessary interventions to 2.5%. Leave-One-Category-Out (LOCO) evaluation recovers held-out categories with 85.1% Correct@1 and 93.8% Correct@3.

Figures & tables

Figure 1: Closed-set detection versus MARS-Gov open-set screening. Fixed taxonomies may miss local targets or force them into a known class. MARS-Gov retrieves legal evidence, detects standpoint gaps, adds a dynamic 10th juror when needed, and sends grounded reports to the Judge for an auditable verdict and minimal rewrite.

Figure 2: Overview of MARS-Gov. The framework retrieves normative evidence, gathers juror risk reports, routes only risky cases to a Judge, and verifies any minimal rewrite. Separating evidence, routing, adjudication, and restoration keeps the decision trace inspectable.

Fine-tuned
	Prec.	Rec.	F1	F1 ${}_{\text{OOD}}$	$\Delta$ Drop	OGR $\downarrow$	BMR	SF	BGS
BERTje de Vries et al. (2019)	0.842 $\pm$ .017	0.784 $\pm$ .021	0.812 $\pm$ .014	0.554 $\pm$ .032	-31.8% $\pm$ 1.62	3.1% $\pm$ 0.51	-	-	-
RobBERT Delobelle et al. (2020)	0.839 $\pm$ .019	0.784 $\pm$ .018	0.811 $\pm$ .013	0.499 $\pm$ .034	-38.5% $\pm$ 1.94	3.6% $\pm$ 0.62	-	-	-
I. Prompted LLMs (Zero-/Few-shot)
GPT-4o	0.514 $\pm$ .025	0.882 $\pm$ .021	0.649 $\pm$ .017	0.621 $\pm$ .024	-4.3% $\pm$ 1.07	16.4% $\pm$ 0.83	0.872 $\pm$ .016	0.821 $\pm$ .021	0.631 $\pm$ .021
Grok-3	0.484 $\pm$ .031	0.920 $\pm$ .017	0.634 $\pm$ .019	0.598 $\pm$ .029	-5.7% $\pm$ 1.18	19.1% $\pm$ 0.94	0.864 $\pm$ .019	0.808 $\pm$ .024	0.642 $\pm$ .022

Table 1: Main DGDB results. Darker green marks stronger performance; darker red marks larger

\Delta

Drop or OGR. F1

{}_{\text{OOD}}

is DGDB-Unseen F1;

\Delta

Drop is the relative drop from DGDB F1. Single-Call was evaluated in-domain only. MARS-Gov gives the best mean F1/BGS with low governance intrusion.

Config.	Metric	B1	B2	B3	B4	B5	B6	B7	B8	B9
		(Disability)	(Migration)	(Colonialism)	(Religion)	(Gender)	(LGBTQ+)	(Culture)	(Education/Class)	(Welfare)
w/o Scout	Rec.	0.794 $\pm$ .027	0.814 $\pm$ .024	0.778 $\pm$ .031	0.776 $\pm$ .028	0.821 $\pm$ .023	0.788 $\pm$ .026	0.807 $\pm$ .024	0.764 $\pm$ .029	0.812 $\pm$ .022
w/o Scout	F1	0.804 $\pm$ .021	0.826 $\pm$ .018	0.794 $\pm$ .025	0.789 $\pm$ .022	0.831 $\pm$ .017	0.806 $\pm$ .019	0.813 $\pm$ .018	0.782 $\pm$ .023	0.819 $\pm$ .017
w/ Scout (Ours)	Rec.	0.882 $\pm$ .018	0.901 $\pm$ .015	0.874 $\pm$ .021	0.871 $\pm$ .019	0.894 $\pm$ .014	0.881 $\pm$ .018	0.889 $\pm$ .016	0.866 $\pm$ .021	0.896 $\pm$ .015
	F1	0.873 $\pm$ .015	0.891 $\pm$ .012	0.867 $\pm$ .017	0.864 $\pm$ .015	0.886 $\pm$ .011	0.872 $\pm$ .014	0.881 $\pm$ .013	0.854 $\pm$ .017	0.884 $\pm$ .012
	Correct@1 (%)	83.4 $\pm$ 2.4	87.6 $\pm$ 1.9	82.7 $\pm$ 2.7	86.5 $\pm$ 2.1	88.9 $\pm$ 1.7	84.3 $\pm$ 2.2	81.6 $\pm$ 2.6	83.2 $\pm$ 2.5	87.4 $\pm$ 1.8

Table 2: Leave-One-Category-Out (LOCO) evaluation on Gemini 2.5 Pro. Best F1 scores are bold. Significance: all w/ Scout gains over w/o Scout remain significant after Bonferroni correction over 9 categories (two-sided paired t-test over 3 runs; max corrected

p=2.4{\times}10^{-2}

Configuration	Detection Quality			Governance	Efficiency
Configuration	Prec.	Rec.	F1	BGS	Tokens/Doc
Full MARS-Gov	0.865 $\pm$ .015	0.895 $\pm$ .013	0.880 $\pm$ .012	0.808 $\pm$ .013	2,450 $\pm$ 58
w/o Verify Loop	0.865 $\pm$ .015	0.895 $\pm$ .013	0.880 $\pm$ .012	0.646 $\pm$ .019 ( $\downarrow$ 20.0%) ∗∗	2,100 $\pm$ 49
w/o Reasoning	0.842 $\pm$ .019	0.870 $\pm$ .017	0.856 $\pm$ .016	0.765 $\pm$ .021 ( $\downarrow$ 5.3%) ∗	2,380 $\pm$ 62
w/o NEC	0.802 $\pm$ .021	0.847 $\pm$ .019	0.824 $\pm$ .018	0.710 $\pm$ .022 ( $\downarrow$ 12.1%) ∗∗	2,250 $\pm$ 54
w/o Scout (10th)	0.851 $\pm$ .017	0.840 $\pm$ .019	0.845 $\pm$ .015	0.760 $\pm$ .018 ( $\downarrow$ 5.9%) ∗	2,150 $\pm$ 44

Table 3: Ablation on DGDB with Gemini 2.5 Pro, averaged over 3 runs. Significance: marked BGS drops vs. Full MARS-Gov remain significant after Bonferroni correction (two-sided paired t-test;

{}^{\ast\ast}p<0.01

{}^{\ast}p<0.05

; max corrected

p=2.7{\times}10^{-2}

). w/o ARG is cost-only.

Figure 4: Recall vs. Over-Governance Rate (OGR) trade-off as threshold

\tau

increases.

Baselines (Gemini 2.5 Pro)
Configuration	Calls	Lat. (s)	Clear%	$/1k
Zero-shot LLM	1.0	1.8 (0.3)	—	0.85
Standard Debate	4.0	7.2 (0.9)	—	4.62
SA-RAG	1.0	3.4 (0.5)	—	2.46
MARS-Gov (with ARG)
GPT-4o	12.6 (0.6)	20.5 (2.3)	64.2 (1.7)	10.04

Table 4: Per-sentence inference cost on DGDB (1,000 sentences, 3 runs; mean with

\sigma

). Calls counts all backbone calls; Clear% is the early-stop share.

Configuration	DALC (Dutch, Social Media)			SBIC (English, Social Media)			KoBBQ (Korean, General)
Configuration	Prec.	Rec.	F1	Prec.	Rec.	F1	Prec.	Rec.	F1
Zero-shot LLMs	0.513 $\pm$ .028	0.873 $\pm$ .021	0.646 $\pm$ .018	0.486 $\pm$ .033	0.844 $\pm$ .023	0.617 $\pm$ .019	0.451 $\pm$ .036	0.803 $\pm$ .027	0.578 $\pm$ .024
Standard Debate	0.607 $\pm$ .025	0.851 $\pm$ .020	0.709 $\pm$ .019	0.583 $\pm$ .027	0.819 $\pm$ .021	0.681 $\pm$ .021	0.553 $\pm$ .029	0.811 $\pm$ .024	0.658 $\pm$ .023
SA-RAG	0.653 $\pm$ .021	0.862 $\pm$ .018	0.743 $\pm$ .016	0.626 $\pm$ .023	0.836 $\pm$ .019	0.716 $\pm$ .018	0.593 $\pm$ .027	0.821 $\pm$ .021	0.689 $\pm$ .019
Ours: MARS-Gov	0.823 $\pm$ .015	0.886 $\pm$ .013	0.853 $\pm$ .012	0.803 $\pm$ .018	0.847 $\pm$ .016	0.824 $\pm$ .015	0.776 $\pm$ .021	0.829 $\pm$ .018	0.802 $\pm$ .017

Table 5: Generalization on DALC, SBIC, and KoBBQ using Gemini 2.5 Pro. Results average 3 runs; best F1 scores are bold. Significance: MARS-Gov beats SA-RAG after Bonferroni correction over 3 datasets (two-sided paired t-test; max corrected

p=1.1{\times}10^{-2}

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Statistics	DGDB	DGDB-Unseen
Language	Dutch	Dutch
Domain	Gov. docs	Gov. docs
Unit	Sentence	Sentence
Corpus size	3,747	DGDB-derived
Evaluation size	Test split	Held-out subset
Construction	Original split	Bias-term substitution

Appendix

Table 6: Statistical overview of the DGDB-derived datasets.

Statistics	DALC	SBIC	KoBBQ
Language	Dutch	English	Korean
Domain	Twitter	Social	QA
Unit	Tweet	Post / tuple	MCQ
Corpus size	8,156	44,671 / 147,139	76,048
Evaluation size	1,000	1,000	1,000
Evaluation balance	500 / 500	500 / 500	500 / 500

Appendix

Table 7: Statistical overview of the external evaluation datasets.

Model Family	Model Name	Standard Zero-shot			Ours: MARS-Gov			Governance
Model Family	Model Name	Prec	Rec	F1	Prec	Rec	F1	BGS
Frontier SOTA	Gemini 2.5 Pro	0.524 $\pm$ .024	0.907 $\pm$ .018	0.664 $\pm$ .016	0.865 $\pm$ .015	0.895 $\pm$ .013	0.880 $\pm$ .012	0.808 $\pm$ .013
	DeepSeek-R1	0.549 $\pm$ .023	0.887 $\pm$ .019	0.678 $\pm$ .015	0.872 $\pm$ .016	0.884 $\pm$ .014	0.878 $\pm$ .013	0.806 $\pm$ .015
	Claude 3.5 Sonnet	0.562 $\pm$ .022	0.872 $\pm$ .020	0.683 $\pm$ .016	0.860 $\pm$ .017	0.882 $\pm$ .015	0.871 $\pm$ .013	0.787 $\pm$ .016
	Grok-3	0.484 $\pm$ .031	0.920 $\pm$ .017	0.634 $\pm$ .019	0.836 $\pm$ .019	0.910 $\pm$ .013	0.871 $\pm$ .014	0.779 $\pm$ .018
	GPT-4o	0.514 $\pm$ .025	0.882 $\pm$ .021	0.649 $\pm$ .017	0.844 $\pm$ .017	0.863 $\pm$ .015	0.853 $\pm$ .013	0.757 $\pm$ .014

Appendix

Table 8: Extended Leaderboard on DGDB (In-Domain). Models are ranked by their MARS-Gov F1 score. The red dashed line marks the performance threshold of the fine-tuned BERTje baseline (

F1=0.812

). Observation: While MARS-Gov improves performance across model families (avg.

\Delta

+0.20

, larger for frontier backbones than for smaller ones), a minimum reasoning capability threshold (approx. GPT-4 Turbo level) is required to surpass supervised baselines.

Figure 5: Comparison of F1 scores between Native Dutch and Translated English prompts. Native prompting consistently outperforms English across all backbones, avoiding the semantic loss associated with translating specific administrative terminology ( $p<0.05$ , paired Student’s t-test over 3 independent runs).

Dimension	DGDB NEC retrieval
Primary Objective	Detecting implicit bias in official policy texts.
Core Domain	Legal and political administrative text ( high-stakes ).
Inventory	Dutch constitutional and equal-treatment law; antidiscrimination policy; accessibility and election communication guidance; online-discrimination, institutional-racism, emancipation, and inclusive-language resources; DGDB reasoning entries.
Selection Rationale	Grounds the review in legal norms while covering administrative, accessibility, and inclusive-language contexts that commonly shape implicit bias.

Appendix

Table 9: NEC retrieval construction for DGDB. MARS-Gov prioritizes normative sources and official writing guidance for administrative bias review.

Figure 6: ARG Hyperparameter Grid Search. (a) F1 Score and (b) Over-Governance Rate (OGR) across varying $\tau_{\mathrm{high}}$ and juror-vote thresholds. The selected configuration ( $\tau_{\mathrm{high}}=0.6,N_{\mathrm{soft\_bias}}=2$ ) is highlighted.

Explore similar work

UnBias-Plus: Detect, Explain, and Rewrite Bias
Jun 22, 2026Ahmed Y. Radwan, Ahmed ElKady, Sindhuja Chaduvula +3Explainable Artificial Intelligence Social Bias in Language Models
Vector Institute for Artificial Intelligence, Toronto, Canada · Independent Researcher
NewsLens: A Multi-Agent Framework for Adversarial News Bias Navigation
May 17, 2026Joy BoseMulti-Agent Systems
Independent Researcher, Bengaluru, India
FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge
Feb 6, 2026Bo Yang, Lanfei Feng, Yunkui Chen +3LLM Evaluation LLM-as-a-Judge
College of Computer Science, Zhejiang University, Hangzhou, China · School of Software Technology, Zhejiang University, Hangzhou, China

Jun 22, 2026, cs.CLJ/K move · Enter open · S save

UnBias-Plus: Detect, Explain, and Rewrite Bias

Ahmed Y. Radwan, Ahmed ElKady, Sindhuja Chaduvula, Mohamed Hafez+2

Vector Institute for Artificial Intelligence, Toronto, Canada · Independent Researcher

Explainable Artificial Intelligence Social Bias in Language Models Language Model Bias Evaluation

Open paper AskPDF arXiv

Bias in natural language remains a persistent challenge in both human-written and AI-generated content, affecting domains such as journalism, education, and AI research. Most existing detection methods identify only the presence of bias, with limited support for granular detection, interpretable explanations, neutral rewriting, and openly available trained models. We present UnBias-Plus, an open-source toolkit unifying (1) segment-level multi-class bias classification, (2) biased span localization, (3) neutral text rewriting, and (4) reasoning for each decision. Available via Python, CLI, REST API, and web interfaces, UnBias-Plus supports accessible bias analysis. The toolkit, source code, models, datasets, and documentation are publicly available.

Abstract

Explore similar work

UnBias-Plus: Detect, Explain, and Rewrite Bias

NewsLens: A Multi-Agent Framework for Adversarial News Bias Navigation

FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge