Invisibilization as a Source of Probability Bias in Language Models

Christian Arnold; Ernesto Calvo; Guadalupe Gonzalez; Sebastián Vallejo Vera; Tiago Ventura

cs.CLOct 22, 2026

Invisibilization as a Source of Probability Bias in Language Models

Authors: Christian Arnold, Ernesto Calvo, Guadalupe Gonzalez, Sebastián Vallejo Vera, Tiago Ventura

Organizations: University of Birmingham · University of Maryland, College Park · University of Western Ontario · Georgetown University

Abstract

Language models are aligned to avoid using protected attributes such as race, ethnicity, and gender to inform their responses. Therefore, they increasingly refuse or hedge when such attributes appear in a prompt. However, if a system must decide how much to learn from what people tell it, it is costly to refuse valuable information because of its source. In this article, we measure the cost of declining information in a two-urn game with an exact Bayesian answer. We present leading LLM models from two developers (OpenAI and Anthropic) with a prompt that requires them to decide how far to trust an unverifiable report from a participant whose description we randomize across ten configurations (6,600 sessions). We show that leading models often decline to assess the reporter, treating information as worthless at rates set by the description rather than by the evidence. At its widest, models decline a Black-described reporter 46% of the time and a white-described reporter 9%; yet, when they do assess the information, the Black-described reporter receives slightly more trust. The bias is invisible to audits that compare average trust or look only at assessed cases. It grows with reasoning effort, follows words rather than categories (models invoke race for "Black" but not for its Spanish translation negro, and never for "Indigenous"), and worsens under warning instructions a practitioner would try first. We also present a scoring option that reduces the refusal rate bias. When a model's trust is scored against revealed truth over repeated rounds, it credits identical records equally. In four of five configurations from Anthropic and OpenAI, the racial gap in trust falls below 0.02 by the last rounds.

Figures & tables

Figure 1: What a report is worth. The log-likelihood ratio of a report,

\ell(\hat{q})

in Eq. 2 , against the reliability assigned to the reporter, for three urn compositions

p

. Declining (

\hat{q}=\frac{1}{2}

, dashed line) deletes the report; values in the shaded region discount it.

Configuration	judged	declined	full trust (1.0)	refused	unparsed
luna, low	0.63	0.31	0.06	—	—
luna, high	0.63	0.30	0.07	—	—
terra, low	0.02	0.39	0.59	—	—
terra, high	0.01	0.33	0.66	—	—
Haiku 4.5	0.27	0.73	0.01	—	—
Sonnet 5, low	0.92	0.08	0.01	—	—

Table 1: Decision to judge or to decline by type of configuration. Share of 660 sessions.

Figure 2: Rate of decline by trait and configuration. Fable 5.1 also includes a separate refuse rate at the API.

Figure 3: Estimated selection effect $\hat{\gamma}$ and evaluation effect $\hat{\delta}$ of a Black versus white description, factorial cells, with 95% intervals. For Fable 5.1 we show the difference in refusal rates.

	$\pi(B)$	$\pi(W)$	$d(B)$	$d(W)$	$\Delta(B)$	$\Delta(W)$	$\hat{\iota}$	naive gap
luna, low	0.34	0.13	0.84	0.81	0.116	0.042	0.075	$-0.044$
luna, high	0.46	0.09	0.85	0.81	0.162	0.027	0.135	$-0.094$
terra, low	0.35	0.47	0.99	1.00	0.174	0.231	$-0.057$	$+0.055$
terra, high	0.30	0.33	1.00	1.00	0.149	0.165	$-0.017$	$+0.017$
Haiku 4.5	0.76	0.73	0.93	0.93	0.330	0.313	0.017	$-0.014$
Sonnet 5, low	0.15	0.13	0.79	0.77	0.044	0.034	0.011	$+0.013$

Table 2: Components of Eq. 5 by configuration, Black- (

B

) and white-described (

W

) sources pooled over five cells each (

n=150

Figure 4: The institution. (A) Round-1 decline rate for Black- and white-described elementary-school pupils at Luna High, unscored and scored with revealed truth; 60 sessions per bar. (B) Mean $\hat{q}$ by round under the three sets of rules, luna low. (C) Learning curves for Black- and white-described pupils under scoring with revealed truth, on identical records.

Demographic Bias in Language Models

Demographic bias in language models studies how identity cues shape model outputs, decisions, and representations, and how to detect and reduce resulting disparities.

See all 135 papers

OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgment
Jul 29, 2026Seonglae Cho, Adriano KoshiyamaLLM Alignment Language Model Bias Evaluation
1Holistic AI · University College London
Language Bias under Conflicting Information in Multilingual LLMs
Apr 8, 2026Robert Östling, Murathan KurfalıLinguistic Bias in Language Models Language Model Bias Evaluation
Stockholm University Sweden · RISE Research Institutes of Sweden Sweden
TrustMI: Causally controlling how assistants trust their users
Oct 5, 2026Théo Lasnier, Romain Froger, Maxence Lasbordes +1LLM Safety Trust in AI
Inria Paris · Sorbonne Université · Meta SuperIntelligence Labs +1

Jul 29, 2026J/K move · Enter open · S save

OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgment

Seonglae Cho, Adriano Koshiyama

1Holistic AI · University College London

LLM Alignment Language Model Bias Evaluation Cognitive Bias in Language Models

Open paper AskPDF arXiv

Large language models are increasingly used as decision aids whose probability judgments shape downstream choices. Whether those judgments carry a systematic directional tilt has been hard to detect: calibration metrics aggregate unsigned errors, and naturalistic uncertainty offers no ground-truth probability. When an LLM rates a startup's success at 70% but its failure at 15%, the missing 15 points expose a distortion no aggregate score flags. We introduce OptimismBench, which detects directional bias with inverted pairs: each scenario elicits both P(success) and P(failure), and asymmetry between the two framings yields a signed bias score without ground truth. Across 16 models from 8 providers, fourteen are optimistic; pessimism appears only in Anthropic's frontier tier. Eleven matched base-versus-chat pairs across four families show post-training sets the sign of the bias, with opposite shifts in different families. The pattern survives prompt, temperature, perspective, and self-debiasing ablations. A seventeen-model six-language comparison further shows model identity dominates language, with inter-model variance at 4.7x inter-language variance. We release 3,870 items across 10 languages for per-model directional-bias auditing. When alignment makes a model more helpful, it also tilts its probabilities; downstream pipelines inherit the tilt by default.

Abstract

Demographic Bias in Language Models

Explore similar work

OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgment

Language Bias under Conflicting Information in Multilingual LLMs

TrustMI: Causally controlling how assistants trust their users