BiasTrace: Linking Reasoning Behaviours to Biased Outputs in LLMs

Varsha Ramineni; Hossein A. Rahmani; Jerome Ramos; Karin Sevegnani; Emine Yilmaz

cs.AIAug 14, 2026

BiasTrace: Linking Reasoning Behaviours to Biased Outputs in LLMs

Authors: Varsha Ramineni, Hossein A. Rahmani, Jerome Ramos, Karin Sevegnani, Emine Yilmaz

Organizations: Centre for Artificial Intelligence, University College London · NVIDIA

Abstract

LLMs exhibit social biases that can produce inaccurate and discriminatory inferences, posing risks in high-stakes applications. While prior work has made progress in measuring and mitigating bias, it largely focuses on final outputs of models, with limited understanding of the mechanisms that produce biased outcomes. Recent advances in LLM reasoning offers a new lens for investigating bias, yet the link between reasoning and bias remains poorly understood. Existing approaches focus primarily on final answer correctness or explicitly biased language, overlooking different behaviours in reasoning that can drive biased outcomes. We introduce BiasTrace, an annotation scheme for labelling reasoning behaviours in model-generated traces and linking them to biased outcomes. BiasTrace captures bias-specific behaviours (e.g., unsupported demographic assumptions) as well as general reasoning patterns that may implicitly contribute to bias (e.g. overthinking). We apply BiasTrace to reasoning traces in bias-sensitive contexts, scaled using validated LLM-as-a-judge methods, producing a large annotated dataset. Our analysis shows that biased outputs often stem from subtle reasoning behaviours rather than explicitly biased language, and that reasoning-level annotations improve bias detection. We further show that BiasTrace behaviours can be exploited for inference-time mitigation. These findings underscore the importance of examining a broader range of reasoning patterns to better understand bias in LLMs.

Figures & tables

Figure 1: Our framework applies the BiasTrace annotation scheme to link reasoning behaviours to biased outcomes i.e. biased final conclusion from the model. We present a redacted trace from Qwen3-8B on the BBQ dataset where the model should have selected answer option ‘unknown’ but instead chose the stereotypical answer. This example demonstrates how biased outputs can arise from different reasoning behaviours, such as overthinking, rather than only from the use of stereotypical language.

Label	Description	Example Behaviour
Bias-Specific
Group Assumption	Reasoning attributes behaviours, traits, or likelihoods to a social or demographic group (e.g., gender, race, age, class, body type) without correct contextual justification.	“ So maybe the obese guy is too lazy to play, hence he’s waiting outside but not participating. ”
Bias Acknowledgment	Reasoning recognises the possibility of bias, stereotyping, or unfair assumptions within its own thought process.	“ I should be careful not to assume based on stereotypes. ”
Meta-Cognitive
Meta Reflection	Reasoning reflects on or questions the intent, scope, or framing of the question.	“ Wait, but the question is from a multiple-choice exam, so the answer is likely based on historical context. ”
Out-of-Context Knowledge

Table 1: BiasTrace reasoning behaviour annotation scheme with description and examples. Labels are on a binary scale, indicating whether the reasoning exhibits the behaviour (1) or not (0). Full description of labels can be found in Appendix D.1 .

Figure 2: Odds ratio from Logistic Regression of BiasTrace reasoning behaviours and of guided prompt and ambiguous context. Significance is indicated with stars (

*p<0.05,**p<0.01,***p<0.001

). Bias Acknowledgement was removed due to unstable convergence, details in Appendix F .

Figure 3: Pearson correlations between each reasoning evaluation method and biased outcomes on the BBQ dataset, comparing the BiasTrace prompt against baseline methods. For Baseline FRM, we report the absolute value. Baseline0–5 bin: binary version (>0 labelled as 1).

Score	$r$	5%	10%	20%
BiasTrace Prompt	0.38	0.36	0.52	0.62
Baseline 0-5	0.32	0.55	0.52	0.49
Baseline 0/1	0.30	0.27	0.35	0.57
Baseline FRM	0.01	0.09	0.09	0.17

Table 2: Bias score performance: correlation of reasoning bias score with per-sample fairness contribution (Pearson

r

) and Top-K% (5, 10, 20%) overlap with fairness-critical samples.

Model	Method	Acc.	Bias $\downarrow$
Qwen3-1.7B	Single	85.45	3.64
	Maj -All	90.00	2.55
	Maj - BiasTrace	90.73	1.73
Qwen3-4B	Single	89.45	4.27
	Maj -All	90.00	4.00
	Maj - BiasTrace	91.18	2.55

Table 3: Bias mitigation results on BBQ dataset. Single selects the first sampled trace, Maj -All applies majority voting over all samples, and Maj - BiasTrace applies majority voting over BiasTrace -filtered unbiased samples. Accuracy and bias rates in percentages (%).

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Qwen3-8B	Qwen3-8B	Qwen3-14B	Qwen3-14B	GPT-OSS-120B
Prompt	Simple	Full	Simple	Full	Full, Simple
Hardware	2 GPUs	2 GPUs	2 GPUs	2 GPUs	4 GPUs
Batch size	16	32	32	16	32
Max generation length	2048	2048	2048	2048	2048
Temperature	0.6	0.6	0.6	0.6	1.0
Top-p	0.95	0.95	0.95	0.95	1.0

Appendix

Table B.1: Model inference configurations, including sampling parameters. Only Qwen3-8B, Qwen-14 were used for developing BiasTrace , and for predictive analysis. GPT-OSS-120B was used in subsequent analysis of improved reasoning evaluation.

Reasoning Behaviour	Initial $\kappa$	Final $\kappa$
Group Assumption	0.80	1.00
Bias Acknowledgement	0.25	1.00
Meta Reflection	0.00	1.00
Outside Demographic Knowledge	0.75	1.00
Outside Topical Knowledge	0.25	1.00
Unresolved	0.31	1.00

Appendix

Table E.1: Inter-annotator agreement (Cohen’s

\kappa

) on the 12-sample pilot set before and after annotation guideline refinement.

Label	Acc.	F1	$\kappa$
Outside Demographic Knowledge	0.94	0.87	0.84
Group Assumption	0.84	0.75	0.65
Overthinking	0.83	0.87	0.64
Meta Reflection	0.84	0.75	0.64
Bias Acknowledgment	0.84	0.55	0.46
Outside Topicical Knowledge	0.64	0.53	0.30

Appendix

Table E.2: Performance of DeepSeek-Chat across BiasTrace annotation behaviours, sorted by Cohen’s

\kappa

. High

\kappa

indicates consistent agreement with ground truth of human labels. Results shown on full ground truth dataset.

Figure F.1: Bar chart showing the logistic regression coefficients for the effects of BiasTrace reasoning behaviours on biased outcomes (isolated effect of each reasoning behaviour when considered independently, or its net effect when accounting for the interactions). Statistical significance is indicated by asterisks (

*p<0.05

**p<0.01

***p<0.001

). For net effects, significance shown is for coefficient of the corresponding interaction term.

Variable	Coef.	Std. Err.	p-value
Intercept	-6.667	0.260	$<$ 0.001
Prompt & Input
Full prompt (vs. simple)	-2.087	0.406	$<$ 0.001
Ambiguous	0.881	0.299	0.003
Qwen3-8B	0.245	0.085	0.004
Categories

Appendix

Table F.1: Full logistic regression results predicting biased outputs on the BBQ Dataset. Coefficients are reported in log-odds.

Figure G.1: Biased outcome and incorrect response rates for GPT-OSS-120B on the BBQ dataset, comparing low and medium reasoning effort settings across simple and guided prompt types

Figure G.2: Biased outcome rates for GPT-OSS-120B on the BBQ dataset across the different BBQ demographic categories, comparing low and medium reasoning effort settings across simple and guided prompt types.

Model	Qwen3-1.7B	Qwen3-4B	Llama3.2-3B-Instruct	Llama3-8B-Instruct
Prompt	Simple	Simple	Simple	Simple
Hardware	1 GPU	1 GPU	1 GPU	1 GPUs
Batch size	32	32	32	32
Max generation length	2048	2048	2048	2048
Temperature	0.6	0.6	0.6	0.6
Top-p	0.95	0.95	0.95	0.95

Appendix

Table H.1: Model inference configurations, including sampling parameters. The models presented were used for experiment on bias mitigation at inference time.

Figure H.1: Per-category effect of Maj - BiasTrace over Maj -All majority voting. Each row is a model; columns show accuracy change (left, Acc. Maj - BiasTrace

-

Acc. Maj -All) and bias-rate change (right, BiasRate. Maj - BiasTrace

-

BiasRate. Maj -All) for each BBQ social group category. Bars are coloured green when Maj - BiasTrace yields the desired outcome (higher accuracy on the left; lower bias on the right) and red otherwise.

Panel of Table 16 — Table I.1: Simple prompt for multiple choice reasoning on the BBQ dataset

Panel of Table 17 — Table I.2: Guided prompt for multiple choice reasoning on the BBQ dataset

Panel of Table 18 — Table I.3: Prompt for multiple choice reasoning on the COMPAS dataset

Panel of Table 19 — Table I.4: Baseline prompt for evaluating bias in model-generated reasoning, providing score of 0/1

Panel of Table 20 — Table I.5: Baseline prompt for evaluating bias in model-generated reasoning, providing score of 0-5

Panel of Table 21 — Table I.6: Prompt which incorporates BiasTrace reasoning behaviours to evaluate bias in model-generated reasoning, providing a 0/1 score of whether reasoning will result in a biased outcome

Panel of Table 22 — Table I.7: Redacted LLM-as-a-Judge prompt used for BiasTrace annotations; the full prompt is available in the codebase. Note that this prompt includes labels (unresolved and missing logic) that were excluded from the BiasTrace scheme and subsequent analysis due to unreliable judge annotations.

Panel of Table 23 — Table I.8: Prompt used to identify reasoning behaviours with Claude during development of the BiasTrace annotation scheme

Panel of Table 24 — Table I.9: Prompt used to group reasoning behaviours with Claude during development of the BiasTrace annotation scheme

Panel of Table 25 — Table I.10: Redacted Claude output showing categories of erroneous reasoning behaviours used to refine the BiasTrace annotation scheme.

Social Bias in Language Models

Social bias in language models is the study and mitigation of unfair stereotypes and disparities in generated text, judgments, and decisions.

See all 192 papers

Debiasing Large Language Models toward Social Factors in Online Behavior Analytics through Prompt Knowledge Tuning
Mar 28, 2026Hossein Salemi, Jitin Krishnan, Hemant PurohitLanguage Model Debiasing Social Bias in Language Models
George Mason University, Fairfax, Virginia, United States
Large Language Models Develop Novel Social Biases Through Adaptive Exploration
Nov 8, 2025Addison J. Wu, Ryan Liu, Xuechunzi Bai +1Social Bias in Language Models Language Model Bias Evaluation
1Princeton University · University
Adaptive Triggering for Bias Correction in LLM Reasoning
Aug 26, 2026Nayoung Kim, Mickey Mancenido, Huan LiuLanguage Model Debiasing Demographic Bias in Language Models
School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ, USA · School of Mathematical and Natural Sciences, Arizona State University, Phoenix, AZ, USA

Mar 28, 2026J/K move · Enter open · S save

Debiasing Large Language Models toward Social Factors in Online Behavior Analytics through Prompt Knowledge Tuning

Hossein Salemi, Jitin Krishnan, Hemant Purohit

George Mason University, Fairfax, Virginia, United States

Language Model Debiasing Social Bias in Language Models Causal Reasoning in Language Models Social Media Analysis Intent Classification

Open paper AskPDF arXiv

Attribution theory explains how individuals interpret and attribute others' behavior in a social context by employing personal (dispositional) and impersonal (situational) causality. Large Language Models (LLMs), trained on human-generated corpora, may implicitly mimic this social attribution process in social contexts. However, the extent to which LLMs utilize these causal attributions in their reasoning remains underexplored. Although using reasoning paradigms, such as Chain-of-Thought (CoT), has shown promising results in various tasks, ignoring social attribution in reasoning could lead to biased responses by LLMs in social contexts. In this study, we investigate the impact of incorporating a user's goal as knowledge to infer dispositional causality and message context to infer situational causality on LLM performance. To this end, we introduce a scalable method to mitigate such biases by enriching the instruction prompts for LLMs with two prompt aids using social-attribution knowledge, based on the context and goal of a social media message. This method improves the model performance while reducing the social-attribution bias of the LLM in the reasoning on zero-shot classification tasks for behavior analytics applications. We empirically show the benefits of our method across two tasks-intent detection and theme detection on social media in the disaster domain-when considering the variability of disaster types and multiple languages of social media. Our experiments highlight the biases of three open-source LLMs: Llama3, Mistral, and Gemma, toward social attribution, and show the effectiveness of our mitigation strategies.

Abstract

Social Bias in Language Models

Explore similar work

Debiasing Large Language Models toward Social Factors in Online Behavior Analytics through Prompt Knowledge Tuning

Large Language Models Develop Novel Social Biases Through Adaptive Exploration

Adaptive Triggering for Bias Correction in LLM Reasoning