Debiasing Large Language Models toward Social Factors in Online Behavior Analytics through Prompt Knowledge Tuning

Hossein Salemi; Jitin Krishnan; Hemant Purohit

cs.CLMar 28, 2026

Debiasing Large Language Models toward Social Factors in Online Behavior Analytics through Prompt Knowledge Tuning

Authors: Hossein Salemi, Jitin Krishnan, Hemant Purohit

Organizations: George Mason University, Fairfax, Virginia, United States

Abstract

Attribution theory explains how individuals interpret and attribute others' behavior in a social context by employing personal (dispositional) and impersonal (situational) causality. Large Language Models (LLMs), trained on human-generated corpora, may implicitly mimic this social attribution process in social contexts. However, the extent to which LLMs utilize these causal attributions in their reasoning remains underexplored. Although using reasoning paradigms, such as Chain-of-Thought (CoT), has shown promising results in various tasks, ignoring social attribution in reasoning could lead to biased responses by LLMs in social contexts. In this study, we investigate the impact of incorporating a user's goal as knowledge to infer dispositional causality and message context to infer situational causality on LLM performance. To this end, we introduce a scalable method to mitigate such biases by enriching the instruction prompts for LLMs with two prompt aids using social-attribution knowledge, based on the context and goal of a social media message. This method improves the model performance while reducing the social-attribution bias of the LLM in the reasoning on zero-shot classification tasks for behavior analytics applications. We empirically show the benefits of our method across two tasks-intent detection and theme detection on social media in the disaster domain-when considering the variability of disaster types and multiple languages of social media. Our experiments highlight the biases of three open-source LLMs: Llama3, Mistral, and Gemma, toward social attribution, and show the effectiveness of our mitigation strategies.

Figures & tables

Fig. 1 : Example of our Goal-based prompt for intent detection task to alleviate the social-attribution bias of Llama3 model by providing the knowledge aid of social factors, i.e., the user’s goal in this prompt.

Fig. 2 : Distribution of significant social attribution causalities in the Intent task across models, highlighting the underestimation of dispositional causality.

Method	Social-attribution Knowledge	Knowledge Source	Instance-Specific
[ 4 ]	×	Implicit (LLM prior)	×
[ 8 ]	×	Human-crafted	×
[ 9 ]	×	External knowledge	×
[ 10 ]	Partial (context only)	Human-crafted	×
Ours	Context + User Goal	LLM-generated	✓

TABLE I : Comparison of our method with prior context injection and external knowledge prompting approaches.

Fig. 3 : Our proposed framework to investigate the social-attribution bias of LLMs in the behavioral analytics tasks.

Fig. 4 : Pipeline of our proposed method.

Task	Causality	Description	Prompt Strategy
Intent	Dispositional	Model’s capability of understanding the user’s goal	Goal-based
Theme	Situational	Model’s capability of understanding the task domain	Context-based

TABLE II : Motivation for designing and selecting prompt strategies based on the matching of the expected social-attribution knowledge aid for a task.

Classify the given tweet into one of the two given categories. - Donations and volunteering: Tweets requesting … - Affected individuals: Tweets reporting injuries … - Infrastructure and utilities: Tweets reporting … Tweet: New to #yyc and doing my part to clean up. Amazing support. #abflood #yycflood #calgarystrong <URL>
	Zero-CoT	Context-based
Knowledge Aid	-	The summary of the tweet: A newcomer to Calgary is helping with the flood cleanup efforts and is impressed with the community’s support.
Reasoning	Let’s break down the tweet: * "New to #yyc and doing my part to clean up" - This suggests that the person is participating in cleanup efforts, which implies that they are affected by the flood. * "Amazing support" - This could be referring to the support from others, which is not directly related to the flood.	The tweet mentions the individual is "doing my part to clean up", which indicates they are volunteering their time to help with the flood cleanup efforts. Additionally, the tweet expresses appreciation for the community’s support , which suggests that the individual is part of the response efforts.
Answer	Affected Individual ✗	Donations and volunteering ✓

TABLE III : Samples of theme detection using Llama3 model

Fig. 5 : Examples of Context and Goal knowledge aids generated by Llama3 model.

Task	Lang.	Event	Labels (#Samples)
Intent	Mono	Lilac (en)	Help-seeking (128), Help-offering (263)
		Thomas (en)	Help-seeking (156), Help-offering (177)
	Multi	Haiti (ht)	Requests (324), Others (196)
		Chile (es)	Requests or Needs (64), Others (149)
Theme	Mono	Alberta (en)	Donations and volunteering (220), Infrastructure and utilities (199), Affected individuals (78)
		Colorado (en)	Donations and volunteering (149), Infrastructure and utilities (175), Affected individuals (205)

TABLE IV : Dataset Statistics

Panel of Figure 10 — Fig. 6 : Templates for context and goal knowledge generation.

Panel of Figure 11 — Fig. 7 : Prompt template for multi-class classification task.

Task	Lang	Dataset	Llama3			Mistral			Gemma
			Zero-CoT	Self-Debias	Ours	Zero-CoT	Self-Debias	Ours	Zero-CoT	Self-Debias	Ours
Intent	Mono	Lilac	80.8 $\pm$ 1.4	79.6 $\pm$ 3.2	88.7 $\pm$ 0.4	77.6 $\pm$ 2.7	74.0 $\pm$ 3.4	81.8 $\pm$ 1.2	74.0 $\pm$ 1.7	76.7 $\pm$ 0.4	83.8 $\pm$ 0.7
		Thomas	82.5 $\pm$ 0.3	82.4 $\pm$ 1.1	86.2 $\pm$ 0.6	79.4 $\pm$ 2.7	79.2 $\pm$ 2.2	87.4 $\pm$ 0.3	79.3 $\pm$ 0.7	80.8 $\pm$ 1.1	86.2 $\pm$ 0.9
	Multi	Haiti (ht)	67.2 $\pm$ 0.2	64.1 $\pm$ 4.1	69.8 $\pm$ 0.5	67.5 $\pm$ 1.7	61.5 $\pm$ 0.6	68.5 $\pm$ 0.7	51.7 $\pm$ 1.4	54.0 $\pm$ 1.2	65.4 $\pm$ 0.8
		Chile (es)	70.0 $\pm$ 0.4	72.1 $\pm$ 1.5	70.9 $\pm$ 0.8	71.4 $\pm$ 0.6	72.4 $\pm$ 1.1	71.0 $\pm$ 0.6	62.6 $\pm$ 0.5	62.6 $\pm$ 0.5	64.4 $\pm$ 0.8
Theme	Mono	Alberta	74.0 $\pm$ 3.0	75.7 $\pm$ 4.9	79.3 $\pm$ 3.2	81.0 $\pm$ 2.1	79.1 $\pm$ 3.0	82.8 $\pm$ 2.0	75.5 $\pm$ 2.8	76.9 $\pm$ 1.7	77.0 $\pm$ 2.6

TABLE V : The performance (macro-F1 score) of our proposed strategy in enhancing the prediction accuracy of the target models in intent detection and theme detection tasks on mono-lingual and multi-lingual datasets. The best result is bolded .

Panel of Figure 13 — Fig. 8 : The average of relative improvement of our strategies over Zero-CoT by different LLMs across the target tasks.

Panel of Figure 14 — Fig. 9 : The average improvement of our strategies over Zero-CoT by different LLMs in mono-lingual setting.

Panel of Figure 15 — Fig. 10 : The effect of temperature parameter on the model performance in the intent detection task.

Fig. 11 : Comparison of the order bias of methods based on the standard deviation of Macro-F1 scores across model/tasks.

Task	Dataset	Llama3		Gemma
		Self	Cross	Self	Cross
Intent	Lilac	88.7 $\pm$ 0.4	84.5 $\pm$ 0.8	83.8 $\pm$ 0.7	86.0 $\pm$ 0.7
	Thomas	86.2 $\pm$ 0.6	85.0 $\pm$ 0.7	86.2 $\pm$ 0.9	85.6 $\pm$ 0.2
Theme	Alberta	79.3 $\pm$ 3.2	80.3 $\pm$ 1.8	77.0 $\pm$ 2.6	76.8 $\pm$ 2.1
	Colorado	83.2 $\pm$ 1.5	83.7 $\pm$ 1.5	84.2 $\pm$ 1.1	83.8 $\pm$ 1.3

TABLE VI : The performance (macro-F1) comparison between the impact of using knowledge generated by model itself (Self) and by another model (Cross) for Llama3 and Gemma.

Task	Model	Helpfulness (%)	Faithfulness (%)
Intent	Llama3	0.88	0.98
	Mistral	0.88	0.98
	Gemma	0.85	0.98
Theme	Llama3	0.85	0.98
	Mistral	0.95	0.98
	Gemma	0.88	0.98

TABLE VII : Human evaluation results of AI-generated knowledge quality across tasks and models.

Language Model Debiasing

See all 26 papers

Examining Social Attribution in LLM Reasoning: A Theory-Guided Probing Methodology
Oct 8, 2026Zhaoxin Yu, Qingchao Kong, Dajun Zeng +1Causal Attribution
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
BiasTrace: Linking Reasoning Behaviours to Biased Outputs in LLMs
Aug 14, 2026Varsha Ramineni, Hossein A. Rahmani, Jerome Ramos +2Social Bias in Language Models Language Model Bias Evaluation
Centre for Artificial Intelligence, University College London · NVIDIA
Knowing Bias, Doing Better: Mitigating Social Bias in LLMs via Know-Bias Neuron Enhancement
Jan 29, 2026Jinhao Pan, Chahat Raj, Anjishnu Mukherjee +4Debiased ML Social Bias in Language Models
Department of Computer Science, George Mason University, Fairfax, VA, USA · Lightridge High School, Aldie, VA, USA

Oct 8, 2026J/K move · Enter open · S save

Examining Social Attribution in LLM Reasoning: A Theory-Guided Probing Methodology

Zhaoxin Yu, Qingchao Kong, Dajun Zeng, Wenji Mao

State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences

Causal Attribution

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly deployed in sociotechnical systems where social attribution, the reasoning process attributing external events to the causes and reasons of agents' social behaviors, plays a critical role. These processes involve judgments of social cause, responsibility, and blame/credit to agents. Although attributional models are well-studied in social psychology and cognition through Attribution Theory, social attribution remains underexplored in AI, particularly LLM social reasoning. This paper provides the first systematic exploration of LLM social attribution. Our work focuses on responsibility and blame attributions, examining current LLMs' judgments and their underlying internal mechanisms. Guided by attribution theory, we construct a social attribution benchmark consisting of a Vignette subset based on classic scenarios from attribution theory research and a Reality subset based on real-world social narratives, yielding 7,639 responsibility/blame judgment questions. On this basis, we evaluate 32 representative LLMs and 5 basic non-LLM baselines. To further explore the internal mechanisms underlying the LLM judgment process, we develop a probing-based methodology to investigate the latent-space representations of 5 key attribution dimensions and the consistency of their influences on LLM judgments compared to those in human social attribution. Our research findings reveal that current LLMs exhibit measurable but incomplete agreement with human responsibility and blame judgments, and meanwhile, this agreement is positively correlated with model size. Some attribution dimensions are systematically decodable from specific positions in LLM hidden states, and their influences on the final judgment are consistent with those indicated by human Attribution Theory. The dataset and associated code are available at https://github.com/Yuzhaoxin946/SAB-Bench.

Abstract

Language Model Debiasing

Explore similar work

Examining Social Attribution in LLM Reasoning: A Theory-Guided Probing Methodology

BiasTrace: Linking Reasoning Behaviours to Biased Outputs in LLMs

Knowing Bias, Doing Better: Mitigating Social Bias in LLMs via Know-Bias Neuron Enhancement