Belief-Sim: Towards Belief-Driven Simulation of Demographic Misinformation Susceptibility

Angana Borah; Zohaib Khan; Rada Mihalcea; Verónica Pérez-Rosas

cs.CLMar 3, 2026

Belief-Sim: Towards Belief-Driven Simulation of Demographic Misinformation Susceptibility

Authors: Angana Borah, Zohaib Khan, Rada Mihalcea, Verónica Pérez-Rosas

Organizations: University of Michigan - Ann Arbor, USA · Texas State University

Abstract

Misinformation is a growing societal threat, and susceptibility to misinformative claims varies across demographic groups due to differences in underlying beliefs. As Large Language Models (LLMs) are increasingly used to simulate human behaviors, we investigate whether they can simulate demographic misinformation susceptibility, treating beliefs as a primary driving factor. We introduce BeliefSim, a simulation framework that constructs demographic belief profiles using psychology-informed misinformation taxonomies and survey priors. We study prompt-based conditioning and post-training adaptation, and conduct a multi-fold evaluation using: (i) susceptibility alignment and (ii) counterfactual demographic sensitivity. Across both datasets and modeling strategies, we show that beliefs provide a strong prior for simulating misinformation susceptibility, with alignment up to 92%.

Figures & tables

Figure 1: BeliefSim Framework. (1) Participant Data, Observed and Imputed Beliefs (mapped to BeliefSim-Tax) are collected from surveys, (2) Methods consist of prompt-conditioning and post-training adaptation and (3) Evaluation is performed using Susceptibility Alignment, Counterfactual and Thematic Analysis.

Dimension	Description	Example from WVS
Worldview & Identity	Group-based and political identities shape how individuals interpret and endorse information.	How proud are you to be from your country?
Epistemic Trust	Beliefs about what sources are credible and how knowledge should be evaluated (e.g., trust in science, media, institutions).	Would you say that the world is better off, or worse off, because of science?
Cognitive Style	Preferred thinking approach, ranging from analytical, accuracy-driven reasoning to intuitive impression-based judgments.	Nowadays, does one often have trouble deciding which moral rules to follow?
Conspiracy Mentality	General tendency to perceive hidden plots or malevolent intent behind significant social or political events.	How many people who are in the media do you believe are involved in corruption?
Morals & Values	Core moral values and cultural worldviews influence judgments of right, wrong, fairness, and societal norms.	Is it justifiable - claiming government benefits to which you are not entitled?
Emotion-Related	Emotion-driven beliefs related to fear, perceived risk, or threat sensitivity that heighten susceptibility to alarming misinformation.	How satisfied are you with your life as a whole these days?

Table 1: BeliefSim-Tax for modeling demographic-aware misinformation susceptibility in LLMs. We provide representative WVS items as examples of how survey questions align with each belief dimension.

Figure 2: Data Example. We map participant demographics to WVS responses to derive group-level imputed beliefs, and use claim-level evaluations as participant observed beliefs These are then used to predict observed beliefs.

Figure 3: Susceptibility Alignment , averaged across demographic groups and models. Imputed + Demo(graphic) (best) performs the best. All belief-based settings do better than zero-shot and demo-only. (Fig 10 shows results of all settings.)

Model	PANDORA	MIST
Qwen2.5-14B	62.28	95.97
Llama-3-8B	57.49	90.53
Llama-3.2-3B	53.89	85.14
Grok-4-Fast	57.07	87.20
Mistral-7B-v0.2	55.99	70.07
OLMo-2-7B	50.60	76.95

Table 2: Model performance. Qwen performs the best followed by Llama models while Deepseek has the worst alignment.

Figure 4: Demographic-based Counterfactual Evaluation. Note that we perform a demographic swap within the same group, i.e., swap male with female, rural with urban, etc. keeping the claim constant.

Figure 5: Flip-Rates for Counterfactual Evaluation : Mistral and Qwn models have lower flip-rates whereas Llama and Deepseek have higher rates. Darker (red) colors mean higher flip rates while lighter (yellow) colors mean lower flips.

Figure 6: Complementarity Analysis. Adding demographics has mixed effects (mostly negative) when partial belief information is added. Positive values mean that adding demographics increases accuracy, while negative values mean that it decreases accuracy.

Figure 7: BeliefSim-FT framework. Green-shaded components correspond to Phase 1 (Belief Modeling), and blue-shaded components are Phase 2 (Susceptibility Fine-Tuning).

Model	PANDORA+MIST-1		MIST-2
Model	Acc	F1	Acc	F1
Baseline (1-phase)
Llama-3-8b	0.750	0.751	0.680	0.673
Qwen-2.5-14b	0.760	0.761	0.720	0.712
Mistral-7B	0.740	0.761	0.650	0.653
LoRA-FT (2-phase)

Table 3: Fine-tuning performances. 2-phase belief training improves performance across both datasets.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: Dataset Examples - MIST and PANDORA

Panel of Figure 9 — Figure 9: Prompt Conditioning Experiments

Figure 10: Susceptibility Alignment across settings and datasets

Dataset	Dimension	Gender	Age	Education	Rural/Urban
Prolific-dataset	worldview_and_identity	55.69	55.88	55.51	55.80
	epistmetic_and_trust_beliefs	58.23	58.82	58.90	58.65
	cognitive_style_beliefs	55.24	57.35	55.72	56.10
	conspiracy_mindset	59.13	58.24	59.53	58.90
	moral_and_value_beliefs	55.99	55.00	54.66	55.20
	emotion_related	55.39	58.24	54.66	56.10

Appendix

Table 4: Belief Dimension Analysis using modal runs.

Demographic	Model	Topic/Theme	Effect and Notes
Age	Llama-3-8B	New study & ideology (T1)	+2.8 ppts (60+– $<30$ ). Older: 0.984 vs younger: 0.956.
	Llama-3-8B	Global threats & population (T2)	+2.0 ppts. Older: 0.961 vs younger: 0.941.
	Llama-3-8B	Marijuana/new-study (T4)	+1.7 ppts. 60+ group: 0.947 vs <30: 0.930.
	Qwen-2.5-14B	Marijuana/new-study (T4)	+2.5 ppts. 60+ group: 0.917 vs <30: 0.892.
	Mistral-7B	All topics	$\leq$ 0.5 ppts. Age differences negligible.
Education	Llama-3-8B	Sleep/blue-light health (T2)	+8.4 ppts. Completed: 0.918 vs not: 0.833 (six cases).

Appendix

Table 5: Thematic Analysis across Demographics.

Data	Model	Belief Type	Zero-shot	Demo-only	Imputed	Observed	Imp+Obs	Imp+Demo
PANDORA	Qwen2.5-14B	modal	62.28	62.28	64.51	62.08	63.49	64.81
PANDORA	Qwen2.5-14B	dist	62.28	62.28	63.51	62.08	62.79	63.91
PANDORA	Llama-3-8B	modal	57.49	56.89	60.63	58.98	60.01	60.93
PANDORA	Llama-3-8B	dist	57.49	56.89	59.63	58.98	59.31	60.03
PANDORA	Llama-3.2-3B	modal	53.89	57.49	56.76	60.78	58.81	55.96
PANDORA	Llama-3.2-3B	dist	53.89	57.49	60.43	60.78	59.77	58.74

Appendix

Table 6: Ablations comparing modal belief conditioning and belief-distribution conditioning across datasets and models. Scores are averaged over demographic axes. Best value per row is in bold.

Axis	Dist. JS	Modal JS	Modal Disagree.
Age	0.0421	0.2511	25.53
Education	0.0498	0.2454	24.96
Gender	0.0083	0.1175	11.95
Rural/Urban	0.0072	0.0958	9.76

Appendix

Table 7: Average demographic divergence for distribution-based and modal belief prompt representations. JS divergence is reported in bits; modal disagreement reports the percentage of WVS items for which demographic groups have different modal responses.

Dataset	Low	Mid	High
PANDORA	58.24	51.03	42.95
MIST	86.13	78.13	87.77

Appendix

Table 8: Modal prompt-conditioning accuracy by claim-level entropy.

Dataset	Model	Veracity	Belief-Susc.	$\Delta$
PANDORA	Qwen2.5-14B	62.28	64.81	+2.53
PANDORA	Llama-3-8B	57.49	60.93	+3.44
PANDORA	Llama-3.2-3B	53.89	60.78	+6.89
PANDORA	Grok-4-Fast	66.77	58.55	-8.22
PANDORA	Mistral-7B-v0.2	55.99	56.12	+0.13
PANDORA	OLMo-2-7B	50.60	58.38	+7.78

Appendix

Table 9: Comparison between zero-shot veracity accuracy and belief-conditioned susceptibility alignment. Veracity measures agreement with the gold truth label, while belief-susceptibility measures agreement with human judgments under belief- conditioned prompting.

Model	Panel A: Utility				Panel B: Shortcut				Panel C: Complementarity
Model	Gender	Area	Age	Edu.	Gender	Area	Age	Edu.	Gender	Area	Age	Edu.
Mistral-7B-v0.2	0.13	0.67	0.07	0.64	2.38	0.00	2.78	0.00	3.11	0.35	3.25	2.39
Qwen3-4B	0.40	0.67	0.59	2.41	0.60	1.25	2.78	0.00	1.56	12.32	7.02	5.38
OLMo-2-7B	2.00	7.67	1.45	2.20	7.74	12.66	0.00	0.00	5.20	9.51	11.34	10.76
Qwen2.5-14B	4.41	2.33	0.72	1.99	2.98	7.50	0.00	0.00	5.02	4.93	2.91	3.59
Llama-3.2-3B	10.28	6.00	12.15	12.78	8.33	5.00	2.78	16.67	2.42	9.64	3.81	5.62

Appendix

Table 10: Counterfactual flip rates (%) across demographic groups. Panel A measures demographic-swap sensitivity on the original distribution, Panel B measures shortcut reliance on balanced slices, and Panel C measures complementarity under degraded beliefs.

Condition	Llama	Qwen	Mistral
Zero-shot	$0.115$	$0.107$	$0.081$
Demo	$0.042$	$0.107$	$0.028$
Imputed	$-0.023$	$0.107$	$0.028$
Imputed + Demo	$-0.023$	$0.107$	$0.028$
Observed	$-0.025$	$0.031$	$0.034$
Observed + Demo	$-0.023$	$0.036$	$0.034$

Appendix

Table 11: Spearman

\rho

between zero-shot factual confidence and human-label match rate on MIST-1, restricted to false claims. Positive values indicate that higher model confidence in a claim’s falsity correlates with better matching of human judgments overall; near-zero or negative values indicate decoupling from the factual prior.

Dataset	Model	Axis	Hit rows	Representative raw-output evidence
MIST-1	Mistral-7B-Instruct-v0.2	Education	36/36	“A person with a completed education is less likely to believe in such a sensational and unsubstantiated claim .”
MIST-1	Mistral-7B-Instruct-v0.2	Education	36/36	“This person, with not completed education , is less likely to have the knowledge or ability to make an informed judgment .”
MIST-1	Mistral-7B-Instruct-v0.2	Gender	30/54	“This person, being male , is likely to believe headlines that discuss international politics and economic transactions. ”
MIST-1	Mistral-7B-Instruct-v0.2	Gender	29/54	“This person, a female , is likely to be informed about global news and economics. ”
PANDORA	OLMo-2-1124-7B-Instruct	Living Area	2/80	“The person living in a rural area might be more inclined to trust the news due to their exposure to fewer alternative sources .”
PANDORA	Llama-3.2-3B-Instruct	Living Area	10/80	“The person’s perspective is influenced by their living area, which is rural, and may not be familiar with the specifics of the Ferguson incident .”

Appendix

Table 12: Qualitative examples of explicit demographic references in raw outputs for Panel B shortcut-reliance cases. Counts indicate rows whose raw output mentions the corresponding demographic axis; excerpts are illustrative and are not used as the primary metric.

Model	Head Tuning	Full Fine-tuning
Qwen	0.0511	0.0732
Mistral	0.087	0.107
Llama	0.287	0.487

Appendix

Table 13: KL divergence comparison across different models and tuning methods.

Model	PANDORA+MIST-1		MIST-2
Model	Acc	F1	Acc	F1
Baseline (1-phase)
Llama	0.750	0.751	0.680	0.673
Qwen	0.760	0.761	0.720	0.712
Mistral	0.740	0.761	0.650	0.653
LoRA-FT (2-phase)

Appendix

Table 14: Fine-tuning performances

Method	Model	PANDORA+MIST-1			MIST-2
Method	Model	Flip Rate	Prob Delta	Acc Drop	Flip Rate	Prob Delta	Acc Drop
Full-FT	Llama	0.0239	0.0148	0.0049	0.0000	0.0000	0.0000
	Qwen	0.0173	0.0476	0.0012	0.1098	0.0700	0.1027
	Mistral	0.0323	0.0024	0.0000	0.0000	0.0129	0.0000
Head Tuning	Llama	0.0000	0.0125	0.0000	0.0000	0.0129	0.0000
	Qwen	0.0080	0.0261	0.0012	0.0000	0.0427	0.0000

Appendix

Table 15: Shortcut reliance comparison between Full Fine-tuning and Head Tuning across different models on PANDORA+MIST-1 and MIST-2 datasets.

Panel of Figure 11 — Figure 11: Prompt for Phase-1 Belief Modeling

Dimension	Q ID	Question
WorldView and Identity Beliefs
	Q6	Importance of religion (1=Very important to 4=Not at all important)
	Q4	Importance of politics (1=Very important to 4=Not at all important)
	Q171	Frequency of religious service attendance (1=More than once/week to 7=Never)
	Q173	Religious self-identification (1=Religious person, 2=Not religious, 3=Atheist)
	Q240	Political left-right scale (1-10)

Appendix

Table 16: World Values Survey Questions mapped to Misinformation Taxonomy Dimensions

Human Behavior Simulation

Human behavior simulation models how individuals and groups decide, interact, move, and adapt in social, economic, educational, and physical settings.

See all 229 papers

LLMs struggle to simulate human belief updates in controlled environments
Jul 30, 2026Sebastian Pohl, Harsh Mehta, Pranav Mambayil +4Human Behavior Simulation LLM Reliability
IT:U, Interdisciplinary Transformation University Austria, Altenberger Straße 66c, 4040 Linz, Austria
Can LLMs Emulate Human Belief Dynamics?
May 5, 2026Adiba Mahbub Proma, Neeley Pate, James N. Druckman +3Opinion Dynamics Normative Conformity in Language Models
Department of Computer Science University of Rochester · Department of Political Science University of Rochester · Department of Physics and Astronomy University of Rochester
Beyond the Mean: Three-Axis Fidelity for Aligning LLM-Based Survey Simulators from Small Pilot Data
Jun 27, 2026Eun Cheol Choi, Youngrae Kim, Prabhu Pugalenthi +2LLM Alignment LLM Fine-Tuning
University of Southern California, CA, USA.

Jul 30, 2026J/K move · Enter open · S save

LLMs struggle to simulate human belief updates in controlled environments

Sebastian Pohl, Harsh Mehta, Pranav Mambayil, Abdul Ghafoor+3

IT:U, Interdisciplinary Transformation University Austria, Altenberger Straße 66c, 4040 Linz, Austria

Human Behavior Simulation LLM Reliability Belief Updating in LLMs Large Language Model-Based Social Simulation

Open paper AskPDF arXiv

LLMs are increasingly deployed as proxies for human study participants in social science experiments, yet the fidelity of this practice has rarely been tested directly. We test whether six LLMs can simulate individual human belief updates, comparing LLM outputs 1-to-1 against ground truth data from 391 UK participants on Prolific, who updated their stances on three discussion topics after reading Reddit comments. Each participant was simulated by an LLM conditioned on a persona derived from their demographic and personality trait data. We find that some LLMs (Qwen3-32B and GPT-5-Mini) can match the human post-stance distribution, but only when given participants' actual initial stances. All six models fail to simulate initial stances themselves and to produce faithful belief updates from self-generated stances. Three systematic biases emerge across all models: overrepresentation of neutral positions, more frequent but smaller belief shifts than humans, and a failure to rank comments by convincingness. Demographic and personality trait personas had no consistent effect on fidelity. LLM simulations of human belief dynamics are only reliable when grounded in realistic starting conditions, that current multi-round social media simulations rarely provide.

Abstract

Human Behavior Simulation

Explore similar work

LLMs struggle to simulate human belief updates in controlled environments

Can LLMs Emulate Human Belief Dynamics?

Beyond the Mean: Three-Axis Fidelity for Aligning LLM-Based Survey Simulators from Small Pilot Data