Misinformation is a growing societal threat, and susceptibility to misinformative claims varies across demographic groups due to differences in underlying beliefs. As Large Language Models (LLMs) are increasingly used to simulate human behaviors, we investigate whether they can simulate demographic misinformation susceptibility, treating beliefs as a primary driving factor. We introduce BeliefSim, a simulation framework that constructs demographic belief profiles using psychology-informed misinformation taxonomies and survey priors. We study prompt-based conditioning and post-training adaptation, and conduct a multi-fold evaluation using: (i) susceptibility alignment and (ii) counterfactual demographic sensitivity. Across both datasets and modeling strategies, we show that beliefs provide a strong prior for simulating misinformation susceptibility, with alignment up to 92%.
Figures & tables
Figure 1: BeliefSim Framework. (1) Participant Data, Observed and Imputed Beliefs (mapped to BeliefSim-Tax) are collected from surveys, (2) Methods consist of prompt-conditioning and post-training adaptation and (3) Evaluation is performed using Susceptibility Alignment, Counterfactual and Thematic Analysis.
Dimension
Description
Example from WVS
Worldview & Identity
Group-based and political identities shape how individuals interpret and endorse information.
How proud are you to be from your country?
Epistemic Trust
Beliefs about what sources are credible and how knowledge should be evaluated (e.g., trust in science, media, institutions).
Would you say that the world is better off, or worse off, because of science?
Cognitive Style
Preferred thinking approach, ranging from analytical, accuracy-driven reasoning to intuitive impression-based judgments.
Nowadays, does one often have trouble deciding which moral rules to follow?
Conspiracy Mentality
General tendency to perceive hidden plots or malevolent intent behind significant social or political events.
How many people who are in the media do you believe are involved in corruption?
Morals & Values
Core moral values and cultural worldviews influence judgments of right, wrong, fairness, and societal norms.
Is it justifiable - claiming government benefits to which you are not entitled?
Emotion-Related
Emotion-driven beliefs related to fear, perceived risk, or threat sensitivity that heighten susceptibility to alarming misinformation.
How satisfied are you with your life as a whole these days?
Table 1: BeliefSim-Tax for modeling demographic-aware misinformation susceptibility in LLMs. We provide representative WVS items as examples of how survey questions align with each belief dimension.
Figure 2: Data Example. We map participant demographics to WVS responses to derive group-level imputed beliefs, and use claim-level evaluations as participant observed beliefs These are then used to predict observed beliefs.
Figure 3: Susceptibility Alignment , averaged across demographic groups and models. Imputed + Demo(graphic) (best) performs the best. All belief-based settings do better than zero-shot and demo-only. (Fig 10 shows results of all settings.)
Model
PANDORA
MIST
Qwen2.5-14B
62.28
95.97
Llama-3-8B
57.49
90.53
Llama-3.2-3B
53.89
85.14
Grok-4-Fast
57.07
87.20
Mistral-7B-v0.2
55.99
70.07
OLMo-2-7B
50.60
76.95
Table 2: Model performance. Qwen performs the best followed by Llama models while Deepseek has the worst alignment.
Figure 4: Demographic-based Counterfactual Evaluation. Note that we perform a demographic swap within the same group, i.e., swap male with female, rural with urban, etc. keeping the claim constant.
Figure 5: Flip-Rates for Counterfactual Evaluation : Mistral and Qwn models have lower flip-rates whereas Llama and Deepseek have higher rates. Darker (red) colors mean higher flip rates while lighter (yellow) colors mean lower flips.
Figure 6: Complementarity Analysis. Adding demographics has mixed effects (mostly negative) when partial belief information is added. Positive values mean that adding demographics increases accuracy, while negative values mean that it decreases accuracy.
Figure 7: BeliefSim-FT framework. Green-shaded components correspond to Phase 1 (Belief Modeling), and blue-shaded components are Phase 2 (Susceptibility Fine-Tuning).
Model
PANDORA+MIST-1
MIST-2
Acc
F1
Acc
F1
Baseline (1-phase)
Llama-3-8b
0.750
0.751
0.680
0.673
Qwen-2.5-14b
0.760
0.761
0.720
0.712
Mistral-7B
0.740
0.761
0.650
0.653
LoRA-FT (2-phase)
Table 3: Fine-tuning performances. 2-phase belief training improves performance across both datasets.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Dataset Examples - MIST and PANDORA
Figure 9: Prompt Conditioning Experiments
Figure 10: Susceptibility Alignment across settings and datasets
Dataset
Dimension
Gender
Age
Education
Rural/Urban
Prolific-dataset
worldview_and_identity
55.69
55.88
55.51
55.80
epistmetic_and_trust_beliefs
58.23
58.82
58.90
58.65
cognitive_style_beliefs
55.24
57.35
55.72
56.10
conspiracy_mindset
59.13
58.24
59.53
58.90
moral_and_value_beliefs
55.99
55.00
54.66
55.20
emotion_related
55.39
58.24
54.66
56.10
Appendix
Table 4: Belief Dimension Analysis using modal runs.
Demographic
Model
Topic/Theme
Effect and Notes
Age
Llama-3-8B
New study & ideology (T1)
+2.8 ppts (60+– <30 ). Older: 0.984 vs younger: 0.956.
Llama-3-8B
Global threats & population (T2)
+2.0 ppts. Older: 0.961 vs younger: 0.941.
Llama-3-8B
Marijuana/new-study (T4)
+1.7 ppts. 60+ group: 0.947 vs <30: 0.930.
Qwen-2.5-14B
Marijuana/new-study (T4)
+2.5 ppts. 60+ group: 0.917 vs <30: 0.892.
Mistral-7B
All topics
≤ 0.5 ppts. Age differences negligible.
Education
Llama-3-8B
Sleep/blue-light health (T2)
+8.4 ppts. Completed: 0.918 vs not: 0.833 (six cases).
Appendix
Table 5: Thematic Analysis across Demographics.
Data
Model
Belief Type
Zero-shot
Demo-only
Imputed
Observed
Imp+Obs
Imp+Demo
PANDORA
Qwen2.5-14B
modal
62.28
62.28
64.51
62.08
63.49
64.81
PANDORA
Qwen2.5-14B
dist
62.28
62.28
63.51
62.08
62.79
63.91
PANDORA
Llama-3-8B
modal
57.49
56.89
60.63
58.98
60.01
60.93
PANDORA
Llama-3-8B
dist
57.49
56.89
59.63
58.98
59.31
60.03
PANDORA
Llama-3.2-3B
modal
53.89
57.49
56.76
60.78
58.81
55.96
PANDORA
Llama-3.2-3B
dist
53.89
57.49
60.43
60.78
59.77
58.74
Appendix
Table 6: Ablations comparing modal belief conditioning and belief-distribution conditioning across datasets and models. Scores are averaged over demographic axes. Best value per row is in bold.
Axis
Dist. JS
Modal JS
Modal Disagree.
Age
0.0421
0.2511
25.53
Education
0.0498
0.2454
24.96
Gender
0.0083
0.1175
11.95
Rural/Urban
0.0072
0.0958
9.76
Appendix
Table 7: Average demographic divergence for distribution-based and modal belief prompt representations. JS divergence is reported in bits; modal disagreement reports the percentage of WVS items for which demographic groups have different modal responses.
Dataset
Low
Mid
High
PANDORA
58.24
51.03
42.95
MIST
86.13
78.13
87.77
Appendix
Table 8: Modal prompt-conditioning accuracy by claim-level entropy.
Dataset
Model
Veracity
Belief-Susc.
Δ
PANDORA
Qwen2.5-14B
62.28
64.81
+2.53
PANDORA
Llama-3-8B
57.49
60.93
+3.44
PANDORA
Llama-3.2-3B
53.89
60.78
+6.89
PANDORA
Grok-4-Fast
66.77
58.55
-8.22
PANDORA
Mistral-7B-v0.2
55.99
56.12
+0.13
PANDORA
OLMo-2-7B
50.60
58.38
+7.78
Appendix
Table 9: Comparison between zero-shot veracity accuracy and belief-conditioned susceptibility alignment. Veracity measures agreement with the gold truth label, while belief-susceptibility measures agreement with human judgments under belief- conditioned prompting.
Model
Panel A: Utility
Panel B: Shortcut
Panel C: Complementarity
Gender
Area
Age
Edu.
Gender
Area
Age
Edu.
Gender
Area
Age
Edu.
Mistral-7B-v0.2
0.13
0.67
0.07
0.64
2.38
0.00
2.78
0.00
3.11
0.35
3.25
2.39
Qwen3-4B
0.40
0.67
0.59
2.41
0.60
1.25
2.78
0.00
1.56
12.32
7.02
5.38
OLMo-2-7B
2.00
7.67
1.45
2.20
7.74
12.66
0.00
0.00
5.20
9.51
11.34
10.76
Qwen2.5-14B
4.41
2.33
0.72
1.99
2.98
7.50
0.00
0.00
5.02
4.93
2.91
3.59
Llama-3.2-3B
10.28
6.00
12.15
12.78
8.33
5.00
2.78
16.67
2.42
9.64
3.81
5.62
Appendix
Table 10: Counterfactual flip rates (%) across demographic groups. Panel A measures demographic-swap sensitivity on the original distribution, Panel B measures shortcut reliance on balanced slices, and Panel C measures complementarity under degraded beliefs.
Condition
Llama
Qwen
Mistral
Zero-shot
0.115
0.107
0.081
Demo
0.042
0.107
0.028
Imputed
−0.023
0.107
0.028
Imputed + Demo
−0.023
0.107
0.028
Observed
−0.025
0.031
0.034
Observed + Demo
−0.023
0.036
0.034
Appendix
Table 11: Spearman ρ between zero-shot factual confidence and human-label match rate on MIST-1, restricted to false claims. Positive values indicate that higher model confidence in a claim’s falsity correlates with better matching of human judgments overall; near-zero or negative values indicate decoupling from the factual prior.
Dataset
Model
Axis
Hit rows
Representative raw-output evidence
MIST-1
Mistral-7B-Instruct-v0.2
Education
36/36
“A person with a completed education is less likely to believe in such a sensational and unsubstantiated claim .”
MIST-1
Mistral-7B-Instruct-v0.2
Education
36/36
“This person, with not completed education , is less likely to have the knowledge or ability to make an informed judgment .”
MIST-1
Mistral-7B-Instruct-v0.2
Gender
30/54
“This person, being male , is likely to believe headlines that discuss international politics and economic transactions. ”
MIST-1
Mistral-7B-Instruct-v0.2
Gender
29/54
“This person, a female , is likely to be informed about global news and economics. ”
PANDORA
OLMo-2-1124-7B-Instruct
Living Area
2/80
“The person living in a rural area might be more inclined to trust the news due to their exposure to fewer alternative sources .”
PANDORA
Llama-3.2-3B-Instruct
Living Area
10/80
“The person’s perspective is influenced by their living area, which is rural, and may not be familiar with the specifics of the Ferguson incident .”
Appendix
Table 12: Qualitative examples of explicit demographic references in raw outputs for Panel B shortcut-reliance cases. Counts indicate rows whose raw output mentions the corresponding demographic axis; excerpts are illustrative and are not used as the primary metric.
Model
Head Tuning
Full Fine-tuning
Qwen
0.0511
0.0732
Mistral
0.087
0.107
Llama
0.287
0.487
Appendix
Table 13: KL divergence comparison across different models and tuning methods.
Model
PANDORA+MIST-1
MIST-2
Acc
F1
Acc
F1
Baseline (1-phase)
Llama
0.750
0.751
0.680
0.673
Qwen
0.760
0.761
0.720
0.712
Mistral
0.740
0.761
0.650
0.653
LoRA-FT (2-phase)
Appendix
Table 14: Fine-tuning performances
Method
Model
PANDORA+MIST-1
MIST-2
Flip Rate
Prob Delta
Acc Drop
Flip Rate
Prob Delta
Acc Drop
Full-FT
Llama
0.0239
0.0148
0.0049
0.0000
0.0000
0.0000
Qwen
0.0173
0.0476
0.0012
0.1098
0.0700
0.1027
Mistral
0.0323
0.0024
0.0000
0.0000
0.0129
0.0000
Head Tuning
Llama
0.0000
0.0125
0.0000
0.0000
0.0129
0.0000
Qwen
0.0080
0.0261
0.0012
0.0000
0.0427
0.0000
Appendix
Table 15: Shortcut reliance comparison between Full Fine-tuning and Head Tuning across different models on PANDORA+MIST-1 and MIST-2 datasets.
Figure 11: Prompt for Phase-1 Belief Modeling
Dimension
Q ID
Question
WorldView and Identity Beliefs
Q6
Importance of religion (1=Very important to 4=Not at all important)
Q4
Importance of politics (1=Very important to 4=Not at all important)
Q171
Frequency of religious service attendance (1=More than once/week to 7=Never)
Q173
Religious self-identification (1=Religious person, 2=Not religious, 3=Atheist)
Q240
Political left-right scale (1-10)
Appendix
Table 16: World Values Survey Questions mapped to Misinformation Taxonomy Dimensions
LLMs are increasingly deployed as proxies for human study participants in social science experiments, yet the fidelity of this practice has rarely been tested directly. We test whether six LLMs can simulate individual human belief updates, comparing LLM outputs 1-to-1 against ground truth data from 391 UK participants on Prolific, who updated their stances on three discussion topics after reading Reddit comments. Each participant was simulated by an LLM conditioned on a persona derived from their demographic and personality trait data. We find that some LLMs (Qwen3-32B and GPT-5-Mini) can match the human post-stance distribution, but only when given participants' actual initial stances. All six models fail to simulate initial stances themselves and to produce faithful belief updates from self-generated stances. Three systematic biases emerge across all models: overrepresentation of neutral positions, more frequent but smaller belief shifts than humans, and a failure to rank comments by convincingness. Demographic and personality trait personas had no consistent effect on fidelity. LLM simulations of human belief dynamics are only reliable when grounded in realistic starting conditions, that current multi-round social media simulations rarely provide.
Sebastian Pohl, Harsh Mehta, Pranav Mambayil +4
IT:U, Interdisciplinary Transformation University Austria, Altenberger Straße 66c, 4040 Linz, Austria
Can LLMs simulate how humans form and change beliefs in social networks? We put this to the test by replicating an established study on belief dynamics, evaluating 12 LLMs across multiple model families and parameter sizes. The answer is a clear no, and in systematic ways. LLMs fail to capture initial human belief distributions and tend to be overall more conformist than humans, shifting their responses to align with those around them. They also take a nuanced approach to emulating human homophilic tendencies within networks. Our findings carry a double payoff: they highlight fundamental properties of LLM behavior, and they raise a sharp warning against deploying LLMs as human proxies in social simulations.
Adiba Mahbub Proma, Neeley Pate, James N. Druckman +3
Department of Computer Science University of Rochester · Department of Political Science University of Rochester · Department of Physics and Astronomy University of Rochester
Large language models (LLMs) are increasingly used to simulate social survey responses, yet their outputs exhibit systematic biases: marginal distributions are skewed, response variance is poorly calibrated, and predictor-outcome relationships are attenuated. We ask a simple question: given a small pilot sample of human responses, can an LLM recover the statistical characteristics of a broader population? We decompose recovery along three axes: structural fidelity, marginal fidelity, and individual fidelity. Using a COVID-19 misinformation survey as a case study, we benchmark three families of approaches: prompting, rectification, and fine-tuning. The findings suggest that fine-tuning on small pilot samples offers a balanced approach for achieving multiple forms of fidelity, but the levels of such fidelity can vary across subsamples, potentially threatening pluralistic alignment.
Eun Cheol Choi, Youngrae Kim, Prabhu Pugalenthi +2