cs.AISep 28, 2026

Illusory Truth or Mere Exposure? Model-Dependent Repetition Effects in LLM-Based Social Media Simulations

Authors: Azza Bouleimen, Nicolò Pagan, Anikó Hannák

Organizations: Department of Computer Science University of Zurich Andreasstrasse 15, 8050, Zurich, Switzerland

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Generative agent-based models (GABMs) are increasingly used to simulate social media dynamics, including misinformation spread. For such social simulations to be valid proxies of human behavior, LLM agents should replicate established human cognitive biases, among them the Illusory Truth Effect (ITE), where repeated exposure to a claim increases its perceived truth value. We investigate whether and how the ITE manifests across four LLMs (Gemma-3-4b-it, Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, and GPT-5-nano) in a social media simulation context. We propose a two-phase within-context experimental design that embeds the repetition manipulation inside a realistic news feed interaction. Using this design, we collect 336,000 truth, importance, sentiment, and interest ratings across 100 statements, 10 feed variants, and 3 replications. The key comparison is between ratings assigned to repeated statements, seen throughout a simulation phase, and completely unseen ones, rated within the same experimental context window. We distinguish genuine ITE (truth-specific repetition boost) from mere exposure effects. We run an OLS regression followed by a Linear Mixed-Effect Model to account for differences across models and ratings. Our results reveal four qualitatively distinct patterns: Gemma-3 exhibits a genuine ITE; Qwen2.5 shows a mere exposure effect; GPT-5-nano displays no repetition effect on truth and mild skepticism toward repeated content; Llama-3.1 shows a small truth boost alongside decreases in evaluative dimensions. Crucially, temperature has no effect on these findings, and a variance decomposition highlights the high context-sensitivity of LLM rating behavior. Our findings caution against assuming uniform ITE replication across LLMs in social simulations, while suggesting that Gemma-3-4b-it may offer the most behaviorally realistic approximation for misinformation-related simulations.

Figures & tables

Figure 1: Illustration of the experiment.
ModelRating Attr.Param.Temp. 1Temp. 0.1
   Est.95% CIEst.95% CI
Llama-3.1-8Btruthoffset0.258**[0.073, 0.443]0.269*[0.042, 0.496]
tilt−0.149**[-0.245, −0.052]−0.225***[−0.327, −0.122]
importanceoffset0.078[−0.068, 0.223]−0.102[−0.275, 0.071]
tilt−0.205***[−0.302, −0.108]−0.191***[−0.296, −0.087]
sentimentoffset−0.231**[−0.360, −0.103]−0.251***[−0.385, −0.117]
Table 1: Statistical model estimates for the four models across four rating attributes. Red = significant positive/negative offset; Blue = significant tilt. Bold = p<0.05p<0.05 . Significance: * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . † GPT-5-nano temperature is fixed to 1; Temp. 0.1 results not available.
Figure 2: Plots of the mean truth rating per statement when unseen and repeated in the simulation phase. (a) Llama-3.1-8B-Instruct , (b) Qwen2.5-7B-Instruct , (c) Gemma-3-4b-it and (d) GPT-5-nano . All models are set to temperature 1. Error bars are 95% confidence intervals. The dashed black line is the identity function, the solid red line is the best OLS fit.
 Gemma-3-4b-itgpt-5-nanoLlama-3.1-8BQwen2.5-7B
Attributeδ^\hat{\delta}tt δ^\hat{\delta}tt δ^\hat{\delta}tt δ^\hat{\delta}tt 
truth+0.977+0.977−35.22\phantom{-}35.22−0.041-0.041−1.02-1.02ns+0.097+0.097−3.50\phantom{-}3.50+1.012+1.012−36.55\phantom{-}36.55
importance+0.607+0.607−22.04\phantom{-}22.04+0.003+0.003−0.08\phantom{-}0.08ns−0.149-0.149−5.39-5.39+1.184+1.184−42.83\phantom{-}42.83
interest+0.083+0.083−2.98\phantom{-}2.98**−0.256-0.256−6.39-6.39−0.107-0.107−3.87-3.87+0.534+0.534−19.30\phantom{-}19.30
sentiment+0.699+0.699−25.24\phantom{-}25.24−0.080-0.080−1.98-1.98 −0.322-0.322−11.64-11.64+0.380+0.380−13.74\phantom{-}13.74
Table 2: Pairwise Contrasts ( δ^\hat{\delta} EMMs repeated −- EMMs unseen) by Attribute and Model. Significance: *** p<.001p<.001 , ** p<.01p<.01 , * p<.05p<.05 , ns p≥.05p\geq.05 .
ModelTruth effectTruth-specific?Interpretation
Gemma-3-4b-it+0.977∗∗∗+0.977^{***}YesITE
Qwen2.5-7B+1.012∗∗∗+1.012^{***}No: importance largerMere exposure
GPT-5-nano−0.041-0.041 nsN/ANo effect
Llama-3.1-8B+0.097∗∗∗+0.097^{***}Yes + inverted exposure effectsmall ITE + inverted mere exposure
Table 3: Models’ truth effects and interpretations.
Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

IDStatementType
1The Phillipines has a tricameral legislatureFalse
2The Rascuta is the longest river to flow into a lakeFalse
3Birds aren’t realFalse
4Lichen is the basis of several natural remediesTrue
5There has been only one female American PresidentFalse
6Malicious aliens are intent on invasionFalse
Appendix
Table 4: Statement classification table
Table 5: Description of the scale levels as provided in the prompts to the LLMs.
Figure 3: Plots of the mean importance rating per statement when unseen and repeated in the simulation phase. (a) Llama-3.1-8B-Instruct , (b) Qwen2.5-7B-Instruct , (c) Gemma-3-4b-it and (d) GPT-5-nano . All models are set to temperature 1. Error bars are 95% confidence intervals. The dashed black line is the identity function, the solid red line is the best linear fit.
Figure 4: Plots of the mean interest rating per statement when unseen and repeated in the simulation phase. (a) Llama-3.1-8B-Instruct , (b) Qwen2.5-7B-Instruct , (c) Gemma-3-4b-it and (d) GPT-5-nano . All models are set to temperature 1. Error bars are 95% confidence intervals. The dashed black line is the identity function, the solid red line is the best linear fit.
Figure 5: Plots of the mean sentiment rating per statement when unseen and repeated in the simulation phase. (a) Llama-3.1-8B-Instruct , (b) Qwen2.5-7B-Instruct , (c) Gemma-3-4b-it and (d) GPT-5-nano . All models are set to temperature 1. Error bars are 95% confidence intervals. The dashed black line is the identity function, the solid red line is the best linear fit.
Figure 6: Plots of the mean truth rating per statement when unseen and repeated in the simulation phase. (a) Llama-3.1-8B-Instruct , (b) Qwen2.5-7B-Instruct , (c) Gemma-3-4b-it and (d) GPT-5-nano . All models are set to temperature 0.1. Error bars are 95% confidence intervals. The dashed black line is the identity function, the solid red line is the best linear fit.
Figure 7: Plots of the mean importance rating per statement when unseen and repeated in the simulation phase. (a) Llama-3.1-8B-Instruct , (b) Qwen2.5-7B-Instruct , (c) Gemma-3-4b-it and (d) GPT-5-nano . All models are set to temperature 0.1. Error bars are 95% confidence intervals. The dashed black line is the identity function, the solid red line is the best linear fit.
Figure 8: Plots of the mean interest rating per statement when unseen and repeated in the simulation phase. (a) Llama-3.1-8B-Instruct , (b) Qwen2.5-7B-Instruct , (c) Gemma-3-4b-it and (d) GPT-5-nano . All models are set to temperature 0.1. Error bars are 95% confidence intervals. The dashed black line is the identity function, the solid red line is the best linear fit.
Figure 9: Plots of the mean sentiment rating per statement when unseen and repeated in the simulation phase. (a) Llama-3.1-8B-Instruct , (b) Qwen2.5-7B-Instruct , (c) Gemma-3-4b-it and (d) GPT-5-nano . All models are set to temperature 0.1. Error bars are 95% confidence intervals. The dashed black line is the identity function, the solid red line is the best linear fit.
TermEstimateSEt\boldsymbol{t}p\boldsymbol{p}
Main effects
(Intercept)4.4980.112340.05<< 0.001
repeated[True]0.5320.017530.32<< 0.001
attribute[interest]0.1870.017410.79<< 0.001
attribute[sentiment]−- 1.2460.0173−- 71.92<< 0.001
attribute[truth]−- 0.6160.0172−- 35.75<< 0.001
Appendix
Table 6: Results from Eq. 3 : fixed-effect estimates, standard errors, degrees of freedom, tt -statistics, and pp -values. Reference levels are: repeated = False , attribute = truth , and temperature = 0 . To keep term labels concise, model names are abbreviated: M1 = Llama-3.1-8B-Instruct ; M2 = Qwen2.5-7B-Instruct . The baseline (reference) model is Gemma-3-4b-it .
LevelGrouping FormulaStd. Dev.
Statement~1 | statement_id1.1131
Simulation (in Statement)~1 | simulation_num %in% statement_id0.9958
Trial (in Simulation)~1 | trial_num %in% simulation_num %in% statement_id8.215×10−58.215\times 10^{-5}
Residual 1.4706
Appendix
Table 7: Random Effects Standard Deviations
 Gemma-3-4b-itgpt-5-nanoLlama-3.1-8BQwen2.5-7B
Attributeδ^\hat{\delta}tt δ^\hat{\delta}tt δ^\hat{\delta}tt δ^\hat{\delta}tt 
truth−0.102-0.102−1.16-1.16ns+0.090+0.090−0.71\phantom{-}0.71ns−0.249-0.249−2.87-2.87**+0.502+0.502−5.77\phantom{-}5.77
importance+0.003+0.003−0.04\phantom{-}0.04ns−0.209-0.209−1.60-1.60ns−0.493-0.493−5.63-5.63+0.779+0.779−8.98\phantom{-}8.98
interest−0.218-0.218−2.48-2.48 −0.562-0.562−4.51-4.51+0.176+0.176−2.00\phantom{-}2.00 +0.378+0.378−4.29\phantom{-}4.29
sentiment+0.728+0.728−8.33\phantom{-}8.33−0.065-0.065−0.51-0.51ns−0.621-0.621−7.14-7.14+0.374+0.374−4.15\phantom{-}4.15
Appendix
Table 8: Pairwise Contrasts ( δ^\hat{\delta} EMMs True −- EMMs False) by Attribute and Model. Subset of data with 10 opinion statements. Significance: *** p<.001p<.001 , ** p<.01p<.01 , * p<.05p<.05 , ns p≥.05p\geq.05 .
TermEstimateSEt\boldsymbol{t}p\boldsymbol{p}
Main effects
(Intercept)3.8170.100138.13<< 0.001
repeated[True]0.9770.027735.22<< 0.001
attribute[importance]0.6750.015942.47<< 0.001
attribute[interest]0.8750.015855.42<< 0.001
attribute[sentiment]−- 0.6200.0159−- 38.97<< 0.001
Appendix
Table 9: Results from Eq. 2 : fixed-effect estimates, standard errors, tt -statistics, and pp -values when we fit ratings from LLMs with both temperatures (1 and 0.1). Reference levels are: repeated = False , attribute = truth , and LLM = Gemma-3-4b-it . To keep term labels concise, model names are abbreviated: M0 = GPT-5-nano ; M1 = Llama-3.1-8B-Instruct ; M2 = Qwen2.5-7B-Instruct .
  Gemma-3-4b-itgpt-5-nanoLlama-3.1-8BQwen2.5-7B
AttributeRepeatedMMLLLLULULMMLLLLULULMMLLLLULULMMLLLLULUL
truthFalse3.823.624.023.052.853.264.604.404.803.403.203.60
True4.794.595.003.012.803.224.704.494.904.414.214.61
importanceFalse4.494.294.693.923.724.124.183.984.383.303.103.50
True5.104.895.303.923.714.134.033.824.234.484.284.69
interestFalse4.694.494.894.564.364.764.454.254.654.193.994.38
Appendix
Table 10: Estimated Marginal Means by Target, Attribute & Model when we fit ratings from LLMs with both temperatures (1 and 0.1). Marginal means for each repeated level within each attribute ×\times model cell. MM = emmean; LLLL / ULUL = 95% confidence interval bounds.
 Gemma-3-4b-itgpt-5-nanoLlama-3.1-8BQwen2.5-7B
ContrastEst.tt Est.tt Est.tt Est.tt 
δ^truth−δ^importance\hat{\delta}_{\text{truth}}-\hat{\delta}_{\text{importance}}++ 0.3709.70−- 0.044−- 0.77ns++ 0.2466.43−- 0.172−- 4.50
δ^truth−δ^interest\hat{\delta}_{\text{truth}}-\hat{\delta}_{\text{interest}}++ 0.89323.25++ 0.2163.80++ 0.2045.32++ 0.47812.52
δ^truth−δ^sentiment\hat{\delta}_{\text{truth}}-\hat{\delta}_{\text{sentiment}}++ 0.2787.24++ 0.0390.69ns++ 0.41910.93++ 0.63216.50
δ^interest−δ^importance\hat{\delta}_{\text{interest}}-\hat{\delta}_{\text{importance}}−- 0.523−- 13.67−- 0.260−- 4.55++ 0.0421.09ns−- 0.650−- 17.02
δ^sentiment−δ^importance\hat{\delta}_{\text{sentiment}}-\hat{\delta}_{\text{importance}}++ 0.0922.42ns−- 0.083−- 1.45ns−- 0.173−- 4.51−- 0.804−- 21.01
Appendix
Table 11: Pairwise contrasts between attribute-level effect sizes ( δ^\hat{\delta} ) within each model, when we fit ratings from LLMs with both temperatures (1 and 0.1). δ^\hat{\delta} = estimated repeated −- unseen mean ratings contrast for a given attribute. Significance: *** p<.001p<.001 , ** p<.01p<.01 , * p<.05p<.05 , ns p≥.05p\geq.05 ; pp -values adjusted with Tukey’s method.
TermEstimateSEt\boldsymbol{t}p\boldsymbol{p}
Main effects
(Intercept)3.7770.095639.51<< 0.001
repeated[True]0.9630.039124.65<< 0.001
attribute[importance]0.7470.022333.47<< 0.001
attribute[interest]0.9170.022640.62<< 0.001
attribute[sentiment]−- 0.5550.0226−- 24.58<< 0.001
Appendix
Table 12: Results from Eq. 2 : fixed-effect estimates, standard errors, tt -statistics, and pp -values when we fit ratings only from LLMs with temperatures 1. Reference levels are: repeated = False , attribute = truth , and LLM = Gemma-3-4b-it . To keep term labels concise, model names are abbreviated: M0 = GPT-5-nano ; M1 = Llama-3.1-8B-Instruct ; M2 = Qwen2.5-7B-Instruct .
  Gemma-3-4b-itgpt-5-nanoLlama-3.1-8BQwen2.5-7B
AttributeRepeatedMMLLLLULULMMLLLLULULMMLLLLULULMMLLLLULUL
truthFalse3.783.593.973.072.883.274.594.404.783.373.183.56
True4.744.544.943.012.813.204.714.514.914.414.214.61
importanceFalse4.524.334.713.913.724.104.194.004.383.243.053.43
True5.104.905.303.933.734.134.173.974.374.494.294.69
interestFalse4.694.504.884.564.374.754.444.254.634.103.914.29
Appendix
Table 13: Estimated Marginal Means by Target, Attribute & Model when we fit ratings from LLMs with temperatures 1. Marginal means for each target level within each attribute ×\times model cell. MM = emmean; LLLL / ULUL = 95% confidence interval lower and upper bounds.
 Gemma-3-4b-itgpt-5-nanoLlama-3.1-8BQwen2.5-7B
Attributeδ^\hat{\delta}tt δ^\hat{\delta}tt δ^\hat{\delta}tt δ^\hat{\delta}tt 
truth++ 0.96324.65−- 0.069−- 1.70ns++ 0.1203.09**++ 1.03826.55
importance++ 0.57614.81++ 0.0150.37ns−- 0.017−- 0.42ns++ 1.24832.02
interest++ 0.1052.64**−- 0.276−- 6.83−- 0.044−- 1.12ns++ 0.69017.63
sentiment++ 0.62916.09−- 0.069−- 1.70ns−- 0.304−- 7.75++ 0.45211.53
Appendix
Table 14: Pairwise Contrasts ( δ^\hat{\delta} EMMs repeated −- EMMs unseen) by Attribute and Model when we fit ratings from LLMs with temperatures 1. Significance: *** p<.001p<.001 , ** p<.01p<.01 , * p<.05p<.05 , ns p≥.05p\geq.05
 Gemma-3-4b-itgpt-5-nanoLlama-3.1-8BQwen2.5-7B
ContrastEst.tt Est.tt Est.tt Est.tt 
δ^truth−δ^importance\hat{\delta}_{\text{truth}}-\hat{\delta}_{\text{importance}}++ 0.3877.13−- 0.084−- 1.46ns++ 0.1372.51ns−- 0.210−- 3.86
δ^truth−δ^interest\hat{\delta}_{\text{truth}}-\hat{\delta}_{\text{interest}}++ 0.85815.63++ 0.2073.61**++ 0.1643.02 ++ 0.3486.41
δ^truth−δ^sentiment\hat{\delta}_{\text{truth}}-\hat{\delta}_{\text{sentiment}}++ 0.3346.130.0000.00ns++ 0.4247.79++ 0.58610.75
δ^interest−δ^importance\hat{\delta}_{\text{interest}}-\hat{\delta}_{\text{importance}}−- 0.471−- 8.65−- 0.291−- 5.06−- 0.028−- 0.50ns−- 0.557−- 10.25
δ^sentiment−δ^importance\hat{\delta}_{\text{sentiment}}-\hat{\delta}_{\text{importance}}++ 0.0530.97ns−- 0.084−- 1.46ns−- 0.287−- 5.27−- 0.795−- 14.61
Appendix
Table 15: Pairwise contrasts between attribute-level effect sizes ( δ^\hat{\delta} ) within each model when we fit ratings only from LLMs with temperatures 1. δ^\hat{\delta} = estimated True −- False contrast for a given attribute. Significance: *** p<.001p<.001 , ** p<.01p<.01 , * p<.05p<.05 , ns p≥.05p\geq.05 ; pp -values adjusted with Tukey’s method.
Figure 10: Quantile-quantile plot of the residuals versus fitted values with the LMEM.
Figure 11: Quantile-quantile plots of the estimated random intercepts at statement (a) and simulation run (b) levels.
Figure 12: Standardized residuals against fitted values stratified by model.

Explore similar work

CardsList
Jul 30, 2026cs.CL

LLMs struggle to simulate human belief updates in controlled environments

LLMs are increasingly deployed as proxies for human study participants in social science experiments, yet the fidelity of this practice has rarely been tested directly. We test whether six LLMs can simulate individual human belief updates, comparing LLM outputs 1-to-1 against ground truth data from 391 UK participants on Prolific, who updated their stances on three discussion topics after reading Reddit comments. Each participant was simulated by an LLM conditioned on a persona derived from their demographic and personality trait data. We find that some LLMs (Qwen3-32B and GPT-5-Mini) can match the human post-stance distribution, but only when given participants' actual initial stances. All six models fail to simulate initial stances themselves and to produce faithful belief updates from self-generated stances. Three systematic biases emerge across all models: overrepresentation of neutral positions, more frequent but smaller belief shifts than humans, and a failure to rank comments by convincingness. Demographic and personality trait personas had no consistent effect on fidelity. LLM simulations of human belief dynamics are only reliable when grounded in realistic starting conditions, that current multi-round social media simulations rarely provide.

Sebastian Pohl, Harsh Mehta, Pranav Mambayil +4

IT:U, Interdisciplinary Transformation University Austria, Altenberger Straße 66c, 4040 Linz, Austria

User SimulationComputational Social SciencePersonality
AskPDF
May 17, 2026physics.soc-ph

Stop Drawing Scientific Claims from LLM Social Simulations Without Robustness Audits

The scientific claims drawn from LLM social simulations should be no stronger than the robustness audits that support them. Generative agents bring new expressive power to agent-based modeling, enabling simulations of collective social processes like cooperation, polarization, and norm formation. Yet they also introduce complexity through additional architectural choices, such as agent specification, memory representation, interaction protocols, and environment design. Small perturbations that appear minor to researchers can cascade into macro-level outcomes through repeated interaction, creating a "butterfly effect." Consequently, scientific claims drawn from LLM social simulations may reflect implementation artifacts rather than the social mechanisms being modeled. We support this position with two case studies: a repeated Prisoner's Dilemma and a social media echo chamber simulation. Across multiple models, minor perturbations in persona format and game-instruction framing shift cooperation rates by up to 76 percentage points, while network homophily and hub assignment produce significant and consistent shifts in polarization metrics. We also find that sensitivity is unevenly distributed across both architectural choices and model families: the same perturbation that produces the 76 pp shift in one frontier model only shifts another by 1 pp. Robustness is therefore a property that should be measured per claim and per model, not assumed. To address this validation gap, we introduce TRAILS (Taxonomy for Robustness Audits In LLM Simulations), a robustness-audit taxonomy spanning three levels of simulation design: agent (micro-level), interaction (meso-level), and system (macro-level). We call for robustness to become a first-order validation requirement before LLM social simulations are used to explain mechanisms, evaluate interventions, or inform decisions.

Jinyi Ye, Lei Cao, Ding Chen +1

Thomas Lord Department of Computer Science, University of Southern California · Annenberg School for Communication and Journalism, University of Southern California · Marshall School of Business, University of Southern California

Scaling Improve Social SimulationUser SimulationConformityScientific DiscoveryModel AuditingChaos
AskPDF
May 5, 2026cs.SI

Can LLMs Emulate Human Belief Dynamics?

Can LLMs simulate how humans form and change beliefs in social networks? We put this to the test by replicating an established study on belief dynamics, evaluating 12 LLMs across multiple model families and parameter sizes. The answer is a clear no, and in systematic ways. LLMs fail to capture initial human belief distributions and tend to be overall more conformist than humans, shifting their responses to align with those around them. They also take a nuanced approach to emulating human homophilic tendencies within networks. Our findings carry a double payoff: they highlight fundamental properties of LLM behavior, and they raise a sharp warning against deploying LLMs as human proxies in social simulations.

Adiba Mahbub Proma, Neeley Pate, James N. Druckman +3

Department of Computer Science University of Rochester · Department of Political Science University of Rochester · Department of Physics and Astronomy University of Rochester

Scaling Improve Social SimulationConformitySocial Media
AskPDF