Organizations: Università della Svizzera italiana (USI), Lugano, Switzerland · SUPSI, DTI-ISIN, Viganello, Switzerland · USC Information Sciences Institute, Marina Del Rey, CA, USA
Agent-based social media simulators offer a controlled environment to study content moderation, yet their value hinges on how faithfully they reproduce real platform dynamics. We develop a calibrated extension of SimSoM, an agent-based model of information diffusion on social networks, grounded in a real-world dataset of online vaccine discourse during the COVID-19 pandemic. Our approach replaces ad-hoc parametrisations with empirically fitted distributions, optimised via CMA-ES (Covariance Matrix Adaptation Evolution Strategy) and validated against real data across temporal, distributional, and structural dimensions. Using this validated simulator, we provide three key contributions. First, we show that the calibrated model reproduces key statistical signatures of the empirical data, including activity distributions, post/reshare ratios, and temporal patterns. Second, we apply established misinformation-spreader detection and prevention methods to both empirical and simulated data, progressively removing top-ranked users and showing that the resulting decline in low-quality content is consistent across the two. Third, comparing static (retroactive) and dynamic (in-simulation) moderation across 30 network realisations, we show that static evaluation significantly overestimates the effectiveness of user bans for the most effective detectors: when moderation is applied in real time, compensatory resharing by the remaining users dampens the expected reduction in low-quality content, so static estimates should be read as an upper bound. These findings highlight the necessity of simulation-based evaluation for content moderation policies and contribute a reusable, empirically grounded simulation framework.
Figures & tables
Metric
Real
Simulated
Dataset statistics
Total actions
819,952
973,539 ± 73,558
Total users
74,234
72,915 ± 1,605
Duration (days)
306.5
306.0 ± 0.0
Content composition
Orphan rate
0.114
0.000 ± 0.000
Table 1: Summary statistics for real and simulated datasets. Dataset statistics : total actions (posts and reshares), unique users, observation window. Content composition : orphan rate = fraction of reshares whose parent post is absent from the collected data; post / reshare fraction = share of actions that are posts vs. reshares; unique targets ratio = fraction of distinct content items ever reshared. User activity : per-user action counts (median, p90), activity inequality (Gini), burstiness [ 12 ] of inter-event times, and median response time = delay between a post and its first reshare (see the sequential-scheduling discussion in the text). Simulation values are mean ± std across 50 independent 90k-node runs.
Figure 1: Temporal comparison: 7-day rolling average of actions per user. Real series (dark), median across the 50 runs (dashed) with its 5th–95th percentile envelope (shaded band), and the single run closest to the real series by mean absolute error (solid).
Figure 2: Targeted user removal: remaining fraction of LQ reshare weight as users are removed one by one in ranked order, for real (solid) and simulated (shaded CI) data.
Figure 3: Temporal effectiveness of banning the top-5 ranked users: percentage of LQ reshares attributable to banned users at increasing days after moderation, for real (left) and simulated (right, with CI) data.
Method
ΔS
ΔD
Gap [95% CI]
p
TASH-Index
↓ 0.075
↓ 0.059
-0.016
[-0.024, -0.009]
< 0.001
Random Forest
↓ 0.074
↓ 0.066
-0.009
[-0.016, -0.002]
0.050
Repost Count
↓ 0.006
↓ 0.005
-0.000
[-0.009, +0.008]
0.715
Coordination Centrality
↓ 0.003
↓ 0.004
+0.002
[-0.011, +0.012]
0.119
Table 2: Static (S) vs. dynamic (D) moderation across 30 independent 50k-node networks (moderation at day 186, top-5 banned). Δ : change in mean daily LQ fraction ( ↓ = less misinformation). Gap = S − D with a 95% bootstrap CI over networks; a negative gap means static overestimates real-time effectiveness (an upper bound). p : Wilcoxon signed-rank test (H 0 : median gap = 0). Bold : significant at the 5% level. Δ and the gap are rounded independently; the gap, CI and p are computed on unrounded values, so column differences may differ by up to 0.001.
Figure 4: Running within-network static − dynamic gap in daily LQ fraction per ranking method (mean over the 30 networks, 95% bootstrap CI; dotted line: tmod=186 ). Referenced to each network’s pre-moderation baseline, the gap is zero before moderation and afterwards converges to the paired gap of Table 2. Negative values mean static evaluation overestimates real-time effectiveness. Only TASH-Index and Random Forest separate from zero; the other two rankers stay within the 95% CI of zero.
LLM agents are increasingly used to simulate real-world interactions, but it remains unclear whether simulated behaviors preserve the content patterns and interaction dynamics of real human behaviors. Existing evaluations often focus on individual comments or task performance, rather than whether simulated behaviors match human behavior at thread level. In this paper, we use Reddit discussions as a concrete testbed for real-world social simulation. Reddit provides public, topic-grounded, multi-party interactions with diverse opinions, experiences, and interaction patterns, making it a useful setting for evaluating whether LLM agents reproduce the distributional and structural properties of real online discussions. We introduce ThreadBench, a benchmark for Reddit discussion simulation built from 9,817 real Reddit threads spanning 8 diverse domains and 116 subdomains. ThreadBench uses statistical analyses to compare generated and real discussions across 12 metrics grouped in 4 families: uniformity, expression, tone, and interaction. We further develop Thread-Prompt-based Discussion Generation (ThreadPDG), a Planner-Writer framework that learns how a real discussion grows. The Planner produces thread- and comment-level plans from a real seed discussion, and the Writer generates comments from these plans. For each seed, we generate multiple candidate threads and combine their best branches using thread-level statistics. Across all domains and multiple LLMs, ThreadPDG achieves ThreadBench pass rates of 58-100%, compared with 0-17% for the baselines. Human evaluation with 436 annotators further shows that ThreadPDG is not distinguished from real discussions in 78.7% of pairs, outperforming existing simulation methods.
Social platforms serve as central hubs for information exchange, where user behaviors and platform interventions jointly shape opinions. However, intervention policies like recommendation and content filtering, can unintentionally amplify echo chambers and polarization, posing significant societal risks. Proactively evaluating the impact of such policies is therefore crucial. Existing approaches primarily rely on reactive online A/B testing, where risks are identified only after deployment, making risk identification delayed and costly. LLM-based social simulations offer a promising pre-deployment alternative, but current methods fall short in realistically modeling platform interventions and incorporating feedback from the platform. Bridging these gaps is essential for building actionable frameworks to assess and optimize platform policies. To this end, we propose PolicySim, an LLM-based social simulation sandbox for the proactive assessment and optimization of intervention policies. PolicySim models the bidirectional dynamics between user behavior and platform interventions through two key components: (1) a user agent module refined via supervised fine-tuning (SFT) and direct preference optimization (DPO) to achieve platform-specific behavioral realism; and (2) an adaptive intervention module that employs a contextual bandit with message passing to capture dynamic network structures. Experiments show that PolicySim can accurately simulate platform ecosystems at both micro and macro levels and support effective intervention policy.
Renhong Huang, Ning Tang, Jiarong Xu +6
Zhejiang University, Hangzhou, China · Fudan University, Shanghai, China · HKUST, Hong Kong, China +3
Large Language Model (LLM) agents offer a potentially-transformative path forward for generative social science but face a critical crisis of validity. Current simulation evaluation methodologies suffer from the "stopped clock" problem: they confirm that a simulation reached the correct final outcome while ignoring whether the trajectory leading to it was sociologically plausible. Because the internal reasoning of LLMs is opaque, verifying the "black box" of social mechanisms remains a persistent challenge. In this paper, we introduce SLALOM (Simulation Lifecycle Analysis via Longitudinal Observation Metrics), a framework that shifts validation from outcome verification to process fidelity. Drawing on Pattern-Oriented Modeling (POM), SLALOM treats social phenomena as multivariate time series that must traverse specific SLALOM gates, or intermediate waypoint constraints representing distinct phases. By utilizing Dynamic Time Warping (DTW) to align simulated trajectories with empirical ground truth, SLALOM offers a quantitative metric to assess structural realism, helping to differentiate plausible social dynamics from stochastic noise and contributing to more robust policy simulation standards.
Juhoon Lee, Joseph Seering
School of Computing, KAIST, Daejeon, Republic of Korea