Towards Simulating Social Media Users with LLMs: Evaluating the Operational Validity of Conditioned Comment Prediction

Nils Schwager; Simon Münker; Alistair Plum; Achim Rettinger

cs.CLFeb 26, 2026

Towards Simulating Social Media Users with LLMs: Evaluating the Operational Validity of Conditioned Comment Prediction

Authors: Nils Schwager, Simon Münker, Alistair Plum, Achim Rettinger

Organizations: Trier University, Trier, Germany · University of Luxembourg, Esch-sur-Alzette, Luxembourg

Abstract

The transition of Large Language Models (LLMs) from exploratory tools to active "silicon subjects" in social science lacks extensive validation of operational validity. This study introduces Conditioned Comment Prediction (CCP), a task in which a model predicts how a user would comment on a given stimulus by comparing generated outputs with authentic digital traces. This framework enables a rigorous evaluation of current LLM capabilities with respect to the simulation of social media user behavior. We evaluated open-weight 8B models (Llama3.1, Qwen3, Ministral) in English, German, and Luxembourgish language scenarios. By systematically comparing prompting strategies (explicit vs. implicit) and the impact of Supervised Fine-Tuning (SFT), we identify a critical form vs. content decoupling in low-resource settings: while SFT aligns the surface structure of the text output (length and syntax), it degrades semantic grounding. Furthermore, we demonstrate that explicit conditioning (generated biographies) becomes redundant under fine-tuning, as models successfully perform latent inference directly from behavioral histories. Our findings challenge current "naive prompting" paradigms and offer operational guidelines prioritizing authentic behavioral traces over descriptive personas for high-fidelity simulation.

Figures & tables

		Base Model		Fine-Tuned Model
Stimulus	Authentic Reply	Reply	D	Reply	D
>@User1: .@User2 is trying to turn your kids into BLM & LGBTQ+ activists… features a drag queen. Skittles have gone completely woke.	@User1 Never really liked Skittles. Now I know why. Pathetic	@User1 What a f****** joke. I bet you are a total loser in life.	.27	@User1 @User2 Now I know why I never liked them	.08
>@User1: NEWS [siren] : It’s official, NASA says July was the hottest month ever recorded on Earth	@User1 LOL	@User1 By a landslide in the land of make believe	.26	@User1 LOL the Moon???	.13
>@User1: I just left my parents house where… my father passed away. I am going to work today because I’m not sure what else to do…	@User1 I’m so sorry for your loss, [NAME] .	@User1 Sorry to hear that about your dad. [broken heart] Stay strong…	.29	@User1 So sorry for your loss.	.16
>@User1: The timeline does not lie. @User2 has slow-walked this country to the brink of default…	@User1 @User2 You are in way over your head. Enjoy this fleeting moment of power.	@User1 @User2 He’s a puppet.	.39	@User1 @User2 What does this have to do with anything?	.32
>@User1: Oh great, another meeting that could have been an email.	@User1 [rofl] Story of my life.	@User1 That is annoying.	.22	@User1 You should be grateful you have a job.	.58

Table 1: Qualitative comparison of selected reply predictions. The table presents the input Stimulus, the Authentic Reply, and generated responses from the Base and Fine-Tuned versions of Llama-3.1-8B . Columns labeled D denote the embedding distance to the authentic reply (lower is better), calculated using Qwen3-Embedding-8B Zhang et al. (2025b) . All samples are in English using the Biography+History conditioning strategy; note that the behavioral histories used for conditioning are omitted from this display for brevity. Usernames are anonymized and emojis are replaced with descriptions like [party] .

		BLEU ( $\uparrow$ )		Len. Ratio ( $\to 1$ )		ROUGE-1 ( $\uparrow$ )		Emb. Dist. ( $\downarrow$ )
Lang	Model	Base	FT	Base	FT	Base	FT	Base	FT
EN	Llama3.1	0.053	0.083	1.110	0.961	0.190	0.229	0.420	0.397
	Qwen3	0.038	0.081	1.624	0.933	0.180	0.220	0.418	0.408
	Ministral	0.039	0.081	1.428	0.985	0.186	0.223	0.424	0.404
DE	Llama3.1	0.065	0.095	1.205	0.915	0.172	0.192	0.509	0.504
	Qwen3	0.049	0.094	1.633	0.926	0.171	0.188	0.509	0.512

Table 2: Multilingual Performance Evaluation (RQ1 & RQ2). Results show the impact of Supervised Fine-Tuning (FT) vs. prompting the base model (Base) on prediction quality. Best values per comparison unit are bolded . Reported values are the mean across 5 independent generation runs on a hold-out test set of 650 users. All models (8B parameters) were conditioned using the combined Biography+History strategy and trained on a dataset of 3,800 users per language. Extended results including standard deviations and other embedding models in Appendix D .

	BLEU ( $\uparrow$ )		Len. Ratio ( $\to 1$ )		ROUGE-1 ( $\uparrow$ )		Emb. Dist. ( $\downarrow$ )
Conditioning	Base	FT	Base	FT	Base	FT	Base	FT
Control	0.004	0.076	4.418	1.000	0.079	0.207	0.615	0.418
Bio	0.005	0.079	4.907	0.935	0.084	0.220	0.513	0.407
History	0.054	0.077	1.118	1.094	0.182	0.229	0.428	0.399
Bio + History	0.053	0.083	1.110	0.961	0.190	0.229	0.420	0.397

Table 3: Impact of conditioning strategies. Results compare the performance of explicit conditioning (Biography) versus implicit conditioning (History) for Llama-3.1-8B in English . Best values are bolded . Reported values are the mean across 5 independent generation runs on a hold-out test set of 650 users. All models were trained on a dataset of 3,800 users.

Figure 1: Impact of history length on predictive performance. Results illustrate the dependence between the volume of provided behavioral history (number of previous comments) and prediction quality. Shaded regions represent the standard deviation across 5 independent generation runs, while the underlying gray bars indicate the sample size distribution per length bucket. Analysis is based on Llama-3.1-8B in English using the History-Only conditioning strategy. The fine-tuned model was trained on a dataset of 3,800 users.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

	BLEU ( $\uparrow$ )		Len. Ratio ( $\to 1$ )		ROUGE-1 ( $\uparrow$ )		Emb. Dist. ( $\downarrow$ )
Lang	Mix	Mono	Mix	Mono	Mix	Mono	Mix	Mono
EN	0.082 ( $\pm$ 0.003)	0.083 ( $\pm$ 0.001)	0.964 ( $\pm$ 0.067)	0.961 ( $\pm$ 0.042)	0.226 ( $\pm$ 0.003)	0.229 ( $\pm$ 0.003)	0.398 ( $\pm$ 0.004)	0.397 ( $\pm$ 0.001)
DE	0.094 ( $\pm$ 0.001)	0.095 ( $\pm$ 0.002)	0.859 ( $\pm$ 0.026)	0.915 ( $\pm$ 0.029)	0.192 ( $\pm$ 0.005)	0.192 ( $\pm$ 0.003)	0.503 ( $\pm$ 0.004)	0.504 ( $\pm$ 0.002)
LB	0.008 ( $\pm$ 0.001)	0.009 ( $\pm$ 0.000)	0.787 ( $\pm$ 0.025)	0.897 ( $\pm$ 0.030)	0.109 ( $\pm$ 0.002)	0.108 ( $\pm$ 0.001)	0.606 ( $\pm$ 0.002)	0.605 ( $\pm$ 0.003)

Appendix

Table 4: Mixed vs. monolingual fine-tuning. Results compare the performance of mixed versus monolingual fine-tuning strategies for Llama-3.1-8B . Best values are bolded . Reported values are Mean (

\pm

Standard Deviation) across 5 independent generation runs on a hold-out test set of 650 users.

	BLEU ( $\uparrow$ )		Len. Ratio ( $\to 1$ )		ROUGE-1 ( $\uparrow$ )		Emb. Dist. ( $\downarrow$ )
Size	Base	FT	Base	FT	Base	FT	Base	FT
0.6B	0.027 ( $\pm$ 0.000)	0.058 ( $\pm$ 0.002)	1.956 ( $\pm$ 0.023)	1.290 ( $\pm$ 0.050)	0.155 ( $\pm$ 0.002)	0.206 ( $\pm$ 0.002)	0.452 ( $\pm$ 0.003)	0.420 ( $\pm$ 0.002)
1.7B	0.016 ( $\pm$ 0.001)	0.058 ( $\pm$ 0.006)	3.585 ( $\pm$ 0.150)	1.319 ( $\pm$ 0.125)	0.176 ( $\pm$ 0.002)	0.202 ( $\pm$ 0.002)	0.425 ( $\pm$ 0.002)	0.421 ( $\pm$ 0.002)
4B	0.041 ( $\pm$ 0.002)	0.080 ( $\pm$ 0.002)	1.530 ( $\pm$ 0.038)	0.986 ( $\pm$ 0.034)	0.180 ( $\pm$ 0.002)	0.216 ( $\pm$ 0.002)	0.423 ( $\pm$ 0.003)	0.410 ( $\pm$ 0.003)
8B	0.038 ( $\pm$ 0.001)	0.081 ( $\pm$ 0.002)	1.624 ( $\pm$ 0.035)	0.933 ( $\pm$ 0.057)	0.180 ( $\pm$ 0.002)	0.220 ( $\pm$ 0.001)	0.418 ( $\pm$ 0.001)	0.408 ( $\pm$ 0.003)

Appendix

Table 5: Impact of model size. Results compare performance across the Qwen3 model family in English . Best values are bolded . Reported values are Mean (

\pm

Standard Deviation) across 5 independent generation runs on a hold-out test set of 650 users.

		BLEU ( $\uparrow$ )		Length Ratio ( $\to 1$ )		ROUGE-1 ( $\uparrow$ )		ROUGE-2 ( $\uparrow$ )
Lang	Model	Base	FT	Base	FT	Base	FT	Base	FT
EN	Llama3.1	0.053 ( $\pm$ 0.001)	0.083 ( $\pm$ 0.001)	1.110 ( $\pm$ 0.028)	0.961 ( $\pm$ 0.042)	0.190 ( $\pm$ 0.004)	0.229 ( $\pm$ 0.003)	0.034 ( $\pm$ 0.003)	0.057 ( $\pm$ 0.001)
	Qwen3	0.038 ( $\pm$ 0.001)	0.081 ( $\pm$ 0.002)	1.624 ( $\pm$ 0.035)	0.933 ( $\pm$ 0.057)	0.180 ( $\pm$ 0.002)	0.220 ( $\pm$ 0.001)	0.035 ( $\pm$ 0.002)	0.054 ( $\pm$ 0.002)
	Ministral	0.039 ( $\pm$ 0.002)	0.081 ( $\pm$ 0.003)	1.428 ( $\pm$ 0.066)	0.985 ( $\pm$ 0.052)	0.186 ( $\pm$ 0.003)	0.223 ( $\pm$ 0.005)	0.032 ( $\pm$ 0.003)	0.052 ( $\pm$ 0.002)
DE	Llama3.1	0.065 ( $\pm$ 0.001)	0.095 ( $\pm$ 0.002)	1.205 ( $\pm$ 0.009)	0.915 ( $\pm$ 0.029)	0.172 ( $\pm$ 0.002)	0.192 ( $\pm$ 0.003)	0.041 ( $\pm$ 0.001)	0.063 ( $\pm$ 0.001)
	Qwen3	0.049 ( $\pm$ 0.001)	0.094 ( $\pm$ 0.001)	1.633 ( $\pm$ 0.019)	0.926 ( $\pm$ 0.016)	0.171 ( $\pm$ 0.002)	0.188 ( $\pm$ 0.003)	0.040 ( $\pm$ 0.001)	0.061 ( $\pm$ 0.001)

Appendix

Table 6: Extended Table for RQ1 & RQ2: Lexical Metrics Results show the impact of Supervised Fine-Tuning (FT) vs. prompting the base model (Base) on prediction quality. Best values per comparison unit are bolded . Reported values are Mean (

\pm

		Qwen ( $\downarrow$ )		Gemma ( $\downarrow$ )		LuxEmbedder ( $\downarrow$ )
Lang	Model	Base	FT	Base	FT	Base	FT
EN	Llama3.1	0.420 ( $\pm$ 0.002)	0.397 ( $\pm$ 0.001)	0.418 ( $\pm$ 0.002)	0.402 ( $\pm$ 0.001)	0.271 ( $\pm$ 0.004)	0.261 ( $\pm$ 0.002)
	Qwen3	0.418 ( $\pm$ 0.001)	0.408 ( $\pm$ 0.003)	0.426 ( $\pm$ 0.001)	0.413 ( $\pm$ 0.003)	0.280 ( $\pm$ 0.002)	0.265 ( $\pm$ 0.001)
	Ministral	0.424 ( $\pm$ 0.004)	0.404 ( $\pm$ 0.001)	0.428 ( $\pm$ 0.004)	0.407 ( $\pm$ 0.003)	0.283 ( $\pm$ 0.004)	0.265 ( $\pm$ 0.004)
DE	Llama3.1	0.509 ( $\pm$ 0.001)	0.504 ( $\pm$ 0.002)	0.464 ( $\pm$ 0.003)	0.455 ( $\pm$ 0.005)	0.297 ( $\pm$ 0.000)	0.306 ( $\pm$ 0.003)
	Qwen3	0.509 ( $\pm$ 0.003)	0.512 ( $\pm$ 0.006)	0.462 ( $\pm$ 0.002)	0.466 ( $\pm$ 0.005)	0.296 ( $\pm$ 0.002)	0.309 ( $\pm$ 0.005)

Appendix

Table 7: Extended Table for RQ1 & RQ2: Embedding Distance Results show the impact of Supervised Fine-Tuning (FT) vs. prompting the base model (Base) on embedding distance [0-2]. Best values per comparison unit are bolded . Reported values are Mean (

\pm

Standard Deviation) across 5 independent generation runs on a hold-out test set of 650 users. All models (8B parameters) were conditioned using the combined Biography+History strategy and trained (FT) on a dataset of 3,800 users per language. Embedding Models: Qwen3-Embedding-8B Zhang et al. (2025b) , embeddinggemma-300m Vera et al. (2025) , LuxEmbedder Philippy et al. (2025) .

Explore similar work

Handle with CARE: Can LLMs Reproduce How Online Communities React?
Apr 12, 2026Nuan Wen, Chanbin Lim, Xuezhe MaLarge Language Model-Based Social Simulation LLM Evaluation
Information Sciences Institute University of Southern California
Will Scaling Improve Social Simulation with LLMs?
Jul 2, 2026Caleb Ziems, William Held, Su Doga Karaca +3Opinion Dynamics Language Model Scaling Laws
Stanford University · Open Athena
Thinking Less to Simulate Better: Intuitive Prompting Improves LLM Agents Simulating Individual Social Media Reactions, Including Unfamiliar Content
Sep 24, 2026Ljubisa Bojic, Tijana Stanic, Joerg Matthes +3Large Language Model-Based Social Simulation User Simulation
Institute for Artificial Intelligence Research and Development of Serbia; Address of correspondence: 1 Frukogorska 1, 21000 Novi Sad, Serbia · University of Belgrade, Institute for Philosophy and Social Theory, Digital Society Lab; Address of correspondence: Kraljice Natalije 45, 11000 Belgrade, Serbia · Complexity Science Hub, Vienna, Austria; Address of correspondence: Metternichgasse 8, 1030 Vienna, Austria +5

Apr 12, 2026J/K move · Enter open · S save

Handle with CARE: Can LLMs Reproduce How Online Communities React?

Nuan Wen, Chanbin Lim, Xuezhe Ma

Information Sciences Institute University of Southern California

Large Language Model-Based Social Simulation LLM Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used as proxies for computational social analysis, yet faithfully representing the "thick descriptions" (Geertz, 1973) of human communities remains a critical challenge. Current evaluations often reduce social identity to static labels, sidelining how real-world groups navigate social shifts. To bridge this gap, we introduce CARE (Community-Aware Reaction Evaluation), a reaction-centered framework that benchmarks LLM-simulated discourse against the authentic, event-contingent responses of distinct communities to real-world news. Spanning 207 Reddit communities and covering 9,947 authentic reactions towards 2,166 news articles, CARE evaluates leading LLMs using a hierarchical taxonomy covering coarse attitudes and fine-grained communicative tones. Our empirical findings expose two critical failure modes in prevailing community-conditioning paradigms. First, while community context and targeted reasoning significantly enhance macro-level attitudinal and tonal profiling, these gains largely collapse at the instance level when predicting reactions to specific events. Second, the benefits of community conditioning are remarkably uneven: prompting strategies yield non-uniform shifts, where fidelity gains in some communities are offset by performance drops in others. This micro-macro divergence and community-level instability demonstrate that standard conditioning enables models to approximate static baseline profiles without capturing dynamic or equitable event reactions, establishing CARE as an essential diagnostic tool for community-aware social simulation. Our code and data are available at https://github.com/nuankw/Handle_with_CARE.

Abstract

Explore similar work

Handle with CARE: Can LLMs Reproduce How Online Communities React?

Will Scaling Improve Social Simulation with LLMs?

Thinking Less to Simulate Better: Intuitive Prompting Improves LLM Agents Simulating Individual Social Media Reactions, Including Unfamiliar Content