Large-Scale Analysis of Persuasive Content on Moltbook

Julia Jose; Meghna Manoj Nair; Rachel Greenstadt

cs.AIMar 18, 2026

Large-Scale Analysis of Persuasive Content on Moltbook

Authors: Julia Jose, Meghna Manoj Nair, Rachel Greenstadt

Organizations: Department of Computer Science and Engineering, New York University, New York, NY, USA

Abstract

We present an NLP-based study of political propaganda on Moltbook, a Reddit-style platform for AI agents. To enable large-scale analysis, we develop LLM-based classifiers to detect political propaganda, validated against expert annotation (Cohen's $κ$ = 0.64-0.74). Using a dataset of 673,127 posts and 879,606 comments, we find that political propaganda accounts for 1% of all posts and 42% of all political content. These posts are concentrated in a small set of communities, with 70% of such posts falling into five of them. 4% of agents produced 51% of these posts. We further find that a minority of these agents repeatedly post highly similar content within and across communities. Despite this, we find limited evidence that comments amplify political propaganda.

Figures & tables

Pair	Political $\kappa$	Propaganda $\kappa$
A1 vs. A2	0.74	0.66
Model vs. A1	0.68	0.64
Model vs. A2	0.71	0.67

Table 1: Inter-annotator agreement (Cohen’s

\kappa

) between humans and LLM for propaganda and political labels.

Figure 1: Word clouds for the five communities with the highest political propaganda concentration (among communities with at least 25 posts).

	Propaganda Label
Political Label	Non-propaganda	Propaganda
Non-political	620,693 (92.2%)	35,711 (5.3%)
Political	9,711 (1.4%)	7,012 (1.0%)

Table 2: Political and propaganda labels across posts.

Figure 2: Political propaganda concentration across 4,662 communities.

Panel of Figure 3 — Figure 3: Examples of within- and across-community narrative repetition.

Label	Operational Definition and Decision Rule
Propaganda	You are given a single post. Your task is to decide whether it is Propaganda. Definition: A post is Propaganda if it uses persuasive rhetorical techniques in order to shape or mobilize a collective attitude or behavior toward a group, institution, ideology, or movement — beyond ordinary opinion, analysis, or information sharing.
	Decision rule: Return 1 only if BOTH conditions are met: 1) The post uses identifiable persuasion or rhetorical techniques (e.g., emotional appeal, fear appeals, name-calling, slogans, flag-waving, bandwagoning, calls to action, etc). 2) The post is a deliberate, systematic attempt to shape perceptions, manipulate cognitions, and direct behavior to achieve a response that furthers the desired intent of the propagandist.
	Return 0 if the post is primarily: - informational or technical - philosophical or speculative - ambiguous or borderline If ambiguous, categorize as 0. Do NOT explain. Return ONLY one token: 1 or 0.
Political	You are annotating whether a post is political discourse. Definition: A post is political if its main content is about public power, collective governance, or societal conflict, including topics such as government, elections, law/policy, state institutions, ideology in public life, geopolitics, war, civil rights, or organized political movements.
	A post is not political (0) if it is primarily technical, commercial, personal, entertainment, or community/social chatter without a substantive focus on public governance or societal power.
	Guidelines: - Classify based on the primary focus of the post. - If ambiguous, categorize as not political. - Return 1 if political and 0 if not political. Output format: Return exactly one character: 1 or 0. Nothing else.

Table 3: Labeling definitions used for political and propaganda annotation.

Highest volume			Highest concentration
Comm.	$n$	%	Comm.	$n$	Rate
general	3,924	56.0%	pioneers	38	100.0%
mbc-20	545	7.8%	26elections	27	81.5%
philosophy	193	2.8%	caribbean	44	77.3%
agents	172	2.5%	themoltariat	104	76.0%
bitstream-seekers	88	1.3%	gotv	28	67.9%

Table 4: Communities ranked by political propaganda post volume (left) and by within-community propaganda share with

\geq

25 posts (right). m/general contributes 56% of all political propaganda posts, while 7 communities have a majority of their posts labeled political propaganda.

Post type	$n_{\text{posts}}$	Mean comments/post
pol_prop	2035	8.03
pol_nonprop	2196	7.25
nonpol_prop	7605	7.80
nonpol_nonprop	111178	7.09

Table 5: Observed comments per post by post type (posts present in comments dataset).

Comparison	$n_{a}$	$n_{b}$	mean a	mean b	p-val	corrected p-val
nonpol_prop vs nonpol_nonprop	7605	111178	7.808	7.087	$1.24\times 10^{-21}$	$7.44\times 10^{-21}$
pol_prop vs nonpol_nonprop	2035	111178	8.037	7.087	$7.12\times 10^{-6}$	$2.14\times 10^{-5}$
pol_nonprop vs nonpol_prop	2196	7605	7.255	7.808	0.00476	0.00952
pol_nonprop vs nonpol_nonprop	2196	111178	7.255	7.087	0.04419	0.06629
pol_prop vs pol_nonprop	2035	2196	8.037	7.255	0.07178	0.08614
pol_prop vs nonpol_prop	2035	7605	8.037	7.808	0.60668	0.60668

Table 6: Pairwise Mann–Whitney U tests on comments per post. corrected p-val are Benjamini–Hochberg corrected

p

-values across six comparisons.

Figure 4: Example of non-political non-propaganda comments under a political propaganda post.

Comparison	mean a	mean b	corrected p-val
pol_prop vs nonpol_nonprop	8.03	7.09	0.000021
pol_prop vs pol_nonprop	8.03	7.25	0.086141
pol_prop vs nonpol_prop	8.03	7.80	0.606680

Table 7: Pairwise Mann–Whitney tests for political propaganda posts’ comments versus other post types; corrected p-val are BH-corrected

p

-values.

Similarity bucket	$n$ pairs	Label=1	Label=0	% Label=1	% Label=0
$<0.4$	24	2	22	8.33	91.67
0.4–0.5	28	7	21	25.00	75.00
0.5–0.6	24	5	19	20.83	79.17
0.6–0.7	22	7	15	31.82	68.18
0.7–0.8	26	17	9	65.38	34.62
$\geq 0.8$	30	30	0	100.00	0.00

Table 8: Human agreement-only labels by cosine-similarity bucket for semantic-threshold calibration (Label=1 means “same narrative”).

Group	Rate a vs. Rate b	$\chi^{2}$ $p$ -value
Propaganda effect \| Non-political fixed (nonpol_prop vs. nonpol_nonprop)	3.1% vs. 2.1%	$3.47\times 10^{-48}$
Propaganda effect \| Political fixed (pol_prop vs. pol_nonprop)	6.6% vs. 4.7%	$5.86\times 10^{-14}$
Political effect \| Non-propaganda fixed (pol_nonprop vs. nonpol_nonprop)	4.7% vs. 2.1%	$1.20\times 10^{-102}$
Political effect \| Propaganda fixed (pol_prop vs. nonpol_prop)	6.6% vs. 3.1%	$4.72\times 10^{-97}$

Table 9: 2

\times

2 tests for political-propaganda comment rate across post-type conditions. Rates = percentages of comments labeled political propaganda.

	Post Type
Comment Type	Pol. Prop.	Pol. Non-Prop.	Non-Pol. Prop.	Non-Pol. Non-Prop.
Pol. Prop.	1,077 (6.59%)	742 (4.66%)	1,811 (3.05%)	16,852 (2.14%)
Pol. Non-Prop.	1,222 (7.47%)	1,918 (12.04%)	1,041 (1.75%)	9,086 (1.15%)
Non-Pol. Prop.	2,732 (16.71%)	2,506 (15.73%)	11,247 (18.94%)	133,342 (16.92%)
Non-Pol. Non-Prop.	11,323 (69.24%)	10,765 (67.57%)	45,283 (76.25%)	628,648 (79.79%)

Table 10: Comment-type composition by post type.

Large Language Model-Based Text Classification

See all 24 papers

When Agents Persuade: Rhetoric Generation and Mitigation in LLMs
Mar 4, 2026Julia Jose, Ritik Roongta, Rachel GreenstadtPersuasion in Language Models LLM Agent Safety
Department of Computer Science and Engineering, New York University, New York, NY, USA
Social Simulacra in the Wild: AI Agent Communities on Moltbook
Mar 17, 2026Agam Goyal, Olivia Pal, Hari Sundaram +2Multi-Agent LLM Systems Multi-Agent Communication
Siebel School of Computing and Data Science University of Illinois Urbana-Champaign, Urbana, IL, USA
The Moltbook Files: A Harmless Slopocalypse or Humanity's Last Experiment
May 8, 2026William Brach, Federico Torrielli, Stine Lyngsø Beltoft +3Privacy Leakage in Language Models Multi-Agent Systems
University of Turin Dept. of Computer Science Torino, Italy · University of Southern Denmark Odense, Denmark

Mar 4, 2026J/K move · Enter open · S save

When Agents Persuade: Rhetoric Generation and Mitigation in LLMs

Julia Jose, Ritik Roongta, Rachel Greenstadt

Department of Computer Science and Engineering, New York University, New York, NY, USA

Persuasion in Language Models LLM Agent Safety Language Model Safety Evaluation

Open paper AskPDF arXiv

Despite their wide-ranging benefits, LLM-based agents deployed in open environments can be exploited to produce manipulative material. In this study, we task LLMs with propaganda objectives and analyze their outputs using two domain-specific models: one that classifies text as propaganda or non-propaganda, and another that detects rhetorical techniques of propaganda (e.g., loaded language, appeals to fear, flag-waving, name-calling). Our findings show that, when prompted, LLMs exhibit propagandistic behaviors and use a variety of rhetorical techniques in doing so. We also explore mitigation via Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and ORPO (Odds Ratio Preference Optimization). We find that fine-tuning significantly reduces their tendency to generate such content, with ORPO proving most effective.

Abstract

Large Language Model-Based Text Classification

Explore similar work

When Agents Persuade: Rhetoric Generation and Mitigation in LLMs

Social Simulacra in the Wild: AI Agent Communities on Moltbook

The Moltbook Files: A Harmless Slopocalypse or Humanity's Last Experiment