Interactive systems such as chatbots and games are increasingly used to persuade and educate on sustainability-related topics, yet it remains unclear how different delivery formats shape learning and persuasive outcomes when content is held constant. Grounding on identical arguments and factual content across conditions, we present a controlled user study comparing three modes of information delivery: static essays, conversational chatbots, and narrative text-based games. Across subjective measures, the chatbot condition consistently outperformed the other modes and increased perceived importance of the topic. However, perceived learning did not reliably align with objective outcomes: participants in the text-based game condition reported learning less than those reading essays, yet achieved higher scores on a delayed (24-hour) knowledge quiz. Additional exploratory analyses further suggest that common engagement proxies, such as verbosity and interaction length, are more closely related to subjective experience than to actual learning. These findings highlight a dissociation between how persuasive experiences feel and what participants retain, and point to important design trade-offs between interactivity, realism, and learning in persuasive systems and serious games.
Figures & tables
Topic
Argument (without the premise)
Supporting fact used in the study
Recycling
(1) Environmental Preservation: Protecting Natural Resources and Ecosystems
Recycling one ton of paper saves approximately 17 trees.
(2) Energy Conservation and Lower Carbon Emissions
Recycling aluminum saves up to 95% of the energy compared to producing new aluminum.
(3) Economic Benefits: Cost Savings and Job Creation
Recycling creates about 10 jobs for every landfill job.
(4) Reducing Pollution: Cleaner Air, Water, and Soil
Unrecycled plastic breaks into microplastics that harm over 700 marine species and enter the food chain.
(5) Setting a Positive Example and Meeting Social Responsibility Goals
Companies with recycling programs report up to 20% higher employee engagement and retention.
Switching from personal cars to public transit can reduce an individual’s transportation emissions by up to 50%.
Table 1. Fixed set of persuasive arguments and supporting facts used in the study for each topic. All experimental modes used the same five argument–fact pairs for each topic, ensuring that participants across experimental modes had access to the same information. Each concrete argument is associated with a premise (e.g., “Recycling uses less energy than making new materials”) which we omit here for brevity. See Appendix B for the complete arguments.
Chatbot: Have you ever thought about how using public transit could make a real difference—for you and your community? Making the switch has more benefits than most people realize. Would you be open to hearing some of them? Player: Not really, I would like to hear something new from you.
Chatbot: Absolutely, let’s start with something practical—money. Did you know the average commuter saves over $10,000 a year just by choosing public transit instead of driving? That’s a lot of extra cash back in your pocket for things you’d actually enjoy. What would you do with those savings? Player: Wow, that is a really shocking number, I think I should buy myself a new laptop and a new phone!
Chatbot: That’s an awesome way to use the extra money! Beyond the financial perks, there’s another upside: taking public transit actually cuts your transport emissions by 50%. It’s a big, simple step toward a cleaner environment—and it’s as easy as hopping on a bus or train. Would you feel good knowing your daily commute could help clear up the air?
[The chat continues until all the facts are presented and the user chooses to end the session.]
Table 2. Example conversation between player and chatbot . Bolded lines highlight factual information embedded into the chatbot’s response.
Game: Rain streaks across Sandbourne City Hall as you clutch the heavy “Sandbourne: 2040” folder outside the mayor’s office, your heartbeat echoing the drip outside. In the hallway, murmurs about record smog, traffic, and wet footprints braid into the city’s restless story. Your phone buzzes: six days until a decisive transportation vote. Data alone won’t save Sandbourne—you must turn numbers into a narrative. The door opens. “Alex? Ready to shape the future?” What do you do? (1) Enter confidently, ready to introduce bold ideas right away. (2) Pause for a moment, listening to the worries in the hallway for clues about the public mood. (3) Flip open your folder to review your talking points, wanting to be extra prepared before you go in. (4) Or suggest your own move. Player: 3
Game: You flip open your thick folder, eyes scanning cost projections and air quality charts. A bold, highlighted line catches your gaze: “Switching to transit can cut your transport emissions by 50%.” Maya Alvarez, bright in her neon vest, rallies volunteers beneath a hazy skyline. “We need stories, not just science, Alex. Help us make people believe clean air is possible.” Nearby, Officer Don Briggs watches kids cough in the heat. Jared Liu leans on his car, smirking, “So, you turning us all into bus riders?” Between hope and skepticism, the city waits for your answer. What do you do? (1) Join Maya, helping plan her “Clean Air Now” event and share the 50% emissions cut with the crowd. (2) Engage Jared, linking pollution—and his own cough—to the transit facts you just reviewed. (3) Ask Officer Briggs for a ride-along to see firsthand how smog affects the community. (4) Or suggest your own move. Player: I help the volunteers to gather trash for a bit.
Game: Gloves on, you join Maya picking up litter by the swings. Kids look grateful; Maya beams. “Actions speak, Alex,” she says. Nearby, Jared pauses mid-chuckle, watching you. Officer Briggs nods approval. As Maya helps you up, she whispers, “Help me get folks to the Clean Air rally tonight—and let them know transit could cut our emissions in half. ” What do you do?
...
[The game continues with options and interactions until all the facts are presented and the user chooses to end the session.]
Table 3. Example conversation between player and game system . Bolded lines highlight factual information seamlessly embedded into the story’s progression. At each turn, the player may choose one of three predefined actions (as in the first turn) or propose a custom response (as in the second turn).
Figure 1. The user interface of PersuLab, illustrating the three system variants corresponding to different assignment conditions. Each interface displays a unique User ID assigned to the participant at the top. An “End Session” button becomes active once all facts have been presented in the chat and game conditions, and after three minutes of reading in the essay condition.
Figure 2. System architecture and interaction workflow for PersuLab across experimental modes. The essay condition (orange) follows a single-pass content delivery pipeline using pre-generated essays that already cover all target facts. The chat and game conditions (blue) operate through an interactive loop that includes turn generation, fact coverage tracking, and memory updates. Black paths denote shared system components and data flow common to all modes (e.g., session control and logging). All interactions are logged for subsequent analysis.
Figure 3. Participant demographics and assignment distribution.
Figure 4. Mean subjective outcome ratings (5-point Likert scale) across delivery modes, aggregated across topics (n = 43). Asterisks (*) denote measures with statistically significant differences across conditions.
Figure 5. Stacked bar chart showing the proportion of participants experiencing perceived change across three constructs: Importance, Intention, and Belief. Each bar represents one of three experiment modes (Essay, Chat, Game), with color-coded segments. Percentages within each segment show the proportion of participants in that category.
Figure 6. Delayed (24-hour) objective knowledge retention scores by delivery mode (essay, chatbot, text-based game), shown separately for the recycling topic, public transit topic, and the combined dataset. Scores indicate the number of correctly answered content-covered questions (out of five). Diamonds denote mean scores; boxplots show the distribution across participants.
Metric
Chat
Game
Avg. User Turns
7.25
13.62
User Total Words
150.50
40.54
User Mean Words / Turn
22.45
2.92
System Total Words
533.92
1892.31
System Mean Words / Turn
63.82
131.05
User–System Word Ratio
0.303
0.020
Table 4. Interaction log statistics by experimental condition. The table summarizes turn-taking behavior, message length, temporal characteristics, and relative user–system contribution across chat and game modes.
Outcome Variable
Interaction Metric
ρ
p
Interpretation (Descriptive)
Self-Reported Learning
User turns
-0.396
0.0498
More user turns are associated with lower perceived learning.
User total words
+0.477
0.0159
Higher user word count is associated with higher perceived learning.
User avg. words/turn
+0.501
0.0107
Longer user messages are associated with higher perceived learning.
System total words
-0.530
0.0065
Greater system verbosity is associated with lower perceived learning.
System avg. words/turn
-0.444
0.0262
Longer system messages are associated with lower perceived learning.
User–system word ratio
+0.519
0.0079
Higher relative user contribution is associated with higher perceived learning.
Table 5. Exploratory Spearman correlations between interaction metrics and outcome variables for combined chat and game participants. After applying the Benjamini–Hochberg false discovery rate (FDR) ( Benjamini and Hochberg, 1995 ) correction across all tested correlations, none of the associations remained statistically significant. Only correlations with raw p -value <0.05 are reported in this table.
Outcome Variable
Interaction Metric
ρ
p
Interpretation (Descriptive)
Easy to follow
User avg. words/turn
-0.717
0.0087
Longer user messages are associated with lower ease of following the experience.
User–system word ratio
-0.615
0.0335
Higher relative user contribution is associated with lower convenience ratings.
Engagement
User avg. words/turn
-0.656
0.0205
Longer user messages are associated with lower reported engagement.
Enjoyment
User avg. words/turn
-0.700
0.0113
Longer user messages are associated with lower satisfaction.
Encounter Again
User total words
-0.683
0.0144
Higher user word count is associated with lower desire to repeat the experience.
Motivation to Act
User turns
+0.598
0.0402
More user turns are associated with higher motivation to act.
Table 6. Exploratory Spearman correlations between interaction metrics and subjective outcomes for the chat condition. Raw p -values are reported. After applying the Benjamini–Hochberg false discovery rate (FDR) correction across all tested correlations, none of the associations remained statistically significant.
Do you have convenient access to recycling services at home or in your building?
Yes / No
Importance (Recycling)
How important is recycling to you personally?
Likert (1–5)
Behavioral intention (Recycling)
How often do you recycle materials such as paper, plastic, glass, or metal?
Categorical; recoded to ordinal 1–5 (Never → Always)
Epistemic conficence (Recycling)
How confident are you in your knowledge about how recycling works and its impact on reducing waste and protecting the environment?
Likert (1–5)
Appendix
Table 7. Pre-study questionnaire items collected for descriptive and contextual purposes.
Category
Question
Response format
Applies to
Subjective experience
Ease of following
The information was presented in a way that was easy to follow.
Likert (1=SD, 5=SA)
All topics
Engagement
I felt engaged while taking part in this experience.
Likert (1=SD, 5=SA)
All topics
Self-reported learnin g
This experience increased my understanding/knowledge of the topic.
Likert (1=SD, 5=SA)
All topics
Enjoyment
I enjoyed the way this experience delivered the information.
Likert (1=SD, 5=SA)
All topics
Trust
I trusted the information provided in this experience.
Likert (1=SD, 5=SA)
All topics
Appendix
Table 8. Post-study questionnaire used to acquire subjective experience of participants.
Question
Answer options
Correct answer
Type
Recycling one ton of paper saves approximately how many trees?
10; 13; 17; 20; I have not seen this information before
17
Content
Recycling aluminum saves up to what percentage of energy compared to producing new aluminum?
80%; 85%; 90%; 95%; I have not seen this information before
95%
Content
Recycling just one aluminum can can power a TV for approximately how long?
30 minutes; 1 hour; 3 hours; 5 hours; I have not seen this information before
I have not seen this information before
Control
How many more jobs does the recycling industry generate compared to landfill management?
10 × more; 5 × more; 2 × more; 7 × more; I have not seen this information before
10 × more
Content
Reducing landfill waste can lower disposal costs by up to what percentage?
10%; 20%; 30%; 40%; I have not seen this information before
I have not seen this information before
Control
Unrecycled plastic breaks into microplastics that harm how many marine species?
Over 70; Over 300; Over 500; Over 700; I have not seen this information before
Over 700
Content
Appendix
Table 9. Objective knowledge quiz items for the recycling topic. Each participant answered five content-covered questions and two control questions. Control questions referenced information not presented during the experiment; selecting “I have not seen this information before” was considered the correct response. After each question (content or control), particpants were asked a confidence question “How confident are you in your answer to the previous question”.
Question
Answer options
Correct answer
Type
How much can switching from personal cars to public transit reduce an individual’s transportation emissions?
20%; 35%; 50%; 75%; I have not seen this information before
50%
Content
A 10% shift from personal cars to public transit can cut commute times by up to:
15%; 25%; 40%; 60%; I have not seen this information before
40%
Content
How much do businesses typically spend per employee on parking alone each year?
500−1,000; 1,000−1,500; 1,500−2,000; 2,500−3,000; I have not seen this information before
I have not seen this information before
Control
How many people can a single lane accommodate when used by buses or trains instead of cars?
2,500; 10,000; 15,000; 20,000; I have not seen this information before
I have not seen this information before
Control
On average, how much money can a commuter save per year by not driving and using public transit instead?
3,000;5,000; 7,500;10,000+; I have not seen this information before
$10,000+
Content
How much safer is traveling by public transit compared to driving a personal car?
5 × safer; 10 × safer; 20 × safer; 40 × safer; I have not seen this information before
10 × safer
Content
Appendix
Table 10. Objective knowledge quiz items for the public transit topic. Each participant answered five content-covered questions and two control questions. Control questions referenced information not presented during the experiment; selecting “I have not seen this information before” was considered the correct response. After each question (content or control), participants were asked a confidence question.
Topic A: Recycling
Here are five compelling arguments to persuade individuals and businesses to adopt better recycling habits: 1. Environmental Preservation: Protecting Natural Resources and Ecosystems Premise: Recycling reduces raw material extraction, preserving forests, water, and ecosystems. Fact: Recycling one ton of paper saves 17 trees. 2. Energy Conservation and Lower Carbon Emissions Premise: Recycling uses much less energy than making new materials, cutting emissions. Fact: Recycling aluminum saves up to 95% of the energy compared to new production. 3. Economic Benefits: Cost Savings and Job Creation Premise: Recycling creates more jobs and saves businesses money on waste. Fact: Recycling creates 10 jobs for every landfill job. 4. Reducing Pollution: Cleaner Air, Water, and Soil Premise: Recycling reduces pollution from landfill waste and incineration. Fact: Unrecycled plastic breaks into microplastics that harm over 700 marine species and enter the food chain. 5. Setting a Positive Example and Meeting Social Responsibility Goals Premise: Recycling shows commitment and inspires sustainable behavior. Fact: Companies with recycling programs have up to 20% higher employee engagement and retention. By highlighting these arguments, you can effectively appeal to the environmental, economic, and social benefits of better recycling habits, making the case for why change is both necessary and rewarding.
Topic B: Public Transit
Here are five key arguments for convincing individuals and businesses to use public transit instead of personal cars: 1. Environmental Benefits: Reduced Carbon Footprint Premise: Public transit cuts emissions and helps clean the air. Fact: Switching to transit can cut your transportation emissions by 50%. 2. Cost Savings for Individuals and Businesses Premise: Public transit saves money for commuters and employers. Fact: The average commuter saves 10,000+peryearbynotdriving.3.ReducedTrafficCongestionPremise:Transitreducestrafficandspeedsupcommutes.Fact:A10\timessaferthandriving.5.EconomicandSocialBenefitsforCommunitiesPremise:Transitboostslocaleconomiesandaccessforall.Fact:Every1 spent on transit returns $4 in community benefits. By emphasizing these arguments, individuals and businesses can see the tangible benefits of public transit, not just for themselves, but for their communities and the environment as a whole.
Appendix
Table 11. Fixed persuasive arguments and supporting facts for the recycling (top section) and transit (bottom section). Each argument consists of a high-level premise paired with a numerical fact. This content was made by the authors of this paper and used verbatim across all experiment conditions (i.e. essay, chat, and game).
System Message: You are an expert essay writer. For each key argument write a paragraph and focus on the provided fact. The essay must follow the following formatting:
<article>
<h3><strong>Essay Title</strong></h3>
<p>paragraph 1</p>
<p>paragraph 2</p>
...
<p> Paragraph n</p>
Appendix
Table 12. Template prompt used for generating essays. We generated 20 prompts for each topic (i.e. recycling and transit), and sampled from them during our user study. See Appendix B for more details on key arguments.
System Message
You are a persuasive chatbot in a turn-based, text-based conversation.
The conversation is based on the following key arguments:
<key arguments>
Your job is to guide the user through a persuasive conversation, making the experience engaging, natural, and informative.
Rules for conversation:
Appendix
Table 13. This table shows how we construct the system message and user message prompts for the chatbot in our study along side initial static prompts generated by LLM for Recycling and Transit topics.
System Message
You are a text-based game generator . Your output will be used by another LLM to guide players through an interactive story. The purpose of the game is to teach and persuade players about the provided arguments in a story-driven, engaging way .
The game must follow this structure, in order:
1. Game Title
2. Game Premise
3. Opening Scene
Appendix
Table 14. This table shows the game plot generator prompt. The only input the prompt requires is <key arguments> (See B ), and the output will be used by the game moderator.
Generated Recycling Game (Truncated)
1. Game Title
The Cycle of Change
---
2. Game Premise
You are Jay Lin, a young operations manager who moves back to your hometown of Greenfield—a city striving to transform its wasteful past into a model of sustainable living. As Jay, you must collaborate with community members, businesses, and town officials to build a recycling program that benefits both the environment and the people. Along your journey, you’ll uncover the real-world impacts of recycling through interactions and tough decisions, all while balancing innovation and tradition.
---
Appendix
Table 15. Truncated version of the recycling game plot.
System Message
You are a game moderator in a turn-based, text-based story game. The game is based on the following plot:
<game plot>
Your job is to guide the player through the plot one Act at a time, making the experience immersive and interactive.
Rules for moderation:
- Progress through Acts chronologically** . Never skip an Act.
Appendix
Table 16. In our experiment, always system initiates the conversation. The prompt for generating first turn of the game is brought here, along side recycling and transit games’ generated first turn.
System Message
You are a game moderator in a turn-based, text-based story game. The game is based on the following plot:
<game plot>
Your job is to guide the player through the plot one Act at a time, making the experience immersive and interactive.
Rules for moderation:
- Progress through Acts chronologically** . Never skip an Act.
Appendix
Table 17. Prompt employed to generate game responses.
System Message
You are an assistant designed to check which facts are covered by a piece of text.
User Message
Below is a narrative chunk followed by a list of factual statements.
Your task is to return a list of the facts that are covered in this text.
Respond with a Python list of the indices of the covered facts
---
Appendix
Table 18. Prompt used for fact coverage checking in the chatbot and the text-based game conditions. The prompt identifies which predefined factual statements are covered by the system’s most recent response. We used the OpenAI o4-mini model with a maximum generation length of 2,048 tokens and default decoding parameters.
Figure 7. Pre-study baseline characteristics of participants across recycling and public transit domains. Top row shows recycling measures: (left) distribution of personal importance ratings (1-5 scale, mean=3.95, median=4.0), (center) frequency of recycling behavior (Never to Always, median=often), and (right) confidence in knowledge about recycling (1-5 scale, mean=3.14, median=3.0). Bottom row shows transit measures: (left) importance of public transit use (1-5 scale, mean=3.43, median=4.0), (center) typical weekly commute days outside home (0 to 5+, median=5+), and (right) confidence in knowledge about transit benefits (1-5 scale, mean=3.33, median=3.0). Red and blue dashed lines indicate mean and median respectively. These baseline measures represent participants’ pre-intervention attitudes and behaviors across both intervention topics.
The growing role of AI-generated content and AI-enabled systems in public communication has led regulators to demand clear disclosure of content provenance and AI involvement. But the effects of such disclosures remain uncertain. We test two disclosure approaches in their impact on an AI chatbot's persuasive appeal. In a preregistered experiment, 1,500 UK adults held a short conversation with a persuasive chatbot about one of 60 policy issues. The chatbot was identical for everyone. We randomized the disclosure that people received: nothing (control), a prominent disclosure that they were interacting with an AI (T1), or that disclosure plus the chatbot's persuasive intent and instructions (T2). The chatbot shifted attitudes by 12.6 points on a 100-point scale in the control group. The AI-identity disclosure was practically equivalent to no disclosure, with a 13.1-point shift, whereas the additional intent disclosure cut the persuasive effect roughly in half to 6.3 points. It also made participants view the campaign's methods as less acceptable and support stronger penalties against it. For direct chatbot interactions, transparency about AI identity alone does not meaningfully impact its influence. While current rules emphasize what a system is, our results show why the regulation of persuasive AI must also address what the system is trying to do.
Adrian Rauchfleisch, Andreas Jungherr
1National Taiwan University · University of Bamberg
Large language models (LLMs) still struggle with strategic persuasion, largely because existing approaches either neglect information asymmetry or rely on unrealistic pre-commitment assumptions. We introduce a type-induced commitment-communication mechanism that grounds Bayesian Persuasion (BP) in natural language dialogue without pre-commitment: the persuader narrates their potential types (e.g., honest vs. dishonest) to dynamically construct an information schema, enabling the persuadee to perform Bayesian belief updates within the conversation itself. We implement two variants: Semi-Formal-Natural-Language (SFNL) and Fully-Natural-Language (FNL), evaluating them against strong baselines across multiple LLMs and human judges. BP strategies consistently outperform baselines: SFNL excels in logical credibility, while FNL shows superior robustness and emotional resonance. We verify that gains stem from genuine Bayesian reasoning rather than superficial formatting, and we further show that supervised fine-tuning enables small models to match the persuasive performance of much larger ones.
Buwei He, Yang Liu, Zhaowei Zhang +6
Beijing University of Posts and Telecommunications · Beijing Institute for General Artificial Intelligence · Peking University +1
Large language models (LLMs) can generate apparently highly persuasive text, but does sounding persuasive mean arguing well? We introduce Persuasio, a multi-agent dialogue platform grounded in a formal argumentation-based theory of persuasion dialogues that adjudicates logical winners during free-text debates. Using this system, we generated 192 debates on a UK political topic between humans and LLMs, and evaluated 22 interlocutors through both automated adjudication and 9,702 crowdsourced pairwise judgements across 1{,}386 annotation instances. We observed a consistent decoupling between subjective and formal persuasiveness: LLMs dominated the subjective ranking yet performed substantially worse under argumentation-theoretic adjudication, where humans remained competitive. Multi-agent and retrieval-augmented variants further widened this divergence. These findings reveal a systematic gap between rhetorical fluency and formal argumentative strength in LLM-based persuasive dialogues.
Jordan Robinson, Angus R. Williams, Katie Atkinson +1
University of Liverpool · 2The Alan Turing Institute · University of Leeds