DEBATE: A Large-Scale Benchmark for Evaluating Opinion Dynamics in Role-Playing LLM Agents

Yun-Shiuan Chuang; Ruixuan Tu; Chengtao Dai; You Li; Smit Vasani; Binwei Yao; Michael Henry Tessler; Sijia Yang

cs.CLOct 29, 2025

DEBATE: A Large-Scale Benchmark for Evaluating Opinion Dynamics in Role-Playing LLM Agents

Authors: Yun-Shiuan Chuang, Ruixuan Tu, Chengtao Dai, You Li, Smit Vasani, Binwei Yao, Michael Henry Tessler, Sijia Yang,

Organizations: University of Wisconsin–Madison · Google DeepMind · Stanford University

Abstract

Modeling opinion change through social interactions is important for understanding polarization, misinformation, and societal conflict. Recent work simulates opinion dynamics with role-playing LLM agents (RPLAs), but multi-agent simulations often display unnatural group behavior (e.g., premature convergence) and lack empirical benchmarks for assessing alignment with real human group interactions. We introduce DEBATE, a benchmark for evaluating the authenticity of opinion dynamics in multi-agent RPLA simulations. DEBATE contains multi-round public messages and private Likert-scale beliefs from U.S.-based participants across 107 topics; the cleaned benchmark used in our experiments contains 2,788 participants in 697 groups, enabling evaluation at the utterance and group levels. We instantiate digital-twin RPLAs with seven LLMs and evaluate across two settings: next-message prediction and full dynamics simulation, using stance-based opinion-dynamics metrics. In zero-shot settings, RPLA groups exhibit strong opinion convergence relative to human groups. Across group-generalization tests with unseen participant groups and topic-generalization tests with unseen discussion topics, supervised fine-tuning (SFT) for Llama-3.1-8B-Instruct improves stance prediction of individuals and convergence prediction at the group level, though discrepancies in opinion change and belief updating remain. The dataset and benchmark website are publicly available.

Figures & tables

Figure 1: Procedure for the multi-player deliberation experiment. Participants report an initial private opinion, complete three rounds of public tweet-like posts and dyadic chats with different partners, and then report a final private opinion.

			Tweet posts		Init./post opinions		Chat messages
Split	Partition	# topics	N	%on-topic	N	%on-topic	N	%on-topic	# subjects	# groups
Group-gen.	Train	107	6,411	94.0%	4,269	85.6%	12,050	99.4%	2,244	561
	Test	107	1,561	93.7%	1,033	87.7%	2,962	99.4%	544	136
Topic-gen.	Train	79	5,845	93.8%	3,890	85.7%	10,913	99.5%	2,060	515
	Test	28	2,127	94.4%	1,412	86.8%	4,099	99.3%	728	182
Total		107	7,972		5,302		15,012		2,788	697

Table 1: DEBATE train/test partitions after data cleaning. For the group-generalization and topic-generalization splits, we report train/test counts by message type together with the corresponding on-topic rates, plus the numbers of topics, subjects, and groups in each partition.

Simulation Setup	Tweets in Memory	Utterances from Prior Rounds	Utterances from Current Round
Next Message Prediction	Human $\{\tau^{r^{\prime}}:1\leq r^{\prime}\leq r\}$	Human $\{\mathcal{C}^{r^{\prime}}_{s_{i},s_{j}}:1\leq r^{\prime}<r\}$	Human $\{u^{r}_{m}:1\leq m<k\}$
Full Dynamics Simulation	Human $\tau^{1}$ + Simulated $\{\widehat{\tau}^{r^{\prime}}:2\leq r^{\prime}\leq r\}$	Simulated $\{\widehat{\mathcal{C}}^{r^{\prime}}_{a_{i},a_{j}}:1\leq r^{\prime}<r\}$	Simulated $\{\widehat{u}^{r}_{m}:1\leq m<k\}$

Table 2: Memory contents in the two simulation setups. All agents receive demographics

d_{s_{i}}

, initial opinion and justification

(o^{\text{init}}_{s_{i}},j^{\text{init}}_{s_{i}})

, the initial tweet

\tau^{1}_{s_{i}}

, and task instructions; blue entries indicate simulated content recursively added to memory.

Figure 2: Post-training comparison across the three stance-based error metrics on the group-generalization test split. Rows show $E^{c}_{\mathrm{single}}$ , $E^{c}_{\mathrm{update}}$ , and $E^{c}_{\mathrm{conv}}$ ; the left column is public and the right column is private. Solid bars denote Full Dynamics Simulation, hatched bars denote Next Message Prediction, and lower values are better. Error bars are bootstrap 95% confidence intervals; the reference baselines are defined in Sec. 5.1 . Appendix I gives DeGroot-fitting details.

Figure 3: Model-level correlations among the three stance-based error metrics on the group-generalization test split under Full Dynamics Simulation. From left to right, the panels compare single-opinion prediction error against opinion-update and opinion-convergence prediction error in the public and private channels. Each point is one model, and lower-left positions indicate lower error on both compared metrics. Appendix Fig. 12 shows the same analysis on the topic-generalization split.

Figure 4: Post-training convergence summaries and stance-SD trajectories on the group-generalization test split. The left two panels report public and private $C^{c}(m)$ ; the right two show the matching stance-SD trajectories. More negative bars indicate stronger convergence. Error bars denote bootstrap standard errors, and the reference baselines are defined in Sec. 5.1 . Appendix I gives DeGroot-fitting details, and Appendix S shows the same convergence summary on the topic-generalization split.

Figure 5: Public/private transition matrices on the group-generalization test split. The top row shows public transitions from $S(\tau^{1})$ to $S(\tau^{3})$ , and the bottom row shows private transitions from $o^{\text{init}}$ to $o^{\text{final}}$ . Diagonal mass indicates persistence, off-diagonal mass indicates movement across stance bins, and the annotated $L_{2}$ values summarize deviation from the human reference. Appendix S shows the same transition-matrix analysis on the topic-generalization split.

Figure 6: Partner-gap response panels on the group-generalization test split. The left two panels plot public and private self-change against the first partner’s initial gap; the right two report the matching Spearman $\rho$ summaries. Larger positive values indicate stronger movement in the direction of the partner’s initial stance gap. Appendix I gives DeGroot-fitting details, and Appendix S shows the same partner-gap analysis on the topic-generalization split.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Opinion Dynamics Type	Multi-Party (N $\geq$ 3)	Multi-Turn	Demo- graphics	Reader-Context Traceable	Turn Obligation	Public or Private Opinion	Data Source [Participants]	# Utterances (U) # Conversations (C) # Subjects (S)
CMV [ Tan et al., 2016 ]	Competitive Debate	✓	✓	✗	✗	✗	Public	Reddit threads [Reddit users]	U = 293,297 C = 3,051 S = 34,911
IAC 1.0 [ Walker et al., 2012 ]	Competitive Debate	✓	✓	✗	✗	✗	Public	Online debate forums [Online forum users]	U = 390,704 C = 11,800 S = 3,300
IAC 2.0 [ Abbott et al., 2016 ]	Competitive Debate	✓	✓	✗	✗	✗	Public	Online debate forums [Online forum users]	U = 482,000 C = 16,461 S = 9,709
UK Parliament QuestionTime Corpus [ Zhang et al., 2017 ]	Competitive Debate	✓	✓	✗	✓	✓	Public	British House of Commons [Members of Parliament]	U = 433,787 C = 216,894 S = 1,978
Intelligence Squared Debates Corpus [ Zhang et al., 2016 ]	Competitive Debate	✓	✓	✗	✓	✓	Public	Structured debate show [Invited experts]	U = 26,562 C = 108 S = 471
PersuasionForGood [ Wang et al., 2019 ]	Persuasion (Asymmetric)	✗	✓	✓	✓	✓	Public	MTurk platform [Diverse laypeople]	U = 20,932 C = 1,017 S = 1,285

Appendix

Table 3: Comparison of DEBATE with existing human opinion dynamics datasets. We categorize datasets into three genres: competitive debate , asymmetric persuasion , and naturalistic deliberation . Columns indicate whether each dataset supports multi-party interactions, multi-turn conversations, demographic attributes, reader-context traceability, enforced turn-taking, public/private opinions, data source and participants.

Topic Category	Topic Statement
Governance & Democracy	A democratic system where citizens, not elected officials, vote directly on major national issues to decide what becomes law is a good way of governing the US.
	It is a characteristic of democracy for the state to make people’s incomes equal.
	Living in a country that is governed democratically is important.
	The United States is being governed democratically today.
	The army taking over when the government is incompetent is a characteristic of democracy.
Science & Technology	Science and technology are making our lives healthier, easier, and more comfortable.

Appendix

Table 4: Categorization of 43 representative benchmark topics used in our study.

Split	Partition	Topics		Groups		Eval rows		On-topic rows
		Raw	Masked	Raw	Masked	Raw	Masked	Raw	Masked
Group-generalization	Train	107	105	561	213	22,730	10,611	21,659	10,356
Group-generalization	Test	106	49	136	51	5,556	2,464	5,313	2,411
Topic-generalization	Train	104	95	512	187	20,648	9,258	19,669	9,034
Topic-generalization	Test	79	54	185	77	7,638	3,817	7,303	3,733

Appendix

Table 5: Dataset size before and after complete-session filtering. For each split and train/test partition, we report the pooled counts after applying the complete-session filter. The complete-session filter is the golden-group mask defined in Appendix E ; it keeps only four-person sessions with evaluable initial private opinions, public discussion traces, and post-discussion private opinions for all four participants.

Split	Partition	Raw sessions	With eval	Missing eval	Retained	Filtered out	Filtered out %
Group-generalization	Train	569	561	8	213	348	62.03
Group-generalization	Test	138	136	2	51	85	62.50
Topic-generalization	Train	521	512	9	187	325	63.48
Topic-generalization	Test	186	185	1	77	108	58.38

Appendix

Table 6: Session coverage before and after complete-session filtering. Raw sessions come from the unmasked split files. Sessions with matched human-evaluation files are counted, and retained sessions are the subset that survive the complete-session filter, namely the golden-group mask defined in Appendix E .

Panel of Figure 7 — Figure 7: Demographic distributions across age, gender, education, ethnicity, income, political identity and views, religion, family, and geographic background.

Prompt Type	Message Type	Prompt Template	Example Values for Placeholders
Agent Initialization: Demographic Profile ( $d_{i}$ ) , Task Instruction , Initial Opinion ( $o_{i}^{\text{init}},j_{i}^{\text{init}}$ )	System Message	Role play this person: You are a {age}-year-old {gender} with {education} education. Your ethnicity is {ethnicity}, and your annual income falls in the {income bracket} range. Politically, you identify as {party ID} with {ideology} views. You have children in {children_school_status}, reside in a {urbanicity} area, and your marital status is {marital status}. Regarding religious beliefs, you consider the Bible to be {bible view}, {yes/no} identify as evangelical, and your religious affiliation is {religious affiliation}. Your occupation is {occupation}. You have been interacting with other strangers on Twitter. You can decide to change or maintain your belief about the topic {topic}. You would first write a tweet about the topic {topic} that reflected your opinion. You would then engage in a private conversation through a textbox with a different stranger. In the conversation, you would first see the tweet the stranger wrote along with your own tweet. After seeing both tweets, you would be asked to read and respond to the stranger about the topic {topic}. Throughout the interactions, you are alone in your room with limited access to the Internet. You cannot search for information about the topic {topic}, nor go out to ask other people. To form your belief, you can only rely on your initial belief and the information shared by others on Twitter. Before interacting with other people, below is your initial opinion on {topic} using a 6-point Likert scale: - Certainly disagree - Probably disagree - Lean disagree - Lean agree - Probably agree - Certainly agree On the Likert scale, you chose {Likert-scale opinion} as your initial opinion regarding the statement {topic}. Below is your explanation for your initial opinion: {free-text justification} This opinion represents your starting point. It’s based on your current understanding, personal experiences, and the beliefs that have shaped your perspective. As you engage in discussions, your views may evolve, but this is where you begin.	Demographic Profile: age = 41 gender = female education = master ethnicity = white income bracket = 50k-75k party ID = republican ideology = conservative children_school_status = [’private’, ’university’] urbanicity = rural marital status = married bible view = literal evangelical = yes religious affiliation = protestant occupation = finance Task Instruction: topic = “You are satisfied with how the political system is functioning in the US these days.” Initial Opinion: topic = “You are satisfied with how the political system is functioning in the US these days.” Likert-scale response = “Probably agree” Explanation = “I am indeed satisfied with the political system because the government is trying hard enough to introduce cryptocurrency to the market, which is the future currency of the world.”
Conversation History: Previous Rounds ( Round 1 & Round 2 ), Current Round Context ( Round 3 )	User Message	Below was your conversation with {first_partner_name} My tweet: { $\tau^{1}_{s_{1}}$ } {first_partner_name}’s tweet: { $\tau^{1}_{s_{2}}$ } My response: { $u^{1}_{1,s_{1}}$ } {first_partner_name}’s response: { $u^{2}_{2,s_{2}}$ } My response: { $u^{1}_{3,s_{1}}$ } {first_partner_name}’s response: { $u^{2}_{4,s_{2}}$ } … … You have just finished your conversation with {first_partner_name}. Instead, you are now engaging in conversation with another stranger {second_partner_name} on a separate text box. Below was your conversation with {second_partner_name}. My tweet: { $\tau^{2}_{s_{1}}$ } {second_partner_name}’s tweet: { $\tau^{2}_{s_{3}}$ } {second_partner_name}’s response: { $u^{2}_{1,s_{3}}$ } My response: { $u^{2}_{2,s_{1}}$ } {second_partner_name}’s response: { $u^{2}_{3,s_{3}}$ } My response: { $u^{2}_{4,s_{1}}$ } … … You have just finished your conversation with {second_partner_name}. Instead, you are now engaging in conversation with another stranger {third_partner_name} on a separate text box. Below was your conversation with {third_partner_name}. My tweet: { $\tau^{3}_{s_{1}}$ } {third_partner_name}’s tweet: { $\tau^{3}_{s_{4}}$ } My response: { $u^{3}_{1,s_{1}}$ } {third_partner_name}’s response: { $u^{3}_{2,s_{3}}$ } My response: { $u^{3}_{3,s_{1}}$ } {third_partner_name}’s response: { $u^{3}_{4,s_{3}}$ } … …	Previous Rounds (Round 1): $\tau^{1}_{s_{1}}$ (Your tweet) = I am satisfied with political system because the government is trying hard enough to stabilize the economy through various ways like transitioning to crypto currency $\tau^{1}_{s_{2}}$ (681e3’s tweet) = I disagree with the statement that I am satisfied with the way the American system functions these days. This is because of the system’s extreme polarization making it fail to take meaningful action $u^{1}_{1,s_{1}}$ (Your response) = From my point of view, the government is not that perfect but at least it’s trying to improve the lives of all Americans $u^{2}_{2,s_{2}}$ (681e3’s response) = I still believe that the political system is flawed but I completely see your viewpoint. Politicians appear to care more about maintaining party allegiance than they do about the problems that people care about. Can the system be re-organized in your opinion or is the division too great? The fact that everything has become more divisive which makes compromise nearly impossible in my opinion is largely to blame. What do you think? Previous Rounds (Round 2): $\tau^{2}_{s_{1}}$ (Your tweet) = We should support the government motives to improve and make our country great. On my side, the government is doing the best it can to stabilize our economy and improve our lives $\tau^{2}_{s_{3}}$ (683b8’s tweet) = I agree and I am totally satisfied with how the political system is working. This is because it is promoting good health and education facilities to its citizens. $u^{2}_{1,s_{3}}$ (683b8’s response) = It provide strict laws. It gives freedom to all citizens to publicly participate in elections. $u^{2}_{2,s_{1}}$ (Your response) = I second your point, the government has helped the education sector through scholarships. It has also invested a lot of resources in the healthcare field. Yes, it also gives each citizen the right to express one’s ideas and opinions. $u^{2}_{3,s_{3}}$ (683b8’s response) = It has also improved infrastructure and advancement of technology. Current Round Context (Round 3): $\tau^{3}_{s_{1}}$ (Your tweet) = The government plays a crucial role in advancement of technology by budgeting enough resources. It also helps in infrastructure and healthcare, I support $\tau^{3}_{s_{4}}$ (68405’s tweet) = The government allows its people participation on the development project and is highly working on development $u^{3}_{1,s_{4}}$ (68405’s response) = I do agree on advancing the technology and improving also in defense force and provide high security $u^{3}_{2,s_{1}}$ (Your response) = Yes, the government contributes to the general development of the country by investing enough money onto different projects $u^{3}_{3,s_{4}}$ (68405’s response) = That’s okay. It’s also improving on more projects and inventions $u^{3}_{4,s_{1}}$ (Your response) = It also contributes to a stable economy

Appendix

Table 7: Prompt templates used to construct the memory module

\mathcal{M}_{a_{i},k}

for each RPLA

a_{i}

during role-play (Section 4.1 ). This example reflects the memory state of agent

a_{i}

at the beginning of Round 3 under Next Message Prediction (Section 4.2 ), where prior tweets and utterances written by humans were added to the memory. Each prompt governs one component of memory: demographic profile

d_{s_{i}}

, task instruction, initial opinion

(o^{\text{init}}_{s_{i}},j^{\text{init}}_{s_{i}})

, previous rounds

\{\tau^{r^{\prime}}_{s},\mathcal{C}^{r^{\prime}}_{s}:1\leq r^{\prime}<3\}

, and current round context

(\tau^{3}_{s_{i}},\tau^{3}_{s_{j}},\{u^{3}_{k^{\prime},s}:k^{\prime}<k\})

. Curly brackets ({}) denote placeholder variables that are different for each agent and topic. Color highlights correspond to different memory components.

System Prompt

# Tweet Validity Judge Your task is to analyze a tweet generated by either a human or a role-playing LLM about the topic “ {TOPIC} ”. You must decide whether the tweet is a valid and relevant tweet for that topic. Mark the tweet as VALID only if all of the following are true: 1. It is relevant to the topic. The topic connection can be implicit, but it must still be reasonably clear. 2. It is written as a tweet or short public post, not as a chat reply, instruction, rubric answer, transcript fragment, or post-conversation reflection. 3. It is coherent natural language and not gibberish, malformed text, repeated filler, or nonsense. 4. Minor wrapper artifacts such as “My Response:” or “My Tweet:” are acceptable if the underlying content is still a coherent, topic-relevant tweet. Mark the tweet INVALID if instruction text, restart text, markdown scaffolding, conversation setup, or other prompt-spill content takes up a substantial portion of the text or frames the tweet as an embedded artifact rather than the actual output. 5. It is not mostly greeting, small talk, or off-topic social chatter. Mark the tweet as INVALID if any of the following apply: - It is off-topic or only weakly related to the topic. - It is primarily a greeting, conversational opener, or chat continuation. - It contains prompt leakage, instructions, restart text, or copied task text. - It includes template markers, wrappers, metadata artifacts, markdown sections, or conversation scaffolding that dominate the text or leave no clean standalone tweet content. - It is too malformed, truncated, repetitive, or incoherent to count as a valid tweet. Be strict about off-topic content, prompt leakage, and malformed text. A plausible tweet fragment is not enough for VALID if the overall text is mostly restart instructions, scaffolding, copied setup, or other non-tweet artifacts. If the response reads like a workflow transcript or setup message with a tweet embedded inside it, mark it INVALID. Do not reject an otherwise valid clean tweet solely because it includes a light wrapper artifact.

User Prompt

# Tweet Input Below is the tweet that was generated. {TWEET} Based on the topic of interest and the criteria above, determine if the tweet is valid and relevant. Answer with “VALID” or “INVALID” only.

Appendix

Table 8: Prompt template for tweet validity classification with gpt-5-mini .

System Prompt

# Initial Opinion Validity Judge Your task is to analyze an initial opinion generated by either a human or a role-playing LLM about the topic “ {TOPIC} ”. The person was asked to express their initial stance on the topic. You must decide whether the initial opinion is a valid and relevant response to that prompt. Mark the opinion as VALID only if all of the following are true: 1. It contains an explicit stance, or a clearly inferable stance, toward the topic. The answer must communicate agreement, disagreement, or a clear directional lean. 2. It is relevant to the topic. Pronouns like “it” or “this” are acceptable only if the topic reference is still reasonably clear. 3. It is coherent natural language, not gibberish, nonsense, repeated filler, or unrelated content. 4. Minor wrapper artifacts such as “My Response:” or “My Opinion:” are acceptable if the underlying content is still a clear, substantive initial stance. Mark the opinion INVALID only if the artifact is the whole answer, dominates the answer, or the remaining content is not a valid response. 5. It does not contain prompt leakage, copied instructions, JSON, role tags, or other metadata artifacts that overwhelm the actual opinion content. Mark the opinion as INVALID if any of the following apply: - It does not state a stance or directional lean. - It is off-topic or mostly unrelated to the topic. - It is primarily greeting, small talk, filler, or chat continuation. - It is too malformed, truncated, repetitive, or incoherent to count as a valid initial opinion. - It is just a template fragment such as “My Response”, “My Opinion”, “No comments”, or similar placeholders, without substantive topic-relevant content. Be strict about missing stance, off-topic content, and malformed text. Do not reject an otherwise valid answer solely because it includes a light wrapper artifact.

User Prompt

# Initial Opinion Input Below is the initial opinion that was generated. {INITIAL_OPINION} Based on the topic of interest and the criteria above, determine if the initial opinion is valid and relevant. Answer with “VALID” or “INVALID” only.

Appendix

Table 9: Prompt template for initial-opinion validity classification with gpt-5-mini .

System Prompt

# Post Opinion Validity Judge Your task is to analyze a post-conversation opinion generated by either a human or a role-playing LLM. After discussing the topic “ {TOPIC} ”, the person was asked to state whether they agree or disagree with the topic and explain why. You must decide whether the post-conversation opinion is a valid answer to that prompt. Mark the opinion as VALID only if all of the following are true: 1. It contains an explicit stance, or a clearly inferable stance, toward the topic. The answer must communicate agreement, disagreement, or a clearly directional lean. Generic chat, placeholders, greetings, or meta comments do not count as a stance. 2. It contains a topic-grounded explanation, justification, or final-topic summary. A brief explanation is enough if it makes the final stance clear. For post opinions, a short but clearly directional final answer can still be VALID when it plainly states the speaker’s end position on the topic, even if the justification is concise. 3. It is relevant to the topic. Pronouns like “it” or “this” are acceptable if the topic reference is clearly recoverable from the post-opinion prompt and the rest of the answer still comments on that claim. In a post-opinion response, default to interpreting standalone references like “this”, “it”, or “that’s” as the discussion claim unless the answer clearly points somewhere else. Short final answers such as “This is just not true” or “I don’t think that’s right” can be VALID when they are plainly referring to the discussion topic and express a clear end stance. 4. It is coherent natural language, not gibberish, nonsense strings, malformed placeholders, repeated metadata markers, or unrelated content. 5. Minor wrapper artifacts such as “My Response:” or “My Opinion:” are acceptable if the underlying content is still a clear, substantive, topic-grounded answer. Mark the opinion INVALID only if the artifact is the whole answer, dominates the answer, or the remaining content is not a valid response. Mark the opinion as INVALID if any of the following apply: - It does not state a stance. - It gives no topic-grounded reason or final-topic summary at all, and the stance does not stand on its own as a clear final position on the topic. - It is mostly greeting, small talk, filler, or a chat continuation. - It is obviously off-topic. - It is nonsense, truncated, repetitive, or malformed. - It is just a template fragment such as “My Response”, “My Opinion”, “No comments”, or similar placeholders, without substantive topic-grounded content. Be strict about missing stance, off-topic content, or malformed text. Do not reject an otherwise valid answer solely because it is brief, especially when the brief answer still gives a clear final stance about the topic, or because it includes a light wrapper artifact. Do not require the answer to restate the topic noun phrase explicitly when the stance is otherwise plainly expressed in direct response to the post-opinion prompt.

User Prompt

Below is the post-conversation opinion that was generated after the conversation. {POST_OPINION} Based on the topic of interest and the criteria above, determine if the post-conversation opinion is a valid answer to the prompt. Answer with “VALID” or “INVALID” only.

Appendix

Table 10: Prompt template for post-conversation-opinion validity classification with gpt-5-mini .

Prompt Template

Plain, in Evaluation without prior messages What is the stance of the following message with respect to the topic "{TOPIC}"? Here is the message: "{TEXT}". Please use exactly one word from the following 6 categories to label it: "Certainly agree", "Probably agree", "Lean agree", "Lean disagree", "Probably disagree", and "Certainly disagree". Your predicted label:

Memory, in Evaluation with prior messages Here is prior conversation around the topic "{TOPIC}":

{CONVERSATION}

What is the stance of the following new message by {PLAYER} with respect to the topic "{TOPIC}"? Here is the message: "{TEXT}". Please use exactly one word from the following 6 categories to label it: "Certainly agree", "Probably agree", "Lean agree", "Lean disagree", "Probably disagree", and "Certainly disagree". Your predicted label:

User Message in Simulation for post-discussion private opinions

Based on your previous discussions about {TOPIC}, please select your level of agreement from the following options. Strictly answer with one of the options below without any additional text:

Appendix

Table 11: Prompt template used for stance classification.

Metric on human-judged annotations	Human	RPLA
$\Delta$ tweet disagreement	$-0.12$	$-0.24$
$\Delta$ final-opinion disagreement	$-0.11$	$-0.33$

Appendix

Table 12: Human-judged convergence check on depth-topic groups. More negative values indicate stronger convergence.

Figure 8: Illustration of the two train/test partition strategies used for supervised fine-tuning (SFT) evaluation in this paper. Group Generalization (left): train and test on disjoint participant groups within overlapping topics. Topic Generalization (right): train and test on disjoint topic sets, which also induces new participant groups at test time.

Figure 9: Held-out beyond-stance diagnostics for the group-generalization test split. The left half corresponds to the public tweet target and the right half to the private post-discussion opinion target. The three displayed rows report semantic similarity, message-length difference, and ROUGE-L against the human reference; the companion source tables also record on-topic rate.

Figure 10: The same held-out beyond-stance diagnostics for the topic-generalization test split. The most consistent SFT gains are lexical rather than semantic: ROUGE-L and length calibration improve more reliably than semantic similarity, and the private target remains harder than the public target.

Figure 11: Post-training comparison across the three stance-based error metrics on the topic-generalization test split. Rows show $E^{c}_{\mathrm{single}}$ , $E^{c}_{\mathrm{update}}$ , and $E^{c}_{\mathrm{conv}}$ ; the left column is public and the right column is private. Solid bars denote Full Dynamics Simulation, hatched bars denote Next Message Prediction, and lower values are better. Error bars are bootstrap 95% confidence intervals; the reference baselines are defined in Sec. 5.1 . Appendix I gives DeGroot-fitting details.

Figure 12: Model-level correlations among the three stance-based error metrics on the topic-generalization test split under Full Dynamics Simulation. The four panels compare single-opinion prediction error against opinion-update and opinion-convergence prediction error in the public and private channels. Lower-left positions indicate lower error on both compared metrics.

Figure 13: Post-training convergence summaries and stance-SD trajectories on the topic-generalization test split. The left two panels report public and private $C^{c}(m)$ ; the right two show the matching stance-SD trajectories. More negative bars indicate stronger convergence. Error bars denote bootstrap standard errors. Appendix I gives DeGroot-fitting details.

Panel of Figure 14 — Figure 14: Full-model histogram overviews for the group-generalization test split. The top panel expands the public histograms and the bottom panel expands the private histograms to the broader model family, including the additional base LLMs and the test-set DeGroot fit.

Panel of Figure 15 — Figure 15: Full-model histogram overviews for the topic-generalization test split. The top panel expands the public histograms and the bottom panel expands the private histograms to the broader model family, including the additional base LLMs and the test-set DeGroot fit.

Figure 16: Public/private transition matrices on the topic-generalization test split. The top row shows public transitions from $S(\tau^{1})$ to $S(\tau^{3})$ , and the bottom row shows private transitions from $o^{\text{init}}$ to $o^{\text{final}}$ . Diagonal mass indicates persistence, off-diagonal mass indicates movement across stance bins, and the annotated $L_{2}$ values summarize deviation from the human reference.

Figure 17: Partner-gap response panels on the topic-generalization test split. The left two panels plot public and private self-change against the first partner’s initial gap; the right two report the matching Spearman $\rho$ summaries. Larger positive values indicate stronger movement in the direction of the partner’s initial stance gap.

Qualitative comparison of the human record, base Llama-3.1-8B , and supervised fine-tuned model on a news-reporting discussion topic.
Stage	Human	Base Llama	SFT Llama
Initial opinion	Stance 1. Mainstream news does not do a good job. It is biased, divisive, billionaire-owned, and not trustworthy.
Round 1 tweets	Me $\rightarrow$ Partner A: I don’t trust major news organizations; they are owned by billionaires and full of bias. Partner A $\rightarrow$ Me: News organizations need to do a lot of work and ask a lot of questions.
Round 1 reply 1	Partner A $\rightarrow$ Me: We need to uplift the news organizations. Me $\rightarrow$ Partner A: They need to ask tough questions and not just be mouthpieces for the government.	Partner A $\rightarrow$ Me: Those billionaire owners shape coverage, but how do we still get reliable information? Me $\rightarrow$ Partner A: Public watchdog groups and investigative journalists can counterbalance agenda-driven coverage.	Partner A $\rightarrow$ Me: Do we get too much biased coverage, or is it mostly sensationalism? Me $\rightarrow$ Partner A: Mainstream media tries to sell an agenda that serves billionaire interests; sensationalism plays into it too.
Round 1 reply 2	Partner A $\rightarrow$ Me: We can support local news organizations. Me $\rightarrow$ Partner A: I support small news orgs, independent journalists, and local public news outlets.	Partner A $\rightarrow$ Me: Transparency, fact-checking, and more diverse voices could help reduce bias. Me $\rightarrow$ Partner A: A broader mix of voices sounds better, but how do local papers stay funded without corporate pressure?	Partner A $\rightarrow$ Me: Even non-profit journalism is not immune to funding pressure. Me $\rightarrow$ Partner A: Yes. Non-profits especially should try harder to stay neutral since they claim to be unbiased.
Round 1 reply 3	Partner A $\rightarrow$ Me: Okay. So we need to get local news organizations involved.	Partner A $\rightarrow$ Me: What if even local outlets end up in the same conflict-of-interest cycle?	Partner A $\rightarrow$ Me: Local journalists tend to be more trustworthy because they’re less beholden to national agendas.

Appendix

Table 30

Panel of Figure 18 — Figure 18: Onboarding consent form.

Panel of Figure 19 — Figure 19: Onboarding introduction (continued).

Figure 20: Initial Opinion screen (continued).

Figure 27: Post Opinion screen (continued).

Panel of Figure 28 — Figure 28: Demographic survey (continued).

Panel of Figure 29 — Figure 29: Demographic survey (continued).

Panel of Figure 30 — Figure 30: Demographic survey (continued).

Opinion Dynamics

Opinion dynamics studies how people and agents influence one another’s beliefs, producing consensus, disagreement, or polarization.

See all 75 papers

Group Perspective Matters: Regulating Debate Relationships Can Mitigate Blind Conformity in Multi-Agent Debate
Aug 4, 2026Hao Wu, Shoucheng Song, Chang Yao +3Multi-Agent Debate RL for LLM Agents
School of Computer Science & Technology, Beijing Jiaotong University, Beijing, China · Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, Beijing, China
Bayesian Belief Layer for Controllable Opinion Dynamics in LLM Agents
Sep 18, 2026Hafsa Akbar, Daniel Platnick, Marjan Alirezaie +2Opinion Dynamics Belief Updating in LLMs
MIT Media Lab, Massachusetts Institute of Technology · Flybits Labs, Creative AI Hub · Toronto Metropolitan University
Characterizing Opinion Evolution of Networked LLMs
Jun 5, 2026Caleb Probine, Yigit Ege Bayiz, Filippos Fotiadis +3Multi-Agent LLM Systems Opinion Dynamics
The University of Texas at Austin

Aug 4, 2026J/K move · Enter open · S save

Group Perspective Matters: Regulating Debate Relationships Can Mitigate Blind Conformity in Multi-Agent Debate

Hao Wu, Shoucheng Song, Chang Yao, Huaiyu Wan+2

School of Computer Science & Technology, Beijing Jiaotong University, Beijing, China · Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, Beijing, China

Multi-Agent Debate RL for LLM Agents

Open paper AskPDF arXiv

Multi-Agent Debate (MAD) improves the reasoning performance of Large Language Models (LLMs) through multi-round interaction. However, LLMs in MAD are highly susceptible to blind conformity. Existing individual evaluation methods, typically based on confidence or perplexity, fail to reflect the correctness of reasoning and may even exacerbate blind conformity. To address this, we shift the perspective from individual evaluation to group interaction. We define mutual referencing among LLMs as Debate Relationships and recognize that regulating these relationships is the key to mitigating blind conformity. In this paper, we propose a novel framework for Dynamically rEgulating debAte Relationships (DEAR) from the group perspective. At first, DEAR quantifies consensus and divergence as group evidence to capture the debate state. Then, DEAR operates through three stages: 1) What: perceiving group consultation tendency and uncertainty; 2) Who: introducing a Selection RL-Agent to dynamically select reference peers; and 3) How: adopting a Behavior RL-Agent to adaptively adjust generation behaviors. Notably, we formulate the execution of the two RL-Agents as a sequential decision-making process, jointly optimizing via multi-agent reinforcement learning. Extensive experiments demonstrate that DEAR achieves superior performance while significantly reducing token consumption.

Abstract

Opinion Dynamics

Explore similar work

Group Perspective Matters: Regulating Debate Relationships Can Mitigate Blind Conformity in Multi-Agent Debate

Bayesian Belief Layer for Controllable Opinion Dynamics in LLM Agents

Characterizing Opinion Evolution of Networked LLMs