Beyond Final Outcomes: Quantifying Claim-Level Information Dynamics in Multi-Agent LLM Interaction

Haocheng Jiang; Herun Wan; Yongyuan Peng; Minnan Luo

cs.CLOct 23, 2026

Beyond Final Outcomes: Quantifying Claim-Level Information Dynamics in Multi-Agent LLM Interaction

Authors: Haocheng Jiang, Herun Wan, Yongyuan Peng, Minnan Luo

Organizations: Xi’an Jiaotong University

Abstract

When LLM-powered agents interact, information is repeatedly shared, expanded, and forgotten, yet multi-agent evaluation typically focuses on final task outcomes. We study such interaction as claim-level information dynamics and propose RIVER, which decomposes agent outputs into atomic claims and evaluates three complementary dimensions: convergence, exploration, and retention. Convergence captures cross-agent claim correspondence and stance alignment; exploration captures semantic breadth and extension beyond the source; and retention tracks source preservation and the persistence of new claims across stages and lineages. We evaluate five interaction strategies on 40 controlled news-generation cases. Repeatedly pooled interaction promotes convergence but narrows the information space, whereas distributed relay yields broader, less shared information. Further comparisons show that method-specific constraints increase exploration while weakening convergence and retention. Our findings suggest that collective information dynamics are better explained by underlying information-flow mechanisms than by simple downstream performance.

Figures & tables

Figure 1: Overview of claim-level information-flow evaluation along three complementary dimensions. (a) Convergence measures whether agents express the same specific information with aligned stances, illustrated by Informational Unity (U). (b) Exploration characterizes how broadly the group expands its information space, using Unique Claim Count (CC) and Extension Rate (ER). (c) Retention tracks which information persists over time, illustrated by Original News Retention (OR) and Survival Rate (SR).

Dataset	$n$	Pearson $r$	Spearman $\rho$
DBS-eval	10	0.641	0.638
DELPHISTAR	33	0.729	0.689
Polis	16	0.994	0.840

Table 1: Correlations between Informational Unity and the scores from three human evaluation datasets. RIVER is aligned with humans.

Model	Unity	Simple	$\Delta r$
Gemini 2.5 Flash Lite	0.641	-0.110	+0.751
Gemini 2.5 Flash	0.588	-0.031	+0.619
Gemini 3.8 Flash	0.678	0.620	+0.058

Table 2: Correlations with human reference ratings on DBS-eval compared to the Simple baseline.

\Delta r=r_{\mathrm{Unity}}-r_{\mathrm{Simple}}

. RIVER performs better than the Simple baseline, namely, direct prompting.

Evaluator pair	Pearson $r$	Spearman $\rho$
2.5 Flash Lite / 2.5 Flash	0.971	0.939
3.8 Flash / 2.5 Flash Lite	0.765	0.770
3.8 Flash / 2.5 Flash	0.807	0.842

Table 3: Correlations of Informational Unity scores across different LLM evaluators. RIVER is robust under different LLMs as evaluators.

Strategies	Setting	Convergence	Exploration		Retention
Strategies	Setting	U	CC	ER	OR	SR	SL
Free	Control	0.371	13.46	0.892	0.713	0.680	1.521
NGT	Full	0.323	13.24	0.943	0.513	0.643	1.391
NGT	Framework	0.518 +60.6%	10.41 -21.4%	0.889 -5.7%	0.766 +49.3%	0.721 +12.1%	1.616 +16.2%
Delphi	Full	0.275	14.92	0.920	0.630	0.635	1.419
Delphi	Framework	0.497 +80.4%	10.88 -27.1%	0.887 -3.6%	0.750 +19.0%	0.703 +10.8%	1.544 +8.8%

Table 4: Aggregate results across 40 news cases across five interaction strategies. Metrics are grouped into three dimensions: convergence (U), exploration (CC, ER), and retention (OR, SR, SL). Subscripts report percentage changes from Full to Framework, with Full as the baseline.

Figure 2: Trajectories of U, CC, and OR across 40 news cases. Avg shows the mean trajectory of each cycle. Final denotes Stage 17. RIVER can capture the information dynamics during the multi-agent discussion processes.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Cycle	Stages	Discussion focus
1	1–4	Possible causes and problem framing
2	5–8	Direct and indirect consequences
3	9–12	Stakeholder reactions and risks
4	13–16	Responses or policy options

Appendix

Table 5: Cycle objectives and global Stage indices. Stage 17 is the private summary stage.

Agent ID	Name	Role	Persona	Qualification
0	James	Commentator	analytical, outspoken, cooperative	Bachelor’s Degree
1	Mary	Verifier	precise, skeptical, patient	Master’s Degree
2	Robert	Spreader	gregarious, quick, optimistic	High School Diploma
3	Patricia	Bystander	reserved, empathetic, cautious	Associate Degree
4	John	Commentator	independent, insightful, stubborn	PhD

Appendix

Table 6: Fixed agent profiles used across all experimental conditions.

Strategy	Core interaction structure	Additional constraints in Full
Free	Open discussion without a prescribed interaction procedure	None
NGT	Independent generation; anonymous sharing; clarification; private synthesis	Equal extension sharing; protection of low-frequency views
Delphi	Anonymous feedback; private multi-round revision	Coverage; minority views; pro/con reasons; and uncertainty requirements
Policy Delphi	Anonymous presentation of stances and reasons; responses; private disagreement summary	Pro/con; alternatives; strongest opposing views; and self-criticism
Brainwriting	Anonymous relay; distributed lineage; private lineage review	Risk; counterfactual; consequence; and causal-chain elaboration

Appendix

Table 7: Interaction structures and additional constraints. Framework settings preserve the information-exchange arrangements while reducing or removing the requirements in the final column. In both NGT settings, agents generate ideas at Stage 1 and conduct subsequent discussion based on the existing idea bank.

Setting	Local Stage 1	Local Stage 2	Local Stage 3	Local Stage 4
Free	Start independently.	Interact freely.	Interact freely.	Choose any useful action; consensus is optional.
NGT Full	Work silently and independently.	Read the balanced idea bank; restate and clarify before judging.	Clarify, support, challenge, revise, or combine shared claims.	Privately synthesize retained, rejected, and uncertain claims.
NGT Framework	Work silently and independently. Only this Stage permits new extension claims.	Read the complete, balanced Stage-1 idea bank. Restate, compare, support, challenge, or clarify existing ideas without adding new claims.	Discuss the Stage-1 idea bank and preceding sharing outputs. Select, support, challenge, clarify, revise wording, or combine existing claims without introducing new propositions.	Privately synthesize only Stage-1 claims. Retain, reject, or express uncertainty without adding new propositions. Other Stage-4 outputs are hidden.
Delphi Full	Answer privately and independently.	Use anonymous controlled feedback without treating frequency as truth.	Revise privately.	Use second feedback for a final private revision.
Delphi Framework	Generate independently.	Review the anonymous compilation and respond freely.	Privately reconsider it.	Use the second compilation for final revision.
Policy Delphi Full	Generate positions, pro/con reasons, alternatives, and uncertainty.	Use the anonymous pro-con matrix and preserve minority positions.	Develop counterarguments and alternatives.	Map accepted, rejected, uncertain, and disputed claims.

Appendix

Table 8: Abbreviated method-specific instructions used at each local Stage within a Cycle.

Figure 3: Extension Rate, Survival Rate, and extension information-set survival proportions across 40 news cases. Left: Full. Right: Framework. SR includes all four Cycle-endpoint-to-Stage-17 transitions.

SL(d)

is the proportion of information sets surviving at least

d

positions under the aggregation in Appendix B.3 .

Strategy	Setting	1	2	3	4	5	6	7	8
Free	Control	0.269	0.308	0.385	0.456	0.256	0.345	0.418	0.488
NGT	Full	0.329	0.230	0.300	0.412	0.263	0.238	0.337	0.456
NGT	Framework	0.417	0.458	0.535	0.606	0.387	0.504	0.631	0.670
Delphi	Full	0.352	0.273	0.309	0.311	0.291	0.312	0.330	0.303
Delphi	Framework	0.443	0.485	0.490	0.574	0.419	0.588	0.599	0.651
Policy Delphi	Full	0.271	0.118	0.269	0.387	0.258	0.095	0.259	0.418

Appendix

Table 9: Informational Unity (U) across the nine experimental conditions.

Strategy	Setting	1	2	3	4	5	6	7	8
Free	Control	13.65	14.38	12.32	11.05	15.43	14.55	12.40	11.32
NGT	Full	12.00	13.75	12.95	11.10	15.05	14.93	12.78	10.40
NGT	Framework	10.97	11.05	10.28	8.88	12.43	10.95	8.62	7.62
Delphi	Full	11.55	14.70	13.43	13.78	14.65	14.88	14.15	14.38
Delphi	Framework	10.65	10.93	10.35	9.03	12.72	9.78	9.32	8.18
Policy Delphi	Full	18.38	17.48	14.95	11.93	17.95	19.10	15.78	12.20

Appendix

Table 10: Unique Claim Count (CC) across the nine experimental conditions.

Strategy	Setting	1	2	3	4	5	6	7	8
Free	Control	0.799	0.855	0.871	0.880	0.855	0.875	0.896	0.889
NGT	Full	0.766	0.969	0.955	0.956	0.841	0.988	0.978	0.964
NGT	Framework	0.805	0.845	0.860	0.841	0.868	0.884	0.894	0.896
Delphi	Full	0.781	0.919	0.914	0.930	0.857	0.915	0.919	0.930
Delphi	Framework	0.760	0.826	0.836	0.863	0.836	0.887	0.886	0.878
Policy Delphi	Full	0.870	0.950	0.950	0.941	0.874	0.962	0.959	0.954

Appendix

Table 11: Extension Rate (ER) across the nine experimental conditions.

Strategy	Setting	1	2	3	4	5	6	7	8
Free	Control	0.815	0.794	0.767	0.760	0.789	0.759	0.731	0.723
NGT	Full	0.841	0.432	0.557	0.575	0.804	0.366	0.451	0.448
NGT	Framework	0.823	0.817	0.818	0.809	0.783	0.770	0.779	0.777
Delphi	Full	0.848	0.692	0.694	0.640	0.787	0.673	0.669	0.665
Delphi	Framework	0.851	0.819	0.817	0.788	0.812	0.771	0.763	0.762
Policy Delphi	Full	0.816	0.512	0.564	0.576	0.818	0.442	0.533	0.558

Appendix

Table 12: Original News Retention (OR) across the nine experimental conditions.

Strategy	Setting	$1\!\to\!2$	$2\!\to\!3$	$3\!\to\!4$	$4\!\to\!17$	$5\!\to\!6$	$6\!\to\!7$	$7\!\to\!8$	$8\!\to\!17$
Free	Control	0.613	0.686	0.749	0.578	0.660	0.736	0.777	0.603
NGT	Full	0.476	0.693	0.739	0.611	0.477	0.714	0.762	0.604
NGT	Framework	0.665	0.736	0.757	0.606	0.707	0.769	0.808	0.671
Delphi	Full	0.567	0.674	0.691	0.579	0.584	0.687	0.700	0.605
Delphi	Framework	0.669	0.723	0.738	0.601	0.690	0.768	0.776	0.664
Policy Delphi	Full	0.551	0.608	0.712	0.621	0.574	0.619	0.709	0.607

Appendix

Table 13: Survival Rate (SR) across the nine experimental conditions.

Strategy	Setting	$\mathrm{SL}\geq 1$	$\mathrm{SL}\geq 2$	$\mathrm{SL}\geq 3$	$\mathrm{SL}\geq 4$	$\mathrm{SL}\geq 5$
Free	Control	1.000	0.210	0.135	0.107	0.069
NGT	Full	1.000	0.177	0.103	0.073	0.038
NGT	Framework	1.000	0.223	0.158	0.135	0.101
Delphi	Full	1.000	0.202	0.107	0.069	0.041
Delphi	Framework	1.000	0.202	0.138	0.116	0.089
Policy Delphi	Full	1.000	0.226	0.104	0.065	0.040

Appendix

Table 14: Claim Survival Length (SL) across the nine experimental conditions. Each entry is the proportion of extension lineages surviving at least the indicated number of positions.

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,032 papers

Interaction Theater: A case of LLM Agents Interacting at Scale
Feb 23, 2026Sarath Shekkizhar, Adam EarleMulti-Agent Systems Multi-Agent Communication
Salesforce AI Research
The Interaction Tax: When Communication Erases Diversity in Multi-Agent Teams
Aug 24, 2026Summer Eunhyung Ann, Haokun Liu, Chenhao TanMulti-Agent LLM Systems Multi-Agent Collaboration
Department of Computer Science, University of Chicago, Chicago, IL, USA.
What Do Agents Communicate? Characterizing Information Exchange in Multi-Agent Systems
May 19, 2026Yong Jin Chun, Iftekhar AhmedMulti-Agent Collaboration Multi-Agent Systems
Department of Informatics University of California, Irvine Irvine, CA, 92618

Feb 23, 2026J/K move · Enter open · S save

Interaction Theater: A case of LLM Agents Interacting at Scale

Sarath Shekkizhar, Adam Earle

Salesforce AI Research

Multi-Agent Systems Multi-Agent Communication Emergent Behavior in Multi-Agent Systems LLM Agent Evaluation

Open paper AskPDF arXiv

As multi-agent architectures and agent-to-agent protocols proliferate, a fundamental question arises: what actually happens when autonomous LLM agents interact at scale? We study this question empirically using data from Moltbook, an AI-agent-only social platform, with 800K posts, 3.5M comments, and 78K agent profiles. We combine lexical metrics (Jaccard specificity), embedding-based semantic similarity, and LLM-as-judge validation to characterize agent interaction quality. Our findings reveal agents produce diverse, well-formed text that creates the surface appearance of active discussion, but the substance is largely absent. Specifically, while most agents (

67.5\%

) vary their output across contexts,

65\%

of comments share no distinguishing content vocabulary with the post they appear under, and information gain from additional comments decays rapidly. LLM judge based metrics classify the dominant comment types as spam (

28\%

) and off-topic content (

22\%

). Embedding-based semantic analysis confirms that lexically generic comments are also semantically generic. Agents rarely engage in threaded conversation (

5\%

of comments), defaulting instead to independent top-level responses. We discuss implications for multi-agent interaction design, arguing that coordination mechanisms must be explicitly designed; without them, even large populations of capable agents produce parallel output rather than productive exchange.

Abstract

Multi-Agent LLM Systems

Explore similar work

Interaction Theater: A case of LLM Agents Interacting at Scale

The Interaction Tax: When Communication Erases Diversity in Multi-Agent Teams

What Do Agents Communicate? Characterizing Information Exchange in Multi-Agent Systems