cs.CLAug 23, 2026

From Diagnosis to Redesign: Using Quantitative Ethnography to Improve Multi-Agent LLM Reasoning

Authors: Vedant Khatri, Anthony Cusimano, Zachari Swiecki, Zhen Xu, Xiner Liu, Renzhe Yu

Organizations: University of California, Irvine, USA · University of California, Los Angeles, USA · Monash University, Australia · Columbia University, USA · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multi-agent large language model (LLM) systems are designed to improve reasoning by decomposing tasks across multiple agents with specialized functions, but the presence of multiple agents does not inherently guarantee coherent reasoning or outputs that align with task objectives. This paper introduces a quantitative ethnographic (QE) approach for diagnosing and redesigning multi-agent LLM systems based on the discourse produced through agent interactions. We test this approach using automated essay scoring as an example context, applying Epistemic Network Analysis (ENA) to model a five-agent multi-agent debate system and examine differences between debates that produced correct versus incorrect scoring decisions. Results show that, in the initial system, correct scoring decisions were characterized by rubric-grounded justification, agreement, and elaboration. Incorrect scoring decisions, in contrast, were characterized by extended proposition-challenge-response exchanges that were less consistently tied to rubric criteria. We then used the findings to revise the agents' prompts. The revised system improved exact scoring accuracy from 27.78% to 40.28% and shifted the discourse of incorrect debates toward the rubric-grounded pattern of correct ones, making the two nearly indistinguishable. Based on these results, we argue that QE can support a diagnostic-to-redesign loop for AI reasoning by tracing how patterns of agent interaction relate to system performance, informing prompt redesign, and evaluating whether those redesigns change both outcomes and interaction patterns.

Figures & tables

Figure 1: TSMD architecture. Each essay is processed sequentially through five agents corresponding to Toulmin’s argumentative components and a final Judge Agent. Every agent receives the essay text, the ASAP 2.0 rubric, and the full transcript of prior agent turns before producing its contribution.
CodeDefinitionExample utterance
AssertionStatements of belief or opinion.“This essay seems to deserve a score of 2.” [Claim Agent]
PropositionStatements that call for discussion or action.“The current score of 4 is reasonable, but I would lean towards a 3.” [Rebuttal Agent]
ElaborationStatements that support other statements by providing evidence or clarification.“The essay presents a vague and underdeveloped argument against the idea that aliens created the Face on Mars.” [Grounds Agent]
AmplificationExplicit inferential statements.“These factors suggest the essay has a point of view but is weakly supported, aligning with a score of 3.” [Grounds Agent]
JustificationStatements that offer norms, values, or rules of logic to support the validity of other statements.“The essay fits the rubric description of a 2 because it has a vague, very limited, or poorly supported point of view.” [Rebuttal Agent]
AgreementStatements that show agreement.“The proposed score of 2 is justified.” [Warrant Agent]
Table 1: CACS codes used in this study, with definitions and example agent utterances. Code definitions are taken from Canary and Seibold [ 9 ] , Table 2 (revised version).
Figure 2: ENA mean networks for accurate and inaccurate debates in the initial round.
Figure 3: Revised-round ENA model comparing accurate and inaccurate debates.
More in this topic

Multi-Agent Reasoning

Multi-Agent Reasoning studies how multiple AI agents share evidence, model one another, and coordinate decisions on complex tasks.

See all 232 papers

Explore similar work

CardsList
Jun 9, 2026

Early-Token Confidence Predicts Reasoning Quality in Multi-Agent LLM Debate

Evaluating reasoning quality in multi-agent LLM systems is challenging, especially for open-ended tasks without reference answers. We investigate whether intrinsic confidence signals, token-level log-probabilities from decoding, can predict reasoning quality as assessed by LLM-as-judge evaluation. Using a debate-based essay scoring framework, we compare confidence proxies against rubric-based judge scores across two ASAP essay sets. We find that early-token confidence, particularly within the first few generated tokens, is consistently the strongest predictor of reasoning quality, outperforming full-sequence statistics. Analysis of log-probability trajectories shows that the opening phase of generation is the most heterogeneous and therefore most informative. We also observe a systematic asymmetry between agent roles, with stronger alignment between confidence and quality for supportive reasoning than for adversarial critique. These results suggest that early decoding dynamics provide a lightweight and effective signal for estimating reasoning reliability in multi-agent LLM systems.

Ali Keramati, Justin Cheok, Jacob Horne +1

University of California, Irvine

LLM Agent EvaluationMulti-Agent DebateLLM Uncertainty Estimation
AskPDF
Jun 9, 2026

The Confident Liar: Diagnosing Multi-Agent Debate with Log-Probabilities and LLM-as-Judge

Multi-agent debate systems are typically evaluated only on whether the final answer is correct, overlooking the quality of the intermediate reasoning that debate is designed to produce. This paper studies the relationship between three signals in multi-agent debate: token-level log-probability distributions over reasoning tokens, LLM-as-judge rubric scores assigned to those tokens, and final task accuracy. We examine whether internal confidence signals predict externally evaluated reasoning quality, and whether either signal aligns with task correctness, across three domains: rubric-based scoring, mathematical reasoning, and factual question answering. Our framework pairs a two-agent debate architecture -- a Constructor and an Auditor -- with an LLM-as-judge that scores each agent's reasoning along instruction following, justification quality, and evidence grounding, together with a critical-failure flag. Experiments in the rubric-scoring domain reveal a consistent four-phase confidence trajectory and a substantial role asymmetry: confidence aligns with judged reasoning quality roughly twice as strongly for the Constructor as for the Auditor, and confidence-based detection of critical reasoning failures is markedly more reliable for the Constructor (AUROC 0.804) than for the Auditor (0.634). These findings motivate the broader cross-domain investigation proposed in this paper.

Ali Keramati, Justin Cheok, Jacob Horne +1

University of California, Irvine

LLM Agent EvaluationMulti-Agent DebateLLM Uncertainty Estimation
AskPDF
Oct 23, 2026

Understanding Communication Failures in Multi-Agent LLM Systems

Multi-agent LLM systems aim to improve reasoning through debate, evidence sharing, and mutual correction, but interaction can also introduce failures that final accuracy alone cannot explain. Agents may converge too early, ignore dissent, suppress correct minority answers, or preserve a wrong framing across rounds. We present a corpus-grounded taxonomy of communication failures, built from 3,211 failed multi-round trajectories across medical QA, coding, finance, multi-hop reasoning, ethics, and embodied text tasks. Using this taxonomy, we analyze failure distributions across datasets and system designs, round-level error dynamics, recovery hardness, majority transitions, minority-signal suppression, and self-reported confidence. We find that failures are dominated by recovery/correction and consensus/decision problems rather than isolated first-round mistakes, with many emerging after agents first observe peer responses. Based on these findings, we recommend stronger disagreement handling, evidence verification, minority-signal preservation, and recovery checks before finalization.

Priya Pitre, Jun Hou, Yi Fang +2

Virginia Tech, Blacksburg, VA 24061, USA

Multi-Agent LLM SystemsMulti-Agent CommunicationLLM Agent ReliabilityFailure Attribution for LLM AgentsMulti-Agent Consensus
AskPDF