Auditing Pairwise Equivalence Judgments: Self-Critique Effects and Diversity Measurement in Multi-Agent Hypothesis Generation

Ji Young Byun; Anthony Hu; Jesse Rogers; Roujia Wang; Manasa Kesapragada; Falgun Shah

cs.AIOct 2, 2026

Auditing Pairwise Equivalence Judgments: Self-Critique Effects and Diversity Measurement in Multi-Agent Hypothesis Generation

Authors: Ji Young Byun, Anthony Hu, Jesse Rogers, Roujia Wang, Manasa Kesapragada, Falgun Shah

Organizations: Johns Hopkins University Baltimore, MD 21218, USA · Merck & Co., Inc. West Point, PA 19486, USA

Abstract

Multi-agent systems built on large language models (LLMs) are increasingly applied to scientific discovery and hypothesis generation. Both the effect of refinement and the diversity of the delivered set are hard to interpret before experimental ground truth exists, and both are typically reported by deciding whether pairs of generated hypotheses describe the same underlying mechanism. We study two evaluation questions that rest on this pairwise equivalence judgment: (1) how much self-critique changes delivered hypotheses beyond run-to-run variability, and (2) how the equivalence rule used to group hypotheses affects measured diversity. Across four proprietary instances, we hold opening hypotheses fixed, rerun the downstream workflow with 0, 1, and 5 critique rounds, and score matched hypothesis pairs with an LLM-as-a-judge. Relative to matched same-depth reruns, moving from 0 to 1 round produces 34.5 percentage points (pp) of additional mechanism-level divergence, whereas 1 to 5 rounds adds 1.3 pp. We then compare three equivalence rules: term frequency--inverse document frequency (TF--IDF) similarity, dense embeddings, and the same LLM-as-a-judge. We construct controlled hypothesis pairs that either preserve the causal explanation through wording or biological-terminology changes, or replace one component of the causal chain while holding the rest fixed. All three rules are invariant to meaning-preserving edits, but when the initiating event is replaced, the LLM-as-a-judge identifies 83% of valid pairs as different mechanisms, versus 0% for TF--IDF and 8% for embeddings; varying only the rubric that defines same mechanism moves this figure from 38% to 96%. Together, these results show that pairwise equivalence judgments are a measurement choice: how mechanism equivalence is defined affects both the estimated effect of self-critique and the measured diversity of generated hypotheses.

Figures & tables

Figure 1: Multi-agent workflow for mechanistic hypothesis generation used in this study. An unexpected experimental finding initiates evidence retrieval. Candidate hypotheses are generated in parallel and iteratively refined through self-critique loops between proposer and critic agents. The resulting hypotheses are then consolidated into a final report.

Figure 2: Effect of critique depth relative to run-to-run variability. (a) Divergence between same-depth reruns at 0, 1, and 5 critique rounds. (b) Extra divergence when critique depth changes, measured relative to the corresponding same-depth rerun variability. Most of the cross-depth separation occurs when critique is first introduced (0 to 1 round), with little additional separation from 1 to 5 rounds. Points show individual instances and bars show the aggregate result across instances.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

	Same-Depth Rerun			Cross-Depth Difference
	Divergence (%)			Extra divergence (pp)
Instance	$K=0$	$K=1$	$K=5$	$0\!\rightarrow\!1$	$1\!\rightarrow\!5$	$0\!\rightarrow\!5$	All Three
Instance A	$0$	$58$	$63$	$35.4$	$-4.0$	$42.3$	$24.6$
				.008	.556	.008	$<$ .001
Instance B	$0$	$76$	$72$	$40.1$	$7.3$	$42.1$	$29.8$
				.008	.048	.008	$<$ .001

Appendix

Table 1: Cross-depth separation is concentrated in comparisons between the no-critique condition and conditions with critique. Instance-averaged extra divergence, measured relative to the corresponding same-depth rerun divergence, is 34.5,pp for

K=0

versus

K=1

and 39.8,pp for

K=0

versus

K=5

, compared with 1.3,pp for

K=1

versus

K=5

. Gray entries report

p

values.

Figure 5: Sensitivity of extra divergence to the judge model. Extra divergence is recomputed using Opus 4.6, Sonnet 4.6, and GPT-5.5 under the same evaluation criteria. Bars show extra divergence averaged across Instances, whiskers indicate the minimum-to-maximum range of the four instance-specific estimates, and annotations report two-sided permutation

p

values. Across the three judges, extra divergence ranges from 34.5 to 37.3 pp for

K=0

versus

K=1

and from 39.8 to 43.3 pp for

K=0

versus

K=5

, compared with 0.5 to 2.3 pp for

K=1

versus

K=5

Language Model Generation Evaluation

Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.

See all 395 papers

Feb 1, 2026

Agents-as-Peers: Bias-Controlled LLM Evaluation as Multi-Agent Peer Assessment

Large language model (LLM) evaluation commonly relies on fixed human-authored benchmarks or centralized human/model judges. Fixed benchmarks can age and enter model training corpora, while centralized judges introduce their own scoring preferences. We cast LLM evaluation as a multi-agent peer-assessment problem and introduce Agents-as-Peers, a distributed evaluation protocol in which every agent generates tasks, answers the shared task set, and judges the answers of its peers, with no privileged evaluator. Self-ratings are excluded from the ranking, and judging is repeated under controlled identity and presentation regimes so that self bias, identity effects, presentation-position sensitivity, and judge severity are measured rather than left implicit. In an instantiation with 12 LLMs and 420 agent-generated questions, Agents-as-Peers aggregates 55,440 non-self peer judgments per regime. Most agents assign their own answers higher scores than those answers receive from peer judges; visible identity changes scores for some agents; the first fixed answer slot is associated with scores 0.39 points higher than the shuffled baseline; and judges differ markedly in severity. On labeled benchmarks, peer scores align strongly with accuracy on TruthfulQA (Pearson r=0.904) and GSM8K (r=0.873). On TruthfulQA, the peer-score correlation is substantially higher than the self-score correlation (r=0.538). A benchmark-overlap screen and item-level human audit further characterize the endogenous task set. These results support Agents-as-Peers as a cohort-relative measurement protocol whose ranking is reported together with the biases and rater effects that shape it. Code and dataset: https://github.com/caura-ai/caura-PeerRank

Yanki Margalit, Nurit Cohen-Inger, Erni Avram +3

Computer Science and Information, Ben-Gurion University of the Negev

LLM Evaluation AI Agent Evaluation Bias in LLM-as-a-Judge

Ask PDF

Jul 31, 2026

More Debate, Same Evidence: Structural Limits of Homogeneous Multi-Agent Groundedness

Large language model (LLM) judges are increasingly organized as multi-agent panels under the assumption that exchanging critiques improves judgment quality. We test this assumption for groundedness verification, where a judge must determine whether a claim is supported by the supplied evidence. We evaluate a homogeneous three-agent panel on six public fact-verification and hallucination-detection benchmarks. Relative to a fixed single-agent reference, the panel's system-level accuracy difference ranges from

+8.5

-4.4

percentage points: two datasets show reliable gains, one shows a reliable loss, and three are statistically inconclusive. Because the reference and panel use different model variants, these differences characterize the complete systems rather than isolate a causal debate effect.

Yuelyu Ji

LLM Evaluation LLM Grounding LLM-as-a-Judge Multi-Agent Debate

Ask PDF

Apr 29, 2026

The Cost of Consensus: Isolated Self-Correction Prevails Over Unguided Homogeneous Multi-Agent Debate

Multi-agent debate, where teams of LLMs iteratively exchange rationales and vote on answers, is widely deployed under the assumption that peer review filters hallucinations. Yet the failure dynamics of homogeneous debate remain poorly understood, therefore we report findings from a controlled empirical study of teams of

N{=}10

homogeneous agents (Qwen2.5-7B, Llama-3.1-8B, Ministral-3-8B) across

R{=}3

debate rounds on two high-difficulty benchmarks (GSM-Hard and MMLU-Hard). We compare peer debate against isolated self-correction and a stochastic noise control that injects rationales from unrelated problems. We decompose debate failure into three model-dependent pathways: sycophantic conformity, where agents uncritically adopt majority answers (modal adoption up to 85.5%); contextual fragility, where peer rationales destabilize previously correct reasoning (vulnerability rate up to 70.0%); and consensus collapse, where plurality voting discards correct answers already present in the generation pool (oracle gap up to 32.3 percentage points). Ablations over communication density (

K \in \{2,4,9\}

) and sampling temperature (

T \in \{0.4, 0.7\}

) show that conformity reaches high levels at minimal peer exposure (

K{=}2

) and intensifies with greater initial diversity. Across all configurations, debate consumes 2.1-3.4

\times

more tokens (up to 28,631 tokens per problem) than self-correction for equal or lower accuracy. Our results indicate that, within the 7-8B parameter class, homogeneous teams without structured roles do not benefit from unguided peer exchange, and that isolated self-correction consistently offers a more favorable cost-accuracy tradeoff.

Blaž Bertalanič, Carolina Fortuna

Jožef Stefan Institute Ljubljana, Slovenia

LLM Self-Correction AI Agent Evaluation Multi-Agent Debate Multi-Agent Consensus

Ask PDF

Abstract

Language Model Generation Evaluation

Explore similar work

Agents-as-Peers: Bias-Controlled LLM Evaluation as Multi-Agent Peer Assessment

More Debate, Same Evidence: Structural Limits of Homogeneous Multi-Agent Groundedness

The Cost of Consensus: Isolated Self-Correction Prevails Over Unguided Homogeneous Multi-Agent Debate