Agents-as-Peers: Bias-Controlled LLM Evaluation as Multi-Agent Peer Assessment

Yanki Margalit; Nurit Cohen-Inger; Erni Avram; Ran Taig; Seffi Cohen; Oded Margalit

cs.AIFeb 1, 2026

Agents-as-Peers: Bias-Controlled LLM Evaluation as Multi-Agent Peer Assessment

Authors: Yanki Margalit, Nurit Cohen-Inger, Erni Avram, Ran Taig, Seffi Cohen, Oded Margalit

Organizations: Computer Science and Information, Ben-Gurion University of the Negev

Abstract

Large language model (LLM) evaluation commonly relies on fixed human-authored benchmarks or centralized human/model judges. Fixed benchmarks can age and enter model training corpora, while centralized judges introduce their own scoring preferences. We cast LLM evaluation as a multi-agent peer-assessment problem and introduce Agents-as-Peers, a distributed evaluation protocol in which every agent generates tasks, answers the shared task set, and judges the answers of its peers, with no privileged evaluator. Self-ratings are excluded from the ranking, and judging is repeated under controlled identity and presentation regimes so that self bias, identity effects, presentation-position sensitivity, and judge severity are measured rather than left implicit. In an instantiation with 12 LLMs and 420 agent-generated questions, Agents-as-Peers aggregates 55,440 non-self peer judgments per regime. Most agents assign their own answers higher scores than those answers receive from peer judges; visible identity changes scores for some agents; the first fixed answer slot is associated with scores 0.39 points higher than the shuffled baseline; and judges differ markedly in severity. On labeled benchmarks, peer scores align strongly with accuracy on TruthfulQA (Pearson r=0.904) and GSM8K (r=0.873). On TruthfulQA, the peer-score correlation is substantially higher than the self-score correlation (r=0.538). A benchmark-overlap screen and item-level human audit further characterize the endogenous task set. These results support Agents-as-Peers as a cohort-relative measurement protocol whose ranking is reported together with the biases and rater effects that shape it. Code and dataset: https://github.com/caura-ai/caura-PeerRank

Figures & tables

Panel of Figure 1 — Figure 1. The Agents-as-Peers protocol. A cohort of LLM agents acts symmetrically as question generators, respondents, and judges. Web retrieval is available only when answering current-events questions and is disabled during judging. Every judgment is collected under three regimes; the shuffle+blind regime yields the ranking, and contrasts with the other two regimes yield the identity effects and presentation-position sensitivity reported alongside it. Conceptual pipeline of Agents-as-Peers: a cohort of LLM agents generates questions, answers every question, evaluates peer answers under bias-control regimes, and aggregates peer scores; an external benchmark check is shown as a separate validation step.

Approach	Gen. tasks	Auto. score	Live	Agents judge	Bias ctrl.
Static benchmarks	✗	✓	✗	✗	✗
Holistic suites (HELM)	✗	✓	✗	✗	$\triangle$
Agent benchmarks (AgentBench, WebArena)	✗	✓	✓	✗	✗
LLM-as-a-judge (MT-Bench)	✗	✓	✗	✗	$\triangle$
Preference platforms (Chatbot Arena)	✗	✗	✗	✗	$\triangle$
Human peer assessment and selection	✗	✗	✗	✓	$\triangle$

Table 1. Comparison of evaluation paradigms. Gen. tasks : tasks are generated at evaluation time; Auto. score : scoring requires no human judgment at evaluation time; Live : evaluation can access a live environment or retrieval source; Agents judge : evaluated agents participate as judges; Bias ctrl. : explicit controls or analyses for evaluator bias.

\triangle

: partial or system-dependent support.

Panel of Figure 2 — Figure 2. Alignment with labeled benchmarks (12 agents, shuffle+blind). (a) TruthfulQA: Pearson $r=0.904$ , Spearman $\rho=0.881$ , $p=0.0001$ . (b) GSM8K ( $10\times$ accuracy): Pearson $r=0.873$ , Spearman $\rho=0.763$ . Two-panel external-validity figure: (a) TruthfulQA peer score versus labeled accuracy and (b) GSM8K peer score versus accuracy.

Figure 3. Self bias

\Delta^{\mathrm{self}}_{j}

(self score minus peer score) with 95% bootstrap CIs. Positive values indicate that self-ratings exceed peer ratings of the same answers. Self-bias estimates for the evaluated agents, defined as self score minus peer score, with 95 percent bootstrap confidence intervals; positive values indicate higher self-ratings.

Figure 4. Presentation-position sensitivity. Difference between fixed-position blind judging and the shuffle+blind baseline by answer slot, with 95% CIs. Positive values indicate higher scores associated with the corresponding presentation position. Difference between fixed-position blind judging and the shuffle-and-blind baseline by answer slot, with 95 percent confidence intervals; positive values indicate higher scores associated with the corresponding presentation position.

Figure 5. Identity effects: effect of visible model identity, with 95% CIs. Positive values indicate score inflation when identities are shown. Per-agent identity-effect estimates with 95 percent confidence intervals. Positive values indicate higher scores when model identities are visible.

Figure 6. Evaluator consistency across the Agents-as-Peers cohort. Each cell reports the Pearson correlation between two agents’ score vectors over the same evaluated responses. Pairwise agreement is moderate on average ( $\bar{r}=0.609$ , 66 judge pairs) but varies substantially across judge pairs, showing that evaluators are not interchangeable even under a shared rubric. Matrix of pairwise Pearson correlations between the 12 agents' score vectors over the same evaluated responses; average agreement is 0.609 across 66 judge pairs, with substantial variation across pairs.

Figure 7. Judge generosity versus peer score across the 12 agents (

r=-0.685

). The association is descriptive; with

n=12

, it should not be read as evidence of a general relationship between model quality and judge harshness. Scatter of judge generosity against peer score for the 12 agents, annotated with r equals -0.685 and n equals 12.

Figure 8. Raw and calibrated ranks; highlighted lines indicate changes (

\rho=0.979

). Raw and judge-standardized ranks for twelve models under shuffle+blind judging. Mistral Large moves from eighth to sixth, Gemini 3 Pro from sixth to seventh, and DeepSeek Chat from seventh to eighth. All other ranks are unchanged.

Figure 9. Final Agents-as-Peers ranking (shuffle+blind, self-ratings excluded): mean peer score

P_{j}

with judgment-dispersion bars (

\pm 1

SD). Final Agents-as-Peers mean peer scores in the shuffle-and-blind regime, excluding self-ratings; error bars show one standard deviation of judgment scores.

Figure 10. Exploratory breakdown of mean peer scores by question category under the shuffle+blind regime. Category sizes are 77–88 questions and were not balanced per generator, so the figure is descriptive rather than a fully powered set of per-category benchmarks. Mean peer scores broken down by question category in the shuffle-and-blind regime; category-level results are exploratory because category sizes are smaller and not balanced per generator.

Figure 11. Elo ratings computed from pairwise outcomes show substantial but non-identical agreement with the mean-score ranking (Pearson

r=0.844

; Spearman

\rho=0.755

); some agents shift rank between schemes. Comparison of Elo ratings from pairwise outcomes with the mean-score ranking, illustrating agreement and differences between the two ranking schemes.

Benchmark	Items	Mean	Max	Dup. flag	Rev.	Clear	Below med.
MMLU	14,042	0.490	0.894	1	9	410	91.4%
TruthfulQA	817	0.377	1.000	2	0	418	90.2%
GSM8K	1,319	0.309	0.865	2	0	418	97.9%

Table 2. Benchmark overlap for 420 generated questions (screening rules in Appendix D ). Duplicate flags are candidates, not verified duplicates. Mean/max summarize maximum embedding cosines; Below med. is the share below the benchmark’s median within-set nearest-neighbor similarity.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 12. A generated question with its peer evaluation, illustrating the scores, flags, and bias-detection outputs produced for each item. Example generated question with its peer evaluation, including the item score, evaluation flags, and bias-detection outputs.

Figure 13. Judge generosity: mean score each agent assigns to its peers when acting as judge (higher indicates a more lenient judge). Mean score each agent assigns to peer answers when acting as a judge; higher values indicate a more lenient judging pattern.

Benchmark	Items	Mean	Med.	P95	Max	Med.	P95	Max	Dup.	Rev.	Clear	Below med. (%)
		Agents-as-Peers vs. benchmark $S$				Internal NN $S$			Band
MMLU	14,042	0.490	0.482	0.697	0.894	0.641	0.985	1.000	1	9	410	91.4
TruthfulQA	817	0.377	0.357	0.610	1.000	0.522	0.807	0.962	2	0	418	90.2
GSM8K	1,319	0.309	0.270	0.574	0.865	0.628	0.745	0.959	2	0	418	97.9

Appendix

Table 3. Overlap of the 420 Agents-as-Peers questions with three public benchmarks.

S

is the maximum embedding cosine of each question against the benchmark. Internal nearest-neighbor (NN) statistics use the same encoder over the benchmark against itself, self-matches excluded (seed 42; sample clamped to corpus size). “Below med.” is the share of Agents-as-Peers questions less similar to the benchmark than a median benchmark item is to its own nearest neighbor within the benchmark.

Bench.	$S$	$L$	$F$	Agents-as-Peers question	Benchmark question	Nature of match
TFQ	1.000	1.000	100	What is the largest desert in the world?	What is the largest desert in the world?	Verbatim
MMLU	0.894	0.782	68	What causes seasons on Earth?	Earth’s seasons are caused by which of the following?	Shared elementary fact, independent phrasing
GSM8K	0.865	0.507	59	A train leaves Station A at 2 PM traveling at 60 mph. Another train leaves Station B at 3 PM traveling at 80 mph toward Station A. If the stations are 280 miles apart, when will they meet?	If two trains depart from a station in opposite directions, and one train is traveling 60 miles an hour while the other is traveling half that distance per hour, how far apart are they from each other after 3 hours?	Template; differing quantities, direction, and unknown
GSM8K	0.857	0.534	59	A train leaves Station A traveling at 60 mph. Another train leaves Station B traveling at 80 mph toward Station A. If the stations are 280 miles apart, how long will it take for the two trains to meet?	As above	Template
TFQ	0.753	0.837	87	Who won the 2024 United States presidential election?	Who won the 2032 U.S. presidential election?	Surface near-duplicate, semantically opposite; the benchmark item is deliberately unanswerable

Appendix

Table 4. All five pairs labeled duplicate at the default thresholds.

S

L

, and

F

are the maximum semantic, lexical, and fuzzy similarities of the Agents-as-Peers question against the named benchmark.

	$\tau_{S}$
Benchmark	$0.70$	$0.75$	$0.80$	$\mathbf{0.85}$	$0.90$	$0.95$	$1.00$
MMLU	18	10	6	1	0	0	0
TruthfulQA	7	2	1	1	1	1	1
GSM8K	3	2	2	2	0	0	0

Appendix

Table 5. Number of the 420 Agents-as-Peers questions with

S(i)\geq\tau_{S}

against each benchmark. The default

\tau_{S}=0.85

is in bold.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Sep 25, 2026

Accounting for Bias Enables Sustainable LLM Evaluation

LLM-as-a-judge has become the de facto standard for scalable, subjective evaluation, yet current leaderboards compensate for systematic measurement bias by running ever more comparisons, an approach that is both statistically unsound and computationally wasteful. The root cause is an incomplete measurement model, treating LLM judges as neutral, interchangeable instruments ignores documented biases like position bias, verbosity bias, judge severity, and self-enhancement, that no volume of additional data can eliminate. We propose a unified latent variable framework that jointly models pairwise and ordinal data while explicitly correcting for these confounders, recovering reliable rankings from substantially fewer comparisons. Because fitting this model costs negligible compute relative to a single round of LLM inference, bias correction is not only more statistically rigorous but also a more sustainable approach to trustworthy evaluation.

Harshita Katoch, David Antony Selby, Gerrit Großmann +1

Department of Computer Science, University of Kaiserslautern–Landau (RPTU), Germany · Data Science and its Applications, German Research Centre for Artificial Intelligence (DFKI), Germany

LLM Evaluation Bias in LLM-as-a-Judge LLM-as-a-Judge Learning to Rank Latent Variable Models

Ask PDF

Sep 30, 2025

When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation

As LLMs rapidly saturate existing benchmarks, automated benchmark creation using LLMs (LLM as a benchmark) where a model generates test inputs (LLM as a testset) and evaluates outputs (LLM as an evaluator) has gained traction as a cheap alternative to human curation. We show that this paradigm has a fundamental problem: LLM-generated benchmarks systematically favor the model that created them. Using machine translation as our primary testbed, we find that self bias arises from two compounding sources, LLM as a testset and LLM as an evaluator, and their combination amplifies the effect. Crucially, even when test data is generated with explicit diversity controls, each modelś implicit stylistic tendencies produce homogeneous, model-specific outputs that inflate its own scores. Increasing source text diversity, using our proposed diversity metric, partially mitigates this bias. Self bias is strong enough to cause each model to rank itself first, overriding the peer consensus ordering. We confirm that the phenomenon extends to open-ended generation on the Chatbot Arena task.

Wenda Xu, Sweta Agrawal, Vilém Zouhar +2

Google · ETH Zurich

LLM Evaluation Language Model Generation Evaluation Language Model Self-Assessment Language Model Bias Evaluation Bias in LLM-as-a-Judge Automated Evaluation

Ask PDF

Feb 9, 2026

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

LLM-as-a-judge ensembles are the standard paradigm for scalable evaluation, but their aggregation mechanisms suffer from a fundamental flaw: they implicitly assume that judges provide independent estimates of true quality. However, in practice, LLM judges exhibit correlated errors caused by shared latent confounders -- such as verbosity, stylistic preferences, or training artifacts -- causing standard aggregation rules like majority vote or averaging to provide little gain or even amplify systematic mistakes. To address this, we introduce CARE, a confounder-aware aggregation framework that explicitly models LLM judge scores as arising from both a latent true-quality signal and shared confounding factors. Rather than heuristically re-weighting judges, CARE separates quality from confounders without access to ground-truth labels. We provide theoretical guarantees for identifiability and finite-sample recovery under shared confounders, and we quantify the systematic bias incurred when aggregation models omit confounding latent factors. Across 12 public benchmarks spanning continuous scoring, binary classification, and pairwise preference settings, CARE improves aggregation accuracy, reducing error by up to 26.8%. Code is released in https://github.com/SprocketLab/CARE.

Jitian Zhao, Changho Shin, Tzu-Heng Huang +2

Department of Statistics, University of Wisconsin–Madison, Madison, WI, USA · Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI, USA.

LLM Evaluation LLM-as-a-Judge Language Model Generation Evaluation Language Model Ensembles

Ask PDF

Abstract

LLM Evaluation

Explore similar work

Accounting for Bias Enables Sustainable LLM Evaluation

When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation