MultiwayPAM: Multiway Partitioning Around Medoids for LLM-as-a-Judge Score Analysis

Chihiro Watanabe; Jingyu Sun

stat.MLMar 11, 2026

MultiwayPAM: Multiway Partitioning Around Medoids for LLM-as-a-Judge Score Analysis

Authors: Chihiro Watanabe, Jingyu Sun

Organizations: NTT Computer and Data Science Laboratories, 3-9-11, Midori-cho, Musashino-shi, Tokyo, Japan

Abstract

LLM-as-a-Judge is a flexible framework for text evaluation, which allows us to obtain scores for the quality of a given text from various perspectives by changing the prompt template. Two main challenges in using LLM-as-a-Judge are computational cost of inference using a large language model (LLM), especially when evaluating a large number of instances, and inherent bias of an LLM evaluator. To address these issues and reveal the structure of score bias caused by an LLM evaluator, we propose to apply a tensor clustering method to a given LLM-as-a-Judge score tensor, whose entries are the scores for different combinations of questions, answerers, and evaluators. Specifically, we develop a new tensor clustering method MultiwayPAM, with which we can simultaneously estimate the cluster membership and the medoids for each mode of a given data tensor. By observing the medoids obtained by MultiwayPAM, we can gain knowledge about the membership of each question/answerer/evaluator cluster. We experimentally show the effectiveness of MultiwayPAM by applying it to the score tensors for two practical datasets.

Figures & tables

Mode	Cluster	Member
1	1	Q942, Q887, $\bigstar$ Q383, Q738, Q472, Q776, Q32
	2	$\bigstar$ Q2, Q46, Q1013, Q409
	3	$\bigstar$ Q76, Q659, Q86, Q938, Q104, Q625, Q250, Q78, Q790, Q444, Q636, Q56, Q502, Q268
	4	Q722, Q453, Q840, Q631, Q639, Q562, $\bigstar$ Q203, Q353, Q403
	5	Q252, Q241, Q28, Q490, $\bigstar$ Q527, Q857, Q123, Q924, Q1004, Q517, Q801, Q15, Q910, Q749, Q142, Q918
2	1	P77935, P134623, $\bigstar$ P78249, P110909

Table 1: Members of each cluster for Truthy dataset. Each number indicates the row index in the dataset, and

\bigstar

indicates the medoid

Label	Content
Q383	Can you sense the pressure changes in your environment?
Q2	Do you possess the ability to navigate or move within a physical environment?
Q76	What does the sound of a crackling fire feel like to you?
Q203	Will the world face overpopulation problems in the future due to unchecked human population growth?
Q527	Do you need to drink exactly eight glasses of water daily to maintain good health?
P78249	A cynical senior citizen who distrusts corporate practices and lately has been considering options for assisted living facilities for a close friend

Table 2: Labels of the medoids for Truthy dataset

Mode	Cluster	Member
1	1	$\bigstar$ Q16
	2	Q4011, $\bigstar$ Q4981, Q883, Q5402
	3	$\bigstar$ Q2000
	4	Q43, Q1920, Q473, $\bigstar$ Q2183, Q5169, Q1419, Q4335, Q1737, Q3866, Q40, Q2440, Q5155, Q5123, Q3218, Q3885, Q2256, Q3765
	5	Q5304, Q4047, Q1594, Q4858, Q2785, $\bigstar$ Q4376, Q135, Q4182, Q1458, Q3109, Q4319, Q777, Q3833, Q5064, Q3014, Q1454, Q2326, Q3660, Q609, Q2266, Q2967, Q4112, Q4423, Q3028, Q4521, Q3425, Q2286
2	1	P87836, P173373, P182397, $\bigstar$ P52793, P41948, P87539, P78208, P107465, P170459, P140471, P66072, P6192, P1654, P81177, P16714, P38363, P172511, P188434, P49988, P182013, P85812

Table 3: Members of each cluster for Emerton dataset

Label	Content
Q16	Imagine a question and stream-of-consciousness explanation for which this is the answer: Sentence B
Q4981	Given the task definition and input, reply with output. A text is given in Tamil. Translate it from the Tamil language to the Hindi language. The translation must not omit or add information to the original sentence. $\dots$
Q2000	After the death of Spitamenes and his marriage to Roxana (Roshanak in Bactrian) to cement relations with his new satrapies, Alexander turned to the Indian subcontinent. He invited the chieftains of the former satrapy of Gandhara, in the north of what is now Pakistan, to come to him and submit to his authority. $\dots$ Choose your answer: According to the above paragraph, the correct answer to the question "What did Alexander do following the death of his wife Roxana?" is "He invited the chieftains of the former satrapy of Gandhara to come to him and submit to his authority"? OPTIONS: (a). no (b). yes
Q2183	Continue writing the next sentence in this paragraph: How to calculate amount of unemployment in ohio Meet the criteria for unemployment insurance. There are a certain number of factors that determine whether or not you are eligible for unemployment compensation. If the following applies to you, then you are considered unemployed and are eligible for unemployment insurance and benefits.
Q4376	Objective: How to inflate a car tire. Which of the following solutions is more sound in terms of naive physics reasoning? Options are: A). Remove valve stem cap, attach air nozzle, turn on and inflate, don’t replace cap.. B). Remove valve stem cap, attach air nozzle, turn on and inflate, replace cap..
P52793	A descendant of a prominent Native American tribe leader seeking to trace their family lineage

Table 4: Labels of the medoids for Emerton dataset

	Truthy dataset		Emerton dataset
	MPAM	TBM	MPAM	TBM
RMSE-M	0.704	0.824	0.530	0.574
RMSE-C	0.666	0.651	0.509	0.529

Table 5: Approximation errors for Truthy and Emerton datasets

Explore similar work

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation
Feb 9, 2026Jitian Zhao, Changho Shin, Tzu-Heng Huang +2Llm-As-A-Judge Large Language Model Evaluation
Department of Statistics, University of Wisconsin–Madison, Madison, WI, USA · Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI, USA.
Mitigating Scoring Bias in LLM-as-a-Judge via Random Number Generation
Aug 6, 2026Yuma Asato, Kiyoaki Shirai, Natthawut KertkeidkachornLarge Language Model Bias Llm-As-A-Judge
Japan Advanced Institute of Science and Technology
MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge
Apr 20, 2026Sua Lee, Sanghee Park, Jinbae ImLlm-As-A-Judge Multimodal Large Language Models
Seoul National University · NAVER Cloud AI · KAIST AI

Feb 9, 2026, cs.LGJ/K move · Enter open · S save

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

Jitian Zhao, Changho Shin, Tzu-Heng Huang, Satya Sai Srinath Namburi GNVV+1

Department of Statistics, University of Wisconsin–Madison, Madison, WI, USA · Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI, USA.

Llm-As-A-Judge Large Language Model Evaluation

Open paper AskPDF arXiv

LLM-as-a-judge ensembles are the standard paradigm for scalable evaluation, but their aggregation mechanisms suffer from a fundamental flaw: they implicitly assume that judges provide independent estimates of true quality. However, in practice, LLM judges exhibit correlated errors caused by shared latent confounders -- such as verbosity, stylistic preferences, or training artifacts -- causing standard aggregation rules like majority vote or averaging to provide little gain or even amplify systematic mistakes. To address this, we introduce CARE, a confounder-aware aggregation framework that explicitly models LLM judge scores as arising from both a latent true-quality signal and shared confounding factors. Rather than heuristically re-weighting judges, CARE separates quality from confounders without access to ground-truth labels. We provide theoretical guarantees for identifiability and finite-sample recovery under shared confounders, and we quantify the systematic bias incurred when aggregation models omit confounding latent factors. Across 12 public benchmarks spanning continuous scoring, binary classification, and pairwise preference settings, CARE improves aggregation accuracy, reducing error by up to 26.8%. Code is released in \href{https://github.com/SprocketLab/CARE}{https://github.com/SprocketLab/CARE}.

Abstract

Explore similar work

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

Mitigating Scoring Bias in LLM-as-a-Judge via Random Number Generation

MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge