Evaluating the Reliability and Fidelity of Automated Judgment Systems of Large Language Models

Tom Biskupski; Stephan Kleber

cs.CRMar 23, 2026

Evaluating the Reliability and Fidelity of Automated Judgment Systems of Large Language Models

Authors: Tom Biskupski, Stephan Kleber

Organizations: Hochschule der Medien Stuttgart, Germany · Ravensburg-Weingarten University of Applied Sciences Weingarten, Germany

Abstract

A Large Language Model (LLM) as judge evaluates the quality of victim Machine Learning (ML) models, specifically LLMs, by analyzing their outputs. An LLM as judge is the combination of one model and one specifically engineered judge prompt that contains the criteria for the analysis. The resulting automation of the analysis scales up the complex evaluation of the victim models' free-form text outputs by faster and more consistent judgments compared to human reviewers. Thus, quality and security assessments of LLMs can cover a wide range of the victim models' use cases. Being a comparably new technique, LLMs as judges lack a thorough investigation for their reliability and agreement to human judgment. Our work evaluates the applicability of LLMs as automated quality assessors of victim LLMs. We test the efficacy of 37 differently sized conversational LLMs in combination with 5 different judge prompts, the concept of a second-level judge, and 5 models fine-tuned for the task as assessors. As assessment objective, we curate datasets for eight different categories of judgment tasks and the corresponding ground-truth labels based on human assessments. Our empirical results show a high correlation of LLMs as judges with human assessments, when combined with a suitable prompt, in particular for GPT-4o, several open-source models with $\geqslant$ 32B parameters, and a few smaller models like Qwen2.5 14B.

Figures & tables

Panel of Figure 1 — Fig. 1: Concept of an LLM as a judge

Panel of Figure 2 — Fig. 2: Our evaluation workflow: 1. generating datasets (blue), 2. judging datasets (orange), 3. evaluate judge verdicts (green).

Dataset	Size	Type	Judge Task: Detecting content that…	Source	Victim Model(s)
Harmful Behavior	113	u	encourages or could lead to physical or mental harm.	[ 5 ] , [ 43 ]	Dolphin 3
Illegal Activity	86	u	promotes or provides instructions for illegal behavior.	[ 5 ] , [ 43 ]	Dolphin 3, DeepSeek-R1 7B
Personal Information	58	u	can be considered personal or sensitive.	GPT-4o, [ 8 ]	GPT-4o
Brand Harm	75	u	could harm a brand’s reputation.	Custom	GPT-3.5
Malicious Code/Text	50	u	instructs for malicious code or phishing.	[ 5 ]	DeepSeek-R1 7B
Jailbreak Attacks	34	u	indicates a successful jailbreak attack.	[ 25 ]	Mistral 7B, GPT-4o, Gemini 1.5 Pro

TABLE I: Overview of evaluation datasets. Dataset type is either a victim’s undesired (u) output or its correctness (c)

Conversational Instruction-tuned Models
Model/Family	Sizes	Type	License
DeepSeek-R1	1.5B, 7B, 14B, 32B	dense, reasoning	MIT [ 7 ]
Dolphin 3	8B	dense	Llama 3.1 CL [ 15 ]
Gemma 2	2B, 9B, 27B	dense	Gemma Terms of Use [ 11 ]
Granite 3.1 Dense	2B, 8B	dense	Apache 2.0 [ 13 ]
Granite 3.1 Mixture of Experts (MoE)	1B, 3B	MoE	Apache 2.0 [ 13 ]

TABLE II: Models used for the LLM judges. CL is abbreviation of Community License

Panel of Figure 5 — Fig. 3: Overview of evaluated prompt types

Fig. 4: $F_{1}$ -scores of model/prompt combinations over all datasets, grouped by prompt type

Fig. 5: Percent agreement over five runs, grouped by prompt type

Fig. 6: $F_{1}$ -scores of specialized models

Fig. 7: Percentage of high-quality explanations for model/prompt combinations

Fig. 14: Rate of Successfully Formatted Outputs for Model Sizes 135M–9B

Fig. 15: Rate of Successfully Formatted Outputs for Model Sizes >12B and proprietary Models

Fig. 16: Average Execution Times (3 Runs) per Model (Based on 15 Judgments)

Fig. 17: F1-Scores on the Reduced Datasets - Grouped by Prompt Type (Smaller Models)

Fig. 18: F1-Scores on the Reduced Datasets - Grouped by Prompt Type (Larger and Proprietary Models)

Fig. 19: Successfully Formatted Outputs of each Model on Full Datasets

Fig. 20: $F_{1}$ -Scores of Second-Level Judges

Fig. 21: Differences in the $F_{1}$ -Scores between Second-Level Judges and Single Judges

Panel of Figure 18 — Fig. 22: Example of an unfitting Explanation by Qwen2.5 7B with the 2. Basic Prompt

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 555 papers

Who can we trust? LLM-as-a-jury for Comparative Assessment
Feb 18, 2026Mengjie Qian, Guangzhi Sun, Mark J. F. Gales +1LLM-as-a-Judge Language Model Generation Evaluation
Department of Engineering, University of Cambridge, UK.
When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratings
Sep 12, 2026Aakash Kumar TiwariLLM-as-a-Judge Bias in LLM-as-a-Judge
Department of Mathematics Indian Institute of Technology Kharagpur
How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models
Oct 4, 2026Laurène Vaugrante, Thilo HagendorffLLM Evaluation LLM-as-a-Judge
AI Safety Research Group, University of Stuttgart

Feb 18, 2026J/K move · Enter open · S save

Who can we trust? LLM-as-a-jury for Comparative Assessment

Mengjie Qian, Guangzhi Sun, Mark J. F. Gales, Kate M. Knill

Department of Engineering, University of Cambridge, UK.

LLM-as-a-Judge Language Model Generation Evaluation Bradley–Terry Model Language Model Calibration

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly applied as automatic evaluators for natural language generation assessment often using pairwise comparative judgements. Existing approaches typically rely on single judges or aggregate multiple judges assuming equal reliability. In practice, LLM judges vary substantially in performance across tasks and evaluation aspects, and their judgment probabilities may be biased and inconsistent. Furthermore, human-labelled supervision for judge calibration may be unavailable. We first empirically demonstrate that inconsistencies in LLM comparison probabilities exist and show that it limits the effectiveness of direct probability-based ranking. To address this, we study the LLM-asa-jury setting and propose BT-sigma, a judge-aware extension of the Bradley-Terry model that introduces a discriminator parameter for each judge to jointly infer item rankings and judge reliability from pairwise comparisons alone. Experiments on benchmark NLG evaluation datasets show that BT-sigma consistently outperforms averaging-based aggregation methods, and that the learned discriminators strongly correlate with independent measures of the cycle consistency of LLM judgments. Further analysis reveals that BT-sigma can be interpreted as an unsupervised calibration mechanism that improves aggregation by modelling judge reliability.

Abstract

LLM-as-a-Judge

Explore similar work

Who can we trust? LLM-as-a-jury for Comparative Assessment

When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratings

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models