Adaptive Rigor in AI System Evaluation using Temperature-Controlled Verdict Aggregation via Generalized Power Mean

Aleksandr Meshkov

cs.CLApr 4, 2026

Adaptive Rigor in AI System Evaluation using Temperature-Controlled Verdict Aggregation via Generalized Power Mean

Authors: Aleksandr Meshkov

Organizations: Independent Researcher, Budva, Montenegro.

Abstract

Existing evaluation methods for LLM-based AI systems, such as LLM-as-a-Judge, verdict systems, and NLI, do not always align well with human assessment because they cannot adapt their strictness to the application domain. This paper presents Temperature-Controlled Verdict Aggregation (TCVA), a method that combines a five-level verdict-scoring system with generalized power-mean aggregation and an intuitive temperature parameter T [0.1, 1.0] to control evaluation rigor. Low temperatures yield pessimistic scores suited for safety-critical domains; high temperatures produce lenient scores appropriate for conversational AI. Experimental evaluation on three benchmark datasets with human Likert-scale annotations (SummEval and USR) shows that TCVA achieves correlation with human judgments comparable to RAGAS on faithfulness (Spearman = 0.667 vs. 0.676) while consistently outperforming DeepEval. The method requires no additional LLM calls when adjusting the temperature parameter.

Figures & tables

Level	Weight $w$	Semantics	Example of usage
Fully	1.0	Fully satisfied	The atomic statement (part of the AI response text) is fully supported by facts from the context of the AI system
Mostly	0.9	Mostly satisfied	The atomic statement is completely based on facts, but the structure of the text is slightly changed
Partially	0.7	Partially satisfied	The atomic statement is about half generated from facts and half made up by the AI model, but overall the answer is still relevant to the query
Minor	0.3	Minimally affected	The atomic statement is not explicitly confirmed in the facts used for generation, but a number of phrases or individual words are still present in the facts
None	0.0	Not satisfied	There is no connection between the generated atomic statement and the facts used by the AI model to generate the answer

Table 1: Five-level verdict system

Figure 1: Five-level verdict scale with non-uniform gaps between semantic levels.

$p$	Name	Function	Characteristics
$-\infty$	Minimum	$\min(x_{1},\ldots,x_{n})$	Extreme pessimism: only the worst value is taken into account
$-2$	—	$\left(\frac{1}{n}\sum_{i=1}^{n}x_{i}^{-2}\right)^{-1/2}$	Strong shift to minimum
$-1$	Harmonic	$\frac{n}{\sum_{i=1}^{n}1/x_{i}}$	Shift to small values
$0$	Geometric	$\sqrt[n]{x_{1}\cdot\ldots\cdot x_{n}}$	Multiplicative mean
$1$	Arithmetic	$\frac{1}{n}\sum_{i=1}^{n}x_{i}$	Standard average
$2$	Quadratic	$\sqrt{\frac{1}{n}\sum_{i=1}^{n}x_{i}^{2}}$	Root Mean Square

Table 2: Special cases of power mean

Figure 2: Effect of the power parameter $p$ on the aggregated score $M_{p}(w)$ for $w=[1.0,0.9,0.7]$ .

$p$	$M_{p}(w)$	Interpretation
$-10$	0.773	Approximately $\min=0.7$ , strict estimate
$-5$	0.809	Strong bias towards the minimum
$-2$	0.838	Noticeable bias towards the minimum
$-1$	0.848	Harmonic mean
$0$	0.857	Geometric mean
$1$	0.867	Arithmetic mean

Table 3: Influence of parameter

p

on the final estimate

$T$	$p$	Type of mean	Explanation
0.1	$-8$	Close to minimum	Maximum strict, even one bad verdict is critical
0.3	$-3.5$	Close to Harmonic	Strict, focus on worst verdicts
0.5	$1$	Arithmetic	Balanced, each verdict has equal contribution
0.7	$5.5$	Quadratic	Forgiving, focus on best verdicts
0.9	$10.0$	Close to maximum	Maximum lenient, good verdicts dominate the final score

Table 4: Mapping temperature to power-law mean

Figure 3: The complete TCVA evaluation pipeline

Method	SummEval	SummEval-Rel	USR
	(Faithfulness)	(Relevancy)	(Dialogue)
	$N=172$	$N=196$	$N=198$
TCVA ( $T=0.2$ )	0.642	0.476	0.153
TCVA ( $T=0.3$ )	0.642	0.476	0.153
TCVA ( $T=0.5$ )	0.663	0.480	0.168
TCVA ( $T=0.7$ )	0.664	0.452	0.173

Table 5: Spearman’s

\rho

correlation with human judgments. Best result per dataset in bold . TCVA shows the best-performing temperature.

	SummEval		SummEval-Rel		USR
Method	$\tau$	MAE	$\tau$	MAE	$\tau$	MAE
TCVA (best $T$ )	0.527	0.356	0.369	0.211	0.143	0.501
RAGAS	0.535	0.259	0.333	0.365	0.140	0.516
DeepEval	0.302	0.325	0.239	0.352	$-0.046$	0.367

Table 6: Kendall’s

\tau

and MAE for the best-performing configurations.

Figure 4: Spearman’s $\rho$ comparison across datasets. TCVA achieves comparable results to RAGAS on faithfulness and outperforms it on relevancy.

Figure 5: Effect of temperature $T$ on Spearman’s $\rho$ . Dashed lines show RAGAS baselines. Optimal temperature varies by evaluation dimension.

Weight scheme	SummEval	SummEval-Rel	USR
Default	0.667	0.480	0.226
Linear	0.668	0.472	0.230
Aggressive	0.664	0.492	0.226
Conservative	0.669	0.478	0.230
$\Delta$ (variation)	0.005	0.020	0.004

Table 7: Weight sensitivity analysis: Spearman’s

\rho

across weight schemes.

\Delta

denotes the range (max

-

min) across schemes.

Figure 6: Weight sensitivity: Spearman’s $\rho$ remains stable across four weight schemes ( $\Delta<0.02$ ), confirming that results are not an artifact of a specific weight choice.

Dataset	Method	$\rho$	95% CI	Paired test
SummEval	TCVA (best $T$ )	0.667	[0.561, 0.755]	$\Delta=-0.009$ , $p=0.759$
	RAGAS	0.676	[0.571, 0.762]	$\Delta=-0.009$ , $p=0.759$
	DeepEval	0.395	[0.251, 0.525]
SummEval-Rel	TCVA (best $T$ )	0.480	[0.367, 0.583]	$\Delta=+0.062$ , $p=0.041^{*}$
	RAGAS	0.411	[0.286, 0.528]	$\Delta=+0.062$ , $p=0.041^{*}$
	DeepEval	0.315	[0.183, 0.439]

Table 8: Bootstrap 95% confidence intervals for Spearman’s

\rho

and paired significance tests (TCVA vs. RAGAS).

Figure 7: Bootstrap 95% confidence intervals for Spearman’s $\rho$ . On SummEval, TCVA and RAGAS CIs overlap fully ( $p=0.759$ ). On SummEval-Rel, TCVA significantly outperforms RAGAS ( $p=0.041$ ).

	SummEval		SummEval-Rel		USR
Configuration	$\rho$	$\Delta$	$\rho$	$\Delta$	$\rho$	$\Delta$
A. Full TCVA	0.667	—	0.480	—	0.226	—
B. w/o Penalty	0.610	$-0.057$	0.485	$+0.005$	0.221	$-0.005$
C. Arithmetic Mean	0.663	$-0.004$	0.480	$0.000$	0.218	$-0.008$
D. Binary Verdicts	0.615	$-0.052$	0.236	$-0.244$	0.256	$+0.030$

Table 9: Ablation study: Spearman’s

\rho

when removing individual TCVA components. Deltas are relative to the full method (Config A).

Figure 8: Ablation study: removing the None-penalty primarily affects faithfulness ( $-0.057$ ), while collapsing to binary verdicts devastates relevancy ( $-0.244$ ).

Figure 9: TCVA scores ( $T=0.5$ ) vs. human consistency ratings on SummEval. The positive trend confirms that TCVA rankings align with human judgments.

Automated Evaluation

Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.

See all 396 papers

Necessary but Not Sufficient: Temperature Control and Reproducibility in LLM-as-Judge Safety Evaluations
Jun 24, 2026Hiroki TambaLLM Evaluation LLM-as-a-Judge
Tamba Research Academy
CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation
Feb 9, 2026Jitian Zhao, Changho Shin, Tzu-Heng Huang +2LLM Evaluation LLM-as-a-Judge
Department of Statistics, University of Wisconsin–Madison, Madison, WI, USA · Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI, USA.
Towards Provably Unbiased LLM Judges via Bias-Bounded Evaluation
Mar 5, 2026Benjamin Feuer, Lucas Rosenblatt, Oussama ElachqarBias in LLM-as-a-Judge LLM-as-a-Judge
1Stanford University · 3New York University

Jun 24, 2026J/K move · Enter open · S save

Necessary but Not Sufficient: Temperature Control and Reproducibility in LLM-as-Judge Safety Evaluations

Hiroki Tamba

Tamba Research Academy

LLM Evaluation LLM-as-a-Judge LLM Reliability LLM Safety Evaluation Benchmark Reproducibility

Open paper AskPDF arXiv

LLM-as-judge ("grader") components are now standard in evaluation harnesses, including safety evaluations where a pass/fail verdict may gate downstream deployment decisions. A widespread assumption is that setting the grader's sampling temperature to 0 makes grading deterministic. We test this assumption against a real safety-evaluation codebase (Japan AISI's open-source aisev) and show it fails on two levels. First, the harness invokes its grader without setting temperature or seed; the underlying provider silently applies its default of 1.0, so items near the decision boundary flip pass/fail across identical runs (per-item disagreement up to ~50% over 20 runs). Second, pinning temperature=0 reduces but does not eliminate flips: across 690 API calls spanning two providers, three model tiers, and five sampling configurations, 1-2 of 7 borderline items remain non-reproducible even under forced greedy decoding (top_k=1). Claude Opus 4.7/4.8 has since deprecated temperature entirely, rendering the primary mitigation inapplicable to newer model generations. These findings expose a structural gap: evaluation harnesses that report single-run verdicts without variance or grader-disagreement metrics can present noise as a safety property. We release a reproduction harness (690 calls, 7 conditions) and recommend that harnesses treat grader disagreement as a first-class health metric alongside the scores themselves.

Abstract

Automated Evaluation

Explore similar work

Necessary but Not Sufficient: Temperature Control and Reproducibility in LLM-as-Judge Safety Evaluations

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

Towards Provably Unbiased LLM Judges via Bias-Bounded Evaluation