Large Language Models (LLMs) are increasingly used as automated judges, yet decoding temperature is often set by intuition rather than evidence. We run over one million large scale evaluations, multiple open-source models, prompting strategies, and judging paradigms to examine how temperature affects judge reliability. Low temperatures yield consistent outputs with few errors, especially in smaller models, while higher temperatures reduce consistency and increase errors. Yet consistency alone does not guarantee sound judgments: low temperatures can narrow the range of reasoning explored. We characterize judge behavior through four reasoning personas and find that temperature shifts their distributions in model-dependent ways, most visibly in ambiguous cases where competing answers are hard to distinguish. Repeated high-temperature sampling reveals this hidden uncertainty, and combining it with majority voting or using high temperature for reasoning and low temperature for the final decision, improves agreement on ambiguous cases without hurting clear ones, at the cost of extra inference. Our findings suggest a simple rule: keep temperature low for stable outputs, but explore more when the judgment is hard.
Figures & tables
Figure 1: LLM-as-a-Judge Paradigm
Figure 2: Pearson correlation matrices between decoding temperature (T) and evaluation metrics—agreement (A), consistency (C), and error rate (E)—across four LLMs.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Comprehensive ATE Analysis Across Moderators, Temperature, and Feature Importance. Row 1: Temperature trends of different models (shared legend), showing mean tendencies across temperature ranges (lines indicate averages, not ATE values). Row 2: Boxplots of Average Treatment Effect (ATE) distributions across three moderators for each outcome metric (dashed line denotes ATE = 0). Row 3: SHAP beeswarm plots ranking feature importance by impact magnitude for each outcome metric. All panels share consistent bold styling and bright color palettes for clarity.
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.