CLM-as-a-Judge: Evaluating an Open Contrastive Decision Model on Public Judge Benchmarks

Gowthamkumar Nandakishore

cs.LGOct 5, 2026

CLM-as-a-Judge: Evaluating an Open Contrastive Decision Model on Public Judge Benchmarks

Authors: Gowthamkumar Nandakishore

Organizations: Independent researcher

Abstract

An open contrastive decision model is near chance as a judge on the hard public benchmarks: Contrastive-LM/CLM-v0.1-8B scores between 0.351 (best- of-four, chance 0.250) and 0.593 (pairwise, chance 0.500), is statistically indistinguishable from coin flipping on RM-Bench and JudgeBench, and answers every HaluEval item with one constant label, matching the trivial always-first baseline at 0.581. Judges with the same parameter count score far higher everywhere: a reward model reaches 0.764 to 0.976 and a generative judge 0.611 to 0.778, and every gap to CLM is significant after Benjamini-Hochberg correction. Two properties do work. Raw confidences are overconfident by up to +0.401, yet one pooled temperature fit on held-out calibration items repairs expected calibration error to at most 0.062, and the repaired confidence ranks the model's own errors above chance on three of six benchmarks. The decision order-flip rate is 0.0002 against 0.2188 for the generative judge, and the length-preference shift is -0.023 against -0.217. The confidence-gated cascade, however, escalates between 0.923 and 1.000 of items to the strong judge at the preregistered 0.97 retention bar: calibrated confidence about a near-chance judge has almost nothing to keep. The design: five public preference benchmarks and one hallucination benchmark with real labels, scored under a preregistration frozen before any test item was seen, against generative, reward-model, and trivial baselines, with per-item predictions released.

Figures & tables

Benchmark	CLM (2048)	laya (512)	Generative (4096)	RM (4096)
RewardBench	0.01 / 0.99	0.40 / 0.76	0.00 / 1.00	0.00 / 1.00
RewardBench 2	0.56 / 0.83	0.93 / 0.27	0.09 / 1.00	0.06 / 1.00
RewardBench 2 pairs	0.06 / 1.00	0.77 / 0.50	0.00 / 1.00	0.00 / 1.00
RM-Bench	0.05 / 0.99	0.84 / 0.51	0.01 / 1.00	0.00 / 1.00
JudgeBench	0.15 / 0.97	0.99 / 0.42	0.06 / 0.99	0.06 / 0.99
HaluEval	0.19 / 0.71	0.25 / 0.54	0.19 / 0.71	0.19 / 0.71

Table 1 : Truncated items per judge family and benchmark: fraction of items with any truncated text (left of slash) and fraction of input tokens kept (right), measured with each family’s own tokenizer under the single frozen policy. Source: anc/metrics/gate05_truncation.json .

Judge	RewardBench	RewardBench 2	RB2 pairs	RM-Bench	JudgeBench	HaluEval
CLM	0.549	0.351	0.593	0.508	0.475	0.581
CLM (tuned heads)	0.521	0.387	0.647	0.539	0.535	0.581
laya	0.512	0.280	0.526	0.494	0.498	0.544
Qwen3-8B (generative)	0.778	0.675	0.758	0.676	0.611	0.728
Qwen3-32B (strong, hosted)	0.868	0.732	0.827	0.726	0.615	0.745
Skywork-Reward-V2-8B	0.976	0.876	0.950	0.923	0.764	—

Table 2 : Evaluation accuracy per judge and benchmark (measured). Chance is 0.250 on RewardBench 2 best-of-four and 0.500 on the pairwise benchmarks. The strong judge is hosted; all other judges ran locally. Source: anc/metrics/comparator_table.json , which also carries the BCa 95% intervals.

Benchmark	Identity ECE	Signed gap	Pooled $T$	Per-bench $T$	Isotonic
RewardBench	0.235	+0.235	0.032	0.032	0.015
RewardBench 2	0.402	+0.401	0.061	0.023	0.060
RewardBench 2 pairs	0.283	+0.283	0.049	0.027	0.020
RM-Bench	0.343	+0.343	0.042	0.042	0.028
JudgeBench	0.354	+0.351	0.062	0.083	0.122
HaluEval	0.255	+0.254	0.060	0.009	0.040

Table 3 : Expected calibration error on evaluation items under the four confidence maps, all fit on calibration items only. The identity column also shows the signed confidence-minus-accuracy gap. Source: anc/metrics/calibration_table.json .

Figure 1 : Reliability diagrams for CLM confidence on evaluation items, identity map against the pooled temperature map, per benchmark. Generated by anc/code/build_figures.py from released predictions.

Benchmark	CLM	Strong	Agree	Cascade	Escal.	Retained	Random
RewardBench	0.549	0.868	0.532	0.854	0.933	0.983	0.848
RewardBench 2	0.351	0.732	0.339	0.732	1.000	1.000	0.732
RewardBench 2 pairs	0.593	0.827	0.580	0.818	0.923	0.989	0.809
RM-Bench	0.508	0.760	0.475	0.741	0.954	0.976	0.749
JudgeBench	0.475	0.615	0.482	0.615	0.979	1.000	0.613
HaluEval	0.581	0.745	0.662	0.745	1.000	1.000	0.745

Table 4 : Cascade as agreement with the strong judge, per benchmark: accuracy of each tier alone, rate at which both judges emit the same decision, cascade accuracy at the retention-selected threshold

\tau^{*}

, escalation fraction, retained fraction of strong-judge accuracy, and the random-escalation control at the matched fraction. Strong, Cascade, Random, and Retained all use one convention, the item-level mean on order-1 evaluation items; RM-Bench’s value in Table 2 instead uses the benchmark’s official domain-average aggregate. Sources: anc/metrics/comparator_table.json , cascade_agreement.json , h4_cascade.json .

Figure 2 : Retained accuracy against escalation fraction for the confidence-gated cascade (per grid threshold) and the matched random-escalation control, per benchmark. The horizontal line is the preregistered 0.97 retention bar. Generated by anc/code/build_figures.py .

Benchmark	laya	CLM	Skywork-Reward-V2-8B	Qwen3-8B (generative)
RewardBench	23	338	244	746
RewardBench 2	29	674	558	1,058
RewardBench 2 pairs	24	233	359	898
RM-Bench	22	106	230	780
JudgeBench	32	292	331	937
HaluEval	23	63	—	635

Table 5 : Measured median latency per item in milliseconds for the local judges (20-item probes, first two warm-up items skipped, one Apple M5 Max), shown as context only. The artifact stores medians at one-tenth-millisecond resolution; printed integers round the stored values. Source: anc/metrics/gate03_latency.json .

Explore similar work

Sep 22, 2026cs.AI

JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

LLM-as-a-judge scales evaluation, but reasoning judges are slow and costly. We study JEV-as-a-Judge: evaluation with JEV, a decision-only judge that returns label probabilities instead of text, and whose confidence decides whether to accept its verdict or escalate to a reasoning judge. Against sixteen generative and reward-model judges, with blinded human adjudication, JEV comes within three points of GPT-6 wherever a verdict can be read off the text, at 0.36% of its fee and a 0.15-second median latency, and falls behind where the verdict must be derived, as in math, code, and logic. Its confidence marks this boundary. With a threshold frozen in advance, accepting confident verdicts and escalating the rest is 0.9 points more accurate than GPT-6 on 1,610 held-out pairs at 41% of its fee, and in a pre-specified live test on two new workloads the cascade matches GPT-6's accuracy exactly. Confidence routing weakens on style-adversarial pairs and reference-free prose; we close with a simple recipe for validating thresholds locally.

Yubo Li, Yidi Miao, Ramayya Krishnan +1

Carnegie Mellon University

Large Language Model Judges Judges Acceptance Uncertain Inference Cost

Ask PDF

Oct 5, 2026cs.LG

Will the Judge Flip? Predicting Position-Sensitive LLM Judgments from Residual Stream Activations

The order in which candidate responses are presented can change an LLM judge's verdict. Detecting such a position flip ordinarily requires judging each pair in both orders, which doubles the number of judgments. We investigate whether residual stream activations recorded immediately before the initial verdict can predict a flip. We use nested grouped cross-validation to evaluate regularized linear probes on 534 JudgeBench pairs for three Qwen3 judges and Llama-3.1-8B. The linear probes achieve AUROCs of .621-.850 and outperform a combined baseline that uses verbalized confidence, verdict-label logits, response lengths, and the judge's initial choice by .062-.113 AUROC. Linear probes trained on JudgeBench and then frozen achieve AUROCs of .685-.853 on 1,802 MT-Bench comparisons without MT-Bench fitting or recalibration. These results show that pre-verdict activations support prediction of susceptibility to candidate order and outperform the non-activation predictors evaluated here.

Hashmath Shaik, Gnaneswar Villuri, Alex Doboli

Department of Electrical and Computer Engineering Stony Brook University Stony Brook, NY 11794, USA

Topic pending

Ask PDF

Apr 26, 2026cs.CL

JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems

Large language models are widely adopted as automated evaluation judges, yet the stability of their verdicts under semantically equivalent prompt rephrasings remains largely unexamined. We conduct a systematic empirical study of prompt-induced decision instability across multiple evaluation tasks and judge architectures. To facilitate this analysis, we release JudgeSense, a benchmark comprising hand-validated prompt-paraphrase pairs spanning factuality, coherence, relevance, and preference, drawn from established NLP benchmarks and accompanied by comprehensive decision logs. The benchmark enables the measurement of judge stability across equivalent prompts, allowing researchers to assess whether stability correlates with model scale or instruction-tuning, and to identify which tasks are most sensitive to prompt wording. Our evaluation reveals that coherence remains the primary task for distinguishing judge behavior, while factuality judgments demonstrate high stability under standard conditions. Pairwise evaluation tasks consistently exhibit position bias. Crucially, we find that model scale is not a reliable proxy for consistency; notably, as an interesting result in our analysis, the largest and newest models are not the most consistent.

Rohith Reddy Bellibatlu, Edward Raff, Wenbin Zhang

Florida International University, Miami, FL · CrowdStrike, New York, USA · University of Maryland Baltimore County, Baltimore, USA

Llm-As-A-Judge Judges Legal Natural Language Processing Benchmarks

Ask PDF