PRISM: Calibrating Black-Box Judges with Counterfactual Probes

Kakei Yamamoto; So Kuroki

cs.LGOct 13, 2026

PRISM: Calibrating Black-Box Judges with Counterfactual Probes

Authors: Kakei Yamamoto, So Kuroki

Organizations: EECS, Massachusetts Institute of Technology · Sakana AI

Abstract

Large language models are increasingly used as judges to reduce reliance on costly human evaluation, yet their scores can systematically disagree with human judgments. We ask whether responses to an explicitly imposed evaluative bias can help correct a judge's implicit scoring biases. We introduce PRISM (Probe-based Residual Inference of Scoring Misalignment), which holds the evaluated content fixed and measures score changes when attributes are emphasized or de-emphasized. A local sensitivity model motivates representing these responses as a surrogate Jacobian. Ridge regression on human labels learns a correction to the baseline score. The resulting scoring protocol is rubric-free: it uses overall scores without requesting criterion-level ratings. In the main Jev and Sol comparisons on SummEval summarization and HANNA story evaluation, PRISM and its learned-baseline variant significantly improve agreement with human rankings over uncalibrated scores while remaining competitive with rubric-based methods. A controlled SummEval comparison yields similar agreement for rubric-derived quality probes and style probes, supporting calibration without reproducing rubric criteria. The learned coefficients reveal shared and opposing correction directions across judges.

Figures & tables

Figure 1:

\mathtt{PRISM}

overview. (1) Holding the evaluated content fixed, we ask the judge to change its emphasis on individual attributes. Differences from the Vanilla score form the surrogate Jacobian

\widehat{J}_{x}

: the illustrated scores

6,5,4

give responses

+1,0,-1

relative to the baseline

5

. Only increased-emphasis probes are pictured; the method retains both increased- and decreased-emphasis responses. (2) The tangent plane illustrates the local sensitivity interpretation: a small change in evaluative weights produces an approximately linear score change. (3) Ridge regression on human-labeled examples learns shared correction coefficients

\widehat{\beta}

; new inputs are scored as

\widehat{y}=s^{0}-\widehat{J}_{x}\widehat{\beta}

using only baseline and probed judge outputs. The scores, surface, and displacement

-\widehat{\beta}

are schematic, rather than recovered internal model quantities.

Panel of Table 1 — Table 1: Recorded coverage-probe example. The article is excerpted, but all three calls received the same full article and candidate summary.

Panel of Figure 2 — Figure 2: Probe signal and presentation-order sensitivity for Jev on 100 SummEval documents. (a) Probe RMS relative to Vanilla repeat variation; the vertical band pools properties. (b) Probe-response correlation between disjoint sets of averaged candidate orders, using length and formality probes of both signs. The blue reference uses repeated pointwise calls. (c) Uncalibrated Spearman gain over pointwise scoring. Joint scoring here is batched pointwise: one rating per candidate, in one call per order, versus 16 single-candidate calls per document; budgets differ. Bars and bands are 95% document-bootstrap intervals. These diagnostics use separate runs from Table 2 ; protocols appear in sections B.6 and B.2 .

Panel of Figure 3 — Figure 3: Weight instructions for Jev on SummEval. (a) Responses projected onto each attribute’s standard half-difference. Zero requests the usual weight; $\pm 1$ , $\pm 2$ , and $\pm 3$ denote slightly , unmodified, and much more/less weight. These are wording categories, not numerical steps. (b) Reproducible RMS of $O$ and $E^{\mathrm{c}}$ , pooled over the same six properties. Both panels use attribute-specific controls and cross-products between ten repeated runs on the same 100 documents; bars and bands are 95% document-bootstrap intervals ( section B.4 ).

Rubric-based scoring
	SummEval		HANNA
Method	Jev	Sol	Jev	Sol
SAJA broad	$0.703$ [-1pt] $[0.672,0.732]$	$0.659$ [-1pt] $[0.627,0.688]$	$0.524$ [-1pt] $[0.484,0.565]$	$0.443$ [-1pt] $[0.390,0.492]$
SAJA granular	$\mathbf{0.745}$ [-1pt] $[0.719,0.770]$	$0.707$ [-1pt] $[0.679,0.732]$	$0.549$ [-1pt] $[0.508,0.590]$	$0.528$ [-1pt] $[0.482,0.574]$
G-Eval	$0.701$ [-1pt] $[0.670,0.729]$	$\mathbf{0.721}$ [-1pt] $[0.696,0.745]$	$0.528$ [-1pt] $[0.483,0.575]$	$\mathbf{0.587}$ [-1pt] $[0.543,0.632]$
Rubric-free scoring

Table 2: Agreement with human judgments on SummEval and HANNA. Cells show mean within-document (SummEval) or within-prompt (HANNA) Spearman correlation (higher is better), with 95% document/prompt-bootstrap intervals. Bold and underline mark the best and second-best displayed means, including ties. Groups distinguish scoring protocols, not probe provenance;

\mathtt{PRISM}

fixes

\alpha=1

and

\alpha\text{-}\mathtt{PRISM}{}

learns it. Full four-judge results appear in tables 13 and 14 .

		Rubric-based		Rubric-free
Benchmark	$n$	SAJA broad	SAJA granular	Vanilla	$\mathtt{PRISM}$
Topical-Chat	34	$0.799$	$0.831$	$\underline{0.835}$	$\mathbf{0.837}$
HelpSteer2	188	$0.665$	$0.710$	$\underline{0.739}$	$\mathbf{0.755}$
MT-Bench	81	$0.808$	$0.805$	$\underline{0.818}$	$\mathbf{0.828}$
JudgeBench	287	$0.676$	$0.707$	$\underline{0.789}$	$\mathbf{0.791}$

Table 3: Accuracy

\uparrow

on additional benchmarks, using Jev. Groups follow table 2 .

\mathtt{PRISM}

uses every signal-screened unit from the expanded pool.

n

counts confirmation contexts with non-tied reference pairs; predicted ties receive half credit. Bold and underline mark the best and second-best displayed values, including rounding ties. All variants and paired comparisons appear in section A.5 .

Panel of Figure 4 — Figure 4: Correction profiles and robustness on SummEval. (a) Normalized directional components of $\widehat{\beta}$ for Jev/Sol; bars span the five overlapping training folds, not confidence intervals. Positive coefficients subtract the response. (b) Jev’s agreement at fixed relative penalties $\gamma$ ( $\lambda$ multiplied by a training-fold constant); bands are 95% document-bootstrap intervals. Main results select penalties inside training folds. (c) Median gains over Vanilla across all probe subsets of each size; bands are 10th–90th percentiles across subsets, not confidence intervals. Dashed lines mark Vanilla.

Table 4: Ablations on SummEval: mean within-document Spearman and 95% document-bootstrap intervals. (a) All representations use the same calls and folds, with the Vanilla coefficient fixed at one. Diagonal Taylor fits a second-order correction; paired

\mathtt{PRISM}

already contains its second-difference inputs ( section D.1 ). (b) Jev with pointwise scoring. Parentheses give attribute counts; Combined is the union. Predictor Vanilla scores, folds, and calibration are held fixed.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Judge	Recorded model identifier	Reasoning effort	Temperature Vanilla/ $\mathtt{PRISM}$ /SAJA	Temperature G-Eval
Jev	typesafe/jev-1.13-20260917	Not set	–	–
Sonnet	anthropic/claude-sonnet-4.5	Not set	$0$	$1$
Sol	gpt-5.6-sol	none	$0$	$1$
Gemma	google/gemma-4-31b-it	Not set	$0$	$1$

Appendix

Table 5: Model versions and requested inference settings across experiments. “Not set” and – denote omitted API fields, not confirmed provider defaults. Sol’s OpenRouter identifier adds the prefix openai/ . Jev returns typed-rating probabilities; the other judges use chat completions with top_p=1 . Probe diagnostics use the corresponding Vanilla/

\mathtt{PRISM}

settings.

Judge	Probe-score format	Attributes	Conditions	Calls/document
Jev	Individual expected ratings	$8$	$38$	$16$
Sonnet	Joint rankings; two orders	$7$	$34$	$68$
Sol	Joint integer ratings	$8$	$38$	$38$
Gemma	Joint rankings; two orders, two blocks	$7$	$34$	$136$

Appendix

Table 6: Nominal acquisition designs for the score conditions used in the calibration analysis. Conditions include five neutral clauses, which are not columns of the paired-response predictor. Additional surface perturbations and Section 2 diagnostics are excluded. Each SAJA variant uses one rubric call per summary, or 16 calls per document. Query and token budgets are not matched.

Judge	Quality bank (8 attributes)	Style bank (7 attributes)
Jev	Primary: expected ratings	Different bank: 8 attributes, 6 shared
Sonnet	Additional: joint ratings	Primary: joint rankings
Sol	Primary: joint ratings	Additional: joint rankings
Gemma	Not found in inspected records	Primary: joint rankings

Appendix

Table 7: Probe-bank coverage in the inspected SummEval records. Primary denotes the source of the main result and correction profile; additional denotes an existing run outside that comparison. All listed runs cover 100 documents. These records do not constitute a matched bank-by-judge experiment; presentation and repetition counts also differ.

Judge	SummEval: 6,400 criterion scores	HANNA: 6,336 criterion scores
Jev	Expected rating from typed probabilities; no imputation.	Expected rating from typed probabilities; no imputation.
Sonnet	First 10 valid draws, at most 20 attempts per item. All scores available; 97 items average 5–9 valid draws.	First 10 valid draws per item; all items complete.
Sol	Top-5 token probabilities for 6,229 items; sampling fallback for 168 (up to 20 valid draws in 40 attempts); 3 midpoint imputations. Of sampled items, 53 reach the cap below 20 valid draws.	Score-token probabilities for all items; no sampling fallback or imputation.
Gemma	Top-20 token probabilities, with up to 5 retries; fallback to 10 valid draws in 20 attempts. 2,128 midpoint imputations, including 1,525 fluency items.	First 10 valid draws per item; all items complete.

Appendix

Table 8: Recorded G-Eval acquisition protocols. Midpoint imputation fills items with no valid score; it does not recover the missing judgment. These are model-specific adaptations of the published procedure, with unmatched sampling budgets across methods.

Benchmark	Explore	Confirm	Eval. pairs	Units	Columns
HANNA	43	53	2860	74	126
Topical-Chat	26	34	468	82	134
HelpSteer2	251 (185)	268 (188)	188	85	137
MT-Bench	79	81	1280	79	131
JudgeBench	333	287	287	84	136

Appendix

Table 9: Recorded splits and primary response dimensions. Parentheses give the eligible contexts after excluding reference ties from pairwise evaluation; all other contexts contribute evaluation pairs. MT-Bench counts individual human judgments, so a response pair may contribute multiple labels. All candidates have parsed scores in every primary measurement bank.

Data source	Reported license
SummEval mirror	MIT (dataset card)
HANNA release	MIT (repository)
HelpSteer2	CC-BY-4.0
MT-Bench judgments	CC-BY-4.0
JudgeBench	MIT
Topical-Chat base data	CDLA-Sharing 1.0

Appendix

Table 10: Release-level data notices, checked on 13 October 2026. Linked sources distinguish the underlying Topical-Chat data from the additional USR judgments.

Attribute	Attribute clause	$s^{+}$	$s^{-}$	$\Delta^{+}$	$\Delta^{-}$
Grounding	whether the summary’s claims are supported by the source article	$7$	$6$	$+1$	$0$
Fluency	the grammar, spelling and word choice of the summary	$7$	$7$	$+1$	$-1$
Accuracy	whether the summary’s claims are factually correct	$6$	$6$	$0$	$0$

Appendix

Table 11: Additional recorded Sol examples for the same candidate as table 1 , with

s^{0}=6

throughout. Here

s^{\pm}=s(x;p_{k}^{\pm})

\Delta^{+}=s^{+}-s^{0}

, and

\Delta^{-}=s^{0}-s^{-}

. Grounding changes one side, fluency moves both scores upward, and accuracy leaves both unchanged. These illustrate response patterns for one input, not average effects of the attributes.

Figure 5: Probe displacements relative to neutral instructions for Jev-1.13 on the same 100 SummEval documents as fig. 2 . Each point divides a property’s RMS displacement, pooled over both signs and three runs, by the RMS displacement of five neutral clauses. All displacements are centered within document. The dashed line marks the neutral reference; the vertical band pools all 14 properties. Bars and the band are 95% document-bootstrap intervals.

Figure 6: Repeatability of centered displacements on the same three Jev runs as fig. 2 . Points are question-level correlations; diamonds and bars summarize the mean and its 95% document-bootstrap interval. Vanilla scores have repeat correlation

0.997

, which measures score stability, not displacement repeatability.

Panel of Figure 7 — Figure 7: Joint-instruction diagnostics. Gray points correspond to all 15 attribute pairs; diamonds pool their second moments. (a) Projection of joint-instruction main effects onto the corresponding single-probe half-differences. The reference at one indicates the same projected effect. (b) RMS of the interaction divided by the root mean square of the two main effects. Bars on the pooled estimates are 95% document-bootstrap intervals. Neither panel assumes a numerical step size for the instructions.

Figure 8: Pairwise judgments relative to pointwise scoring for Jev-1.13 on SummEval, before calibration. Each point is the mean within-document Spearman difference from the pointwise reference; bars are paired 95% document-bootstrap intervals. The two conditions use one presentation order per pair or average both orders.

Benchmark	RMS ratio	Reproducibility probe / neutral	Kendall reverse / repeat	Even/odd at $\pm 1$ (smooth reference)
HANNA	$3.12$	$0.90/0.56$	$0.57/0.92$	$0.50\;(0.19)$
Topical-Chat	$2.24$	$0.85/0.44$	$0.79/0.95$	$0.89\;(0.62)$
HelpSteer2	$2.53$	$0.88/0.55$	$0.62/0.97$	$0.74\;(0.36)$
MT-Bench	$2.48$	$0.87/0.47$	$0.72/0.96$	$0.71\;(0.25)$
JudgeBench	$2.46$	$0.86/0.53$	$0.70/0.97$	$0.99\;(0.24)$

Appendix

Table 12: Probe diagnostics on the confirmation split from the 24-unit study. The signal ratio pools its 16 properties; reproducibility compares independent calls. Kendall compares reversed candidate order with an unchanged-order repeat in joint scoring. The final column uses four diagnostic properties chosen on exploration data and the unchanged-weight instruction as the zero of the wording ladder; reproducible components are estimated by between-run cross-products. These are separate diagnostic calls, not the 89-unit calibration pool.

Rubric-based scoring
Method	Jev	Sonnet	Sol	Gemma
SAJA broad	$0.703$ [-1pt] $[0.672,0.732]$	$0.657$ [-1pt] $[0.624,0.689]$	$0.659$ [-1pt] $[0.627,0.688]$	$0.687$ [-1pt] $[0.657,0.716]$
SAJA granular	$\mathbf{0.745}$ [-1pt] $[0.719,0.770]$	$0.671$ [-1pt] $[0.637,0.705]$	$0.707$ [-1pt] $[0.679,0.732]$	$0.675$ [-1pt] $[0.643,0.705]$
G-Eval	$0.701$ [-1pt] $[0.670,0.729]$	$\mathbf{0.712}$ [-1pt] $[0.683,0.739]$	$\mathbf{0.721}$ [-1pt] $[0.696,0.745]$	$0.682^{\dagger}$ [-1pt] $[0.647,0.716]$
Rubric-free scoring
Vanilla	$0.654$ [-1pt] $[0.615,0.691]$	$0.645$ [-1pt] $[0.607,0.681]$	$0.693$ [-1pt] $[0.661,0.722]$	$0.672$ [-1pt] $[0.639,0.708]$

Appendix

Table 13: Full four-judge results on SummEval, measured by mean within-document Spearman correlation (higher is better). Brackets give 95% document-bootstrap confidence intervals. Bold and underline mark the best and second-best displayed means, including ties. Groups distinguish scoring protocols, not probe provenance;

\mathtt{PRISM}

fixes

\alpha=1

and

\alpha\text{-}\mathtt{PRISM}{}

learns it.

\dagger

Gemma’s G-Eval includes midpoint imputation for 33.3% of criterion scores ( section A.4 ).

Rubric-based scoring
Method	Jev	Sonnet †	Sol	Gemma †
SAJA broad	$0.524$ [-1pt] $[0.484,0.565]$	$0.389$ [-1pt] $[0.333,0.445]$	$0.443$ [-1pt] $[0.390,0.492]$	$0.364$ [-1pt] $[0.309,0.414]$
SAJA granular	$0.549$ [-1pt] $[0.508,0.590]$	$\mathbf{0.500}$ [-1pt] $[0.450,0.548]$	$0.528$ [-1pt] $[0.482,0.574]$	$0.466$ [-1pt] $[0.410,0.517]$
G-Eval	$0.528$ [-1pt] $[0.483,0.575]$	$\underline{0.497}$ [-1pt] $[0.452,0.543]$	$\mathbf{0.587}$ [-1pt] $[0.543,0.632]$	$\mathbf{0.497}$ [-1pt] $[0.449,0.544]$
Rubric-free scoring
Vanilla	$0.459$ [-1pt] $[0.409,0.505]$	$0.432$ [-1pt] $[0.384,0.478]$	$0.462$ [-1pt] $[0.428,0.497]$	$0.437$ [-1pt] $[0.398,0.474]$

Appendix

Table 14: Full four-judge results on HANNA: mean within-prompt Spearman correlation (higher is better), with 95% prompt-bootstrap intervals. All methods use the same 96 prompts; fitted methods share five outer folds. SAJA uses 1–10 scores. Vanilla/probe scores use 0–100 for Sonnet, 1–10 for Sol and Gemma, and expected ratings for Jev. G-Eval uses six HANNA criteria with model-specific response-format adjustments ( section A.4 ). Bold and underline mark the best and second-best displayed means.

\dagger

Sonnet and Gemma include mean-imputed failed scores.

Benchmark	$\Delta$ Vanilla	$\Delta$ SAJA broad
HANNA	$+0.063\;[+0.041,+0.085]$	$+0.041\;[+0.019,+0.062]$
Topical-Chat	$+0.002\;[+0.000,+0.005]$	$+0.038\;[-0.003,+0.079]$
HelpSteer2	$+0.016\;[-0.003,+0.040]$	$+0.090\;[+0.027,+0.154]$
MT-Bench	$+0.010\;[-0.011,+0.033]$	$+0.021\;[-0.013,+0.059]$
JudgeBench	$+0.002\;[-0.003,+0.009]$	$+0.115\;[+0.057,+0.171]$

Appendix

Table 15: Primary

\mathtt{PRISM}

minus Vanilla and SAJA broad, with paired 95% context-bootstrap intervals. The 2,000 resamples hold fitted models and selected banks fixed; intervals are unadjusted for the family of reported comparisons. Topical-Chat’s interval against the Vanilla has an exact lower endpoint of zero.

Relative penalty $\gamma$	Jev	Sonnet	Sol	Gemma
$10^{-3}$	$0.7001$	$0.6842$	$0.7084$	$0.6952$
$10^{-2}$	$0.6996$	$0.6849$	$0.7079$	$0.6943$
$10^{-1}$	$0.7047$	$0.6846$	$0.7081$	$0.6949$
$1$	$0.7016$	$0.6822$	$0.7062$	$0.6922$
$10$	$0.6768$	$0.6561$	$0.6984$	$0.6810$
$100$	$0.6575$	$0.6479$	$0.6952$	$0.6752$

Appendix

Table 16: Regularization sensitivity of paired-response calibration: mean within-document Spearman correlation on the same outer folds. A finite, strongly shrunk correction can still break baseline ties, so its rank correlation need not equal the baseline’s.

Benchmark	CV maximizers $\gamma$	Selected $\gamma$
HANNA	$\{1\}$	$1$
Topical-Chat	$\{1000\}$	$1000$
HelpSteer2	$\{0.01,0.1,10,100\}$	$100$
MT-Bench	$\{1\}$	$1$
JudgeBench	$\{100,1000\}$	$1000$

Appendix

Table 17: Regularization selected on exploration data for each primary expanded-bank fit. The strongest penalty among exact CV-score maximizers is selected; confirmation scores do not enter this selection.

Quality bank (eight attributes)
Attribute	Exact attribute clause inserted in the templates
Accuracy	whether the summary’s claims are factually correct
Grounding	whether the summary’s claims are supported by the source article
Coverage	how much of the article’s important content the summary includes
Coherence	how well-organised the summary is and how it flows
Fluency	the grammar, spelling and word choice of the summary

Appendix

Table 18: Complete probe banks in the matched Jev–SummEval robustness comparison of table 4(b) . Quality includes rubric-derived criteria; Style uses eight distinct named properties without those explicit criteria. Distinct names do not imply that style is statistically independent of quality. This eight-attribute style bank differs from the seven-attribute Sonnet/Gemma bank in section A.2 .

Judge	Full bank	After one omission	Four-probe gain	Above baseline
Jev	$0.705$	$[0.687,0.706]$	$+0.043$	$70/70$
Sonnet	$0.684$	$[0.678,0.686]$	$+0.036$	$35/35$
Sol	$0.706$	$[0.704,0.710]$	$+0.011$	$68/70$
Gemma	$0.695$	$[0.691,0.697]$	$+0.020$	$35/35$

Appendix

Table 19: Probe-selection robustness. Full-bank and omission values are mean within-document Spearman correlations; the omission brackets give the full range across removed attributes. The four-probe gain is the median across all four-attribute subsets. The last column counts subsets whose mean exceeds the Vanilla judge, without asserting significance for every subset.

Figure 9: Directional corrections for Sonnet and Gemma. Points are mean normalized coefficients

a_{k}

; bars span five overlapping folds. Positive coefficients subtract the response, negative ones add it. This bank and elicitation format differ from Jev/Sol in fig. 4 (a); see table 7 . A variable sign does not imply no effect.

Judge	Attribute	Directional $a_{k}$	Shared $e_{k}$
Jev	Accuracy	$-0.095\;[-0.122,-0.082]$	$0.004\;[-0.007,0.016]$
	Coherence	$-0.003\;[-0.018,0.004]$	$0.003\;[-0.015,0.021]$
	Coverage	$0.062\;[0.039,0.078]$	$-0.018\;[-0.024,-0.008]$
	Engagement	$-0.042\;[-0.054,-0.033]$	$-0.096\;[-0.119,-0.073]$
	Fluency	$-0.087\;[-0.105,-0.055]$	$-0.144\;[-0.151,-0.134]$
	Grounding	$-0.021\;[-0.029,-0.013]$	$0.046\;[0.039,0.054]$

Appendix

Table 20: Full profiles of the paired-response correction. Each entry is the mean over five training folds followed by the minimum and maximum in brackets; these ranges are not confidence intervals. Positive contributions are subtracted in the deployed score.

Rubric-based
Method	HANNA	Topical-Chat	HelpSteer2	MT-Bench	JudgeBench
SAJA broad	$0.682$	$0.799$	$0.665$	$0.808$	$0.676$
SAJA granular	$0.695$	$0.831$	$0.710$	$0.805$	$0.707$
Rubric-free
Vanilla	$0.659$	$0.835$	$0.739$	$0.818$	$0.789$
$\mathtt{PRISM}$ : all passing (primary)	$0.723$	$0.837$	$0.755$	$0.828$	$0.791$

Appendix

Table 21: Probe-pool strategy comparisons (accuracy

\uparrow

). Every fitted predictor uses exploration labels only, with the penalty selected separately by grouped cross-validation and exact ties favoring stronger regularization. The primary bank retains all signal-screened units.

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 555 papers

From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges
Jan 13, 2026Yihan Hong, Huaiyuan Yao, Bolin Shen +3LLM-as-a-Judge Rubric-Based Evaluation
Washington University in St. Louis, St. Louis, MO, USA · Arizona State University, Tempe, AZ, USA · Florida State University, Tallahassee, FL, USA
Small Language Models as Judges for Rubric-Based Reinforcement Learning
Aug 30, 2026Fengyu Xie, Yilun Zhao, Bingsen Chen +2LLM-as-a-Judge Language Model Generation Evaluation
New York University · Yale University
Self-Evaluation Is Already There: Eliciting Latent Judge Calibration in Base LLMs with Minimal Data
Jun 3, 2026XiuYu Zhang, Yi Shan, Junfeng Fang +1LLM Evaluation LLM-as-a-Judge
National University of Singapore · Beijing University of Technology

Jan 13, 2026J/K move · Enter open · S save

From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges

Yihan Hong, Huaiyuan Yao, Bolin Shen, Wanpeng Xu+2

Washington University in St. Louis, St. Louis, MO, USA · Arizona State University, Tempe, AZ, USA · Florida State University, Tallahassee, FL, USA

LLM-as-a-Judge Rubric-Based Evaluation

Open paper AskPDF arXiv

Rubric-based text evaluation increasingly relies on large language models (LLMs) as scalable judges, yet frozen black-box models can interpret the same criteria inconsistently, produce score attributions that are difficult to audit, and map judgments poorly onto human scoring scales. We define this challenge as criteria transfer: translating human rubric intent into a stable, auditable inference-time scoring protocol. We introduce Rulers, which locks a task-level rubric specification, executes it through structured, evidence-grounded judgments, and calibrates the resulting signals to human score boundaries. Across four rubric-governed benchmarks and multiple frozen backbone models, Rulers achieves stronger agreement with human scores in most evaluated settings, while better matching empirical score distributions and remaining more stable under semantically equivalent rubric perturbations. Calibration controls and component ablations show that these gains cannot be attributed to post-hoc alignment alone, but depend on the combination of fixed criteria, traceable evidence, and calibrated score interpretation. These findings suggest that reliable LLM judging requires faithfully operationalizing human evaluation standards rather than relying on prompt-level scoring alone. Our code is available at https://github.com/LabRAI/Rulers.git.

Abstract

LLM-as-a-Judge

Explore similar work

From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges

Small Language Models as Judges for Rubric-Based Reinforcement Learning

Self-Evaluation Is Already There: Eliciting Latent Judge Calibration in Base LLMs with Minimal Data