Rubrics Carry the Standard: Shapley Attribution for Checklist-Based LLM Judges

Saeid Asgari; Rui Ying; Sara Malvar; Rakshanda Agarwal; Bruce Sun; Claire Wang; Leonardo de Oliveira Nunes; Ranveer Chandra

cs.CLOct 18, 2026

Rubrics Carry the Standard: Shapley Attribution for Checklist-Based LLM Judges

Authors: Saeid Asgari, Rui Ying, Sara Malvar, Rakshanda Agarwal, Bruce Sun, Claire Wang, Leonardo de Oliveira Nunes, Ranveer Chandra

Abstract

Checklist rubrics applied by large language model (LLM) judges have become a standard instrument for evaluating open-ended outputs and for rewarding them during training. Two assumptions surround this practice: that leniency or strictness is a property of the judge model, and that a rubric can be improved by asking a model to review it. We test both in a controlled grading task with planted flaws and on two human-labeled datasets, SummEdits and RAGTruth. Holding the judge and the outputs fixed, rubric content moves the pass rate of correct outputs from 2% to 100%, whereas changing the judge model or the rendering of the same checks moves it by a few points. Model self-review then drifts the rubric toward the reviser's own standard: it adds requirements the task never stated and deletes checks that human labels had taught, with three reviser models and on all three datasets. To measure and control this, we show that under all-must-pass aggregation the Shapley value of each check has an exact closed form, which yields per-check catch credit, false-fail blame and unique catches in linear time. Attribution shows that self-review deletes the checks that do unique work and keeps the checks that cause false fails. A change-control gate that combines these quantities with a provenance freeze on label-derived checks holds nearly every drifting revision chain at its starting quality, with identical decisions under three judge models.

Figures & tables

Figure 1: Strictness on Recon-72 (24 correct, 48 flawed reports). (a) Operating point of each rubric under a Sonnet judge. Hand-written and generated checks form two clusters that rendering does not move. (b) Correct reports passed under the same rubrics by three judges, which agree within a few points except with a generic scale and a pass threshold of 5. Takeaway: what the checks ask for moves correct passed from about 4% to about 96%, while rendering and judge model move it by a few points, as claimed: strictness is set by rubric content.

Figure 2: Generic checklists of increasing length under a Sonnet judge, mean of two runs. (a) On SummEdits each added check trades a few consistent summaries passed for inconsistent ones caught. (b) On RAGTruth, going from 3 to 10 checks costs 36 points of faithful responses passed for at most 4 points of catches; the same 10 checks read as a paragraph pass 75%. Takeaway: the content effect replicates on human labels, small where generic checks rarely demand anything a correct output lacks (SummEdits) and large where they forbid the light synthesis annotators accepted (RAGTruth); RAGTruth is also the one place where rendering mattered, through the aggregator.

Figure 3: Accuracy of three Recon rubrics as a function of the flawed share of the validation set, computed from their two rates. The generated draft has balanced accuracy 52, the level of chance, yet outscores the hand-written rubric whenever more than 80% of the set is flawed. Takeaway: accuracy on a defect-heavy validation set can rank a rubric that fails almost everything above one that works, which is why we report the two rates throughout.

Output	Label	Checks answering No ( $F_{i}$ )	Share of the outcome
R1	flawed	A	A: 1 (unique catch)
R2	flawed	A, B	A, B: $\tfrac{1}{2}$ each
R3	flawed	B, C, D	B, C, D: $\tfrac{1}{3}$ each
R4	correct	D	D: 1 of blame
R5	correct	none	none

Table 1: A four-check rubric on five outputs. Totals: A has credit 1.5 and one unique catch; B 0.83 and C 0.33, both without unique catches; D has credit 0.33 and blame 1. Credits sum to the 3 flawed outputs caught and blame to the 1 correct output failed. Takeaway: only deleting A would lose a catch and only deleting D would let a correct output pass, so unique catches and blame, not raw fail counts, identify which checks matter; these are the rules the Shapley gate applies in Section 7 .

Figure 4: Self-review over four rounds on Recon-108. (a) From the hand-written rubric, free revision adds requirements and correct reports stop passing. (b) From the label-taught rubric, free and fixed-count revision remove or narrow the open coverage check and unplanned misses stop being caught. Delete-only revision is flat in both. Takeaway: drift runs in opposite directions from the two starting rubrics, too strict from one and too lenient on omissions from the other, each time toward the reviser’s own idea of a good report; only the arm that cannot add requirements avoids it, consistent with the claim that self-review moves the standard rather than refining it.

Figure 5: (a) Shapley catch credit and false-fail blame per check on Recon-108, grouped by the origin of the check. Label-derived checks carry the most credit per check and no blame; checks added by Opus review of the hand rubric carry the most blame. (b) Of the 186 checks deleted across the 12 Recon revision chains, the number that held a unique catch and the number that carried any false-fail blame, by origin. Takeaway: 78 of the 186 deleted checks held a unique catch and only 2 carried any blame, so self-review removes checks that do work no other check does and keeps the ones that fail correct outputs, the opposite of what an improving revision would do.

Figure 6: Quality of the gated rubric after four rounds of revision, scored on the 8 evaluation environments. (a) From the hand-written rubric the failure mode is strictness; (b) from the label-taught rubric it is lost coverage. Only Gate S+P stays near the starting level (dotted) in both panels. Takeaway: each single signal protects one failure direction (Gate R only what its calibration labels see, Gate P coverage, Gate S strictness), and only their combination protects both, with one exception (Sonnet’s label-taught chain) discussed in the text; this supports gating revisions check by check rather than trusting either labels or attribution alone.

Chain	Rubric or gate	Haiku judge	Sonnet judge	Opus judge
H, Opus seed 1	start	100.0 / 83.3 / 0.0	97.2 / 83.3 / 0.0	97.2 / 83.3 / 0.0
	ungated (round 4)	5.6 / 98.6 / 91.7	5.6 / 98.6 / 91.7	5.6 / 98.6 / 91.7
	Gate S+P, leave-one-out	97.2 / 83.3 / 0.0	94.4 / 83.3 / 0.0	91.7 / 83.3 / 0.0
L, Opus seed 2	start	80.6 / 98.6 / 91.7	83.3 / 100 / 100	83.3 / 100 / 100
	ungated (round 4)	83.3 / 84.7 / 16.7	80.6 / 84.7 / 8.3	80.6 / 86.1 / 16.7
	Gate S+P, leave-one-out	83.3 / 98.6 / 91.7	80.6 / 100 / 100	80.6 / 100 / 100

Table 2: Gate replay under three judge models on all 12 Recon environments. Each cell: correct passed / flawed caught / unplanned caught (%). Takeaway: under every judge the ungated rubric collapses in the same way, and Gate S+P keeps each metric within 6 points of that judge’s own starting value, so the gate’s decisions do not depend on which model applies the rubric.

SummEdits (51 consistent, 49 inconsistent evaluation items)
Chain	start	ungated	Gate R	Gate P	Gate S	Gate S+P
Generated, free	72.5 / 79.6	58.8 / 87.8	64.7 / 83.7	58.8 / 87.8	68.6 / 81.6	68.6 / 81.6
Label-taught, free	78.4 / 81.6	58.8 / 85.7	58.8 / 85.7	62.7 / 85.7	64.7 / 85.7	68.6 / 85.7
Label-taught, fixed count	78.4 / 81.6	60.8 / 85.7	78.4 / 81.6	70.6 / 83.7	68.6 / 83.7	70.6 / 83.7
Label-taught, delete only	78.4 / 81.6	70.6 / 85.7	70.6 / 85.7	74.5 / 81.6	72.5 / 83.7	74.5 / 81.6
RAGTruth (56 faithful, 56 hallucinated evaluation items)

Table 3: Gates on the human-labeled datasets after two rounds of Opus revision, decided with calibration labels and scored on the evaluation split. Each cell: correct outputs passed / flawed outputs caught (%). Takeaway: on every label-taught chain Gate S+P passes more correct outputs than the ungated rubric, at a cost of at most 4.1 points of catches; it returns RAGTruth to its starting value and recovers part of the SummEdits loss, consistent with the gates protecting what their calibration labels and frozen checks can see.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Report	Label	Hand-written	Label-taught	After Opus review
correct 1	pass	none	none	{3, 6, 7}
correct 2	pass	none	none	{3, 6, 7}
correct (Part B)	pass	none	none	{3, 6, 7}
missed finding	fail	{6}	{5}	{3, 6, 7}
fabricated	fail	{3}	{2, 4, 8}	{2, 3, 5, 6, 7, 8}
wrong resource	fail	{1, 2, 3, 4}	{1, 2, 4, 8}	{1, 2, 3, 5, 6, 7, 8}

Appendix

Table 4: Checks answered No (

F_{i}

) for the storage environment under three rubric versions.

Item	Label	Generated 10 checks	Label-taught	After self-review
L2M1E	faithful	none	none	none
3AZYQ	faithful	{6, 9}	none	none
C0YWR	hallucinated	{1, 6, 8, 9}	{1, 7}	{1, 2, 11, 13}
KJE63	hallucinated	{1, 6, 8, 9}	{1, 7, 10}	{1, 2, 11, 12}

Appendix

Table 5: Checks answered No for the four responses of the example under three RAGTruth checklists.

Rubric-Based Evaluation

Rubric-based evaluation uses explicit criteria to assess model outputs, agent behavior, and student work in a structured, interpretable way.

See all 198 papers

Self-Preference Bias in Rubric-Based Evaluation of Large Language Models
Apr 8, 2026José Pombal, Ricardo Rei, André F. T. MartinsBias in LLM-as-a-Judge LLM Evaluation
Sword Health · Instituto de Telecomunicações · Instituto Superior Técnico, Universidade de Lisboa +2
Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge
Mar 11, 2026Mingyang Song, Mao Zheng, Chenning XuLLM-as-a-Judge Rubric-Based Evaluation
Large Language Model Department, Tencent, China
Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluation
Aug 31, 2026Anshul Bagaria, Sowmya S Sundaram, Gokul S Krishnan +1LLM-as-a-Judge Language Model Generation Evaluation
Centre for Responsible AI (CeRAI), Wadhwani School of Data Science and AI (WSAI), IIT Madras, Chennai, India

Apr 8, 2026J/K move · Enter open · S save

Self-Preference Bias in Rubric-Based Evaluation of Large Language Models

José Pombal, Ricardo Rei, André F. T. Martins

Sword Health · Instituto de Telecomunicações · Instituto Superior Técnico, Universidade de Lisboa+2

Bias in LLM-as-a-Judge LLM Evaluation Rubric-Based Evaluation

Open paper AskPDF arXiv

LLM-as-a-judge has become the de facto approach for evaluating LLM outputs. However, judges are known to exhibit self-preference bias (SPB): they tend to favor outputs produced by themselves or by models from their own family. This skews evaluations and, thus, hinders model development, especially in settings of recursive self-improvement. We present the first study of SPB in rubric-based evaluation, an increasingly popular benchmarking paradigm where judges issue binary verdicts on individual evaluation criteria, instead of assigning holistic scores or rankings. Using IFEval and LiveCodeBench, benchmarks with programmatically verifiable rubrics, we show that SPB persists even when evaluation criteria are entirely objective: among rubrics where generators fail, judges can be more than 50% more likely to incorrectly mark them as satisfied when the output is their own. We also find that, similarly to other evaluation paradigms, ensembling multiple judges helps mitigate SPB, but without fully eliminating it. On HealthBench, a medical chat benchmark with subjective rubrics, we observe that SPB skews model scores by up to 10 points, a potentially decisive margin when ranking frontier models. We analyze the factors that drive SPB in this setting, finding that negative rubrics and subjective topics like communication and emergency referrals are particularly susceptible.

Abstract

Rubric-Based Evaluation

Explore similar work

Self-Preference Bias in Rubric-Based Evaluation of Large Language Models

Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge

Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluation