Evaluating Rubric Generation with Interventional Transfer

Erik Skalnes; Layne C. Price; Raviteja Anantha; Michael Oberst

cs.LGOct 7, 2026

Evaluating Rubric Generation with Interventional Transfer

Authors: Erik Skalnes, Layne C. Price, Raviteja Anantha, Michael Oberst

Organizations: Johns Hopkins Computer Science · Amazon

Abstract

Instance-specific rubrics are common in AI benchmarks where reliable evaluation requires specific expert knowledge. This approach is difficult to scale, prompting research into the generation of rubrics with large language models (LLMs). However, even when expert rubrics are available as references, it is unclear how to productively evaluate the quality of generated rubrics at scale. In this paper, we introduce a method for the evaluation of rubric generation, which we call Interventional Transfer (IT), based on the idea that two rubrics are similar if they move together when a response is perturbed to pass/fail one of them. In contrast to existing approaches for evaluation of rubric generation, we argue that different forms of interventional transfer can be used to evaluate the utility of generated rubrics for different tasks. For instance, we apply this approach in a case study on HealthBench, where we demonstrate an asymmetry in rubrics generated by Qwen3.8-27B, Deepseek-V4-Flash, and Opus-5, used to evaluate responses from GPT-5.6-Terra. Perturbations that degrade responses according to the generated/expert rubric transfer into lower scores on the corresponding expert/generated rubric, but perturbations that improve on one rubric do not reliably transfer into higher scores on the other. We argue that this finding has implications for the usage of LLM-generated rubrics for performance monitoring and hill-climbing. We contrast our approach with existing approaches for rubric evaluation, which do not surface the same asymmetry that we observe.

Figures & tables

Panel of Figure 1 — Figure 1: Overall setting and response perturbation. (a) Model $M$ produces a response $R$ to query $X$ , rubric-generator $G$ generates a rubric $\mathcal{C}^{g}$ from $X$ alone, and humans generate the expert rubric $\mathcal{C}^{*}$ . Both rubrics evaluate the same response. The boxes / cells represent criteria, and are filled (in grey) when passed and empty when failed. (b) The response model $M$ edits $R$ to satisfy all criteria in $\mathcal{C}^{g}$ , and the resulting $R^{\prime}$ is evaluated on $\mathcal{C}^{*}$ . Blue cells show generated criteria that are passed by design after intervention; green and red show an expert criterion satisfied post-intervention and one still failed. Outcomes are illustrative.

Panel of Figure 2 — Figure 2: Illustration of three approaches for evaluating rubric generation. $\mathcal{C}^{g}$ is a 5-criterion generated rubric and $\mathcal{C}^{*}$ is a 4-criterion expert rubric. Scores are aggregated with $\mathrm{frac}(\mathcal{C};R)$ , the fraction of passed criteria over total criteria. (a) Semantic matching of criteria, which enables the calculation of precision and recall metrics. (b) Correlation on a population of responses. (c) An example of an Interventional Transfer metric (ours), which measures the fraction of previously missed expert criteria that are passed after we intervene to pass all generated criteria.

Panel of Figure 3 — Figure 3: The Interventional Transfer metrics described in Section 3 , each of which examines the extent to which intervention on a response to pass/fail a target rubric transfers to a corresponding improvement/degradation in the measurement rubric. Each cell has an interpretation in the context of a particular application of generated rubrics. Measuring with $\mathcal{C}^{*}$ the intervention to pass $\mathcal{C}^{g}$ corresponds to the quality of $\mathcal{C}^{g}$ as a training target; measuring the intervention to fail corresponds to the trustworthiness of a degradation flag from $\mathcal{C}^{g}$ . Measuring with $\mathcal{C}^{g}$ the intervention to pass/fail $\mathcal{C}^{*}$ corresponds to the ability of $\mathcal{C}^{g}$ to detect good/bad responses.

Panel of Figure 4 — Figure 4: Published approaches to rubric evaluation semantically match criteria (left), or relate generated scores to expert scores with rank-correlation on a panel of responses (left-mid), rank-correlation across physician responses (right-mid), or by discrimination between high/low quality responses on physician responses versus artificially degraded responses (right).

Panel of Figure 5 — Figure 5: Application of our approach to HealthBench data. Each group represents a distinct whole-rubric intervention on responses from GPT-5.6-Terra to either pass or fail the generated rubric $\mathcal{C}^{g}$ or the expert rubric $\mathcal{C}^{*}$ . The $Y$ -axis is $Q^{\pm}(\mathcal{C}^{t}\to\mathcal{C}^{m})$ from Equation 1 : the net number of measurement-rubric criteria that moved to $b$ after intervening to move the target rubric from $a$ to $b$ , divided by the number of measurement criteria initially at $a$ . Differences where the pairwise confidence interval does not include zero are indicated by $*$ . We observe that intervening to fail target criteria that currently pass ( $1$ on the response) has a far greater effect on the measurement rubric than intervening to pass criteria that currently fail.

Panel of Figure 6 — Figure 6: Criteria-level transfer between and within rubrics. Each bar is one rubric’s criteria on responses from GPT-5.6-Terra, with columns designating whether the intervention is to pass ( $R_{+c}$ , left) or to fail ( $R_{-c}$ , right). Every criterion $c$ gets its own intervention on the original response. In the upper chart, the first three rows intervene on criteria from each generated rubric $\mathcal{C}^{g}$ and measure the resulting change on the expert rubric $\mathcal{C}^{*}$ . The next three rows intervene on expert criteria and measure changes in each generated rubric. In the top six rows, each criterion is categorized by how many criteria of the other rubric move with it. A target criterion is irrelevant if no criterion in the measurement rubric moves in the intervention direction, matched if one criterion moves, and general if several criteria move. The lower chart measures within-rubric effects. After one target criterion is made to pass or fail, it reports whether zero, one, or several other criteria (excluding itself) in that same rubric also move in the intervention direction. Numbers are percentages of the row’s criteria. We observe that irrelevant criteria make up $82$ – $87\%$ of initially failing generated criteria but only $47$ – $57\%$ of initially passing ones, consistent with the pass/fail asymmetry of Figure 5 , and that passing criteria are more redundant than failing criteria in every rubric.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Description	Location
RubricRAG [ 6 ] , with $\mathrm{sim}(c^{g},c^{*})\in[0,1,\dots,9]$ scored by an LLM
LLM-matching	$R:=\frac{1}{\|\mathcal{C}^{}\|}\sum_{c^{}}\max_{c^{g}}\mathrm{sim}(c^{g},c^{})$ $P:=\frac{1}{\|\mathcal{C}^{g}\|}\sum_{c^{g}}\max_{c^{}}\mathrm{sim}(c^{g},c^{*})$ $F_{1}:=\frac{2PR}{P+R}$	§3.3.1; prompt in Fig. 3
$n$ -gram-matching	$P$ , $R$ , $F_{1}$ as above with $n$ -gram similarity in place of $\mathrm{sim}$	§3.3.1
Missed@ $\tau$ , Hallucinations@ $\tau$ , Redundancy@ $\tau$	$\frac{1}{\|\mathcal{C}^{}\|}\sum_{c^{}}\mathds{1}[\max_{c^{g}}\mathrm{sim}(c^{g},c^{})<\tau]$ $\frac{1}{\|\mathcal{C}^{g}\|}\sum_{c^{g}}\mathds{1}[\max_{c^{}}\mathrm{sim}(c^{g},c^{*})<\tau]$ $\frac{2}{\|\mathcal{C}^{g}\|(\|\mathcal{C}^{g}\|-1)}\sum_{c^{g}<c^{g\prime}}\mathds{1}[\mathrm{sim}(c^{g},c^{g\prime})>\tau]$	§3.3.2
Score-correlation	$\mathrm{corr}\big(\mathrm{frac}(\mathcal{C}^{g};R_{\mathrm{human}}),\,\mathrm{frac}(\mathcal{C}^{*};R_{\mathrm{human}})\big)$ , Spearman and Pearson $R_{\mathrm{human}}$ is the human-authored response	§4.3, Table 3
Discrimination between high/low quality responses	$\mathds{E}\big[\mathds{1}[\mathrm{frac}(\mathcal{C};R_{\mathrm{human}})>\mathrm{frac}(\mathcal{C};R_{\mathrm{bad}})]\big]$ for each of $\mathcal{C}\in\{\mathcal{C}^{g},\mathcal{C}^{*}\}$ $R_{\mathrm{bad}}$ is generated by conditioning on another example’s rubric	§3.4; Fig. 8

Appendix

Table A.1 : Metrics used to evaluate generated rubrics against a reference rubric or human labels.

Metric	Description	Location
GenRubric [ 3 ] , over $n=8$ responses $R_{k}$ from different models for each example, with $s^{g}_{k}:=\mathrm{frac}(\mathcal{C}^{g};R_{k})$ and $s^{}_{k}:=\mathrm{frac}(\mathcal{C}^{};R_{k})$
Score-correlation	$\mathrm{corr}\big(\mathrm{rank}(s^{g}),\mathrm{rank}(s^{*})\big)$ called “sorting consistency”	App. B.1
Kendall’s $\tau_{b}$	$\frac{C-D}{\sqrt{(C+D+T^{g})(C+D+T^{})}}$ , for $C$ concordant and $D$ discordant response pairs, and $T^{g}$ , $T^{}$ the pairs tied under $s^{g}$ , $s^{*}$	App. B.2
Top-1 consistency	$\mathds{1}\big[\argmax{k}\,s^{g}_{k}\in\argmax{k}\,s^{*}_{k}\big]$	App. B.3
Pairwise accuracy	$\frac{1}{\binom{n}{2}}\sum_{k<l}\mathds{1}\big[\mathrm{sign}(s^{g}_{k}-s^{g}_{l})=\mathrm{sign}(s^{}_{k}-s^{}_{l})\big]$	App. B.4
aggregation	per query, averaged over generation runs, then over $700$ queries	App. B.5

Appendix

Table A.2 : Metrics that evaluate a generated rubric through the responses it ranks or the judge it produces.

Setting	Choice
Judge $J$ (scores a response against a rubric)	Gemini 3.8 Flash, reasoning effort medium
Judge votes	majority of 3 samples
Negative criteria restated as positive	Gemini 3.8 Flash, reasoning effort low
LLM-matching framings ( Figure A.1 )	Gemini 3.8 Flash, reasoning effort medium
RubricRAG similarity	Gemini 3.8 Flash, reasoning effort low
Generators	Qwen3.8-27B, Deepseek-V4-Flash, Opus-5

Appendix

Table A.3: Model choices and implementation settings for the experimental pipeline

	Deepseek-V4-Flash $-$ Qwen3.8-27B	Opus-5 $-$ Qwen3.8-27B	Opus-5 $-$ Deepseek-V4-Flash
LLM-matching $F_{1}$	0.04 (0.02, 0.05)	0.02 (0.01, 0.04)	-0.01 (-0.03, 0.00)
Score Correlation	0.04 (-0.05, 0.13)	0.06 (-0.02, 0.15)	0.02 (-0.06, 0.09)
Score Correlation (physician)	-0.05 (-0.21, 0.09)	0.03 (-0.14, 0.19)	0.07 (-0.10, 0.27)
Discrimination between high/low quality responses	0.05 (-0.03, 0.15)	0.13 (0.03, 0.23)	0.08 (-0.02, 0.18)

Appendix

Table A.4: Paired differences between rubric generators for each existing approach in Figure 4 with 95% paired bootstrap intervals. Most comparisons are inconclusive, but the few that are indicate that large models generate slightly better rubrics.

Panel of Figure 11 — Figure A.1 : Absolute rubric quality implied by semantic matching with LLMs is sensitive to the definition of semantic similarity in the prompt. We show the quality implied by instructing the LLM to say that two criteria are semantically similar if they measure the same thing (left), changing one would change the other (mid), or one logically entails the other (right). The three prompts are ask_llm_match_rubric , ask_llm_simulate_intervention and ask_llm_implies in Appendix E .

Panel of Figure 12 — Figure A.2 : For different choices of response panel, each corresponding to something one might reasonably do, the value of the score correlation metric changes.

	expert score
response model	ordinary	foreign rubric	difference	share of passing $\mathcal{C}^{*}$ lost
Qwen	0.58 (0.54, 0.62)	0.41 (0.37, 0.46)	0.17 (0.13, 0.21)	0.27 (0.22, 0.33)
Llama-4-Scout-17B	0.45 (0.41, 0.50)	0.34 (0.30, 0.38)	0.11 (0.07, 0.15)	0.25 (0.17, 0.31)
Qwq-32B	0.58 (0.54, 0.62)	0.46 (0.42, 0.51)	0.12 (0.07, 0.16)	0.21 (0.16, 0.26)
Glm-5.3-Flash	0.61 (0.57, 0.65)	0.55 (0.50, 0.59)	0.07 (0.03, 0.11)	0.09 (0.04, 0.15)
Grok-4.20	0.58 (0.53, 0.62)	0.36 (0.32, 0.41)	0.22 (0.16, 0.27)	0.37 (0.28, 0.45)

Appendix

Table A.5: RubricRAG’s method for degrading responses, putting an irrelevant rubric into the response-model prompts, assessed across models of varying sizes. Big/strong models are less affected than small/weak models, and the amount of degradation is limited for both.

Panel of Figure 14 — Figure A.3 : Illustration of the difference between typical score correlation approaches and our proposed approach. Rows are user queries $X_{i}$ and columns are responses from different models $M_{k}$ , and each cell has fraction of passing criteria in the generated (above diagonal) and expert (below diagonal) rubrics. The left figure considers existing responses and the right side considers perturbed responses. Existing score-correlation approaches differ in which population of responses they consider. The method used in Dhole & Agichtein [6] (RubricRAG) is the correlation across queries for a single model, illustrated by the vertical slice. The method used in Chen et al. [3] (GenRubric) is the expected correlation across models, illustrated by the horizontal slice. Our method examines the expected difference between responses perturbed to satisfy one rubric, measured by the other rubric. The example illustrates a possible failure mode of score correlation, where the correlation is nearly perfect on a population ( $1.00$ down the first column and $0.95$ across the second row), but does not substantially move the expert rubric score.

rubric generator	passing	$n$	failing	$n$	passing $-$ failing
Qwen3.8-27B	4.42	825	3.28	134	1.14 (0.67, 1.57)
Deepseek-V4-Flash	4.80	754	3.87	174	0.92 (0.47, 1.36)
Opus-5	4.09	1269	3.43	397	0.65 (0.31, 1.02)

Appendix

Table A.6: Mean LLM-matching similarity (0–9) between each generated criterion and its closest expert criterion, split by whether the criterion initially passes or fails on the GPT-5.6-Terra response. The difference carries a 95% bootstrap interval over examples.

	passing / failing
similarity $\geq$	Qwen3.8-27B	Deepseek-V4-Flash	Opus-5
1	95% / 91%	98% / 98%	93% / 92%
2	87% / 75%	91% / 86%	83% / 78%
3	75% / 54%	81% / 69%	69% / 59%
4	58% / 39%	66% / 48%	52% / 40%
5	46% / 28%	53% / 34%	41% / 30%

Appendix

Table A.7: Share of initially passing and failing generated criteria on the GPT-5.6-Terra response whose LLM-matching similarity to the closest expert criterion is at or above each cutoff.

	Deepseek-V4-Flash $-$ Qwen3.8-27B	Opus-5 $-$ Qwen3.8-27B	Opus-5 $-$ Deepseek-V4-Flash
Training target	0.05 (-0.02, 0.13)	0.11 (0.04, 0.17)	0.06 (-0.02, 0.13)
Detect good responses	-0.03 (-0.18, 0.09)	-0.09 (-0.22, 0.04)	-0.05 (-0.18, 0.10)
Detect bad responses	0.10 (0.05, 0.14)	-0.00 (-0.04, 0.04)	-0.10 (-0.14, -0.06)
Degradation flag	-0.03 (-0.07, 0.01)	0.03 (0.00, 0.07)	0.06 (0.03, 0.10)

Appendix

Table A.8: Paired differences between rubric generators for each IT metric in Figure 5 on GPT-5.6-Terra responses with 95% paired bootstrap intervals.

Panel of Figure 18 — Figure A.4 : How do IT metrics vary based on response model? For responses from the model on the x-axis, we measure (from top to bottom) the four tasks. Specifically, the top is the training target: $\mathcal{C}^{*}$ gained on $R_{+\mathcal{C}^{g}}$ . Second is detecting good responses: $\mathcal{C}^{g}$ gained on $R_{+\mathcal{C}^{*}}$ . Third is detecting bad responses: $\mathcal{C}^{g}$ lost on $R_{-\mathcal{C}^{*}}$ . Bottom is the degradation flag: $\mathcal{C}^{*}$ lost on $R_{-\mathcal{C}^{g}}$ . The paper uses the last column.

			whole rubric, $R_{\pm\mathcal{C}}$			single criterion, $R_{\pm c}$
target	measured	$\|c\|$	$R^{\prime}-R$	$\frac{R^{\prime}-R}{\|\mathcal{C}\|}$	$\frac{\|R^{\prime}\|}{\|R\|}$	$R^{\prime}-R$	$\frac{R^{\prime}-R}{\|c\|}$	$\frac{\|R^{\prime}\|}{\|R\|}$
$\mathcal{C}^{g}$ , $+$	$\mathcal{C}^{*}$	$23$	$78$	$1.13$	$1.46$	$29$	$1.28$	$1.15$
$\mathcal{C}^{*}$ , $+$	$\mathcal{C}^{g}$	$26$	$118$	$1.13$	$1.23$	$33$	$1.26$	$1.06$
$\mathcal{C}^{*}$ , $-$	$\mathcal{C}^{g}$	$24$	$277$	$1.79$	$0.62$	$60$	$2.48$	$0.92$
$\mathcal{C}^{g}$ , $-$	$\mathcal{C}^{*}$	$22$	$339$	$1.70$	$0.42$	$63$	$2.85$	$0.92$

Appendix

Table A.9: How much each intervention changes the response, on responses from GPT-5.6-Terra to HealthBench queries. Each row is one of the four interventions, showing both criteria-level and rubric-level, with target the rubric intervened on with direction, and measured the measurement rubric.

|c|

is the mean length in words of a targeted criterion.

R^{\prime}-R

is the number of words deleted plus the number of words inserted to turn

R

into

R^{\prime}

, by a word-level diff, averaged over interventions, so it is positive even when the edit shortens the response, and a reworded word counts twice.

\frac{R^{\prime}-R}{|\mathcal{C}|}

and

\frac{R^{\prime}-R}{|c|}

divide those words by the words of the targeted criteria, summed over the row, for the whole-rubric and single-criterion interventions.

\frac{|R^{\prime}|}{|R|}

is the length in words of the perturbed response over that of the original, averaged over interventions, so values below

1

mean the edit shortened the response. We observe that intervening to fail criteria takes more editing per word of targeted criterion than intervening to pass them, and that these edits shorten the response, to

42

–

62\%

of its original length for whole-rubric interventions and to

92\%

for single-criterion ones, while intervening to pass lengthens it.

Explore similar work

May 28, 2026cs.CL

Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge

LLM-as-a-Judge is a scalable alternative to human evaluation, yet existing rubric-based methods rely on human-annotated data such as reference answers or expert-crafted rubrics. We propose to automatically generate fine-grained evaluation rubrics without any human annotation. Our training-free method generates rubrics at dataset-specific and instance-specific granularities, achieving performance competitive with existing methods across four benchmarks. We further present a method that iteratively fine-tunes a rubric generator model via meta-judge reward signals. The fine-tuned generator outperforms all existing baselines in both pairwise and pointwise evaluation. Notably, a fine-tuned 14B rubric generator outperforms a much larger proprietary model at rubric generation, showing the effectiveness of our fine-tuning strategy.

Zijie Wang, Eduardo Blanco

University of Arizona Department of Computer Science

LLM-as-a-Judge Language Model Generation Evaluation Rubric-Based Evaluation

Ask PDF

Aug 30, 2026cs.CL

GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation

Large language models are increasingly used as scalable evaluators for open-ended tasks. However, many LLM judges derive query-specific criteria during scoring, leaving the evaluation requirements insufficiently specified and their coverage difficult to audit. Query-specific rubrics make these requirements explicit, but expert-written rubrics are costly to construct, while existing automatic methods typically rely on inference-time refinement or external supervision. We introduce GenRubric, a self-evolving framework that improves rubric generation from unlabeled queries without requiring additional human annotations during self-evolution. Our approach is based on rubric-induced self-consistency: independently sampled rubrics for the same query provide partial views of its latent evaluation requirements, and a comprehensive rubric should induce a response that generalizes across these complementary evaluation views. We implement this principle through reinforcement learning, combining a cross-rubric comprehensiveness signal with group-level and criterion-level rewards for rubric quality. We train GenRubric models at 4B, 8B, and 14B scales across multiple domains. Experiments on human-annotated rubric benchmarks show that self-evolution improves the agreement between evaluations induced by generated rubrics and those induced by expert-written rubrics. The improvements further generalize to held-out domains, demonstrating the potential of self-evolving rubric generation for scalable and query-specific LLM evaluation. Code and models are publicly available at https://github.com/foggpoy/GenRubric.

Yifan Chen, Haitao Li, Qingyao Ai +4

Department of Computer Science and Technology, Tsinghua University · National University of Singapore

LLM-as-a-Judge Language Model Generation Evaluation Rubric-Based RL Rubric-Based Evaluation Rubric-Guided Generation

Ask PDF

Jul 16, 2026cs.CL

Rubrics on Trial: Evolving Rubrics from a Single Query via Synthetic Pairwise Evidence

Rubric evolution offers a promising approach to improving the quality of rubrics generated by large language models (LLMs). Central to this process is rubric comparison, which identifies the better of two rubrics and guides the direction of evolution. However, accurate rubric comparison is difficult, which presents two challenges. (1) It should reflect downstream task performance, which is essential for assessing rubric utility but often prohibitively expensive to evaluate. (2) It should discourage unnecessary criteria, which increase verification costs and may dilute the influence of essential criteria. To address these challenges, we introduce Rubrics on Trial, a multi-agent framework that evolves rubrics by comparing synthetic response pairs. To address challenge 1, the framework compares synthetic responses that satisfy the respective rubrics, providing a proxy for downstream performance without training a separate policy for each rubric. To address challenge 2, it assesses the necessity of a candidate criterion by independently generating high-quality alternative responses that violate it and comparing them with edited versions that satisfy it. A rubric is favored when it improves response quality in both comparisons, and the resulting comparison signal is further incorporated for rubric evolution. Extensive experiments demonstrate that Rubrics on Trial improves the quality of generated rubrics and leads to better downstream task performance.

Haocheng Yang, Licheng Pan, Yuan Lu +7

School of Computing, National University of Singapore · Xiaohongshu Inc. · School of Cyber Science and Technology, Zhejiang University +3

Rubric-Based Evaluation

Ask PDF