Rubricon: Automated Rubric Generation and Refinement for LLM Evaluation and Training

Ulyana Tkachenko; Hui Wen Goh; Guram Gogia; Jonas Mueller

cs.AIOct 28, 2026

Rubricon: Automated Rubric Generation and Refinement for LLM Evaluation and Training

Authors: Ulyana Tkachenko, Hui Wen Goh, Guram Gogia, Jonas Mueller

Organizations: Handshake AI

Abstract

Rubrics enable reward assignment on tasks without verifiable answers, but are expensive to manually produce from scratch. We present Rubricon, a pipeline that generates discriminative rubrics without requiring human preference labels, rubric annotation, or generator-model training. Using any LLM, Rubricon samples multiple rubrics, selects the most discriminative candidate, adds criteria to capture overlooked requirements or response flaws, and prunes criteria that are misaligned or shallow. Alongside LLM self-critique and test-time scaling, Rubricon uses observed criterion pass rates on example model responses to guide refinement. Across five professional, consumer, clinical, and general-use LLM benchmarks, Rubricon produces rubrics whose scores of frontier LLM responses exhibit 9% greater alignment with labor-intensive expert-authored rubrics, compared to the best alternative synthetic rubric generation method among RRD, QSRG, OpenRubrics, and agentic generation via harnesses like Codex. On HealthBench, post-training a Qwen3.5-9B model with Rubricon rubrics yields 4.5% better performance than training with rubrics from the strongest alternative synthetic rubric generator, RRD.

Figures & tables

Figure 1: The Rubricon pipeline. Given a task prompt, example strong-model responses, and optional reference material, Generate-and-Select drafts $K=5$ candidate rubrics and selects the rubric with the largest average strong–weak edge across criteria. Add sequentially expands the selected rubric through gap fill, prompt coverage, negative probing, and completeness probing, with each step operating on the rubric from the prior step. The probe steps generate responses intended to satisfy the current rubric, then convert independently identified defects or omissions into new criteria. Prune removes misaligned criteria and those that are satisfiable by evasive or weak responses. The resulting rubric can undergo further Add then Prune refinement rounds.

Method	ProfBench	ACE	PRBench	HealthBench	RubricBench
Metric	$\rho$	$\rho$	$\rho$	$\rho$	Accuracy (%)
QSRG	$0.361\pm 0.082$	$0.225\pm 0.043$	$0.226\pm 0.013$	$0.172\pm 0.008$	$52.9\pm 1.6$
QSRG+Context	$0.267\pm 0.070$	$0.301\pm 0.038$	$0.353\pm 0.012$	$0.279\pm 0.009$	$52.4\pm 1.6$
OpenRubrics	$0.202\pm 0.068$	$0.276\pm 0.038$	$0.071\pm 0.014$	$0.119\pm 0.010$	$57.5\pm 1.7$
Agentic	$0.283\pm 0.063$	$0.333\pm 0.039$	$0.363\pm 0.014$	$0.294\pm 0.009$	$59.3\pm 1.9$
RRD WU	$0.394\pm 0.076$	$0.316\pm 0.040$	$0.497\pm 0.011$	$0.365\pm 0.009$	$58.7\pm 1.7$

Table 1: Alignment with human rubrics and human preferences. Rubricon achieves the highest observed scores across all five benchmarks. Across datasets, it improves macro-average

\rho

by 9.0% relative to the strongest baseline, RRD WU (

0.428

vs.

0.393

), and selective preference accuracy by 0.5 percentage points over the strongest preference baseline, Agentic (

59.8\%

vs.

59.3\%

). The first four benchmarks report

\rho

between scores under generated and expert-authored rubrics. RubricBench reports selective preference accuracy (%), evaluated on response pairs whose normalized rubric scores differ by more than

0.10

(retained counts in Table 7 ). The

\pm

values indicate standard errors estimated by bootstrapping tasks (

B=500

resamples).

Stage	Macro-avg. $\rho$	Avg. Rubric Size
Generation	0.359	20.6
Selected Rubric	0.364	20.6
Gap Fill	0.362	23.9
Prompt Coverage	0.362	24.6
Negative Probing	0.388	28.3
Completeness Probing	0.395	31.0

Table 2: Alignment with expert-authored rubrics (macro-averaged

\rho

across datasets) and average rubric size after each stage of the first refinement round. Rows show cumulative refinement. Values in the top row are averaged over

K=5

independently generated rubrics.

Figure 2: Rubricon rewards translate into stronger held-out post-training performance. Qwen3.5-9B trained with Rubricon rubric rewards outperforms the same model trained with RRD WU rewards at every evaluated post-training checkpoint, reaching a 4.5% higher score at the last evaluated checkpoint. Scores are averaged over three independently sampled responses per prompt, then across 500 held-out HealthBench prompts. The Rubricon-trained model achieves a 20.5% higher held-out score than the starting model checkpoint.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Domain(s)	Human annotations	Tasks	Ref. material
ProfBench	Finance, Consulting, Physics	Human rubrics	28	—
PRBench	Finance, Law	Human rubrics	656	expert step-by-step reasoning
ACE	Shopping, DIY, Gaming, Food	Human rubrics	80	—
HealthBench	Health	Human rubrics	1,704	example response
RubricBench	Chat, IF, STEM, Code, Safety	Human preferences	1,147	—

Appendix

Table 3: Evaluation benchmarks. Human annotations specifies how generated rubrics are assessed: alignment with scores from human-authored rubrics or human preferences between responses. Tasks gives the number of tasks evaluated after preprocessing. Ref. material lists the inputs provided for each task in the rubric generation prompt (see Appendix F.1 ).

	Cohen’s $\kappa$		F1
Verifier Model	Our Prompt	ProfBench	Our Prompt	ProfBench
GPT-5 mini	0.581	0.549	0.802	0.786
GPT-5.2	0.560	0.556	0.777	0.777
Claude Sonnet 4.6	0.556	0.533	0.785	0.760
Gemini 3.1 Pro	0.580	0.582	0.791	0.794

Appendix

Table 4: Using our verifier prompt, GPT-5 mini achieves the highest observed Cohen’s

\kappa

and F1 among the tested verifiers. We report criterion-level agreement (Cohen’s

\kappa

and F1) with expert annotations on ProfBench using our judge prompt and the original ProfBench judge prompt. On a 367-criterion subset that we independently re-annotated, (non-expert) human agreement with the ProfBench expert annotations is

\kappa=0.281

(64.0% raw agreement). The GPT-5 mini verifier reaches

\kappa=0.455

(72.5% raw agreement) against the expert labels on the same subset.

Verifier Model	RRD WU	Rubricon	$\Delta\pm\mathrm{SE}_{\mathrm{paired}}$
Gemini 3.5 Flash-Lite	0.7077	0.7392	$0.0315\pm 0.0057$
GPT-5 mini	0.6482	0.6753	$0.0271\pm 0.0052$
Claude Sonnet 5	0.5357	0.5710	$0.0353\pm 0.0059$

Appendix

Table 5: Verifier sensitivity on held-out HealthBench evaluation dataset. Each verifier scores the same saved responses from the two policies after eight training epochs against physician-authored rubrics.

\Delta

denotes Rubricon minus RRD WU . Standard errors are computed across paired prompt-level differences.

Method	ProfBench	ACE	PRBench	HealthBench
Expert-authored	30.5	7.4	16.6	14.7
QSRG	22.1	19.5	20.3	19.9
QSRG+Context	21.2	20.3	20.6	20.1
OpenRubrics	11.8	9.3	9.3	7.9
Agentic	33.9	24.3	28.4	24.1
RRD WU	26.0	26.7	25.4	23.4

Appendix

Table 6: Average number of criteria per rubric on the four datasets where we report alignment against human-expert rubrics. The average size of these expert-authored rubrics is included for reference.

Method	Retained comparisons	Avg. rubric size
QSRG	1,966	20.0
QSRG+Context	2,046	20.1
OpenRubrics	2,133	7.2
Agentic	1,576	20.0
RRD WU	1,938	20.5
Rubricon	1,760	17.3

Appendix

Table 7: Number of retained comparisons and average rubric size on RubricBench ( Zhou et al., 2026 ) . We retain comparisons with an absolute rubric score difference above

0.10

(Section 5.1 ). Counts include three scoring runs per task, giving 3,441 comparisons across 1,147 tasks before filtering.

Stage	ProfBench	ACE	PRBench	HealthBench
Selected Rubric	$0.345\pm 0.074$	$0.352\pm 0.037$	$0.421\pm 0.012$	$0.339\pm 0.009$
After Add Stage (round 1)	$0.358\pm 0.081$	$0.376\pm 0.037$	$0.486\pm 0.012$	$0.360\pm 0.009$
After Prune Stage (round 1)	$0.405\pm 0.084$	$0.354\pm 0.037$	$0.500\pm 0.012$	$0.371\pm 0.009$
After Add Stage (round 2)	$0.374\pm 0.086$	$0.354\pm 0.036$	$0.500\pm 0.012$	$0.372\pm 0.009$
After Prune Stage (round 2)	$\mathbf{0.424\pm 0.083}$	$\mathbf{0.385\pm 0.035}$	$\mathbf{0.516\pm 0.011}$	$\mathbf{0.388\pm 0.009}$

Appendix

Table 8: Iterative refinement improves alignment on every benchmark. A second refinement round increases macro-average

\rho

by 17.6% over the selected draft rubric. Rows are cumulative, with each stage operating on the preceding rubric. Results are reported as

\rho

± task-bootstrap standard error.

Method / stage	ProfBench		ACE		PRBench		HealthBench
	$\rho$	Size	$\rho$	Size	$\rho$	Size	$\rho$	Size
Selected Rubric	$0.345\pm 0.074$	23.6	$0.352\pm 0.037$	19.4	$0.421\pm 0.012$	21.3	$0.339\pm 0.009$	17.9
Gap Fill	$0.327\pm 0.077$	27.0	$0.364\pm 0.039$	22.8	$0.417\pm 0.012$	24.8	$0.340\pm 0.009$	21.0
Prompt Coverage	$0.323\pm 0.080$	29.0	$0.368\pm 0.038$	23.1	$0.417\pm 0.013$	25.0	$0.339\pm 0.009$	21.1
Negative Probing	$0.346\pm 0.077$	33.3	$\mathbf{0.379\pm 0.036}$	26.0	$0.471\pm 0.012$	30.2	$0.356\pm 0.009$	23.5
Completeness Probing	$0.358\pm 0.081$	37.3	$0.376\pm 0.037$	27.5	$0.486\pm 0.012$	34.1	$0.360\pm 0.009$	25.0

Appendix

Table 9: Alignment with expert-authored rubrics (

\rho

) and average rubric size (criteria per task) across the first Rubricon refinement round. Bold marks the highest observed correlation per benchmark. Uncertainty is the task-bootstrap standard error (500 resamples).

Figure 3: Rubric category composition across different benchmarks and pipeline stages during Rubricon’s first refinement round. Each benchmark has a different category distribution more representative of its domain, but all show a $33$ – $45\%$ relative increase in the share of Reasoning criteria during refinement.

	Round 1			Round 2
Dataset	Generate-and-Select	Add	Prune	Add	Prune
ProfBench	23.6	37.3	32.8	46.1	41.9
PRBench	21.3	34.1	31.4	40.5	38.8
ACE	19.4	27.5	22.0	29.5	25.0
HealthBench	17.9	25.0	20.5	26.2	23.4
RubricBench	15.8	21.0	17.1	21.5	18.8

Appendix

Table 10: Mean rubric size (criteria per task) at each pipeline stage, averaged over all tasks in each evaluation dataset. In Round 1, Add expands rubrics by 33–60% and Prune removes 8–20%, leaving rubrics 8–47% larger than the generated baseline. In Round 2, Add expands rubrics by a further 26–41% and Prune removes 4–15%, leaving final rubrics 19–82% larger than the generated baseline. RubricBench sizes here are averaged over all 1,147 tasks; Table 7 reports 17.3 because it averages over the comparisons retained by the

\delta=0.10

margin.

Setting	Value
Base model	Released Qwen3.5-9B post-trained checkpoint
Training budget	8 epochs; 152 updates
Batch size	Up to 64 prompts
Responses per prompt	8 sampled; at least 6 valid required
Objective	GRPO with no KL penalty or ratio clipping
Optimizer	Adam, constant learning rate $10^{-5}$

Appendix

Table 11: Shared post-training configuration. This configuration was identified to be effective by initially training on the original (expert-authored) HealthBench rubrics, rather than tuning the configuration based on training against automatically generated rubrics.

	ProfBench		ACE		PRBench		HealthBench
Rubric	$\rho$	Size	$\rho$	Size	$\rho$	Size	$\rho$	Size
Selected rubric	0.345	23.6	0.352	19.4	0.421	21.3	0.339	17.9
Merged rubric	0.391	119.2	0.387	96.0	0.462	107.3	0.367	90.2
Rubricon	0.424	41.9	0.385	25.0	0.516	38.8	0.388	23.4

Appendix

Table 12: Alignment with expert-authored rubrics (Spearman

\rho

) and average rubric size (criteria per task). Merging improves alignment over the selected rubric at approximately

5\times

the rubric size, while Rubricon achieves higher macro-average alignment with a

3.2\times

smaller rubric than the merge.

Panel of Figure 4 — Figure 4: Iterative refinement expands coverage and restores expert alignment. Add introduces missing requirements, while Prune removes misleading criteria, increasing Spearman’s correlation with the physician-authored rubric from $0.613$ initially to $0.925$ .

Panel of Figure 5 — Figure 5: Refinement keeps useful additions and removes misleading criteria. Prune retains seven of Add’s ten criteria, increasing Spearman’s correlation with the physician-authored rubric from $0.632$ initially to $0.682$ .

Automatic Rubric Generation

See all 30 papers

Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge
May 28, 2026Zijie Wang, Eduardo BlancoAutomatic Rubric Generation LLM-as-a-Judge
University of Arizona Department of Computer Science
GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation
Aug 30, 2026Yifan Chen, Haitao Li, Qingyao Ai +4LLM-as-a-Judge Language Model Generation Evaluation
Department of Computer Science and Technology, Tsinghua University · National University of Singapore
RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation
Mar 21, 2026Kaustubh D. Dhole, Eugene AgichteinAutomatic Rubric Generation LLM Evaluation
Department of Computer Science, Emory University, Atlanta, GA, USA

May 28, 2026J/K move · Enter open · S save

Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge

Zijie Wang, Eduardo Blanco

University of Arizona Department of Computer Science

Automatic Rubric Generation LLM-as-a-Judge Language Model Generation Evaluation Rubric-Based Evaluation

Open paper AskPDF arXiv

LLM-as-a-Judge is a scalable alternative to human evaluation, yet existing rubric-based methods rely on human-annotated data such as reference answers or expert-crafted rubrics. We propose to automatically generate fine-grained evaluation rubrics without any human annotation. Our training-free method generates rubrics at dataset-specific and instance-specific granularities, achieving performance competitive with existing methods across four benchmarks. We further present a method that iteratively fine-tunes a rubric generator model via meta-judge reward signals. The fine-tuned generator outperforms all existing baselines in both pairwise and pointwise evaluation. Notably, a fine-tuned 14B rubric generator outperforms a much larger proprietary model at rubric generation, showing the effectiveness of our fine-tuning strategy.

Abstract

Automatic Rubric Generation

Explore similar work

Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge

GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation

RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation