From Structure to Validity: Mechanism-Guided Generation and Refinement of LLM Rubrics

Feifei Qian; Xuan Chen; Lixin Cui; Chaoyang Mei; Fanlin Meng; Shuyao Xiao; Ming Cui; Junxi Yin

cs.CLOct 9, 2026

From Structure to Validity: Mechanism-Guided Generation and Refinement of LLM Rubrics

Authors: Feifei Qian, Xuan Chen, Lixin Cui, Chaoyang Mei, Fanlin Meng, Shuyao Xiao, Ming Cui, Junxi Yin,

Organizations: School of Artificial Intelligence, Beijing Normal University · KE Holdings Inc. · School of Information, Central University of Finance and Economics

Abstract

Rubrics have been widely used to evaluate and optimize language models on open-ended tasks. However, existing work primarily focuses on generating better rubrics, with limited understanding of what makes rubric design effective. In this work, we conduct a systematic mechanistic analysis of rubric design across criterion organization, criterion operationalization, and score aggregation. Our analysis shows that structural organization enables more comprehensive criterion construction, semantic specification alone does not ensure effective criterion-level judgments, and sophisticated score aggregation provides limited additional gains. Guided by these findings, we introduce Adaptive Rubric Tree Search with Hybrid refinement (ARTS-Hybrid), which combines adaptive tree construction with semantic-first, response-calibrated refinement. Experiments across benchmarks show that ARTS-Hybrid improves both end-to-end evaluation performance and intrinsic rubric quality. Further experiments suggest that, beyond evaluation, rubrics generated by ARTS-Hybrid provide effective signals for model optimization.

Figures & tables

Judge Model	Coverage (%) $\uparrow$			Novelty (%) $\uparrow$
Judge Model	Flat	Flat-Multi	Tree	Flat	Flat-Multi	Tree
DeepSeek-V4-Flash-GA	52.17	56.78	57.39	9.69	9.16	11.46
Llama3.1-405B-Instruct	42.11	41.13	43.90	22.37	25.09	27.49

Table 1: Comparison between tree-structured and flat rubric generation.

Judge Model	Counterfactual Type	Exact-Match Accuracy (%) $\uparrow$
Judge Model	Counterfactual Type	Flat	Tree	Path
DeepSeek-V4-Flash-GA	Single-leaf	26.84	26.78 (-0.06)	30.77 (+3.93)
DeepSeek-V4-Flash-GA	Within-dimension	11.67	10.73 (-0.94)	12.21 (+0.54)
DeepSeek-V4-Flash-GA	Cross-dimension	23.92	16.44 (-7.48)	28.18 (+4.26)
Llama3.1-405B-Instruct	Single-leaf	33.91	32.35 (-1.56)	37.25 (+3.34)
Llama3.1-405B-Instruct	Within-dimension	14.17	13.73 (-0.44)	16.46 (+2.29)

Table 2: Criterion-level violation detection under different uses of rubric hierarchy. All results are reported as percentages, and colored values indicate absolute changes from Flat in percentage points.

Method	Criterion-Level Properties			Pairwise Accuracy (%)
Method	Disc. $\uparrow$	Judge Var. $\downarrow$	Invalid (%) $\downarrow$	Llama	Qwen	Avg. $\uparrow$
Direct	0.5105	0.2749	–	48.27	47.87	48.07
Semantic	0.5207	0.2726	0.00	49.87	47.00	48.44
Calibration	0.5450	0.2510	17.84	48.27	49.73	49.00
Semantic + Calibration	0.5295	0.2626	0.00	49.33	50.53	49.93

Table 3: Effects of semantic specification and response-based calibration on criterion-level properties and pairwise evaluation accuracy, evaluated using Llama-3.3-70B-Instruct and Qwen2.5-72B-Instruct ( Qwen et al., 2025 ) as judge models. Invalid denotes the percentage of selected candidate revisions that fail the semantic-validity constraint and is not applicable to Direct .

Aggregation Scheme	DeepSeek-V4-Flash-GA (%) $\uparrow$	Llama3.1-405B-Instruct (%) $\uparrow$
Flat	73.20	73.00
Tree	72.60 (–0.60)	71.33 (–1.67)
Flat + WU	72.20 (–1.00)	73.27 (+0.27)
Tree + WU	72.53 (–0.67)	71.33 (–1.67)

Table 4: Pairwise evaluation accuracy under different score aggregation schemes.

Figure 1: Overview of the ARTS-Hybrid framework.

Benchmark	Direct Judge	Query-Only	Response-Aware	Q-World	ARTS-Hybrid (ours)
JudgeBench	63.14	58.57	61.43	63.71	64.29
RM-Bench Chat	61.24	60.47	59.69	62.79	64.34
RB Chat	87.15	82.40	86.87	81.28	89.39
Chat-Hard	58.11	64.47	59.87	65.79	68.20
PreciseIF	52.92	56.25	55.00	56.04	55.63
Focus	79.80	80.27	84.44	83.03	85.59

Table 5: Pairwise accuracy (%) across benchmarks. The best result is highlighted in bold.

Method	Coverage	Concreteness	Non-red.	Actionability	Overall
Query-Only	1.926	2.720	2.349	2.871	2.049
Response-Aware	1.966	2.677	2.329	2.734	2.029
Q-World	2.082	2.434	1.041	2.222	1.997
ARTS-Hybrid (ours)	2.163	2.871	2.851	2.929	2.206

Table 6: Comparison of intrinsic rubric quality. Higher values indicate better performance.

SFT Supervision	JudgeBench Accuracy (%) $\uparrow$
Qwen3-4B	69.14
Direct-Prompt Rubrics	66.86
OpenRubrics Expert Rubrics	66.57
Qworld Rubrics	64.86
ARTS-Hybrid Rubrics	69.43

Table 7: The performance of Qwen3-4B rubric generators under different SFT supervision sources.

Reward Source	Thinking	Non-Thinking
Qwen3-4B	38.14	34.86
Direct-Prompt Rubric	38.08	34.76
OpenRubrics Expert Rubric	38.33	34.85
Qworld Rubric	37.64	34.95
ARTS-Hybrid Rubric (ours)	38.93	35.18

Table 8: The score performance on HealthBench under different rubric-based rewards.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Distribution of task categories and rubric counts in the mechanism analysis set.

DeepSeek-V4-Flash-GA
	Seed 1		Seed 2
Counterfactual Type	Correct	Shuffled	Correct	Shuffled
Single-leaf	23.77	23.63 (–0.13)	21.23	21.50 (+0.27)
Within-dimension	12.55	11.61 (–0.94)	12.28	11.88 (–0.40)
Cross-dimension	27.66	25.67 (–1.99)	27.09	25.11 (–1.99)
Llama3.1-405B-Instruct

Appendix

Table 9: Comparison between correct and shuffled hierarchical paths. Values in parentheses denote changes from Correct to Shuffled .

Role	Target Quality	Model
Response Generator	Strong	Qwen3-30B-A3B-Instruct-2507
Response Generator	Medium	Llama-3.1-70B-Instruct
Response Generator	Weak	Mistral-7B-Instruct-v0.2
Calibration Judge	–	Qwen2.5-72B-Instruct
Calibration Judge	–	Llama-3.3-70B-Instruct

Appendix

Table 10: Calibration response generators and judges used in ARTS-Hybrid. Each response generator produces two responses, and each calibration judge evaluates all six responses.

Figure 3: Independent validation of calibration response quality. DeepSeek-V4-Flash-GA and Gemini-3.8-Flash independently evaluate responses generated at the intended strong, medium, and weak quality levels.

Hyperparameter	Value
Leaf budget	16
Candidates per expansion	{6, 10}
Minimum children per expansion	{2, 3}
Maximum tree depth	3
Depth discount $\gamma$	0.85
Calibration responses per query	6

Appendix

Table 11: The hyperparameter configurations used by ARTS-Hybrid.

Method	Coverage $\uparrow$	Concreteness $\uparrow$	Non-red. $\uparrow$	Actionability $\uparrow$	Overall $\uparrow$
Query-Only	3.679	4.251	3.420	4.219	3.609
Response-Aware	3.711	4.262	3.248	4.216	3.600
Q-World	4.478	4.047	1.948	4.064	3.119
ARTS-Hybrid (ours)	4.009	4.405	4.073	4.402	3.938

Appendix

Table 12: Comparison of intrinsic rubric quality. Higher values indicate better performance.

Figure 4: Training dynamics under different rubric-based reward signals, including informative group rate, response length, and KL loss.

Method	Qwen3.5-397B-A17B	Llama3.1-405B-Instruct
Direct Judge	77.14	57.40
Query-Only	80.57	41.50
Response-Aware	79.43	56.00
Q-World	80.00	63.86
ARTS-Hybrid (ours)	85.67	69.71

Appendix

Table 13: Pairwise accuracy (%) across judge models. The best result is highlighted in bold.

Method	RM-Bench Chat	PreciseIF	Focus	JudgeBench
ARTS-Hybrid	64.34	55.63	85.59	64.29
w/o Semantic	50.39	54.17	77.04	61.71
w/o Calibration	58.14	55.00	86.13	62.86

Appendix

Table 14: Ablation study of semantic specification and response-based calibration. All results are pairwise evaluation accuracies (%). The best result in each column is highlighted in bold.

Automatic Rubric Generation

See all 26 papers

From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape
Jun 7, 2026Hao Chen, Ziyu Han, Yukun Yan +3LLM Evaluation Rubric-Based Evaluation
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology · Department of Computer Science and Technology, Institute for AI, Tsinghua University
Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge
May 28, 2026Zijie Wang, Eduardo BlancoAutomatic Rubric Generation LLM-as-a-Judge
University of Arizona Department of Computer Science
GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation
Aug 30, 2026Yifan Chen, Haitao Li, Qingyao Ai +4LLM-as-a-Judge Language Model Generation Evaluation
Department of Computer Science and Technology, Tsinghua University · National University of Singapore

Jun 7, 2026J/K move · Enter open · S save

From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape

Hao Chen, Ziyu Han, Yukun Yan, Qingfu Zhu+2

Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology · Department of Computer Science and Technology, Institute for AI, Tsinghua University

LLM Evaluation Rubric-Based Evaluation

Open paper AskPDF arXiv

As Large Language Models (LLMs) advance toward open-ended autonomous agents, the mechanisms used to evaluate and guide their behavior must evolve accordingly. This work introduces the rubric as a unifying framework capturing this evolution, characterizing rubrics as a dynamic response to successive LLM paradigm shifts that recurs across otherwise independent efforts in evaluation, reinforcement learning, and safety alignment. We define rubrics as explicit criteria sets that transform complex quality judgments into structured and actionable standards, and demonstrate that their recurrence across these research threads is not coincidental. We systematically organize existing rubric designs, examine their construction and optimization, and analyze their role across evaluation and training. Rubrics manifest at three progressively deeper levels: at the evaluative level, they decompose holistic judgments into verifiable dimensions; at the training level, they serve as dense feedback signals providing process-level guidance where scalar rewards fall short; at the intrinsic level, they emerge dynamically from model behaviors, driving self-improvement. We further assess rubric reliability across generation quality, execution fidelity, theoretical constraints, and security threats, before surveying rubric-based benchmarks across diverse domains. By rendering assessment transparent and decomposable, rubrics translate human value expectations into machine-learnable signals, serving as the enduring bridge between human intentions and machine behavior.

Abstract

Automatic Rubric Generation

Explore similar work

From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape

Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge

GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation