RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following

Tianjun Pan; Xuan Lin; Wenyan Yang; Qianyu He; Shisong Chen; Licai Qi; Wanqing Xu; Hongwei Feng

cs.AIMar 26, 2026

RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following

Authors: Tianjun Pan, Xuan Lin, Wenyan Yang, Qianyu He, Shisong Chen, Licai Qi, Wanqing Xu, Hongwei Feng,

Organizations: College of Computer Science and Artificial Intelligence, Fudan University · Ant Group · Donghua University

Abstract

Rubric-based evaluation has become a prevailing paradigm for evaluating instruction following in large language models (LLMs). Despite its widespread use, the reliability of these rubric-level evaluations remains unclear, calling for meta-evaluation. However, prior meta-evaluation efforts largely focus on the response level, failing to assess the fine-grained judgment accuracy that rubric-based evaluation relies on. To bridge this gap, we introduce RubricEval. Our benchmark features: (1) the first rubric-level meta-evaluation benchmark for instruction following, (2) diverse instructions and responses spanning multiple categories and model sources, and (3) a substantial set of 3,486 quality-controlled instances, along with Easy/Hard subsets that better differentiates judge performance. Our experiments reveal that rubric-level judging remains far from solved: even GPT-4o, a widely adopted judge in instruction-following benchmarks, achieves only 55.97% on Hard subset. Considering evaluation paradigm, rubric-level evaluation outperforms checklist-level, explicit reasoning improves accuracy, and both together reduce inter-judge variance. Through our established rubric taxonomy, we further identify common failure modes and offer actionable insights for reliable instruction-following evaluation.

Figures & tables

Figure 1: Existing rubric-based instruction-following evaluation and our rubric-level meta-evaluation task.

Figure 2: Overview of our data construction pipeline.

Figure 3: Preliminary experiments and results on human reference set.

Category	Benchmark	Easy	Hard	Total
Constrained	InfoBench_hard	72	47	119
	ComplexBench	54	46	100
	CFBench	54	71	125
	AdvancedIF	188	152	340
	Subtotal	368	316	684
Compositional	ComplexBench	130	110	240

Table 1: Statistics of RubricEval by instruction category and source benchmark.

Figure 4: Distribution of instances across rubric categories in our taxonomy.

RubricEval-Easy
Model	Constrained		Compositional		Multi-turn		System		Overall
Model	BAcc	mF1	BAcc	mF1	BAcc	mF1	BAcc	mF1	BAcc	mF1
Llama-3.1-8B-Instruct	65.71	61.18	57.52	48.21	65.80	58.25	66.12	64.59	63.79	58.06
Llama-3.3-70B-Instruct	80.48	81.83	81.65	83.42	82.82	83.83	86.83	87.24	82.94	84.08
Qwen2.5-7B-Instruct	63.70	63.05	69.85	62.46	60.06	55.04	67.11	65.89	65.18	61.61
Qwen2.5-32B-Instruct	75.78	76.26	75.64	70.01	75.72	76.96	79.38	79.17	76.63	75.60

Table 2: Main results on RubricEval . We report performance on the Easy (top) and Hard (bottom) splits of RubricEval across four instruction types and Overall. Each setting is evaluated with balanced accuracy ( BAcc ) and macro-F1 ( mF1 ). Bold indicates the best score in each column within the same split, and underline indicates the second-best.

Granularity	Reasoning	Constrained		Compositional		Multi-turn		System		Overall
Granularity	Reasoning	Qwen	GPT	Qwen	GPT	Qwen	GPT	Qwen	GPT	Qwen	GPT
Rubric-level	✗	62.55	69.99	75.62	78.37	68.09	74.62	69.55	78.89	68.95	75.47
Rubric-level	✓	74.19	82.90	83.17	80.83	76.80	83.48	75.34	81.45	77.38 (+8.4)	82.17 (+6.7)
Checklist-level	✗	54.06	60.80	72.38	68.21	53.39	56.11	63.98	68.48	60.95	63.40
Checklist-level	✓	66.71	69.59	77.06	74.60	60.90	57.88	74.94	79.67	69.90 (+9.0)	70.44 (+7.0)

Table 3: Comparison of evaluation paradigms on a subset of RubricEval , sampled from both Easy and Hard subsets. We vary granularity and reasoning during evaluation, reporting Balanced Accuracy ( BAcc ) for Qwen (Qwen2.5-32B-Instruct) and GPT (GPT-4.1). Green values show improvement from reasoning. Bold indicates the best score in Overall column. Results on Easy and Hard subsets are in Appendix H .

Figure 5: Inter-judge analysis on CFBench. Judge variance decreases from vanilla (*) to rubric-level with reasoning.

Rubric Type	Qwen3	gpt-oss	GPT-4o	o3-mini
Content	76.9	86.1	72.2	81.0
Content Inclusion	77.1	86.8	72.3	80.6
Content Exclusion	80.4	87.8	71.4	83.9
Topic Scope	72.9	83.5	72.9	78.4
Form	77.6	87.5	67.0	87.3
Quantity Limit	78.6	86.6	68.7	88.6

Table 4: Judges accuracy (%) by rubric type. Underlined values are below the dimension average for that model.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Benchmark	#Inst.	#Rub.	R/I	H.
Constrained	InfoBench_hard	228	1,453	6.37	✓
	ComplexBench	238	1,027	4.32	✓
	CFBench	243	1,035	4.26	✓
	AdvancedIF	243	1,816	7.47	✓
Compositional	ComplexBench	435	1,651	4.49	✓
Multi-Turn	StructFlowBench	643	1,775	2.76	✓

Appendix

Table 5: Instruction sources and rubric statistics in RubricEval . Statistics are computed over the benchmark subsets used in our experiments. #Inst.: instructions; #Rub.: rubrics; R/I: rubrics per instruction; H.: human-crafted/verified.

Subset	Samples	Judge Inst.	Pos.	Neg.
Right-to-Wrong	120	240	120	120
Wrong-to-Right	133	266	133	133
Total	253	506	253	253

Appendix

Table 6: Human-annotated reference set statistics.

Figure 6: T-SNE visualization of rubric instances in the embedding space, colored by rubric category.

Model	Family	Total	Active	Arch.	Mode
Qwen3-4B-Instruct-2507	Qwen	4B	4B	Dense	Instruct
Qwen3-4B-Thinking-2507	Qwen	4B	4B	Dense	Thinking
Qwen2.5-7B-Instruct	Qwen	7B	7B	Dense	Instruct
Llama-3.1-8B-Instruct	Llama	8B	8B	Dense	Instruct
DeepSeek-R1-0528-Qwen3-8B	Qwen+DeepSeek	8B	8B	Dense	Thinking
Qwen3-30B-A3B-Instruct-2507	Qwen	30B	3B	MoE	Instruct

Appendix

Table 7: Model pool for response generation, covering 3 families, scales from 4B to 70B, Dense and MoE architectures, and Instruct/Thinking inference modes.

Category	Source Benchmark	# Instr.	# Rubric-Level Labels			Category Summary
Category	Source Benchmark	# Instr.	Easy	Hard	Total	Easy	Hard	Total
Constrained	InfoBench_hard	95	72	47	119	368	316	684
	ComplexBench	80	54	46	100
	CFBench	112	54	71	125
	AdvancedIF	219	188	152	340
Compositional	ComplexBench	188	130	110	240	130	110	240

Appendix

Table 8: Statistics of the RubricEval Benchmark.

Easy Subset
Granularity	Reasoning	Constrained		Compositional		Multi-turn		System		Overall
Granularity	Reasoning	Qwen	GPT	Qwen	GPT	Qwen	GPT	Qwen	GPT	Qwen	GPT
Rubric-level	w/o CoT	70.01	84.05	81.50	85.49	82.28	85.74	84.81	90.71	79.65	86.50
Rubric-level	w/ CoT	86.44	97.89	91.65	96.99	88.75	94.88	85.13	93.13	87.99	95.72
Checklist-level	w/o CoT	53.07	67.45	79.17	72.86	56.83	56.82	74.83	79.07	65.98	69.05
Checklist-level	w/ CoT	76.18	81.08	80.60	86.99	67.86	61.88	84.08	89.04	77.18	79.75

Appendix

Table 9: Evaluation paradigm comparison on Easy and Hard subsets.

Figure 7: Rubric taxonomy for RubricEval with 4 high-level dimensions and 13 fine-grained categories.

Type	Benchmark	Description	Used Subset	Total		Used
Type		Description	Used Subset	Inst.	Rub.	Inst.	Rub.
Constrained	InfoBench Qin et al. (2024b)	Breaks instructions into decomposed questions; evaluates instruction-following with DRFR metrics.	Hard	500	2,250	228	1,453
	ComplexBench Wen et al. (2024)	Tests multi-constraint, complex instruction following using hierarchical constraint types and combinations.	Multi-Constraint	1,150	5,297	238	1,027
	CFBench Zhang et al. (2025a)	Large-scale Chinese constraint-following benchmark spanning 200+ real scenarios and 50+ NLP tasks.	Filtered	1,000	4,273	243	1,035
	AdvancedIF He et al. (2025)	Expert-rubric benchmark for advanced instruction following (complex, multi-turn, system-level); supports rubric-based RL.	Single-turn	1,645	12,442	243	1,816
Compositional	ComplexBench Wen et al. (2024)	Tests multi-constraint, complex instruction following using hierarchical constraint types and combinations.	Compositonal	1,150	5,297	435	1,651

Appendix

Table 10: Detailed source statistics for RubricEval . We list the descriptions, used subset, and the counts of instructions/rubrics (Total available vs. Used).

Figure 8: Single-model Judge Performance on the Human-labeled Set (Accuracy)

Dimension	Rubric type	Definition	Example
Content	Content Inclusion	The response must include specific content elements (keywords, entities, components).	For each destination in the itinerary, does the generated text include the recommended duration of stay?
	Content Exclusion	The response must NOT include specific content elements.	Does the generated text free of using the letter ’e’?
	Topic Scope	The response must stay within a specified topic, domain, or area.	Were four questions related to Greek mythology?
Form	Quantity Limit	The response or its elements must meet explicit numeric limits (counts, lengths, frequencies).	Did the model cite at least 3 of the clues in each explanation?
	Format Structure	The output must follow a required format, structure, template, or match an exact output.	Is the generated text formatted as a travelogue video script?
	Ordering Sequence	Elements in the response must follow a specified order or arrangement.	Was the list appropriately organized by publishing date, from oldest to newest?

Appendix

Table 11: Rubric taxonomy of RubricEval

Figure 9: Case study on automated labeling. In some cases, we observe that our RAF framework produces more objective judgments than human annotators.

Panel of Table 12 — Table 12: Prompt used in RubricEval to evaluate rubrics belong to Constrainted instruction category.

Rubric-Based Evaluation

Rubric-based evaluation uses explicit criteria to assess model outputs, agent behavior, and student work in a structured, interpretable way.

See all 197 papers

IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation
Mar 5, 2026Bosi Wen, Yilin Niu, Cunxiang Wang +5LLM-as-a-Judge Instruction Following
The Conversational Artificial Intelligence (CoAI) Group, Tsinghua University · Zhipu AI · University of Electronic Science and Technology of China
RubricBench: Aligning Model-Generated Rubrics with Human Standards
Mar 2, 2026Qiyuan Zhang, Junyi Zhou, Yufei Wang +8LLM Evaluation Rubric-Based Evaluation
City University of Hong Kong · Tencent Hunyuan · McGill - Mila & Quebec AI Institute +2
RubriX: A Fast, Fine-Tunable Non-Autoregressive Rubric Judge
Oct 26, 2026Balamurugan Paramasivam, Sudalai RajkumarRubric-Based Evaluation LLM-as-a-Judge
CobuildX

Mar 5, 2026J/K move · Enter open · S save

IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation

Bosi Wen, Yilin Niu, Cunxiang Wang, Xiaoying Ling+4

The Conversational Artificial Intelligence (CoAI) Group, Tsinghua University · Zhipu AI · University of Electronic Science and Technology of China

LLM-as-a-Judge Instruction Following Reward Model Evaluation Learning to Rank

Open paper AskPDF arXiv

Instruction-following is a foundational capability of large language models (LLMs), with its improvement hinging on scalable and accurate feedback from judge models. However, the reliability of current judge models in instruction-following remains underexplored due to several deficiencies of existing meta-evaluation benchmarks, such as their insufficient data coverage and oversimplified pairwise evaluation paradigms that misalign with model optimization scenarios. To this end, we propose IF-RewardBench, a comprehensive meta-evaluation benchmark for instruction-following that covers diverse instruction and constraint types. For each instruction, we construct a preference graph containing all pairwise preferences among multiple responses based on instruction-following quality. This design enables a listwise evaluation paradigm that assesses the capabilities of judge models to rank multiple responses, which is essential in guiding model alignment. Extensive experiments on IF-RewardBench reveal significant deficiencies in current judge models and demonstrate that our benchmark achieves a stronger positive correlation with downstream task performance compared to existing benchmarks. Our codes and data are available at https://github.com/thu-coai/IF-RewardBench.

Abstract

Rubric-Based Evaluation

Explore similar work

IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation

RubricBench: Aligning Model-Generated Rubrics with Human Standards

RubriX: A Fast, Fine-Tunable Non-Autoregressive Rubric Judge