Mubric: Mutation Testing-Guided Rubric Generation for LLM Evaluation

Jiayuxuan Yang; Jie M. Zhang; Yiling Lou; Zhenpeng Chen

cs.AISep 29, 2026

Mubric: Mutation Testing-Guided Rubric Generation for LLM Evaluation

Authors: Jiayuxuan Yang, Jie M. Zhang, Yiling Lou, Zhenpeng Chen

Organizations: Tsinghua University · King’s College London · University of Illinois Urbana-Champaign

Abstract

Rubric-based evaluation is widely used to assess LLM-based systems by decomposing response quality into task-specific scoring criteria. However, automatically generating rubrics that reliably capture task-specific quality requirements remains challenging. We introduce Mubric, a mutation testing-guided approach to rubric generation. Mutation testing, a classic software testing methodology, evaluates a test suite by injecting faults into programs and checking whether the tests detect them. We draw an analogy between test suites and rubrics: if a rubric captures an important quality requirement, introducing a corresponding defect into an otherwise high-quality response should reduce its score. Mubric first mines common defects from real pairs of preferred and dispreferred responses and abstracts these defects into reusable mutation operators, each specifying how to introduce a particular type of response defect. For a new task, it applies relevant operators to a reference response, checks whether the injected defects reduce response quality, and uses insufficiently penalized defects to refine the rubric. We evaluate Mubric on 703 tasks across four representative domains against six advanced rubric generation methods. Mubric achieves the highest overall evaluation accuracy, outperforming the strongest baseline by 7.48 percentage points.

Figures & tables

Method	GPT-5.4-mini					Gemini-2.5-flash-lite					Overall
Method	STEM	Code	IF	RB-Chat	RM-Chat	STEM	Code	IF	RB-Chat	RM-Chat
TICK	39.33	36.13	38.71	38.67	32.56	34.90	28.67	33.87	24.00	32.56	33.84
RLCF-C	47.26	33.90	51.61	43.33	35.66	39.86	42.67	41.13	32.00	30.47	39.87
RLCF-R	53.54	42.11	58.33	48.30	22.22	45.19	37.31	38.18	32.87	34.21	41.38
RLCF-B	46.03	43.08	61.54	41.10	25.24	42.22	43.28	40.00	32.87	28.07	40.32
Dynamic	49.66	46.98	45.16	51.01	29.46	42.86	36.24	36.29	48.00	17.05	40.74

Table 1: (RQ1) Evaluation accuracy (%) of Mubric and existing methods. Overall is computed as the fraction of correct predictions over all task–LLM evaluation instances across the five datasets. Bold indicates the best result in each column.

Method	STEM	Code	IF	RB-Chat	RM-Chat	Overall
Initial	45.33	50.67	44.35	58.00	39.53	47.94
Direct-Refine	48.00	46.00	40.32	52.67	37.98	45.38
On-the-Fly Ops	42.67	48.00	43.55	53.33	31.78	44.24
Mubric	51.33	53.33	50.00	60.00	42.64	51.78

Table 2: (RQ2) Accuracy (%) of Mubric and its ablation variants, evaluated with Gemini-2.5-flash-lite. Bold indicates the best result in each column.

Figure 2: (RQ3) Distribution of 201 reusable mutation operators across quality aspects and transformation types. Blank cells denote zero counts; darker cells indicate larger counts.

Dataset	Aspect
STEM	Factuality (43%) , Logical correctness (33%) , Comprehension (12%)
Code	Logical correctness (70%) , Factuality (17%) , Comprehension (13%)
IF	Comprehension (48%) , Completeness (22%) , Factuality (15%)
RB-Chat	Comprehension (51%) , Factuality (31%) , Logical correctness (5%)
RM-Chat	Factuality (56%) , Comprehension (15%) , Logical correctness (14%)

Table 3: (RQ3) Top three quality aspects and transformation types by operator selection frequency in each dataset. Percentages denote their shares among all selections within each dataset.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Task. “Hi, I have some falafel, but no tahini to put on them. Can you give me a recipe for making tahini?”
Selected operator. Fabricate $\times$ Factuality. Replace a correct factual statement with a false claim. Here, the operator changes the explanation about bitterness in the sesame-toasting step and leaves the rest of the recipe unchanged.
Reference response $y^{\mathrm{ref}}$ (excerpt)	Mutated response $\tilde{y}$ (same passage)
Toast the sesame seeds Add sesame seeds to a dry skillet over medium-low heat. Toast for 3–5 minutes, stirring frequently, until they become lightly golden and fragrant. Be careful not to brown them too much, as burnt sesame seeds can taste bitter. Let them cool for a few minutes.	Toast the sesame seeds Add sesame seeds to a dry skillet over medium-low heat. Toast for 3–5 minutes, stirring frequently, until they become lightly golden and fragrant. Toasting the sesame seeds fully removes their natural oils, which helps prevent the tahini from becoming bitter. Let them cool for a few minutes.
Injected defect. The claim that toasting fully removes sesame oils is false. The rest of the response is unchanged, so the recipe retains its ingredients and preparation sequence despite the new factual error.
Relevant items in the initial rubric	Relevant changes in the refined rubric
Item 2, weight 3. “Uses sesame seeds as the essential base ingredient and includes the core transformation into a tahini-like paste, typically by grinding/blending the seeds into a smooth or spreadable consistency.” Item 4, weight 2. “Gives clear, usable preparation steps in a sensible order, including any important preparation details such as toasting the seeds if recommended and blending until smooth.”	Added item, weight 2. “ Avoids introducing false or misleading factual claims about tahini ingredients, preparation methods, or culinary properties; all cooking guidance should be consistent with how tahini is actually made. ”

Appendix

Table 4: Example 1: Checking factual accuracy in a recipe. RM-Chat, sample 38.

Task. “How many words are spoken daily on Earth? Try to explain your answer. Your explanation should take the reader through your reasoning step-by-step.”
Selected operator. Break logic $\times$ Logical robustness. Introduce an arithmetic error while keeping the stated assumptions and final estimate unchanged. Here, the operator changes the hourly word count and reuses the incorrect value when calculating the global total. This tests whether the rubric catches errors in the calculations supporting a conclusion.
Assumptions in both responses. Both responses assume 6 billion daily speakers, an average of 1 hour of speech per speaker, and 130 words per minute.
Reference response $y^{\mathrm{ref}}$ (excerpts)	Mutated response $\tilde{y}$ (aligned excerpts)
Step 3: Estimate words spoken per minute $[\ldots]$ In one hour: $130\times 60=7{,}800$ words So an average speaking person might say about 7,800 words per day. Step 4: Multiply by the number of speakers $6{,}000{,}000{,}000\times 7{,}800$ $=46{,}800{,}000{,}000{,}000$ That gives about: $4.7\times 10^{13}$ or 47 trillion spoken words per day.	Step 3: Estimate words spoken per minute $[\ldots]$ In one hour: $130\times 60=\mathbf{78{,}000}$ words So an average speaking person might say about 78,000 words per day . Step 4: Multiply by the number of speakers $6{,}000{,}000{,}000\times\mathbf{78{,}000}$ $=46{,}800{,}000{,}000{,}000$ That gives about: $4.7\times 10^{13}$ or 47 trillion spoken words per day.
Injected defect. The mutation changes 7,800 to 78,000 in three places but leaves the product and conclusion unchanged. Thus $130\times 60$ is miscomputed, and the subsequent multiplication no longer supports the stated 47-trillion estimate.
Relevant item in the initial rubric	The same item in the refined rubric

Appendix

Table 5: Example 2: Making intermediate calculation checks explicit. RM-Chat, sample 770.

Task. “i have a form with two input: job_title, work_city, and a button which can submit make a django controller accept these two input argument, and print it on result page”
Selected operator. Break logic $\times$ Logical correctness. Change the target of a function call so that it performs the wrong action. Here, the operator changes which page the code displays after the user submits the form.
Reading the code. Django is a Python web framework. The render function displays the named page using the supplied values. In both responses, job_form.html contains the input form, and result.html displays the submitted job title and city. The excerpts show the code after it has read these two values.
Reference response $y^{\mathrm{ref}}$ (excerpt)	Mutated response $\tilde{y}$ (same code)
return render(request, " result.html ", { "job_title": job_title, "work_city": work_city })	return render(request, " job_form.html ", { "job_title": job_title, "work_city": work_city })
Injected defect. The submitted values are read correctly, but the code returns the input form instead of the result page. That form has no code to display the submitted values, so the user sees empty input fields. The mutation changes the page name in the code and updates the corresponding sentence in the explanation.
Relevant item in the initial rubric	The same item in the refined rubric

Appendix

Table 6: Example 3: Checking that form submission displays the results. Code, sample 577.

Explore similar work

GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation
Aug 30, 2026Yifan Chen, Haitao Li, Qingyao Ai +4Rubrics Large Language Model Evaluation
Department of Computer Science and Technology, Tsinghua University · National University of Singapore
Rubrics on Trial: Evolving Rubrics from a Single Query via Synthetic Pairwise Evidence
Jul 16, 2026Haocheng Yang, Licheng Pan, Yuan Lu +7Rubrics Task-Specific Rubrics
School of Computing, National University of Singapore · Xiaohongshu Inc. · School of Cyber Science and Technology, Zhejiang University +3
Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge
May 28, 2026Zijie Wang, Eduardo BlancoRubrics Llm-As-A-Judge
University of Arizona Department of Computer Science

Aug 30, 2026, cs.CLJ/K move · Enter open · S save

GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation

Yifan Chen, Haitao Li, Qingyao Ai, Fengbin Zhu+3

Department of Computer Science and Technology, Tsinghua University · National University of Singapore

Rubrics Large Language Model Evaluation

Open paper AskPDF arXiv

Large language models are increasingly used as scalable evaluators for open-ended tasks. However, many LLM judges derive query-specific criteria during scoring, leaving the evaluation requirements insufficiently specified and their coverage difficult to audit. Query-specific rubrics make these requirements explicit, but expert-written rubrics are costly to construct, while existing automatic methods typically rely on inference-time refinement or external supervision. We introduce GenRubric, a self-evolving framework that improves rubric generation from unlabeled queries without requiring additional human annotations during self-evolution. Our approach is based on rubric-induced self-consistency: independently sampled rubrics for the same query provide partial views of its latent evaluation requirements, and a comprehensive rubric should induce a response that generalizes across these complementary evaluation views. We implement this principle through reinforcement learning, combining a cross-rubric comprehensiveness signal with group-level and criterion-level rewards for rubric quality. We train GenRubric models at 4B, 8B, and 14B scales across multiple domains. Experiments on human-annotated rubric benchmarks show that self-evolution improves the agreement between evaluations induced by generated rubrics and those induced by expert-written rubrics. The improvements further generalize to held-out domains, demonstrating the potential of self-evolving rubric generation for scalable and query-specific LLM evaluation. Code and models are publicly available at https://github.com/foggpoy/GenRubric.

Abstract

Explore similar work

GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation

Rubrics on Trial: Evolving Rubrics from a Single Query via Synthetic Pairwise Evidence

Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge