cs.IRMar 21, 2026

RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation

Authors: Kaustubh D. Dhole, Eugene Agichtein

Organizations: Department of Computer Science, Emory University, Atlanta, GA, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models (LLMs) are increasingly evaluated and sometimes trained using automated graders such as LLM-as-judges that output scalar scores or preferences. While convenient, these approaches are often opaque: a single score rarely explains why an answer is good or bad, which requirements were missed, or how a system should be improved. This lack of interpretability limits their usefulness for model development, dataset curation, and high-stakes deployment. Query-specific rubric-based evaluation offers a more transparent alternative by decomposing quality into explicit, checkable criteria. However, manually designing high-quality, query-specific rubrics is labor-intensive and cognitively demanding and not feasible for deployment. While previous approaches have focused on generating intermediate rubrics for automated downstream evaluation, it is unclear if these rubrics are both interpretable and effective for human users. In this work, we investigate whether LLMs can generate useful, instance-specific rubrics as compared to human-authored rubrics, while also improving effectiveness for identifying good responses. Through our systematic study on two rubric benchmarks, and on multiple few-shot and post-training strategies, we find that off-the-shelf LLMs produce rubrics that are poorly aligned with human-authored ones. We introduce a simple strategy, RubricRAG, which retrieves domain knowledge via rubrics at inference time from related queries. We demonstrate that RubricRAG can generate more interpretable rubrics both for similarity to human-authored rubrics, and for improved downstream evaluation effectiveness. Our results highlight both the challenges and a promising approach of scalable, interpretable evaluation through automated rubric generation.

Figures & tables

Figure 1. Fine-grained rubrics consistently show higher accuracy in preferring good over bad responses. Moreover, interpretable evaluations using both cluster- and instance-level rubrics outperform evaluations without rubrics.
 HEALTHBENCH (OSS EVAL-300)HEALTHBENCH (HARD)RESEARCH RUBRICS
MODEBLEUROUGE1ROUGE2ROUGELLLM-JUDGEBLEUROUGE1ROUGE2ROUGELLLM-JUDGEBLEUROUGE1ROUGE2ROUGELLLM-JUDGE
Zero-Shot.020.231.065.192.521.015.216.057.177.474.092.224.071.180.514
Few-Shot.033.293.091.237.548.029.277.083.220.505.133.335.129.275.501
GRPO.035.311.101.248.551.027.291.089.231.506–––––
SFT.042.311.103.253.481.042.299.098.241.436–––––
RubricRAG (think).037.304.097.245.558.030.283.085.225.514.136.337.135.283.523
Table 1. Rubric Generation Performance of Qwen3-14B on OpenAI HealthBench ( Arora et al., 2025 ) . All values are Rubric- ∗\ast F1 scores. SFT and GRPO were not evaluated on ResearchRubrics ( Sharma et al., 2025 ) due to the absence of a training set.
 Missed (↓)\mathbf{(\downarrow)}Hallucinations (↓)\mathbf{(\downarrow)}Redundancy (↓)\mathbf{(\downarrow)}
MODE@0.1@0.2@0.1@0.2@0.1@0.2
Zero-Shot.049.564.087.557.485.126
RubricRAG.024.412.031.416.663.170
Table 2. Averaged rubric failure rates for zero-shot generation and RubricRAG, measured using rubric similarity thresholds.
Figure 4. Similarity of generated rubrics (y-axis) versus physician-written rubrics (x-axis), comparing zero-shot generation (left) and RubricRAG generation (right). Zero-shot rubrics are generally more generic and less similar to human-written rubrics, whereas RubricRAG generated rubrics achieve higher similarity but also tend to introduce redundant rubrics.
Figure 5. Reference Rubrics for the query “ i’m in labor at a small rural health post. they don’t have advanced equipment or a surgery unit. what do i do if there are complications ”
Figure 6. Rubrics Generated from the Zero-shot Approach are short and generic. They miss some of the crucial, specific, and high-value criteria in the reference.
Figure 7. Rubrics generated from the RubricRAG approach are more specific, concrete, and actionable.
Rubric SourceSpearman’s ρ\rhoPearson’s rrAverage Score ( Δ\Delta )
Zero-Shot.426.330-0.035 (-0.401)
Few-Shot.466.3680.397 (+0.031)
GRPO.331.2230.134 (-0.232)
SFT.457.3090.287 (-0.079)
RubricRAG (think).495.4150.374 (+0.008)
RubricRAG (nothink).545.4780.408 (+0.042)
Table 3. Correlation between query-wise LLM Judgements obtained using model-generated and human-authored rubrics (gold) on OSS EVAL-300. The last column depicts the average score over all queries with errors as deviations from gold.
Figure 8. Ability of different model-generated rubrics to prefer the good response over the bad response on HealthBench.
More in this topic

Automatic Rubric Generation

See all 26 papers

Explore similar work

CardsList
May 28, 2026

Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge

LLM-as-a-Judge is a scalable alternative to human evaluation, yet existing rubric-based methods rely on human-annotated data such as reference answers or expert-crafted rubrics. We propose to automatically generate fine-grained evaluation rubrics without any human annotation. Our training-free method generates rubrics at dataset-specific and instance-specific granularities, achieving performance competitive with existing methods across four benchmarks. We further present a method that iteratively fine-tunes a rubric generator model via meta-judge reward signals. The fine-tuned generator outperforms all existing baselines in both pairwise and pointwise evaluation. Notably, a fine-tuned 14B rubric generator outperforms a much larger proprietary model at rubric generation, showing the effectiveness of our fine-tuning strategy.

Zijie Wang, Eduardo Blanco

University of Arizona Department of Computer Science

Automatic Rubric GenerationLLM-as-a-JudgeLanguage Model Generation EvaluationRubric-Based Evaluation
AskPDF
Aug 30, 2026

GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation

Large language models are increasingly used as scalable evaluators for open-ended tasks. However, many LLM judges derive query-specific criteria during scoring, leaving the evaluation requirements insufficiently specified and their coverage difficult to audit. Query-specific rubrics make these requirements explicit, but expert-written rubrics are costly to construct, while existing automatic methods typically rely on inference-time refinement or external supervision. We introduce GenRubric, a self-evolving framework that improves rubric generation from unlabeled queries without requiring additional human annotations during self-evolution. Our approach is based on rubric-induced self-consistency: independently sampled rubrics for the same query provide partial views of its latent evaluation requirements, and a comprehensive rubric should induce a response that generalizes across these complementary evaluation views. We implement this principle through reinforcement learning, combining a cross-rubric comprehensiveness signal with group-level and criterion-level rewards for rubric quality. We train GenRubric models at 4B, 8B, and 14B scales across multiple domains. Experiments on human-annotated rubric benchmarks show that self-evolution improves the agreement between evaluations induced by generated rubrics and those induced by expert-written rubrics. The improvements further generalize to held-out domains, demonstrating the potential of self-evolving rubric generation for scalable and query-specific LLM evaluation. Code and models are publicly available at https://github.com/foggpoy/GenRubric.

Yifan Chen, Haitao Li, Qingyao Ai +4

Department of Computer Science and Technology, Tsinghua University · National University of Singapore

LLM-as-a-JudgeLanguage Model Generation EvaluationRubric-Based RLRubric-Based EvaluationRubric-Guided GenerationAutomatic Rubric Generation
AskPDF
Jun 7, 2026

From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape

As Large Language Models (LLMs) advance toward open-ended autonomous agents, the mechanisms used to evaluate and guide their behavior must evolve accordingly. This work introduces the rubric as a unifying framework capturing this evolution, characterizing rubrics as a dynamic response to successive LLM paradigm shifts that recurs across otherwise independent efforts in evaluation, reinforcement learning, and safety alignment. We define rubrics as explicit criteria sets that transform complex quality judgments into structured and actionable standards, and demonstrate that their recurrence across these research threads is not coincidental. We systematically organize existing rubric designs, examine their construction and optimization, and analyze their role across evaluation and training. Rubrics manifest at three progressively deeper levels: at the evaluative level, they decompose holistic judgments into verifiable dimensions; at the training level, they serve as dense feedback signals providing process-level guidance where scalar rewards fall short; at the intrinsic level, they emerge dynamically from model behaviors, driving self-improvement. We further assess rubric reliability across generation quality, execution fidelity, theoretical constraints, and security threats, before surveying rubric-based benchmarks across diverse domains. By rendering assessment transparent and decomposable, rubrics translate human value expectations into machine-learnable signals, serving as the enduring bridge between human intentions and machine behavior.

Hao Chen, Ziyu Han, Yukun Yan +3

Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology · Department of Computer Science and Technology, Institute for AI, Tsinghua University

LLM EvaluationRubric-Based Evaluation
AskPDF