RubriX: A Fast, Fine-Tunable Non-Autoregressive Rubric Judge

Balamurugan Paramasivam; Sudalai Rajkumar

cs.CLOct 26, 2026

RubriX: A Fast, Fine-Tunable Non-Autoregressive Rubric Judge

Authors: Balamurugan Paramasivam, Sudalai Rajkumar

Organizations: CobuildX

Abstract

Rubric-based evaluation asks which level of a rubric a response meets. It is usually done by prompting a large language model (LLM) as a judge, which needs a generative call per judgment and reads every level description, so its cost grows with the rubric. We present RubriX, a rubric judge whose architecture is inspired by contrastive language models (CLM): a LoRA-adapted encoder embeds the input and each level description, and a softmax over scaled cosine similarities gives a probability for every level without text generation. Trained on public rubric data and proprietary synthetic data, RubriX reaches 75.4% exact accuracy on held-out rubrics, against 57.1% for TypeSafe's Jev and 49.8% for a Qwen3-8B judge. On eight human-labelled benchmark sets it is within 6 percentage points of Jev's accuracy on most of them, while Jev keeps a larger lead on two and ranks responses best on average; against the results reported in the G-Eval paper, RubriX outperforms G-Eval with GPT-3.5 on every SummEval dimension and both QAGS subsets. With level embeddings cached, RubriX's latency stays at about 28 ms per judgment as level descriptions grow from 10 to 3,000 tokens, while Jev's (approximate network time subtracted) rises from 44 ms to 211 ms and the Qwen3-8B judge's from 32 ms to 516 ms, making RubriX 7.7x faster than Jev and 18.8x faster than the Qwen3-8B judge at 3,000 tokens per level.

Figures & tables

Panel of Figure 1 — Figure 1: The RubriX scorer. The input and each level description are encoded separately by the same encoder, projected by two heads, and compared by temperature-scaled cosine similarity. A softmax over the rubric’s own $K$ levels gives a probability for each level. Level embeddings do not depend on the input, so they can be computed once per rubric and cached.

Source	Criteria	Levels	Examples
Feedback Collection [ 15 ]	898	5	22,450
HelpSteer3, principles [ 19 ]	1,046	2	3,372
HelpSteer, HelpSteer2 [ 27 , 28 ]	8	5	200
HelpSteer3, feedback [ 19 ]	3	5	75
UltraFeedback [ 8 ]	4	5	100
ASAP 2.0 [ 7 ]	1	6	25

Table 1: Training data. Public sources are capped at 25 examples per criterion, level-balanced.

Benchmark	What is judged	Labels from	Judgments
HealthBench [ 1 ]	health conversations, consensus criteria (met / not met)	2–5 physicians	29,511
BiGGen [ 16 ]	responses with instance-specific rubrics, nine capabilities (5 levels)	human raters	3,194
ELLIPSE [ 6 ]	learner essays, overall proficiency and six traits (5 levels)	two trained raters	45,374
SummEval [ 11 ]	news summaries, four dimensions (5 levels)	three experts	6,400
JudgmentBench, binary [ 14 ]	legal answers, checklist items (met / not met)	practising lawyers	17,365
JudgmentBench, multi-level [ 14 ]	legal answers, counted errors (4 levels)	practising lawyers	2,782

Table 2: Evaluation benchmarks.

Figure 2: Held-out test set: 6,749 judgments on rubrics whose criteria and level descriptions never appear in training.

Figure 3: Exact accuracy on the human-labelled benchmarks.

Figure 4: Within-one accuracy on the multi-level benchmarks.

		RubriX		Jev		Qwen3-8B judge		G-Eval-4		G-Eval-3.5
Benchmark	Category	$\rho$	$\tau$	$\rho$	$\tau$	$\rho$	$\tau$	$\rho$	$\tau$	$\rho$	$\tau$
SummEval	Coherence	0.581	0.450	0.649	0.510	0.432	0.330	0.582	0.457	0.440	0.335
	Consistency	0.496	0.413	0.559	0.473	0.458	0.377	0.507	0.425	0.386	0.318
	Fluency	0.435	0.360	0.484	0.402	0.461	0.381	0.455	0.378	0.424	0.347
	Relevance	0.422	0.328	0.474	0.370	0.360	0.280	0.547	0.433	0.385	0.293
	AVG	0.483	0.388	0.541	0.439	0.428	0.342	0.514	0.418	0.401	0.320

Table 3: Ranking correlation with human scores: summary-level Spearman

\rho

and Kendall

\tau

within groups of responses to the same input, averaged per category; AVG is the mean over a benchmark’s categories. G-Eval values are copied from its paper (Table 1 for SummEval, Table 3 for QAGS); it does not report the other benchmarks. Bold marks the best judge in each row and underline the second best, separately for

\rho

and

\tau

. A group is one article’s summaries for one dimension (SummEval), the essays written to one prompt for one trait (ELLIPSE), one conversation’s replies for one criterion (HealthBench), one instance’s responses (BiGGen) or the answers to one task for one rubric item (JudgmentBench); for QAGS each subset is one group.

Figure 5: Median latency per judgment, one request at a time, against the number of levels (1,500 tokens each); labels give the median in ms. Latency axis on a log scale. Jev’s line has an approximate network time subtracted: 232.6 ms, the median time, over 50 fresh connections from our client, from the end of the connection handshake to the first byte of the reply to an empty request to its API. RubriX and the Qwen3-8B judge were timed on the H100 itself and show processing only. Jev accepts at most 10 levels and the Qwen3-8B judge’s 32k context holds at most 20 levels of this length.

Figure 6: Median latency per judgment against the length of each level description (5 levels), as in Figure 5 .

Rubric-Based Evaluation

Rubric-based evaluation uses explicit criteria to assess model outputs, agent behavior, and student work in a structured, interpretable way.

See all 197 papers

ExecRubrics: Executable Tool-Augmented Rubrics for Verifiable and Efficient Long-Form Evaluation
Aug 23, 2026Kaustubh D. Dhole, Charles L. A. Clarke, Eugene Y. AgichteinLLM Evaluation Rubric-Based Evaluation
Department of Computer Science Emory University Atlanta, USA · School of Computer Science University of Waterloo Waterloo, Canada
Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge
May 28, 2026Zijie Wang, Eduardo BlancoAutomatic Rubric Generation LLM-as-a-Judge
University of Arizona Department of Computer Science
GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation
Aug 30, 2026Yifan Chen, Haitao Li, Qingyao Ai +4LLM-as-a-Judge Language Model Generation Evaluation
Department of Computer Science and Technology, Tsinghua University · National University of Singapore

Aug 23, 2026J/K move · Enter open · S save

ExecRubrics: Executable Tool-Augmented Rubrics for Verifiable and Efficient Long-Form Evaluation

Kaustubh D. Dhole, Charles L. A. Clarke, Eugene Y. Agichtein

Department of Computer Science Emory University Atlanta, USA · School of Computer Science University of Waterloo Waterloo, Canada

LLM Evaluation Rubric-Based Evaluation Automated Evaluation

Open paper AskPDF arXiv

Rubrics aim to make language-model evaluation transparent by decomposing response quality into interpretable criteria. However, natural-language rubrics are often ambiguous, require LLM judges, and typically assume criteria aggregated through linear weighted sums, limiting their ability to capture dependencies, alternatives, penalties, and override conditions. We propose ExecRubrics, a framework for representing rubrics as compact executable programs. ExecRubrics encodes evaluation logic as verifiable Python scoring functions, giving natural-language rubric intent an operational semantics: a fixed decision procedure that can be inspected, executed, and edited. On three long-form response benchmarks -- HealthBench, HelpSteer, and ArgQuality -- we show that ExecRubrics can recover substantial preference signal without an LLM judge at evaluation time. On ArgQuality and HelpSteer, the strongest executable variants are within 1.1 and 4 percentage points, respectively, of the direct GPT-5.5 agentic baseline. Executable rubrics are also considerably faster, achieving a 192x average speedup. We show that incorporating external logic and resources from text processing libraries such as NLTK and spaCy can further improve preference accuracy. Our results suggest a novel way of approaching automated evaluation, by offering a faster, more explainable, and less ambiguous alternative to black-box rubric evals, particularly in high-stakes domains such as healthcare and banking where precision and auditability are critical.

Abstract

Rubric-Based Evaluation

Explore similar work

ExecRubrics: Executable Tool-Augmented Rubrics for Verifiable and Efficient Long-Form Evaluation

Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge

GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation