Evaluating language agents on open-ended, artifact-producing tasks is difficult: model-based graders can be inconsistent, while exhaustive programmatic checks are often impractical. Textual rubrics make evaluation criteria explicit, but writing them requires domain expertise. Rubrics generated from task descriptions alone can miss requirements and failure modes that become apparent in attempted solutions. We introduce RubricCraft, which treats rubric construction as the generation of natural-language specifications guided by candidate solutions. Given a task description and independently sampled solver trajectories, RubricCraft compares how each attempt satisfies or violates the task requirements and uses these differences to construct rubric criteria. It applies the criteria to the collected solutions and revises checks that overlook errors or penalize valid alternatives. The resulting rubrics capture partial correctness without requiring any single solution to serve as a complete reference. On SpreadsheetBench v2, against a prompt-only rubric written and applied by the same model, RubricCraft rubrics pass fewer wrong outputs (29.5% against 42.6%), fail none of the outputs the benchmark's own checker accepts, and rank outputs by correctness more often, while separating models as often. Repair on a single attempt leaves grading unchanged, but repair driven by attempts from three model families acts on different tasks, and an evidence audit judges most of those repairs improvements, many on tasks whose reference solution is itself wrong.
Figures & tables
Figure 1: The running example: a rubric that looks complete in isolation, read against three attempts. All three attempts put the right totals in a new Total column, so c1 and c2 pass everywhere; green marks live formulas and passing verdicts. Red : a1 and a2 differ on whether the totals stay live, a formula against a pasted value, yet no criterion checks it, so they tie (incompleteness). Amber : c3 does not say whether restyling a header modifies Sheet2, so a3 ’s verdict depends on the judge (non-checkability).
Figure 3: RC against the naive rubric on the four axes, with the two views behind the marked rows. (a) Values are percentages unless marked pp, and n counts outputs unless marked; the seed-gap and stability rows grade GPT-6 Astra’s outputs three times. Intervals are 95% paired task-bootstrap intervals for RC’s advantage over naive, signed so that right of zero favors RC (green: excludes zero in its favor; blue: in naive’s; gray: includes zero); bold marks the better value where the interval excludes zero. AUC is the probability that a near-complete output is graded above a wrong one. ∗ Zero by construction. (b) Share of the 777 outputs graded as done, by the seed’s cell accuracy; the row marked b pools the three wrong bands at 3.5/4 , and the seed’s task bit (red) passes nothing below its all-cells threshold. Dashed lines move RC’s threshold to 3/4 and 4/4 . (c) One of the 37 outputs in the row marked c, verified from the stored cells (Appendix E.1 ).
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: The repair loop converges fast and does not inflate scores. (a) Cumulative share of the 305 gradable rubrics accepted by fix round on the full 321-task run (16 infrastructure-skipped tasks excluded): 80% pass unrepaired, 96% within one repair, and 99% within two, with three left unresolved; the third permitted repair round was never needed. (b) For the 39 repairs carrying a clean analyzer verdict at both ends and a genuinely flagged initial defect, the change in the grader’s score from the first to the final round; because each round re-sampled the attempt, the change mixes rubric and attempt differences. The distribution centers on zero (median 0 ; 13 lower, 12 higher, 14 unchanged; mean +0.16 of four points), so repair does not systematically raise scores. A further 16 of the 60 multi-round cases were retries after a transient non-JSON analyzer response rather than substantive repairs, and are excluded from (b).
Figure 5: Four rubric conditions, each grading the same 242 GPT-6 Astra outputs three times, from the uncached submissions (hollow) and from their Excel-recalculated copies (filled). (a) Seed gap: mean grade on seed-passed minus seed-failed outputs. (b) False pass: share of the 105 outputs with fewer than 80% of target cells right that are graded as done; dots are the three recalculated runs. (c) Verdict flips: share of outputs whose pass/fail verdict changes across the three gradings. Recalculated evidence roughly halves the RubricCraft conditions’ flips and removes a third of their false passes.
Condition
Construction
Mean (runs)
Seed gap
False pass
Flips
Loop + diversity
repaired, 3-model attempts
86.0 (85.5–86.3)
16.4
40.6%
9.5%
Loop (RC)
repaired, Sol attempt
85.0 (84.9–85.2)
18.0
32.1%
9.1%
No loop
first-round R0
84.2 (84.0–84.4)
19.0
32.7%
6.2%
Naive
prompt-only
89.6 (89.5–89.7)
12.2
47.3%
7.4%
Appendix
Table 1: Rubric conditions graded three times from recalculated copies of the same 242 GPT-6 Astra outputs (percent of maximum grade; ranges over the three runs). The seed gap is the mean grade on seed-passed minus seed-failed outputs, false pass uses the 105 outputs with fewer than 80% of target cells right, and flips are outputs whose pass/fail verdict changes across the three runs. No condition fails any of the 61 seed-passed outputs in any run.
Error
n
Seed
RC
Naive
RC − naive
Passes an output with <80% of target cells right
312
0% ∗
29.5%
42.6%
[−19.5,−6.7]
Fails an output the seed passes
190
0% ∗
0.0%
2.6%
[−5.2,−0.5]
Fails an output with ≥95% of target cells right
331
42.6%
2.4%
6.6%
[−7.5,−1.6]
Fails an output whose only errors are in preserved cells
37
100%
2.7%
8.1%
[−13.9,0.0]
Ties or reverses a pair ≥5 pts apart in cell accuracy
258
84.9%
34.1%
53.9%
[−28.2,−11.1]
Strictest rubric setting, pass only at 4/4 (seed row: its task bit)
Appendix
Table 2: Grading errors against the golden workbooks on the 259-task cohort (all three models’ outputs). A rubric passes an output at a grade of at least 3.5/4 and fails it below 3/4 ; the seed passes and fails by its own rule. Intervals are 95% paired task-bootstrap intervals for RC minus naive, in pp. ∗ Zero by construction: the reference is the seed’s own cells.
All 259 tasks
Without formula mode (250)
Measure
RC
Naive
Interval
RC
Naive
Interval
Wrong outputs passed
29.5%
42.6%
[−19.5,−6.7]
29.5%
44.4%
[−21.7,−8.2]
Seed-passed outputs failed
0.0%
2.6%
[−5.2,−0.5]
0.0%
2.1%
[−4.7,−0.5]
Near-complete outputs failed
2.4%
6.6%
[−7.5,−1.6]
2.1%
5.5%
[−6.8,−0.6]
Pairs tied or misordered
34.1%
53.9%
[−28.2,−11.1]
33.3%
55.4%
[−30.8,−13.8]
Appendix
Table 3: The main agreement and fidelity comparisons without the nine formula-comparison tasks (250 tasks), against the full cohort (259 tasks). Intervals are 95% paired task-bootstrap intervals for RC minus naive, in pp.
Figure 6: Figure 3 b split by task family. Bins with few outputs (for example the four seed-passed debugging outputs) have wide intervals and should not be read individually. On financial-model and template tasks, RC at 3.5/4 passes fewer low-accuracy outputs than naive; it matches naive on near-complete template outputs but passes fewer near-complete financial-model outputs that the seed fails (75% against 91% of 53). On debugging, where almost no output fixes every seeded error, both rubrics pass few low-accuracy outputs, and RC passes more of the seed-failed outputs with at least 95% of target cells right (40% against 10% of 20). Moving RC’s threshold to 4/4 lowers its curve most on financial-model tasks, where many outputs earn high but not perfect grades.
Figure 7: RubricCraft rubrics agree with the seed’s requirements; naive rubrics enumerate more of them. Semantic coverage on the 238 tasks where every rubric set completed (task means with 95% task-bootstrap intervals; only full matches count). (a) Share of fix-level requirements (what must be fixed, and where) that a rubric fully covers. (b) Share of rubric criteria fully supported by value-level requirements (each fix with up to two key expected values). (c) Share of value-level requirements that a rubric explicitly contradicts.
Cell
Golden
Submitted
LBO!C33
0.053 (constant)
=’Ex 12 - Debt Schedule’!H9
Ex 12 - Debt Schedule!H9
=F9+G9
=F9+G9
Ex 12 - Debt Schedule!F9
0.018
0.018
Ex 12 - Debt Schedule!G9
0.035
0.035
Appendix
Table 4: Cell-level comparison for debugging:03_02 , read directly from the stored golden and submitted workbooks. Only the audited cell differs, and it differs in representation rather than value: the two component cells are identical in both files, so both workbooks evaluate the rate to 0.053 .
Rubrics enable reward assignment on tasks without verifiable answers, but are expensive to manually produce from scratch. We present Rubricon, a pipeline that generates discriminative rubrics without requiring human preference labels, rubric annotation, or generator-model training. Using any LLM, Rubricon samples multiple rubrics, selects the most discriminative candidate, adds criteria to capture overlooked requirements or response flaws, and prunes criteria that are misaligned or shallow. Alongside LLM self-critique and test-time scaling, Rubricon uses observed criterion pass rates on example model responses to guide refinement. Across five professional, consumer, clinical, and general-use LLM benchmarks, Rubricon produces rubrics whose scores of frontier LLM responses exhibit 9% greater alignment with labor-intensive expert-authored rubrics, compared to the best alternative synthetic rubric generation method among RRD, QSRG, OpenRubrics, and agentic generation via harnesses like Codex. On HealthBench, post-training a Qwen3.5-9B model with Rubricon rubrics yields 4.5% better performance than training with rubrics from the strongest alternative synthetic rubric generator, RRD.
Language model-generated rubrics are increasingly used as reward signals for rubric-based reinforcement learning, LLM-as-a-judge evaluation, and automated grading. Such rubrics are reliable only if they reward honest answers over adversarial answers optimized to exploit them. Yet their robustness to such optimization remains poorly understood. We isolate the hardest regime: impossible tasks, where the prompt pressures the model toward an unsupported conclusion, so the only honest response is to acknowledge the impossibility. We introduce ImpossibleRubrics, a benchmark of 169 impossible tasks spanning six impossibility categories, each paired with a verifiable oracle certificate specifying what an honest answer may and may not claim, together with 48 answerable controls. Rather than providing fixed rubrics, ImpossibleRubrics provides task environments and certificates, allowing rubrics to be generated downstream and then adversarially tested for whether they reward certificate-violating answers. Eleven generators are exploited 8--26% of the time on the unbiased 150-of-169 environment cut; on a deliberately selected stress cut the strongest generator we measured is still exploited 36% while a certificate-faithful rubric is exploited 0%, so what we measure is a rubric-quality gap, not task impossibility. One result runs against intuition. A single generic rubric ("be decisive, penalize hedging") used unchanged for every task is exploited 64% of the time, and seven of the eleven generators are exploited more often than that while writing a rubric tailored to each one. The tailored criteria appear to tell an attacker which claim to fabricate. The problem is not that rubrics are vague; it is that they are specific about the wrong things.
Bowen Qin, Yi Xie, Yesheng Liu +1
National University of Singapore · Peking University · Institute of Automation, Chinese Academy of Sciences
Rubric-based evaluation asks which level of a rubric a response meets. It is usually done by prompting a large language model (LLM) as a judge, which needs a generative call per judgment and reads every level description, so its cost grows with the rubric. We present RubriX, a rubric judge whose architecture is inspired by contrastive language models (CLM): a LoRA-adapted encoder embeds the input and each level description, and a softmax over scaled cosine similarities gives a probability for every level without text generation. Trained on public rubric data and proprietary synthetic data, RubriX reaches 75.4% exact accuracy on held-out rubrics, against 57.1% for TypeSafe's Jev and 49.8% for a Qwen3-8B judge. On eight human-labelled benchmark sets it is within 6 percentage points of Jev's accuracy on most of them, while Jev keeps a larger lead on two and ranks responses best on average; against the results reported in the G-Eval paper, RubriX outperforms G-Eval with GPT-3.5 on every SummEval dimension and both QAGS subsets. With level embeddings cached, RubriX's latency stays at about 28 ms per judgment as level descriptions grow from 10 to 3,000 tokens, while Jev's (approximate network time subtracted) rises from 44 ms to 211 ms and the Qwen3-8B judge's from 32 ms to 516 ms, making RubriX 7.7x faster than Jev and 18.8x faster than the Qwen3-8B judge at 3,000 tokens per level.