Rubric optimization improves LLM grading by revising scoring instructions while keeping the grader's parameters fixed. Existing optimizers often judge edits by aggregate validation performance, limiting the supervision they retain about which errors an edit corrects or introduces. Testing each candidate also requires rescoring validation answers, which limits how many edits can be explored within a fixed budget. We propose RubricWorld, a framework that learns from previous edit evaluations to guide further search. Its world model predicts an edit's correction and degradation rates at score boundaries, conditioned on the current rubric state. The training signal comes from paired grading results: scores for the same answers before and after an edit. Predictions select one candidate for direct evaluation, and held-out confidence bounds determine whether it corrects enough target errors while keeping new errors within prespecified limits. Measured effects update the model even when the edit is rejected. Across EIR (National), ASAP2.0, and ASAP-SAS with three frozen LLM graders, RubricWorld achieves the highest score in 20 of 27 metric comparisons. On ASAP2.0, it raises gpt-5-mini accuracy from 18.20% with the original rubric to 46.70%, exceeding the strongest baseline, APO, by 22.60 percentage points. code will be released upon acceptance.
Figures & tables
Figure 1: Overview of RubricWorld . (a) The current rubric and observed grading errors guide the generation of edits with a limited scope. (b) The world model predicts corrections and degradations for each candidate and guides the choice of an edit for evaluation. Paired grading decisions before and after the edit provide learning feedback, including for edits that are later rejected. (c) Confidence bounds on held-out answers determine whether the edit is accepted. The LLM grader’s parameters remain fixed throughout.
Method
EIR (National)
ASAP2.0
ASAP-SAS
ACC
QWK
F1
ACC
QWK
F1
ACC
QWK
F1
Meta-Llama-3.1-8B-Instruct-AWQ-INT4
Zero-shot ( Brown et al., 2020 )
55.42 ± 0.54
54.10 ± 0.46
54.85 ± 1.92
12.50 ± 2.15
5.75 ± 2.13
4.55 ± 0.70
1.99 ± 0.64
1.44 ± 0.87
1.20 ± 2.08
Few-shot ( Brown et al., 2020 )
57.12 ± 2.00
52.80 ± 0.56
55.15 ± 0.20
9.40 ± 0.54
4.78 ± 0.79
4.74 ± 1.45
19.92 ± 1.39
0.01 ± 2.25
7.75 ± 2.01
APO ( Pryzant et al., 2023 )
61.35 ± 1.17
60.15 ± 1.97
59.90 ± 1.17
15.45 ± 1.68
8.92 ± 1.27
7.98 ± 0.79
21.51 ± 0.75
8.90 ± 1.56
14.50 ± 0.61
GradeOpt ( Chu et al., 2025 )
54.20 ± 0.83
56.45 ± 0.18
53.10 ± 1.99
10.48 ± 0.68
7.37 ± 2.11
5.44 ± 1.53
33.07 ± 1.76
9.15 ± 1.53
21.09 ± 0.43
Table 1: Final grading scores across three benchmarks and frozen LLM graders. Three-seed mean ± SD; ACC/F1 in %, QWK on a 0–100 scale. Blue/red denotes changes from zero-shot; bold marks the best displayed score.
Variant
QWK@5 ↑
ACC@5 ↑
Tokens (M) ↓
5 rounds
20 rounds
Full RubricWorld
38.43
40.18
6.84
8.32
w/o State
33.16
35.42
6.41
8.68
w/o Learned Model
24.85
27.31
6.72
8.15
Table 2: Edit-effect model ablation on ASAP2.0. Average across three frozen graders with shared generator and certification.
Figure 2: Prediction, selection, and certification on ASAP2.0 ( gpt-5-mini ). (a) Held-out-state MAE (pp); (b) selected-edit correction and degradation (%); (c) test QWK (0–100) and degradation (%). Three-seed mean ± SD; variants in Appendix N .
Figure 3: Quality–cost trade-off on ASAP2.0 ( gpt-5-mini ). Three-seed means including warm-start tokens; outlined points are Pareto-optimal.
Figure 4: Illustrative rubric-editing trajectories (ASAP2.0 scale; gpt-5-mini setting). The same edit C is applied after A or B.
Figure 7
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Item
n
Scale
Train
Val/Test
U41Q1
199
0–3
159
20 / 20
U41Q2
230
0–4
184
23 / 23
U42Q1
248
0–4
198
25 / 25
U42Q2
262
0–3
209
26 / 27
U43Q1
434
0–4
347
43 / 44
Total
1,373
–
1,097
137 / 139
Appendix
Table 3: EIR (National) aggregate split statistics. Items use fixed 80/10/10 partitions, stratified by reference score where class counts permit.
Split
#Samples
Score distribution
Train
14,711
1058 / 4023 / 5326 / 3350 / 827 / 127
Test
2,596
194 / 700 / 954 / 576 / 143 / 29
Appendix
Table 4: ASAP2.0 split statistics. Score distributions are ordered from score 1 to score 6.
Data role
Answers
Ddyn,train
60 / 200 / 400
Ddyn,val
150
Dcert
1,500
Dtest
2,596
Appendix
Table 5: ASAP2.0 data allocation for edit-effect collection. The three training counts are alternative collection budgets; the certification count is the total pool size, not a per-round eligible sample count.
Split
#Samples
Score distribution
Train
1,170
266 / 301 / 366 / 237
Valid
252
Stratified from training data
Test
251
57 / 64 / 79 / 51
Appendix
Table 6: ASAP-SAS Set 1 split statistics. Score distributions are ordered from score 0 to score 3.
Method
Text supplied or changed
Selection basis
Zero-shot
Original rubric
No update
Few-shot
Scored examples appended to prompt
Validation agreement
APO
Rewritten scoring instructions
Aggregate validation score
GradeOpt
Guidelines and examples derived from errors
Validation improvement
GEPA
Prompt variants generated from reflection
Candidate-pool selection
RubricWorld
Edit to a chosen score distinction
Repair and per-boundary breakage
Appendix
Table 7: What each comparison method changes and evaluates.
Parameter
Value
Optimization rounds
5
Main-table repetitions
3 seeds; mean ± std
Long-horizon analysis rounds
100
Candidate patches per round
5
Validation sampling
By reference score
Data used to find errors
Training partition
Appendix
Table 8: Reported optimization settings for RubricWorld .
Reliable rubric grading requires more than accurate score prediction. Each judgement must be grounded in the mark scheme and evidence from the student answer. Existing credit-assignment and intervention methods, primarily designed for self-contained reasoning tasks such as mathematics reasoning, struggle in this setting because they do not identify where grading reasoning goes wrong or how the model's belief about the final mark changes during reasoning. We propose Evidence-Diagnosed Intervention Training (EDIT), a two-phase framework for training more rubric-faithful LLM graders. First, EDIT-SFT locates problematic reasoning steps using internal model signals: posterior belief over the final mark and input-grounding scores. It then revises only these local steps with help from a rubric checklist. Second, EDIT-RL calibrates the grader with belief-guided reward shaping, penalising large harmful belief drifts while still allowing helpful exploration. Experiments on two real-world, multi-subject grading benchmarks demonstrate that EDIT consistently outperforms strong supervised fine-tuning and reinforcement learning baselines on both in-domain and out-of-domain splits, with ablation studies confirming that internal-state diagnostics drive these gains. Under deterministic rubric-edit interventions, EDIT-SFT is the most rule-responsive of all evaluated systems, and EDIT-RL largely retains this responsiveness while improving accuracy.
Zhihao Wu, Linhai Zhang, Taiyi Wang +4
1King’s College London · University of Cambridge · 3AQA +1
Rubrics enable reward assignment on tasks without verifiable answers, but are expensive to manually produce from scratch. We present Rubricon, a pipeline that generates discriminative rubrics without requiring human preference labels, rubric annotation, or generator-model training. Using any LLM, Rubricon samples multiple rubrics, selects the most discriminative candidate, adds criteria to capture overlooked requirements or response flaws, and prunes criteria that are misaligned or shallow. Alongside LLM self-critique and test-time scaling, Rubricon uses observed criterion pass rates on example model responses to guide refinement. Across five professional, consumer, clinical, and general-use LLM benchmarks, Rubricon produces rubrics whose scores of frontier LLM responses exhibit 9% greater alignment with labor-intensive expert-authored rubrics, compared to the best alternative synthetic rubric generation method among RRD, QSRG, OpenRubrics, and agentic generation via harnesses like Codex. On HealthBench, post-training a Qwen3.5-9B model with Rubricon rubrics yields 4.5% better performance than training with rubrics from the strongest alternative synthetic rubric generator, RRD.
Rubrics have been widely used to evaluate and optimize language models on open-ended tasks. However, existing work primarily focuses on generating better rubrics, with limited understanding of what makes rubric design effective. In this work, we conduct a systematic mechanistic analysis of rubric design across criterion organization, criterion operationalization, and score aggregation. Our analysis shows that structural organization enables more comprehensive criterion construction, semantic specification alone does not ensure effective criterion-level judgments, and sophisticated score aggregation provides limited additional gains. Guided by these findings, we introduce Adaptive Rubric Tree Search with Hybrid refinement (ARTS-Hybrid), which combines adaptive tree construction with semantic-first, response-calibrated refinement. Experiments across benchmarks show that ARTS-Hybrid improves both end-to-end evaluation performance and intrinsic rubric quality. Further experiments suggest that, beyond evaluation, rubrics generated by ARTS-Hybrid provide effective signals for model optimization.
Feifei Qian, Xuan Chen, Lixin Cui +6
School of Artificial Intelligence, Beijing Normal University · KE Holdings Inc. · School of Information, Central University of Finance and Economics