Best-of-N selection for code generation is only as good as the judge that picks among the candidates, and the judges accurate enough for this role are costly frontier LLMs. A small judge can be trained on abundant earlier data labeled by unit tests, but it is deployed on newer problems: the earlier problems are easier and teach it little, their tasks differ, and their candidates come from weaker policies, while data close to deployment are scarce. We present JURIST, a 1.7B reasoning judge that uses a small meta set close to deployment to learn how much each earlier training sample should count. In our experiments, we find that training on the meta set alone, or adding it to the training data, does not help, while using it to weight the earlier data through a bi-level objective adds 3.9 Hard points over adding it to the training data. On LiveCodeBench, JURIST reaches 84.7% Overall and matches Claude Sonnet 4.6 on Hard problems (73.8%) at 463x lower cost; on BigCodeBench, it surpasses all test-time scaling baselines (35.9%). Abundant earlier data, guided by a small deployment-aligned meta set, can thus train a compact judge that selects accurately at low cost.
Figures & tables
Figure 1: Cost-accuracy comparison among LLM judges on LiveCodeBench Hard problems. The x-axis is logarithmic; marker area grows with model size.
Figure 2: Bi-level data-reweighted training framework.
Figure 3: How JURIST is trained and how it selects a candidate at inference (a), with its prompt and output (b).
(i) Explains
(ii) Execution
(iii) Shift
(iv) Cost
Method
Reasoning
Compares
Exec. labels
Outcome RL
Domain
Instance
Size
Generative judges
Auto-J ( Li et al., 2024 )
✓
✓
✗
✗
✗
✗
13B
Prometheus 2 ( Kim et al., 2024 )
✓
✓
✗
✗
✗
✗
7B, 8 × 7B
Self-Taught Eval. ( Wang et al., 2024b )
✓
✓
✓
✗
✗
✗
70B
GenRM ( Zhang et al., 2025a )
✓
✗
✓
✗
✗
✗
2B–9B
Table 1: Prior work on code Best-of-N selection against the four requirements of Appendix A . ✓ : holds in part.
Method
Granularity
Params
Weight formula
Domain reweighting
domain
K
αk assigned to domain Dk
Instance table
sample
N
αi stored per (Xi,Yi)
Instance net
sample
∣θ∣
αi=MLPθ(L(Xi,Yi;ϕ))
Table 2: Data-reweighting strategies.
Objective
Margin mi
Learning signal ψi
DPO
log-ratio, chosen minus rejected
1−qi
KTO
implicit reward minus reference
λqi(1−qi)
ORPO
log odds ratio
λO(1−qi)
GRPO
logpi/(1−pi)
pi(1−pi)
Table 3: Margin mi and learning signal ψi of each objective; qi=σ(mi) , pi is the probability that the judge selects a passing candidate on sample i ( Appendix G ); λ and λO are the KTO class weight and the ORPO odds-ratio weight, respectively.
Figure 7
Figure 5: Overall pass@1 (%) on (a) LiveCodeBench and (b) BigCodeBench: JURIST against zero-shot models that solve each problem directly (grey shading) and test-time scaling baselines; (H), (M): high, medium reasoning effort. (c) JURIST against each of the four DeepSeek V3.2 Speciale candidates and an oracle judge on LiveCodeBench. The y-axes do not start at zero.
Method
o4-mini
V3.2
Qwen2.5
Qwen3-30B
Avg.
Skywork-o1 PRM 7B
74.1
68.7
25.2
36.6
51.2
DeepSeek GRM 16B
75.6
83.2
25.2
38.2
55.6
JURIST 1.7B (ours)
81.7
84.7
26.7
38.9
58.0
Table 5: Overall accuracy across policy models on LiveCodeBench. o4-mini: o4-mini (high); V3.2: DeepSeek V3.2 Speciale; Qwen2.5: Qwen2.5-Coder-32B; Qwen3-30B: Qwen3-Coder-30B-A3B.
Method
DPO
KTO
ORPO
Average
Overall
No reweighting
81.7
83.2
81.7
82.2
Domain reweighting
83.2
83.2
82.4
82.9
Instance table
82.4
84.0
83.2
83.2
Instance net
83.2
84.0
83.2
83.5
Hard
Table 7: Reweighting under three preference-optimization objectives on LiveCodeBench; all difficulty levels in Table 15 .
Method
Overall
p
Hard
p
Reweighting strategy
No reweighting
83.36 ± 1.29
.027
69.02 ± 2.25
.008
Domain reweighting
84.73 ± 0.80
–
71.64 ± 1.11
–
Single-level alternatives
Meta set only
82.14 ± 1.73
.001
66.72 ± 3.00
.002
Pooled training and meta sets
82.82 ± 0.83
.001
67.70 ± 1.90
.002
Table 8: GRPO ablation on LiveCodeBench, mean ± standard deviation over ten inference seeds; p : paired t -test against domain reweighting over the seeds.
Figure 6: Training time on 2 × A100; the upper level adds 20.36% wall-clock.
Judge
Tokens per judgment
Minutes
Overall
Hard
Untrained Qwen3-1.7B
5.7k (5.3k–6.4k)
42
79.85
61.97
RM-R1 7B
4.0k (3.1k–4.4k)
18
82.52
68.31
JURIST 1.7B (ours)
1.2k (1.2k–1.3k)
7
84.43
71.48
Table 10: Inference cost on LiveCodeBench. Tokens per judgment: mean (range) over four policy models; Minutes (one evaluation pass), Overall, and Hard: DeepSeek V3.2 Speciale candidates. All entries are means over five inference seeds.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Judge
Protocol
Overall
Hard
Cost (USD)
vs. JURIST
Random
Uniform selection
76.0
54.9
$0.00
–
Claude Haiku 4.5
Listwise
83.2
70.5
$10.97
48 ×
Claude Sonnet 4.6
Listwise
84.7
72.1
$31.19
136 ×
Claude Haiku 4.5
Pairwise ( R=8 )
85.5
72.1
$62.85
273 ×
Claude Sonnet 4.6
Pairwise ( R=8 )
86.3
73.8
$106.48
463 ×
JURIST 1.7B (ours)
Pairwise ( R=8 )
84.7
73.8
$0.23
1 ×
Appendix
Table 12: Cost and accuracy of frontier LLM judges on LiveCodeBench. Last column: cost relative to JURIST .
LiveCodeBench
BigCodeBench
Method
Overall
Easy
Medium
Hard
Method
Overall
Zero-shot Methods
o4-mini (high)
77.1
100.0
89.7
57.4
o3-mini (medium)
33.1
Gemini-2.5-Pro-06-05
72.5
100.0
82.1
52.5
o1 (high)
32.4
o4-mini (medium)
71.8
96.8
79.5
54.1
o3-mini (high)
32.4
o3 (high)
71.8
100.0
71.8
57.4
Claude-3.7-Sonnet
32.4
Appendix
Table 13: Full per-method, per-difficulty performance on LiveCodeBench and BigCodeBench.
Method
Overall
Easy
Medium
Hard
o4-mini (high)
Random
75.8
98.4
80.8
61.1
Skywork-o1 PRM 7B
74.1
96.8
82.1
57.4
DeepSeek GRM 16B
75.6
100.0
79.5
60.7
JURIST 1.7B
81.7
100.0
89.7
67.2
DeepSeek V3.2 Speciale
Appendix
Table 14: Results on LiveCodeBench by policy model.
Method
Overall
Easy
Medium
Hard
DPO
No reweighting
81.7
100.0
92.3
65.6
Domain reweighting
83.2
100.0
94.9
67.2
Instance table
82.4
100.0
94.9
65.6
Instance net
83.2
100.0
92.3
68.9
KTO
Appendix
Table 15: Full per-difficulty results of the preference-optimization ablation on LiveCodeBench.
Method
Overall
Hard
Untrained Qwen3-1.7B
79.85 ± 1.28
61.97 ± 2.13
SFT warmup
82.44 ± 1.53
67.21 ± 2.59
No reweighting
83.21 ± 1.08
68.52 ± 1.80
Domain reweighting
84.43 ± 0.87
71.48 ± 1.47
Instance table
83.97 ± 0.54
69.51 ± 1.47
Instance net
84.27 ± 0.68
69.83 ± 1.47
Appendix
Table 16: GRPO judges on LiveCodeBench over five inference seeds (mean ± standard deviation): the initialization and the instance-level reweighting strategies, with no reweighting and domain reweighting.
Figure 7: Public leaderboard results on LiveCodeBench and BigCodeBench.
Figure 8: Voting-round scaling on LiveCodeBench, averaged over four policy models.
Figure 9: Learned domain weights during training on (a) LiveCodeBench and (b) BigCodeBench.
Figure 11: Learning signal ψi of each objective ( Table 3 ), normalized to its maximum, from samples the judge solves ( q=1 ) to samples it gets completely wrong ( q=0 ). For DPO, KTO and ORPO, q=σ(mi) ; for GRPO, q=pi in the large-group limit.
Shift
Training
Meta
Test
Hard problems
23.0%
27.9%
46.6%
AtCoder-hard
9.3%
21.3%
34.4%
Trajectory source
Qwen2.5/3-Coder
o4-mini, DeepSeek
DeepSeek
Appendix
Table 17: Distribution shifts on LiveCodeBench. Test set: 31 easy, 39 medium and 61 hard problems.
Figure 12: Share of training samples at each value of q , the share of 8 judgments that are correct, placed on the learning-signal curve ψ=q(1−q) of GRPO, for the 1.7B and 8B base judges. Bubble area and number: percentage of the samples of that difficulty. Bold: samples at q=1 and q=0 , which carry no signal.
Figure 13: Share of training samples with no learning signal ( q=1 or 0 : all 8 judgments correct or all wrong) for Qwen3 base judges of three sizes before training.
Judge
Difficulty / domain
p
ψ
By difficulty
1.7B base
Easy
0.663
0.257
Medium
0.505
0.326
Hard
0.497
0.284
8B base
Easy
0.849
0.118
Medium
0.668
0.238
Appendix
Table 18: Mean probability p that the judge selects a passing candidate over 8 judgments, and mean learning signal ψ=p(1−p) , on the training samples, for the 1.7B and 8B base judges before training and for the 1.7B judge after SFT warmup, where GRPO starts (120 samples per difficulty). Bold: largest ψ per judge.
Figure 14: Training samples of each difficulty on the update map of Figure 4 , for the judge midway through training with domain reweighting. Each point is a sample with ψi>0 , jittered vertically. Top: share of samples whose weight falls or rises; bottom: share with ψi=0 . The three shares of a panel sum to 100%.
Figure 15: Alignment of training samples with meta samples of each group, after the common direction of the meta gradient is removed, double-centered ( ×10−3 , P=I ). (a) Platform and (b) candidate source, for the judge midway through training with domain reweighting. Diagonal: same-group excess with a 95% interval from resampling problems; p : permutation test of the diagonal mean.
Figure 29
Figure 16: (a) Learning signal ψ , (b) alignment and (c) their product by difficulty over training, for a run of domain reweighting whose meta set holds only hard problems ( P=I ). The alignment of a difficulty is the ψ -weighted mean alignment of its samples, and the product is the mean of ψi times the alignment.
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.