LLM judges increasingly serve as quality gates for code changes that have no test oracle, but a judge prompt must be calibrated for its task and recalibrated as the system it judges evolves, while hand labels for the calibration are resource intensive and go stale. Development workflows emit outcome labels for free (tests pass or fail, changes land or are abandoned), but these labels are noisy. We present AutoRefine, a loop that rewrites a deployable judge's prompt to reproduce outcome labels, with an informed teacher that sees the judge's input plus the outcome, explains each label it keeps, and abstains when the outcome cannot be explained from what the judge sees. On a held-out dataset with an execution oracle, calibration raised chance-corrected agreement (Scott's π) from 0.36 to 0.60. On a deployed agent's outcome labels, with 25% flipped at random to introduce noise, the teacher set aside flipped fail labels at more than twice the rate of genuine ones, cutting overall label noise from 25% to 20%. Recalibrating that agent's judge using AutoRefine raised its agreement with merge/abandon outcomes on 17 independently held-out diffs from the current workflow (π 0.16 to 0.51); on 16 diffs from older workflows, the previous judge stayed ahead, illustrating the need for recalibration as the agent evolves. By abstaining, the teacher not only removes noisy labels but also flags changes whose outcome the judge's input cannot explain, telling a team when a judge needs better inputs rather than a better prompt.
Figures & tables
Fig. 1: The AutoRefine loop (orange: LLM calls; gray: data). The informed teacher curates the training labels once: it keeps a label with a justification grounded in the judge’s view, or abstains ( skip ) and drops the sample. Each iteration runs the deployable judge on the kept samples and scores it against their labels; disagreement analysis contrasts the judge’s reasoning with the teacher’s justification and proposes balanced prompt edits. An uncurated validation split selects the final prompt.
Judge
fail rec.
fail prec.
pass rec.
π
Rubric, uncal. (default thr.)
0.50
0.42
0.89
0.36
Rubric, uncal. (train thr.)
0.69
0.35
0.80
0.33
Rubric, calibrated
0.64
0.68
0.95
0.60
Pass/fail, uncalibrated
0.38
0.80
0.99
0.46
Pass/fail, calibrated †
0.73
0.53
0.90
0.54
TABLE I: Held-out test results (85 tasks, 13 failures), before and after calibration. Each replay rescores the same recorded patches; values are means over three replays, excluding tasks whose judge timed out or errored (about 5% of tasks per replay). † One replay.
Fig. 2: Where the performance-optimization agent’s diffs come from. The class-balanced calibration set supplies the splits of Sec. IV-C ; adding the 327 new diffs of an uncurated four-month pull gives the 512 diffs of Sec. IV-B .
Fig. 3: Where the teacher abstains. On the oracle benchmark’s validation split, it abstains on 12 of the 13 hidden-test failures and on none of the 72 passes; with injected noise on merge/abandon labels, it sets aside landed diffs relabeled abandoned more than twice as often as genuinely abandoned ones; and it abstains most on diffs abandoned after review. Labels on bars give abstentions out of n .
Slice
n
π
Landed
Abandoned
Current (sel.+test)
43
0.07→0.38
12/20→18/20
11/23→12/23
indep. test
17
0.16→0.51
6/8→8/8
4/9→5/9
before v1
36
0.05→0.44
9/15→14/15
10/21→12/21
Legacy test ⋄
16
0.50→−0.02
6/8→5/8
6/8→3/8
TABLE II: Judge performance before ( v1 ) and after ( v2 ) recalibration, on held-out diffs. Each cell gives v1→v2 . Landed (abandoned) recall: landed diffs judged pass (abandoned diffs judged fail ); landed recall is the deployment metric (target 0.90). Only the test slices were never used for selection. ⋄ Older diffs from the earlier pipeline v1 was tuned on, assigned to the test split in advance; v2 ’s drop here is consistent with distribution shift, and is why we recalibrate as the agent evolves.
Test-time scaling for large language models (LLMs) can involve generating multiple candidate solutions for selection or revising an earlier attempt. To guide selection and revision, we need a verifier that can estimate candidate correctness and provide evidence of failure. In competitive programming, however, official judging systems provide authoritative verdicts under finite submission budgets, without revealing failure explanations or failing inputs. To provide informative feedback without consuming official submissions, we formulate a task in which a model judges and diagnoses a candidate program using only the problem statement and source code, with generated counterexamples serving as checkable evidence. To evaluate this task, we build USACO-JUDGE from the USACO 2026 contests, using accepted and unaccepted programs written by human experts and multiple LLMs. We then train ARBITER, a 35B mixture-of-experts (MoE) agentic verifier, with reinforcement learning in a generic execution sandbox. Training raises counterexample success on incorrect programs from 40.61% to 62.67% and mean binary verdict accuracy on USACO-JUDGE from 71.41% to 87.41%, approaching frontier verification performance. Beyond verification, ARBITER outperforms baselines in both code selection and refinement. In an IOI 2026 case study, ARBITER-guided refinement raises GPT-OSS-120B's expected score above the gold-medal threshold under the official submission limit.
LLM judges, models that score another system's output, can be gamed by the systems they score. Recent work identifies one defence that works: the judge solves the task itself first and commits to that answer, then accepts a candidate only if the two match. We call this commit-first judging, and ask whether shipped software implements it, and what it costs. We audit the default judge configurations of eight widely used evaluation frameworks. Of the 24 configurations in scope, none implement it. Nine implement a variant the literature measures as ineffective, and share one ancestor prompt, traceable through a copied typographical error. In a controlled experiment, an ordinary best-of-N search with no access to correct answers optimises code against one of these configurations, used exactly as documented. On an interval merging task the judge accepted 90 of 96 candidates in one seed and 93 of 96 in the other; every accepted candidate passed every test the search could see and failed a held-out suite it could not. The judge identified the defective line and cited it as grounds for a perfect score. Commit-first judging removed the effect: 0 of 96 in both seeds. On a second task it made matters worse in both seeds: the judge's committed answer was wrong, and in one seed the population converged on it. This is our main finding. Commit-first judging does not remove the anchor that gets gamed, it moves it from the candidate to the judge's own answer, so evaluation is only as good as the judge is at the task. That precondition is cheap to measure in advance, and is task local rather than scale dependent: a smaller judge solved a task the frontier judge failed and resisted gaming where it did not. We also validate our own instruments: five of fifteen claims in our criteria were wrong against verbatim sources, and two held-out checks were unjustified by their specifications.
LLMs have substantially improved software engineering yet real-world development requires architectural understanding. Such understanding is prohibitively expensive to label manually and impossible to verify through tests alone. We propose an agentic judging pipeline using a strong LLM as a scalable proxy for expert architectural evaluation, comprising two judges: the Architecture Complexity Judge (ACJ), which estimates codebase-specific architectural understanding a task demands, and the Architecture Quality Judge (AQJ), which evaluates patch conformance to repository-specific architectural conventions via source-grounded rubrics. Fine-tuning Qwen3-8B/14B/32B on 3,360 curated instances achieves resolved rates of up to 27.2% on SWE-bench Verified - up to 540% over the base model and 256% over unfiltered fine-tuning. Meanwhile, the trained models achieve strong cross-language generalization and consistent improvements in architectural patch quality.
Kirill Vasilevski, Ximing Dong, Benjamin Rombaut +8
Justina · Centre for Software Excellence, Huawei Canada · School of Computing, Queen’s University, Canada +1