Beyond the Verdict: Agentic Code Judging with Verifiable Feedback

Zheye Deng; Sha Li; Changlong Yu; Xin Liu; Qin Lu; Priyanka Nigam; Yangqiu Song

cs.AIOct 13, 2026

Beyond the Verdict: Agentic Code Judging with Verifiable Feedback

Authors: Zheye Deng, Sha Li, Changlong Yu, Xin Liu, Qin Lu, Priyanka Nigam, Yangqiu Song

Organizations: HKUST · Amazon

Abstract

Test-time scaling for large language models (LLMs) can involve generating multiple candidate solutions for selection or revising an earlier attempt. To guide selection and revision, we need a verifier that can estimate candidate correctness and provide evidence of failure. In competitive programming, however, official judging systems provide authoritative verdicts under finite submission budgets, without revealing failure explanations or failing inputs. To provide informative feedback without consuming official submissions, we formulate a task in which a model judges and diagnoses a candidate program using only the problem statement and source code, with generated counterexamples serving as checkable evidence. To evaluate this task, we build USACO-JUDGE from the USACO 2026 contests, using accepted and unaccepted programs written by human experts and multiple LLMs. We then train ARBITER, a 35B mixture-of-experts (MoE) agentic verifier, with reinforcement learning in a generic execution sandbox. Training raises counterexample success on incorrect programs from 40.61% to 62.67% and mean binary verdict accuracy on USACO-JUDGE from 71.41% to 87.41%, approaching frontier verification performance. Beyond verification, ARBITER outperforms baselines in both code selection and refinement. In an IOI 2026 case study, ARBITER-guided refinement raises GPT-OSS-120B's expected score above the gold-medal threshold under the official submission limit.

Figures & tables

Panel of Figure 1 — Figure 1: Comparison of Arbiter with an online judge ( left ) and the IOI 2026 case study ( right ). Under the same 50-submission cap per problem, the expected Arbiter score crosses the gold threshold in round 4; the expected online-judge-feedback score remains below it.

Panel of Figure 2 — Figure 2: A verification instance. The verifier’s answer and the judge-side evaluation are shown in order; judge-only results are not revealed during the verifier’s episode.

Panel of Figure 3 — Figure 3: The verification pipeline assigns partial rewards according to each outcome and combines them into the final reward used to train Arbiter , with each component tied to a separate check.

Panel of Figure 4 — Figure 4: Reward ablations. Left : reward, verdict accuracy, and tool use under verdict-only training. Right : Accepted precision and recall for false-positive penalties $\lambda=0$ and $0.5$ .

Panel of Figure 5 — Figure 5: Adaptive length control. The panels compare mean reward, response length, and abort rate across two runs continued from the same checkpoint. Shading marks the active penalty phases.

Panel of Figure 6 — Figure 6: USACO-Judge composition by difficulty, verdict, failure type, and official score band.

Model	Params	UOJ-Bench (Hacking)				USACO-Judge				Overall Hack Succ.
		Verdict Acc.		Hack Succ.		Verdict Acc.		Hack Succ.
		Easy	Hard	Easy	Hard	Easy	Hard	Easy	Hard
Non-agentic
Qwen3.6-35B-A3B	35B	86.85	87.57	16.70	10.85	66.42	64.55	32.04	21.65	16.10
GPT-OSS-120B	120B	79.12	77.06	29.09	20.07	60.74	66.88	33.89	36.84	26.59
DeepSeek-V4-Flash-Preview	284B	73.49	73.10	33.49	22.03	74.94	68.68	55.00	41.52	31.26

Table 1: Results on UOJ-Bench (Hacking) and USACO-Judge , reported by difficulty split (Easy / Hard) and grouped by inference mode: non-agentic ( , single-shot) versus agentic ( , tool-using).

Panel of Figure 7 — Figure 7: Test-time scaling on the Qwen3.6-35B-A3B backbone. We draw eight samples per candidate and report Hack@ $k$ , the expected success rate of a uniformly chosen subset of $k$ samples.

Panel of Figure 8 — Figure 8: Human audit of Arbiter diagnoses. Counterexample success tracks correctness ( Left ), while correct diagnoses usually locate the bug and over half suggest a fix ( Right ).

Condition	Qwen3.6-35B-A3B						GPT-OSS-120B
	Easy (9)		Medium (9)		Hard (12)		Easy (9)		Medium (9)		Hard (12)
	@1	@8	@1	@8	@1	@8	@1	@8	@1	@8	@1	@8
Starting code	69.5	69.5	52.4	52.4	0 9.5	0 9.5	73.8	73.8	62.8	62.8	12.8	12.8
Previous code only	52.4	93.6	38.9	59.8	0 5.8	16.6	88.1	89.6	75.9	81.6	21.8	28.5
Online Judge	68.9	98.1	49.2	64.7	10.2	17.4	80.7	94.3	73.0	86.0	20.5	35.5
Arbiter : $\mathbf{V}+\mathbf{X}$	90.1	92.5	62.4	65.9	0 6.4	13.4	95.8	96.8	84.0	84.1	26.3	35.8

Table 2: Refinement from shared, already judged seeds on 30 non-Bronze USACO 2026 problems. The tiers are Silver, Gold, and Platinum

+

Open. Budgets count one or eight additional judge submissions; inference compute is unmatched. Bold and underlined entries mark the top two conditions.

Selection method	Qwen3-30B-A3B-Thinking				Qwen3-235B-A22B-Thinking
	LiveCodeBench v6		OJBench		LiveCodeBench v6		OJBench
	Pool=8	Pool=64	Pool=8	Pool=64	Pool=8	Pool=64	Pool=8	Pool=64
Uniform selection	66.2	66.2	28.9	28.9	74.7	74.7	37.1	37.1
Grading RM ( Liu et al., 2025 )	67.1	68.0	28.7	29.3	75.9	74.9	37.6	34.9
MBR-Exec (hard) ( Shi et al., 2022 )	68.5	69.2	29.6	32.4	77.9	77.7	37.7	39.6
MBR-Exec (soft) ( Shi et al., 2022 )	70.0	70.0	31.4	31.8	78.5	78.5	40.5	41.4

Table 3: Pass@

1

following Ma et al. (2026) . Baselines are published; Arbiter and perfect selection use regenerated pools with unmatched inference compute.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

	Judge verdict
Score band	AC	WA	TLE	MLE	RE	CE	Total
100 (accepted)	3,000	–	–	–	–	–	3,000	15.0%
76–99 (near-miss)	–	4,460	3,107	220	163	–	7,950	39.8%
1–75 (partial)	–	4,600	1,600	400	600	–	7,200	36.0%
0	–	1,000	–	–	–	–	1,000	5.0%
Compile Error	–	–	–	–	–	850	850	4.3%

Appendix

Table 4: Composition of the training pool by sampling score band and judge verdict. Margins report the share of the pool for each row and column.

CPID	What the checker validates
1540	Verifies the reported optimum and checks that the supplied position assignment reconstructs valid square subsequences.
1563	Simulates the submitted keystrokes and checks that they reconstruct the target string.
1566	Tests the submitted permutation and truth-teller/liar assignment against every directional claim.
1589	Replays the submitted swaps, enforces the operation limit, and checks the final string.
1597	Confirms the minimum attack count, replays the submitted attacks, and verifies that every target reaches zero HP.
1598	Validates infeasibility claims or tests the submitted array against every min, max, sum, and xor query.

Appendix

Table 5: Custom output checkers in USACO-Judge , indexed by official USACO CPID . The other 33 problems use UOJ built-in token comparators.

Model	Params	Token Price ($/M Tokens)	UOJ-Bench		USACO-Judge		Overall Hack Succ.	Output Cost per Verification
Model	Params	Token Price ($/M Tokens)	Easy	Hard	Easy	Hard	Overall Hack Succ.	Output Cost per Verification
Non-agentic
Qwen3.6-35B-A3B	35B	$0.10	0 9.6K	10.6K	11.4K	10.9K	16.10	$0.0011
GPT-OSS-120B	120B	$0.17	0 3.8K	0 4.5K	0 4.3K	0 4.1K	26.59	$0.0007
DeepSeek-V4-Flash-Preview	284B	$0.66	16.6K	19.4K	18.8K	22.0K	31.26	$0.0129
Qwen3.5-397B-A17B	397B	$0.90	16.7K	18.4K	28.1K	28.2K	42.73	$0.0195

Appendix

Table 6: Mean output tokens per candidate (K), overall hack success, output-token price, and estimated cost per verification using the reported average output lengths. Lower output-token usage and cost are better.

Non-agentic
Model	Easy				Hard
Model	Att.	Exec.	Valid	Succ.	Att.	Exec.	Valid	Succ.
Qwen3.6-35B-A3B	$86.85$	$96.49$	$92.03$	$22.89$	$88.45$	$96.95$	$89.92$	$16.23$
GPT-OSS-120B	$83.23$	$91.82$	$92.80$	$41.02$	$83.12$	$89.34$	$90.04$	$30.01$
DeepSeek-V4-Flash-Preview	$80.04$	$90.40$	$95.38$	$48.51$	$79.60$	$90.03$	$94.84$	$32.41$
Qwen3.5-397B-A17B	$98.47$	$89.26$	$95.02$	$49.91$	$98.14$	$88.75$	$95.05$	$37.18$

Appendix

Table 7: Verification funnel and acceptance quality by benchmark split. Each funnel stage is conditional on the preceding stage. Prec. and Rec. treat Accepted as the positive class.

Panel of Table 8 — Table 8: Percentage composition of validator-approved input sizes and successful-hack verdicts. Each bar runs from $<1$ KB or WA (left), through other cases, to $\geq 1$ MB or TLE (right).

Model	Params	UOJ-Bench		USACO-Judge
Model	Params	Easy	Hard	Easy	Hard
Qwen3.6-35B-A3B	35B	16.3	17.1	12.1	16.2
DeepSeek-V4-Flash-Preview	284B	21.1	21.6	15.3	22.2
Arbiter (Ours)	35B	17.5	18.7	14.2	17.8

Appendix

Table 9: Mean interaction turns per candidate in the agentic setting. Non-agentic baselines use one turn and are omitted.

Turn	Reasoning and observation
①	Identifies that S[i][j] stores the sum of the window whose top-left corner is $(i,j)$ . It derives the exact range of windows affected by an update and compares the candidate with brute force on 100,000 random tests, finding no mismatch.
②	Checks the worst-case cost. Each update touches at most $K^{2}\leq 625$ windows, giving at most 18.75 million updates over the full input. It then generates a constraint-scale timing instance.
③	Repairs the timing input so repeated updates to a cell remain strictly increasing, compiles the candidate, and measures 0.088 seconds for 30,000 updates.
④	Rechecks boundary cases, integer range, memory use, and input format, then returns Accepted with no counterexample generator.

Appendix

Table 10: An Arbiter trajectory for USACO-Judge submission 913421. Four turns of analysis and testing correctly support its Accepted verdict.

Turn	Reasoning and observation
①	Derives the geometric interpretation of the task, compiles the candidate, and confirms that it passes the supplied examples.
②	Checks the angular grouping and two-pointer count, then begins randomized differential testing against a brute-force solver.
③	Repairs an error in the brute-force harness. The process-per-case version is too slow to complete the planned search.
④	Batches 2,000 random instances into one candidate invocation. No wrong answer is found, so the search shifts from functional correctness to runtime.
⑤	Spots the $c^{2}$ loop after grouping and constructs one group of size 200,000. Running the candidate on this input exceeds the local timeout.
⑥	Checks a small all-identical instance by enumeration and confirms that its correct answer is zero.

Appendix

Table 11: A successful Arbiter trajectory for USACO-Judge submission 912773, identifying and triggering a Time Limit Exceeded failure.

Turn	Reasoning and observation
①	Traces the DSU-based construction and notices that zero denotes both a legal lower bound and the initial state. It compiles the candidate and observes -1 on the minimal instance.
②	Connects the failure to the K_min[j] != 0 guard, then checks the candidate on the first supplied examples.
③	Runs the second supplied example and confirms that alternative printed arrays still satisfy its constraints.
④	Verifies that $N=Q=1$ and $k=0$ are legal, then checks directly that the array $[0]$ satisfies the constraint.
⑤	Replays the three-line input and again observes -1 , fixing the failure mechanism and expected output before submission.
⑥	Returns Wrong Answer , identifies the sentinel collision, and submits the generator. UOJ validates the input and confirms the wrong answer.

Appendix

Table 12: A successful Arbiter trajectory for USACO-Judge submission 913195, identifying and triggering a Wrong Answer failure.

	Qwen3-30B-A3B-Thinking		Qwen3-235B-A22B-Thinking
Reference Verdict \ Arbiter Prediction	Accepted	Unaccepted	Accepted	Unaccepted
Accepted	$88.81$	$11.19$	$88.28$	$11.72$
Unaccepted	$3.64$	$96.36$	$6.33$	$93.67$

Appendix

Table 13: Row-normalized binary verdict confusion matrices for code selection. Rows are reference verdicts, columns are Arbiter predictions, and values are percentages.

Panel of Figure 9 — Figure 9: Pass@ $1$ as Arbiter calls per candidate increase from one to eight. Rows show candidate pools of 8 and 64. Uniform and perfect selection mark the lower and upper reference levels.

	GPT-OSS-120B		Qwen3.6-35B-A3B
Reference Verdict \ Arbiter Prediction	Accepted	Unaccepted	Accepted	Unaccepted
Accepted	$97.20$	$2.80$	$96.37$	$3.63$
Unaccepted	$9.68$	$90.32$	$2.65$	$97.35$

Appendix

Table 14: Row-normalized binary verdict confusion matrices during refinement. Rows are reference verdicts, columns are Arbiter predictions, and values are percentages.

Diagnosis audit	Count	Share
Correct	332	83.0%
Incorrect	65	16.2%
Vacuous	3	0.8%

Appendix

Table 15: Human labels for the division-balanced random audit. Shares are computed within the audit sample.

Depth	Count	Share	Operational rule
Actionable fix	88	54.0%	Cites candidate code and states the correct behavior.
Located cause	46	28.2%	Cites candidate code and explains the failure, but gives no fix.
Mechanism only	17	10.4%	Explains the failure without locating it in the code.
Symptom only	12	7.4%	Reports the failure without locating or explaining it.

Appendix

Table 16: Depth of the 163 correct diagnoses that identify a bug. Each level follows the operational rule shown in the table.

Depth and CPID	Audited case
Actionable fix CPID 1545	On the generated five-node instance, cow 2 hosts a C party on the first night. The diagnosis identifies that range_chmax updates the queried depth before oldOwner is recovered, so the program mistakes the updated node for the previous owner and prints 0 0 0 instead of 2 0 0 . It gives the precise repair: call point_query before range_chmax .
Located cause CPID 1545	The generated instance is a two-node cycle. Cow 2 first hosts C and cow 1 then hosts O. The diagnosis traces the failure to the second cycle host: when prevIdx+1 >= Lsize , process_interval reads past the cycle boundary. The candidate consequently prints 0 2 0 instead of 1 1 0 . The diagnosis locates the failing condition and explains its effect, but does not prescribe a repair.
Mechanism only CPID 1590	For cyclic costs [7,10,2] and a time budget of 34, the diagnosis explains that the candidate incorrectly buys later wins in cheapest-first order. The fixed queue requires cost 36 for the third win, whereas the candidate assumes cost 28 and reports three wins instead of two. This explains the failure mechanism and the correct behavior without pointing to a particular expression or branch.
Symptom only CPID 1573	On a two-position instance with equal input probabilities, the diagnosis reports that the candidate assigns probability 3/4 to each position instead of 1/2 , producing a total of 3/2 . It only speculates about the DP sweep and residue contributions, without identifying a responsible operation or establishing a causal account of the failure.

Appendix

Table 17: Representative audited diagnoses at each depth. Each generated input is accepted by the validator and breaks the corresponding candidate.

	Easy		Hard
$\lambda$	Prec.	Rec.	Prec.	Rec.
0	97.56	88.89	77.78	76.67
0.5	100.00	83.33	80.98	62.86

Appendix

Table 18: Accepted -class precision and recall on USACO-Judge for the two

\lambda

settings. All values are percentages.

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 537 papers

LLM-as-a-Verifier: A General-Purpose Verification Framework
Jul 6, 2026Jacky Kwok, Shulu Li, Pranav Atreya +6LLM Answer Verification LLM Agent Evaluation
Stanford University · UC Berkeley · NVIDIA Research
CP-Agent: A Calibrated Risk-Controlled Agent for Feedback-Driven Competitive Programming
May 23, 2026Peisong Wang, Bowen Liu, Zehua Li +4Code Generation LLM-Based Program Synthesis
The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China
Verifier-Grounded Agentic RL for Provably Correct Code
Oct 13, 2026George Ma, Yuhao Zhang, Bing He +5RL for Code Generation Verified Code Generation
UC Berkeley · AWS AI Labs

Jul 6, 2026J/K move · Enter open · S save

LLM-as-a-Verifier: A General-Purpose Verification Framework

Jacky Kwok, Shulu Li, Pranav Atreya, Yuejiang Liu+5

Stanford University · UC Berkeley · NVIDIA Research

LLM Answer Verification LLM Agent Evaluation Test-Time Scaling LLM Agent Verification

Open paper AskPDF arXiv

Scaling pre-training, post-training, and test-time compute have become the central paradigms for improving the capabilities of LLMs. In this work, we identify verification, the ability to determine the correctness of a solution, as a new scaling axis. To unlock this and demonstrate its effectiveness, we introduce LLM-as-a-Verifier, a general-purpose verification framework that provides fine-grained feedback for agentic tasks without requiring additional training. Unlike standard LM judges that prompt LLMs to produce discrete scores for candidate solutions, LLM-as-a-Verifier computes the expectation over the distribution of scoring token logits to generate continuous scores. This probabilistic formulation enables verification to scale along multiple dimensions: (1) score granularity, (2) repeated evaluation, and (3) criteria decomposition. In particular, we show that scaling the scoring granularity leads to better separation between positive and negative solutions, resulting in more calibrated comparisons. Moreover, scaling repeated evaluation and criteria decomposition consistently lead to additional gains in verification accuracy through variance and complexity reduction. We further introduce a cost-efficient ranking algorithm for selecting the best solution among candidates using the verifier's continuous scores. LLM-as-a-Verifier achieves state-of-the-art performance on Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4%), and MedAgentBench (73.3%). Beyond verification, the fine-grained signals from LLM-as-a-Verifier can also serve as a proxy for estimating task progress. We build an extension for Claude Code, enabling developers to monitor and improve their own agentic systems. Finally, we show that LLM-as-a-Verifier can provide dense feedback for RL, improving the sample efficiency of SAC and GRPO on robotics and mathematical reasoning benchmarks.

Abstract

LLM-as-a-Judge

Explore similar work

LLM-as-a-Verifier: A General-Purpose Verification Framework

CP-Agent: A Calibrated Risk-Controlled Agent for Feedback-Driven Competitive Programming

Verifier-Grounded Agentic RL for Provably Correct Code