A coding agent repeatedly makes decisions as it repairs code. These decisions shape repair trajectories and affect the token cost of successive model requests. We ask which repair decisions can be delegated to a low-cost probabilistic judge to reduce token use while maintaining repair quality. The judges answer questions with probabilities over predefined answers, without generating explanations. We evaluate judgment quality across repair stages and test how individual interventions change subsequent repair work. We compare these interventions with unguided repair, which uses no judge, and with the same workflow using the repair model itself as the judge. The benefit depends on the work an intervention changes, rather than the cost of the judgment alone. Based on these studies, we propose JevCoder, a workflow built around Jev, a low-cost probabilistic judge. It combines code-snippet recommendations with action-type guidance and uses judgments to determine when to submit. We evaluate three repair models, each on the same 400 tasks: 200 from SWE-bench Verified and 200 from SWE-Bench Pro Verified. Compared to unguided repair, JevCoder reduces mean repair-model token use by 21.6%-50.6%, with higher observed repair rates across all three models.
Figures & tables
Figure 1: In a simplified recorded SWE-Bench Pro Verified case, JevCoder uses Jev’s relevance probabilities to rank candidate snippets and supply selected code as the coding agent’s initial context.
Figure 2: JevCoder delegates judgments to a shared probabilistic judge: Jev. Before repair, a program retrieves candidate code for the issue, Jev scores each snippet’s relevance, and the highest-scoring snippets enter the repair model’s initial context. After each command, program rules select relevant state questions for Jev and use its answers to suggest an action type. The repair model chooses the specific command. The stopping policy depends on the model class. For the larger API models (DeepSeek, Gemini), a Choice question on the latest check output (Pass: a previously failing check now passes; Rest: other outcomes) triggers direct submission. For the smaller model (Qwen), repeated No answers to whether the next step adds information trigger a reminder and then submission.
Judgment
Metric
Jev
Comparator
Δ
Comparator model
Opening — Initial context
File relevance (Binary)
Top-1 hit (%) ↑
75.9
29.0
+46.9
Qwen3.5-2B
File relevance (Score)
Top-1 hit (%) ↑
94.9
88.1
+6.8
Jev (Choice) ‡
Snippet recommendations
Selection recall (%) ↑
81.6
78.0
+3.6
DeepSeek
During repair — State and action judgments
Code located
AUROC ↑
0.738
0.701
+0.038
DeepSeek/Qwen
Table 1: Judgment quality and snippet selection. Higher is better; bold marks the better point estimate. Δ is Jev minus the comparator (percentage points for percentage metrics; absolute differences for AUROC), computed before rounding. File relevance uses 100 candidate files for Binary and 10 for Score. Selection recall is measured over reference lines covered by candidate snippets. State AUROCs pool DeepSeek and Qwen inputs, each evaluated against its same-model judge. ‡ Jev (Choice) returns probabilities over the same ten candidate files.
Table 2: Component comparisons relative to matched unguided repair on N tasks. Values are changes in repair rate (pp) and in mean repair-model tokens (M); bars show the direction and size of each change. Green and red mark improvement and degradation; grey marks small changes (absolute change below 1 pp or 0.01M). A check mark denotes adoption in JevCoder. 95% intervals and full comparisons appear in Appendix D .
Condition
Repair (%) ↑
Repair-model tokens (M) ↓
Runtime (s) ↓
Repair
Judging
Total
DeepSeek-V4.1-Flash
Baseline
76.25
1.978
—
1.978
181.7
Same-model judging
77.00 ( +0.75 )
1.441
0.265
1.706 ( −13.7% )
287.6 ( +58.2% )
JevCoder
77.25 ( +1.00 )
1.550
†
1.550 ( −21.6% )
162.7 ( −10.5% )
Qwen3.8-27B
Table 3: JevCoder evaluation on 400 matched tasks per condition. Tokens are per-task means; runtimes are medians. Parentheses indicate changes from Baseline (repair: pp; token means and runtime medians: %). Bold marks the best repair rate, Total tokens and runtime within each model. † : Jev tokens are excluded from repair-model totals and reported separately (§ 7 ).
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Judgment
Answers
Question
Snippet relevance
yes / no
Considering only state.blocks.{id} : does this exact source directly implement, branch on, carry data for, call, or test the behavior described in state.issue ? The current buggy implementation counts; shared vocabulary alone does not. Source in other blocks is not evidence for this block.
Code located
yes / no
Has the agent opened code the issue names or clearly implies (function, class, file)?
Issue reproduced
yes / no
Does the latest output show the misbehaviour the issue describes?
Post-edit check
yes / no
Since the last source edit, does any output show the issue’s scenario behaving as the issue expects?
Stalled repair
yes / no
Do the last 6 commands pursue the same goal without new information?
Issue-related activity
yes / no
Do the agent’s last few commands still work on code related to the behavior the issue describes?
Appendix
Table 4: Fixed judge questions. Stalled repair, code located, issue reproduced, and post-edit check drive action-type guidance; the proposed-command question drives command blocking and Qwen’s no-progress submission; the successful-check question drives automatic submission; snippet relevance drives code-snippet recommendations. Issue-related activity and patch outcome are evaluated offline only. Field names in typewriter font refer to fields of the evidence payload.
DeepSeek trajectories
Qwen trajectories
Judgment
Positive/total
Jev
Same-model
Positive/total
Jev
Same-model
Code located
1,525/1,598
0.751
0.687
1,494/1,599
0.731
0.776
Issue reproduced
115/1,234
0.832
0.845
130/1,599
0.816
0.771
Stalled repair
44/1,578
0.914
0.810
322/1,599
0.971
0.972
Post-edit check
158/443
0.725
0.735
56/508
0.845
0.831
Appendix
Table 5: Repair-state AUROC on identical inputs for each repair model. Counts give positive labels and total sampled states; each model’s states combine both benchmarks. The pooled results appear in Table 1 .
Comparison
Repair model
Tasks
Δ Repair rate (pp)
Mean Δ Tokens (M)
File recommendations (Binary)
DeepSeek
118
+0.50
−0.354
[−4.97,+5.78]
—
File recommendations (Score)
DeepSeek
118
−3.58
−0.372
[−10.06,+2.89]
[−0.730,−0.050]
Snippet recommendations
DeepSeek
118
+3.89
−0.206
[−2.28,+10.47]
[−0.566,+0.167]
Appendix
Table 8: Paired component comparisons behind Table 2 , with 95% intervals below the estimates. Each change is the component condition minus matched unguided repair, except snippet withdrawal, which is withdrawal minus full retention. The 118- and 116-task SWE-Bench Pro Verified samples are reweighted to the original failure/success proportions; the 72-task samples are unweighted. For binary file recommendations, intervals are reported separately for input and output tokens in the text.