Coding agents spend substantial computation generating actions and processing long interaction histories. Poor action choices can steer agents into low-value states, wasting computation and increasing API costs. Sampling and verifying alternative actions can improve the agent's decisions, but applying this procedure at every step adds substantial overhead. We introduce SWE-Verifier, a lightweight generative model that selectively triggers candidate sampling and verification. It scores the coding agent's proposed action to determine when additional candidate sampling is needed, then ranks the resulting actions before execution. We train SWE-Verifier from Qwen3.5-4B using preference data that groups contrasting actions with their resulting environment observations and rubric-guided teacher scores. Using the mini-swe-agent harness, SWE-Verifier improves resolution from 75.47% to 77.20% on MiniMax-M2.7 at 1.09 times cost and generalizes to DeepSeek-V4-Flash without training on its actions, improving resolution from 72.60% to 74.27% at 1.13 times cost. Across both models, SWE-Verifier verifies fewer than 4% of steps and achieves comparable or higher resolution than the trajectory and all-step baselines at substantially lower cost. These results demonstrate that selective candidate sampling and lightweight verification provide complementary gains in coding-agent performance and inference efficiency. Our model, code, and data are available at https://github.com/zkjzou/SWE-Verifier.
Figures & tables
Figure 1: Verification strategies for coding agents on the full 500-task SWE-Bench Verified set. (a) Trajectory-level verification selects among completed rollouts, all-step verification compares candidates at every step, and selective verification compares candidates only at selected steps. (b) Resolution gains and (c) total costs on MiniMax-M2.7: SWE-Verifier improves resolution by 1.73 percentage points at 1.09 × total cost. Both are relative to the no-verifier baseline.
Figure 2: Overview of SWE-Verifier . (a) We construct execution-aware preference data by collecting successful trajectories, sampling and filtering candidate actions, replaying them for environment feedback, and obtaining rubric-guided teacher labels. (b) We train a Qwen3.5-4B verifier to generate outcome predictions, progress assessments, rubric evaluations, and action scores. (c) At inference, the verifier scores the initial action to decide whether to generate alternatives, then ranks candidates when verification is invoked.
Policy
Strategy
Verifier Setting
Resolve (%) ↑
Total Cost ( × ) ↓
Steps (Verified %)
MiniMax-M2.7
None
No verification
75.47 ± 1.40
1.00
46.38 (0.00%)
Trajectory
LLM-as-a-Verifier@3
75.85 ± 0.87
3.50
–
Oracle@3
82.20
3.00
–
All-step
Self-Verification
75.27 ± 1.86
3.35
38.51 (100.00%)
Qwen3.5-4B
76.40 ± 0.69
2.26
39.98 (100.00%)
Selective
Untrained
76.93 ± 1.21
1.20
44.05 (12.61%)
Table 1: Verification strategies on the full 500-task SWE-Bench Verified set. DeepSeek-V4-Flash uses correct-high reasoning. Untrained uses the base Qwen3.5-4B model. Steps are mean executed actions, with verification rates in parentheses. Costs are normalized to each policy’s no-verifier baseline; e.g., 1.09 × means 9% more cost than no verification. Among non-baseline, non-oracle settings, bold/underline mark the best/second-best resolution, steps, and costs within each policy. Across both policies, selective verification improves resolution with substantially lower cost than trajectory-level or all-step verification.
Gate
Selector
Resolve (%) ↑
Steps (Verified %)
Policy Cost ( × )
Total Cost ( × )
No verification
–
68.48 ± 3.48
57.89 (0.00%)
1.00
1.00
Threshold τ=0.2
SWE-Verifier (Untrained)
MiniMax-M2.7
71.12 ± 3.48
55.50 (5.41%)
1.05
1.18
SWE-Verifier (Untrained)
70.63 ± 0.57
59.43 (5.54%)
1.16
1.16
SWE-Verifier (Score Only)
MiniMax-M2.7
73.27 ± 2.62
59.21 (5.13%)
1.14
1.29
SWE-Verifier (Score Only)
71.95 ± 3.78
61.45 (5.79%)
1.24
1.24
Table 2: Gate/selector ablation for MiniMax-M2.7 on 202 tasks, comparing Untrained and Score Only gates paired with either MiniMax-M2.7 or the corresponding lightweight verifier as selector at two thresholds. The trained Score Only gate attains the highest resolution at both thresholds when paired with MiniMax-M2.7, indicating that a lightweight learned gate can reserve a stronger selector for the decisions where it is most useful.
Figure 5
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Resolve
Δ Resolve
Steps
Verify Rate
Policy Cost
Total Cost
MiniMax-M2.7 baseline
75.5%
+0.0
46.4
0%
1.00 ×
1.00 ×
Trajectory-level upper bound
Oracle@3
82.2%
+6.7
139.1
–
3.00 ×
3.00 ×
Step-level verifier
All Steps
verifier ( n=3 )
76.2%
+0.7
38.2
100%
1.15 ×
2.16 ×
Appendix
Table 3: MiniMax-M2.7 results on the full 500-task SWE-Bench Verified set. Each step-level and small-model verification setting reports a single run on all 500 tasks. The no-verifier baseline averages three runs, and Oracle@3 selects a successful rollout whenever one exists among those three runs. Δ Resolve is measured against the averaged no-verifier baseline. Verify Rate is the fraction of trajectory steps where candidate actions are verified. Policy Cost counts only MiniMax-M2.7 policy tokens; Total Cost includes verifier-selection calls but excludes label-gate cost for selective-step rows. Small trained verifier calls are treated as near-zero relative to frontier policy calls, so their Total Cost equals Policy Cost.
Figure 5: Phase composition and cross-setting phase stability over normalized trajectory position. Colored areas show the baseline phase distribution. Lines show the fraction of steps whose phase matches the baseline within each position bin. The verifier largely preserves the baseline workflow structure, while verifier-as-tool differs most in the early and middle bins before converging later.
Verification strategy
Policy input cost
Policy output cost
Verifier input cost
Verifier output cost
Baseline
x
y
0
0
Trajectory-level
nx
ny
Vtrajin
Vtrajout
Step-level
(1−z)x[1+αc(n−1)]
(1−z)y[1+c(n−1)]
(1−z)cx
(1−z)cβy
Small step-level verifier
(1−z)x[1+αc(n−1)]
(1−z)y[1+c(n−1)]
≈0
≈0
Appendix
Table 4: Cost decomposition for the verification strategies. Here x and y are the baseline policy input and output costs, n is the number of trajectories or candidate actions, z is the fractional trajectory-length reduction, c is the realized verification rate with c=1 for every-step verification, α is the cached-input price multiplier, β is the verifier scoring cost ratio, and Vtrajin and Vtrajout are the outcome-verifier costs for trajectory-level scaling. The first candidate incurs input cost x ; each later candidate incurs cached-input cost αx .
Figure 6: cookiecutter trajectory prefix before candidate selection. Continued parts use the same figure number.
Figure 10
Figure 7: cookiecutter candidate-action slate shown to the teacher verifiers before selection, continued.
Figure 8: cookiecutter teacher outputs from the basic blind and observation-aware verifiers.
Figure 9: cookiecutter checklist-based verifier summary, checklist, candidate scores, and final choice.
Figure 11: Resolved and unresolved MiniMax evidence around the post-validation patch-construction decision.
Coding agents are increasingly deployed for end-to-end software engineering tasks, where a readily available oracle evaluator that defines the success criteria is often absent. This motivates verifications for coding agents that can reliably judge and explain their behavior. However, verification is non-trivial as coding agents are increasingly capable of solving long-horizon software engineering tasks. Their execution trajectories can be extremely long, containing hundreds of iterative interactions with code repositories and external tools. Existing verifiers either rely on general-purpose LLMs without specialized training or train dedicated critics that predominantly focus on producing scalar rewards, highlighting the relatively limited exploration of training-based generalist verification. We introduce SWE-Critix, a post-training pipeline for training generalist verifiers of coding-agent trajectories. Given a task and its execution trajectory, the learned verifier predicts the completion label and generates a natural-language analysis that explains its judgment. Our pipeline combines supervised fine-tuning (SFT) with reinforcement learning (RL) and trains a series of 30B models on trajectories from CodeForge-Preview. On a held-out test set, SWE-Critix improves classification accuracy from 57% for the base Qwen3-30B-A3B reasoning model to 66% after SFT and 78% after RL. We release our models, datasets, and training code to facilitate further research.
Jiajun Hu, Chun Yong Chong
Leibniz Research Center, Huawei, Hong Kong SAR, China · School of Information Technology, Monash University Malaysia, Subang Jaya, Malaysia
A classical intuition holds that verifying a solution is easier than producing one. For today's coding agents, this intuition is being inverted: as foundation models develop stronger reasoning capabilities and engineering harnesses grow more sophisticated, generating complex candidate solutions is no longer difficult -- reliably verifying them has become the harder problem. Every verifier we can build is only a proxy for human intent, never the intent itself. This makes verification subject to a twofold difficulty: first, intent is underspecified by nature, making it inherently hard to faithfully check whether it has been fulfilled; second, during model training, optimization widens the gap between proxy and intent -- manifesting as reward hacking or signal saturation. To address this, we characterize the quality of verification signals along three dimensions -- scalability, faithfulness, and robustness -- and argue that achieving all three simultaneously is the central challenge. We further study four reward constructions: a test verifier for general coding tasks, a rubric verifier for frontend tasks, the user as verifier for real-world agent tasks, and an automated agent verifier for long-horizon tasks. Across different task types and policy capability levels, we conduct in-depth analysis and experiments on the core challenges of reward design and how to more effectively leverage reward signals. Experiments show that targeted verification design can effectively suppress reward hacking, improve task completion quality, and achieve significant gains across multiple internal and public benchmarks. These experiences collectively point to a core observation: no fixed reward function can remain effective as policy capability continues to grow; and verification must co-evolve with the generator.
Autonomous coding agents solve repository issues by reading code, running commands, editing files, and submitting patches. Extra inference-time compute yields gains only when it produces a useful repair and supplies reliable evidence for choosing one. Three behaviors decide both, and we argue they are teachable rather than byproducts of scale, so a policy can carry them instead of a scaffold. Location diversity remains narrow, since attempts return to the same site and extra samples add no coverage. Edit diversity is left unexploited, since methodologies that differ resolve complementary issues no single run reaches. Verification misleads, since a test the agent writes for its own patch accepts many incorrect ones. Directing search by execution feedback and scoring each patch against its own reverted tree resolves 52.8% of SWE-bench Verified using 48.1% of the agent-steps an eight-sample baseline spends. Training moves these behaviors into the policy. On the 270 issues held out from SFT and RL training, weighted supervised fine-tuning raises pass@1 from 31.9% to 35.2% and pass@8 from 46.7% to 51.1%. A reinforcement objective then trains the verifier against gold-labeled repairs and incorrect variants, crediting the assertions that detect them. It raises pass@1 to 43.0% and pass@8 to 60.7%, lifts verifier precision from 26.8% to 41.7%, and more than halves false acceptance. Resolution improves on two of three out-of-distribution suites and verifier precision on all three, and the gains hold at 7B, 14B, and 30B against published coder baselines.
Muhammad Ahmed Mohsin, Myeongsoo Kim, Kangrui Ruan +3