PreciseCoder: Faithful and Generalizable Debugging Beyond Unit-Test Rewards

Yunkai Zhan; Miaosen Chai; Shangshang Wang; Jike Zhong; Yuqing Yang; Deqing Fu; Serina Chang; Wang Bill Zhu

cs.SEOct 13, 2026

PreciseCoder: Faithful and Generalizable Debugging Beyond Unit-Test Rewards

Authors: Yunkai Zhan, Miaosen Chai, Shangshang Wang, Jike Zhong, Yuqing Yang, Deqing Fu, Serina Chang, Wang Bill Zhu

Organizations: University of Southern California · University of Chicago · Google · University of California, Berkeley

Abstract

Reinforcement learning (RL) from unit tests is the common recipe for post-training code models, but a unit test only asks whether the final program passes. We analyze what this unit-test-only reward teaches a model during debugging and find that, while it raises pass rates, it also teaches the model to edit more and to hallucinate bugs in code that is already correct. We introduce PreciseCoder, a training algorithm that augments the unit-test reward with a signal for semantically verified, localized edits, which correlates with smaller edits and higher pass rates. On 4B, 9B, and 27B models, PreciseCoder reduces the average number of edit lines from 7-15to 3 at a pass rate on par with unit-test-only training. Trained only on single-line bugs, PreciseCoder can generalize to repository-level program repair, where it resolves more issues with smaller patches. Its precision also transfers from weaker to stronger models: given the reasoning trace of a 9B PreciseCoder model, frontier models such as GPT-5.6 Sol and Qwen3.8-Max gain 4.6-12.4 points in edit precision with no additional training. Linguistic analysis of the thinking traces shows that our reward encourages the model to reason about individual lines, name the buggy ones, and form consistent bug hypotheses.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Illustration of over-editing and bug hallucination phenomenon; (b) Training dynamics of GRPO vs. PreciseCoder on Qwen3.5-4B, reach the same pass rate (left), but GRPO edits more lines (middle) and leaves fewer bug-free programs untouched (right).

	Diagnosis		Reward signals				Generalizability
	Over-	Bug	Unit	Edit	Location	Partial	Weak-to-
	editing	hallucination	tests	extent	verified	fixes	strong
FixedBench ( Gloaguen et al., 2026 )	✗	✓	–	–	–	–	✗
PDB ( Zhu et al., 2026b )	✓	✗	–	–	–	–	✗
SWE-RL ( Wei et al., 2025 )	✗	✗	✗	✗	✗	✗	✗
EA-GRPO ( Ke et al., 2026 )	✓	✗	✓	✓	✗	✗	✗

Table 1: PreciseCoder vs. representative related work. “–” marks evaluation-only studies.

Figure 2: Pearson $r$ averaged over all training tasks (circle 4B, square 9B; $n{=}16$ samples per task). $\mathrm{unit}$ is constant when passing and is plotted at $y{=}0$ .

Methods	PDB Test				GT Debug
	Pass (%) $\uparrow$	Precision (%) $\uparrow$	Recall (%) $\uparrow$	Lines $\downarrow$	No-op (%) $\uparrow$	Damage (%) $\downarrow$
Model: Qwen3.5-4B
Base	36.5	35.8	48.5	15.3	30.6	19.3
SFT	41.2	39.7	54.7	10.2	34.9	19.4
DPO	51.5	40.5	61.2	8.7	30.6	17.7
GRPO	55.1	30.0	60.1	13.8	19.3	17.8

Table 2: Prior methods trade pass rate for smaller edits; PreciseCoder keeps pass rate on par with GRPO, while editing

4\times

fewer lines on PDB Test and, without seeing correct code, hallucinates fewer bugs on GT Debug. Bold: best; underline: second.

Figure 4: On PDB-Wild multi-line and repository-level repairs, PreciseCoder matches or outperforms unit-test-only GRPO and EA-GRPO across all metrics.

	With 9B trace			$\Delta$
Receiver	Pass	Prec.	Lines	Pass	Prec.
	(%)	(%)		(%)	(%)
Donor, own trace
PreciseCoder -9B	64.6	67.9	3.0	—	—
Own base, prefilled
Qwen3.5-9B	58.3	58.4	6.5	$+$ 17.1	$+$ 14.5

Table 4: Thinking trace carries correctness and precision for weak-to-strong transfer on PDB Test.

\Delta

: change vs. the receiver’s own reasoning.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

	PDB Test				GT Debug
Model	Pass (%) $\uparrow$	Prec. (%) $\uparrow$	Recall (%) $\uparrow$	Lines $\downarrow$	No-op (%) $\uparrow$	Damage (%) $\downarrow$
Qwen3.5-4B	36.5	35.8	48.5	15.3	30.6	19.3
+ GRPO	55.1	30.0	60.1	13.8	19.3	17.8
+ GRPO + clean rows	50.6	53.9	66.0	3.8	73.7 ∗	7.9 ∗
PreciseCoder -4B	55.3 $\pm$ 0.9	64.9 $\pm$ 1.3	72.9 $\pm$ 1.1	3.0 $\pm$ 0.1	48.0 $\pm$ 5.8	13.5 $\pm$ 1.5
Qwen3.5-9B	41.2	43.9	53.1	13.7	36.6	18.7

Appendix

Table 5: GRPO + clean rows against the base model, unit-test-only GRPO and PreciseCoder (the latter three as in Table 2 ). ∗ Trained on bug-free inputs. Bold: best within each size.

4B
	From scratch
Model	BCB (901)	LCB (806)	HumanEval+	CruxEval	DebugBench
Qwen3.5-4B (base)	43.2	42.4	86.6	85.9	67.0
+ GRPO	44.2	49.8	86.6	64.6	75.5
PreciseCoder-4B (ours)	46.5	53.3	92.1	90.5	72.9
9B

Appendix

Table 6: Out-of-distribution generalization: unit-test pass rate (%), one sample per task. PreciseCoder -9B is the mean of the three Table 2 seeds.

Figure 9: Reward-weight sweep on Qwen3.5-4B GRPO, PDB Test. Lines are seed means; whiskers span three seeds at

\lambda\in\{1/6,1/3,1/2\}

	Abstain-or-fix		$\Delta$
Model	No-op	Dmg.	No-op	Dmg.
Qwen3.5-4B	68.3	11.5	$+$ 37.7	$-$ 7.8
GRPO-4B	51.4	10.9	$+$ 32.2	$-$ 6.9
EA-GRPO-4B	57.0	14.9	$+$ 13.7	$-$ 3.9
PreciseCoder-4B	71.9	7.6	$+$ 23.8	$-$ 5.9
Qwen3.5-9B	71.0	9.6	$+$ 34.4	$-$ 9.2

Appendix

Table 7: GT Debug under the abstain-or-fix prompt;

\Delta

= change from the minimal prompt.

	PDB Test					$\Delta$
Receiver input	Pass	Prec.	Rec.	$F_{1}$	Lines	Pass	Prec.
Own reasoning (20k)	73.8	51.0	72.6	56.4	6.7	—	—
+ think block of the 9B donor
Qwen3.5-9B (base)	66.4	53.6	70.0	56.6	7.6	$-$ 7.4	$+$ 2.6
+ GRPO	74.2	58.4	76.6	62.7	5.5	$+$ 0.4	$+$ 7.4
+ EA-GRPO	76.1	62.1	77.7	65.9	5.9	$+$ 2.3	$+$ 11.1

Appendix

Table 8: Chain-of-thought transfer to Qwen3.6-27B from every 9B donor, PDB Test.

\Delta

= change from the receiver’s own reasoning; bold = positive

\Delta

	Qwen3.5-4B			Qwen3.5-9B			Frontier
	Base	+ GRPO	PreciseCoder	Base	+ GRPO	PreciseCoder	GLM-5.2	DeepSeek-V4-Pro
Pass (%)	36.5	55.1	56.2	41.2	63.9	64.6	70.8	67.6
Precision (%)	35.8	30.0	64.6	43.9	41.7	67.9	65.0	40.8
Think tokens (mean)	2535	3617	4096	2469	4096	4087	7375	12043
At the thinking budget (%)	50	79	100	50	100	100	7	16
Prose words	932	1179	1432	908	1494	1581	2919	4975

Appendix

Table 9: Thinking on PDB Test: length, lexical rates per 1,000 prose words, and grounding in the buggy line, for the base, unit-test-only GRPO and PreciseCoder models of Table 2 ( PreciseCoder seed 42, one sample per task) and two frontier models. Pass and precision are those of this single sample. DeepSeek-V4-Pro here is DeepSeek-V4-Pro Preview, against DeepSeek-V4-Pro (0813) in Table 2

	Qwen3.5-4B			Qwen3.5-9B
	Base	+ GRPO	PreciseCoder	Base	+ GRPO	PreciseCoder
No-op (%)	30.6	19.3	48.0	36.6	30.7	51.6
Damage (%)	19.3	17.8	13.5	18.7	11.4	11.8
Edited lines when edited	3.8	5.2	1.6	3.8	2.6	1.1
Think tokens (mean)	9773	14102	14813	9234	15868	16336
At the thinking budget (%)	54	77	68	51	91	99

Appendix

Table 10: Thinking on GT Debug (1,512 correct programs, one sample per task, all rows): outcome, length, and the marks of a bug hunt with nothing to find. Rates per 1,000 prose words unless noted.

Figure 10: Common word markers in the thinking traces.

Figure 11: What each training changed in the 9B’s thinking on the same PDB Test task: unit-test-only GRPO minus base (purple) and PreciseCoder minus base (blue), in standard deviations of the feature within the 9B triple; bars are 95% confidence intervals.

Figure 12: PDB Test edit precision against pass rate (bubble area = edited lines). Arrows join each base model to its PreciseCoder model; error bars are

\pm 1

sd over three runs.

Figure 13: Chain-of-thought transfer on PDB Test. (a) A PreciseCoder think block prefilled into its own base model. (b) The PreciseCoder -9B think block handed to larger receivers; dashed arrows mark the trace given in the user turn.

Figure 14: GT Debug under the minimal-edit (circles) and abstain-or-fix (triangles) prompts. Up and right is better.

Reinforcement learning (all sizes)
	Value
Advantage estimator	GDPO, $\lambda=\tfrac{1}{3}$ (§ 3.3 )
Group size $G$	6
Prompts per step / PPO mini-batch	32 / 16
PPO epochs per step	1
Optimizer	AdamW ( $\beta_{1}{=}0.9$ , $\beta_{2}{=}0.999$ , weight decay 0.01)

Appendix

Table 11: Training and evaluation hyperparameters. The upper block is shared by every reinforcement-learning run, including the GRPO and EA-GRPO controls; the middle block lists what changes with model size. In the CoT-transfer block, “trace +

n

” means the donor’s trace is prefilled into the receiver’s thinking, which may continue for

n

more tokens before it is closed. Settings not listed are verl defaults.

Automated Program Repair

Automated program repair develops methods that locate software faults and produce validated fixes with limited human effort.

See all 153 papers

Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
Apr 19, 2026Miaosen Chai, Wang Bill Zhu, Shangshang Wang +5LLM Evaluation Automated Program Repair
University of Southern California · Microsoft · University of Wisconsin–Madison +1
QiMeng-PRepair: Precise Code Repair via Edit-Aware Reward Optimization
Apr 7, 2026Changxin Ke, Rui Zhang, Jiaming Guo +10Automated Program Repair LLM Code Generation
State Key Lab of Processors, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · Institute of Microelectronics, CAS
When Models Edit Too Much: On the Fidelity of Minimal Code Edits
Sep 3, 2026Tongyao Zhu, Wei Hern Lim, Min-Yen KanAutomated Program Repair Language Model Editing
National University of Singapore

Apr 19, 2026J/K move · Enter open · S save

Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?

Miaosen Chai, Wang Bill Zhu, Shangshang Wang, Yejia Liu+4

University of Southern California · Microsoft · University of Wisconsin–Madison+1

LLM Evaluation Automated Program Repair

Open paper AskPDF arXiv

Unlike code completion, debugging requires localizing faults and applying targeted edits. We observe that frontier LLMs often regenerate correct but over-edited solutions during debugging. To evaluate how far LLMs are from precise debugging, we introduce the Precise Debugging Benchmark (PDB) framework, which automatically converts any coding dataset into a debugging benchmark with precision-aware evaluation. PDB generates buggy programs by synthesizing verified atomic bugs and composing them into multi-bug programs. We define two novel metrics, edit-level precision and bug-level recall, which measure how many necessary edits are made and how many bugs are resolved. We release two evaluation benchmarks: PDB-Single-Hard on single-line bugs, and PDB-Multi on multi-line bugs. Experiments show that frontier models, such as GPT-5.1-Codex and DeepSeek-V3.2-Thinking, achieve unit-test pass rates above 76% but exhibit precision below 45%, even when explicitly instructed to perform minimal debugging. Finally, we show that iterative and agentic debugging strategies do not substantially improve precision or recall, highlighting the need to rethink post-training pipelines for coding models.

Abstract

Automated Program Repair

Explore similar work

Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?

QiMeng-PRepair: Precise Code Repair via Edit-Aware Reward Optimization

When Models Edit Too Much: On the Fidelity of Minimal Code Edits