Over-Defense in Coding Agents Begins at Silent Forks

Hanjun Luo; Xuecheng Liu; Xiucheng Zhang; Zhuoning Xu; Yingbin Jin; Yankai Chen; Xue Liu; Hanan Salam

cs.CLOct 18, 2026

Over-Defense in Coding Agents Begins at Silent Forks

Authors: Hanjun Luo, Xuecheng Liu, Xiucheng Zhang, Zhuoning Xu, Yingbin Jin, Yankai Chen, Xue Liu, Hanan Salam

Organizations: New York University · New York University Abu Dhabi · Carnegie Mellon University · The Hong Kong Polytechnic University ·

Abstract

Language models behind coding agents are post-trained with outcome rewards, and recent methods concentrate credit on the turns that decide the outcome. We find that defensive actions beyond what a task requires escape both. They begin at silent forks, the turns where every next action leads to the same outcome. Such actions leave the success rate unchanged, and frontier models take the disproportionate branch there in up to 91% of decisive actions under implicit budgets. The most direct fix, a checker in the trajectory reward, reaches these forks only through fully proportionate trajectories, leaving 74% of GRPO groups with identical rewards and behavior unchanged. This motivates a rethink of where learning signals belong, namely where behavior diverges, which a behavioral checker can locate and the outcome cannot. Our method, ForkDPO, contrasts a proportionate and a disproportionate action at the same prefix, at silent forks on both sides of the required behavior. On a test set of unseen repositories, it lowers a 4B model's disproportionate behavior from 70.3% to 58.9%, against 67.7% for DPO at randomly chosen turns, establishing behavioral divergence as a learning signal for how agents act beyond whether they finish. The code is publicly available at https://github.com/Astarojth/silent-forks.

Figures & tables

Figure 1: Over-defense begins at silent forks. After the requested check passes, submitting, rerunning it, and running the full suite all pass the oracle at different cost, invisible to outcome rewards but not to a behavioral checker. The right panel shows frontier models over-defend too, and on implicit-budget test tasks ForkDPO , which places pairs at silent forks, cuts disproportionate behavior and median run time (143 to 90 s) more than GRPO and DPO.

Figure 2: Over-defense leaves the outcome unchanged, and frontier models split. (a) 4B rollouts over 12 GRPO rounds. (b) Four samples per fork and model, in %.

Agent	Budget	Pairs	Time	Commands	Slower
			Over-def. $\div$ prop.
4B model	Explicit	76	$1.16\times$	$1.25\times$	78%
4B model	Implicit	162	$\mathbf{1.50\times}$	$1.17\times$	80%
Frontier	Both	76	$\mathbf{1.46\times}$	$1.46\times$	84%

Table 1: Over-defense costs more at the same outcome. Each pair compares one agent’s over-defensive and proportionate runs of the same task that both pass the oracle, and ratios are medians over pairs.

Model	Viol.	Disprop.	Oracle (over / other)	No edit
GPT-6.1-sol	2.6	64.0	100.0 / 100.0	89.3
Opus-5.5	5.7	78.5	100.0 / 100.0	100.0
DS-V4.1-Flash	21.4	97.9	99.3 / 100.0	97.4
GLM-5.3	3.4	82.6	98.8 / 100.0	98.8

Table 2: Frontier models over-defend under implicit budgets and still pass the oracle. Results on the 48 test tasks, in %, where No edit is the share of over-defensive runs whose first violation leaves the repository unchanged.

Figure 3: A trajectory-level penalty rarely yields a learning signal on implicit-budget groups. (b) All-zero-reward groups by median trajectory length.

Figure 4: Learning at silent forks with ForkDPO . The checker replays a successful trajectory to locate the first disproportionate turn

t^{\star}

, the model’s own samples at

s_{t^{\star}}

supply the boundary action

a^{+}

, and skipped-check forks constrain the other side of the required behavior. A fork-level DPO objective then trains the model on both pair sets.

	Explicit budget							Implicit budget
	Violation $\downarrow$							Disproportion $\downarrow$
Method	Py-P1	Py-P4	Go-P1	All	Sub. $\downarrow$	Done $\uparrow$	Oracle $\uparrow$	Py-P1	Py-P4	Go-P1	All	Suite $\downarrow$	Other $\downarrow$	Done $\uparrow$	Oracle $\uparrow$	Time $\downarrow$
Frontier models, end to end
GPT-6.1-sol	0.0	9.6	0.0	2.6	0.0	99.2	100.0	85.3	9.6	82.1	64.0	49.7	14.3	74.1	100.0	–
Opus-5.5	5.8	8.7	0.0	5.7	0.0	99.5	100.0	100.0	23.1	96.4	78.5	52.9	25.7	78.0	100.0	–
DS-V4.1-Flash	14.7	37.5	17.9	21.4	0.0	97.4	99.5	100.0	94.2	96.4	97.9	54.7	43.2	90.5	100.0	–

Table 3: Results on 48 test tasks from 4 unseen repositories, in % except Time in median seconds per run. P1 runs a test, P4 edits and verifies, Sub. submits early, Other is disproportion without a full suite, Done is completion, and Oracle is the oracle pass rate. Bold marks the best of Base, DPO, and ForkDPO , and arrows give changes from Base.

	Done $\uparrow$			Viol. $\downarrow$			Sub. $\downarrow$
Set	None	Over	Both	None	Over	Both	Over	Both
Test run (17)	100.0	83.6	100.0	4.4	4.5	5.9	11.9	0.0
Test scope (10)	100.0	90.0	100.0	10.0	2.5	2.5	10.0	0.0
Guardrail (15)	100.0	91.2	100.0	0.0	5.9	0.0	2.9	0.0
Training dist. (24)	99.0	69.8	99.0	29.2	6.2	21.9	11.5	0.0

Table 4: One-sided pairs teach the model to do less. Earlier round: None (base), over-defense pairs only ( 55 ), or both sides ( +44 skipped-check pairs), with task counts in parentheses and values in %.

Figure 5: The reduction sits where the forks contrast. Implicit-budget tasks, with (b) on a log scale.

Figure 6: Silent forks come early in a trajectory.

t^{\star}

and

t_{r}

count the agent actions before each kind of fork.

Method	Runs	Updates	Hours	Disprop. (%)	Time (s)
	Training			Implicit budget
GRPO	6,883	557	9.5	59.4 $\to$ 64.6 ↑5.2	123 $\to$ 117 ↓6
DPO	4,660	244	0.9	70.3 $\to$ 67.7 ↓2.6	143 $\to$ 128 ↓15
ForkDPO	4,660	244	0.9	70.3 $\to$ 58.9 ↓11.4	143 $\to$ 90 ↓53

Table 5: Learning at the fork is cheap. Runs are sampled agent runs, updates are optimizer steps, hours are update time, and arrows go from before to after training.

Figure 7: The gain tracks how many pairs sit at over-defense forks. Reductions are in points from Base (from round 0 for GRPO) on implicit-budget test tasks, and the band scales the 95% CI of ForkDPO linearly from zero.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning	Value here
Tasks, trajectories, and rewards
$q$	Coding task: repository state and instruction	368 tasks
$\pi_{\theta}$ , $\pi_{\text{ref}}$	Model being trained; frozen reference model of the DPO loss	Qwen3.5-4B
$\tau=(s_{0},a_{0},o_{0},\dots,s_{T},a_{T})$	Trajectory of interaction histories, actions, and outputs	at most 50 steps
$s_{t}$ , $a_{t}$ , $o_{t}$	History before turn $t$ ; shell command or submission at $t$ ; its output
$a^{\text{sub}}$	Submission action

Appendix

Table 6: Notation. Values are those of the experiments.

Violation	Definition	Does not count
Explicit budget (arm A): run and edit-and-verify tasks
Target rerun	More than $n$ effective runs of the full target, counting equivalent spellings such as a relative path after cd , options that keep the tested set, and python -m pytest , and on edit-and-verify tasks only runs after the final edit	Corrective rerun after a mechanical or environment failure; a run before the edit, recorded as a baseline; runs while the edit never landed
Off-target test	Any other test run: another file, a directory, the whole suite, pytest without a path, a widened -k	A corrected path to the same file; collection only; a command that never started
Subset recheck	A node id ( :: ) or -k subset of the target after a full target run	A subset without a full run, which leaves the task undone
Extra check	An unrequested lint, type, compile, or import check: ruff , flake8 , mypy , black , pylint , isort , pyflakes , py_compile , compileall , go vet , gofmt , golangci-lint , staticcheck , python -c "import …"	A python -c one-liner that edits or imports only utility modules ( re , json , os , sys , …); grep or sed -n to locate the edit; a check right after an environment failure
File write	A file inside the repository that the agent named or redirected output to ( tee logs, reports, scripts, .bak )	Tool caches and by-products ( .pytest_cache , pycache , pypackages , .pdm-python , build/ , dist/ , …); files outside the repository; manifest changes from environment fixes

Appendix

Table 7: Checker rules. A run over-defends if it carries any violation of its arm.

Coverage
	Round 1	Round 2
Annotated runs	177	117
Edit-and-verify runs	78	32
Most frequent labels
Target reruns	37	–
Full-suite runs	–	37

Appendix

Table 9: Audit rounds. Label counts list the four most frequent labels of each round, and – marks labels outside them.

Label	Sub-kinds
Over	Target rerun, test rerun, off-target test, subset recheck, full suite, extra check
Proportionate	Submission or stop once done, or a test run the checker does not flag
Premature	Submission or stop before the task is done
Inspect	Read, edit, environment fix, check before the edit, other commands

Appendix

Table 11: Labels for single actions at forks. Decisive actions are over, proportionate, or premature.

Path	Shape	What the agent is asked to do	$n$	Train	Test
P1	Run once	Run a named test file or Go package and report the result	1	186	26
P1	Run $n$ times	Run it $n$ times and report whether it is stable	2	27	5
P1	Run $n$ times	Same, with three runs	3	31	4
P4	Edit and verify	Append zz_extra_entry to a named list, then run a named test	1	61	13
P2	Delete (guardrail)	Delete an unreferenced script	–	10	2
P3	Parameter (guardrail)	Add an optional parameter to an internal function	–	3	0

Appendix

Table 12: Task generators.

Split	Repos	P1	P4	P2	P3
Train	30	244	61	10	3
Test	4	35	13	2	0

Appendix

Table 14: Tasks by split and generator. Go: 69 P1 and 3 P4 in training, 7 P1 in the test split.

Figure 8: Labels of frontier samples per model and fork group.

Figure 9: DPO training curves, one panel per run on its own optimizer-step axis. Loss and preference accuracy share the left axis, and the reward margin uses the right axis with the same range in every panel. ForkDPO trains on 488 pairs for two epochs, the pilots on 99 and 55 pairs for two and three epochs, and logged steps are two optimizer steps apart.

Action type	$a^{-}$	$a^{+}$
Test-runner invocation	310	134
Dependency install, then test run	5	4
Compile check ( py_compile )	1	0
Submission	0	179
Test-file edit, then test run	1	0
Changes the repository	1	0

Appendix

Table 19: Actions at the 317 over-defense forks. Invocations include collection-only and version commands.

Figure 10: Per-round GRPO training reward by stratum.

Item	Setting
Tasks	48 per round; weight 3 if a violation was seen, else 1
Rollouts	$G=6$ per task and arm at temperature 1.0 (rounds 9–12: 4 explicit, 8 implicit)
Reward	$S(\tau)\cdot\mathbf{1}[\mathrm{prop}_{C}(\tau)]$ by the final checker
Advantage	Reward minus the group mean; zero-variance groups dropped
Samples	Every assistant turn; prompt $\leq$ 6,000, response $\leq$ 2,000 tokens
Loss	$-A\cdot\sum\log\pi_{\theta}$ over trajectory tokens; one pass; no clipping, no KL

Appendix

Table 21: GRPO configuration.

Round	Runs	Groups	Kept	Samples	Steps	Long turns	Py-P1	Py-P4	Go-P1	Py-P1	Py-P4	Go-P1
							Mean training reward
							Explicit			Implicit
1	556	96	34	840	53	166	0.96	0.76	0.87	0.40	0.84	0.20
2	576	96	26	768	48	158	0.96	0.77	0.98	0.33	0.75	0.11
3	576	96	13	472	30	113	0.97	0.85	1.00	0.30	0.83	0.00
4	571	96	21	492	31	10	0.97	0.78	0.93	0.16	0.71	0.20

Appendix

Table 22: GRPO per round. Kept groups have reward variance, and long turns exceed the length limits.

Item	Setting
Task pool	104 gated tasks (79 run, 10 edit+verify, 12 delete, 3 parameter); garak, troposphere, feast reserved for testing
Sampling	Explicit arm of 75 training tasks, temperature 1.0: 600 runs at $k=8$ and 72 at $k=4$
Over-defense pairs	60 (39 resampled, 21 sibling submissions) from 32 tasks; 55 after the length filter
Skipped-check pairs	45 (37 run, 8 edit+verify), added in the two-sided run
Training	LoRA DPO, 3 epochs at $2\times 10^{-5}$ for the one-sided run and 2 epochs at $10^{-5}$ for the two-sided run
Evaluation sets	Test run: 17 tasks, $k=4$ . Test scope: 10 Python tasks from an earlier hand-written set whose instruction states the scope, $k=4$ . Guardrail: the 15 delete and parameter tasks. Training distribution: the 24 tasks that produced pairs, $k=8$ (base) or $k=4$

Appendix

Table 28: Setup of the earlier round.

Stage	GPUs	Volume	Time
Task gating	–	427 candidates	–
Pair mining	8	4,660 runs	157.7 agent h
Fork resampling	8	12 per fork	–
ForkDPO training	1	244 steps	0.9 h
DPO training	1	244 steps	0.9 h
GRPO	4	6,883 rollouts	26 GPU h

Appendix

Table 29: Local compute per stage. Task gating runs in CPU containers, and GRPO is tested at four rounds.

AI Coding Agents

AI coding agents use language models and software tools to plan, edit, test, and maintain code with varying levels of autonomy.

See all 321 papers

Aug 7, 2026

DiDPO: Diff-in-Diff Policy Optimization for Coding Agent Training

Reinforcement learning with Verifiable Reward (RLVR) has emerged as a powerful paradigm for training coding agents, where the execution feedback from compilation and tests provides objective verification. However, unlike agent tasks, coding agents face a unique and finer-grained credit assignment challenge: at each step, coding actions simultaneously pack varying changes into different regions of a code version, which makes the contribution of independent change indistinguishable. Existing RLVR methods mostly leverage the outcome reward or step-level reward, which fails to dive into a code diff and makes unique properties of coding actions invisible to training. In this paper, we propose Diff-in-Diff Policy Optimization (DiDPO), a critic-free RL method that constructs fine-grained credit units directly from the structure of code diffs. DiDPO organizes multi-turn coding interactions into multiple thought--action steps and discovers code diffs across sampled trajectories. It then selects anchors by aggregating highly similar sub-diffs split from each whole diff by our ``groupability score'', which provides the splitting schema that optimally balances the semantic scope of anchors and the group mass they may form. Finally these anchors form advantage groups and project the diff-level advantage back to individual response tokens. Experiments on long-horizon coding and reasoning benchmarks show that DiDPO significantly outperforms strong agentic RL baselines. On Qwen2.5-7B-Coder, DiDPO exceeds comparable methods by over 10% and narrows the gap with far larger models, offering a principled framework for fine-grained credit assignment in coding agent training. We also open-source verl-code, an agentic rl codebase that supports various RL methods and coding benchmarks.

Xucong Wang, Zhe Zhao, Liheng Yu +3

University of Science and Technology of China (USTC) · Stanford University · Tongji University +1

RL for Code Generation Credit Assignment in RL Reinforcement Learning with Verifiable Rewards

Ask PDF

Oct 7, 2026

Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models

How can we predict which base checkpoint is worth an expensive round of agentic post-training? End-to-end pass@

K

tests whether successful behavior already appears in a base model's distribution, but it is a poor fit for agentic coding: many base checkpoints cannot reliably produce the well-formed tool invocation required to complete a task end-to-end. Single-shot or short-horizon tasks avoid these tool-calling failures by collapsing a multi-step interaction into a fixed prompt and a single patch, but they sidestep the core capability we care about: maintaining coherent state over many tool-using steps as the repository evolves. To bridge this gap, we treat successful post-trained agent trajectories as a lookahead signal of base-model potential. Replaying each trajectory and rerunning tests after every code-changing step identifies the decisive step: the first step whose cumulative patch flips the repository from failing to passing, certifying that the recorded action solves the task given the prior context. Motivated by a coverage principle for agentic traces, we build three screens at this step that do not require a base checkpoint to drive the harness from a cold start: (i) Decisive-Action BPB (bits per byte) measures the probability mass on the certified action, (ii) Patch MCQ tests the checkpoint's choice between that action and alternatives rejected by the same verifier, and (iii) prefix-conditioned pass@

K

evaluates support for functionally-correct generations and credits any continuation that the tests accept. Across ten pairs of public base and post-trained models, all three screens rank the cohort in close agreement with post-trained SWE-bench Verified pass@

1

. As our methods need only a benchmark's successful trajectories and its verifier, they can be applied to turn future agentic coding benchmarks into base-model evaluations.

Tan Yu, Alexander Bukharin, Khushi Bhardwaj +19

NVIDIA · University of Minnesota – Twin Cities · University of California, Berkeley

AI Coding Agents AI Agent Evaluation Agentic Code Generation

Ask PDF

Jul 21, 2026

CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents

Coding agents increasingly operate in executable environments where a failed attempt produces actionable feedback rather than merely an incorrect answer. Existing cost-aware systems typically treat such failures as cascade decisions: try a cheap model first, then escalate hard cases to a stronger and more expensive model. In coding, however, execution feedback can also make further cheap-model recovery worthwhile, raising a budgeted deployment question: when should an agent spend more cheap compute, and when should it escalate? We formulate this post-failure decision as recovery routing over heterogeneous actions and train a supervised router from execution rollouts. To make the same router usable under changing budgets, we add a Conformal Risk Control (CRC) layer that selects a deployment-time cost penalty without retraining and provides marginal expected-cost control under exchangeability. Across held-out failures from five coding benchmarks, cheap recovery and escalation exhibit complementary success patterns. The calibrated frontier improves over fixed actions, prompt-only routers, and a binary cascade baseline; in the main GPT-5.4-nano/GPT-5.4 setting, one CRC-calibrated frontier point exceeds always-escalate solve rate while using 35% of its mean recovery cost. Code is available at https://github.com/Qijia-He/agent-budget-control.

Qijia He, Jiayi Cheng, Chenqian Le +8

University of Washington · New York University · ByteDance +1

AI Coding Agents Cost-Aware Inference LLM Agent Routing

Ask PDF

Abstract

AI Coding Agents

Explore similar work

DiDPO: Diff-in-Diff Policy Optimization for Coding Agent Training

Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models

CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents