Coding agents increasingly share a workstation while drawing on separate providers and subscription allowances. A second agent can inspect a completed answer, but the call spends another pool and may provide no substantive finding. We describe an advisory cross-provider review contract: distinct resource pools, bounded execution, restricted reviewer capabilities, complete input delivery, usable semantic output, explicit failure states and durable per-attempt evidence. In a controlled, agent-authored pilot of 20 paired development turns, eight had a material reviewer finding (95% exact interval 19.1-63.9%). A boundary-condition scan across both reviewer backends reproduced a previously discovered false success on partial input: four truncation levels passed historically and failed after repair. The scan also found and repaired cancellation during process reaping. In real CLI probes, Claude had no writing tools; Codex attempted writes in five of five read-only trials, each write tool failed, and no disposable repository changed. These tests cover specified paths and versions, not field reliability. A preregistered shadow study of metadata-only review allocation accrued 25 formal observations before an exact-runtime regression found a third defect: a reviewer exiting nonzero with a well-formed verdict was counted as complete. Exit status was not recorded per attempt, so exposure cannot be resolved retrospectively. The 25 formal and two pending records remain an audit cohort; the measurement-valid cohort restarted at zero and collection has begun. No gate result is reported.
Figures & tables
Figure 1: Implemented agent interface and cross-provider review versus the prospective metadata-only gate. In shadow evaluation, the reviewer still runs on every eligible turn. The gate corresponds to OTA’s Allocator role and the reviewer to its Reviewer role.
Property
Failure exercised
Protected measurement
Distinct resource pool
Same-pool configuration refused before spawn
Independent reviewer-call cost
Bounded lifetime
Child does not drain input; cancellation during review
Attempt denominator and timeout/cancellation attrition
Restricted capabilities
Write-enabled launch arguments rejected; real CLI asked to write in disposable repos
Observed working-tree integrity under tested backends
Complete input delivery
Plausible clean verdict after four truncation levels, from 1% delivered to one character missing
Valid review denominator
Semantic result and typed failure
Empty, malformed, exit-zero empty output, and nonzero exit with well-formed verdict
No failure miscounted as "no material finding"
Durable per-attempt evidence
Exit status not recorded when defect found after collection
Retrospective scoping of affected observations
Table 1: Runtime contract, injected failure and measurement protected.
Direction
Pairs
Mat.
Nonmat.
P med.
R med.
Codex → Claude
10
2
8
77.7
117.5
Claude → Codex
10
6
4
282.9
24.7
Table 2: Controlled pilot by provider direction; medians in seconds.
Condition
Observed classification or filesystem outcome
Clean-looking verdict after 1%, 50%, 99%, or all but one character delivered
Both backends: historical completed 4/4; current input_delivery_failed 4/4
Cancellation during input write, output read, or process reap
Both: pre-reap-repair accepted the reap case; current cancelled in all three phases
Nonzero exit with complete, well-formed verdict
Both: every collection-era revision completed; post-repair candidate process_crash ; all six exit/output scenarios classified as specified per backend (12/12)
Output at 0.8, 1.0, or 1.2 s with 1.0 s budget
Both: completed, timeout , timeout , respectively, in this timed scan
Malformed JSON; exit-zero empty result
Claude: malformed_output , empty_response ; Codex: empty_response for either; no clean verdict
Same-pool or write-enabled launch configuration
same-pool refusal or unsafe_argv , before launch
Table 3: Boundary scan and real CLI probe. Each synthetic boundary is one deterministic case; CLI rows are five distinct disposable repositories per backend.
AI coding agents increasingly accept assigned software tasks, modify repositories under bounded authority, and return work packages for review. Prior work proposed the software delegation contract, covering the task, authority, returned work package, and acceptance context, as the unit of analysis for delegated coding work, but did not measure its effects. This paper reports a controlled pilot study of explicit delegation contracts for coding agents. We built a dependency-free TypeScript API task environment with seeded defects and documentation gaps, authored ten tasks across five families, and ran 64 agent executions across two model tiers under three conditions: a realistic issue-style prompt, an explicit delegation contract, and a contract with a required evidence bundle. Each run was scored with hidden acceptance tests, mutation checks, and scope analysis, then reviewed by three independent condition-blinded model-based reviewers using a fixed rubric, for 192 reviews. Explicit contracts did not improve objective task outcomes: all 64 runs passed hidden acceptance checks, with zero scope violations. They did improve reviewability. Evidence sufficiency improved in 22 of 30 paired comparisons and worsened in none (+0.83 on a 5-point scale, p < 0.0001, Cliff's delta = 0.66); reviewer ambiguity decreased (p = 0.035); changed-file lists, known-limitations sections, residual-risk sections, and reviewer checklists appeared mostly or only when demanded by the contract. Contracts cost +13% agent tokens and +38% wall-clock time, with larger effects for the weaker model tier. On these small tasks, delegation contracts bought reviewability rather than correctness.
Coding agents can return plausible patches that omit required behavior. These failures are hard to review because long traces and confident summaries often hide what was missed. We ask when a nominally weaker reviewer can reliably decide whether a patch solves its issue. We study 411 execution-labeled traces from three agents and 101 controlled cases. On 154 GPT-5.4 traces, structured but unchecked evidence raises both defect catch and over-rejection. We then provide official execution evidence as an upper-bound diagnostic. After choosing and freezing one of two formats per reviewer, five of six reviewers improve both rates on 122 held-out traces; two classify every trace correctly. Reviewer size is not a consistent predictor of quality. Because official tests are unavailable in deployment, we also evaluate a frozen cascade with patch-caused static errors and generated tests that first fail on the unpatched repository. On 121 scored held-out GPT-5.4 traces and 59 Gemini traces, its coverage is 0.89 and 0.86, risk is 0.33 and 0.26, catch is 0.76 and 0.80, and over-rejection is 0.66 and 0.67. Most false rejections occur when unresolved cases reach the reviewer. Official execution evidence shows the potential of weak review when decisive checks are available. Producing equally reliable checks without official tests remains the main bottleneck.
Junyu Guo, Shangding Gu, Ming Jin +1
University of California, Berkeley · Virginia Tech
Developers increasingly use two coding agents together: one writes a draft, and the other reviews it. However, it is not clear whether the pairing is worth its cost and time, or whether the order of the pairing matters. We run a controlled experiment on 116 recent hard and medium lcb tasks with Claude and Codex across six conditions to approximate a software practitioner's workflow: both solo baselines, both cross-model orderings, and both same-model orderings. The reviewer sees the problem and the writer's draft but cannot execute tests, which approximates a code review step. Claude review raises Codex drafts from 71.6% to 89.7% (pBH=.001); Codex self review raises them to 84.5% (pBH=.022). The reverse direction does not pay off: Codex reviewing Claude drafts drops the pass rate from 91.4% to 82.8% (pBH=.046), and Claude self review leaves the 91.4% baseline unchanged. Our evaluation indicates that the useful pairing is asymmetric: use Claude to review Codex, not the other way around.
Zuodong Xiang, Yike Zhang, YueMing Zhang +1
University of California, Davis USA · Johns Hopkins University USA · California State University, Long Beach USA