ADAPT: Agentic Diff-Aware Patch Testing for Pull Requests
Organizations: Bilkent University, Ankara, Turkey
Abstract
Modern software development relies on pull requests (PRs) that introduce small, targeted code changes. Once merged, these changes should be protected by focused tests that exercise the behavior the PR introduces, yet developers often submit PRs without such tests, leaving the new behavior unguarded against future regressions. Most test generation techniques target entire software units and optimize objectives such as code coverage rather than the behavior a change introduces, and prior diff-aware approaches have had limited success in capturing it. We propose ADAPT (Agentic Diff-Aware Patch Testing), a regression test generation approach that uses an autonomous coding agent to produce change-focused tests for PRs. Given a PR, the agent analyzes the diff between the base and head versions to identify the modified functions and their context, then plans and generates candidate unit tests. A dedicated fail-to-pass (F2P) verification tool runs each candidate on both versions and, after F2P success, reports which executable changed lines remain uncovered; guided by this feedback, the agent iteratively refines its tests. We evaluate ADAPT on the same 426 real-world PRs used by the prior state-of-the-art baseline, with GPT-4o and GPT-5-mini. ADAPT+GPT-4o reaches an F2P rate of 62.0% (264/426), compared with BLAST's reported 35.4% (151/426), while ADAPT+GPT-5-mini reaches 89.9% (383/426). On 137 PRs shared with Change And Cover (ChaCo), ADAPT achieves full patch coverage on 83.9% of PRs, compared with 79.6% for ChaCo, at considerably lower time and cost. A manual evaluation confirms that the F2P-valid tests target each PR's intended change, with mean scores of up to 4.54/5 for importance, 4.35/5 for assertion strength, and 4.49/5 for readability, often covering behavior beyond the developer-written tests.
Figures & tables
| ID | Question | |
| Per-test quality (1–5 scale) | ||
| Q1 | Importance: does the test check behavior that actually matters for the change—the intended behavior—rather than a trivial or irrelevant case? | |
| Q2 | Assertion strength: how precise are the checks? A weak smoke test that only confirms the code “runs without error” scores low, whereas a precise oracle that pins down the exact expected result scores high. | |
| Q3 | Readability: is the test clean, idiomatic, and easy to follow—would it pass code review? | |
| Comparison with the developer-written test (yes/no) | ||
| Q4 | Gen. Human: does the generated test cover any behavior the developer-written test does not? | |
| Approach | F2P | No F2P | Error | F2P Rate |
|---|---|---|---|---|
| BLAST (GPT-4o) | 151 | 275 | – | 35.4% |
| ADAPT (GPT-4o) | 264 | 133 | 29 | 62.0% |
| ADAPT (GPT-5 mini) | 383 | 4 | 39 | 89.9% |
| Subset | Total$ | $/PR | Tok/PR | Steps | Time |
|---|---|---|---|---|---|
| GPT-4o | |||||
| All | 478.95 | 1.12 | 698.8 | 22.5 | 169.8 |
| Passed | 127.40 | 0.48 | 266.6 | 13.1 | 106.0 |
| GPT-5 mini | |||||
| All | 16.62 | 0.04 | 413.6 | 16.6 | 196.9 |
| Passed | 16.15 | 0.04 | 450.8 | 16.5 | 192.9 |
| Trajectory Type | GPT-4o | GPT-5 mini |
| Single verification-tool call | 128 | 252 |
| Repeated verification-tool calls | 292 | 135 |
| Total runs with verification | 420 | 387 |
| Avg. verification calls per run | 5.4 | 1.8 |
| GPT-4o | GPT-5 mini | |||
| Category | Count | % | Count | % |
| F2P test cases | 384 | 9.2 | 608 | 23.8 |
| Non-F2P test cases | 3,778 | 90.8 | 1,943 | 76.2 |
| Total test cases | 4,162 | 100.0 | 2,551 | 100.0 |
| Question | Mean | AC | MAD |
|---|---|---|---|
| GPT-4o | |||
| Importance (Q1) | 4.01 | 0.732 | 0.895 |
| Assertion strength (Q2) | 3.79 | 0.653 | 0.976 |
| Readability (Q3) | 4.16 | 0.802 | 0.845 |
| Gen. Human (Q4) | 0.66 | 0.371 | 0.348 |
| Human Gen. (Q5) | 0.81 | 0.460 | 0.319 |
| Approach | Overall | pandas | SciPy | Qiskit |
|---|---|---|---|---|
| ( ) | ( ) | ( ) | ( ) | |
| ADAPT | 115 (83.9%) | 40 (90.9%) | 40 (81.6%) | 35 (79.5%) |
| ChaCo | 109 (79.6%) | 35 (79.5%) | 38 (77.6%) | 36 (81.8%) |
| Approach | PRs | Input | Cached | Output | Total tok. | Tok./PR | Total$ | $/PR | Time/PR |
|---|---|---|---|---|---|---|---|---|---|
| ADAPT | 137 | 265.7M | 255.4M | 5.4M | 271.1M | 1.98M | 19.82 | 0.145 | 17.4/13.9 |
| ChaCo | 137 | 58.5M | – | 23.0M | 81.6M | 0.60M | 60.28 | 0.44 | 56.7/46.2 |