Current software engineering agents are evaluated predominantly inside isolated sandboxes against synchronous local tests. However, deploying features in collaborative repositories requires modifying continuous integration (CI/CD) pipelines, containers, and provisioning manifests alongside application logic. Because agents lack deployment awareness, attempting cross-domain modifications introduces unverified dependencies into asynchronous integration pipelines. To isolate the causal integration overhead of dual-scope pull requests (PRs)-modifying both application code and deployment infrastructure-we evaluated 26,999 PRs across two large datasets (AIDev v5 and MOSAIC-3M) encompassing six agent architectures and an aligned human baseline. Using a deterministic four-tier file taxonomy and an exact 4D strata-matching framework, we isolated integration metrics under conditional exchangeability while controlling for application volume. We find agents operate primarily as application coders or chore workers, with maintenance tasks capturing 17.70% to 25.41% of agent PRs. Dual-scope modifications represent only 4.48% to 4.55% of agentic PRs, yet trigger bimodal integration outcomes. In the first mode, explicitly reviewed contributions experience protracted delays, adding a median penalty of +13.13 hours and multiplying time-to-merge by 3.24x, driven mechanically by CI/CD and provisioning manifests. In the second mode, maintainer gatekeeping exhibits a structural bypass: even for external contributors lacking repository write access, agent identity reduces the odds of human review by 63.7% compared to human peers. Consequently, 43.1% to 56.2% of dual-scope PRs bypass inspection to merge silently in a median of under 31 minutes. These uninspected modifications compromise build verification, demonstrating the need for deployment-aware benchmarks and strict path-based access controls.
Figures & tables
Dimension / Pipeline Stage
AIDev v5 ( pop )
MOSAIC-3M
A. Dataset Scope & Repository-Level Curation
Native Collection Timeline
Jan 2025 – Mar 2026
Jun 2025 – Aug 2025
Profiled Agent Architectures
Codex, Claude, Copilot, Cursor, Devin, Jules
Codex, Claude, Copilot, Devin, Jules
Human Baseline
None
Included
Repository Popularity Threshold
>100⋆ (Inherent to pop )
>100⋆ (Applied filter)
Curated Repositories ( >100⋆ )
25,402
4,553 (of 50,515)
Table 1. Cross-Dataset Structural Parity and Cohort Funnel Matrix Table mapping dataset provenance, structural boundaries, experimental controls, and the progressive cohort filtering funnel from raw PRs down to the cleaned evaluated populations across the AIDev-pop subset and MOSAIC-3M.
Figure 1. Operational Landscape of Agentic Pull Requests: (a) Macro PR scope distribution across cleaned cohorts; (b) Infrastructure subdomains modified within dual-scope PRs (non-mutually exclusive). Two-panel visualization: Panel (a) is a grouped horizontal bar chart comparing AIDev v5 and MOSAIC-3M across macro PR scopes, showing application-only PRs at 63 to 70 percent, dual-scope PRs at roughly 4.5 percent, and maintenance chores at 21 to 28 percent. Panel (b) shows dual-scope subdomains, dominated by CI/CD manifests at 72 to 77 percent, followed by Containers and Automation (15 to 20 percent each), and Provisioning (7 to 10 percent).
AIDev v5
MOSAIC-3M
Author / Architecture
Total PRs
Dual-Scope
Rate (%)
Total PRs
Dual-Scope
Rate (%)
Claude Code
291
52
17.87% ∗
903
55
6.09%
Copilot
2,675
413
15.44%
505
23
4.55%
Devin
751
91
12.12%
858
33
3.85%
Google Jules
27
3
11.11%
26
1
3.85%
Cursor
1,176
103
8.76%
Not Tracked
Table 2. Cross-Dataset Agent-Level Dual-Scope Prevalence (PRs Created Jun–Jul 2025) Table comparing dual-scope PR prevalence across agent architectures side-by-side between AIDev v5 and MOSAIC-3M cohorts, reporting total PRs, dual-scope counts, and percentage prevalence.
Figure 2. Empirical Cumulative Merge Latency Curves: Comparing single-domain application PRs (Control) against dual-scope PRs (Treatment) across AIDev v5 ( N=687 pairs) and MOSAIC-3M ( N=80 pairs). Empirical cumulative distribution function (ECDF) curves comparing time-to-merge. In AIDev v5, the dual-scope curve shows a distinct horizontal shift to the right, reflecting the 13.13-hour median latency penalty. In MOSAIC-3M, the dual-scope curve rapidly reaches 50 percent within the first 31 minutes due to unreviewed silent merges, crossing the control curve.
Model / Factor
PR Count
Time Ratio [95% CI]
p -value
Model 1: Global Treatment Effect
AIDev v5 Dual-Scope
687
3.237 [2.545, 4.116]
<0.001
MOSAIC-3M Dual-Scope
80
0.794 [0.377, 1.672]
0.544
Model 2: Subdomain Decomposition
CI/CD Manifests
532
3.827 [2.918, 5.020]
<0.001
System Provisioning
61
2.727 [1.315, 5.655]
0.0070
Table 3. Log-Normal Parametric Survival Models on PR Merge Latency Log-Normal AFT survival models estimating the time-to-merge multiplier for dual-scope PRs. Model 1 compares the global treatment effect across AIDev v5 and MOSAIC-3M; Model 2 decomposes the treatment effect into four infrastructure subdomains in AIDev v5, demonstrating that CI/CD manifests and system provisioning drive statistically significant integration delays.
PR Scope
Formal Review (%)
Silent Merge (%)
Silent OR [95% CI]
p -value
MOSAIC-3M Human ( N=79 )
Application-Only
72.2
21.5
1.00 (Ref)
—
Dual-Scope
73.4
19.0
0.85 [0.38, 1.89]
0.839
MOSAIC-3M Agents ( N=80 )
Application-Only
52.5
42.5
1.00 (Ref)
—
Dual-Scope
36.2
56.2
2.38 [1.04, 5.43]
0.052
Table 4. Cross-Sectional Interaction: The Intervention Bifurcation Table comparing external review intervention rates and silent integration rates across Human Baseline, MOSAIC-3M Agentic, and AIDev v5 Agentic cohorts, including McNemar Odds Ratios.