cs.SEOct 9, 2026

Co-Modifying Application Code and Deployment Infrastructure: A Causal Study of AI Agent Pull Requests

Authors: Mohammad Wathiq Soualhi, Basem Suleiman

Organizations: Independent Researcher, Malaysia · University of New South Wales, Australia

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Current software engineering agents are evaluated predominantly inside isolated sandboxes against synchronous local tests. However, deploying features in collaborative repositories requires modifying continuous integration (CI/CD) pipelines, containers, and provisioning manifests alongside application logic. Because agents lack deployment awareness, attempting cross-domain modifications introduces unverified dependencies into asynchronous integration pipelines. To isolate the causal integration overhead of dual-scope pull requests (PRs)-modifying both application code and deployment infrastructure-we evaluated 26,999 PRs across two large datasets (AIDev v5 and MOSAIC-3M) encompassing six agent architectures and an aligned human baseline. Using a deterministic four-tier file taxonomy and an exact 4D strata-matching framework, we isolated integration metrics under conditional exchangeability while controlling for application volume. We find agents operate primarily as application coders or chore workers, with maintenance tasks capturing 17.70% to 25.41% of agent PRs. Dual-scope modifications represent only 4.48% to 4.55% of agentic PRs, yet trigger bimodal integration outcomes. In the first mode, explicitly reviewed contributions experience protracted delays, adding a median penalty of +13.13 hours and multiplying time-to-merge by 3.24x, driven mechanically by CI/CD and provisioning manifests. In the second mode, maintainer gatekeeping exhibits a structural bypass: even for external contributors lacking repository write access, agent identity reduces the odds of human review by 63.7% compared to human peers. Consequently, 43.1% to 56.2% of dual-scope PRs bypass inspection to merge silently in a median of under 31 minutes. These uninspected modifications compromise build verification, demonstrating the need for deployment-aware benchmarks and strict path-based access controls.

Figures & tables

Dimension / Pipeline StageAIDev v5 ( pop )MOSAIC-3M
A. Dataset Scope & Repository-Level Curation
Native Collection TimelineJan 2025 – Mar 2026Jun 2025 – Aug 2025
Profiled Agent ArchitecturesCodex, Claude, Copilot, Cursor, Devin, JulesCodex, Claude, Copilot, Devin, Jules
Human BaselineNoneIncluded
Repository Popularity Threshold>100⋆>100\star (Inherent to pop )>100⋆>100\star (Applied filter)
Curated Repositories ( >100⋆>100\star )25,4024,553 (of 50,515)
Table 1. Cross-Dataset Structural Parity and Cohort Funnel Matrix Table mapping dataset provenance, structural boundaries, experimental controls, and the progressive cohort filtering funnel from raw PRs down to the cleaned evaluated populations across the AIDev-pop subset and MOSAIC-3M.
Figure 1. Operational Landscape of Agentic Pull Requests: (a) Macro PR scope distribution across cleaned cohorts; (b) Infrastructure subdomains modified within dual-scope PRs (non-mutually exclusive). Two-panel visualization: Panel (a) is a grouped horizontal bar chart comparing AIDev v5 and MOSAIC-3M across macro PR scopes, showing application-only PRs at 63 to 70 percent, dual-scope PRs at roughly 4.5 percent, and maintenance chores at 21 to 28 percent. Panel (b) shows dual-scope subdomains, dominated by CI/CD manifests at 72 to 77 percent, followed by Containers and Automation (15 to 20 percent each), and Provisioning (7 to 10 percent).
 AIDev v5MOSAIC-3M
Author / ArchitectureTotal PRsDual-ScopeRate (%)Total PRsDual-ScopeRate (%)
Claude Code2915217.87% ∗903556.09%
Copilot2,67541315.44%505234.55%
Devin7519112.12%858333.85%
Google Jules27311.11%2613.85%
Cursor1,1761038.76%Not Tracked
Table 2. Cross-Dataset Agent-Level Dual-Scope Prevalence (PRs Created Jun–Jul 2025) Table comparing dual-scope PR prevalence across agent architectures side-by-side between AIDev v5 and MOSAIC-3M cohorts, reporting total PRs, dual-scope counts, and percentage prevalence.
Figure 2. Empirical Cumulative Merge Latency Curves: Comparing single-domain application PRs (Control) against dual-scope PRs (Treatment) across AIDev v5 ( N=687N=687 pairs) and MOSAIC-3M ( N=80N=80 pairs). Empirical cumulative distribution function (ECDF) curves comparing time-to-merge. In AIDev v5, the dual-scope curve shows a distinct horizontal shift to the right, reflecting the 13.13-hour median latency penalty. In MOSAIC-3M, the dual-scope curve rapidly reaches 50 percent within the first 31 minutes due to unreviewed silent merges, crossing the control curve.
Model / FactorPR CountTime Ratio [95% CI]pp -value
Model 1: Global Treatment Effect
AIDev v5 Dual-Scope6873.237 [2.545, 4.116]<0.001<0.001
MOSAIC-3M Dual-Scope800.794 [0.377, 1.672]0.5440.544
Model 2: Subdomain Decomposition
CI/CD Manifests5323.827 [2.918, 5.020]<0.001<0.001
System Provisioning612.727 [1.315, 5.655]0.00700.0070
Table 3. Log-Normal Parametric Survival Models on PR Merge Latency Log-Normal AFT survival models estimating the time-to-merge multiplier for dual-scope PRs. Model 1 compares the global treatment effect across AIDev v5 and MOSAIC-3M; Model 2 decomposes the treatment effect into four infrastructure subdomains in AIDev v5, demonstrating that CI/CD manifests and system provisioning drive statistically significant integration delays.
PR ScopeFormal Review (%)Silent Merge (%)Silent OR [95% CI]pp -value
MOSAIC-3M Human ( N=79N=79 )
Application-Only72.221.51.00 (Ref)—
Dual-Scope73.419.00.85 [0.38, 1.89]0.839
MOSAIC-3M Agents ( N=80N=80 )
Application-Only52.542.51.00 (Ref)—
Dual-Scope36.256.22.38 [1.04, 5.43]0.052
Table 4. Cross-Sectional Interaction: The Intervention Bifurcation Table comparing external review intervention rates and silent integration rates across Human Baseline, MOSAIC-3M Agentic, and AIDev v5 Agentic cohorts, including McNemar Odds Ratios.
More in this topic

Software Engineering Agents

Software Engineering Agents are AI systems that use code, tools, and feedback to perform software development tasks.

See all 347 papers

Explore similar work

CardsList
Feb 23, 2026

When AI Teammates Meet Code Review: Collaboration Signals Shaping the Integration of Agent-Authored Pull Requests

Autonomous coding agents increasingly contribute to software development by submitting pull requests on GitHub; yet, little is known about how these contributions integrate into human-driven review workflows. We present a large empirical study of agent-authored pull requests using the public AIDev dataset, examining integration outcomes, resolution speed, and review-time collaboration signals. Using logistic regression with repository-clustered standard errors, we find that reviewer engagement has the strongest correlation with successful integration, whereas larger change sizes and coordination-disrupting actions, such as force pushes, are associated with a lower likelihood of merging. In contrast, iteration intensity alone provides limited explanatory power once collaboration signals are considered. A qualitative analysis further shows that successful integration occurs when agents engage in actionable review loops that converge toward reviewer expectations. Overall, our results highlight that the effective integration of agent-authored pull requests depends not only on code quality but also on alignment with established review and coordination practices.

Costain Nachuma, Minhaz Zibran

Idaho State University, USA

AI Coding AgentsAI-Assisted Software DevelopmentHuman-AI Collaboration
AskPDF
Jun 21, 2026

Beyond Simpson's Paradox: A Cascade of Confounders in AI Agent Pull-Request Co-Authorship

Pooled across five AI coding agents, pull requests (PRs) with a human Co-Authored-By trailer merge less often than purely-autonomous ones (53.8% vs. 79.8%) -- yet this aggregate finding is a textbook Simpson's Paradox. Stratifying 33,596 PRs from the AIDev dataset by agent identity reverses the conclusion: Copilot and Devin show large positive within-agent gaps (+41.2 and +33.5 pp, both p<0.001), while Cursor, Claude Code, and Codex show small effects whose cross-sectional 95% CIs span zero. The paradox is driven entirely by agent composition: Codex, which dominates 64.9% of the dataset, achieves high merge rates while rarely using co-authorship. But Simpson's Paradox is only the first layer of a cascade of confounders: within-repo controls eliminate Devin's gap (+33.5 to +1.6 pp, p=0.73); a commit-count control further halves Copilot's within-repo gap (+36.2 to +24.4 pp); restricted to multi-commit PRs, the Copilot within-repo effect dissolves to +4.8 pp (p=0.59). No agent retains a clear co-authorship effect once both repository selection and PR structure are controlled. Our findings caution against reporting agent-pooled statistics without stratification and demonstrate that cross-sectional co-authorship associations are largely selection and PR-structure artefacts rather than evidence of a causal benefit.

Haoran Yu, Xiaochong Jiang, Lifei Liu +3

Independent Researcher Seattle, WA, USA · Carnegie Mellon University Pittsburgh, PA, USA · Georgia Institute of Technology Atlanta, GA, USA

Software EngineeringAI Coding AgentsCausal ReasoningHuman-AI CollaborationMining Software Repositories
AskPDF
Mar 29, 2026

Safer Builders, Risky Maintainers: A Comparative Study of Breaking Changes in Human vs Agentic PRs

AI coding agents are increasingly integrated into modern software engineering workflows, actively collaborating with human developers to create pull requests (PRs) in open-source repositories. Although coding agents improve developer productivity, they often generate code with more bugs and security issues than human-authored code. While human-authored PRs often break backward compatibility, leading to breaking changes, the potential for agentic PRs to introduce breaking changes remains underexplored. The goal of this paper is to help developers and researchers evaluate the reliability of AI-generated PRs by examining the frequency and task contexts in which AI agents introduce breaking changes. We conduct a comparative analysis of 7,191 agent-generated PRs with 1402 human-authored PRs from Python repositories in the AIDev dataset. We develop a tool that analyzes code changes in commits corresponding to the agentic PRs and leverages an abstract syntax tree (AST) based analysis to detect potential breaking changes. Our findings show that AI agents introduce fewer breaking changes overall than humans (3.45% vs. 7.40%) in code generation tasks. However, agents exhibit substantially higher risk during maintenance tasks, with refactoring and chore changes introducing breaking changes at rates of 6.72% and 9.35%, respectively. We also identify a "Confidence Trap" where highly confident agentic PRs still introduce breaking changes, indicating the need for stricter review during maintenance oriented changes regardless of reported confidence score.

K M Ferdous, Dipayan Banik, Kowshik Chowdhury +1

Kennesaw State University Marietta, Georgia, USA · Quanta Technology Raleigh, North Carolina, USA

AI Coding AgentsLLM Agent Reliability
AskPDF